How this site's AI agent is evaluated
The agent on my portfolio speaks for me, so it's graded like a model, misses included. These are its published results, how each was measured, and what the eval caught.
Results
Run on 2026-10-28 against the production model, claude-sonnet-5-5, through the same handler visitors use.
| Measure | Result |
|---|---|
| Agent eval, pass@1 | 94.9%: 168 of 177 answers (59 questions, 3 runs each) |
| Every run passed, pass^3 | 52 of 59 questions |
| Held-out questions | 75.6% on 15 questions written without sight of the prompt and never used to tune it |
| Latency and cost | p95 4.6 s · $0.006 an answer |
| Red team | 0 of 88 attacks changed what the agent did, end to end (2026-10-28) |
| Usage limits | 12 of 12 abuse and cost scenarios contained (2026-10-29) |
| Fit scorer | 75.5% agreement with model-written hiring reviews on 53 fresh postings; 52.8% on skills alone |
How the agent eval works
Each case runs through the real /api/chat handler with the page's tool loop. Code checks length, required and forbidden content, tool calls, links and every number against the record; a judge model then grades grounding and the case's criterion against the agent's documents. Every failure an eval finds becomes a case it keeps, so the same mistake is tested on every run.
Each question runs 3 times. pass@1 is the share of single answers that pass; pass^3 counts the questions that passed all 3 times, which is the number that matters for an agent acting for a user. Why pass^k, not pass@k.
Checking the judge
The judge, claude-opus-5-5, is from the agent's own model family, and a judge can favour its own family's answers. So I labelled 31 recorded answers blind, with the model and the judge's verdict hidden, weighted toward answers the judge failed. My verdict matched the judge's on 61.3% of them (Cohen's κ 0.13). The labels are published with the results, so anyone can check both of us.
What the eval caught
The most recent of 34 failures it found, each with its fix:
- Location mattered to the verdict after all: he ranks remote, then Canada, then the US, and remote before hybrid before onsite, and a skills-only verdict ignored that. fit eval v3: location order added (a US hybrid or onsite role drops one step; outside Canada and the US, not remote, is not a fit), reviewers record where the work is and the same code applies the order to both sides, and 53 new fresh postings measured once: 75.5% overall, 52.8% on skills alone. A trained ordinal model was tried and didn't beat the rules on unseen postings
- The fit labels mixed two questions: 65 of 96 reviewer rejects cited location, which the scorer leaves out by design, so it was graded against a different question. The held-out set had also been read while the scorer was being changed. fit eval v2: every posting re-reviewed under a written protocol (skills and experience only), one rule added on the 153 old postings (a title's specialty, required and missing), and 58 fresh postings measured once after that. Held-out agreement fell from the 63.4% shown before to 56.9%, which the page showed until fit eval v3 replaced it
- Real postings read through ?for= links showed the fit scorer's section reader missing headings ("What you need", "What we require") and counting scam warnings and tech-stack lines as requirements; clearance wording such as "willing to obtain" blocked two roles. fit scorer: a wider section reader, boilerplate dropped before matching, clearance read by sentence, six specialist requirement terms, a lighter penalty for solutions titles. Held-out agreement fell from 70.7% to 63.4% because the score curve was calibrated on the old reader; tuning set 71% to 74%. The 41 held-out results were seen while making these changes, so the next refresh adds fresh postings
- A held-out set of 15 questions, written by a separate agent without sight of the prompt or these cases, scored Sonnet 60.0% on first grading: the judge treated every fact a criterion listed as required, which no answer within 120 words can hold. grader: one rule for all 15 criteria (listed facts are examples; only what a criterion says the answer must or must not do is required), and the saved answers re-judged with nothing rerun. What is left is real: long answers to detail-heavy questions, and a five-row table cap that can't hold six jobs, fixed through a new case in the tuned set
- Across two runs, answers called The Cabin Depot's installer partners "enterprise clients" and credited SQL work to The Cabin Depot. the record's list of easily confused facts now says the partners are installer partners, and that SQL work belongs to Geoffrey Stephens
- Asked about relocating, one answer in three left out that he is also open to remote or hybrid work; asked what is still in development, one said "two things" and then listed more. the prompt now says to name remote and hybrid alongside relocation, and to list work in development without a count. The first version of that rule led one answer to infer that a remote role avoids US sponsorship, so it now keeps visa answers to what the record says
- The list check allowed at most eight bullets, but he has used nine languages, so a complete answer failed. the check allows up to 10, the same as the prompt
- Fit answers for roles with two sides (agents for enterprise clients, AI security) kept running past 100 words, because the evidence they need doesn't fit. the limit is now 120 words, in the prompt and the grader alike, so a full fit answer fits
All 34 are in agent-eval.json.
The data
- agent-eval.json: results by model and category, judge agreement, calibration and every caught failure
- agent-cases.json: every question, its code checks, the judge's criterion and the judge's prompt
- redteam.json: the red-team attacks and what each did
- shield.json: the usage-limit scenarios
The same approach runs on Garm, my open agent-security benchmark, which is scored by deterministic checks against the database instead of a judge.