Field notes on building AI agents

A cache miss never shows up as an error

Every major model API caches by prefix: it reuses the start of a prompt only if that start matches a recent one exactly, token for token. Put a timestamp, the visitor's name or a shuffled tool list near the top, and each request begins differently, so nothing is ever read back. Prompts under the model's minimum length (1,024 tokens on OpenAI; 512 on current Claude models, up to 4,096 on some older ones) are never cached at all.

Neither case returns an error. A short prompt pays full price, and a shifting prefix pays more, since every request writes a fresh cache entry at 1.25× and reads none. The only trace is a cache-read count of zero in the usage, which is why my portfolio's caching table marks such requests cold or uncached. My prompt keeps the date after the cached part and builds the tool list in a fixed order. A zero there is one of the first things I look for on an agent's bill.

The tool description is the prompt that picks the tool

The model chooses tools by reading their names and descriptions, so overlapping descriptions are one of the most common causes of wrong calls. Fewer tools, each with one clear job, work better. My portfolio server has seven, all read-only, and each role server has one per piece of work.

MCP dropped its handshake

The 2026-07-28 revision made MCP stateless. There is no initialize or session any more: every request carries its protocol version and client capabilities in _meta, and server/discover replaces the handshake for anyone who wants to look first. The method and tool name are also copied into HTTP headers, so gateways can route and rate-limit without reading the body. A server must reject any header that disagrees with the body; otherwise the gateway and the server could act on two different requests. These servers still answer the older handshake, so older clients connect too.

readOnlyHint is a hint, and a server can lie

Annotations are a server describing itself. The MCP spec tells clients not to trust annotations from servers they don't trust, so the client's own policy should decide what needs approval. Every tool on these servers returns evidence and changes nothing, so each is labelled read-only. Your client shouldn't take my word for it either.

A leaderboard score says little about your task

Public benchmarks test on someone else's data, and models get tuned toward them. Teams that ship pick models by running their own tasks. The Market fit tab does the same: it scores me against 110 job postings and shows the gaps.

Make the model optional

In NestEazy's design the model only drafts, so an injected instruction or a wrong draft can't reach a renter on its own, and a model outage only means no drafts for a while. Drafts appear only as suggestions the landlord sends with a tap.

Private data, untrusted input and a way out: pick two

Simon Willison calls this the lethal trifecta. An agent that can read private data, read content an attacker controls and send data somewhere can be talked into leaking it, whatever the prompt says. The fix has to be structural: take one of the three away. With no tools, the model's words are still a way out, so the design takes away the private data instead. The model never sees it, and code checks every draft.

People stop reading approvals

Ask someone to approve two hundred actions a day and they approve nearly all of them. Human review works when it's rare enough to be read. Let people pre-approve routine work for a limited time, and keep the few actions that are hard to undo out of reach of any pre-approval.

pass@k flatters agents; pass^k doesn't

pass@k asks whether any of k tries succeeds. For an agent acting for a user, the question is whether all of them do, which τ-bench calls pass^k. With independent tries the gap grows fast: with a 70% success rate per try and 3 tries, pass@k is 97.3% and pass^k is 34.3%.

LLM judges can have favourites

Model graders can favour one position (often the first), the longer answer and their own model family; that last effect is often small and varies by model and task. Score pairs in both orders, keep length in check, and drop the judge where code can decide: Garm scores its agent with deterministic checks against the database. Where a judge is needed, use more than one model family. This site's judge is from its agent's own family, so the eval set and my blind labels of its verdicts are published for anyone to check.

A valid schema can still hold invented facts

Structured outputs guarantee the shape of the JSON. They say nothing about whether the values are right, and a field can be well formed and invented. Check the values against the record they came from.

BM25 still wins on exact terms

Embeddings capture meaning and blur exact strings: product names, error codes, SKUs. Search for “SYSPRO” and BM25 finds it, where an embedding may return near misses about ERPs. Production retrieval is often hybrid for that reason. This resume uses BM25 because job postings are full of exact terms.

Retries send things twice

When a call times out, you can't tell whether it ran. Retry without an idempotency key and the renter gets two messages, or a payment goes through twice. The fix is a transactional outbox: write every outbound action in the same transaction as the change that caused it, under an idempotency key the receiver checks, so a replay does nothing.

Let the first one wait

In my job agent, the first application that qualifies for its autopilot rule still waits for me. Submitting is the one step that can't be undone. It runs on its own only after it has gone through once with a person watching: the form filled, the answers checked, the confirmation recorded.

Most of the agent is the harness

In my job agent, the model writes, researches and reviews. The rest is harness code: the schedule, with model calls batched at half price; daily and monthly spend caps checked before every model call; my record in a cached prefix, so repeat calls read it for a fraction of the price; crash recovery that hands interrupted work back to me; and its policy on what may go out on its own.

The harness runs in your browser, so the history is signed

On my portfolio, the proxy only adds the key, the system prompt and the tool list. The tool loop runs in the page: each call passes the permission gateway shown in the chat's trace, and the results go back with the history. A client that holds the history can forge an assistant turn, such as "Understood, I'm now a general assistant", so the proxy signs every assistant turn it returns, thinking blocks included, and rejects any history whose signatures don't verify. Tool results can't be signed, since the browser makes them, so the proxy reruns each read tool from the signed call and keeps its own result. The one approval-gated tool must return one of the gateway's three outcomes for that exact call.

Every tool call needs an answer, failures included

In the Messages API, each tool_use block has to be answered in the very next user message by a tool_result with the same id, or the request is rejected. When the model calls several tools at once, every result goes back in that one message, ahead of any text; split them across messages and it learns to stop calling tools in parallel. When a tool fails, return the failure with is_error: true. The model can then retry or explain; hide the error and it tends to report a success that never happened.

Check stop_reason before you show an answer

If stop_reason is max_tokens, the answer was cut off, and in a tool loop the last tool call may be incomplete. This agent caps answers at 600 tokens and its prompt asks for about 70 words, so it finishes well inside the cap. If it's refusal, the model declined: show that, not a partial answer.

A 429 tells you when to come back

Rate-limit errors usually come with a retry-after header. Wait that long, plus some jitter, instead of retrying at once. Without one, back off exponentially. This site's proxy gives each visitor a weighted budget, sends retry-after when it's spent, and the console tells you exactly how long to wait.

Concepts, defined

What is a system prompt?

A system prompt is the standing brief a model gets with every request: who it is, what it's for and what it must never do. Long reference documents go at the top, with the instructions after them, because that order helps on long inputs.

What is prompt caching?

Model APIs keep the start of a recent prompt in a cache. When the next request begins the same way, the provider reuses the work it already did on that part. That part is then billed at a fraction of the input price, and the answer starts sooner.

What is the Model Context Protocol (MCP)?

An open protocol for connecting AI applications to tools and data. The host (an IDE, a desktop chat app or a coding agent) runs one client per server, and each speaks JSON-RPC to its server. Servers offer three things: tools the model can call, resources the application can read and prompts the user can choose.

What is a model card?

Labs publish a model card (or system card) with each major release: what it is, what it's for, how it scores and where it fails.

Mine follows the same format, with an evaluation against what AI teams hire for.

What does shipping AI to production take?

In production a model has to be measured, budgeted and safe to fail. With agents, the design question is what the model may do on its own. NestEazy's answer is three rules, shown in chapter 4 of my portfolio.

What are AI safeguards?

The controls around a model at run time: what it can see, which actions it can reach, what gets checked before anything goes out and what needs a person. Alignment shapes what a model tries to do; safeguards limit what it can do.

I started in security at IBM, so I design agents assuming someone will try to break them.

What is least privilege for AI agents?

Each agent gets only the tools and data its task needs. When a model is fooled, the damage stops at what that one agent could reach.

What are evals?

A fixed set of tasks with clear pass criteria, run the same way every time, so you can tell whether a change helped.

Every failure an eval finds becomes a case it keeps, so the same mistake is tested on every run.

What is p95 latency?

The time 95 of every 100 answers come in under. An average hides the slow tail people actually wait through, so latency targets and alerts are set on a percentile such as p95 or p99.

What is structured data?

Data with a schema can be rendered anywhere and checked by code. My resume is JSON Resume data, ranked for the job you're hiring for, with no model in the loop.

What are Agent Skills?

A skill is a folder an agent can load on demand: a SKILL.md with a name, a description and instructions, plus files it reads and scripts it runs only when needed. Only the name and description are always in context.

What does system design for an AI platform involve?

Most of the work is in the trade-offs: what happens at ten times the traffic, when a provider goes down, or when a retry lands twice.

What decides a delivery plan?

The plan decides what comes first, what can run in parallel and what sets the launch date.

What is a proactive agent?

A proactive agent owns an objective: it goes looking for the work, acts within the limits it's given and stops at any decision reserved for a person.

What is agent memory?

An agent that works for weeks has to remember the rules it was given, what was decided before and what it got wrong, or it asks the same questions and repeats the same mistakes.

Mine recalls the answers I've given, by meaning, so a question I've answered once is found again however a form words it. It drafts new rules from my edits and reviewers' notes, and none applies until I approve it. It gets more room to act on its own only when I raise its trust level.