How I evaluate agents

An agent can give a fluent answer and still fail the task, so a single accuracy number is the wrong place to start. For each task I would want four things, reported separately:

• Task success. Did the run do what was asked, judged against an expected outcome written down before the run, not against whether the answer sounds right?
• Groundedness. Is each claim supported by context the agent retrieved or a tool result it actually received? A correct answer with no support is a lucky answer.
• Tool-call errors. Wrong tool, wrong or malformed arguments, and failed calls the agent quietly ignored.
• Latency and cost per successful task. Totals divided by the tasks that succeeded, not by all runs. An agent that is cheap per call but needs retries to finish is not cheap.

A single number also hides where the errors fall. In the evaluation lab, at the default threshold, model B has the higher F1 overall, 0.803 against 0.795. On the IMDb slice it is the less accurate of the two, 0.750 against 0.775. I evaluated a skill-extraction model on 2,400+ records with precision, recall, F1 and slice analysis, not one score.

A threshold is a decision, not a default. Raising it trades recall for precision, and the right point depends on what a false positive costs. The lab’s slider shows the trade on the test set, but a threshold you deploy should come from validation data. For an agent, the same choice decides when it acts on its own and when it stops and asks.

A small gap is often noise. In the lab, B leads A by 0.008 F1 on 596 test sentences. Across 1,000 bootstrap resamples of the test set, the interval for that gap runs from −0.021 to +0.038, and B comes out ahead in 70% of them. The interval crosses zero, so a 0.008 F1 gap is not a win, and I would not switch models on it. A hand-built agent test set with fewer cases gives a wider interval, so it needs one even more.

Traces come first. I set up LLM tracing and evaluation with Arize AX and OpenTelemetry/OpenInference. In OpenTelemetry terms a run is a trace and each step inside it is a span; OpenInference adds conventions for LLM steps such as model calls, retrievals and tool calls. When spans record inputs, outputs, timing and errors, a failed run shows which step went wrong instead of leaving you guessing from the final answer.

Then put each failure in one bucket:

• Wrong tool. The agent called a tool that cannot answer the question, or answered without the tool it needed.
• Bad arguments. The right tool with the wrong parameters or format.
• Missing evidence. The answer states something that no span supports.
• Timeouts and infrastructure. The call never had a fair chance. Some agent integration failures I have debugged were of this kind and had nothing to do with the model: MCP IAM request signing, OAuth credentials, missing deployment secrets and network rules.

The exporter behind these figures does a rough version of this step from the error text alone: the 162 failed calls split into not found 48, permission 44, timeout 30, other 20, network 16 and bad arguments 4. It cannot see a wrong tool or missing evidence. The dashboard’s failure analysis splits the classes by tool.

Each classified failure should become a case in a small, fixed test set with an expected outcome. If the set does not change between runs, a moving score means the agent changed. Next to it, keep a failure log: the trace, the bucket, the fix, and the test that now covers it. Learning from what went wrong also shaped the onboarding guide for the workshops I co-delivered. Each workshop showed where developers got stuck, such as missing secrets, access groups and network rules, and the guide changed to match. It has been revised more than 40 times.

The recorded run that built this site shows the pattern on a small scale. Of its 52 steps, 8 ended in an error, and 6 of those are followed by a recorded retry or fix. Two were delegated tasks that stopped midway, one interrupted and one after a lost connection, and both were resumed. One was a measurement: the first attempt to hold a figure on screen longer overshot, 6.4 s against 3.8 s before, and the saved version settled at about 5.2 s. The dependency download never cleared: it kept failing with a 503, and the trace records the normal full test command as blocked. Separately, a grounding test caught two homepage labels that the text did not support, and both were reworded before the commit.

Each handoff between agents costs something. The receiving agent gets a summary instead of the full context, and a task that dies midway has to be handed off again. So start with one agent, and add a specialist only when it does a job the first one does badly and has a clear input and output.

In the build trace, one orchestrator, my primary agent working under my direction, recorded 37 of the 52 steps and made all 16 commits. The implementation-worker recorded 9, the quality-reviewer 4 and the ui-design-director 2. Both tasks that stopped midway and left partial edits behind were delegated worker tasks.

The specialist reviewers added value at review points. The quality-reviewer’s first pass found frozen review notes, blank waits and copy hidden on short screens. A later pass on the integrated changes found no high-severity issues and six smaller findings. A second pass on the security changes reported five low-priority findings, all fixed before saving. The ui-design-director’s review found tiny text, crowded labels and uneven tour pacing. A reviewer with fresh context reads the change, not the conversation that produced it.

There is no second run with more or fewer agents, so the trace cannot tell you whether this split beat one agent doing everything. It counts role types, not separate agent instances. Most times are estimates, so it says nothing about latency or cost per role. And it covers one project, the Nov 10–11 redesign only.

My conclusion is narrow. Keep one agent in charge of integration and commits, and add specialists where an independent check is worth more than the handoff. The multi-agent workflow I built is set up that way: one subagent implements, one reviews, one verifies, and they get read-only access to delivery tools.