AI evaluation: how to test agents before users find the failure

Teams can get surprisingly far with manual testing. A builder tries a few prompts, watches the agent complete a task, and adjusts the instructions when something looks wrong. That approach is reasonable for a prototype. It breaks down when the system has many tools, multiple steps, changing models, retrieval, or real users. A change that fixes one visible failure can quietly create a different failure in another workflow. Without evaluations, the team is left with memory and anecdotes.

Anthropic's 2026 guidance describes the value clearly: evaluations make behavior changes visible before they affect users, and their value compounds across the agent lifecycle. They turn failures into reusable test cases. They give product and engineering a shared definition of success. They allow teams to compare a new model or prompt against a baseline without waiting for production complaints. The evaluation suite becomes a measurement instrument for a system whose output is otherwise variable.

An evaluation is not simply a score. It is a task with clear inputs, a defined success condition, a trial, a grader, and a recorded result. For an agent, the transcript and the final environment state both matter. An agent can say a flight was booked while no reservation exists. It can produce a polished answer while citing an unapproved source. It can reach the right answer after taking an unsafe route. The harness must inspect what happened, not only what was said.

The first practical step is to define the behavior before building a sophisticated test system. Write down what a successful run does, what it must never do, and when it should ask for help. If two people interpret the requirement differently, the agent will be evaluated against an unstable target. A small set of precise cases often exposes that product ambiguity faster than another week of prompt tweaking.

Start with the workflow's ordinary cases. If you are evaluating a customer-support agent, include the common billing question, the common cancellation, and the common request for account information. Make the expected outcome observable. Then add cases that represent the conditions under which the system must show judgment: missing identity information, conflicting records, a request outside policy, an angry customer, a prompt injection in an attachment, or a request that needs a human.

Balanced problem sets matter. If every test asks the agent to search the web, it may learn to search unnecessarily. Include questions where search is required and questions that should be answered from an approved internal source. If every test rewards completing a transaction, the system may take action when it should ask for approval. Include negative cases where the correct behavior is to decline, clarify, or stop.

Anthropic recommends beginning with 20 to 50 simple tasks drawn from requirements and real failures. That is a useful starting point because it creates a feedback loop quickly. The task description should make the desired outcome clear, and a known reference solution should pass the graders. If every task has a zero score, the problem may be the specification or harness rather than model capability.

Keep a distinction between a test case and a trial. Model behavior can vary across runs, so one case may need multiple attempts. Store the model, prompt, tools, retrieval version, data snapshot, and environment state for each trial. Otherwise a score from last Tuesday cannot be reproduced when the team asks why it changed. Reproducibility is especially important when a vendor silently updates a model or tool behavior.

Deterministic graders are the strongest option when the desired result is objectively checkable. Use exact matches for an identifier, schema validation for structured output, unit tests for code, policy rules for a threshold, and an environment query for whether a record exists. These graders are fast, repeatable, and easy to debug. They should handle as much of the evaluation as the task permits.

Model-based graders can evaluate qualities that are difficult to reduce to one string: whether a response is grounded in supplied evidence, whether it addresses the user's actual request, or whether its tone is appropriate. They need a rubric with observable criteria and examples. A judge that is asked to 'rate helpfulness' without a definition can introduce a second opaque model into the quality problem.

Human graders remain important for calibration and high-stakes decisions. Use them to establish whether an automated rubric reflects expert judgment, review samples where the judge is uncertain, and assess issues involving fairness, nuance, or harm. Human review is slower and more expensive, but a cheap automated score that measures the wrong thing is not efficient. The goal is an appropriate mixture, not zero humans.

Anthropic recommends combining grader types because no single layer catches every issue. A customer-support task might use an environment check for ticket state, a deterministic check for required fields, a model rubric for clarity, and a human sample for calibration. The combination gives a more honest picture than a single overall score. Report the component scores and the failure examples, not only a blended average.

A single-turn answer can often be graded from the text. An agent works over a trajectory: it receives a goal, calls tools, observes results, modifies state, and decides what to do next. Errors can propagate. A wrong retrieval result can lead to a wrong recommendation, which can lead to a tool call that changes an account. Evaluation must make the trajectory visible and decide which intermediate behaviors are allowed.

For tool use, grade selection and arguments separately. Choosing the right tool with an invalid customer ID is not a success. Using the right query but exposing data outside the user's permission is not a success. Calling a read-only tool before an approval step may be fine; calling a write tool without approval is not. These distinctions are useful because they point to the fix: tool description, schema, authorization, state, or model behavior.

The final environment state is often the strongest signal. Check whether a record was really updated, a test really passed, a file really exists, or a support ticket really moved to the correct status. Do not grade the final sentence alone. A model can say 'completed' after a timeout or hallucinate a reference number. The harness should query the system of record and record the evidence that supports the grade.

Consistency matters more than a lucky success for many customer-facing agents. Anthropic distinguishes pass@k, the chance of getting at least one success in k attempts, from pass^k, the chance that all k trials succeed. A coding assistant may benefit from trying multiple solutions. A support agent that performs a financial action needs a much stronger consistency bar. Choose the metric based on what the user expects and what a failure costs.

Capability evaluations ask, 'Can the agent do this?' They should target useful tasks the system currently finds difficult. A low early pass rate is acceptable because the purpose is to identify the hill the team is trying to climb. Capability cases can graduate into regression cases once the system handles them reliably. They then protect a behavior that the product now depends on.

Regression evaluations ask, 'Does the agent still do what worked before?' Their job is to catch damage from a model upgrade, prompt change, retrieval change, tool refactor, or policy update. A regression suite should include the high-value paths and the edge cases that have caused real incidents. Run it before release and, for important systems, on a schedule against the production configuration.

Do not let a high average hide a critical failure. A system that passes 98% overall may still fail every high-value refund case. Weight tasks by risk or set hard gates for safety and policy checks. Report slices by workflow, customer type, language, tool, and outcome. The right aggregation depends on what the organization promises. A single dashboard number is convenient, but it should never be the only view.

When an evaluation fails, preserve the transcript and classify the failure. Was the task ambiguous? Was the expected answer wrong? Was retrieval incomplete? Did the model choose the wrong tool? Did a guardrail block a legitimate action? Did the grader mis-score a valid solution? Fixing the classification is part of evaluation quality. Otherwise the team may optimize the wrong component and create a new regression.

Offline evaluations are controlled and repeatable. They help a team compare changes before users see them. They cannot cover every future input, source update, integration failure, or new misuse pattern. Production monitoring supplies the missing signal: user feedback, sampled traces, action outcomes, escalations, latency, cost, and drift in the distribution of tasks.

The strongest loop combines automated pre-release tests, production monitoring, periodic transcript review, targeted A/B tests, and human calibration for subjective graders. Monitoring should not collect more sensitive information than necessary. Define retention, access, redaction, and incident procedures alongside the evaluation design. A trace is useful only if the right people can inspect it safely when something goes wrong.

User feedback is not a perfect grader, but it reveals gaps that synthetic tests missed. A customer may say the answer was unhelpful because it technically answered the question but ignored the situation. A worker may report that a system is correct but too slow to use. Add representative failures to the suite after reviewing them. That turns live experience into a durable protection instead of a one-off support ticket.

Evaluation suites also need ownership. Product teams should define task success. Domain experts should contribute cases. Engineering should maintain the harness and trace collection. Security and legal teams should review high-impact boundaries. Someone must decide when a failing result blocks release and who is responsible for updating the suite. Without ownership, evaluations become a report that everyone reads and nobody uses.

In week one, define the workflow and collect 20 to 50 tasks. Include common requests, missing information, cases that require a tool, cases that should not trigger a tool, policy boundaries, and a human handoff. Write a success condition for each. Create a known-good reference outcome where possible. This is the most valuable work because it makes the product requirement concrete.

In week two, build the simplest harness. Run the agent against a fixed environment or data snapshot, capture the complete trace, and add deterministic checks first. Verify final state, required fields, permissions, and tool arguments. Add a small rubric-based model grader only where deterministic checks cannot capture the quality. Manually inspect failures and revise ambiguous tasks before blaming the model.

In week three, add regression protection and repetition. Run multiple trials for variable tasks. Establish a baseline for pass rate, consistency, latency, and cost. Split capability and regression cases. Add negative examples and adversarial inputs. Decide which failures are release blockers. Store the prompt, model, retrieval, tools, and evaluation version with each result.

In week four, connect the suite to the release and production loop. Run it on every meaningful model or prompt change, sample live traces, and schedule human calibration. Add failures from real use. Publish a small scorecard that shows outcome success, safety, quality, cost, and the top failure modes. The purpose is not to create a perfect number. It is to replace guesswork with a shared, improving measurement system.