How to build an AI agent that can prove its work.

The first design decision is not which framework to install. It is whether the work needs an agent at all. OpenAI's practical guide describes agents as systems that independently accomplish tasks using a language model to manage workflow execution and tools to interact with external systems. That distinction matters because a single-turn chatbot and a fixed automation have different failure surfaces.

I use a simple test. If the workflow is a fixed sequence with predictable inputs, stable rules, and low maintenance cost, use ordinary software or automation first. If the work requires interpreting documents, choosing among possible next steps, handling exceptions, or interacting with a person in natural language, an agent may be justified. The presence of an LLM is not itself a reason to introduce autonomy.

Write down the cost of being wrong. An agent drafting an internal summary can often operate with lightweight review. An agent issuing refunds, editing medical records, approving a loan, changing production code, or sending a legally meaningful message needs a much stronger control design. The consequence determines how much autonomy, evidence, and human involvement are acceptable.

Choose a workflow with a visible end state. 'Give better advice' is difficult to test. 'Create a support-ticket draft with the correct category, relevant policy citation, proposed reply, and escalation flag' is testable. A visible state lets the application verify completion instead of trusting the agent's final sentence.

Before you write instructions, create a one-page behavior contract. Name the user, the trigger, the inputs, the output, the systems it may access, the actions it may take, and the point at which a human must approve. State the conditions under which it must ask a question, stop, or hand off.

Separate goals from constraints. The goal might be to prepare a complete answer. Constraints might require using only an approved knowledge base, hiding personal data from a third-party tool, never inventing a policy, and escalating when the customer cannot be authenticated. Constraints should be enforced by application code where possible, then reinforced in the instructions.

Define a budget. Set a maximum number of tool calls, a time limit, a token or API spend limit, and a retry policy. A loop that keeps searching is not diligence; it is an unbounded operating cost. A stop condition can be as simple as 'after two failed retrieval attempts, summarize what is missing and ask for a human.'

Create a known-good example and a known-bad example. The good example makes the intended result concrete. The bad example shows a tempting but prohibited behavior, such as taking an action before confirmation or treating an unverified document as policy. These examples become part of both the instructions and the evaluation set.

Tools are the agent's interface to reality. OpenAI groups them broadly into data tools that retrieve context and action tools that change an external system. Keep that distinction visible in your design. Read-only tools can often be available earlier; write tools should carry a higher permission and approval threshold.

Give every tool a single job. 'Manage customer account' is an ambiguous tool. 'Look up an account by verified account ID' and 'create a refund request up to the approved limit' are easier to describe, authorize, test, and audit. A small toolset with little overlap makes tool selection clearer and reduces the number of unsafe combinations.

Define schemas that reject malformed arguments. Validate identifiers, ranges, enum values, required fields, and ownership on the server. The agent's tool call is a request, not proof that the caller is allowed to perform the action. Re-check authentication and authorization in the tool service, especially when the agent can act for more than one user or tenant.

Return useful, compact errors. 'Request failed' forces the agent to guess. 'Account not found; do not retry with a different customer ID; ask the user to confirm the ID' gives the system a safe next step. Do not expose secrets, internal stack traces, or private records in tool errors.

Document when a tool should not be used. Anthropic's context-engineering guidance emphasizes that tool descriptions and tool sets shape the model's available action space. If a human cannot tell which of two overlapping tools is appropriate, an agent will struggle too. Remove redundant tools before adding another instruction paragraph.

Start with the best existing procedure, support script, checklist, or policy. OpenAI recommends using existing documents to create routines, then breaking dense material into smaller, actionable steps. This is usually more reliable than asking a blank model to invent a process from a vague description.

Use a predictable structure: role and purpose, inputs, decision rules, tool guidance, output format, edge cases, escalation, and stopping condition. Keep each instruction at the right level. Overly rigid if-then prose can become brittle; overly general advice leaves the model without a decision signal. Test the minimum clear instruction set, then add text only in response to a documented failure.

Context engineering is broader than prompt writing. The agent may receive system instructions, tool definitions, retrieved documents, conversation history, user data, and intermediate results. Anthropic describes context as a finite resource with diminishing returns when irrelevant material accumulates. Curate the smallest high-signal context for the current decision instead of dumping an entire knowledge base into every turn.

Use examples to show judgment, not to create a giant list of exceptions. Include a normal case, an ambiguous case, a refusal or escalation case, and a case where a tool should not be called. Examples should be representative and maintained like code; stale examples can quietly teach the wrong policy.

A single agent with a clear toolset is the easiest architecture to understand and evaluate. It can receive the task, decide which tool to use, inspect the result, and complete or escalate. Begin here unless the workflow has a strong reason to separate responsibilities.

Split work when the context, permissions, or expertise are genuinely different. A research specialist may search and return a compact evidence packet while a writing agent drafts from that packet. A customer-facing agent may hand a sensitive action to a specialist with narrower permissions. The point of multiple agents is separation of concern, not a larger diagram.

Every handoff needs an explicit contract: what the specialist receives, what it must return, what evidence is attached, and who remains accountable. Handoffs can lose important context, duplicate work, or create loops. Log them as part of the trajectory and add tests for incomplete and conflicting handoff data.

A fixed workflow can be a better choice than agentic orchestration. If the sequence is known but individual steps use language models, define that sequence in application code and use the model within bounded steps. Let the model choose only where flexibility creates value.

A model instruction can explain a boundary, but it cannot be the only boundary. Apply controls before data reaches the model, while the model chooses a tool, and after the tool returns. Filter sensitive fields, restrict retrieval, validate actions, and check the result before writing it to a system of record.

Use least privilege. Create separate credentials for the agent, limit the records and operations it can access, and avoid giving a drafting assistant the ability to send or delete. For consequential actions, require a human confirmation that shows the proposed action and the relevant evidence. Make the confirmation meaningful rather than a blanket 'allow all' button.

Plan for prompt injection and untrusted content. Documents, webpages, emails, and user messages may contain instructions that conflict with the agent's task. Mark external content as data, keep authority rules outside retrieved text, restrict tool permissions, and test malicious instructions. Do not let a retrieved document grant itself permission to call a tool.

Add observability from the beginning. Record the request identifier, model and instruction version, tools called, arguments after redaction, results, approval decisions, errors, duration, and cost. This supports debugging, incident response, and evaluation. Logging everything without a retention and privacy plan creates a different risk, so define access and retention too.

An agent evaluation is a task, a trial, a grader, and evidence of what happened. Test the final outcome, but also inspect the trajectory: whether the right tool was selected, whether arguments were valid, whether the source was authorized, whether the agent repeated a failed action, and whether it stopped when it should.

Build the first suite from 20 to 50 realistic tasks. Include common work, edge cases, incomplete inputs, conflicting instructions, tool failures, adversarial content, and cases where the correct behavior is to ask or refuse. Run multiple trials when the agent is nondeterministic. Separate capability tests, which explore what it can do, from regression tests, which protect behavior that already works.

Use deterministic graders for exact fields, permissions, schemas, unit tests, and final state. Use rubric-based model graders for qualities such as groundedness or completeness only when the rubric is explicit and calibrated against human review. For high-impact workflows, keep expert review in the loop. A high score from a badly designed grader is not evidence of a reliable agent.

Connect offline tests to production learning. Sample traces, capture user corrections, investigate failures, and turn confirmed failures into new regression cases. Re-run the suite when changing the model, prompt, tool, retrieval index, policy, or user population. See our [AI evaluation guide](/ai-evaluation) for a detailed evaluation framework.

The first release should make mistakes cheap to correct. Use draft-only output, shadow mode, a small internal group, or human approval for every action. Give users a clear way to correct the result and report a failure. Do not expand simply because the demo looked good; expand when the workflow evidence supports it.

Define launch metrics before collecting them. Track task success, correction rate, escalation quality, tool error rate, unauthorized-action attempts, latency, cost, and user effort. A time-saving metric is useful only if quality and risk remain acceptable. Watch the distribution, not only the average: a small number of severe failures can matter more than many easy successes.

Set a pause rule. The owner should know what failure rate, incident type, vendor change, cost spike, or data issue causes the agent to revert to the human process. Test the pause and fallback path before launch. An agent that cannot be stopped safely is not ready for broader permissions.

After the pilot, make one of four decisions: retire it, keep it as a supervised assistant, expand the user group, or expand the allowed actions. Record why. The next version should be based on the failure evidence and the value actually observed, not on pressure to make the system appear more autonomous.