Agent Harness vs Eval Harness: What Each Does and How to Test Both

What Is an Agent Harness?

An agent harness is the code around a language model that turns it into an agent. Anthropic's engineering team defines it as "the system that enables a model to act as an agent" in Demystifying evals for AI agents, and also calls it a scaffold. In a running agent, the harness owns these jobs:

• Prompt assembly - system instructions, retrieved context, memory, and the conversation so far, combined into each model call.
• Tool dispatch - parsing tool calls, validating arguments, running the tool, and returning results to the model.
• Permissions - which tools run without approval, which need a human, and which are blocked.
• Context management - deciding what stays in the window and what gets compacted when a long task fills it.
• Loop control - when to call the model again, when to stop, how many retries to allow, and how to detect a loop.
• Execution environment - the sandbox, file system, and network access the agent works inside.

Claude Code and Codex CLI are agent harnesses, and SDKs such as the Claude Agent SDK hand you harness pieces to configure into your own. Harness engineering is the practice of improving that code, through better tool descriptions, verification steps, and context handling, without touching the model. The planner, generator, and evaluator pattern in agentic AI architecture is one common harness design.

What Is an Eval Harness?

An eval harness is the infrastructure that runs an evaluation from start to finish. The same Anthropic post puts it plainly: "An evaluation harness is the infrastructure that runs evals end-to-end." In practice it sets up each task, invokes the agent, records every step of the trial, grades the outcome, and rolls the grades up into scores.

• Tasks - inputs plus success criteria, with the starting state each task assumes.
• Trial runner - starts the agent in an isolated environment, often several times per task, since agents are non-deterministic.
• Recorder - keeps the transcript, tool calls, and the environment state at the end of the trial.
• Graders - code checks, model judges, or human review that turn the record into verdicts.
• Aggregation - pass rates, consistency across trials, cost, and latency per task and per suite.

The term predates agents. A classic test harness wraps code under test with drivers and stubs, and model benchmark runners such as EleutherAI's lm-evaluation-harness score single prompts against fixed answers. Agent eval harnesses add multi-step trials, tools, and environment state, and the walkthrough on how to build an eval harness for AI agents covers building one step by step.

Agent Harness vs Eval Harness: Key Differences

The two share a word and little else. One is part of your product, and the other is part of your test infrastructure:

They meet in every eval run, because the eval harness drives the agent harness. Anthropic's post says it directly: "When we evaluate 'an agent,' we're evaluating the harness and the model working together." A score can therefore move because of the model, the agent harness, or the eval harness, and you can only tell which with tests aimed at each.

Note: TestMu AI Agent Assurance runs from your terminal as rook. It reads your agent's code to work out what it does, writes the scenarios, and grades each one against the files, tool calls, and records the run changed. Create your TestMu AI account

Can the Same Model Score Differently in Different Harnesses?

The opening numbers come from a June 2026 survey on agent system and harness design, which compared public Terminal-Bench 2.0 submissions. Claude Opus 4.6 ranged from 58.0% with Claude Code to 76.4% with another harness, 14 of the 20 models varied by at least 10% across harnesses, and the median within-model range was 13.6%. The authors note the evidence "is not a randomized ablation," since submitters tuned their own setups, but the spread is too large to ignore.

A controlled example points the same way. In OpenAI's cookbook recipe on harness-aware evaluation of plugins, GPT-5-mini scored 0.43 on a plugin's questions inside the Codex product harness and 0.64 in a small standalone tool-calling loop. The authors' conclusion: "A score from one setup doesn't transfer to the other, since it measures the setup as much as it measures the model."

For your own agent, three habits follow:

• Version the harness - record the harness commit next to the model ID in every eval result, so a score change can be traced to one or the other.
• Evaluate in your own harness - a leaderboard number measured in someone else's harness says little about how the model behaves inside yours.
• Change one thing at a time - when you swap the model and change the harness in one release, a regression can't be attributed to either.

How to Test an Agent Harness

Test it like the software it is, because it ships to users and breaks the same way any code breaks. Work from the inside out:

• Component tests with a scripted model - feed the harness fixed model outputs and check tool dispatch, argument validation, unknown-tool handling, retry limits, and stop conditions. The guides to testing Claude Agent SDK agents and OpenAI Agents SDK agents show this with real hooks and scripted models.
• Permission tests - confirm a denied tool stays denied through retries, subagents, and context compaction, and that every tool that can write is covered by a guard.
• Context tests - push a long task past the compaction point and check that the goal, constraints, and open work survive the summary.
• Ablations - remove one harness component at a time, rerun the same tasks, and keep the component only if the scores drop without it.
• A regression gate - rerun a fixed task set on every harness change, with some tasks held out so tuning doesn't overfit to the ones you watch.

Ablations earn their place because harness components cost money. In Anthropic's post on harness design for long-running application development, a solo agent ran for 20 minutes at a cost of $9, while the full harness ran for 6 hours at $200. The author's reasoning applies to every harness: "every component in a harness encodes an assumption about what the model can't do on its own," and those assumptions go stale as models improve.

The same post flags a trap for harnesses that check their own work: "agents reliably skew positive when grading their own work." If your harness includes a self-check step, test that step against cases where the work is wrong, and prefer a separate evaluator for the final verdict.

How to Validate an Eval Harness

Calibrate it like a measuring instrument, because its errors show up as wrong scores rather than crashes. Anthropic's evals post describes the cost of skipping this: Opus 4.5 "initially scored 42% on CORE-Bench" until researchers found problems such as rigid grading that rejected "96.12" when the expected value was 96.124991, and after the fixes the score rose to 95%.

• Read transcripts - sample passing and failing trials and confirm each verdict matches what happened in the run.
• Check graders against humans - label a sample by hand and measure how often each model judge agrees, especially on the failures. The guide to agent-as-a-judge covers graders that inspect the environment instead of the transcript.
• Pin the environment - record CPU, memory, timeouts, and tool versions for every run, since agents use the runtime to solve tasks.
• Start clean - give every trial a fresh environment, so files and history from one trial can't help or hurt the next.
• Report what you couldn't check - when the evidence a grader needs is missing, record the criterion as unverified instead of folding it into pass or fail.

That last habit looks like this in an Agent Assurance run of a release-notes agent. The Unverifiable expectations panel lists 4 expectations across SC-004 and SC-007 that couldn't be checked because the run had no view of the agent's tool calls, and those stay out of the pass rate instead of being guessed.

Testing Both Harnesses With Agent Assurance

TestMu AI Agent Assurance tests how your agents actually behave across workflows, tools, and actions, and catches failures and vulnerabilities before they ship. It works as a ready-made eval harness for agents that act, pointed at your agent harness:

• Tasks from your code - it reads the agent's source, prompts, and tool definitions, or a PRD when there's no code to read, and generates functional and adversarial scenarios.
• Your real harness in the loop - it invokes the agent the way a user would, through a command, an HTTP endpoint, or a stdio MCP server, so the harness under test is the one you ship.
• Graders that read the environment - each criterion is checked against files that changed, artifacts produced, tool calls against the agent's declared tools, and read-only checks through an MCP tool you approve. A claimed action is never proof.
• Two configurations, one suite - an agent can have several profiles, so you can run the same scenarios against two harness or model configurations and compare.
• An honest denominator - criteria it couldn't verify are reported as Unable to Verify and kept out of the pass rate.

Agent Assurance is pre-alpha, so commands and stored file formats can still change, and it runs the agent for real, so point it at staging. The Agent Assurance overview docs explain how discovery, profiles, and verdicts fit together.

Next Steps

Start by writing the harness commit into every eval result you already produce. Then pick the one agent harness component you trust least, ablate it on a fixed task set, and read twenty transcripts from the run to check that your graders agree with what happened.

To run generated scenarios against your real agent harness, install Agent Assurance with npm install -g @testmuai/rook and point it at a staging deployment.