What Is an Agent Harness?

Is an Agent the Same as Its Harness?

No. An agent is a language model plus its harness. The model chooses the next step, such as editing a file or running the test suite, and the agent harness carries that step out: it checks the step against permissions, runs the tool, returns the result to the model and repeats until a stop condition is met.

The June 2026 survey draws the same line: an LLM-based agent "is not the foundation model alone, but a coupled system" of a model and an execution harness. LangChain's Vivek Trivedy wrote in March 2026: "If you're not the model, you're the harness."

Anthropic's guide Demystifying evals for AI agents uses agent harness and scaffold for the same system, one that "processes inputs, orchestrates tool calls, and returns results." Microsoft's Agent Framework documentation says an agent harness is "sometimes called an AI harness," and agentic harness and AI agent harness are variants of the same term.

How Much Can an Agent Harness Change a Model's Benchmark Score?

On Terminal-Bench 2.0, most models with three or more harness results scored 10 points or more higher in their best harness than in their worst. The survey by Jianyuan Guo and colleagues, which drew on the official leaderboard and public submission logs, counted 14 such models out of 20, and Claude Opus 4.6's spread was 18.4 points.

Terminal-Bench 2.0 is a benchmark of 89 tasks that an agent carries out in a terminal. The survey's Table VIII gives Claude Opus 4.6 in four harnesses submitted to the benchmark's leaderboard, with 445 trials each, and its text adds the Claude Code result.

• Spread within one model - Claude Opus 4.6 gained 11.8 points from Terminus 2 to Terminus-KIRA.
• Token use - Meta-Harness used a median of 755.0K input tokens per trial against 79.4K for Terminus 2, about 9.5 times as many, for 13.5 more points.
• Overfitting checks - because search and final evaluation shared one benchmark, the Meta-Harness authors checked for overfitting "by manual inspection and regex-based audits."

In the same survey, the median spread among those 20 models was 13.6 points, and the largest spreads, for Claude Haiku 4.5, GPT-5.1-Codex and Gemini 3.1 Pro, exceeded 20 points.

The authors call this evidence "not a randomized ablation" and conclude that task completion depends on "the fit between the model's interactive skills and the harness's runtime design." Treat a published score as a fact about someone else's pair, and measure the pair you ship on your own tasks.

What Are the Parts of an Agent Harness?

In TestMu AI's breakdown for testing, an agent harness has seven parts: context assembly, tool dispatch, permissions, loop control, memory and state, the execution environment, and recording. The failures listed are typical examples from the coding agent, not measured rates.

The Meta-Harness paper defines a harness as "a stateful program that wraps a language model and determines what context the model sees at each step." The survey splits the same layer into six "coupled runtime responsibilities": observation, context, control, action, state, and verification/governance. In TestMu AI's reading, observation and context fall under context assembly, control under loop control, action under tool dispatch and the execution environment, state under memory and state, and verification/governance under permissions and recording.

Recording sets how much a test can settle afterward. A changed file or record can be checked directly, but a criterion about which tools ran, or which ones the harness refused, needs a record of the calls. AI agent tracing covers capturing tool calls and spans, and AI verifiability covers measuring how much of what an agent did can be confirmed from evidence.

Which Coding Agents and SDKs Are Agent Harnesses?

Claude Code is an agent harness, and it appears as one in the Terminal-Bench 2.0 results above. Anthropic's post on effective harnesses for long-running agents calls the Claude Agent SDK "a powerful, general-purpose agent harness," and the Claude Agent SDK overview says the SDK gives you "the same tools, agent loop, and context management that power Claude Code."

No harness tops every model. In the survey's Table VIII, Claude Opus 4.6 peaks in Meta-Harness (76.4%), GPT-5.3-Codex in SageAgent (78.4%) and Gemini 3.1 Pro in TongAgents (80.2%). For SDK-specific setups, see testing Claude Agent SDK agents and testing OpenAI Agents SDK agents.

How Is an Agent Harness Different From a Framework or Runtime?

A framework is what you build an agent with, a runtime executes its runs, and the agent harness is the layer around the model in the agent you ship, whatever it was built from. LangChain's Harrison Chase wrote in October 2025, "I don't think there is a clear definition of framework vs runtime vs harness," and described agent runtimes as "generally lower level than agent frameworks."

In the examples above, Microsoft's Agent Framework is a framework with a Harness component, the Claude Agent SDK and the OpenAI Agents SDK are agent SDKs, and Claude Code and Codex are ready-made agent harnesses for coding. Each SDK or framework ships real harness parts, but the policies inside those parts stay yours; the guide to agentic AI frameworks compares the main options.

In testing, the older test harness supplies stub and driver modules around code under test, and an eval harness runs an agent against fixed tasks and grades it. Agent harness vs eval harness explains where the eval harness ends and the agent harness begins.

What Is Harness Engineering?

Harness engineering is improving an agent by changing what surrounds the model, such as its instructions, tools, checks and stop rules, rather than the model itself. Mitchell Hashimoto made "Engineer the Harness" step 5 of his AI adoption journey in February 2026: each time an agent makes a mistake, engineer a fix so that it does not happen again, through AGENTS.md updates or tools such as scripts that take screenshots or run filtered tests.

In a February 2026 memo on martinfowler.com, Birgitta Böckeler grouped OpenAI's write-up on harness engineering into three building blocks, a grouping she says is based on her interpretation:

• Context engineering - a knowledge base in the codebase that keeps improving, plus dynamic context such as observability data and browser navigation.
• Architectural constraints - rules checked by the agents and also by custom linters and structural tests.
• Garbage collection - agents that run periodically to find inconsistencies in documentation and breaches of the architectural rules.

In Anthropic's experiment on harness design for long-running apps, Prithvi Rajasekaran ran Claude Opus 4.5 on a prompt beginning "Create a 2D retro game maker," once alone and once inside a harness of three agents:

Rajasekaran called the harness "over 20x more expensive," with a quality difference that was "immediately apparent," and after Opus 4.6 arrived he removed the sprint structure, expecting the newer model to handle the work without it. Re-test each harness part when the model changes. Agentic AI architecture covers that three-agent pattern.

How Do You Test an Agent Harness?

Test each part on its own with a scripted model that returns fixed outputs, then test the whole agent with a real model. Böckeler's April 2026 article, "Harness engineering for coding agent users," leaves harness evaluation as an open question: "We need a way to evaluate harness coverage and quality similar to what code coverage and mutation testing do for tests."

import { posix } from 'node:path';

// The repo policy: no pushes, writes only under src/
// and test/.
const POLICY = {
  deny: ['git_push'],
  writable: ['src/', 'test/'],
};

// What a sub-agent gets when nothing is passed down.
// An empty prefix matches every path.
const DEFAULT_POLICY = { deny: [], writable: [''] };

// Fixed model outputs: edit, test, push, push again
// after the refusal, write outside src/, then hand the
// push to a sub-agent.
const OUTPUTS = [
  { tool: 'write_file', args: { path: 'src/cart.js' } },
  { tool: 'run_command', args: { cmd: 'npm test' } },
  { tool: 'git_push', args: { remote: 'origin' } },
  { tool: 'git_push', args: { remote: 'origin' } },
  { tool: 'write_file', args: { path: 'src/../.env' } },
  { tool: 'spawn_subagent', args: { outputs: [
    { tool: 'git_push', args: { remote: 'origin' } },
  ] } },
];

// A scripted model replays fixed outputs in order and
// ignores whatever the harness sends back to it.
function scriptedModel(outputs) {
  let i = 0;
  return (_lastResult) => outputs[i++] ?? { done: true };
}

function decide(policy, call) {
  if (policy.deny.includes(call.tool)) return 'deny';
  if (call.tool === 'write_file') {
    const p = posix.normalize(call.args.path);
    const ok = policy.writable.some(
      (dir) => p.startsWith(dir));
    if (!ok) return 'deny';
  }
  return 'allow';
}

// The loop: ask the model, decide, run or refuse,
// record every decision, stop at done or maxSteps.
function runAgent(outputs, policy, cfg, rec, depth = 0) {
  const model = scriptedModel(outputs);
  let last = null;
  for (let step = 0; step < cfg.maxSteps; step++) {
    const call = model(last);
    if (call.done) return;
    if (call.tool === 'spawn_subagent') {
      const child = cfg.inheritPolicy
        ? policy
        : DEFAULT_POLICY;
      rec.log.push({ depth, ...call, decision: 'allow' });
      runAgent(call.args.outputs, child, cfg, rec,
        depth + 1);
      last = { ok: true };
      continue;
    }
    const decision = decide(policy, call);
    rec.log.push({ depth, ...call, decision });
    if (decision === 'allow') {
      rec.effects.push(call);
      last = { ok: true };
    } else {
      last = { error: `${call.tool} denied by policy` };
    }
  }
}

config A: sub-agent inherits the policy
PASS git_push refused, then refused on retry
PASS git_push refused inside the sub-agent
PASS no push reached the remote
PASS write to src/../.env refused
PASS 7 of 7 calls logged, each with a decision
config B: sub-agent gets a fresh policy
PASS git_push refused, then refused on retry
FAIL git_push refused inside the sub-agent
     log: sub-agent allow
FAIL no push reached the remote
PASS write to src/../.env refused
PASS 7 of 7 calls logged, each with a decision
failed: A 0 of 5, B 2 of 5

How Do You Compare Two Agent Harness Configurations?

Leaderboard rows differ in more than the harness, so a comparison of two configurations of your own agent changes one thing and holds the rest fixed:

• Label every run - tag each result with the model ID, harness commit and policy revision that produced it.
• Change one variable - a tool description, the compaction threshold, a permission rule or the model, never two at once.
• Fix the task set - same tasks and same starting state, run by an eval harness that sits outside the agent.
• Repeat each configuration - run every task several times, because one run of an agent is a single sample of its behavior; pass@k vs pass^k covers choosing k, the number of runs per task.
• Compare three numbers - pass rate, the fraction of acceptance criteria each run could verify, and cost per task; the token column above shows how far token use can climb with score.
• Read changes by task - a flat pass rate can hide tasks that newly fail while others newly pass; agent regression testing covers reading a run-over-run diff.

Which Agent Harness Tests Can You Run With Agent Assurance?

For the end-to-end layer, TestMu AI's Agent Assurance invokes your real agent, such as the coding agent on this page, and judges each acceptance criterion on evidence from the run rather than on the agent's own account. You drive it from a terminal with Rook CLI, whose command is rook.

• Scenarios from your codebase - Rook CLI reads your code to find your agents and their tools, then drafts functional and adversarial scenarios, each carrying the acceptance criteria it is graded on.
• Profiles - a profile tells Rook CLI how to reach the agent you ship, over HTTP, by running a command or through an MCP tool, using hook scripts that you can read and review.
• Grading from evidence - what the agent said is graded from quotes of what it actually output; what it did is graded from evidence such as hook-reported tool calls and state changes, files that changed, artifacts it produced, and read-only queries made with a tool you approve. A model judge scores some criteria, and a claimed action is never proof.
• Three possible verdicts - a criterion the evidence cannot settle is marked Unable to Verify, and that verdict keeps its own count next to Pass and Fail instead of being merged into either. A profile's access sets how much of a run can be observed.

Most eval and observability tools judge what an agent replied and what its traces show; Agent Assurance checks what a run changed and lists the criteria it could not verify.

To compare two harness configurations, give the agent one profile per configuration, such as two models or two deployments of your harness, and run the same scenario IDs against each; every run keeps the profile it used. After the second run, Rook CLI shows how the pass rate moved between your two most recent runs and names each scenario that started failing or was fixed. Before you call a difference a regression, check both runs' profile revision and scenario selection.

Rook CLI has no undo for what the agent writes during a run, so aim it at a staging copy of your systems. It is a pre-release test, with no view of production traffic and no way to block an agent at runtime.

Agent Assurance is generally available, and a 14-day free trial gives you 4,000 credits; the Agent Assurance plans page lists the paid options. Install Rook CLI from npm (Node.js 20 or newer), and add the Claude Code skill if a coding agent will drive it; the skill's installer requires Node.js 22 or newer.

npm install -g @testmuai/rook@latest
npx @testmuai/rook-skill@latest install \
  --agent claude-code

Homebrew and the shell installer, the alternatives on macOS and Linux, need no Node. Rook CLI covers macOS and Linux, and 64-bit Windows through npm or WSL. Rook CLI's profiles and hooks guide shows how a profile invokes an agent and what its hooks report.

What Should You Test First in Your Own Agent Harness?

Start with the permission that would hurt most if it failed, such as a push to a shared branch or a write outside the repository, and write a scripted-model test that requests it three ways: directly, again after the refusal, and from a sub-agent, the route where the toy harness's planted bug got through.

Then add the context and loop-control rows of the matrix, and finish with the real agent on staging. For that last layer, the Agent Assurance quickstart walks through one small test against a public sample agent before you connect your own.