Agent Harness Testing: What a Harness Can't Verify About Itself

What Is an Agent Harness?

An agent harness (also known as agent scaffolding or an AI harness) is all the stuff around the model that actually turns it into an agent. It's the tool calling loop, the context and memory, the permissions and sandboxing, the retries and the logs.

The standard shorthand is agent = model + harness. Agents like Claude Code and Codex run inside one, as do browser agents and enterprise workflow agents.

The majority of guides stop at that point and just list the bits. This one takes it one stage further and asks the next question: once the harness is built, how do you know it works?

What Each Part of the Harness Can't Verify

Every row has the same shape. The harness governs behaviour at the moment. Correctness lives in the end state, which the harness never inspects.

Every Harness Change Is a Release

Harness engineering is iterative. You've got teams constantly playing with prompts, swapping models, changing how context is compressed, adding tools and fiddling with retry limits. Each of those things can have an impact on what the agent actually ends up doing, and no one individual change is going to be obvious.

• Model swap: the new model calls the tools in a different way, and the workflow that worked last week now goes around a validation step.
• Context compaction: a new summarisation rule which drops a field the agent would have needed three steps later.
• New tool: the agent will go for the new tool and will simply bypass the old one which had the business rule.
• Retry policy: a timeout retry will re-attempt the write and create a duplicate record.

None of these trips a guardrail. Every step is permitted, is logged, and, where appropriate, is approved. They all appear in the end state.

A worked example. Imagine a refund agent that reads a support ticket, looks up the order in the commerce system, and issues a refund. The team improves the model and increases the retry limit.

In testing, everything looks fine, and each refund is approved. When the payment API times out, the harness will retry, and some orders end up being refunded twice. It's only by looking at the end state (one refund per eligible order) that you spot it.

That's why you need regression tests at the workflow level (run on every change) rather than just unit tests of the individual components.

Verify From Outside the Harness

A harness can't certify itself for the same reason a company won't audit its own books. The verifier has to be external to the model, the prompts and the runtime.

That's Agent Assurance (TestMu AI's independent verification layer for agent workflows) in action via Rook. Test agents run your workflow through the real systems and then check the final state (records, data, downstream effects) against deterministic specs which are derived from your Jira tickets, your docs and your recordings.

The same specs run on every harness change, so a regression will fail the release rather than get to the customer. It works with any model or harness, and the evidence stays completely in your own control plane.

How it fits in with the rest of your testing is that the Kane CLI tests the software you build, Agent Assurance tests what your agent workflow does to the systems around it, and the harness keeps each run safe as it happens.

A practical loop:

• Write the anticipated results of the workflow as specs, straight from the requirements.
• Execute them against today's harness to establish a baseline.
• Re-execute them on every harness change: model, prompt, tool, context strategy, retry policy.
• Ship only on a pass and turn every production incident into a new spec.

Conclusion

The double refund in our example didn't trip a guardrail, fail a permission check or show up as an error in the logs. It only showed up in the end state. That's the gap every agent harness leaves open, and it widens each time you swap a model, edit a prompt or add a tool.

Agent Assurance closes it by verifying what each workflow actually changed in the systems around it, from outside the harness, on every change. The harness governs how the agent behaves. Agent Assurance proves what it did.