How to Evaluate a Tool That Claims to Be Agent-Native

What Is a Tool Claiming When It Says Agent-Native?

It is claiming that software, rather than a person, is a first-class user of the product. Every consequential operation a human performs through the interface should be reachable by a program that authenticates as itself, reads a schema to learn what is available, and interprets the response without a screen.

That is a much narrower claim than the marketing usually implies, and it is worth separating from neighbouring terms before an evaluation starts. A useful primer on the distinctions is our breakdown of how agent-native differs from agentic and AI-native, which are three different properties that vendors tend to use interchangeably.

The claim also carries regulatory weight now. In Operation AI Comply, announced on 25 September 2024, the US Federal Trade Commission brought five law enforcement actions against operations that used AI claims deceptively.

One detail from that sweep is worth carrying into every vendor call. In the DoNotPay matter, the FTC complaint alleged that the company "did not conduct testing to determine whether its AI chatbot's output was equal to the level of a human lawyer". The regulator's objection was not that the product used AI badly, but that nobody had measured whether the claim was true.

That sets a usable bar for buyers. Asking a vendor for the evidence behind an agent-native claim is not adversarial procurement, it is the same standard a regulator applied.

Why Can a Demo Not Answer the Question?

Because a demo is driven by a human who is already authenticated, already inside the right workspace, and available to click whatever appears. Those three conditions remove exactly the failures an agent hits first.

The pattern repeats across categories. A product can look fully autonomous on stage and still depend on a browser for the parts that matter:

• Authentication - the demo account was logged in beforehand, hiding the fact that issuing a machine credential requires a support ticket.
• Provisioning - the project, environment, and integrations were configured through a setup wizard that has no API equivalent.
• Approval - the destructive step was confirmed in a modal, which simply does not exist when a pipeline makes the same call.
• Interpretation - the presenter read the result off a dashboard, so nobody noticed the API returns that result only as rendered text.
• Recovery was never shown at all, because demos are built from the path where nothing fails.

The fix is not a longer demo. It is running the workflows yourself, from a machine, in the conditions the tool will actually face.

Which Checks Actually Separate the Two?

Seven, each of which produces a result you can record rather than an impression you have to argue about. Every one is runnable inside a normal trial, and every one has a specific vendor answer that tells you the claim is thin.

What Does a Failed Check Look Like?

Rather than describe one abstractly, we ran check three against our own tooling on 27 August 2026 and published what came back. The first call is a normal success against the TestMu AI Browser Cloud SDK, which returns a typed object a caller can read without parsing anything.

{
  "format": "png",
  "width": 1920,
  "height": 1080,
  "bytes": 142351
}
// elapsed_ms: 11342

That result passes check two comfortably. Dimensions and payload size arrive as numbers, and the format arrives as a stable string rather than a display label.

We then pointed the same call at a domain that does not resolve, to see what a caller receives when the operation fails. The result did not meet the bar this article sets:

name:       Error
message:    net::ERR_NAME_NOT_RESOLVED at https://this-domain-does-not-exist-9f2a.invalid/
own keys:   []

There is no stable error code, no field naming the offending parameter, and no retryable flag. A caller can log that string, but it cannot branch on it without matching text, and text matching breaks the next time the underlying message changes.

We are publishing our own failure here for a reason. A buyer guide where the host product passes every check it invents is worth very little, and this is the shape of finding your trial should be producing about every vendor on the shortlist, including us.

How Much Context Does the Tool Surface Cost?

Tool definitions and responses are billed as input tokens on every turn, so an oversized surface raises cost and lowers accuracy at the same time. This is the check buyers skip most often, and the one vendors are least prepared to answer.

For calibration, Anthropic's guidance on writing tools for agents notes that Claude Code restricts tool responses to 25,000 tokens by default, and states plainly that "more tools don't always lead to better outcomes" because overlapping tools distract agents from efficient strategies.

• Count the surface - serialize the full tool listing and token-count it, then count the tools themselves.
• Measure the heaviest responses - call the three largest operations with production-sized inputs and token-count each result.
• Check the controls - every listing operation should accept limit, cursor, and field selection with sensible defaults.
• Red flag - the vendor cannot tell you the token cost of their own manifest, which usually means no evaluation harness exists behind the claim.

How Do You Run This Inside a Normal Trial?

Two weeks is enough, and most of the signal arrives in the first two days. The sequence below front-loads the checks that disqualify a vendor fastest, so you stop spending time on tools that cannot pass.

• Request a machine credential on day one and record how long it takes to arrive.
• Capture the tool listing or API document, validate it, and token-count it.
• Script the five workflows headlessly in a clean container, logging every browser fallback.
• Run the five deliberate failures and record each response shape.
• Export one trace by API and hand it to somebody who did not watch the run.
• Write the findings up as observed results, not impressions, and compare vendors on the record rather than the pitch.

A tool that expects this treatment makes it easy. TestMu AI Agent Testing ships a Python command line, testmu-a2a-cli, that authenticates from environment variables so no interactive login is required in a pipeline, extracts an agent reply from a custom response shape using a JSONPath expression, and emits results as table, json, or junit so a build server can consume them directly.

export TESTMU_USERNAME=<username>
export TESTMU_ACCESS_KEY=<access_key>

testmu-a2a test \
  --agent https://my-bot.com/api/chat \
  --spec "Customer support bot that handles billing and account issues" \
  --count 10 \
  --threshold 0.80 \
  --format junit \
  --output results.xml

The pass threshold is an explicit flag rather than a dashboard setting, which is the property that lets a pipeline gate a merge on the result. Setup steps are documented in the Agent Testing CLI documentation, which also publishes a machine-readable index for agents and serves a plain Markdown version of any page by appending .md to its URL.

Apply the same scepticism here that this article asks you to apply everywhere. Phone evaluations run five parallel calls per organisation by default, configurable between one and fifty, and batch image analysis accepts fifty images per run, so throughput is bounded and worth confirming against your own volumes during the trial.

What Do These Checks Not Tell You?

They measure whether a tool is usable by a machine, which is a different question from whether the tool is any good. A product can pass all seven and still produce weak results, because interface quality and output quality are independent properties.

• Output quality - none of these checks assess whether the agent's actual work is correct, which needs its own evaluation method, often the approach described in our guide to using an LLM as a judge for agent evaluation.
• Behaviour under load - a clean trial says nothing about how the tool behaves at production concurrency, where useful baselines come from agent performance metrics and benchmarks.
• Cost at volume - per-task pricing that looks reasonable across ten runs can change character across ten thousand.
• Durability of the claim - the checks capture one moment, and agent interfaces are young enough that a passing result in August may not hold in December.

Run the interface checks to decide who stays on the shortlist, then evaluate output quality on the survivors. Doing it in that order saves the expensive work for tools that can actually be operated by the systems you intend to operate them with.

Where to Start

Pick the vendor currently furthest along in your evaluation and ask for one thing before the next call: a scoped machine credential and the address of their tool schema or API document. How quickly that arrives, and in what form, will tell you more than the next hour of demonstration.

Then run check five, because headless first-run parity is the one that fails most often and takes an afternoon to settle. If provisioning and teardown complete in a container with no browser, the remaining checks are worth your time. If they do not, the claim has already answered itself.

You can practise the whole sequence against a product that expects it. Request a credential from TestMu AI, work through the setup in the Agent Testing CLI documentation, and run the seven checks end to end so the process is familiar before you point it at a vendor who has more to lose from the answer.