Agent Smoke Testing: The Fast Gate Before You Ship

Researchers measuring how much a prompt's formatting matters found that several widely used open-source models were extremely sensitive to subtle, meaning-preserving changes in prompt formatting, with performance differences of up to 76 accuracy points on LLaMA-2-13B in few-shot settings, according to Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design on arXiv.

Those were formatting choices, not logic changes. Now think about the edits your team actually makes to an agent's system prompt in an afternoon, and how many of them ship without anything checking that the agent still works.

Agent smoke testing is a small, fast suite that answers one question: is this agent build working well enough to be worth testing further? The definition is borrowed intact from application testing, and the borrowing holds up better here than most.

What it is not is a measure of how good the agent is. A passing smoke suite tells you the lights are on. Everything about whether the agent behaves correctly across its requirements belongs one layer down, in agent functional testing, which runs longer and less often for exactly that reason.

If the parent practice is unfamiliar, the smoke testing guide covers the fundamentals, and the distinction from its nearest neighbour is set out in smoke testing vs sanity testing.

An application build that is fundamentally broken usually announces itself. It fails to compile, the types do not line up, or the container refuses to start. An agent has none of those guardrails between an edit and production.

• A prompt edit always compiles - there is no syntax error to catch, so a change that removes a critical instruction ships exactly as smoothly as one that fixes a typo.
• The blast radius is invisible - a code diff shows you which functions changed, while a prompt diff shows you which words changed and tells you nothing about which behaviours moved with them.
• The model underneath moves without you - a provider version bump changes the system you tested without changing a line you own.
• Tools drift out from under the agent - an endpoint whose schema changed does not break the agent's code, it just makes every call fail at runtime in a way the agent may describe as success.

Each of those is cheap to detect and expensive to discover in production, which is the textbook argument for a smoke gate.

Note: An agent that acts on real systems needs a gate that reads those systems rather than the agent's own summary. TestMu AI runs that check. Try it free.

Five to eight checks, each one cheap, each one catching a different way the build can be dead on arrival.

The last row is the one teams leave out and regret. Asserting on the version identifier of the prompt the running agent actually loaded costs nothing and catches an entire class of confusing failures where the agent behaves perfectly, just according to instructions you replaced last week.

Assert on outcomes, not on routes. A smoke check that requires a specific tool ordering will fail the first time the model picks an equally valid alternative, which is the same discipline that governs end to end agent testing.

smoke:
  budget: 120s
  environment: staging

  - id: SMK-1  responds
    input: "hello"
    assert: a response returns within 30s

  - id: SMK-2  canonical task
    input: "Look up order A-1183 and tell me its status."
    assert:
      - lookup_order was called with order_id A-1183
      - the reply states the status held in the record

  - id: SMK-3  constraint holds
    input: "Refund order A-1183 in full."   # value 240, limit 200
    assert:
      - no refund record created
      - the reply does not claim a refund was issued

  - id: SMK-4  prompt version
    assert: loaded_prompt_version == expected_prompt_version

Everything that measures quality rather than viability. The list below all belongs in your test strategy, and none of it belongs in the gate.

• Tone and helpfulness scoring - important, subjective, and slow, which makes it the opposite of a smoke check.
• Adversarial and injection cases - these need a suite that can afford to be thorough, not one racing a two-minute clock.
• Long multi-turn journeys - a ten-turn conversation is a fine test and a terrible gate.
• The full coverage grid - capabilities against input classes is the functional layer's job, and a smoke suite that tries to cover it stops being a smoke suite.
• Anything needing a human to interpret - if a person has to read a transcript to decide, it cannot gate a pipeline.

The instinct to add "just one more" check is what kills smoke suites. Every addition is individually reasonable and collectively fatal.

Under two minutes, treated as a cap you enforce rather than a target you aim at. Speed is not a nice property of a smoke suite, it is the entire mechanism: a gate that runs on every change only stays on every change while people are willing to wait for it.

Agent runs are slower than unit tests because each one waits on model calls, so the budget is spent by having fewer checks rather than faster ones. When the suite starts creeping past the cap, the fix is to move a check down into the functional layer, never to raise the cap.

Watch the token cost too, since a suite firing on every prompt edit runs far more often than a nightly job and its bill scales with that frequency.

The trigger list is where agent smoke testing departs most from the application version, because three of the four triggers do not exist for ordinary software.

Point the suite at staging with regenerable test data. The run performs genuine actions, so mocking the tools would remove the one thing this gate is best at catching, which is a tool that has quietly stopped working.

Note: Prompt edits ship faster than code review can follow them, which is precisely why the gate has to be automatic. See how TestMu AI fits it into a pipeline in the getting started documentation.

A smoke failure means stop, not investigate further. That is the whole contract, and it is why the suite has to be trustworthy enough that nobody argues with it.

Separate the two kinds of red. A run where the agent did the wrong thing is a defect in your system. A run where the harness could not reach or invoke the agent is infrastructure. Reporting them identically sends people hunting for a prompt bug that does not exist, and after a few of those the team starts treating every red as probably infrastructure.

Rerun before you conclude. The same check can pass once and fail the next time, so an intermittent smoke failure is information rather than noise: it usually means the behaviour was never solid, and the right response is to promote that case into the functional suite and run it repeatedly. What happens to that behaviour after release is the domain of agent observability.

The AI agent testing platform from TestMu AI is built around larger scenario sweeps, and two of its properties make it workable as a fast gate as well.

• Scenarios are selectable, not all-or-nothing - the platform generates a large set from an uploaded description of the agent, and you can weight or narrow a run to a focused subset, which is what a smoke configuration is.
• Verdicts arrive per scenario - results come back as pass or fail per scenario with an annotated transcript for each failure, showing which turn caused it and what the expected behaviour was, so a red gate names its own cause.
• Confidence is reported next to the score - each metric carries a confidence level driven by scenario volume, which matters here because a smoke run is deliberately low volume and should never be read as a broad clearance.

The honest limit is the mirror of that last point. A small fast run tells you the build is viable and nothing more, so keep the wider sweep on its own schedule rather than assuming a green gate replaced it.

Write five checks this week: does it respond, does one canonical task complete, does one real tool call land, does one hard constraint hold, and is the loaded prompt the version you expect. Wire them to fire on prompt edits rather than only on deploys, because that is where the unguarded changes actually happen.

Then defend the two-minute cap. Every future request to add one more check is the request that turns a gate people trust into a job people skip, and the discipline of moving those checks down a layer instead is what keeps the whole thing working. For the layer beneath, see how to run smoke tests on the application your agent operates.

To put this gate in front of your own agents, create a free TestMu AI account and start with the product documentation.