Testing Non-Deterministic AI Outputs: A Practical Guide

Why Do AI Models Give Different Answers to the Same Prompt?

AI models give different answers to the same prompt because inference servers batch requests together, and the batch size changes the order in which floating-point values are summed inside the model. That reordering shifts token probabilities enough to flip a choice.

Thinking Machines Lab traced this behavior directly. Sampling 1,000 completions from Qwen3-235B at temperature 0 with a single fixed prompt produced 80 distinct completions, and the most common one appeared 78 times. The completions stayed identical until token 103, where 992 of them continued with "Queens, New York" and 8 continued with "New York City".^[1]

Your prompt is identical on every run, but the other users' requests batched alongside it are not, so the source of variation sits outside your code and outside your control. Three separate sources stack:

• Sampling randomness: Temperature and top-p draw the next token from a probability distribution, so wording and structure shift between runs even on a stable server.
• Batch-size dependence: Reduction kernels for operations like RMSNorm and attention are not batch-invariant, so the same input under a different server load produces different logits.
• Provider-side changes: Model weights and serving configuration change on the provider's schedule, which moves output without any deploy on your side.

Only the first source is yours to configure. A test suite written as though sampling is the whole problem will pass in development and fail under production load, because production is exactly where batch composition varies most.

Does Setting Temperature to 0 Make an LLM Deterministic?

No. Temperature 0 forces greedy decoding, so the model always takes the highest-probability token, but it leaves the batching and kernel behavior underneath untouched. Greedy decoding narrows variance without removing it, and a fixed seed adds nothing at temperature 0, because greedy decoding makes no sampling draw for the seed to control.

OpenAI states the position plainly in its API documentation: Chat Completions are "non-deterministic by default", and a seeded request returns output that is only "(mostly) deterministic".^[2] The same documentation ties reproducibility to a system fingerprint that tracks model weights and infrastructure configuration, and it changes when the provider updates either.

The measured cost of assuming otherwise is large. Researchers at Penn State University and Comcast AI Technologies ran five models across eight benchmark tasks at temperature 0 with fixed seeds, and recorded accuracy variations up to 15% across runs, with a gap between best and worst possible performance reaching 70%.^[3]

Set temperature to 0 and pin the model version anyway. Both narrow the distribution you have to test against.

Why Do Exact-Match Assertions Fail on AI Output?

Exact-match assertions fail because one question has many correct answers. "Paris", "Paris is the capital of France", and "The capital of France is Paris" are all correct, and a string equality check accepts exactly one of them. Each valid phrasing the model can produce lowers the odds that one expected string matches, so the suite decays as prompts evolve.

This is the test oracle problem. A conventional assertion needs an oracle, meaning a reliable way to decide whether a specific output is correct. Generative systems break that assumption because the correct output cannot be enumerated in advance, so a red test tells you the string changed and nothing about whether the behavior broke.

A script written for this article sent one prompt to gemini-2.5-pro 20 times at temperature 0 and compared each response against the first. The exact-match assertion passed 5 of the 20 runs, and the five distinct strings differed only in wording. Every response was a correct definition, so this run tested exact match on correct output only. The result covers one prompt, one model, and one provider.

The failure is expensive in both directions. False failures train the team to rerun the suite until it passes, which destroys the signal. False passes are worse: an assertion that checks only for the substring "Paris" also passes on "Paris is not the capital of France".

Two rules follow. Assert on properties the answer must hold rather than on the text it happens to use, and keep every deterministic check that still applies. Response schema, status codes, latency bounds, required fields, and banned patterns are all still exactly assertable, and they should stay as strict as they were before the model arrived.

What Assertion Types Work for Testing Non-Deterministic AI Outputs?

Four assertion types work on non-deterministic AI output: structural checks on shape, invariant properties that must always hold, semantic similarity against a reference, and rubric scoring by a judge model. Each costs more than the one before it. Structural checks run locally with no model call, while rubric scoring adds an extra inference to every assertion.

Route each claim to the cheapest type that can actually verify it. Sending a JSON schema check to a judge model spends tokens and adds variance on a problem that a Pydantic model answers instantly and identically every time. Reserve rubric scoring for claims that genuinely need semantic judgment, and read LLM as a judge for the bias controls that keep those scores stable.

In the Reddit thread "You can't test the model, so I gave up and tested everything around it" on r/LLMDevs, the author replaced a hand-tuned instruction file with deterministic gates: a hook that typechecks every file write, and an allowlist that stops the agent before it pushes. The author closes by asking how to test the non-deterministic part itself.

How Many Runs Does It Take Before a Pass Rate Means Anything?

A pass rate needs enough runs that its confidence interval is narrower than the regression you want to catch. An 18-of-20 result looks like 90%, but its exact binomial interval at 95% confidence runs from 68.3% to 98.8%.^[4] Precision improves with the square root of the run count, so doubling runs narrows that interval by far less than half.

That figure uses the exact binomial method, which the NIST Engineering Statistics Handbook specifies for proportions measured on small samples. A range that wide cannot separate a healthy suite from a badly degraded one, so a short gate reporting a high pass rate tells you almost nothing.

Small run counts also hide common defects. The same binomial model says a failure mode firing on 8% of requests survives a 5-run gate whenever all 5 runs pass, and 0.92 to the fifth power is 65.9%.^[4] The defect clears the gate on roughly two attempts out of three, then reaches production as an intermittent complaint nobody can reproduce.

Work backwards from the regression size that matters to you. Applying the NIST sample-size formula for proportions at 95% confidence and 80% power, detecting a drop from a 95% baseline to 85% needs 53 samples.^[5] Halve the difference you want to catch and the requirement roughly quadruples, which is the arithmetic that decides your suite size.

• Smoke gate: Run 5 to 10 repetitions on a handful of critical scenarios for per-commit feedback, and accept that it catches only gross breakage.
• Release gate: Run 55 or more scenarios once each before a release, which buys statistical power through breadth instead of repetition.
• Stability probe: Repeat one scenario 100 times when you specifically need to measure variance for that prompt, such as after a model version change.

Breadth beats repetition for most suites. One hundred different scenarios run once exercises far more of the input space than one scenario run 100 times, and it costs the same in tokens. Repeat a single prompt only when the variance of that specific prompt is the thing under test.

In the Reddit thread "Verifying non-deterministic generative UI in CI" on r/LLMDevs, the author reported that generated interfaces change on every render, so pixel-diff tools fail. Their harness samples N generations, renders each in headless Chromium, and scores data fidelity, accessibility, and layout, then reports consistency as the standard deviation of those scores. The practical takeaway is that spread across runs is itself worth gating on.

What Is Metamorphic Testing, and When Does It Beat a Golden Set?

Metamorphic testing checks that a required relation holds between the outputs of two related inputs, so it needs no expected answer at all. It beats a golden set whenever the correct output cannot be written down in advance. The relation itself is the assertion, which is why it scales to inputs that have no reference answer.

A systematic survey by Zheng Zheng and colleagues describes the technique as oracle-alleviating, because it checks necessary relations among multiple related executions instead of relying on exact expected outputs.^[6] Their worked example is sentiment analysis: replacing a word with a synonym, so "The movie is good" becomes "The movie is excellent", must not change the predicted label.

Metamorphic relations are the cheapest way to generate large numbers of valid test cases, because each one turns a single labeled input into an unlimited supply of derived checks. Four relations transfer directly to a retrieval or support application:

• Paraphrase invariance: Rewording a question must not change the extracted fact, so "When does my plan renew?" and "What is my renewal date?" must return the same date.
• Negation inversion: Negating the input must flip a classification label, which catches the exact failure that embedding similarity scores as a near-match.
• Irrelevant-context invariance: Appending an unrelated sentence to the prompt must not change the answer, which exposes prompt-injection sensitivity and context dilution.
• Order invariance: Shuffling retrieved documents or list items must not change the conclusion, which catches position bias in the retrieval layer.

The relations complement a golden set rather than replacing it. A golden set proves the system gets specific known cases right, and metamorphic relations prove it stays self-consistent across inputs nobody labeled, which is where regressions usually hide. For broader coverage of scoring methods, LLM evaluation maps the metric families these relations feed into.

How Do You Build a Golden Set for Testing Non-Deterministic AI Outputs?

A golden set is a fixed collection of inputs paired with human-approved acceptance criteria rather than exact expected strings. Start with 25 to 50 labeled cases, then grow the set from real production failures. Criteria do not catch a regression nobody anticipated, so the set only covers failure modes someone has already written down.

The design decision that makes a golden set survive is storing criteria instead of answers. An expected-string record breaks on the first harmless rewording, while a criteria record keeps working across model versions because it encodes what correctness means for that input.

• Required facts: The specific values the answer must contain, such as a policy number or a cancellation deadline, checked by substring or extraction.
• Forbidden claims: Statements the answer must never make, such as promising a refund the policy does not allow, checked by pattern or judge.
• Structural requirements: Field presence, value ranges, and format rules that a schema validator can settle without a model.
• Reference answer: One human-written correct response used only as the anchor for semantic similarity scoring, never as an equality target.
• Provenance note: Where the case came from, because cases harvested from production incidents deserve stricter thresholds than synthetic ones.

Keep the set small enough to run often. Scaling past 100 cases before the thresholds are calibrated produces a suite that is expensive to run and still unable to say whether a change made things better. Every production failure that reaches a user should end its life as a new record in this file.

How Do You Catch Regressions When the Output Changes Every Run?

Compare distributions instead of single runs. Record the pass rate and score spread for every scenario on each build, then fail the pipeline when the new mean drops below the stored baseline interval. A build whose wording changed on every scenario but whose pass rate held has not regressed.

A single red response is not a regression signal on a system that samples from a distribution. The measurable signal is a shift in the distribution, and it takes two forms: the mean pass rate drops, or the spread widens while the mean holds steady.

The second form is the one teams miss. An unchanged average with a doubled standard deviation means the system now fails harder on the runs where it fails.

# Gate a build on the distribution, not on one response.
import statistics

BASELINE_PASS_RATE = 0.94
BASELINE_STDDEV = 0.06
MIN_RUNS = 30

def gate(scores, threshold=0.75):
    """scores: one similarity or rubric score per scenario run."""
    if len(scores) < MIN_RUNS:
        raise SystemExit(f"Need {MIN_RUNS} runs, got {len(scores)}")

    pass_rate = sum(s >= threshold for s in scores) / len(scores)
    spread = statistics.pstdev(scores)

    # Wilson-style margin: allow normal sampling noise, block real drops.
    margin = 1.96 * (BASELINE_PASS_RATE * (1 - BASELINE_PASS_RATE) / len(scores)) ** 0.5

    if pass_rate < BASELINE_PASS_RATE - margin:
        raise SystemExit(f"Pass rate {pass_rate:.2%} below baseline")
    if spread > BASELINE_STDDEV * 2:
        raise SystemExit(f"Score spread {spread:.3f} doubled; output destabilized")

    print(f"OK pass_rate={pass_rate:.2%} spread={spread:.3f}")

Store results per scenario rather than as one aggregate. An aggregate score hides the case where two scenarios improve while one collapses, and per-scenario history makes a regression traceable to the prompt that caused it. Re-baseline deliberately after a model upgrade, and record the old numbers so the comparison stays honest.

Sampling several responses and taking the majority answer also raises accuracy on reasoning tasks. Wang and colleagues reported that self-consistency, which samples diverse reasoning paths and selects the most consistent answer, improved GSM8K accuracy by 17.9%.^[7] The same aggregation that stabilizes a test also stabilizes production behavior.

For the wider tooling picture, LLM testing covers the frameworks these gates plug into, and LLM test automation shows the pipeline wiring with code.

How Do You Validate a Non-Deterministic Agent Before It Ships?

Run the agent through many scenario variants, score each on fixed quality dimensions, and require a confidence level tied to scenario volume before approving the release. A verdict from 10 scenarios is not the same evidence as a verdict from 200.

Multi-turn agents make the sample-size problem worse. A five-turn conversation can score well on every individual response and still fail overall because the agent lost context between turns, so per-response scoring misses the defect entirely. Building that scenario volume by hand is where most teams stall, because writing 100 conversational variants and grading them consistently is slow manual work.

TestMu AI provides Agent Testing, which generates the scenarios and scores the transcripts against fixed dimensions:

• Response Consistency: Measures information uniformity across repeated calls that run the same scenario, which is the direct measurement of the variance this guide is about.
• Evaluation Confidence Scoring: Labels each metric High, Medium, or Low based on scenario volume, so a passing score from too few scenarios is marked as indicative rather than decisive.
• Hallucination Detection: Flags responses that state information the agent's knowledge base and context do not support, scored per scenario and aggregated across the run.

Scenario generation produces 60 to 100 or more variants per workflow from uploaded documentation, and each runs as a full multi-turn conversation rather than a single exchange. You can follow the getting started with agent testing platform guide to connect an agent and read the go-live report.

Non-determinism shows up at every stage of a testing pipeline, not only at the model boundary. TestMu AI runs a set of purpose-built AI agents across those stages, including Agent Testing for conversational quality, an Auto Healing Agent for element drift, a Test Insights Agent for result analysis, and a Root Cause Analysis Agent for failure triage.

The metric definitions behind agent evaluation scores are set out in AI agent evaluation.

Validate Non-Deterministic Agent Behavior Before Release

Explore AI Agents

Conclusion

Start by writing down the acceptance criteria for 25 scenarios your application must handle, then pick the cheapest assertion type that can verify each one. Set temperature to 0, pin the model version, and treat the remaining variance as a property to measure rather than a bug to eliminate.

Testing non-deterministic AI outputs comes down to one shift: the assertion moves from the response to the distribution behind it. Pass rates, tolerance bands, and metamorphic relations all answer the question a release actually turns on, which is whether behavior changed since the last build.

Teams planning a broader skills path can follow the AI roadmap for software testers, which places these techniques alongside the rest of the AI testing stack.

Sources and References Used

• [1] Defeating Nondeterminism in LLM Inference - Thinking Machines Lab (September 2025)
• [2] Advanced Usage: Reproducible Outputs and the Seed Parameter - OpenAI API Documentation (2026)
• [3] Non-Determinism of Deterministic LLM Settings - Atil et al., Penn State University and Comcast AI Technologies (arXiv 2408.04667)
• [4] Section 7.2.4.1 Confidence Intervals for Proportions - NIST/SEMATECH e-Handbook of Statistical Methods
• [5] Section 7.2.4.2 Sample Sizes Required for Proportions - NIST/SEMATECH e-Handbook of Statistical Methods
• [6] Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey - Zheng et al. (arXiv 2605.13898)
• [7] Self-Consistency Improves Chain of Thought Reasoning in Language Models - Wang et al., ICLR 2023 (arXiv 2203.11171)