LLM Regression Testing: Gates That Survive a Model Upgrade

The Trouble With a Pass-Rate Delta

A new model version lands, the suite runs green, and the aggregate score comes back a point or two above the frozen baseline. The general practice this builds on is covered in our guide to regression testing. The team reads that as a clean upgrade and ships it the same afternoon. By Friday, support has tickets about a summarization behavior that used to work.

An arXiv preprint paired aggregate accuracy against per-item direction across two version upgrades.

• Method - 2,000 MMLU-Pro items sampled ten times each at temperature 0.7, with the direction of change recorded per item.
• Llama 3 to 3.1 - aggregate accuracy rose by 1.6 points; among the 952 analyzable items, 34% reliably improved and 28% reliably deteriorated.
• Qwen 2.5 to 3 - aggregate accuracy rose by 2.8 points; among the 652 analyzable items, 47% reliably improved and 39% reliably deteriorated.
• Analyzable items - items scoring 0.0 or 1.0 under both models are excluded as undetectable. The preprint separately reports churn across the full benchmark, improvements and deteriorations together, at 21% for Llama and 28% for Qwen.
• Provenance - the study is an arXiv preprint and has not been through peer review, so treat the magnitudes as indicative and act on the direction.

Version drift inside a hosted service moves per-item behavior with no upgrade on your side.

• Fixed inputs - in arXiv:2307.09009, Chen, Zaharia and Zou put the same prime versus composite questions to two dated GPT-4 snapshots.
• March 2023 - 84% accuracy on identifying prime versus composite numbers.
• June 2023 - 51% accuracy on those same questions.
• Attribution - the questions never moved, so the behavior change belongs to the service your gate is pointed at. That finding is also arXiv evidence, outside peer review.

An aggregate reports how many items passed, and a regression gate needs to know which ones.

Both upgrades cleared a delta gate on exactly this arithmetic.

• Gate arithmetic - a delta gate fires only when deterioration outweighs improvement by more than the threshold you set.
• Both upgrades - improvements outnumbered deteriorations, so the gate reports a healthy release while somewhere between a quarter and two fifths of analyzable items get worse.
• Who absorbs it - the cost lands on whichever user journeys sit inside the deteriorating quarter to two fifths, and the aggregate carries nothing that names them.

A pass rate is a summary statistic, and a summary statistic earns its place by discarding variation so that one number can stand in for many results. That trade works for a status dashboard. For a regression gate, the variation being discarded is the same variation you built the gate to catch.

Measuring Churn Instead of Net Movement

Churn is the count of items that changed verdict between two runs, tallied in each direction: one column for fail to pass, one for pass to fail. A netted pass-rate delta folds both into a single figure and throws away the movement underneath.

Take a 400-case suite that scored 340 last week and scores 340 today. The delta is zero, and the same zero appears when 18 cases broke and 18 others started passing. Identical pass rates can sit at opposite ends of the churn range.

Wire an alert to a threshold on that aggregate and it stays silent through exactly that run, because the number it watches never moved. The movement is in the set of cases failing now that were not failing before, which is the pass to fail column arriving as a notification rather than a report. The conditions worth alerting on are that newly failing set and a drop in confidence for a metric that had been stable, and the gate they feed is still yours to set.

The case for sampling each item more than once rests on the same preprint, which is not peer reviewed and re-scored 2,000 MMLU-Pro items across two version upgrades, Llama 3 to 3.1 and Qwen 2.5 to 3. Every item was scored once with greedy decoding, then ten times at temperature 0.7, and an item counted as changed only when its pass rate across those ten samples moved reliably, a stricter bar than one flipped verdict. Greedy single-shot scoring was wrong in both directions.

• Missed changes - single-shot scoring failed to flag 42% of the items that repeated sampling showed had reliably moved in either direction.
• False flags - single-shot scoring also flagged 25% of unchanged items as changed, which sends triage after findings that evaporate on a rerun. Moving the pass threshold recovers neither error.
• Aggregate gains sat on heavy churn - Llama gained 1.6 points while 34% of its 952 analyzable items improved and 28% deteriorated, a median pass-rate shift of 0.50; Qwen gained 2.8 points while 47% of its 652 analyzable items improved and 39% deteriorated, median 0.90.
• Count the undetectable items back in - those figures exclude items that scored 0.0 or 1.0 in both models, and across the full benchmark churn was still 21% and 28%.

The gate that follows holds critical degradations at zero instead of a pass-rate floor.

• Print two columns, drop the net - the run summary carries improved count and degraded count side by side, each with its item IDs underneath, and the netted delta leaves the report.
• Sample each case K times - the study used ten samples per item. Record the K you picked with the result so later comparisons are like for like.
• Price the sampling honestly - K samples costs K times the output tokens, wall clock and rate-limit pressure on every pipeline run. That trade-off is why a high K belongs on the critical list, with the long tail at K=1 or nightly.
• Gate on the degraded column - the failure condition is the number of critical cases that went pass to fail, held at zero, so a healthy average across hundreds of items cannot absorb one break.
• Keep the critical list short - the cases you would roll back for, such as a refusal on a prompt that must be answered, a malformed tool call, or a cited document that stops resolving.
• Watch the improved column too - a jump there on a change meant to be behavior-neutral usually means your scoring moved, which is worth checking before anyone banks the win.

A run summary that reads “improved 31, degraded 19, critical degradations 2” gives the reviewer something to act on before the merge.

Controls You Pinned Are Being Withdrawn

If your suite pins sampling parameters to hold output steady, check whether that pin still works on the model you are about to upgrade to. Anthropic’s API documentation lists temperature, top_p and top_k as deprecated as of Claude Opus 4.7, and on Claude 4.7 and later models, setting any of the three to a non-default value returns a 400 error.

A request that worked yesterday fails outright, and Anthropic’s guidance is to omit the parameters and use prompting to guide behavior. The standing instruction to set temperature to zero now breaks the call on that model family.

What the pin was buying you, on the published record:

• Default behavior - OpenAI’s API docs state that Chat Completions are non-deterministic by default, meaning model outputs may differ from request to request.
• Seed and system_fingerprint - OpenAI warns that even with the seed parameter pinned and all other parameters identical, determinism may be impacted due to necessary changes it makes to model configurations on its end. A changed system_fingerprint in the response is the signal that this happened.
• Temperature 0 - a peer-reviewed ACM TOSEM study of ChatGPT code generation (DOI 10.1145/3697010) found that setting the temperature to 0 does not guarantee determinism, although it does bring less non-determinism than the default configuration at temperature 1.
• Self-hosted deployments - an arXiv preprint (arXiv:2601.19934, 12 January 2026, not peer-reviewed) reports nondeterminism persisting at temperature 0.0 in both deployment types it tested, one API-served model and one open-weight model run locally on vLLM with fixed seeds, at unique-output fractions of 0.24 and 0.09 on exact repeats. Moving the weights onto your own hardware did not buy the property back in that local configuration.

The version pin has a similar character. OpenAI publishes minimum notice periods before it retires a model:

• Generally available models - at least six months of notice.
• Specialized variants - at least three months, covering chat, Codex and deep research variants.
• Preview models - much shorter notice, two weeks being OpenAI’s own example, and OpenAI reserves the right to shorten any of these periods if safety or compliance concerns require a faster timeline.

A minimum tells you how much warning you get before the endpoint moves. For a suite built on repeated identical output:

• Sampling parameters - deprecated on Claude Opus 4.7 and later, with a 400 error where the value used to go.
• Determinism - never promised in OpenAI’s documentation, and measured as incomplete at temperature 0.
• The endpoint - retired on the provider’s schedule. Anthropic retired claude-3-7-sonnet-20250219 on February 19, 2026 and claude-opus-4-1-20250805 on August 5, 2026, defining a retired model as “no longer available for use.” Amazon Bedrock and Google Cloud set their own retirement schedules, so the date differs by platform.

Treat any release check whose green depends on byte-identical repeats as unproven. The work this quarter is finding every check that assumed the pin and deciding it again without one. Re-run one blocking suite with the sampling parameters removed, and record how many assertions still hold on identical input.

Detecting a Backend Move You Were Not Told About

Why identical inputs diverge at all is covered in testing non-deterministic AI outputs. Most of the regression work you already do assumes the cause sits on your side of the wire. A change to the serving configuration breaks that assumption, because nothing in your repository changed on the day the numbers moved.

OpenAI’s docs are direct about this: determinism “may be impacted due to necessary changes OpenAI makes to model configurations on our end,” and the docs point at one field as the way to keep track of those changes: system_fingerprint. When that value differs between two responses, your request is being served by a different configuration than the earlier call.

Capture the fingerprint on every request path, production included.

• Log it on every request - CI runs, local test runs, and production traffic. A fingerprint captured only in your eval pipeline misses the change users actually hit, and production usually sees the move first.
• Put it on the deploy timeline - emit a fingerprint change as an event in the same stream that already carries your prompt edits, retrieval index rebuilds and library bumps, so the provider’s change is visible on the same axis as your own.
• Store it with each run result - the fingerprint belongs in the stored record next to the scores, the prompt hash and the model ID, under the same retention as the run itself. Long after the fact, an investigation needs to know which serving configuration produced which piece of evidence, and a retention window on your logging vendor should not decide that.
• Alert on any change - the value is opaque, so a threshold has nothing to compare against. A single notification saying the serving configuration moved gives the next score drop an obvious first suspect.

The fingerprint reports that the serving configuration changed. What changed, by how much, and whether it touched the behavior your suite measures all stay unknown.

• Magnitude - a reroute to different hardware and a change to decoding produce the same observable, a different opaque string.
• Direction - the field says nothing about whether your outputs got better, worse, or stayed where they were.
• Causality - a fingerprint change and a score drop landing in the same hour is a correlation you still have to test against your stored runs.
• Provider coverage - this is one vendor’s field, surfaced in the guidance quoted above. Other providers may give you nothing equivalent, so your timeline still has to handle silence.

Treat it as a correlation aid that narrows the search. When your suite goes red and the fingerprint changed the day before, the first hour of triage goes to comparing stored run distributions against the new serving configuration. I check the fingerprint before I open the commit log.

Some of this movement is announced in advance.

• Announced retirements - OpenAI publishes minimums of at least six months before retiring a generally available model and at least three months for specialized variants, though it reserves the right to move faster where safety or compliance requires it. Those floors give you planning room.
• Unannounced moves in the serving configuration - no published floor applies, and where a fingerprint exists at all it is the closest thing to a warning.

A Regression Suite Needs a Calendar

Every regression suite I have run fires on a diff, on something a person in the repository changed. A model-backed suite carries a second class of failure that arrives on a date nobody on your team picked. Providers publish those dates, so your pipeline can act on them well ahead of time.

OpenAI’s deprecation documentation commits in writing to minimum notice periods before a model is retired.

• Generally available models - at least six months of notice before retirement.
• Specialized variants - at least three months; OpenAI’s examples are chat, Codex and deep research variants.
• Preview models - anything with “preview” in the name can go on much shorter notice, such as two weeks, which makes it a weak pin for a suite that gates releases; OpenAI advises against preview models for business-critical workloads unless you can migrate on short notice.
• Safety overrides all of it - OpenAI reserves the right to retire sooner when safety or compliance requires, with as much notice as is reasonably possible.

Read every one of those as a floor. The wording is “at least” in every line, so six months is the minimum OpenAI commits to and the actual notice has room to run longer. Planning a migration for exactly six months out treats a minimum as a schedule, which is a common misreading.

Anthropic defines a retired model as no longer available for use, with requests to it failing. Its deprecation page carries dated examples for the Claude API, Claude Platform on AWS and Microsoft Foundry, and the windows below have already closed.

• claude-3-7-sonnet-20250219 - deprecated October 28, 2025, retired February 19, 2026.
• claude-opus-4-1-20250805 - deprecated June 5, 2026, retired August 5, 2026.
• Amazon Bedrock and Google Cloud - both set their own retirement schedules, so the date that binds you is the one for the platform your CI actually calls.

The pin in your config is a dependency with a published expiry, so give that expiry to the pipeline and let a scheduled job hold it.

A schedule that holds an expiry has to survive the weeks when nobody touches the repository, and it has to fire at an hour when someone is around to act, which makes the timezone part of the configuration. TestMu AI’s agent testing platform runs evaluations on a preset daily, weekly or monthly cadence or on full cron syntax with IANA timezone support, so the date check fires in your organization’s local time, and a separate agent update event can start a run when your own agent prompt or underlying model changes. An update event covers the changes your team makes, and a provider’s retirement date is a change made to you, which nothing in your own repository will announce.

• Schedule a date check - a cron job in CI, separate from the diff-triggered suite, that compares the pinned identifier’s published retirement date against today and fails the build once you are inside the window.
• Set the window as policy - choose a period that covers re-baselining, judge recalibration and one full release cycle, write it into the repository once, and let the job enforce it on every run.
• Bring the successor in early - during the deprecation window, run the replacement in a second lane on the same inputs, so you accumulate a churn history across several runs before the cutover.
• Re-baseline on purpose - do it while both models still answer, so item-level movement lands in a report instead of an incident.
• Keep deprecated and retired apart - the gap between the deprecation date and the retirement date is the only period in which both sides of the comparison can be generated, and past that retirement date requests to the old identifier fail, so the old side is gone for good.
• Store the pin with the baseline - keep the model identifier and the platform in the baseline record, because after retirement that record is the only account of the old behavior you have.

Run the date check on the same schedule as your dependency audit, and the migration lands as planned work.

Cost and Latency as Regression Signals

Your suite already records whether each scenario passed, and the scoring methods behind that verdict are covered in LLM evaluation. Cost per scenario and time to first token usually go unlogged, though both come off response metadata for free. They move for the same reasons a graded outcome moves, and neither needs a judge call.

A change can raise spend without flipping a single assertion in the run:

• A prompt edit - one more example or a longer system block adds input tokens to every call, and every answer stays correct.
• A tool definition change - a renamed parameter or a fuller description widens the payload on each turn of an agent loop.
• A retrieval change - a chunker that returns more passages keeps the answer correct and grows context on every request.
• A model swap - the new version may reason longer before it answers, so output tokens and first-token time grow while accuracy holds.

Caching makes the measurement slippery. Anthropic’s prompt cache has a five-minute default lifetime that refreshes each time the cached content is used, and modifying a tool definition name, description, or parameter invalidates the entire cache. A cache hit moves spend and latency while leaving the generated text alone.

• Scenarios that share a prefix and run close together - keep refreshing the same entry, so most of the set reports warm-cache cost and latency.
• A gap longer than five minutes between calls on that prefix - lets the entry lapse, so identical code re-pays for the prefix and reports slower first tokens.
• A queued nightly run against a pull request run - makes scheduling part of the reading, since one starts cold and the other lands warm.
• Any commit that touches tool definitions - starts cold, so the first run after it looks like a cost regression when only cache state changed.
• A warm-or-cold marker on each scenario - lets a reviewer separate a real cost move from a cold start.

Each signal answers a different question about the same run:

Gate cost, latency, and token counts against a stored baseline with a tolerance band, expressed as a ratio to the last accepted run. An absolute ceiling in dollars or milliseconds goes stale the moment a provider changes pricing.

• Band the median across the set - one slow scenario is usually the provider; a shifted median is usually your change.
• Keep the band wide enough to absorb cache state - otherwise the gate fires on scheduling rather than on your change, and the first reviewer to hit that turns it off.
• Rebaseline on accepted cost moves only - when an upgrade raises spend and the team takes the trade, store the new cost and latency baseline with the reason attached.
• Store the baseline next to the suite - a band that lives in someone’s dashboard stops getting updated when the scenarios change.

Triage After a Red Gate

The gate is red at 9 a.m., and the build that tripped it carried a prompt edit, a rebuilt retrieval index, a changed tool definition and a provider alias that moved overnight. Each is plausible, and the order you test them in decides how much of the day this costs you. Run the cheapest discriminator first, where cheap means no suite re-runs.

• First, re-run unchanged - same build, same prompt, same inputs, nothing touched. If the verdict changes while nothing else did, you are looking at variance and the gate sampled too few runs to see it. The peer-reviewed ACM TOSEM study of ChatGPT code generation (DOI 10.1145/3697010) sent five identical requests per task at the provider default and got zero equal test output on 75.76% of CodeContests, 51.00% of APPS and 47.56% of HumanEval tasks.
• Second, read the provider response - the model identifier on the response, and the backend indicator where your provider exposes one. OpenAI’s docs say determinism may be impacted by configuration changes made on their end, and expose the system_fingerprint field so you can see it. One lookup in a log you already write separates a provider-side move from your own change.
• Third, re-run the previous build - the last green commit, untouched, against the endpoint as it stands now. If it fails now too, the change was not yours. Chen, Zaharia and Zou measured GPT-4 at 84% accuracy on prime versus composite identification in March 2023 and 51% on the same questions that June.
• Last, bisect your own inputs - prompt, retrieval context and tooling, one variable at a time. Each candidate costs a full suite re-run, which is why it comes last. On Anthropic, changing a tool name, description or parameter invalidates the entire prompt cache, which otherwise has a 5-minute default lifetime, so cost and latency assertions move even when quality holds.

Record which step resolved it, in the artifact that holds the run. One field and a date is enough. The distribution over time tells you whether your suite is measuring your product or your vendor.

• Mostly re-runs - the gate is under-sampled and the threshold is absorbing noise. An arXiv preprint on MMLU-Pro model-version upgrades (2604.27405, not peer-reviewed) compared greedy single-shot scoring against repeated sampling: it falsely flagged 25% of unchanged items and missed 42% of items that reliably changed.
• Mostly the provider check - the vendor is driving your results, and the follow-up belongs with whoever owns model pinning.
• Mostly the input bisect - the suite is doing its job, and repeat offenders point at the least stable parts of your prompt and retrieval path.

Some cases survive every step: the verdict still flips, and nothing you checked explains it. Quarantine is the right call, and it has to be written down.

• Name an owner - one named person who holds the quarantined case.
• Record what you checked - each step and what it returned, so nobody repeats the morning.
• Set a review date - the day the case returns to the gate or gets removed on purpose, with a reason.

A case that leaves the suite without an owner, a record and a review date is a deletion, whatever the commit message says. Keep the quarantine count beside the pass rate, so a suite hollowing out across releases stays visible while it happens.

Conclusion

Change the shape of the report before changing anything else. Two columns, improved and degraded, costs nothing to produce and answers a question the single number never could.

Then decide which cases deserve repeated sampling, because that decision is where the budget goes and it should follow what breaks loudly in production rather than what is convenient to run.

Put the retirement dates in the build last. A model identifier that stops existing is the one regression with a published date on it, and there is no reason to meet it by surprise.