Eval-Driven Development: Write the Eval Before the Change

What Is Eval-Driven Development, and What Does It Add to Evals?

Eval-driven development (EDD), also written eval driven development or evaluation-driven development, is a way of building LLM apps and agents in which every change to a prompt, model, retrieval setup or tool begins with an eval: the inputs, the scorer and the bar the output must clear. The change ships only when that eval moves the way the spec says and nothing it guards gets worse.

Google's web.dev Learn AI course, in its evaluation-driven development lesson, describes it as "test-driven development (TDD), adapted for the uncertainty of AI". Its loop is to "define the problem, initialize a baseline, evaluate, and optimize", repeated until the work is done.

Without that order, fixes pile up inside the prompt. In the SPADE paper on data quality assertions for LLM pipelines, Shankar and colleagues observed that developers "often identify data quality issues during prototyping prior to deployment, and attempt to address them by adding instructions to the LLM prompt over time", and they tested a way to mine those prompt histories into checks on nine real-world LLM pipelines. Eval-driven development writes each such issue down as an eval item before the instruction goes in.

Having evals is not the same as eval-driven development, which adds these habits:

• The eval comes first - the target and guards are written down before anyone edits the prompt, so the numbers after the change are read against a bar nobody could move to fit them.
• The eval decides - a change that misses the target or trips a guard is blocked, however good a few sample answers look in review.
• Misses become items - every failure found later, in review or in production, is added to the set as an item that must fail first.

The parts of an eval itself, its dataset, graders and harness, are covered in the guide to AI evals.

Eval-Driven Development vs TDD, BDD and Spec-Driven Development

All four write the check before the work. Martin Fowler's Test Driven Development entry credits TDD to Kent Beck in the late 1990s and sums up its steps as "Red - Green - Refactor". Dan North's Introducing BDD, first printed in Better Software magazine in March 2006, adds that "Acceptance criteria should be executable".

The EDDOps paper by Xia and colleagues at CSIRO's Data61 (arXiv 2411.13768, version 3 of November 2025) explains why classic TDD and BDD fall short for LLM agents: in its words they rely on "static requirements and executable oracles", use binary pass/fail assertions, focus on pre-deployment validation, and give "limited support for emergent behaviors".

• Keep TDD for deterministic code - parsers, tool argument schemas and routing rules still get test-driven development, because a unit test is cheaper and never varies.
• Use BDD scenarios as eval inputs - the Given and When lines of a behavior-driven development scenario make good eval inputs, while the Then line usually needs a scorer instead of an exact match.
• Pair EDD with spec-driven work - in spec-driven development an agent writes the code, and an eval-first gate on the agent's prompts and tools decides whether its next change ships.

For the classic methods side by side on ordinary code, see TDD vs BDD.

The Eval-Driven Development Loop, Step by Step

The EDDOps paper's process model runs Define Evaluation Plan, Develop Evaluation Test Cases, Conduct Offline and Online Evaluations, then Analyze and Improve.

Kent Beck's Canon TDD, published in December 2023, keeps each pass small: "Turn exactly one item on the list into an actual, concrete, runnable test", then change the code until that test and every earlier one pass. Applied to one change to an LLM system, the two give a red-then-green loop:

• Write the eval spec - name the behavior you want, the inputs and slices that show it, the metric, the gain that counts and the guards. Commit it before touching the prompt.
• Run it on the current system - the target check must fail. If the baseline already passes, the spec does not describe the change, so rewrite it before going on.
• Record the baseline - store scores per slice with the dataset version, judge model, judge prompt and prompt version, so the comparison after the change is like for like.
• Make the smallest change that could pass - one prompt edit, one model swap or one retrieval setting per loop, so a failed guard points at one cause.
• Rerun the same items with the same judge - compare slice by slice and read the guards, not only the overall number.
• Ship or block from the spec - apply the decision rule as written, whichever way the numbers point.
• Promote and tidy - freeze the items the change was built against into the regression set, then shorten or clean the prompt while every check stays green.

Repeat runs for a non-deterministic system are in pass@k vs pass^k, CI wiring and flaky gates in LLM evaluation, and gates for prompt edits in prompt evaluation.

How Do You Write an Eval That Works as a Specification?

Write it so two people reading the spec would make the same ship or block call from the same numbers. Anthropic's guide to defining success criteria asks for criteria that are specific, measurable, achievable and relevant. Its safety example is "Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter."

• Behavior - one sentence on what should change for the user, such as "shipping answers name the carrier and the delivery window", never "better answers".
• Inputs and slices - real inputs grouped by intent, language or risk, with the slice the change targets marked, so a gain in one group cannot hide a loss in another.
• Scorer - the cheapest check that can fail. Use code for formats, required fields and forbidden strings, and a model judge or human grade only for what code cannot read. For work that lands in files or code, an agent judge with read-only tools can check each requirement against them, a method called agent-as-a-judge.
• Target - the direction and size of change that counts, chosen before the run, such as "+3 points on the shipping slice".
• Guards - what the change may not cost: any slice dropping past a limit, answer length, latency, token cost, refusal rate or a safety check.
• Pinned versions - dataset version, judge model and judge prompt, recorded in the spec so a rerun months later scores the same way.

The worked example in the next section reads this spec file before it scores either candidate. Both candidates' results were already public, so the example is a replay that shows the mechanics, not a blind test:

{
  "name": "more-thorough-answers",
  "change": "prompt asks for more detail",
  "baseline": "Mixtral-8x7B-Instruct-v0.1",
  "metric": "AlpacaEval 2.0 win rate",
  "reference": "gpt4_1106_preview",
  "target": { "overall_gain_points": 3 },
  "guards": {
    "max_slice_drop_points": 2,
    "max_median_length_growth_pct": 25
  }
}

How many items each slice needs, and how to version them, is covered in the golden dataset guide.

A Worked Eval-First Loop on Recorded Model Outputs

The recordings come from the length experiment in the Length-Controlled AlpacaEval paper by Dubois and colleagues (COLM 2024), and the AlpacaEval repository publishes them. To test how much answer length moves the judge, the authors prompted models to "Answer with as much detail as possible." or, in a concise variant, to be "as concise as possible" while keeping the necessary information.

A Node script run on 2026-10-19 replays an eval-first loop on those recordings, without calling any model or judge:

• Inputs - the 805 AlpacaEval instructions, sliced by the five source sets they come from: helpful_base, koala, oasst, selfinstruct and vicuna.
• Scores - AlpacaEval 2.0's recorded GPT-4 Turbo judgments of each Mixtral 8x7B Instruct answer against a gpt4_1106_preview reference answer, averaged into a win rate per slice. The overall figures match the repository's leaderboard file to two decimals.
• Length - the median answer length in characters, read from the recorded answers.
• Cross-check - the length-controlled win rate the repository publishes for each variant, printed as "published LC". The repository's code is Apache-2.0 and its data CC BY-NC 4.0.

node edd-gate.mjs eval-spec.json \
  Mixtral-8x7B-Instruct-v0.1_verbose \
  Mixtral-8x7B-Instruct-v0.1_concise

Spec: more-thorough-answers
Baseline: Mixtral-8x7B-Instruct-v0.1
Prompts: 805, judge verdicts recorded

RUN 0  baseline against its own spec
  target overall gain       +0.00  limit  +3.00  FAIL
  guard worst slice         +0.00  limit  -2.00  PASS
  guard median length %     +0.00  limit +25.00  PASS
  verdict: RED (eval can fail)

RUN 1  candidate: verbose prompt
  slice           n    base    cand   delta
  helpful_base  129   11.19   22.04  +10.85
  koala         156   15.30   19.75   +4.46
  oasst         188   18.84   25.98   +7.14
  selfinstruct  252   24.04   29.16   +5.12
  vicuna         80   15.82   20.70   +4.88
  all           805   18.26   24.61   +6.36
  median chars          1355    2021
  published LC         23.69   23.22   -0.47
  target overall gain       +6.36  limit  +3.00  PASS
  guard worst slice         +4.46  limit  -2.00  PASS
  guard median length %    +49.15  limit +25.00  FAIL
  verdict: BLOCK

RUN 2  candidate: concise prompt
  slice           n    base    cand   delta
  helpful_base  129   11.19    7.88   -3.32
  koala         156   15.30   10.95   -4.34
  oasst         188   18.84   13.09   -5.75
  selfinstruct  252   24.04   21.48   -2.56
  vicuna         80   15.82    5.82  -10.00
  all           805   18.26   13.74   -4.51
  median chars          1355     736
  published LC         23.69   22.96   -0.73
  target overall gain       -4.51  limit  +3.00  FAIL
  guard worst slice        -10.00  limit  -2.00  FAIL
  guard median length %    -45.68  limit +25.00  PASS
  verdict: BLOCK

• Run 0 is the red check - the unchanged prompt fails its own target, which shows the eval can fail and that the behavior is still missing.
• Run 1 hits the target and still blocks - every slice rose, the overall gain of 6.36 points clears the bar, and the median answer grew 49.15% against a 25% limit. The published length-controlled rate fell 0.47 points, so the gain was mostly length.
• Run 2 fails both ways - the concise prompt lost 4.51 points overall, and the vicuna slice fell 10.00 points against a 2-point limit.

Neither prompt ships, and the spec did its job before anyone argued from sample answers. The next red step is a better target, such as a length-controlled score or per-item criteria for what "thorough" must include.

Feeding Production Failures Back Into the Eval-Driven Loop

Production failures are the inputs nobody wrote in advance. The authors of the interview study Operationalizing Machine Learning, who interviewed 18 machine learning engineers, describe putting ML into production as "a continual loop" that runs from data collection and labeling to "monitoring of performance drops in production". The EDDOps review found the opposite habit in research: almost all the academic sources it surveyed evaluate on a static set, and before deployment only.

In eval-driven development, a production failure enters the suite through the same red-then-green loop:

• Capture the input, the context the system saw and the output, with personal data removed.
• Write the criterion the output broke, in the same form as the other items in its slice.
• Run the new item on the current version and confirm it fails. An item that passes did not capture the failure.
• Write the fix against that red item, then rerun the whole set with its guards.
• Tag the item with its incident or trace ID and keep it in the regression set for good.

Finding those failures in live traffic is a monitoring job, covered in LLM prompt monitoring.

Failure Modes of Eval-Driven Development

The first risk is Goodhart's law, which Manheim and Garrabrant's paper Categorizing Variants of Goodhart's Law describes as a metric used to improve a system "to an extent that further optimization is ineffective or harmful". Run 1 above is a measured case: the judged score rose while the length-controlled score fell.

• Tuning to the suite - after enough loops, prompts fit the eval items rather than the task. Keep a held-out slice nobody tunes against and score it only at release.
• An eval that was never red - an eval written after the change, or one the baseline already passed, confirms the change instead of testing it. Keep the Run 0 output with the spec.
• Judge drift - a new judge model or an edited judge prompt between baseline and candidate changes the scale. Rescore the baseline whenever the judge changes; LLM-as-a-judge covers calibrating one.
• Averages over slices - the EDDOps review found that most academic sources it surveyed reported only end-to-end metrics. A gate on one overall number lets a slice fall, as vicuna did by 10.00 points in Run 2.
• Eval rot - items keep encoding last quarter's policy, price or tool schema, so the suite punishes correct new behavior. Review the slices a change touches before trusting their failures.
• Moving the bar after the run - lowering a target because the numbers came in short leaves no bar at all. Change a spec only in its own reviewed commit, then rerun the red check.

Model upgrades raise their own version of these risks, covered in LLM regression testing.

Who Owns the Evals on an Eval-Driven Team?

In Who Validates the Validators?, Shankar and colleagues watched nine industry practitioners build evaluators for an LLM pipeline and found that "developers rapidly iterate over criteria", refining them as they read outputs. That puts the criteria with whoever can recognize a good output, not with whoever writes the scorer.

The EDDOps review found AI-only evaluators in most of the academic sources it surveyed, while its industry (grey-literature) sources used hybrid human and AI review as often as AI-only review.

• Product owner or domain expert - writes the behavior and criteria, grades a sample of outputs by hand, and signs off on targets.
• Engineer - builds the scorers and the harness, pins the judge and dataset versions, and runs the red check before every change.
• QA engineer - owns slices, guard metrics, adversarial items and the promotion of production failures into the set.
• Separation rule - the person who edits a prompt does not edit its target or guards in the same pull request.

Eval-Driven Development for Agents That Act, Using Agent Assurance

When an agent books, edits, files or deletes, its reply is a claim and the changed system is the result. The 2024 agent benchmark OSWorld scored the end state, giving each computer task "a detailed initial state setup configuration and a custom execution-based evaluation script".

• Target written as an end state - "invoice 7342 is marked paid and the ledger holds one matching entry", checked in the billing system, never "the agent says it paid the invoice".
• Tool guards - tools the agent must call and tools it must never call for that task, such as a payment tool on an invoice that is already paid.
• Unverifiable results kept apart - a criterion the run could not observe counts as unverified rather than passed or failed, and stays out of the target score.

TestMu AI's Agent Assurance supplies the scenarios and verdicts for this loop when your agent's work lands in files, tickets or databases. It is pre-alpha and publicly installable through Rook CLI, its command-line tool:

npm install -g @testmuai/rook
rook --version

It runs on macOS and Linux, and 64-bit Windows through npm or WSL. Installing from npm takes Node.js 20 or later, and the Rook CLI installation guide also documents Homebrew and a checksum-verifying shell script for macOS and Linux.

• Spec from the agent - from the agent's code, or from a PRD, docs folder or API spec when the code is out of reach, Rook CLI writes scenarios with acceptance criteria for each. Functional and adversarial classes come by default.
• Frozen before the gate - the Agent Assurance CI/CD guide keeps scenario generation out of the release gate and runs only reviewed scenario IDs, with a strict policy that every selected scenario must pass.
• Graded on what changed - a criterion passes or fails on the artifacts the run produced, files changed under the paths you declare, tool calls compared with the tools the agent declares, and record checks through a read-only tool you approve. Some criteria go to a model judge, but the agent saying it acted is never accepted as evidence.
• Red and green across runs - each criterion ends Pass, Fail or Unable to Verify, and Unable to Verify stays out of the pass rate.
• Exit code 0 either way - the pipeline reads each verdict in rook report --json, because a completed run returns exit code 0 whether its scenarios passed or failed.

Most eval and observability tools score what your agent said and recorded. Agent Assurance checks what the run changed, and reports what it could not verify. It invokes the real agent, whose writes are real, so point it at staging; it tests the agent before release and is not a production monitor.

Starting Eval-Driven Development on One Feature

Take the next prompt or model change already on your board and write its spec before the change: one target, a slice guard and one cost guard such as length or latency. Run it on today's version and keep the red output, as Run 0 above did with its +0.00 gain against a +3.00 target.

• Commit the spec and its red output in the same pull request as the change, so reviewers see the bar before the result.
• Block on the spec's rule, even when sample answers look better, and write the next target if the eval turns out to measure the wrong thing.
• If the feature is an agent that acts, keep the Rook CLI report from each run with the change: from the second run it shows which scenarios broke since the previous run and which ones now pass, the red-to-green record this loop asks for.