What Is a Golden Dataset? How to Build, Size and Version One

What Is a Golden Dataset?

A golden dataset, also written golden data set, is a small, versioned collection of real inputs, each paired with human-reviewed acceptance criteria or a reference answer, that an AI system is scored against on every change to its prompt, model, retrieval or tools. It is test data: none of it is used to train or prompt the system.

Teams also call it a golden set, gold set or reference set. The term is informal, with no standard definition, so check what a vendor means by it before comparing numbers.

Microsoft's Producing Golden Datasets guidance, added to its promptflow-resource-hub repository in November 2023 for Copilot apps, describes one as "a collection of realistic customer questions along with expert answers". It also states that golden datasets "are not used as source data to train or fine-tune an LLM or inject context into an LLM prompt".

A folder of test prompts becomes a golden dataset when it has these properties:

• Reviewed - one person wrote each item's criteria and a second checked them, so a failure points at the system instead of at a disputed label.
• Frozen and versioned - every run records the dataset version it scored, so two runs on one version compare item by item.
• Held out - no item appears in a prompt, a few-shot example or training data, so a pass shows how the system handles an input it has never seen.

When AI evals read from a set like this, the grader, the metrics and the release gate all score the same items, so one wrong item distorts every number built on it.

Golden Dataset vs Ground Truth, Benchmark, Test Set and Golden Record

Golden record and golden master sound like golden dataset but come from outside AI testing.

The MMLU paper says its test "covers 57 tasks including elementary mathematics, US history, computer science, law, and more", and the SWE-bench paper introduces 2,294 software engineering problems "drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories". A high score on either ranks a model against other models, and neither contains your users' questions, your policies or your tools.

What Goes Into a Golden Dataset Item?

Every golden dataset item has three parts: the input, the check that decides pass or fail, and metadata recording where the item came from and who stands behind it. Microsoft's guidance treats question and answer text as the minimum and advises adding "any references to the source documents".

{
  "id": "ans-0142",
  "type": "answer",
  "input": "Can I get a refund 40 days after renewal?",
  "must_include": ["30 days"],
  "must_not_claim": ["refund approved"],
  "reference": "The refund window is 30 days.",
  "slice": "refunds", "risk": "high",
  "source": "production", "status": "active",
  "author": "rk", "reviewer": "ms",
  "created": "2026-09-02", "reviewed": "2026-09-09",
  "added_in": "1.0.0"
}
{
  "id": "rag-0031",
  "type": "rag_answer",
  "input": "Which plans include single sign-on?",
  "must_include": ["Enterprise"],
  "must_not_claim": ["SSO on the Free plan"],
  "expected_sources": ["kb/plans.md#sso"],
  "slice": "plans", "risk": "medium",
  "source": "production", "status": "active",
  "author": "rk", "reviewer": "jt",
  "created": "2026-09-03", "reviewed": "2026-09-10",
  "added_in": "1.0.0"
}
{
  "id": "ext-0207",
  "type": "extraction",
  "input": "files/invoice-2231.pdf",
  "expected": {"invoice_no": "INV-2231",
    "total": "1840.00", "currency": "EUR",
    "due_date": "2026-07-31"},
  "match": "exact",
  "slice": "invoices", "risk": "high",
  "source": "expert", "status": "active",
  "author": "pv", "reviewer": "ms",
  "created": "2026-09-05", "reviewed": "2026-09-11",
  "added_in": "1.1.0"
}
{
  "id": "act-0019",
  "type": "agent_action",
  "input": "Cancel order 5521 and email the customer.",
  "end_state": [
    {"read": "orders.status", "key": "5521",
     "expect": "cancelled"},
    {"read": "outbox.count", "key": "cust-77",
     "expect": 1}],
  "must_not_call": ["issue_refund"],
  "verifier": {"tool": "orders_db.query",
    "access": "read_only", "env": "staging"},
  "slice": "orders", "risk": "high",
  "source": "incident", "status": "active",
  "author": "pv", "reviewer": "jt",
  "created": "2026-09-08", "reviewed": "2026-09-12",
  "added_in": "1.1.0"
}

id,slice,risk,input,criteria,author,reviewer,agreed,status

How Do You Build a Golden Dataset?

Each step ends with an exit criterion: the evidence you can show a colleague that the step is done.

• Name the release decision the set gates - Write one sentence, such as "block any prompt change that makes refund answers worse", and name its owner. The decision sets which slices you need and how small a drop the set must catch.
  Exit criterion - the decision, owner and pass rule are in the dataset's README.
• Pull candidates from production logs, tickets and incidents - Collect real inputs, strip personal data and record each one's source. Microsoft's guidance also puts realism first: start from "the most realistic questions you'd get from your non-expert customers" before looking at the answers your documents hold.
  Exit criterion - a candidate pool larger than the set, with a source on every row.
• Split candidates into slices and risk tiers - Group them by product area or intent and tag each with what a wrong output would cost.
  Exit criterion - every slice meets the per-slice minimum in the sizing section.
• Write acceptance criteria for each item - Required facts, forbidden claims and structure for answers; expected sources for retrieval; exact values for extraction; end states for actions.
  Exit criterion - a second person can decide pass or fail from the item alone.
• Label with two reviewers and record agreement - A second reviewer checks every item without seeing the first verdict, an adjudicator settles disagreements, and the agreement rate goes into the README. Microsoft's guidance makes a second expert's QA pass optional; make it required, because a wrong criterion fails correct outputs on every run until someone finds it.
  Exit criterion - every active item carries two names, and disputed items were rewritten or dropped.
• Add adversarial and edge cases - Inputs that try to override instructions, requests the policy forbids, empty, malformed and very long inputs, and every language your users write in. Tag them as their own slice so they do not dilute the main pass rate.
  Exit criterion - every high-risk tier has adversarial items with a stated pass behavior.
• Freeze version 1.0.0 and record a baseline run - Tag the file, score it with the current prompt, model, retrieval index and grader, and store every item's result as well as the total.
  Exit criterion - a baseline record naming the dataset version and each component version.
• Run the set in CI on every change - Trigger it on each prompt, model, retrieval or tool change, compare every item with the baseline, and fail on a significant paired drop or any high-risk item that flips to fail.
  Exit criterion - every pull request that touches the system shows which items flipped, in both directions.

How Many Examples Does a Golden Dataset Need?

Enough items to detect the smallest regression that would change your decision, counted per slice. Microsoft's guidance for Copilot question-answer apps says "you should typically aim to have about 150 question/answer pairs", and its FAQ adds that "a minimum dataset of 100 question-answer pairs is required". That figure applies to that kind of app; size other systems by the decision they gate.

Each flipped item moves its slice's pass rate by 100 divided by the slice's size, so one failure moves a 10-item slice 10 points and a 20-item slice 5 points.

// paired-gate.mjs: two runs of one golden set, compared
const items = 200;    // golden items scored in both runs
const basePass = 180; // passes in the baseline run
const lost = 12;      // passed in baseline, fail now
const gained = 3;     // failed in baseline, pass now
const candPass = basePass - lost + gained;

function choose(n, k) {
  let r = 1;
  for (let i = 1; i <= k; i++) r = (r * (n - i + 1)) / i;
  return r;
}

// Exact McNemar test: binomial(flips, 0.5), two-sided
function exactPaired(b, c) {
  const n = b + c;
  let tail = 0;
  for (let i = 0; i <= Math.min(b, c); i++) {
    tail += choose(n, i);
  }
  return Math.min(1, (2 * tail) / 2 ** n);
}

// Standard normal CDF (Abramowitz and Stegun 7.1.26)
function phi(z) {
  const x = Math.abs(z) / Math.SQRT2;
  const t = 1 / (1 + 0.3275911 * x);
  const poly = t * (0.254829592 + t * (-0.284496736
    + t * (1.421413741 + t * (-1.453152027
    + t * 1.061405429))));
  const erf = 1 - poly * Math.exp(-x * x);
  return z >= 0 ? (1 + erf) / 2 : (1 - erf) / 2;
}

const flips = lost + gained;
const pExact = exactPaired(lost, gained);
const chi2 = (Math.abs(lost - gained) - 1) ** 2 / flips;
const pChi2 = 2 * (1 - phi(Math.sqrt(chi2)));

// Unpaired view: the two runs as independent samples
const pool = (basePass + candPass) / (2 * items);
const se = Math.sqrt(pool * (1 - pool) * (2 / items));
const z = (basePass - candPass) / items / se;
const pZ = 2 * (1 - phi(Math.abs(z)));

// Fewest flips, all in one direction, with p < 0.05
let d = 1;
while (exactPaired(d, 0) >= 0.05) d++;

const pct = (n) => ((100 * n) / items).toFixed(1) + "%";
const log = (line) => console.log(line);
log("Golden set: " + items + " items, one run each");
log("Baseline   " + basePass + " pass  " + pct(basePass));
log("Candidate  " + candPass + " pass  " + pct(candPass));
log("");
log("Paired view (same items, same grader)");
log("  pass -> fail        " + lost);
log("  fail -> pass        " + gained);
log("  exact McNemar p     " + pExact.toFixed(3));
log("  chi-square (cc)     " + chi2.toFixed(2)
  + ", p " + pChi2.toFixed(3));
log("");
log("Unpaired view (independent samples)");
log("  two-proportion z    " + z.toFixed(2)
  + ", p " + pZ.toFixed(3));
log("");
log("Same " + candPass + " passes, other splits");
for (const [b, c] of [[9, 0], [30, 21]]) {
  const label = "  " + b + " lost, " + c + " gained";
  const p = exactPaired(b, c).toFixed(3);
  log(label.padEnd(24) + "exact p " + p);
}
log("");
log("Fewest one-way flips for p < 0.05: " + d);

Golden set: 200 items, one run each
Baseline   180 pass  90.0%
Candidate  171 pass  85.5%

Paired view (same items, same grader)
  pass -> fail        12
  fail -> pass        3
  exact McNemar p     0.035
  chi-square (cc)     4.27, p 0.039

Unpaired view (independent samples)
  two-proportion z    1.37, p 0.170

Same 171 passes, other splits
  9 lost, 0 gained      exact p 0.004
  30 lost, 21 gained    exact p 0.262

Fewest one-way flips for p < 0.05: 6

Can an LLM Generate a Golden Dataset?

It can draft items, and people still approve each one: a draft stays silver, meaning machine-made and unreviewed, until it passes the same two-reviewer check as a hand-written item.

Microsoft's guidance recommends against LLM-written questions "when trying to measure a quality metric that reflects a real-world customer-experience", and it says "you should never use an LLM answer as-is without proper, deep expert validation".

Generated items are useful for:

• Thin slices - paraphrases of real items bring a small slice up to its minimum while keeping the reviewed criteria of the item each came from.
• Adversarial phrasings - injection attempts, policy-boundary requests and hostile wording that production logs rarely hold in volume.
• Edge-case variants - a real request rewritten with a missing field, a contradiction or another language.

Tag them with source synthetic, cap their share of each slice in the README, and report the pass rate on real items next to the full rate.

Keep golden items out of prompts, few-shot examples and fine-tuning data, as Microsoft's definition states. If a fix depends on one, move it to the training pool, retire it from the golden dataset and write a new item for the same behavior.

How Do You Version a Golden Dataset?

Number dataset versions the way Semantic Versioning 2.0.0 numbers software: increment the major version "when you make incompatible API changes", the minor version "when you add functionality in a backward compatible manner" and the patch version "when you make backward compatible bug fixes". For a golden dataset, compatible means a new score can still be compared with the old ones.

• Pin every run - record the dataset version, model, prompt, retrieval index and grader version with each run.
• Keep a changelog - list what changed, why and who approved it, so a jump in the pass rate can be traced to the set.
• Retire items instead of deleting them - a retired item keeps its row and history, so old runs stay reproducible.
• Keep a held-out slice - run one slice only at release and keep its results away from whoever tunes prompts.

Why Do Golden Datasets Decay, and How Do You Catch It?

A golden dataset decays because the product, its policies and its users change while the items stay fixed: a criterion written around last quarter's refund window fails correct answers after the policy moves. Microsoft's guidance describes the symptom: "significant or continued drift" in scores "might also be an indication that the Golden Dataset is no longer up to date with reality and should be revised".

Re-review the affected slices when one of these triggers fires:

Between triggers, review every slice at least once a quarter, and start with the high-risk slices.

Labels in widely used benchmarks carry errors too. In Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (NeurIPS 2021 Datasets and Benchmarks track), Northcutt, Athalye and Mueller estimated "an average of at least 3.3% errors across the 10 datasets" they studied, with label errors making up at least 6% of the ImageNet validation set.

The same paper reports that on ImageNet with corrected labels, "ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6%". Label errors of that size can change which model a golden dataset tells you to ship.

• Check the item before filing the bug - a failure on a criterion that no longer matches policy is a dataset fix with its own version bump.
• Review items whose result never changes - an item every version passes no longer separates versions, and one every version fails is often stale.

Golden Datasets for Agents That Act: Checking the End State

For an agent that acts, the expected output is a changed record, file or ticket, so its golden items need end-state checks like the order item above. Every tool the agent can call adds paths, each needing its own check and a verifier with read access to the system it changed.

TestMu AI's Agent Assurance tests this kind of agent. It is pre-alpha and publicly installable as Rook CLI:

npm install -g @testmuai/rook
rook --version

Rook CLI installs on macOS and Linux, and 64-bit Windows through npm or WSL. The npm route needs Node.js 20 or newer; the other routes are in the Rook CLI installation guide.

Agent Assurance reads your agent's code or spec and writes scenarios with a check per criterion: functional and adversarial ones by default, non-functional ones when you ask. It invokes the real agent against staging or in CI and grades each criterion on what it can observe:

• Files and artifacts - files the run created or changed on disk, and what the agent produced.
• Tool calls - each call in the run, compared with the set of tools the agent declares.
• Records - read-only checks of the system the agent changed, through a tool you approve.

Each criterion comes back with one verdict: Pass, Fail or Unable to Verify. Only Pass and Fail enter the pass rate. The agent's own report is not proof: a reply saying the order is cancelled passes nothing until the order record shows it.

After a change, a rerun compares its pass rate with the run before and lists the scenarios that started failing and the ones that were fixed. The Rook CLI docs for comparing two saved runs keep changed criteria apart from new failures, for the same reason a major version restarts a golden item's history.

Agent Assurance does its testing before release: it is neither a production monitor nor a runtime guardrail, and it tests your agent rather than the model underneath.

Your First Golden Dataset: Start With One Decision

Start with the next change you plan to ship and the decision it needs. Pull real candidates for its riskiest slice from last month's tickets, write criteria for 20 of them and have a second person review them. Freeze version 1.0.0, record the baseline and add the paired comparison to CI before the change merges.

If the system is an agent that changes records or files, give each item an end-state check, or install Rook CLI (npm install -g @testmuai/rook) and let Agent Assurance derive the scenarios from the agent's code.