Datasets for Training and Evaluating RAG Models Compared

What Makes a Dataset Useful for Training or Evaluating a RAG Model?

A RAG dataset is useful when it lets you score the retriever and the generator separately. That needs a corpus to search, questions, labels that say which passages answer each question, and reference answers, and most public datasets for training and evaluating RAG models ship only some of these parts.

• Corpus - the documents the retriever searches. MultiHop-RAG ships its news articles as a separate corpus config, while nq_open ships questions and answers only, so you supply the Wikipedia passages.
• Questions - written by crowdworkers, drawn from search logs, built from templates (PopQA turns each Wikidata fact into a question) or generated by a model.
• Relevance labels - the passages that answer each question, which make retrieval recall and precision computable. MS MARCO flags each of a query's candidate passages as selected or not.
• Reference answers - the expected output, plus an explicit label when no answer exists, as in SQuAD 2.0.

Relevance labels also give you a baseline to beat. The BEIR benchmark paper collected 18 publicly available retrieval datasets and reports that "BM25 is a robust baseline", so run BM25 on the same questions before you trust a dense retriever's number.

Which parts you need depends on the score. The formulas on the RAG evaluation metrics hub show which scores need a reference answer and which need only the retrieved context.

Which RAG Datasets Have Training Splits and Which Are Evaluation Only?

Training data teaches a retriever, a reranker or a generator; evaluation data is held out to measure it. In Dense Passage Retrieval, Karpukhin and co-authors trained a retriever whose "embeddings are learned from a small number of questions and passages" and beat a Lucene BM25 system by 9% to 19% absolute on top-20 passage retrieval accuracy.

Generators can be trained for retrieval too. RAFT trains a model with distractor documents in its context and reports gains "across PubMed, HotpotQA, and Gorilla datasets", so one public set can train a model in one project and evaluate a different model in another, but it should never do both for the same model.

Split names on the Hub do not always say which job a set was built for:

• Evaluation sets published as train - in the script's output, MultiHop-RAG's 2,556 queries and TechQA-RAG-Eval's 910 rows sit in a split named train, although both cards describe evaluation sets.
• One split, three sources - PubMedQA has "1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA instances", and on the Hub each of its three configs keeps all of its rows in a split named train; only the 1,000 expert rows carry human labels.
• Test splits with no answers - HotpotQA fullwiki, MS MARCO v2.1 and TriviaQA rc publish test questions without answers, so score them on validation and carve your own holdout from train.
• Two configs, same questions - HotpotQA's distractor and fullwiki configs each hold 90,447 training rows in the dataset viewer's size data the script read, the same questions with different contexts, so adding both counts every question twice.

Never fine-tune on a split you report a score on, and treat a set that has only one split as evaluation data unless its card says otherwise.

How Do Public RAG Datasets Compare on License, Size and Splits?

A Node.js script, run on 2026-10-20, read each set's card from the Hugging Face Hub API and its rows per config and split from the Hugging Face dataset viewer API, caching every response. Its core loop:

const hub = 'https://huggingface.co/api/datasets/';
const dsv = 'https://datasets-server.huggingface.co';
for (const [name, id, config] of sets) {
  const card = await getJson(hub + id);
  const size = await getJson(dsv + '/size?dataset='
    + encodeURIComponent(id));
  // license: card.cardData.license
  // gated, lastModified: card.gated, card.lastModified
  // rows: size.size.splits for one config, or summed
  // over every config for the sets marked +
}

The output below is the script's own. The license column is the card's license tag, and rows come from one config per set, except KILT, RAGBench, ChatRAG-Bench and PubMedQA (marked +), whose configs are separate tasks or sources and are added together. FEVER's rows come from its card because the dataset viewer returned no size for it, and its paper_dev and paper_test splits are left out of the other column because they re-split its labelled_dev rows:

Hugging Face Hub, read 2026-10-20
name          license        cfg  train   other  updated
ms_marco      none             2   809k    202k  2024-01
nq_open       BY-SA 3.0        1    88k    3.6k  2024-03
trivia_qa     unknown          8   138k     35k  2024-01
squad_v2      BY-SA 4.0        1   130k     12k  2024-03
PopQA         none             1      -     14k  2022-12
BEIR scifact  BY-SA 4.0        2      -    1.1k  2026-04
KILT          mit             11   3.1M    102k  2024-01+
hotpot_qa     BY-SA 4.0        2    90k     15k  2025-08
FRAMES        Apache           1      -     824  2024-10
MultiHopRAG   ODC-BY           2   2.6k       -  2024-01
ASQA          Apache           1   4.4k     948  2022-09
FEVER         BY-SA 3.0/GPL3   3   311k     78k  2024-01*
RAGTruth      none             1    15k    2.7k  2024-11
RAGBench      BY 4.0          12    73k     22k  2024-06+
ChatRAG-Bench other           11      -     35k  2024-05+
TechQA-RAG    Apache           1    910       -  2025-05
PubMedQA      mit              3   274k       -  2024-03+
other = validation, dev, test and named eval splits
rows: one config; + = all configs added up
* rows from the dataset card (datasets-server had none)

Train split: 13 of 17; gated: 0; no license tag: 3
No license tag: ms_marco, PopQA, RAGTruth

Test splits checked for answers
  hotpot_qa test    0 of 7405    have one (all rows)
  ms_marco test     0 of 101092  have one (all rows)
  trivia_qa test    0 of 500     have one (sampled rows)

GitHub-only sets (license from the GitHub API)
  MTRAG   Apache-2.0   pushed 2026-10-07
  CRAG    NOASSERTION  pushed 2025-06-14

Thirteen of the 17 Hub sets have a split named train, none is gated, and three carry no license tag. MTRAG and CRAG are distributed through GitHub rather than as Hub datasets, so the script read their licenses from the GitHub API. RAGBench now lives at galileo-ai/ragbench; the older rungalileo/ragbench id redirects there.

BEIR SciFact shows no train split only because the script read its corpus and query lists. Its train and test relevance labels sit in a separate Hub dataset, BeIR/scifact-qrels.

Which RAG Datasets Test Retrieval, Multi-Hop Reasoning, Grounding and Conversation?

Pick by the failure you need to catch, because no single family covers every RAG failure. The FRAMES dataset card describes "824 challenging multi-hop questions requiring information from 2-15 Wikipedia articles", while a retrieval set such as MS MARCO scores each query against its marked passages.

• Retrieval - MS MARCO, Natural Questions and BEIR score whether the right passage reaches the top results. Use them for retriever and reranker work, not to judge long-form or grounded answers.
• Multi-hop reasoning - HotpotQA, FRAMES and MultiHop-RAG, whose card puts the evidence for its 2,556 queries across 2 to 4 documents, although the null queries among them have no answer in the corpus. They catch a retriever that finds one fact and misses the second.
• Grounding and hallucination - RAGTruth labels hallucinated spans in model outputs, FEVER labels claims as supported, refuted or lacking enough information, and RAGBench packs 12 source-dataset configs into one Hub dataset in the script's output.
• Multi-turn conversation - the MTRAG paper says it "contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks" and calls for systems that can handle "unanswerable questions". ChatRAG-Bench is built from 10 existing conversational QA sets and tests unanswerable cases drawn from QuAC and DoQA.
• Dynamic and domain facts - CRAG has 4,409 question-answer pairs across five domains and eight question categories, and its authors report advanced LLMs at 34% accuracy or less on their own, rising only to 44% with straightforward RAG. TechQA-RAG-Eval and PubMedQA cover technical support and biomedicine.

A practical starting pair is one retrieval set for the retriever plus one multi-hop or grounding set for the full pipeline, chosen from the license table further down.

Why Are Public RAG Evaluation Datasets Unbalanced Across Question Types?

Each public set was built for its own research question, so its mix of question types follows that research question. IBM Research's Know Your RAG paper labels each question by what its context supports and states that "popular public Q&A datasets can be heavily unbalanced with respect to these labels". Its Table 3 gives the share of each label among 4,972 to 6,910 labeled questions per set:

The shares come from the authors' LLM labeler run on samples they preprocessed: each HotpotQA question kept a single positive context, which most of them cannot be answered from; PubMedQA rows came from its unlabeled subset and SQuAD 2.0 rows from its training set.

The same check on three evaluation sets came from a second script, also run on 2026-10-20, which read label counts from the dataset viewer and counted hallucination spans row by row in the test split of RAGTruth-processed, a processed copy of the RAGTruth corpus that the wandb organization publishes on Hugging Face. Its output, unedited:

FRAMES test: 824 questions
reasoning type (a question can have several)
  Multiple constraints   549  66.6%
  Numerical reasoning    293  35.6%
  Temporal reasoning     278  33.7%
  Tabular reasoning      236  28.6%
  Post processing        107  13.0%
  more than one type     403  48.9%
Wikipedia articles per question: median 3, max 23
  questions needing 4 or more  225  27.3%
  questions needing 16 or more   4  0.5%

MultiHopRAG: 2556 queries (split "train")
  comparison_query       856  33.5%
  inference_query        816  31.9%
  temporal_query         583  22.8%
  null_query             301  11.8%

RAGTruth test: 2700 responses, 6 models
task       responses  with a hallucination span
  Summary      900      204  22.7%
  Data2txt     900      579  64.3%
  QA           900      160  17.8%
  all         2700      943  34.9%

• FRAMES - in the script's output, 48.9% of questions carry more than one reasoning label and only 13.0% need post-processing, so one accuracy number blends very different tasks. The script also found four questions that link more than 15 Wikipedia articles, above the 2 to 15 range the card gives.
• MultiHop-RAG - refusal behavior rests on the 301 null queries the script counted, so a small change in refusals moves the overall score.
• RAGTruth - in the script's span counts, a hallucination detector meets hallucinations in 17.8% of question-answering responses and 64.3% of data-to-text ones, so its score over the whole split depends on the task mix.

Report a score per label next to the overall number, and reweight the labels to the mix your own users send before comparing two pipelines.

How Does Contamination Inflate Scores on Public RAG Datasets?

Contamination means the model saw the questions or their source documents during pretraining, so it can answer without retrieving anything and the RAG score credits the retriever for the model's memory. The general case, with figures for model benchmarks, is covered in LLM benchmarks; for RAG sets the damage is that retrieval quality disappears from the number.

The FRAMES paper names "the potential for pretraining data contamination" as a limitation, since its questions come from Wikipedia. Its baseline reached 0.408 accuracy without retrieval and 0.66 with the authors' multi-step retrieval pipeline, so retrieval added the difference of 0.252, and the 0.66 on its own overstates what retrieval adds.

Popularity matters as well. The PopQA paper found that "unassisted LMs remain competitive in questions about high-popularity entities" across its 14k questions, while models "struggle with less popular factual knowledge".

Date windows help only for a while. The MultiHop-RAG paper picked news published from September 26, 2023, to December 26, 2023, because "This timeframe extends beyond the knowledge cutoff of some widely-used LLMs, including ChatGPT and LLaMA, as of the time of writing". A model trained after 2023 may have read those articles.

• Closed-book run - ask every question with retrieval off and report the RAG score next to it; a small gap means the set measures memory.
• Date check - compare the newest document in the set with the model's training cutoff, and prefer sets built after it.
• Popularity slices - split long-tail entities from popular ones, as PopQA's page-view field allows, and read retrieval gains on the long tail.
• Private slice - hold back questions written from your own corpus that never appear in prompts or fine-tuning data.

Which RAG Datasets Allow Commercial Use?

The license tag on a Hub card is metadata someone typed, and for several sets it disagrees with the card text or the source site, as read on 2026-10-20. A Creative Commons CC BY-SA 4.0 license, the tag on HotpotQA, SQuAD 2.0 and BEIR SciFact, allows use "for any purpose, even commercially", but "If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original".

Treat this table as a reading of the cards, not legal advice. Before a public set touches a commercial model or product, send the license from the dataset's source page to whoever approves licenses in your company, and record the version you used.

When Should You Build Your Own RAG Evaluation Dataset?

Build your own when your corpus is private, when your users ask a different mix than the public sets, when no license fits, or when the system acts on what it retrieves. Public sets stay useful as a second opinion; the decision gate should run on your questions and your documents.

Building, sizing and versioning a reviewed set is covered in the golden dataset hub. The additions below are specific to retrieval:

• Unanswerable questions - include questions your corpus cannot answer. SQuAD 2.0 added "over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones", and a strong system fell from 86% F1 on SQuAD 1.1 to 66% on SQuAD 2.0.
• A quota per question type - generate or collect questions per IBM label rather than with one prompt; the Know Your RAG authors found that questions from a simple prompt strategy were almost all single-fact.
• Reviewed synthetic questions - the FRAMES authors found a high proportion of hallucinated questions and answers when an LLM wrote them, and switched to human annotation.
• Gold passages tied to a corpus snapshot - store the document version each label points to, so a re-index does not break retrieval scores silently.
• Labels your metrics need - decide which Ragas metrics or other scores you will run before labeling, because some need a reference answer and some need only the retrieved context.

What Do RAG Datasets Miss When Agents Act on Retrieved Documents?

Every set in the comparison stops at text: an answer string, a passage id or a hallucination span. A support agent that reads a returns policy from its knowledge base and then cancels an order is judged by the order record, and none of these datasets holds one. Agents built this way are what TestMu AI's Agent Assurance is for: it puts them through generated scenarios before release.

Rook CLI, the terminal client for Agent Assurance, is pre-alpha software that anyone can install from npm with Node.js 20 or later:

npm install -g @testmuai/rook

Packages are published for macOS and Linux, and 64-bit Windows through npm or WSL. The two Unix-like systems can skip npm with Homebrew or a shell installer that verifies checksums, as the Rook CLI installation docs explain.

Point it at the agent's codebase, its spec or the folder of documents it retrieves from. Agent Assurance turns those into scenarios, each with acceptance criteria, then invokes the agent itself against a staging deployment or inside CI and settles each criterion from evidence:

• Tool calls - the tools the run actually invoked, set against the agent's declared toolset and any tool a criterion forbids.
• Records - a read-only lookup, through an MCP tool you supply and approve, shows whether the order really changed.
• Written output - files the run created or changed inside the paths named in its profile.
• Model judges - a model judge scores some criteria, yet an action the agent merely reports is never treated as proof.
• Hallucination scenarios - hallucination is one of the adversarial categories, which are generated by default next to the functional ones.

Verdicts come per criterion in three values, Pass, Fail and Unable to Verify, and only scenarios that reached a decided verdict enter the scenario pass rate. Use staging data for every run, since the writes an agent makes during a test are real and nothing undoes them.

Most eval and observability products score the text an agent returns plus its logged trace, whereas Agent Assurance looks at what changed in files, calls and records and lists every criterion it could not settle. The public sets above keep their job, because Agent Assurance is neither a model benchmark nor a retriever leaderboard, and production monitoring is outside its scope.

How Do You Pick a RAG Training Set, Evaluation Set and Baseline?

Pick one retrieval set with a real train split and one evaluation-only set whose license fits your use, score training sets on their validation splits and evaluation-only sets on the split they ship, and put the closed-book score next to every RAG score. Break the result down by question type before you compare two pipelines. Those habits turn public datasets for training and evaluating RAG models into numbers you can defend.

If your system acts on what it retrieves, write the end state into each criterion; the Agent Assurance scenarios docs show how criteria and verdicts are laid out.