9 Best RAG Evaluation Tools for 2026

A RAG system fails in two places. The retrieval step can fetch the wrong document, or the model can invent details the document never contained. RAG evaluation tools score these two steps separately, so you know which one to fix.

Both scores are built like any other AI eval: a dataset of inputs, a grader that marks each output, and a pass threshold that decides whether the result ships. LLM benchmarks share that shape, but they compare models on a fixed question set and cannot clear your own pipeline for release.

A single overall score hides which step failed. A support bot can quote a refund policy that expired last year, sound perfectly fluent, and pass every latency and error check; an end-to-end score may flag the answer as wrong, but only a retrieval score shows that the wrong document caused it.

Many teams run no offline evaluation at all. LangChain's State of Agent Engineering survey (1,340 responses, collected from 18 November to 2 December 2025) found 57.3% of respondents running agents in production and 32% citing quality as a top barrier, yet only 52.4% running offline evaluations on test sets. This guide compares nine tools against stated criteria, with repository health pulled from GitHub on 16 October 2026 for the eight that are open source.

What Is RAG Evaluation and What Does It Measure?

RAG evaluation, or RAG evals for short, scores a retrieval-augmented generation system in two halves against a fixed set of test questions: the retriever, on whether it fetched the context the question needs, and the generator, on whether its response stays faithful to that context and addresses what was asked.

Each half needs its own score because the failures are independent. A faithful summary of the wrong documents still scores well on faithfulness, and perfect retrieval does not stop a model from inventing details the sources never contained. Models do this for several reasons, and the guide to what causes AI hallucinations pairs each one with the fix that targets it.

Most tools in this list score some version of these metrics:

• Context precision checks whether relevant retrieved chunks were ranked above irrelevant ones. A low score means your reranker is burying good sources.
• Context recall checks whether the retriever found everything needed to answer. A low score points at chunking strategy or embedding choice, which is where the fix belongs.
• Faithfulness tests every claim the answer makes against the retrieved context, so it is the metric that catches confidently invented policy details.
• Answer relevancy checks whether the response addresses the question actually asked, catching answers that are true and grounded but beside the point.

Most implementations score these with an LLM acting as judge, and the same LangChain survey found both review methods in wide use, reporting human review at 59.8% and LLM-as-judge approaches at 53.3%. The judge model is itself a dependency, and swapping it changes your scores without any change to the pipeline being measured. For the formula behind each score, and the places where frameworks define the same metric differently, see the learning hub on RAG evaluation metrics.

For a broader treatment of scoring non-deterministic systems beyond retrieval, the article on AI agent evaluation covers the metric families that apply once an agent takes actions as well as answering questions, and the roundup of the best AI agent evaluation tools compares the platforms that run those checks.

How We Chose the 9 RAG Evaluation Tools

Tool roundups in this category usually rank the publisher's own product first. This list states its criteria up front, so you can disagree with the weighting instead of guessing at it.

• Documented evaluation logic. Every entry publishes how its metrics are computed, whether as open source or as documented method, because a score you cannot audit is hard to defend to a stakeholder. Eight of the nine are open-source AI testing tools; the ninth is TestMu AI's own commercial platform, included for the deployed-agent layer and flagged as ours wherever it appears.
• Genuine RAG-specific metrics. The tool must score retrieval quality, which generic text scoring leaves untouched.
• Verified repository health. Star counts, most recent commit, and license all come from the GitHub API, with the capture date stated in the maintenance table so you can re-run it.
• A path into CI. Evaluation that only runs in a notebook does not prevent regressions.

The entries below run from retrieval-pipeline metrics through to AI agent validation, so the ordering reflects evaluation layer rather than a quality ranking. TestMu AI appears at the end because it addresses the last layer; its position says nothing about its strength.

What this comparison does not do, so you know where to look elsewhere: it runs no latency or throughput benchmarks, models no cost of ownership at scale, and scores no accuracy comparison between the tools' judge implementations. Repository figures are a snapshot taken on 16 October 2026. Anyone choosing on speed or spend should benchmark against their own corpus, because those results depend on your documents and judge model more than on the tool.

9 Best RAG Evaluation Tools for 2026

2. DeepEval

DeepEval treats an evaluation as a unit test. You define a test case with an input, the retrieved context, and an expected output, then assert that a metric clears a threshold. Anyone who has written pytest assertions can read a DeepEval suite without learning a new mental model.

That framing is what makes it the strongest CI fit in this list. DeepEval's CI guide says a metric that falls below its threshold fails the assertion and turns the build red, which makes evaluation a merge gate instead of a report someone reads later. The project ships under Apache 2.0 and keeps releasing: DeepEval 4.3.0 on PyPI was published on 15 October 2026.

• Best for teams that already gate merges on tests and want RAG quality in the same gate.
• Custom criteria via G-Eval cover requirements the built-in metrics miss.
• Requires a judge model budget per run, which grows with suite size.

In this TestMu Conf 2026 session, Testing What Matters: Evaluating LLM Relevancy with DeepEval, Monika Sharma goes under the hood of those metrics, separating answer relevancy, contextual relevancy, and faithfulness as distinct failure axes, and covering how DeepEval scores each one, where those scores break down, and when a model-graded judge can be trusted.

Note: A sound RAG score cannot confirm that the deployed chatbot quotes a customer's real order status or today's price. TestMu AI Agent Testing's Data Validation compares each fact a chat agent states with the value your own API returns, using read-only lookups, and reports a verdict for every fact. Try TestMu AI free!

pip install agent-testing-cli

agent-testing-cli --project "$TESTMU_PROJECT_ID" \
  --json run \
  --workflow "$TESTMU_WORKFLOW_ID" \
  --suite "$TESTMU_SUITE_ID" \
  --yes

Maintenance and Licensing of Open-Source RAG Evaluation Tools

Feature tables age badly. Repository health tells you whether a tool will still be maintained when your evaluation suite depends on it. Every figure below comes from the GitHub repository API for the eight open source entries, pulled on 16 October 2026 with the script after the table, rather than quoted from vendor pages.

GitHub's API reports the license of Phoenix and Langfuse as NOASSERTION, because GitHub's license detection compares each LICENSE file with a short list of known licenses and matched neither; the table gives what each file states. These figures will drift. The script below needs only Node 18 or newer and makes two unauthenticated GitHub API calls per repository, so you can rerun it and judge against current numbers instead of these:

// repo-health.mjs: Node 18 or newer, nothing to install
const repos = [
  "vibrantlabsai/ragas", "confident-ai/deepeval",
  "Arize-ai/phoenix", "langfuse/langfuse",
  "promptfoo/promptfoo", "truera/trulens",
  "run-llama/llama_index", "comet-ml/opik",
];
const api = "https://api.github.com/repos/";
const get = async (url) => (await fetch(url)).json();
for (const r of repos) {
  const repo = await get(api + r);
  const [c] = await get(api + r + "/commits?per_page=1");
  console.log(repo.full_name, repo.stargazers_count,
    repo.license?.spdx_id,
    c.commit.committer.date.slice(0, 10));
}

Its output on 16 October 2026 gives each repository, its stars, its license, and the date of its last default-branch commit:

vibrantlabsai/ragas 16039 Apache-2.0 2026-02-24
confident-ai/deepeval 18905 Apache-2.0 2026-10-16
Arize-ai/phoenix 11873 NOASSERTION 2026-10-16
langfuse/langfuse 35918 NOASSERTION 2026-10-16
promptfoo/promptfoo 26056 MIT 2026-10-15
truera/trulens 3602 MIT 2026-10-13
run-llama/llama_index 52522 MIT 2026-10-14
comet-ml/opik 22654 Apache-2.0 2026-10-16

One caveat if you compare against the GitHub web UI: the API's open_issues_count field bundles pull requests in with issues. For Ragas it reads 638, which GitHub's search API splits into the 384 open issues and 254 open pull requests quoted above.

The Ragas row carries the most risk. The repository is not archived, but almost eight months without a commit, no PyPI release since January 2026, 254 open pull requests, and a change of repository owner add up to a supply-chain question for a dependency that gates your releases. Apache 2.0 licensing keeps a fork open to you if the gap widens.

How to Choose a RAG Evaluation Tool

Start from the question you cannot currently answer, and let that pick the tool. If that question is not about retrieval at all, a wider list of the best LLM evaluation tools is the better starting point.

• You have no retrieval scores at all, and want them fastest. Start with Ragas for metric breadth, and budget for the maintenance risk in the table above.
• You want a bad retrieval change to fail a pull request. DeepEval, because thresholds map directly onto assertions and exit codes.
• You need to explain one specific wrong answer to a stakeholder. Arize Phoenix, because the trace shows which chunks were retrieved for that exact query.
• You are tuning embeddings and chunking over weeks and need comparisons across runs. Langfuse, for dataset versioning and durable history. If both Phoenix and Langfuse fit, our guide to Langfuse vs Phoenix weighs their licenses, deployment footprint, and where evaluation runs in each.
• Your evaluation owners are not Python engineers, or you need red-teaming in the same tool. promptfoo, for YAML configuration and adversarial probes under MIT licensing.
• You want to learn the RAG triad before committing to a platform. TruLens, for a small readable implementation.
• Your pipeline scores well but users still complain. The failure has moved downstream to the agent, and TestMu AI Agent Testing scores that surface.

If you need both views, run two tools: a metrics library gating the pipeline in CI, and an agent-level evaluation gating the deployed experience. Pass thresholds and the CI gate itself get a full treatment in the guide to RAG testing, while the walkthrough on how to scalably test LLMs covers running these suites at volume.

Testing Retrieval-Driven Actions With Agent Assurance

Every RAG metric in this guide grades text: what the retriever returned and what the model wrote about it. Give the support bot from the introduction a refund tool, and the expired policy it retrieves can now trigger a refund the current policy forbids, an outcome none of those scores can see, because the refund is written to the payments system rather than stated in the answer.

TestMu AI's Agent Assurance, currently pre-alpha and publicly installable, puts an agent like this one through real runs before release and inspects their effects. From the agent's code or a written spec it derives functional and adversarial scenarios (plus non-functional ones when you ask for them), calls the actual agent in a staging environment or a CI job, and grades every acceptance criterion against evidence collected from the run:

• Evidence over the agent's account - criteria are graded from what the run left behind: modified files, generated artifacts, and each tool call checked against the agent's declared tool list. A reply saying the refund went through is a claim to check against those records.
• Record checks that change nothing - to confirm what happened to the refund, Rook CLI reads the record with a read-only tool that you have approved, for example a refund-status lookup served by a separate MCP server. Its docs warn that checking with the agent's own issue_refund tool would create the very refund it is meant to observe.
• A verdict per criterion - Pass and Fail are the decided verdicts; Unable to Verify is the third, used when the run left too little evidence to settle a criterion either way. Those undecided results get their own count, and the pass rate covers decided scenarios only.
• Attacks mapped to published lists - adversarial scenarios are mapped to MITRE ATLAS and to OWASP's Top 10 lists for Agentic Applications (2026) and for LLM Applications (2025).
• Changes between runs - after a retrieval or prompt change, rerun the suite; the report shows how the pass rate moved since the previous run and which scenarios now fail after passing last time, alongside those that were fixed.

Rook CLI is the command-line form of Agent Assurance. Install it from npm and check the version:

npm install -g @testmuai/rook
rook --version

Rook CLI targets macOS and Linux, and 64-bit Windows through npm or WSL, and Node.js 20 or newer is required when you install through npm. The Rook CLI installation guide also documents Homebrew and a shell script for macOS and Linux. Point every run at a staging environment, because runs call the real agent and its tools can write real data.

A CI job should decide pass or fail from the output of rook run --json, checking its ok and halted values and the report totals, not from the exit code, which can be 0 for a finished run whose scenarios failed. The docs page on running Agent Assurance in CI/CD has a gate script that fails closed when those totals are missing.

Claude Code users can add the rook skill with one more command. That installer requires Node.js 22 or newer and adds the skill alone, so set up Rook CLI first; the guide to using Rook CLI with Claude Code walks through both.

npx @testmuai/rook-skill@latest \
  install --agent claude-code

For the support bot from the introduction, a scoped first request might read:

/rook Check the billing support agent in this repo.
It reads refund policy from the knowledge base and
can issue refunds through a tool. With the staging
profile, suggest at most three scenarios in which
an outdated or stricter policy should stop a refund,
and list the writes the agent could make before any run.

Conclusion

Pick the one metric you are not measuring today, faithfulness if you score nothing yet, and run it with a pass threshold in the pipeline that ships your retrieval changes. Then test the deployed agent above that pipeline: the Agent Testing features and metrics guide lists the nine scores TestMu AI gives chat and voice agents and the call metrics it applies to phone agents.