A RAG system fails in two places. The retrieval step can fetch the wrong document, or the model can invent details the document never contained. RAG evaluation tools score these two steps separately, so you know which one to fix.
Why separate them? Because a single overall score hides the real problem. A support bot can quote a refund policy that expired last year, sound perfectly fluent, and pass every latency and error check. The answer is wrong only because retrieval pulled the wrong document, and no end-to-end score would tell you that.
Most teams still skip this. In LangChain's State of Agent Engineering survey of 1,300+ professionals, 57% had agents in production and a third named quality as their top blocker, yet only 52.4% ran offline evaluations. This guide compares nine tools against stated criteria, with repository health verified on GitHub for the eight that are open source.
Key Takeaways
- RAG evaluation tools score a retrieval-augmented generation (RAG) pipeline in two separate stages. They check retrieval and generation on their own, not as one combined score.
- Scoring the stages separately shows which one failed. A single accuracy number tells you the answer is wrong but not why.
- Four core metrics do most of the work. Context precision and recall cover retrieval, while faithfulness and answer relevancy cover the answer.
- The top open-source options in 2026 are Ragas, DeepEval, LlamaIndex, Arize Phoenix, Langfuse, promptfoo, Opik, and TruLens. TestMu AI Agent Testing covers the deployed-agent layer above the pipeline.
- Ragas has the most RAG metrics but a fading maintenance signal, with its last commit on 24 February 2026. DeepEval fits CI best, running evaluations as pytest-style assertions that fail a build.
- Arize Phoenix and Langfuse lead on tracing and observability. promptfoo adds YAML tests with red-teaming, and TruLens is the compact take on the RAG triad.
- Pick by the layer you are failing at. Use Ragas or DeepEval for retrieval, Arize Phoenix or Langfuse for debugging, and TestMu AI Agent Testing for a misbehaving deployed chatbot.
What RAG Evaluation Measures
The two surfaces fail independently, which is why they are scored independently. Retrieval can fetch the wrong documents while generation faithfully summarizes them, or retrieval can be perfect while the model invents details the sources never contained.
Four metrics do most of the work across every tool in this list:
- Context precision checks whether relevant retrieved chunks were ranked above irrelevant ones. A low score means your reranker is burying good sources.
- Context recall checks whether the retriever found everything needed to answer. A low score points at chunking strategy or embedding choice, which is where the fix belongs.
- Faithfulness checks whether the answer is supported by the retrieved context. This is the hallucination metric, and it is the one that catches confidently invented policy details.
- Answer relevancy checks whether the response addresses the question actually asked, catching answers that are true and grounded but beside the point.
Most implementations score these with an LLM acting as judge, which the LangChain survey found running alongside human review in most teams. The judge model is itself a dependency, and swapping it changes your scores without any change to the pipeline being measured.
For a broader treatment of scoring non-deterministic systems beyond retrieval, our guide to AI agent evaluation covers the metric families that apply once an agent starts taking actions, not just answering questions.
How We Chose These Tools
Tool roundups in this category usually rank the publisher's own product first. This list states its criteria up front, so you can disagree with the weighting instead of guessing at it.
- Documented evaluation logic. Every entry publishes how its metrics are computed, whether as open source or as documented method, because a score you cannot audit is hard to defend to a stakeholder. Eight of the nine are open-source AI testing tools; the ninth is TestMu AI's own commercial platform, included for the deployed-agent layer and flagged as ours wherever it appears.
- Genuine RAG-specific metrics. The tool must score retrieval quality, which generic text scoring leaves untouched.
- Verified repository health. Star counts, most recent commit, and license all come from the GitHub API, with the capture date stated in the maintenance table so you can re-run it.
- A path into CI. Evaluation that only runs in a notebook does not prevent regressions.
The entries below run from retrieval-pipeline metrics through to deployed-agent validation, so the ordering reflects evaluation layer rather than a quality ranking. TestMu AI appears at the end because it addresses the last layer, not because it is the strongest or weakest entry.
What this comparison does not do, so you know where to look elsewhere: it runs no latency or throughput benchmarks, models no cost of ownership at scale, and scores no accuracy comparison between the tools' judge implementations. Repository figures are a snapshot dated 21 to 22 July 2026. Anyone choosing on speed or spend should benchmark against their own corpus, because those results depend on your documents and judge model more than on the tool.
9 Best RAG Evaluation Tools for 2026
1. Ragas
Ragas carries the widest catalogue of RAG-specific metrics of any library here, including faithfulness, answer relevancy, context precision, context recall, and noise sensitivity, all documented on the Ragas project site. Its reference-free metrics let you score a pipeline without hand-labelling a ground-truth answer for every question, which lowers the cost of getting a first measurement in place.
Its maintenance signal has weakened, though. Opening the original Ragas repository URL now redirects to a different owner account, where the tracker shows 337 open issues and 180 open pull requests. In the same API pull behind the maintenance table below, its most recent commit on the default branch was 24 February 2026, while the other five libraries had all committed within the previous 48 hours.
- Best for teams that want maximum RAG metric coverage with minimal setup code.
- Apache 2.0 licensed, so vendoring or forking is straightforward if maintenance stalls further.
- Weakest on CI ergonomics; you write the harness around it yourself.
2. DeepEval
DeepEval treats an evaluation as a unit test. You define a test case with an input, the retrieved context, and an expected output, then assert that a metric clears a threshold. Anyone who has written pytest assertions can read a DeepEval suite without learning a new mental model.
That framing is what makes it the strongest CI fit in this list. A faithfulness score falling below your threshold fails the build the same way a broken assertion does, which turns evaluation into a merge gate instead of a report someone reads later. The project ships under Apache 2.0 and remains actively released, with version 4.1.2 published on 12 July 2026.
- Best for teams that already gate merges on tests and want RAG quality in the same gate.
- Custom criteria via G-Eval cover requirements the built-in metrics miss.
- Requires a judge model budget per run, which grows with suite size.
3. Arize Phoenix
Phoenix starts from tracing rather than scoring. Built by Arize as an open-source project, it instruments a RAG application through OpenTelemetry, captures each span across retrieval and generation, then layers evaluation on the collected traces. When a score drops, you can open the specific trace and read which chunks were retrieved for that query.
That ordering suits debugging far better than a scoreboard does. Phoenix was actively committed at capture time, as recorded in the maintenance table below, and it runs locally in a notebook or as a persistent server.
- Best when you need to explain why a specific answer was wrong, not just how often answers are wrong.
- OpenTelemetry compatibility means traces can flow into observability tooling you already run.
- Ships under the Elastic License 2.0, which permits internal use but restricts offering the software itself as a managed service. Worth a legal read if you resell tooling.
4. Langfuse
Langfuse is the largest project in this comparison by community size, as the maintenance table below shows. Its centre of gravity, per the Langfuse platform documentation, is trace storage, dataset management, and prompt versioning, with evaluation running against stored traces instead of standing alone as a scoring library.
The practical consequence is that Langfuse answers questions across time. You can compare a dataset run from this week against the same dataset from last month and see which change moved the numbers, which matters when embedding models and chunking strategies are being tuned in parallel.
- Best for teams that need durable evaluation history and self-hosting.
- Dataset versioning makes regression comparisons reproducible across releases.
- Fewer prebuilt RAG-specific metrics than Ragas, so expect to define more scoring yourself.
Note: RAG scores tell you the pipeline is sound. They do not tell you the deployed chatbot holds context across turns or refuses a prompt injection. TestMu AI runs 15+ specialist evaluators against your live agent and returns a Green, Yellow, or Red production-readiness verdict. Try TestMu AI free!
5. promptfoo
promptfoo replaces Python test files with declarative YAML. In the promptfoo configuration model you describe providers, test cases, and assertions in configuration, then run the suite from the command line, which suits teams who want evaluation owned by people who do not write Python daily.
Its second capability is adversarial. promptfoo ships red-teaming that probes for prompt injection and data leakage, so the same tool that scores retrieval quality can also attack the system. It carries MIT licensing alongside same-day commit activity, as recorded in the maintenance table below.
- Best for config-driven evaluation that non-Python contributors can maintain.
- MIT licensing removes most legal friction in regulated environments.
- RAG metric depth trails Ragas, so complex retrieval scoring needs custom assertions.
6. TruLens
TruLens is built around the RAG triad: context relevance, groundedness, and answer relevance, the three scores the TruLens documentation treats as its core. It is the smallest project here by community size, and it is deliberately narrow, which makes it readable end to end. That compactness is why teams often use it to understand what the triad means before adopting something larger.
Its feedback-function model lets you attach a scoring function to any step of a chain, so instrumentation is granular without much ceremony. It remained actively maintained at capture time under an MIT license, as the maintenance table below records.
- Best for small teams wanting the triad without a platform to operate.
- Compact enough to read end to end, which helps when explaining scores internally.
- Smallest community here, so expect fewer worked examples for unusual stacks.
7. LlamaIndex
LlamaIndex is the odd one out here: it is a RAG framework first and an evaluation tool second. Because so many teams already build the pipeline inside it, its built-in evaluation modules score the objects the pipeline already produces, with no separate harness to wire up.
Scoring splits cleanly across the two layers. Response evaluators cover faithfulness, answer relevancy, context relevancy, and correctness against a reference, while retrieval is scored by ranking position with hit-rate and mean reciprocal rank. Those ranking metrics are the retrieval-quality view the pure scoring libraries underweight. The repository is MIT licensed and was actively committed on 21 July 2026 at 51k stars, as the maintenance table below records.
- Best for teams already building on LlamaIndex who want evaluation without adopting a second framework.
- The clearest retrieval-ranking metrics in this list, since hit-rate and mean reciprocal rank score where the right chunk landed, not just whether it appeared.
- Synthetic question generation builds an evaluation set from your own documents when you have no labelled test data.
8. Opik
Opik, from Comet, is the open-source answer to hosted evaluation platforms. Its tracing and evaluation stack runs self-hosted under Apache 2.0, combining trace capture with LLM-as-judge scoring in one tool rather than splitting the two across separate services.
For RAG it ships built-in Hallucination, Answer Relevance, and Context Precision metrics, and a PyTest integration drops those checks into an existing CI suite. That combination puts it between DeepEval and Langfuse: judge-based assertions you can gate a merge on, sitting on top of stored traces. It carries 20.8k stars and a commit dated 22 July 2026, as recorded in the maintenance table below.
- Best for teams that want hosted-platform features like trace storage and dashboards without sending data to a vendor.
- Built-in RAG metrics mean you score context precision and hallucination without writing the judge prompts yourself.
- The PyTest hook gates a merge on an evaluation score, much like DeepEval, if pytest already runs in your CI.
9. TestMu AI Agent Testing (Formerly LambdaTest)
TestMu AI Agent Testing does not compute context precision on your retriever, and it will not replace Ragas or DeepEval for scoring a retrieval pipeline. Its AI agent testing platform evaluates the layer above: the chat, voice, or phone agent that your RAG pipeline ships inside. TestMu AI (formerly LambdaTest) builds it, so treat this entry as the one with a vendor interest and weigh it accordingly.
The platform generates 60 to 100+ scenarios from an uploaded PRD, knowledge base, or policy document, then runs 15+ specialist evaluators against the live agent. Chat and voice agents are scored on nine quality dimensions including hallucination detection, context awareness, completeness, and tone consistency, while phone agents add 30+ call metrics. Each result carries a confidence level, so a passing score backed by thin scenario volume is visibly different from one backed by broad coverage.
Runs roll up into a Green, Yellow, or Red production-readiness verdict, and the same evaluations run from CI through the command line:
pip install testmu-a2a-cli
testmu-a2a test \
--agent https://my-bot.com/api/chat \
--spec "Support agent answering billing questions from a policy knowledge base" \
--count 10 \
--format junit \
--output results.xml
JUnit output plugs into GitHub Actions, GitLab CI, and Jenkins without a custom plugin, and the command exits non-zero on failure so a pipeline can gate on it. Setup details are in the AI agent testing platform documentation.
- Ten prebuilt personas probe behaviour that a fixed test set never exercises, including confused, angry, and off-script users.
- Phone agents add call-specific measurement such as first call resolution and containment rate.
- Credits are consumed per evaluation cycle, so broad persona sweeps cost more than a narrow regression run.
Maintenance and Licensing Comparison
Feature tables age badly. Repository health tells you whether a tool will still be maintained when your evaluation suite depends on it. Every figure below was read directly from the GitHub repository API for the eight open source entries, rather than quoted from vendor pages. The first six were captured on 21 July 2026; LlamaIndex and Opik were added on 22 July 2026, with their star counts shown as GitHub's rounded display figure.
| Tool | Stars | Last commit | License | Strongest at |
|---|
| LlamaIndex | 51k | 21 Jul 2026 | MIT | Framework-native retrieval and response evaluation |
| Langfuse | 31,546 | 21 Jul 2026 | MIT core, enterprise dirs separate | Trace history and dataset versioning across releases |
| promptfoo | 23,452 | 21 Jul 2026 | MIT | Declarative YAML suites plus adversarial red-teaming |
| Opik | 20.8k | 22 Jul 2026 | Apache 2.0 | Open-source tracing with built-in RAG judge metrics |
| DeepEval | 16,991 | 21 Jul 2026 | Apache 2.0 | Pytest-style assertions that gate a CI merge |
| Ragas | 14,927 | 24 Feb 2026 | Apache 2.0 | Widest catalogue of RAG-specific metrics |
| Arize Phoenix | 10,646 | 21 Jul 2026 | Elastic License 2.0 | OpenTelemetry tracing for per-query debugging |
| TruLens | 3,450 | 20 Jul 2026 | MIT | Compact RAG triad reference implementation |
These figures are a snapshot, and they will drift. The pull is one command per repository, so you can re-run it and judge against current numbers instead of ours:
for r in vibrantlabsai/ragas confident-ai/deepeval Arize-ai/phoenix \
langfuse/langfuse promptfoo/promptfoo truera/trulens \
run-llama/llama_index comet-ml/opik; do
curl -sL "https://api.github.com/repos/$r" |
jq -r '"\(.full_name) stars=\(.stargazers_count) license=\(.license.spdx_id)"'
curl -sL "https://api.github.com/repos/$r/commits?per_page=1" |
jq -r '" last commit: \(.[0].commit.committer.date)"'
done
One caveat if you compare against the GitHub web UI: the API's open_issues_count field bundles pull requests in with issues, so it reads higher than the number on the Issues tab. The 337 issues and 180 pull requests quoted above are counted separately.
The Ragas row carries the most risk. It remains the most metric-complete library in the category and it is not archived, but a five-month commit gap alongside 180 unmerged pull requests and a change of repository owner is a real supply-chain consideration for a dependency that gates your releases. Apache 2.0 licensing means forking stays available if that gap widens.
A Test We Ran, and What It Missed
The article opened with a chatbot citing an expired refund policy. That scenario is testable, so we tested it with promptfoo, one of the nine tools above, using three cases against the same prompt: context that answers the question, context that does not, and context that answers it from a document marked superseded.
# promptfooconfig.yaml - case 3, the stale document
- vars:
query: How many days do I have to request a refund?
context: >-
Refund Policy (rev. 2019-03, SUPERSEDED): Customers may request
a full refund within 90 days of purchase.
assert:
- type: llm-rubric
value: >-
The answer reports a 90-day window because that is what the
context says.
All three cases passed. This is the run output, captured on 21 July 2026:
[PASS] You have 30 days from the date of purchase to request a full refund.
[PASS] I do not know.
[PASS] Based on the provided policy, you have 90 days from the time of purchase.
Results:
3 passed (100%)
0 failed (0%)
Duration: 25s (concurrency: 4)
The third result is the one worth sitting with. The model answered 90 days from a document explicitly labelled superseded, and the faithfulness check passed, correctly, because the answer was perfectly grounded in the text it was given. A user reading that reply gets a number that has been wrong since 2019.
Faithfulness measures the distance between the answer and the retrieved context. It says nothing about the distance between the retrieved context and the truth. Catching that failure needs retrieval-side scoring, a freshness check on the index, or evaluation of the deployed agent, and the second case shows the flip side working exactly as intended: with no matching context, the model refused instead of inventing a shipping fee.
Two caveats on this run. It used Gemini as both the model under test and the grader behind the llm-rubric assertion, which is the judge-model dependency described earlier, and three cases is a demonstration rather than a benchmark. The config above is the whole test, so it is quick to re-run against your own model and policy documents.
Where RAG Metrics Stop
The first eight entries all score the retrieval pipeline. Some do it offline against a fixed test set, and Phoenix, Langfuse, and Opik also score live traffic through tracing, but the unit of measurement is the same either way: a query, its retrieved context, and the answer returned. That unit is narrower than the product, which is where agentic AI testing takes over and scores the whole deployed system.
A pipeline can post excellent faithfulness and context recall on every query it handles, and the shipped product can still fail:
- Multi-turn context loss, where the agent answers turn one correctly and forgets the account number by turn four. Single-query test sets never exercise this.
- Behaviour under hostile or confused users, which fixed question sets written by engineers rarely resemble.
- Prompt injection reaching the retrieval layer through user input, turning a document store into an attack surface that security testing is built to probe.
- Tone and escalation failures, where the answer is correct and grounded but the interaction still ends badly.
All four are properties of the deployed agent, so they need instrumentation at that layer. The same principle applies when the interface is voice rather than text, covered in our guide to conversational AI testing, and when the failure surfaces in the interface itself rather than the model, covered in LLM UI testing.
How to Choose a RAG Evaluation Tool
Start from the question you cannot currently answer, and let that pick the tool.
- You have no retrieval scores at all, and want them fastest. Start with Ragas for metric breadth, and budget for the maintenance risk in the table above.
- You want a bad retrieval change to fail a pull request. DeepEval, because thresholds map directly onto assertions and exit codes.
- You need to explain one specific wrong answer to a stakeholder. Arize Phoenix, because the trace shows which chunks were retrieved for that exact query.
- You are tuning embeddings and chunking over weeks and need comparisons across runs. Langfuse, for dataset versioning and durable history.
- Your evaluation owners are not Python engineers, or you need red-teaming in the same tool. promptfoo, for YAML configuration and adversarial probes under MIT licensing.
- You want to learn the RAG triad before committing to a platform. TruLens, for a small readable implementation.
- Your pipeline scores well but users still complain. The failure has moved downstream to the agent, and TestMu AI Agent Testing scores that surface.
Most production teams end up running two tools rather than one: a metrics library gating the pipeline in CI, and an agent-level evaluation gating the deployed experience. For teams extending this into broader model testing, our walkthrough on how to scalably test LLMs covers running these suites at volume.
Conclusion
Start by picking a single metric you are not measuring today, most usefully faithfulness, and wire it into the pipeline that ships your retrieval changes. One scored metric in CI beats six scored in a notebook nobody reruns.
Multi-turn memory, adversarial input, and escalation behaviour decide whether users trust the answer, and none of them appear in a retrieval score. TestMu AI's KaneAI test agent covers the surrounding application flows, and Agent Testing scores the conversation itself against hallucination, context awareness, and completeness before release.
Both layers are worth measuring, and the tools above make it cheap to start with either one. For the wider category these sit in, our roundup of AI testing tools maps the fuller landscape.
