Ragas Metrics Explained: Every Metric, Score and Version

What Are Ragas Metrics and How Do They Score a Sample?

Ragas metrics are the scoring classes in Ragas, an open-source Python library for evaluating LLM applications. Each class takes one test sample, such as a question, the retrieved chunks, a response and a reference answer, and returns a score, usually from 0 to 1. Most of them ask a judge model to split text into claims or to label chunks, then compute the score from those labels with fixed arithmetic.

The Ragas GitHub repository now sits under the vibrantlabsai organization; requests to the old explodinggradients/ragas address redirect there, and the code is licensed Apache-2.0. Ragas evaluation metrics come in three forms in the current release:

• Legacy metrics - classes imported from ragas.metrics, the import path in Ragas tutorials and docs written before the collections API. They still load in 0.4.3 and warn on import.
• Collections metrics - classes imported from ragas.metrics.collections, which take the judge model and embedding model in the constructor and score with an async ascore() call that returns a MetricResult.
• Custom metrics - DiscreteMetric, NumericMetric and RankingMetric, prompt-defined metrics that replaced some fixed classes, such as AspectCritic.

For the general retrieval formulas, independent of any library, see the guide to RAG evaluation metrics.

How Ragas Metric Classes Were Counted and Mapped Across Releases

The counts on this page come from a Node script run on 2026-10-20 against a local cache of public Ragas files. It read the metric exports at tag v0.1.21 and at tag v0.4.3, the latest release, starting from the v0.1.21 metrics module, which exports 14 metric classes. Tag v0.1.21 is the last 0.1 release tagged on GitHub, and its documentation is still online; PyPI also lists an untagged 0.1.22.

• Nine unauthenticated GitHub API requests fetched the repository record, the tag and release lists, the latest commits on main and the file trees at four tags; every response was saved to disk before parsing.
• The metrics modules, every collections metric file at v0.4.3 and the documentation sources at that tag came from raw.githubusercontent.com, and the release dates from the PyPI JSON API.
• The script read the public names in each version's __all__ list and, for v0.4.3, the table of deprecated legacy names, then kept only metric classes.
• It parsed each collections class's constructor to see whether it takes an llm or embeddings argument, and whether that argument has a default.
• Renamed classes were paired by hand from module paths and release notes; every other number is the script's output.

== Public metric classes per version ==
v0.1.21 ragas.metrics          14
v0.4.3 ragas.metrics (legacy)  43
v0.4.3 metrics.collections     37

== v0.1.21 -> v0.4.3 collections ==
Same name in collections: 8
Renamed: 4
No collections class: 2

== v0.4.3 legacy -> v0.4.3 collections ==
Same name in collections: 25
Renamed: 11
No collections class: 7

The mapping compares class names, not scoring behavior. A class that kept its name can still score differently across versions, because the v0.4.0 release moved many metrics to a new prompt architecture, so treat a version change as a new baseline.

Ragas Metrics by Category

The table groups the v0.4.3 collections classes under the category headings of the Ragas metrics list source at v0.4.3; the score ranges come from each class's own docstring or constructor. Three collections names have no entry in that list and stay out of the table: QuotedSpansAlignment, AnswerCorrectness, which has its own docs page at that tag but no list entry, and AgentGoalAccuracy, an alias of AgentGoalAccuracyWithReference.

The docs list still shows Aspect critic and Simple Criteria Scoring under General purpose, although neither has a collections class in v0.4.3; both are legacy-only. The rubric classes ask a judge model for a score on a fixed scale against criteria you write, or against a built-in accuracy rubric when DomainSpecificRubrics gets none, the same family of method as G-Eval.

Which Ragas Metrics Need an LLM?

Most of them. The script read the constructor of every class exported by the v0.4.3 collections module: of the 37 exported names, which count the alias AgentGoalAccuracy beside the class it points to, 26 require an llm argument with no default, 1 needs only embeddings and 10 take no model at all.

== v0.4.3 collections: constructor needs ==
LLM and embeddings: 1
  AnswerRelevancy
LLM only: 25
  (AnswerCorrectness: embeddings at default weights)
Embeddings only: 1
  SemanticSimilarity
Neither: 10
  BleuScore
  CHRFScore
  ExactMatch
  NonLLMStringSimilarity
  QuotedSpansAlignment
  RougeScore
  StringPresence
  ToolCallAccuracy
  ToolCallF1
  DataCompyScore

• No model - the string metrics, BLEU, ROUGE and chrF, the tool-call pair, DataCompyScore and QuotedSpansAlignment are deterministic, so they cost nothing per run and give the same score every time.
• Embeddings only - SemanticSimilarity compares the response and the reference by cosine similarity of their embeddings.
• LLM and embeddings - AnswerRelevancy has the judge write questions from the answer, then compares their embeddings with the original question.
• LLM only - every claim-based metric (Faithfulness, ContextRecall, FactualCorrectness, NoiseSensitivity), every rubric class and the topic and goal metrics. AnswerCorrectness also needs an embedding model at its default weights, which give similarity 0.25; it drops that need only when you set the similarity weight to 0.

A judge-based score inherits the judge's errors, so validate the judge on a sample you labeled before you trust its numbers; the guide to LLM-as-a-judge covers how. The deterministic metrics need no such check, but they only measure overlap with a reference string.

Deprecated Legacy Ragas Metrics and Their Collections Names

In v0.4.3, the legacy metrics module keeps 43 metric classes in a table of deprecated names and loads them on demand. Each one issues a DeprecationWarning whose text says the import "is deprecated and will be removed in v1.0" and gives the ragas.metrics.collections path instead. The script found 25 of the 43 with the same name in collections, 11 renamed and 7 with no collections class.

The deprecation itself shipped in Ragas release v0.4.2, whose notes list "refactor: deprecate legacy metrics" next to the migrations of SQLSemanticEquivalence, DataCompyScore, CHRF Score, quoted spans and the multimodal pair to the collections API.

Ragas Release History and Project Activity

The Ragas release history on PyPI showed 88 versions with uploaded files, pre-releases included, when the script read it on 2026-10-20, from the first 0.0.1 alphas of May 2023 to 0.4.3. Five releases arrived between 11 November 2025 and 13 January 2026, then none:

== Release history (cache read 2026-10-20) ==
PyPI versions with files   88
Latest release             0.4.3 2026-01-13
Days since latest release  280
GitHub tags / releases     90 / 89
Last commit on main        2026-02-24
  0.3.9   2025-11-11
  0.4.0   2025-12-03
  0.4.1   2025-12-10
  0.4.2   2025-12-23
  0.4.3   2026-01-13

The v0.4.3 release is dated 13 January 2026, and 280 days had passed since it on the day of the run. Commits continued on main until 24 February 2026, including one titled "perf: optimize average precision algorithm", which no tagged release contains yet.

• Pin the version - a score is only comparable with scores from the same Ragas version and the same judge model, so record both with every run.
• Expect a break at v1.0 - the deprecation message names v1.0 as the release that removes the legacy imports, and the release notes through v0.4.3 give no date for it.
• Read docs with their version - the documentation site keeps several versions online, and the v0.1.21 pages describe classes that no longer exist under those names.

Core Ragas RAG Metrics Worked by Hand on a Sample

The sample below is an invented exchange with an HR assistant. The response adds a claim the documents never make, and the retriever ranks two irrelevant chunks among four:

user_input:
  How many days of parental leave do I get,
  and how early must I apply?

retrieved_contexts (rank order):
  1. Employees receive 20 days of paid
     parental leave.
  2. The cafeteria is open from 8 am to 3 pm.
  3. Submit leave requests at least 30 days
     before the start date.
  4. Parking permits renew every January.

reference:
  Employees get 20 days of paid parental leave.
  Requests are due 30 days before the start
  date. The leave can be split into two blocks.

response:
  You get 20 days of paid parental leave.
  Apply at least 30 days ahead. Unused days
  carry over to next year.

Ragas asks a judge model for every verdict below; here they are set by hand, and the arithmetic follows the v0.4.3 code exactly. The script's output for this sample:

== Leave-policy example, scored by hand ==
Faithfulness   claims [1,1,0]   0.667
Context recall ref [1,1,0]      0.667
Context precision [1,0,1,0]   0.833
Context precision [1,1,0,0]   1.000
Context precision [0,0,1,1]   0.417
Context precision [0,0,0,0]   0.000

• Faithfulness, 2 / 3 = 0.667 - the response holds three claims: 20 days of paid leave (chunk 1 supports it), apply 30 days ahead (chunk 3) and unused days carry over (nothing supports it).
• Context recall, 2 / 3 = 0.667 - the reference holds three claims, and the split into two blocks appears in no chunk, so the retriever missed one claim out of three.
• Context precision, 0.833 - relevance by rank is 1, 0, 1, 0, so precision at rank 1 is 1 and at rank 3 is 2 / 3; their sum, 1.667, divided by the 2 relevant chunks is 0.833.

The v0.4.3 faithfulness code divides the statements the judge marks as supported by all statements, and returns NaN when the judge extracts none. The count of claims is the judge's choice: if it split the first claim into "20 days" and "paid", the same answer would score 3 / 4 = 0.75.

The v0.4.3 context precision code adds 1e-10 to the count of relevant chunks before dividing, so a retrieval with no relevant chunk scores 0.000 instead of failing. Moving the two relevant chunks to ranks one and two lifts the score to 1.000, and moving them to ranks three and four drops it to 0.417, with the same chunks retrieved.

The v0.4.3 context recall code joins every chunk into one context string before the judge classifies the reference claims, so reordering chunks never changes recall. Recall stays at 0.667 in all three orderings above, which is why precision and recall have to be read together.

Agent and Tool-Call Metrics in Ragas

The agent classes in collections are ToolCallAccuracy, ToolCallF1, TopicAdherence and the AgentGoalAccuracy variants. All of them read a conversation made of Ragas message objects, and the tool-call pair needs no model, so it adds no judge cost to a regression run. Tool-call scoring outside any one library is covered under AI agent evaluation metrics.

== Tool-call metrics, strict order (default) ==
Run                      Acc    F1     TP FP FN
A wrong start date       0.750  0.500  1  1  1
B request filed twice    0.000  1.000  2  0  0
C extra notify call      0.000  0.800  2  1  0

Docs extra-call example: P 0.667, R 1.000
  F1 recomputed            0.800
  F1 printed in agents.md  0.67

Limits of Ragas Metrics in RAG and Agent Testing

Every Ragas metric scores text, labels or recorded calls that you pass in. None of the metric classes in the v0.4.3 collections folder inspects a database, a file system or the agent's target systems after the run, so these questions stay outside the scores:

• Whether a call took effect - the tool-call metrics read the calls in the messages, not the record the call was meant to create or change.
• Whether a faithful answer is true - faithfulness checks support in the retrieved chunks, so a stale chunk produces a faithful wrong answer. The RAG evaluation metrics guide covers this in its section on factuality versus faithfulness.
• Attacks on the agent - no collections class is named for prompt injection, jailbreaks or data leakage; a rubric class scores only the rubric it is given.
• Retrieval quality without labels - every retrieval metric needs a reference answer, reference contexts or the judge's relevance labels; for grounding without a reference, see groundedness.
• Which library to run - for a comparison of Ragas with other libraries and platforms, see the roundup of RAG evaluation tools.

Checking What an Agent Changed Beyond Ragas Scores With Agent Assurance

Run B is the case to plan for: the tool-call F1 was perfect, the transcript reads like a success, and the employee now has two leave requests. TestMu AI's Agent Assurance is built to catch that before release, by inspecting the state a test run leaves behind. Its command-line tool, Rook CLI, is pre-alpha and publicly installable from npm:

npm install -g @testmuai/rook

Rook CLI builds exist for macOS and Linux, and 64-bit Windows through npm or WSL. Its npm package needs a Node.js runtime at version 20 or above, while Homebrew and the shell installer bring their own runtime on macOS and Linux; all three routes are set out in the Rook CLI installation guide. From the agent's code or spec, Agent Assurance generates scenarios with acceptance criteria, invokes the real agent against a staging deployment or inside a CI pipeline, and decides each criterion from evidence like this:

• Records - a read-only probe, run through a tool you supply and approve, confirming that the leave table now holds exactly one 20-day request starting 2 November.
• Tool calls - every call compared with the tool list the agent declares, so a criterion can also forbid a call outright.
• Files and artifacts - output the run left under the paths its profile names.

In most eval and observability tools, the score comes from an agent's reply text and the calls logged in its trace, and with an eval library the record itself is reached only by a check scripted for each task. In Agent Assurance, a claimed action is never proof: the criterion is decided by what the run changed.

The verdicts are Pass, Fail and Unable to Verify; the last marks a criterion the evidence could not settle, and it is left out of the pass rate rather than counted as a failure. Test runs make real writes that nothing reverses, so aim them at a staging copy of the HR system. Agent Assurance is not a production monitor, and it does not intercept calls at runtime.

How to Migrate Ragas Metrics to the Ragas Collections API

If you maintain a Ragas suite, start by running it with warnings shown and listing every class that issues the v1.0 deprecation warning. Map each one with the renamed-classes table above, and rebuild the legacy-only classes as custom metrics or plain code. The Ragas v0.3 to v0.4 migration guide lists the call change: single_turn_ascore() on a SingleTurnSample returned a float, and ascore() takes keyword arguments and returns a MetricResult with .value:

# Before: legacy import, warns on v0.4.3
from ragas.metrics import Faithfulness

# After: collections API, the LLM is passed in
import asyncio
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

async def main(question, answer, chunks):
    llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
    metric = Faithfulness(llm=llm)
    result = await metric.ascore(
        user_input=question,
        response=answer,
        retrieved_contexts=chunks,
    )
    print(result.value)

# question, answer and chunks come from your dataset
asyncio.run(main(question, answer, chunks))

Before switching, score one frozen dataset with the old and new metric classes on the same judge model and keep both baselines, so a score change caused by the migration is not read as a change in the system under test. If the agent behind your RAG system files requests or changes records, each of those actions still needs an end-state check of its own, because no Ragas score reads the record.