LLM Hallucination Detection: Methods and Limits

What Counts as a Hallucination?

A hallucination is model output that is not supported by the model's inputs or by any source it had access to, regardless of whether it happens to be true. The defining property is the missing support, not the falsehood.

That distinction decides which fix applies. An answer that faithfully repeats a wrong document is a data problem, and no amount of generation-side detection will catch it. An answer the model composed with nothing behind it is a generation problem, and that is what the methods below are built for.

The research literature narrows the term further. In Detecting hallucinations in large language models using semantic entropy, published in Nature, Farquhar, Kossen, Kuhn and Gal target a subset they call confabulations, which they define as "arbitrary and incorrect generations". The word does useful work: it separates output that varies randomly between runs from output that is consistently wrong because the model learned something false.

Why Is Detection Hard?

Four properties make this harder than ordinary output validation, and each one rules out an obvious approach.

• Fluency is uncorrelated with accuracy - The fabricated citation is formatted exactly like the real one. Surface features that humans use to judge credibility carry no signal here, which is why reading the output more carefully does not scale.
• There is usually no reference answer - In production you are checking answers to questions nobody has answered before. Any method that requires a known-correct output only works offline, on a curated set.
• Correct answers vary too - Two valid responses can share no wording. String comparison therefore flags legitimate variation and misses confident fabrication, which is the worst possible combination.
• Self-reported confidence is unreliable - Asking a model whether it is sure produces an answer generated by the same process that produced the original claim. It is the cheapest signal available and close to the least informative.

The Detection Methods Compared

Five approaches account for most production deployments. They are not alternatives so much as layers with different costs.

The ordering matters more than the choice. Deterministic checks are free and catch the failures that break downstream code, so they belong first regardless of what else you run. Our guide to LLM evaluation covers assembling the datasets these methods score against.

What Is Groundedness?

Groundedness is the degree to which a generated answer is supported by the source material supplied to the model. A grounded answer can be traced back to the retrieved documents; an ungrounded one cannot, whether or not it is true.

It is the most useful metric available for retrieval systems, because it converts an unanswerable question into an answerable one. Instead of asking whether a claim is true of the world, which needs omniscience, it asks whether the claim follows from the passages you handed over, which needs only the passages.

Scoring usually decomposes the answer into individual claims and checks each against the context, using an entailment model or a judge prompt. Three details decide whether the score means anything:

• Claim-level, not answer-level - A paragraph where four claims are supported and one is invented scores well in aggregate and is still wrong. Decompose before scoring or the average hides the defect.
• Entailment, not similarity - Vector similarity rewards an answer that shares vocabulary with the context. Entailment asks whether the context actually implies the claim, which is the question you meant to ask.
• Citation resolution is separate - That a citation exists, points to a real document, and that the document supports the sentence are three checks. Systems commonly pass the first two and fail the third.

Groundedness says nothing about answers with no retrieval behind them, which is its structural limit rather than a tuning problem. For the retrieval half of the same system, our guide to RAG testing covers evaluating what the retriever returned before the generator ever sees it.

How Does Semantic Entropy Work?

Sample the same prompt several times, group the answers by what they mean rather than how they are worded, and measure how spread out those meaning groups are. Wide spread indicates the model is not constrained by anything, which is the condition under which it fabricates.

The Nature paper describes the mechanism as computing "uncertainty at the level of meaning rather than specific sequences of words", which is what separates it from ordinary token-probability confidence. Two answers can share almost no vocabulary and mean the same thing, and a word-level measure treats that as disagreement when it is agreement.

prompt: "Which year did the Mariner 4 flyby return its first images?"

sample 1  ->  "1965"          |
sample 2  ->  "In 1965."      |  one meaning cluster  ->  low entropy  ->  trust
sample 3  ->  "Nineteen 65"   |

prompt: "Which paper first proposed the Zhang-Liu consistency bound?"

sample 1  ->  "Zhang et al., 2016"   |
sample 2  ->  "Liu and Chen, 2019"   |  three clusters  ->  high entropy  ->  flag
sample 3  ->  "A 2021 NeurIPS paper" |

The property that makes it deployable is independence from your task. The authors report that the method "works across datasets and tasks without a priori knowledge of the task, requires no task-specific data and robustly generalizes to new tasks not seen before", so it needs no labelled examples and no reference answers.

The cost is the obvious limit. Several generations per checked output is affordable on a sampled slice or an offline evaluation set, and rarely affordable on every production response, which is why it tends to sit behind the cheaper layers rather than in front of them.

Where Does Each Method Fail?

Every method is blind to something by construction, and knowing which blindness you are buying is the whole of the design decision.

The row worth staring at is semantic entropy against consistent errors. Sampling methods measure whether the model is sure, not whether it is right, and a model can be reliably, confidently, repeatedly wrong. That is precisely the failure a groundedness check catches and a consistency check cannot.

The judge row has a similar trap. Our guide to LLM-as-a-judge covers the bias and reliability work that has to happen before a judge's verdict is worth gating on.

How Do You Build a Detection Layer?

Cheapest first, and route by risk rather than applying one policy to every response.

• Rank your routes by cost of a miss - A wrong dosage answer and a wrong film recommendation do not warrant the same spend. This ranking determines everything below it, and skipping it produces uniform thresholds that are simultaneously too strict and too loose.
• Run deterministic checks on everything - Schema, required fields, forbidden content, and whether each citation resolves to a document that exists. Free, immediate, and they catch the failures that break downstream code.
• Add groundedness wherever there is retrieval - Claim-level, using entailment rather than similarity. This is the highest-value layer for any RAG product and the one most often skipped in favour of a vague judge prompt.
• Reserve sampling for the top risk tier - Semantic entropy on every response is rarely affordable. On the routes where a miss is expensive, several generations is cheap insurance.
• Measure the detector, not just the model - Build a small labelled set of known-bad outputs and confirm the detector catches them. A detector nobody has seen fail is an assumption rather than a control.

Step five is the one teams omit, and it is the same discipline that applies to any check you intend to trust. Our write-up on prompt evaluation covers baselining a check so a change to it cannot silently degrade what it catches.

Conclusion

Take the single highest-risk answer your product generates and write down what evidence would have to exist for that answer to be trustworthy. If the honest answer is a retrieved passage, you need groundedness. If it is that the model should be sure, you need sampling. If it is both, you have just designed your detection layer.

The one thing not to carry away is that any single method covers you. Sampling misses the confidently wrong, groundedness misses what was never retrieved, and judges inherit the failings of whatever model you picked. Coverage comes from layering methods whose blind spots do not overlap.

For running these checks against conversational systems rather than static outputs, TestMu AI's Agent Testing evaluates agent and chatbot behaviour against expectations you define, and the getting started with Agent Testing documentation covers defining those expectations for a first agent. Our guide to LLM testing covers the wider application layer these detectors sit inside.