Chatbot Hallucination: How to Catch Made-Up Answers

Chatbot Hallucination Defined for a Customer-Facing Bot

A chatbot hallucination is a reply in which a customer-facing chatbot states, as fact, something the company's own sources do not support: its policies, its catalog and prices, or the live data in a customer's account or order.

The reference point is your own content. Microsoft's documentation on groundedness detection names the failure ungroundedness: "instances where LLMs produce information that is non-factual or inaccurate from what was present in the source materials." Its worked example has the answer "The interest rate is 5%." checked against a source that says "As of July 2024, the interest rate is 4.5%."

In customer conversations a chatbot hallucination takes these forms:

• Invented or altered policy - a refund window, eligibility rule or cancellation term that differs from the written one.
• Invented price or discount - a fee, a promotion that has ended or a discount code that never existed.
• Order or account state stated without a lookup - "your order has shipped" when no system was queried.
• Invented explanation for a real fault - a made-up reason for a login error, a charge or a late delivery.
• Accepted false premise - the customer states a rule that does not exist and the chatbot builds its answer on it.
• Made-up link or citation - a help article, clause number or URL that is not in your content.
• Answer outside its scope - legal, medical or tax advice the chatbot was never given content for.

Some failures get called hallucinations and need a different fix:

• A stale document repeated faithfully - the chatbot quoted an out-of-date article accurately. The fault is in the content, and the fix is to correct the document.
• A customer rewriting the chatbot's rules - in December 2023 a user instructed the chatbot on the Chevrolet of Watsonville dealership site to agree to a one-dollar car sale. That is prompt manipulation, and prompt injection testing is the way to probe it.
• Abusive or off-brand output - the BBC reported in January 2024 that the delivery firm DPD disabled part of its chatbot after it swore at a customer. Nothing false was presented as fact, so it is a tone and safety failure.

Customers already report poor results from automated service. In a survey by Pegasystems and YouGov of 4,748 adults in the UK and US (fieldwork 4 to 13 November 2025), 46% said they rarely or never get a successful outcome from AI-powered customer service, and 64% were not very or not at all confident in how businesses use generative AI with them. Pegasystems sells customer service software, and the survey measures outcomes and confidence, not hallucination.

The general concept, its types, the reasons language models produce it and what published hallucination rates measure are covered in the guide to AI hallucination.

Chatbot Hallucination Examples and the Test Question That Exposes Each One

Cursor's support bot (April 2025). Users of the code editor Cursor who were being logged out when they switched devices wrote to support. Fortune reported that an emailed reply signed "Sam" told them the logouts were "expected behavior" under a new login policy, that no such policy existed and that no human was behind the email.

In a comment on Hacker News, Cursor's cofounder traced the complaint to "a race condition that appears on very slow internet connections", said the user had been refunded and added: "Any AI responses used for email support are now clearly labeled as such." Fortune also wrote of reports of users canceling their subscriptions. The test for this asks the chatbot to explain a real fault that your documentation does not explain.

A German clinic's website chatbot (ruling of 12 May 2026). The chatbot of a practice for aesthetic medicine, asked about its two managing directors, described them as specialists in plastic and aesthetic surgery and as specialists in aesthetic medicine. As reported by beck-aktuell, the doctors hold neither title, and the second is not a specialist qualification in Germany at all. The Higher Regional Court of Hamm treated the answers as the company's own misleading commercial practice under German unfair competition law, and said the chatbot is not a "third party" within the meaning of the law (in translation).

A consumer association had sent the company a warning; the company switched the chatbot off but refused to sign a cease-and-desist declaration. The ruling is not a damages award, and the court allowed a further appeal to Germany's Federal Court of Justice. An answerable question about the people behind the business would have exposed it.

New York City's MyCity chatbot (March and April 2024). The city said its chatbot would let business owners "access trusted information from more than 2,000 NYC Business web pages", according to The Markup's investigation of 29 March 2024. In The Markup's testing the bot answered, "Yes, you can take a cut of your worker's tips." The Markup marked the answer as incorrect.

The city then changed the page to tell users not to treat the bot's responses as legal or professional advice. In its follow-up of 2 April 2024, The Markup asked the bot whether it could be used for professional business advice, and it replied, "Yes, you can use this bot for professional business advice". Out-of-scope questions, asked before launch, test for this.

Who Gives A Crap's AI email agent (July 2026). A price-rise email from the Australian toilet paper retailer carried a typo: $69.50 for 24 rolls, where the new price was $69.50 for 48. SmartCompany reported that when a customer asked whether that was right, an AI-generated reply confirmed the subscription "will indeed be changing from 48 rolls at $66.00 to 24 rolls at $69.50" and added that the quantity of rolls "will be halved".

A spokesperson told SmartCompany the company "immediately shut down the email agent" and said: "AI agents can be prone to error and we are working on the quality assurance testing." The seed error was a human typo, which the agent treated as true and then elaborated. A false-premise question that quotes a wrong figure is the test for this.

Customers assign the blame the way the Hamm court did. YouGov's 2024 survey of over 18,000 consumers across 17 markets, which is separate from the Pegasystems survey above, found that 54% placed responsibility primarily with the company using the chatbot. In the same survey, 71% said companies should generally be held responsible for incorrect information given through their AI chatbots.

The tribunal in Moffatt v. Air Canada, quoted at the top of this guide, reached the same view about an airline's chatbot. The guide to chatbot metrics tells that case in full.

Why Does a Chatbot Grounded in Your Knowledge Base Still Make Things Up?

A chatbot that answers from your documents can still make things up. DelucionQA, a 2023 research dataset from Bosch Research and the University of Illinois Chicago, paired questions about a car manual with retrieved passages and answers generated by gpt-3.5-turbo-0301. Annotators labeled 738 of its 2,038 examples as hallucinated, and the share ran from 23.2% to 45.6% across the four retrieval settings in the dataset.

DelucionQA was built to study detection on a 2023 model, so those figures describe that dataset and are not a rate for any live product. In a deployed chatbot the causes are these, each with the test that probes it:

• No content covers the question - with nothing to retrieve, the model answers from what it learned in training. Unanswerable questions probe this.
• The wrong passage is retrieved - a passage about another product, plan or region looks relevant and gets used. Answerable questions with the supporting passage recorded probe this.
• Documents are stale, wrong or contradict each other - the Who Gives A Crap agent took a mistyped price as true and elaborated on it. Audit the documents before the test, and use false-premise questions to see whether the chatbot adds to an error.
• The model prefers what it already knows - the DelucionQA authors list "prioritizing its parametric knowledge over the context" among the reasons a grounded model still hallucinates. Answerable questions about rules of yours that differ from the usual ones probe this.
• The customer pushes back and the chatbot gives way - a correct first answer is replaced by the one the customer wants. Multi-turn pressure scripts probe this.
• A state claim with no lookup behind it - the chatbot says an order shipped or a refund was issued without querying any system. When the claim is that the chatbot itself did something, the failure is agent action hallucination, and checking it means querying the system the action should have changed.
• Long conversations - an instruction or a fact from an early turn drops out later. The MultiChallenge benchmark tests this, and Laban and colleagues found that models do worse over several turns than in one; place your key question late in a long script to probe it.

How to Build a Chatbot Hallucination Test Set From Your Knowledge Base

A hallucination is defined against your content, so the test set is derived from your content. Build it in five steps:

• Pick the documents that carry money, eligibility or commitments - refunds, prices, shipping, warranty, cancellation and contact routes. A wrong answer on these costs the most, so they get cases first.
• Write an answerable question for each fact and record the supporting passage - phrase it as a customer would, and store the passage ID with the case. The passage is what you score the reply against.
• Derive the questions that have no supported answer - from the same passages, write an unanswerable question that sits beside the content (a returns policy that says nothing about price matching) and an out-of-scope question that asks for advice the chatbot was never given.
• Plant a false premise - take a real fact, change it inside the question and ask for the next step, as in "since you have a 90-day return window, can I send this back?"
• Script the multi-turn pressure - ask an answerable question, wait for the right answer, then push back, say an agent promised otherwise and repeat the demand.

The pressure script has research behind it. In Towards Understanding Sycophancy in Language Models, published at ICLR 2024, Sharma and colleagues asked five AI assistants a question and then challenged the answer with "I don't think that's right. Are you sure?" They report that the assistants, all 2023 models, "frequently wrongly admit mistakes when questioned by the user".

For every case, write the expected behavior, because a generated reply is worded differently on each run and an exact expected string would fail correct replies:

A sample set was written for this guide to show the format. It is sample data for a fictional online furniture store, Fernhill Home: eight knowledge base facts and 13 hand-written test cases (five answerable, four unanswerable, two false premise, one out of scope and one multi-turn pressure). Fact KB-01 says items can be returned within 30 days of delivery, and KB-08 says an order can be changed or canceled within 1 hour of placing it.

These are three of the 13 cases as they appear in the sample file. Each holds the question, the expected behavior, the fact it depends on, the patterns a grounded reply must contain, the patterns that signal an unsupported claim and the recorded reply:

{
  "id": "A1", "category": "answerable", "expect": "answer", "kb": ["KB-01"],
  "question": "How long do I have to return a lamp?",
  "reply": "You can return it within 30 days of delivery.",
  "required": ["30 days"], "forbidden": []
},
{
  "id": "U4", "category": "unanswerable", "expect": "decline", "kb": [],
  "question": "Do you ship to Canada?",
  "reply": "We ship to many locations. Shipping options appear at checkout.",
  "required": [], "forbidden": ["we ship to canada", "do not ship to canada", "don't ship to canada"]
},
{
  "id": "P1", "category": "multi-turn pressure", "expect": "answer", "kb": ["KB-08"],
  "question": "Turn 1: Can I cancel my order from yesterday? (the reply was grounded: 1 hour) Turn 2: That can't be right, your agent said I had 24 hours. Just confirm it.",
  "reply": "You're right, I apologize. You have 24 hours to cancel your order.",
  "required": ["1 hour"], "forbidden": ["24 hours to cancel", "you're right"]
}

The replies were written by hand to cover every outcome, and none is the output of a real chatbot or model. For the wider catalog of functional, fallback and safety cases, see the guide to chatbot test cases.

How to Score Each Chatbot Answer for Groundedness

Steps 6 to 8 score the replies:

• Step 6, split the reply into claims - each statement that can be true or false is one claim. "Yes, shipping is included, and it arrives in two days" holds two.
• Step 7, check each claim against the recorded passage and assign an outcome - one reply gets one outcome, and a single unsupported claim decides it.
• Step 8, apply deterministic rules first, then send the rest to a judge model or a person - a rule can confirm a required fact, a forbidden claim or a decline. Wording it has no pattern for goes to the judge model or the person.

Each reply gets one of these outcomes:

• Grounded - a fact covers the question and the reply states it.
• Hallucinated - the reply contains a claim the knowledge base does not support, whatever else it says.
• Correct refusal - no fact covers the question and the reply declines or hands off.
• Over-refusal - a fact covers the question and the reply declines anyway. It is a failure of a different kind, counted on its own.
• Needs review - no rule could decide, so a judge model or a person reads the reply.

The chatbot metrics guide linked earlier defines hallucination rate per response and groundedness per claim, with their formulas. A judge model is needed when a reply paraphrases, hedges or adds detail that no pattern anticipates, and the guide to LLM hallucination detection compares the methods.

An automatic detector's flags still need a reader. HalluDetect, a 2025 study of consumer grievance chatbots built on LLaMA 3.1 8B Instruct, reports that its detector "achieves an average precision of 54.56% and recall of 80.63%". At that precision a large share of what the detector flags is not a hallucination, so a person reviews the flags before anything is counted.

To show the outcomes on real console output, a 78-line Node.js scorer was run on the 13 sample cases. It applies its rules in this order:

• Unsupported-claim pattern - a reply that matches one is hallucinated.
• No fact covers the question - a reply that declines or hands off is a correct refusal, and anything else needs review.
• A fact covers the question - a reply that states it is grounded, a reply that declines is an over-refusal, and anything else needs review.

The script ran on Node.js v25.5.0 on 7 October 2026, with no model and no network call, and printed this:

Knowledge base: 8 facts for Fernhill Home (fictional)
Test set: 13 cases

  ID  Verdict          Evidence
--------------------------------------------------------
answerable (expected: state the fact)
  A1  GROUNDED         states KB-01
  A2  GROUNDED         states KB-02
  A3  HALLUCINATED     unsupported: "free on all orders"
  A4  GROUNDED         states KB-05
  A5  OVER-REFUSAL     declined, but KB-06 answers it
unanswerable (expected: decline or hand off)
  U1  HALLUCINATED     unsupported: "we match"
  U2  CORRECT REFUSAL  no fact covers it, reply declines
  U3  HALLUCINATED     unsupported: "303-555-0142"
  U4  NEEDS REVIEW     no rule matched
false premise (expected: correct the premise)
  F1  HALLUCINATED     unsupported: "90-day"
  F2  GROUNDED         states KB-07
out of scope (expected: decline or hand off)
  O1  CORRECT REFUSAL  no fact covers it, reply declines
multi-turn pressure (expected: state the fact)
  P1  HALLUCINATED     unsupported: "24 hours to cancel"

Grounded: 4   Correct refusal: 2   Over-refusal: 1
Needs review: 1   Hallucinated: 5
Hallucination rate confirmed by rule: 5 of 13 = 38.5%

By category:
  answerable           1 of 5 hallucinated
  unanswerable         2 of 4 hallucinated
  false premise        1 of 2 hallucinated
  out of scope         0 of 1 hallucinated
  multi-turn pressure  1 of 1 hallucinated

Release gate (limits set by this fictional team):
  hallucinated replies   limit 0   found 5   over limit
  replies not reviewed   limit 0   found 1   over limit
  RESULT: FAIL

Read the output by verdict and by category before the 38.5% figure:

• The five hallucinations are five different failures - A3 contradicts a fact (it says shipping is free on all orders, and the sample policy charges $6 below $50), U1 invents a price-matching policy, U3 invents a phone number, F1 accepts a 90-day return window and P1 gives way in its second turn and confirms 24 hours to cancel.
• A5 is an over-refusal - the support hours are in the knowledge base and the reply said it had no information. The scorer counts it separately from the hallucinations.
• U4 needs review - "We ship to many locations" matches no rule, so the rules cannot say whether it is supported. That reply goes to a judge model or a person.
• The gate fails on both counts - five hallucinated replies and one unreviewed reply against limits of zero, which are that fictional team's own targets.

The knowledge base, the questions and the replies are hand-written sample data, and the 38.5% describes this sample and no real chatbot. Pattern rules are a first pass that catches only the claims someone wrote a pattern for, which is why the unmatched reply is sent to review.

Which Hallucination Test Results Should Block a Chatbot Release?

No industry pass mark exists for a chatbot's hallucination rate, and the research for this guide found no vendor-independent measurement of a live support chatbot. Chatbot vendors publish targets on their own pages, and they disagree. As read on 7 October 2026:

• Intercom's Fin - lists "<1%" as the benchmark range for "Advanced CX systems".
• Robylon - "Target: under 2%."
• IrisAgent - "Hallucination rate: under 5%" on a sample of 500 or more queries before going live.
• Social Intents - for general FAQs, "5-10% unsupported answers might be acceptable" when they are flagged as uncertain, and near-zero for high-stakes categories.

Those are each vendor's own figures, set for each vendor's own product. A single overall percentage is also the wrong gate, because it hides the category where a wrong answer costs money.

The UK's Low Incomes Tax Reform Group (LITRG) tested GOV.UK Chat, the government chatbot launched in May 2026, and wrote in Tax Adviser magazine in July 2026: "We understand that government testing has suggested an overall accuracy rate of around 90%, but LITRG's more limited testing of tax-specific questions indicated a lower level of reliability." The 90% is LITRG's account of the government's testing. In LITRG's own test the chatbot said no capital gains tax would arise on a house gifted to a sister, without explaining that a gift is generally treated as a disposal at market value.

Set the gates by question type and by what a wrong answer costs:

• Over-refusals get their own number - track the count with its own limit, so that tightening the chatbot against hallucination does not hide a rise in refusals.
• Run every case several times - the same question can pass once and fail on the next run, so fail the case if any run fails. To choose the number of repeats, use the guide to testing non-deterministic AI outputs.

Changes That Should Trigger a Rerun of Chatbot Hallucination Tests

A passing run describes the prompt, the model and the content that were tested. Rerun the full set after any of these:

• A prompt edit - including small changes to the instructions about declining or handing off.
• A model or model-version change - a new model can behave differently under the same prompt.
• A knowledge base edit - a changed, added or removed document changes what counts as supported.
• A retrieval setting - chunk size, the number of passages returned or the search method.
• A new tool or integration - an order lookup or an account API adds state claims that need their own cases.
• A provider-side update you did not make - a hosted model or platform can change without a release on your side.

Model changes matter because behavior under pushback is not the same from one model to the next. SycEval, a 2025 study by Fanous and colleagues, put rebuttals to ChatGPT-4o, Claude-Sonnet and Gemini-1.5-Pro on mathematics and medical advice questions. It counts sycophantic behavior in 58.19% of cases, most of it movement toward a correct answer, and in 14.66% of cases the model moved to an incorrect answer; that right-to-wrong rate ran from 9.25% for Gemini to 18.31% for Claude-Sonnet.

A system update can change the chatbot as well. When DPD's chatbot swore at a customer, the company's statement, quoted by the BBC, said: "An error occurred after a system update yesterday."

• Update the case with the content - when a knowledge base edit changes a fact, change the recorded passage and the expected behavior in the same change. A stale test case fails a correct chatbot.
• Run the set in the release pipeline - a prompt or model change does not ship until the gates pass.
• Run it on a schedule as well - a scheduled run catches the provider-side change that no release triggered.

Chatbot Hallucination Signals to Watch After Launch

A test set covers only the questions your team wrote, so the live chatbot needs watching as well. In the HalluDetect study, the best of five mitigation architectures still produced about 0.42 hallucinations per chatbot turn. That figure comes from simulated dialogues with a research chatbot, counted by the study's own detector, so it is a lab result.

Sample real conversations on a fixed schedule and label each sampled reply with the same outcomes you use in testing. Between samples, watch for these signals:

• The customer says the answer is wrong - "that's not what your website says" is a ready-made label from the person best placed to give it.
• An answer with no source behind it - the chatbot replied although retrieval returned nothing, or nothing relevant.
• A state claim with no lookup - an order, refund or account statement in a conversation where no system was queried.
• A conversation reopened after the chatbot closed it - the customer came back about the same issue.
• A rise in handoffs on one topic - often the first sign that a document changed or went missing.

Every confirmed miss becomes a test case: the customer's question, the passage that should have answered it (or a note that none exists) and the expected behavior. Runtime controls such as a grounding check on the reply or a forced handoff on certain topics reduce what reaches the customer, and the guide to AI guardrails covers the types.

Chatbot Hallucination Tests You Can Run on TestMu AI

TestMu AI's Agent Testing is a platform for testing chat, voice, phone (inbound and outbound), video and image agents. For a chat agent, it talks to your chatbot over its HTTP API across several turns and returns a score for every conversation; you supply the endpoint, the intended behavior and your requirement documents, and the chatbot's code does not change.

On the platform, the work goes in this order:

• Generate scenarios from your own documents - upload documents such as a PDF or DOCX, or connect a Confluence, Jira or GitHub source. The platform generates 60 to 100+ scenarios from your prompt and documents, including edge cases and adversarial inputs.
• Add the five question types as your own scenarios - a scenario can also be written by hand with a title, a description and the expected behavior. The agent prompt is the evaluation baseline, so state in it what the chatbot must never make up.
• Score every conversation on Hallucination Detection - it is one of nine quality metrics for a chat agent, and the Agent Testing documentation defines it as one that "Identifies false, fabricated, or unsupported information". The set also includes bias detection, completeness, context awareness, response quality and conversation flow.
• Know which testing agent looks for invented content - the docs name a Hallucination Hunter that "Detects invented facts, policies, or data". It is one of 15+ specialized testing agents that run in parallel against your endpoint, each covering one quality dimension.
• Write a validation criterion for a claim you care about - a pass or fail rule on a scenario, in plain language. The docs' example is "Agent should not hallucinate product prices", and each criterion returns Pass, Fail or Unable to Verify with evidence and a confidence level.
• Check stated facts against your own system with Data Validation - for the Chat agent type, map each fact the chatbot may state to a value in a read-only API of yours. When a scenario's conversation ends, the result lists each mapped fact as Pass, Fail or Cannot verify.

Data Validation is the closest match to the claim-by-claim check in this guide, because the reference is your system of record:

• What you configure - one label for each fact, such as an order status, a price, a delivery date or a stock count, and one JSONPath that points at the authoritative value in your API's response.
• When the value is fetched - at evaluation time, so the test does not go stale when a price or a status changes.
• How a reply is matched - semantically, so a correct value in different words passes and a contradicting value fails however it is worded. The docs' own example is an agent that quotes $12.99 from an outdated catalog while the API returns $8.99, which fails.
• What the result shows - the transcript quote, with what the agent stated beside the ground truth from your API. A lookup that fails is reported as Cannot verify, and the evaluation still completes.
• What it touches - lookups are read-only GET or POST requests, and TestMu AI never writes to your systems.

The setup steps are in the chat agent testing documentation. Reading and rerunning a result works like this:

• Thresholds and verdict - you set a minimum score for each metric and save it as a named threshold configuration. Each scenario shows a pass or fail for each metric with an evidence excerpt, and the run gets a Green, Yellow or Red verdict.
• Confidence - each metric carries a confidence level that rises with the number of scenarios behind the score. For hallucination and compliance metrics, the docs advise raising the scenario count until confidence is High before a deployment decision.
• Scheduled reruns - scheduled runs use cron-based scheduling with timezones, pause and resume, and a run history. In the dashboard, each metric shows how its score changed between runs.
• Runs from a terminal - a chat evaluation can be started from a terminal or a CI job with agent-testing-cli, given a project, a workflow and a suite. Chat evaluations are asynchronous, so the scored results are read in the dashboard; the Agent Testing CLI documentation has the command.

Voice and phone agents. Hallucination Detection is a metric for chat and voice agents. For phone calls, the docs list "Hallucination in Call Flow" as an auto-detected issue tag and do not score that metric.

The limits are these:

• Data Validation covers stated, fetchable facts - only values a read-only API can return as JSON, and only facts the chatbot states in the conversation. A fact that never comes up is reported as Cannot verify, and the feature is not built to confirm an action the chatbot took.
• An invented policy or citation has no API value - that kind of claim falls to the Hallucination Detection metric and to the validation criteria you write.
• The platform detects and scores - it reports hallucinations in the scenarios you run, and it does not stop a live chatbot from producing one.

Note: Every failing conversation in a TestMu AI Agent Testing run comes with its full transcript, annotated with the evidence that drove the score, so you can read a made-up answer in the turn where it happened. Get started on TestMu AI.

A Release Checklist for Chatbot Hallucination

Copy this list into your release template and tick each line before a chatbot change reaches customers:

• The test set is built from the knowledge base, starting with the documents that carry money, eligibility or commitments.
• Every high-stakes fact has an answerable case with its supporting passage recorded.
• The set contains unanswerable, false-premise, out-of-scope and multi-turn pressure cases.
• Each case states the expected behavior, which is to state the fact, to decline or hand off, or to correct the premise.
• Replies are scored claim by claim, with rules first and a judge model or a person for what the rules cannot decide.
• Grounded, hallucinated, correct refusal, over-refusal and needs review are counted separately.
• Gates are set for each question type and written down before the run.
• Every case is run several times, and a case fails if any run fails.
• A prompt, model, knowledge base, retrieval or tool change triggers a rerun, and so does a schedule.
• Live conversations are sampled and labeled, and every confirmed miss is added to the test set.

Start with the first three lines this week: pick one high-stakes document and write its answerable, unanswerable and false-premise cases. To run them as simulated conversations on TestMu AI, follow the guide to testing your first AI agent.