Chatbot Metrics: 15 KPIs to Measure Chatbot Performance

What Are Chatbot Metrics?

Chatbot metrics are measurements, mostly ratios, that describe how a chatbot's conversations went: whether the user's goal was met, whether a person had to take over, whether the bot understood each message, what the experience was like and what it cost, and, for LLM chatbots, whether the answers were true to their sources and to the conversation so far.

Every metric is built from the same parts:

• Event definition - what counts as a fallback, a handoff, a resolution or a completed goal.
• Numerator - how many of those events happened in the period.
• Denominator and unit - what the events are divided by, counted in a stated unit such as a session, a bot reply or a claim.

Two dashboards can print different values from the same conversations because they chose different events or units.

The groups in this guide follow a model from 1997. PARADISE, a framework for evaluating spoken dialogue agents by Walker, Litman, Kamm and Abella, models performance as "a weighted function of a task-based success measure and dialogue-based cost measures, where weights are computed by correlating user satisfaction with performance."

Chatbot performance metrics still divide that way. Outcome metrics measure task success; handoff, understanding and effort metrics measure dialogue costs; and CSAT measures satisfaction. Chatbots that generate their answers need one more group, answer quality, which asks whether a reply is supported by its source and consistent with earlier turns.

Chatbot Metrics at a Glance

In the table, numbers 1 to 12 describe how a conversation went; 7 and 8 assume an intent model or a defined fallback event. Numbers 13 to 15 check what the bot said, against a source or against earlier turns.

The denominators in the table rest on these terms:

• Engaged session - a session in which the user asked for something.
• Unengaged session - a session that opens the widget, says hello and leaves. It stays out of every denominator.
• Session and conversation - this guide uses both words for the same unit: one continuous exchange between a user and the bot.
• Volume counts - total sessions, active users and messages per session are not on the list, because they size these denominators and do not say how a conversation went.

Google's Dialogflow CX analytics documentation defines the escalation rate in its intent escalations view as the "Percentage of sessions that requested human escalation" and the no-match rate for each page as the "Percentage of total conversation turns that resulted in a no-match for the page". One analytics panel therefore reports a per-session number and a per-turn number, so write the unit next to every chatbot KPI you publish.

Outcome Metrics: Goal Completion, Resolution, and Containment

Outcome metrics are the chatbot success metrics: they say whether a conversation achieved something. The same word can hide different formulas here, because each platform decides for itself when a conversation counts as resolved or successful.

Microsoft's outcome model is also an accounting identity. A session is unengaged or engaged, and every engaged session ends as resolved (confirmed or implied, which this guide calls assumed), escalated or abandoned, which gives:

engaged sessions = resolved confirmed + resolved assumed + escalated + abandoned

Containment is everything that was not escalated, so it includes every assumed resolution and every abandoned session. Resolution rate and containment rate are both read off that equation, and goal completion rate is the outside check on them.

Handoff and Drop-Off Metrics: Escalation, Abandonment, and Recontact

Handoff and drop-off metrics describe the conversations that left the bot: to a person, to a timeout, or back again a few days later. Each needs a reason or a time window attached before the number means anything.

Understanding and Effort Metrics: Intent Accuracy, Fallback Rate, and Turns

Understanding metrics ask whether the bot recognized what each message meant, and effort metrics ask how much work the user did to be understood. What the log can tell you depends on the kind of chatbot:

• Intent-based chatbot - matches each message to a predefined intent and emits a no-match event when nothing fits.
• LLM chatbot - generates every reply and has no such event unless you define one, so its fallback rate is only as meaningful as that definition.

Experience and Cost Metrics: CSAT, Latency, and Cost per Resolution

Each metric in this group has a denominator that is easy to get wrong: who answered the survey, which replies were timed, and which conversations count as resolved.

Answer Quality Metrics for LLM Chatbots

The first 12 metrics describe how a conversation went, and none of them checks whether what the bot said was true, which was the failure in Moffatt v. Air Canada. Answer quality metrics check the content of a reply against a source (a knowledge base, a policy page or a system of record) or, for context retention, against what was said earlier in the conversation. They matter most for chatbots that generate answers with an LLM, where nobody wrote the reply in advance.

Answer quality scores come from labels instead of logs: a person or a judge model reads each response and decides. A judge model brings its own error. In the 2023 paper Large Language Models are not Fair Evaluators, Wang and colleagues changed only the order in which two answers were shown and found that Vicuna-13B could beat ChatGPT on 66 of 80 test queries with ChatGPT as the evaluator.

Validate any judge against human labels on a sample before you report its rate. The guide to LLM-as-a-judge covers that calibration step.

How to Measure Chatbot Performance From a Conversation Log

• Define a goal event for each intent - an event your own systems can confirm, such as a refund record created or an order status returned.
• Log the outcome of every session - resolved confirmed, resolved assumed, escalated or abandoned, with a fallback flag on every bot reply.
• Compute each rate with one formula - goal completion rate, escalation rate by reason, fallback rate, and CSAT with its response rate.
• Write the formula and the unit beside each number - a rate published without its denominator cannot be checked or compared.
• Add answer quality checks when an LLM generates the answers - hallucination rate and groundedness, scored against a reference for each response.

To show the formulas disagreeing on one set of conversations, this section uses two short Node.js scripts written for this guide:

• Log generator - writes a sample log of 40 sessions for a fictional support chatbot. It sets the mix of outcomes by hand (13 confirmed, 9 assumed, 8 escalated, 6 abandoned and 4 that never engage) and draws every other field from a fixed random seed, with odds chosen for each outcome.
• Metrics script - reads that log and prints each metric with its fraction and formula.
• Sample data - none of the numbers is a measurement of a real chatbot, and none is a benchmark. The gaps in the output illustrate the arithmetic and are not findings.

Each session record carries the fields the formulas need: the platform's outcome label, a back-end goal flag, the labeled and predicted intent, and fallback, latency and token values for every reply. This is one record from the sample log, reformatted for width:

{
  "id": "S016",
  "engaged": true,
  "labeledIntent": "change_address",
  "predictedIntent": "change_address",
  "outcome": "resolved_assumed",
  "escalationReason": null,
  "goalCompleted": false,
  "userTurns": 2,
  "botReplies": [
    { "fallback": false, "latencyMs": 1038, "tokens": 1370 },
    { "fallback": false, "latencyMs": 1759, "tokens": 955 }
  ],
  "csat": null,
  "recontactWithin7d": true
}

Both scripts ran on Node.js 24.21.0 on October 5, 2026, and the second printed this:

Sample log: 40 sessions, 36 engaged, 4 unengaged
Rates are over engaged sessions unless a line says otherwise.

OUTCOME
Goal completion rate              52.8% (19/36)     goal confirmed by the back end / engaged
Resolution rate, confirmed only   36.1% (13/36)     resolved confirmed / engaged
Resolution rate, with assumed     61.1% (22/36)     (confirmed + assumed) / engaged
Containment rate                  77.8% (28/36)     (engaged - escalated) / engaged
Containment rate, all sessions    80.0% (32/40)     (all - escalated) / all, unengaged included
Goal confirmed by the back end, per outcome label:
  resolved confirmed              13 of 13
  resolved assumed                4 of 9
  escalated                       0 of 8
  abandoned                       2 of 6

HANDOFF AND DROP-OFF
Escalation rate                   22.2% (8/36)      escalated / engaged
  user requested                  1
  system unintended               6
  system intended                 1
Abandonment rate                  16.7% (6/36)      abandoned / engaged
Recontact rate, 7 days            27.3% (6/22)      resolved with a repeat contact / resolved

UNDERSTANDING AND EFFORT
Intent recognition accuracy       80.6% (29/36)     first messages matched correctly / labeled
  lowest intent                   57.1% (4/7)       change_address
Fallback rate, per bot reply      14.2% (18/127)    fallback replies / bot replies
Fallback rate, per conversation   30.6% (11/36)     engaged with 1+ fallback / engaged
Average turns to resolution       3.92 (51/13)      user turns in confirmed / confirmed

EXPERIENCE AND COST
CSAT, mean of 1 to 5              3.75 (45/12)      sum of ratings / ratings received
CSAT, satisfied share             58.3% (7/12)      ratings of 4 or 5 / ratings received
CSAT response rate                33.3% (12/36)     ratings received / engaged
Response latency, mean            1,791 ms          over 127 bot replies
Response latency, P50             1,359 ms          nearest-rank percentile
Response latency, P95             6,260 ms          nearest-rank percentile
Tokens per confirmed resolution   11,579            all engaged tokens (150,533) / confirmed
  confirmed sessions only         4,957             their own tokens (64,438) / confirmed

95% WILSON SCORE INTERVALS, 36 ENGAGED SESSIONS
Containment rate                  77.8%             61.9% to 88.3%
Resolution rate, confirmed only   36.1%             22.5% to 52.4%
Goal completion rate              52.8%             37.0% to 68.0%

Every line of that output comes from the same 40 sessions, and the headline changes with the definition:

• Outcome metrics - containment is 77.8%, resolution with assumed is 61.1%, goal completion is 52.8% and confirmed resolution is 36.1%. Of the 28 contained sessions, the back end confirms the goal in 19.
• Goal completion by outcome label - all 13 confirmed resolutions have a completed goal, while only 4 of the 9 assumed resolutions do. Two of the 6 abandoned sessions also completed their goal. That happens when a user gets the answer and leaves without confirming.
• Fallback rate - 14.2% per bot reply and 30.6% per conversation come from the same 18 fallback replies.
• Unengaged sessions - counting the 4 sessions that never engaged lifts containment from 77.8% to 80.0%, because a session that never asked for anything was never escalated.
• Intent recognition accuracy - the overall 80.6% includes a change_address intent at 57.1%.
• CSAT - the 3.75 average rests on 12 ratings. That is a 33.3% response rate.
• Response latency - the mean of 1,791 ms sits between a P50 of 1,359 ms and a P95 of 6,260 ms and describes neither.
• Cost per resolved conversation - counting every engaged session's tokens gives 11,579 tokens per confirmed resolution (the script leaves out the single reply in each of the 4 unengaged sessions), and counting only the confirmed sessions' own tokens gives 4,957.

Metrics 13 to 15 are not in the output: they need a reference and a label for each response, which a session log does not hold.

A rate from 36 sessions is an estimate. In Interval Estimation for a Binomial Proportion, Brown, Cai and DasGupta describe the "chaotic coverage properties" of the standard Wald interval and "recommend the Wilson interval or the equal-tailed Jeffreys prior interval for small n". The Wilson interval for containment here runs from 61.9% to 88.3%, so a move of a few points between two weeks of this size is not evidence of a change.

The script computes the interval with this function:

// 95% Wilson score interval for a proportion of k successes in n trials.
function wilson(k, n, z = 1.96) {
  const p = k / n;
  const denom = 1 + (z * z) / n;
  const center = (p + (z * z) / (2 * n)) / denom;
  const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / denom;
  return [center - half, center + half];
}

Setting Targets for Chatbot KPIs

Containment, fallback and CSAT figures from other companies are not comparable with yours, because the definitions behind them differ. Copilot Studio and Intercom Fin count resolutions the user never confirmed, Amazon Lex V2 categorizes a conversation by the result of its final intent, and sessions that never engaged can be counted or left out.

In the sample log above, the same 36 engaged sessions give a resolution rate of 61.1% with assumed resolutions and 36.1% with confirmed ones only. That is a 25-point gap on identical conversations.

The guide to agent performance lists published call-center figures for first contact resolution and CSAT with their sources. Read them as context for your own baseline: that guide cautions that its first contact resolution figure depends on how "solved in one contact" is defined.

Set chatbot KPI targets from your own baseline:

• Freeze the definitions - write down the event, the unit and the timeout behind each metric, and version them. A changed definition starts a new baseline.
• Baseline each intent - compute every metric for each intent before the change you want to judge, because an overall figure can look healthy while one intent fails.
• Attach an interval - print the Wilson interval with every rate and treat a move inside it as no change.
• Compare like with like - judge a release by its difference from the baseline under the same definitions.
• Pair the metrics - read each headline number with the metric that exposes its blind spot.

Chatbot Metrics You Can Test Before Release

Most of the metrics above can be measured before release, each with a different kind of test, and the rest need real users:

• Simulated conversations - goal completion, escalation and fallback behavior, turn counts and the three answer quality metrics.
• A labeled utterance set - intent recognition accuracy.
• A load test - response latency at your expected peak.
• Production only - containment, resolution, abandonment, recontact, CSAT and cost per resolved conversation.

TestMu AI's Agent Testing runs the first kind of test, simulated conversations, for chat, voice and phone agents. Its testing agents hold multi-turn conversations with your chatbot through the chatbot's own endpoint, as a user would, and score each one on nine quality metrics. Several of them correspond to metrics in this guide:

• Hallucination Detection - whether the agent invents information not supported by its knowledge base or context (metric 13).
• Context Awareness - whether the agent retains and correctly uses information from earlier in the conversation (metric 15).
• Positive User Outcome - whether the interaction is likely to result in the user achieving their goal. It is the closest pre-release reading of goal completion (metric 1).
• Completeness - whether the response fully addresses the user's question or need. It is the pre-release signal for recontact (metric 6).

The remaining metrics are Bias Detection, Response Quality, Conversation Flow, Tone Consistency and Root-Cause Understanding. A run rolls up into a Green, Yellow or Red verdict, and each metric in it returns:

• A pass or fail for the scenario - with an evidence excerpt from the conversation that drove it.
• A score aggregated across the run - one figure for the metric over all scenarios.
• A confidence level - High, Medium or Low, based on how many scenarios were evaluated.

For groundedness against your own data, Data Validation checks the values a chat agent states, such as an order status or a price, against your system of record through read-only API lookups after each conversation. Each fact gets a Pass, Fail or Cannot verify verdict, and the setup is in the chat agent testing documentation.

A chat run does not cover:

• Latency under load - a chat run does not model production-scale concurrent usage, and streaming latency is not measured separately. The product's performance testing is for phone agents, so a chatbot needs a separate load test.
• Intent recognition accuracy - it is not one of the nine chat and voice metrics, so score your labeled utterance set separately.
• Memory across sessions - continuity between separate sessions is not evaluated.

A Green verdict covers the scenarios, personas and metrics you configured and is no guarantee of zero failures in production.

If your team prefers to write evaluation logic in code, an evaluation library gives you the building blocks, and Agent Testing is for teams that want the scenarios, simulated users and scoring without building them in-house. To turn these checks into a test plan, see how to test a chatbot, which walks through the process step by step.

Note: Beyond the nine standard metrics, TestMu AI Agent Testing lets you define custom pass or fail rules for your own chatbot, and its dashboard shows metric score deltas between test runs. Start free with TestMu AI.

Conclusion

Take the number your team reports most often and write its event, formula, unit and timeout on one line. Then compute goal completion rate for the same sessions from a back-end event and put the two side by side. The gap shows which conversations your current chatbot metrics count as wins without the user getting what they came for.

Before the next prompt, model or knowledge-base change, run the answer quality checks on simulated conversations, so that hallucination rate and context retention have a pre-release value to compare against. TestMu AI's Agent Testing runs them from the dashboard or the CLI, and the guide to testing your first AI agent walks through the first run.