LLM Testing: What It Is and How to Test LLM Applications

How LLM Testing Works

Each LLM test pairs one input with acceptance criteria written before the run. The harness runs the input through the application, records the response and any tool calls, and marks each criterion pass or fail. The suite's pass rate then decides the release.

Exact-match assertions break here because LLM output is probabilistic: wording varies with the prompt, model version, temperature and context, so a correct answer rarely repeats character for character. The assertion patterns that survive that variation are covered in testing non-deterministic AI outputs.

The system under test is also larger than the model. Prompts, retrieval (RAG), external APIs, function calling, memory and agents all shape the answer, so a customer support chatbot, for example, is tested on whether it:

• Reads the user's intent correctly.
• Retrieves the right article from its knowledge base.
• Says it does not know instead of inventing an answer.
• Stays inside safety rules and company policy.
• Keeps the same quality after a prompt update or model upgrade.

Model choice is one of the variables a suite settles, and SLM vs LLM compares where smaller models hold up on your own requests. Models that return a typed decision instead of text, such as Jev, the System One model from TypeSafe AI, are tested on whether the confidence threshold behind each decision is set correctly.

LLM Testing vs Traditional Software Testing

Traditional software testing asserts that a known input returns one expected output, while LLM testing scores a response that can vary and asks whether it is correct, grounded and safe rather than identical. LLM evaluation is the scoring layer inside that work: it measures output quality, and testing turns those scores into release decisions.

Types of LLM Testing

The common types of LLM testing are unit, functional and integration, regression, performance, security, bias and fairness, prompt, and human and evaluator testing. Each targets a different failure, so a production suite mixes several.

Key Metrics Used in LLM Testing

LLM testing metrics fall into five groups: task quality, safety and bias, grounding, human and judge scores, and system performance and cost. Pick the few in each group that map to a release risk and give each one a threshold.

Published hallucination figures use different denominators, and the guide to the AI hallucination rate shows how to put them on one scale. The grounding metrics have formulas in RAG evaluation metrics and groundedness, and public model scores are compared in LLM benchmarks.

Latency, cost and error rate show whether an app runs well, not whether its answers are accurate or safe, so pair every performance threshold with a quality one. The free Agent Testing Certification by TestMu AI covers testing AI agents and conversational bots for reliability, safety and real-world accuracy, including the Agent Testing platform, Agent Assurance and Rook CLI.

How to Test LLM Applications Step by Step

To test an LLM application, define what a good answer is, build a dataset that reflects real use, choose metrics, automate the scoring, add human review, gate every release on regressions in CI, and keep scoring production traffic.

# llm_gate.py: score saved replies, fail CI on a regression
import json, sys

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

def check(case, raw):
    try:
        answer = json.loads(raw)["answer"].lower()
    except (ValueError, KeyError, TypeError):
        return "not JSON with an answer field"
    for text in case.get("must_include", []):
        if text.lower() not in answer:
            return f"missing: {text}"
    for text in case.get("must_not_include", []):
        if text.lower() in answer:
            return f"forbidden: {text}"
    return None

golden = load("golden.jsonl")
replies = {r["id"]: r["output"] for r in load("replies.jsonl")}
with open("baseline.json") as f:
    baseline = json.load(f)  # last release: {"id": true}

passed, regressions = 0, []
for case in golden:
    problem = check(case, replies.get(case["id"], ""))
    note = f" ({problem})" if problem else ""
    print(("FAIL " if problem else "PASS ") + case["id"] + note)
    if problem is None:
        passed += 1
    elif baseline.get(case["id"]):
        regressions.append(case["id"])

rate = passed / len(golden)
print(f"pass rate {rate:.0%}, regressions {len(regressions)}")
sys.exit(1 if regressions or rate < 0.9 else 0)

$ python llm_gate.py
PASS refund-window
FAIL cancel-plan (missing: billing page)
PASS eu-shipping
FAIL order-status (not JSON with an answer field)
pass rate 50%, regressions 1
$ echo $?
1

LLM Testing Frameworks and Tools

Common LLM testing frameworks include DeepEval, Ragas, Promptfoo and TruLens, which are open source, and LangSmith, LangChain's platform for tracing and evaluation. Choose by architecture: a RAG pipeline needs retrieval metrics, a prompt-heavy app needs side-by-side prompt comparison, and an agent needs a trace of every step.

A separate reference shows how Ragas metrics such as faithfulness and context precision are calculated, and the roundup of the best LLM evaluation tools compares these and other options by architecture and privacy needs. In the TestMu Conf 2026 session Testing What Matters: Evaluating LLM Relevancy with DeepEval, Monika Sharma shows how to measure relevancy, the failure where an answer sounds fluent and confident but misses what the user asked.

These frameworks score outputs, traces and retrieval at the model and pipeline layer. Once the LLM ships inside a product, the test has to reach it the way users do: TestMu AI's Agent Testing connects to chat, voice, phone, image and video agents through that same channel.

On chat, voice and phone agents it runs 15+ specialized AI testing agents, each probing a different failure mode, and scores every conversation on metrics such as Intent Recognition Accuracy and Hallucination Detection. Each run ends in a Green, Yellow or Red production-readiness verdict with transcripts of the failing scenarios, so an eval framework hardens the model layer during development and Agent Testing checks the application layer before release.

Common Challenges in LLM Testing

The hardest parts of LLM testing are output that varies between runs, hallucinations, prompt sensitivity, long-context drift, adversarial attacks, and the cost of evaluating at scale.

Real-World LLM Testing Use Cases

What a suite checks depends on the application: a chatbot is tested on intent and conversation, a RAG app on retrieval and grounding, a coding assistant on code correctness, and an agent on the actions it takes.

• AI chatbots and virtual assistants - Response accuracy, intent recognition, hallucination checks and safety, plus multi-turn tests that confirm the bot keeps context and follows instructions across a conversation.
• RAG applications - Whether retrieval returns the relevant documents and the answer reflects them, measured with faithfulness, context precision, context recall and answer relevancy.
• AI coding assistants - Code correctness, syntax validity, logical accuracy and security flaws in generated code, across the languages and frameworks the assistant supports.
• AI agents - Whether the agent picks the right tools, completes the task, recovers from failures and reaches the right outcome across a multi-step workflow, not only whether its final message reads well. AI agent testing and AI agent evaluation cover both sides in depth.
• Content generation - Factual accuracy, grammar, tone, brand alignment, originality and compliance with content guidelines.
• Enterprise knowledge assistants - Retrieval quality, factual correctness, citation accuracy and access control, so confidential documents never reach users without permission.
• Healthcare and finance - Higher bars for factual accuracy, bias, explainability and compliance with industry regulation before anything reaches production.

For the conversational surface, chatbot testing covers the test types and case templates, and the guide on how to test a chatbot walks from mapping intents to writing test cases and choosing metrics.

Testing Tool-Calling LLM Applications With Agent Assurance

When an LLM application calls tools, writes files or updates records, the test has to check the effect, because a reply saying the task is done proves nothing alone. TestMu AI's Agent Assurance is built for agents that act: it reads your codebase to learn what the agent does, writes a test suite, invokes the agent for real, and checks each criterion against what the run changed rather than what the agent reported.

• Scenarios from the agent's own code - Discovery reads manifests, prompts, tool tables and MCP servers, then writes functional and adversarial scenarios, including the prompt injection, data exfiltration and policy violation attempts that security testing looks for.
• Evidence over claims - Tool calls are checked against the agent's declared tools, and changed files, produced artifacts and records confirmed by read-only checks are the evidence, so a false completion claim meets what really happened.
• A pass rate that shows its gap - Each criterion ends as Pass, Fail or Unable to Verify. Unable to Verify stays out of the pass rate and is reported beside it, as the share of criteria that could not be verified, which this guide calls the assurance gap.
• Run-over-run regression checks - A rerun shows the pass-rate change between the last two runs and which scenarios newly fail or were fixed, and the JSON report gives a CI job the run totals to gate on.

Agent Assurance runs in your terminal as Rook CLI, the rook command, and needs a TestMu AI account with an Agent Assurance plan or the 14-day free trial. The npm route needs Node.js 20 or newer:

npm install -g @testmuai/rook@latest

Claude Code can drive it too. Install the skill (Node.js 22 or newer), open Claude Code in the agent's repository, and type /rook with the test you want; Claude Code's usual approvals still apply to each command it proposes.

npx @testmuai/rook-skill@latest install --agent claude-code

A bounded first request for an agent that files tickets could read:

/rook Test the ticketing agent in this repository, which opens, updates, and closes tickets through an API tool. Use the staging profile, propose up to three scenarios, including a request it should refuse to act on, and show me the possible writes before invoking the target.

The Claude Code setup guide covers the skill and the CLI install together.

Conclusion

Start with a golden dataset built from real user prompts, give each case a written pass criterion, and run the gate in CI on every prompt, model and retrieval change. If the application calls tools or writes data, install Rook CLI with the Rook CLI install guide and point its first suite at a staging copy of the agent.