How to Test a LangChain or LangGraph Agent

What Is a LangChain or LangGraph Agent?

LangChain is the framework for composing calls to a language model, tools, and data. LangGraph is its stateful sibling for agents: instead of a linear chain, you define a graph where the model decides which node or tool to visit next, keeping state across turns. That control is why teams reach for it, and why testing it is harder than testing a plain chain.

What a real LangGraph agent is made of:

• Nodes that call the model, run a tool, or transform state, wired into a graph with conditional edges the agent traverses at runtime.
• Tools and function calls that reach out to a database, an API, a retriever, or another agent mid-conversation.
• A persisted state object that carries context, message history, and intermediate results across turns.
• A deployment surface, often the LangGraph Platform server or a custom HTTP endpoint, that exposes the agent to a front end or another service.

Because the agent chooses its own path, two runs of the same input can take different routes. That non-determinism is the whole point of an agent, and it is exactly what a fixed input-output assertion cannot capture. For a hands-on build example, our guide to a Playwright LangChain agent walks through wiring one up end to end.

Why Do LangChain Agents Pass Tests but Fail in Production?

LangChain's own testing documentation splits agent testing into unit tests, integration tests, and evals. All three are valuable, and all three run on inputs you chose. Most production failures live in the inputs you did not.

What Should You Test in a LangChain Agent?

Whether the agent is a support copilot, a research assistant, or a data agent, the same four dimensions decide whether it is ready for real users. Each needs a different kind of scenario, not just more curated dataset rows.

How Do You Test a LangChain or LangGraph Agent, Step by Step?

Testing happens in three layers: what you assert in code, how you evaluate the trajectory the agent took, and how you score the deployed agent against the messy real-user traffic none of the first two simulate.

Testing a LangGraph Agent With TestMu AI's Agent Testing

Connecting Agent Testing to a LangGraph agent does not require an SDK or a change to the graph. It calls the agent's HTTP endpoint directly, adapting to a non-standard request shape with a body template and a response path, then layers three inputs on top before generating a full scenario set.

# Point the evaluator at a custom LangGraph endpoint
testmu-a2a test \
  --agent https://your-langgraph-app/invoke \
  --body-template '{"input": {"messages": [{"role": "user", "content": "{{message}}"}]}}' \
  --response-path "output.messages[-1].content" \
  --spec "Support agent that answers billing questions and calls the refund tool" \
  --count 30

• Upload the same PRD, tool spec, or system prompt the agent was built from, and the platform parses it for behavioral requirements and scenario seeds.
• Paste in the agent's system prompt, so the evaluation baseline matches what the agent is actually instructed to do.
• Add focus areas that weight scenario generation toward what matters most this run, such as tool-selection accuracy and refusal behavior.

From there, autonomous evaluators run the generated scenarios against the endpoint and score each on the 9 quality metrics. A tool call is checked against what it actually should have done, not just whether it returned a response, and every run rolls up into a Green, Yellow, or Red verdict, the same model our overview of AI agent testing walks through in depth.

Which Metrics Matter for a LangChain Agent?

A trajectory match tells you the agent took the expected steps. It does not tell you whether the final answer was honest, complete, or safe. Agent Testing scores each conversation on 9 quality metrics that apply to any chat agent, tuned for exactly the ways a tool-using agent goes wrong.

• Hallucination detection catches invented facts and, critically for agents, invented tool arguments the model filled in rather than grounded.
• Context awareness checks whether the agent's persisted state actually held across a long conversation, not just a two-turn eval.
• Completeness and root-cause understanding checks whether the agent solved the real problem or a related one that looked close.
• Tone consistency and bias detection checks for the same response quality regardless of how the user phrased the request.

Every score also carries a confidence level, High, Medium, or Low, based on how many scenarios actually ran, so a Green built on ten scenarios is never mistaken for a Green built on three hundred. That distinction is what separates a demo that passed from an agent that is ready.

How Do You Red-Team a LangChain Agent?

The four dimensions cover a user trying to get help. Red-teaming covers a user, or a poisoned document, trying to break the agent on purpose: override the system prompt, exfiltrate data through a tool, or coax out PII. Agents are especially exposed here because they call real tools and read untrusted content. The testmu-a2a-cli runs adversarial probes against the endpoint, scoring categories including prompt injection, jailbreak, data exfiltration, and PII leakage at basic, intermediate, or advanced intensity.

testmu-a2a redteam \
  --agent https://your-langgraph-app/invoke \
  --intensity advanced \
  --spec "LangGraph support agent with refund and account tools"

# Target specific attack categories only
testmu-a2a redteam \
  --agent https://your-langgraph-app/invoke \
  --categories prompt-injection,jailbreak,pii-leakage

The output is a letter grade, A+ to F, per category rather than one pass or fail, because an agent can resist a direct jailbreak and still follow instructions injected through a retrieved document. For any agent wired to tools that mutate data, that per-category grade is the gap between a vulnerability you found and one an attacker finds.

How Do You Run LangGraph Agent Tests in CI/CD?

A prompt edit, a new tool, or a model upgrade merges the moment the PR does. LangSmith's pytest and Vitest integrations run your evals in the same pipeline, and the testmu-a2a-cli adds the endpoint-level sweep: it installs with pip, authenticates from CI secrets instead of an interactive login, and writes JUnit XML that GitHub Actions, GitLab CI, and Jenkins read natively.

# .github/workflows/langgraph-agent-tests.yml
name: LangGraph Agent Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install TestMu A2A CLI
        run: pip install testmu-a2a-cli
      - name: Run LangGraph agent regression suite
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
        run: |
          testmu-a2a test \
            --agent https://your-langgraph-app/invoke \
            --spec "LangGraph support agent with refund and account tools" \
            --format junit --output results.xml
      - name: Publish results
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: LangGraph Agent Test Results
          path: results.xml
          reporter: java-junit

Gate on exit code: 0 for all scenarios passed, 1 for any failure. Run the full evaluation for prompt, tool, or model changes, and a smaller smoke set for a refactor that does not touch agent behavior, so every commit is not blocked on a full regression pass. The discipline is the same one covered in our guide to AI agent regression testing.

What Won't Automated Testing Catch?

A Green verdict means the agent passed on the scenarios you ran, with the personas you tested, on the metrics you configured. Here is what it does not cover:

• A low-confidence Green is not a cleared launch. It means too few scenarios ran to trust the score yet, so run more, especially on hallucination and injection resistance, before you ship.
• Cross-session memory is not evaluated end to end. If your agent is supposed to recall a user across sessions, test that persistence path separately.
• Non-determinism means one clean run is not proof. Run the scenario set enough times to see the distribution of paths the agent actually takes.
• Highly specialized domains need custom rules. The default 9 metrics cover general quality well, but an agent giving medical, legal, or financial guidance needs custom validation criteria on top.
• A stale spec gives you a stale verdict. If the agent's behavior changed and the prompt describing "correct" did not, some regressions slip through and some good new behavior gets flagged as failure.

None of this is a reason to skip testing. It is a reason to keep testing after launch, not just before it.

Conclusion

Take the LangGraph agent you already have passing its evals. Point it at a harder question than the notebook ever asked: what happens when a user is angry, phrases the request three ways at once, or feeds it a document that tells the agent to ignore its instructions. LangSmith and AgentEvals catch the trajectory bugs. TestMu AI's Agent Testing catches the rest, running the deployed agent through hundreds of personas and adversarial probes and handing back a go-live verdict before a real user finds the gap first.

Point the evaluator at your endpoint, check the setup docs, and run your first evaluation before the next prompt or model change merges.