How to Test a Botpress Chatbot

What Is a Botpress Chatbot?

Botpress is a builder for LLM chatbots and agents, available as an open-source core and a hosted Botpress Cloud. You build in Studio as a flow of nodes, but the flagship is the Autonomous Node, which the docs describe as using an LLM to decide what the bot should say and which tools to run. That autonomy is why testing it is harder than testing a scripted flow.

What a real Botpress chatbot is made of:

• Flows made of nodes and cards, with standard nodes running their cards in sequence and the Autonomous Node deciding actions at runtime.
• The LLMz engine that directs the Autonomous Node and keeps prompts more agnostic to variance between LLMs.
• Knowledge Bases the bot grounds on, queried by the Autonomous Node's built-in search tool.
• A deployment surface, the Chat API on Botpress Cloud or a self-hosted endpoint, that exposes the bot to a web widget or another service.

Because the Autonomous Node chooses its own path, two runs of the same input can route differently. That non-determinism is the point of an LLM-driven bot, and it is exactly what a single fixed input-output assertion cannot capture. For the test types that apply to any bot, our overview of chatbot testing is a useful primer.

Why Do Botpress Chatbots Pass the Emulator but Fail in Production?

The Emulator is a good build-time check. It is also a single conversation, typed by the person who built the bot, close to the path they had in mind. Most production failures live in the inputs nobody typed there.

What Should You Test in a Botpress Chatbot?

Whether the bot is a support assistant, a lead qualifier, or an internal helpdesk, the same four dimensions decide whether it is ready for real users. Each needs a different kind of scenario, not just another Emulator run.

How Do You Test a Botpress Chatbot, Step by Step?

Testing happens in three layers: what you check by hand in Studio, how you drive the deployed bot over the Chat API, and how you score that bot against the messy real-user traffic the first layer never simulates.

# Send a user message over the Botpress Chat API, then read the reply
curl -X POST "https://chat.botpress.cloud/$WEBHOOK_ID/messages" \
  -H "x-user-key: $USER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"conversationId": "$CONV_ID", "payload": {"type": "text", "text": "reset my password"}}'

# The reply is asynchronous: list messages and read payload.text
curl "https://chat.botpress.cloud/$WEBHOOK_ID/conversations/$CONV_ID/messages" \
  -H "x-user-key: $USER_KEY"
# { "messages": [ { "payload": { "type": "text", "text": "Sure, let's reset it..." } } ] }

Testing a Botpress Chatbot With TestMu AI's Agent Testing

Connecting Agent Testing to a Botpress bot does not require an SDK or a change to the flow. It calls the bot's synchronous endpoint directly, adapting to its request shape with a body template and a response path, then layers three inputs on top before generating a full scenario set.

# Point the evaluator at a synchronous Botpress chat endpoint
testmu-a2a test \
  --agent https://your-botpress-proxy/chat \
  --header "x-user-key: $USER_KEY" \
  --body-template '{"payload": {"type": "text", "text": "{{message}}"}}' \
  --response-path "payload.text" \
  --spec "Support bot that answers account questions and calls the reset-password tool" \
  --count 30

• Upload the same PRD, tool spec, or Knowledge Base the bot was built from, and the platform parses it for behavioral requirements and scenario seeds.
• Paste in the Autonomous Node's instructions, so the evaluation baseline matches what the bot is actually told to do.
• Add focus areas that weight scenario generation toward what matters most this run, such as tool-selection accuracy and refusal behavior.

From there, autonomous evaluators run the generated scenarios against the endpoint and score each on the 9 quality metrics. A tool call is checked against what it actually should have done, not just whether it returned a response, and every run rolls up into a Green, Yellow, or Red verdict, the same model our overview of AI agent testing walks through in depth.

Which Metrics Matter for a Botpress Chatbot?

A clean Emulator run tells you one path worked. It does not tell you whether the answer was honest, complete, or safe on inputs you did not try. Agent Testing scores each conversation on 9 quality metrics that apply to any chat bot, tuned for the ways an LLM-driven, tool-using bot goes wrong.

• Hallucination detection catches invented facts and, for agents, invented tool arguments the model filled in rather than grounded.
• Context awareness checks whether the bot's state actually held across a long conversation, not just a two-turn Emulator test.
• Completeness and root-cause understanding check whether the bot solved the real problem or a related one that looked close.
• Tone consistency and bias detection check for the same response quality regardless of how the user phrased the request.

Every score also carries a confidence level, High, Medium, or Low, based on how many scenarios actually ran, so a Green built on ten scenarios is never mistaken for a Green built on three hundred. That distinction is what separates a demo that passed from a bot that is ready.

How Do You Red-Team a Botpress Chatbot?

The four dimensions cover a user trying to get help. Red-teaming covers a user, or a poisoned document, trying to break the bot on purpose: override its instructions, exfiltrate data through a tool, or coax out PII. Botpress bots are exposed here because the Autonomous Node feeds user input into an LLM that can invoke tools and APIs. The testmu-a2a-cli runs adversarial probes against the endpoint, scoring categories including prompt injection, jailbreak, data exfiltration, and PII leakage at basic, intermediate, or advanced intensity.

testmu-a2a redteam \
  --agent https://your-botpress-proxy/chat \
  --header "x-user-key: $USER_KEY" \
  --intensity advanced \
  --spec "Botpress support bot with account and reset-password tools"

# Target specific attack categories only
testmu-a2a redteam \
  --agent https://your-botpress-proxy/chat \
  --categories prompt-injection,jailbreak,pii-leakage

The output is a letter grade, A+ to F, per category rather than one pass or fail, because a bot can resist a direct jailbreak and still follow instructions injected through a retrieved document. For any bot whose Autonomous Node calls tools that mutate data, that per-category grade is the gap between a vulnerability you found and one an attacker finds.

How Do You Run Botpress Chatbot Tests in CI/CD?

A flow edit, a new tool, a Knowledge Base update, or an LLMz change ships the moment you deploy. The testmu-a2a-cli adds the endpoint-level sweep to your pipeline: it installs with pip, authenticates from CI secrets instead of an interactive login, and writes JUnit XML that GitHub Actions, GitLab CI, and Jenkins read natively.

# .github/workflows/botpress-chatbot-tests.yml
name: Botpress Chatbot Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install TestMu A2A CLI
        run: pip install testmu-a2a-cli
      - name: Run Botpress chatbot regression suite
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
          USER_KEY: ${{ secrets.BOTPRESS_USER_KEY }}
        run: |
          testmu-a2a test \
            --agent ${{ vars.BOTPRESS_ENDPOINT }} \
            --header "x-user-key: $USER_KEY" \
            --body-template '{"payload":{"type":"text","text":"{{message}}"}}' \
            --response-path "payload.text" \
            --spec "Botpress support bot with account and reset-password tools" \
            --format junit --output results.xml
      - name: Publish results
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Botpress Chatbot Test Results
          path: results.xml
          reporter: java-junit

Gate on exit code: 0 for all scenarios passed, 1 for any failure. Run the full evaluation for flow, tool, or model changes, and a smaller smoke set for a change that does not touch bot behavior. The discipline is the same one covered in our guide to AI agent regression testing.

What Won't Automated Testing Catch?

A Green verdict means the bot passed on the scenarios you ran, with the personas you tested, on the metrics you configured. Here is what it does not cover:

• A low-confidence Green is not a cleared launch. It means too few scenarios ran to trust the score yet, so run more, especially on hallucination and injection resistance, before you ship.
• Channel-specific rendering is not covered. A web widget, WhatsApp, and Slack each render cards differently, so smoke-test the actual channel you deploy on.
• Non-determinism means one clean run is not proof. Run the scenario set enough times to see the distribution of paths the Autonomous Node actually takes.
• Highly specialized domains need custom rules. The default 9 metrics cover general quality well, but a bot giving medical, legal, or financial guidance needs custom validation criteria on top.
• A stale spec gives you a stale verdict. If the bot's instructions changed and the prompt describing correct behavior did not, some regressions slip through.

None of this is a reason to skip testing. It is a reason to keep testing after launch, not just before it.

Conclusion

Take the Botpress chatbot you already have passing in the Emulator. Point it at a harder question than you typed there: what happens when a user is angry, phrases the request three ways at once, asks something the Knowledge Base never covered, or feeds it a message that tells the bot to ignore its instructions. The Emulator and Event Debugger catch the single-path bugs. TestMu AI's Agent Testing catches the rest, running the deployed bot through hundreds of personas and adversarial probes and handing back a go-live verdict before a real user finds the gap first.

Point the evaluator at your endpoint, check the setup docs, and run your first evaluation before the next flow or model change ships.