How to Test a Kore.ai Bot

What Is a Kore.ai Bot?

Kore.ai's builder is the XO Platform, now positioned inside the broader Agent Platform for building and orchestrating agentic applications. A bot's conversation logic lives in Dialog Tasks, but the current generation layers autonomous agents, an orchestrator, and Tools on top. That shift from scripted dialog to generative reasoning is exactly what changes how you have to test it.

What a real Kore.ai bot is made of:

• Dialog Tasks built from interlinked nodes, driven by intent and entity NLU scored across machine learning, fundamental meaning, and knowledge graph models.
• Search AI, Kore.ai's retrieval-augmented generation layer with graph-RAG and more than 100 pre-built connectors to enterprise data.
• Workflow, Code, and MCP Tools that let agents act on external systems, coordinated by an orchestrator across multi-agent workflows.
• Voice and chat channels, plus a Webhook channel that exposes the bot to any HTTP client.

Because the bot mixes deterministic dialog with generative answers, two parts of it fail in two different ways. The dialog can route correctly and the generated answer can still be wrong. For the shared surface across platforms, our overview of chatbot testing maps the test types that apply to any bot.

What Does Kore.ai's Native Testing Cover?

Kore.ai's native testing is genuinely strong, and worth using first. It is also, by its own descriptions, NLU-accuracy testing. Knowing exactly where it stops is what tells you what to add.

Why Do Kore.ai Bots Pass Batch Testing but Fail in Production?

Batch testing scores curated utterances without conversation context. Real users bring context, paraphrase, and off-script intent, which is where the failures the native metrics cannot see actually live.

How Do You Test a Kore.ai Bot, Step by Step?

Testing happens in three layers: the native NLU tools that catch intent and entity bugs, the Webhook endpoint that lets you drive the deployed bot, and an evaluation platform that scores the generative answers the native tools never grade.

# Send a user message to a Kore.ai Webhook endpoint and read the reply
curl -X POST "https://bots.kore.ai/chatbot/hooks/<bot_id>" \
  -H "Authorization: bearer $KORE_JWT" \
  -H "Content-Type: application/json" \
  -d '{"message": {"type": "text", "val": "reset my password"},
       "from": {"id": "tester"}, "session": {"new": true}}'

# Response: the bot reply is in the data array's "val" field
# { "data": [ { "type": "text", "val": "Sure, let's reset it...", "messageId": "..." } ] }

Testing a Kore.ai Bot With TestMu AI's Agent Testing

Connecting Agent Testing to a Kore.ai bot does not require the BotKit SDK or a change to the bot. It calls the Webhook endpoint directly, adapting to Kore.ai's request shape with a body template and a response path, then layers three inputs on top before generating a full scenario set.

# Point the evaluator at a Kore.ai Webhook endpoint
testmu-a2a test \
  --agent https://bots.kore.ai/chatbot/hooks/<bot_id> \
  --header "Authorization: bearer $KORE_JWT" \
  --body-template '{"message": {"type": "text", "val": "{{message}}"}, "from": {"id": "testmu-eval"}, "session": {"new": false}}' \
  --response-path "data[0].val" \
  --spec "Banking support bot that resets passwords, checks balances, and escalates disputes" \
  --count 30

• Upload the same PRD, policy document, or knowledge base the bot was built from, and the platform parses it for behavioral requirements and scenario seeds.
• Paste in the agent instructions, so the evaluation baseline matches what the bot is actually told to do.
• Add focus areas that weight scenario generation toward what matters most this run, such as Search AI grounding and escalation logic.

From there, autonomous evaluators run the generated scenarios against the endpoint and score each on the 9 quality metrics. A Search AI answer is checked against what the bot should actually know, not just whether NLU matched the intent, and every run rolls up into a Green, Yellow, or Red verdict, the same model our overview of AI agent testing walks through in depth.

Which Metrics Matter for a Kore.ai Bot?

Precision and Recall tell you the NLU matched the intent. They do not tell you whether the answer was honest, complete, or safe. Agent Testing scores each conversation on 9 quality metrics that pick up exactly where the native NLU metrics stop.

• Hallucination detection catches answers the bot invented when Search AI returned a weak or off-topic chunk.
• Context awareness checks whether entities and state actually carried across a branch between dialog tasks, not just within one.
• Completeness and root-cause understanding check whether the bot solved the real problem or a related one that looked close.
• Tone consistency and bias detection check for the same response quality regardless of how the user phrased the request.

Every score also carries a confidence level, High, Medium, or Low, based on how many scenarios actually ran, so a Green built on ten scenarios is never mistaken for a Green built on three hundred. That distinction is what separates a demo that passed from a bot that is ready.

How Do You Red-Team a Kore.ai Bot?

The four dimensions cover a user trying to get help. Red-teaming covers a user, or a poisoned document, trying to break the bot on purpose: override its instructions, exfiltrate data through a Tool, or coax out PII. Kore.ai bots are especially exposed because they call Workflow, Code, and MCP Tools that act on enterprise systems. The testmu-a2a-cli runs adversarial probes against the endpoint, scoring categories including prompt injection, jailbreak, data exfiltration, and PII leakage at basic, intermediate, or advanced intensity.

testmu-a2a redteam \
  --agent https://bots.kore.ai/chatbot/hooks/<bot_id> \
  --header "Authorization: bearer $KORE_JWT" \
  --intensity advanced \
  --spec "Kore.ai banking bot with balance and dispute Tools"

# Target specific attack categories only
testmu-a2a redteam \
  --agent https://bots.kore.ai/chatbot/hooks/<bot_id> \
  --categories prompt-injection,jailbreak,pii-leakage

The output is a letter grade, A+ to F, per category rather than one pass or fail, because a bot can resist a direct jailbreak and still follow instructions injected through a retrieved document. For any bot wired to Tools that touch enterprise data, that per-category grade is the gap between a vulnerability you found and one an attacker finds.

How Do You Run Kore.ai Bot Tests in CI/CD?

A Dialog Task edit, a new Search AI source, or a model change ships the moment it is published. Native Batch Testing gates the NLU, and the testmu-a2a-cli adds the endpoint-level generative sweep: it installs with pip, authenticates from CI secrets instead of an interactive login, and writes JUnit XML that GitHub Actions, GitLab CI, and Jenkins read natively.

# .github/workflows/kore-ai-bot-tests.yml
name: Kore.ai Bot Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install TestMu A2A CLI
        run: pip install testmu-a2a-cli
      - name: Run Kore.ai bot regression suite
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
          KORE_JWT: ${{ secrets.KORE_JWT }}
        run: |
          testmu-a2a test \
            --agent ${{ vars.KORE_WEBHOOK }} \
            --header "Authorization: bearer $KORE_JWT" \
            --body-template '{"message":{"type":"text","val":"{{message}}"},"from":{"id":"testmu-eval"},"session":{"new":false}}' \
            --response-path "data[0].val" \
            --spec "Kore.ai banking bot with balance and dispute Tools" \
            --format junit --output results.xml
      - name: Publish results
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Kore.ai Bot Test Results
          path: results.xml
          reporter: java-junit

Gate on exit code: 0 for all scenarios passed, 1 for any failure. Run the full evaluation for dialog, Search AI, or model changes, and a smaller smoke set for a change that does not touch bot behavior. The discipline is the same one covered in our guide to AI agent regression testing.

What Won't Automated Testing Catch?

A Green verdict means the bot passed on the scenarios you ran, with the personas you tested, on the metrics you configured. Here is what it does not cover:

• A low-confidence Green is not a cleared launch. It means too few scenarios ran to trust the score yet, so run more, especially on Search AI grounding and injection resistance, before you ship.
• Voice quality on the IVR channel is a separate test. A text-endpoint sweep does not exercise speech recognition on accents and noise, so test the voice path with real calls.
• Non-determinism means one clean run is not proof. Run the scenario set enough times to see the distribution of paths the bot actually takes.
• Highly regulated domains need custom rules. The default 9 metrics cover general quality well, but a banking or healthcare bot needs custom validation criteria on top.
• Native NLU testing is still your first gate. The endpoint sweep complements Batch and Conversation Testing; it does not replace them.

None of this is a reason to skip testing. It is a reason to keep testing after launch, not just before it.

Conclusion

Keep running Kore.ai's Batch and Conversation Testing; they are among the best native NLU tools in the market. Then point the bot at the questions those tools were never built to answer: is the Search AI response actually true, does the tone hold when the user is angry, can the bot be talked into ignoring its rules. The native tools grade the intent. TestMu AI's Agent Testing grades the answer, running the deployed bot through hundreds of personas and adversarial probes and handing back a go-live verdict before a real user finds the gap first.

Point the evaluator at your Webhook endpoint, check the setup docs, and run your first evaluation before the next dialog or model change ships.