CrewAI Testing: How to Test Agents, Crews, and Tasks

Why Do CrewAI Crews Fail in Ways Unit Tests Miss?

CrewAI is a role-based multi-agent framework. Its CrewAI GitHub repository shows more than 57,000 stars and requires Python 3.10 or later. You give each agent a role, a goal, and a backstory, hand the crew a list of tasks, and call kickoff().

Under the default sequential process, tasks execute one after another and each task inherits the context of the one before it. That is the design that makes crews easy to assemble, and it is also the design that lets one bad output travel.

That design creates four failure modes a green test suite will not catch:

• Silent contract drift - Task 1 returns prose where task 2 expected JSON. Nothing throws; task 2 simply reasons over the wrong shape and produces a plausible answer built on nothing.
• Compounding error - In a sequential process each task inherits the previous output. A small fabrication in step 1 becomes the premise of steps 2 through 5, and the final report is confidently wrong.
• Tool selection drift - The agent has five tools and picks a different one this run. The answer still arrives, but it came from a cached scrape instead of the live API.
• Delegation loops - Under a hierarchical process the manager agent hands work back and forth. The crew finishes, the token bill triples, and no assertion notices.

None of these raise an exception, which is why assertEqual on the final string is the wrong instrument. Run the same crew twice with an identical prompt and you get two different strings, both acceptable. The useful question is not whether the output matches a fixture, but whether the crew did the right work and respected its contracts. The wider version of that problem is covered in our guide to multi agent testing.

What Testing Hooks Does CrewAI Actually Ship?

Most guides to testing agents reach for an external evaluation tool on the first page. CrewAI already ships three testing surfaces of its own, and together with plain pytest they cover the layers where a bug is cheapest to catch.

Read the verdict column carefully, because it decides where each hook belongs. The first three produce a binary answer, so they can gate a pull request. crewai test returns a distribution, which means a single run cannot tell you whether the crew regressed or the sampler was unlucky.

What Does a CrewAI Testing Pyramid Look Like?

Push as much of the crew as possible into deterministic code, then buy confidence about the rest with scenario coverage. Four layers, cheapest first:

• Deterministic unit tests - Tool bodies, parsers, and guardrail functions, run with pytest on every commit. Milliseconds, no API key, no token spend.
• Contract enforcement - Guardrails attached to each task, so a malformed output never reaches the next agent. These run inside the crew, in production as well as in tests.
• Behavioral assertions - Event listeners capturing tool selection and task completion during a real kickoff. Slower and token-costed, so run these on merge rather than on every push.
• Scenario evaluation - The finished crew scored across many inputs and personas, which is the only layer that answers "is this good enough for users".

Teams usually build layer 4 first, because it is the layer that looks like agent testing, then wonder why the feedback loop takes twenty minutes. Layers 1 and 2 catch a large share of real defects and cost nothing per run. Build up from the bottom.

How Do You Unit Test CrewAI Tools Without Calling an LLM?

A common complaint about CrewAI is that individual components are hard to unit test. The fix is structural rather than technical: keep the logic outside the agent. A guardrail function receives a TaskOutput and returns a tuple, so pytest can construct that input and call the function directly. No agent, no prompt, no key.

Here is a guardrail that rejects a pricing summary quoting a bare number with no currency marker, plus its tests:

from typing import Any, Tuple

from crewai import TaskOutput


def price_table_guardrail(result: TaskOutput) -> Tuple[bool, Any]:
    """Reject a task output that quotes a price without a currency symbol."""
    text = result.raw.strip()
    if not text:
        return (False, "Task returned an empty output")
    for token in text.split():
        if token.replace(".", "", 1).isdigit() and "$" not in text:
            return (False, f"Bare number '{token}' with no currency marker")
    return (True, text)


def make_output(raw: str) -> TaskOutput:
    return TaskOutput(description="pricing summary", raw=raw, agent="analyst")


def test_guardrail_passes_on_currency_marked_output():
    ok, payload = price_table_guardrail(make_output("Starter costs $399 per month."))
    assert ok is True
    assert payload == "Starter costs $399 per month."


def test_guardrail_rejects_bare_number():
    ok, reason = price_table_guardrail(make_output("Starter costs 399 per month."))
    assert ok is False
    assert "no currency marker" in reason


def test_guardrail_rejects_empty_output():
    ok, reason = price_table_guardrail(make_output("   "))
    assert ok is False
    assert reason == "Task returned an empty output"

Running that file against crewai 1.15.16 produces this output, which is the actual terminal result and not an illustration:

$ python -m pytest test_crew_guardrails.py -v
============================= test session starts =============================
platform win32 -- Python 3.13.5, pytest-9.1.1, pluggy-1.6.0
collected 3 items

test_crew_guardrails.py::test_guardrail_passes_on_currency_marked_output PASSED [ 33%]
test_crew_guardrails.py::test_guardrail_rejects_bare_number PASSED       [ 66%]
test_crew_guardrails.py::test_guardrail_rejects_empty_output PASSED      [100%]

============================= 3 passed in 10.89s ==============================

Ten seconds is almost entirely CrewAI's import time; the assertions themselves are instant. Apply the same shape to custom tools: put the HTTP call, the parsing, and the error handling in a plain function, then have the tool wrap it. The wrapper is what the agent sees, and the function is what your tests call.

How Do Task Guardrails Enforce Output Contracts?

A guardrail is the one testing hook that also runs in production. Per the CrewAI tasks documentation, a function-based guardrail accepts exactly one parameter and returns (True, validated_result) on success or (False, error_message) on failure. When it fails, the error goes back to the agent, which retries until it passes or exhausts guardrail_max_retries, which defaults to 3.

That retry loop is why guardrails beat post-hoc validation. A failing assertion in a test tells you the crew was wrong yesterday; a failing guardrail tells the agent, mid-run, exactly what to fix.

from crewai import Task

pricing_task = Task(
    description="Summarize the pricing tiers from the scraped page",
    expected_output="A markdown table with every price carrying a currency symbol",
    agent=analyst,
    guardrails=[price_table_guardrail, no_placeholder_guardrail],
    guardrail_max_retries=2,
)

Three practical rules for writing them:

• Make the error message actionable - It becomes the retry prompt. "Validation failed" wastes a retry; "Bare number '399' with no currency marker" gets fixed on the next attempt.
• Prefer functions over string guardrails for anything checkable - A string guardrail is evaluated by the agent's own LLM, so it is another non-deterministic judgment. Reserve it for genuinely subjective criteria like tone.
• Lower guardrail_max_retries on expensive tasks - Three retries on a task that makes six tool calls is a quiet way to quadruple a token bill when the agent cannot satisfy the rule at all.

The guardrails list takes precedence over the singular guardrail argument, and guardrails run in sequence, so order them cheapest-check-first. A regex that fails in microseconds should never sit behind an LLM-evaluated string guardrail.

How Do You Assert on What Agents Did, Not What They Said?

The most useful assertions about a crew are about behavior. Did the research agent call the live search tool, or did it answer from the model's own memory? Did the writer task run before the editor task? Wording changes every run; the tool-call sequence usually does not.

CrewAI exposes this through an event bus. The CrewAI event listener documentation covers events across crews, agents, tasks, tools, memory, LLM calls, and flows, including ToolUsageStartedEvent, TaskCompletedEvent, and CrewKickoffFailedEvent. You subclass BaseEventListener and register handlers in setup_listeners.

from crewai.events import BaseEventListener, crewai_event_bus, ToolUsageStartedEvent


class ToolCallRecorder(BaseEventListener):
    """Records every tool the crew reaches for, so tests can assert on tool choice."""

    def __init__(self):
        self.calls = []
        super().__init__()

    def setup_listeners(self, event_bus):
        @event_bus.on(ToolUsageStartedEvent)
        def _on_tool_start(source, event):
            self.calls.append(event.tool_name)


def test_recorder_captures_tool_selection():
    recorder = ToolCallRecorder()
    future = crewai_event_bus.emit(
        None,
        ToolUsageStartedEvent(tool_name="serper_search", tool_args={"q": "crewai"}),
    )
    if future is not None:
        future.result(timeout=5)
    assert recorder.calls == ["serper_search"]

Two details in that snippet are not in the documentation, and both cost real debugging time when this test was first written.

• Initialize your state before super().__init__() - BaseEventListener calls setup_listeners from inside its constructor, so anything the handler touches must already exist when super() runs.
• emit() is asynchronous and returns a Future - Handlers dispatch on a thread pool. The first version of this test asserted immediately after emit and failed with an empty list, because the handler had not run yet. Waiting on the returned future fixes it.

In a real suite, attach the recorder before kickoff() and assert on the sequence afterwards: that the search tool was called at least once, that no deprecated tool appeared, that the total call count stayed under a budget. Feeding the same event stream into a tracing backend gives you the production half of this picture, which we cover in agent observability.

What Does the crewai test Command Actually Measure?

CrewAI ships a scoring command. The CrewAI testing documentation describes it as running the crew for a number of iterations and printing a table of per-task scores on a 1 to 10 scale, per-run columns, an average per task, the responsible agent, a crew average, and execution time.

The docs page lists two parameters. The shipped CLI in version 1.15.16 lists three:

$ crewai test --help
Usage: crewai test [OPTIONS]

  Test the crew and evaluate the results.

Options:
  -n, --n-iterations INTEGER  Number of iterations to Test the crew
  -m, --model TEXT            LLM Model to run the tests on the Crew. For now
                              only accepting only OpenAI models.
  -f, --filename TEXT         Crew-only: path to a trained-agents pickle
                              (produced by `crewai train -f`). When set,
                              agents load suggestions from this file instead
                              of the default trained_agents_data.pkl.
  --help                      Show this message and exit.

The documented defaults are out of date. The docs page states 2 iterations and gpt-4o-mini. Reading the defaults straight off the installed command object in 1.15.16 returns n_iterations = 3 and model = gpt-5.4-mini. That gap matters because iteration count drives token spend and the scoring model drives the scores themselves, so a version bump can silently change both your bill and your numbers.

Pin both explicitly and never rely on the default:

crewai test -n 5 -m gpt-4o

Three limits to plan around. The scorer is OpenAI-only, stated verbatim in the CLI help, so a crew running on Claude or a local model is graded by a judge from a different family. The score is an LLM judgment, not ground truth, which makes it useful as a trend and misleading as a gate. And it scores task output only, so it cannot see that the agent reached the right answer through the wrong tool.

Use it as a regression signal: record the crew average per release, and investigate a drop of more than a point rather than reacting to any single low run.

How Do You Test a Crew That Talks to Real Users?

Everything above tests the crew as a Python object. The moment it sits behind a chat endpoint, a support widget, or a voice line, none of it reaches the thing users touch. Guardrails cannot tell you the crew is condescending to a confused customer, and a 1 to 10 task score cannot tell you it leaked a policy detail on turn four of a nine-turn conversation.

That layer needs synthetic users holding real conversations. TestMu AI's Agent Testing platform connects to a live chat, voice, or phone endpoint and runs 15+ specialized evaluators against it in parallel, each probing one failure mode: a Hallucination Hunter for invented facts, a Bias Detector for differential treatment, a Context Specialist for whether the crew still remembers turn two, plus completeness, tone, escalation, and prompt-injection checks. Upload the PRD or policy doc the crew was built from and the platform generates 60 to 100+ scenarios spanning happy paths, edge cases, adversarial inputs, and compliance checks.

Two capabilities matter most for a CrewAI deployment. 10 pre-built personas (Confused Customer, Angry User, International Caller, Off-Script User, and others) exercise the inputs a developer testing their own crew never types. And the run resolves to a single Green, Yellow, or Red readiness verdict instead of a scores dashboard, so a release decision does not require reading every transcript. Connect a crew's HTTP endpoint using the chat agent API integration docs.

Note: A CrewAI crew behind a chat endpoint needs the same scenario coverage as any production agent. TestMu AI runs 15+ autonomous evaluators against your live endpoint and returns a Green, Yellow, or Red readiness verdict before you ship. Try TestMu AI free!

How Do You Wire CrewAI Tests Into CI?

The layers have different costs, so they need different triggers. Running everything on every push burns tokens and slows reviews to the point that people stop reading the results.

A practical trick for the nightly job: use CrewAI's kickoff_for_each, which executes the crew once per item in a list of inputs, to run a fixed regression set of prompts in one command and diff the tool-call sequences against the previous night. Behavioral drift shows up there days before a quality score moves.

Keep the baseline in version control next to the crew definition. A prompt edit that changes the crew average is information; a prompt edit that changes it without anyone noticing is a regression. The same discipline applied to conventional suites is covered in agent functional testing.

Which CrewAI Testing Mistakes Cost the Most Time?

• Asserting on generated text - A test that pins the final string passes once and then fails on every subsequent run for no real reason. Assert on structure, tool calls, and guardrail outcomes instead.
• Treating a crewai test score as a gate - The score comes from an LLM judge sampling a handful of runs. Gating a merge on it produces a flaky pipeline and teaches the team to rerun until green.
• Leaving memory on during tests - A crew configured with memory=True carries state between runs, so test two is not testing the same system as test one. Disable it for deterministic layers and test memory behavior deliberately in its own suite.
• Testing only the happy path you built against - Developers type clean, well-formed prompts. Real inputs are truncated, angry, multilingual, and off-topic, and that gap is where most production incidents live.
• Skipping the hierarchical manager in test coverage - Under a hierarchical process the manager agent makes delegation decisions that nothing else validates. Capture its task events and assert that delegation terminated.
• Assuming documentation defaults match the installed version - The CrewAI docs and the 1.15.16 CLI disagree on both defaults for crewai test. Read the flags off your own install before you write the pipeline.

If your crew is one of several frameworks in play, the trade-offs between them are covered in our guide to agentic AI frameworks, which walks through where role-based crews fit against graph-based and conversational alternatives. For the conversational side specifically, AutoGen testing covers the equivalent hooks in that framework.

Conclusion

Start by moving one tool body out of its CrewAI wrapper into a plain function and writing three pytest cases against it. That takes an afternoon, needs no API key, and gives you the first hard pass or fail signal your crew has ever had. Then attach a guardrail to the task whose output feeds the most downstream work, since that is where a bad shape does the most damage.

Once the deterministic layers are in place and your crew is answering real users, the remaining risk is scenario coverage. Point TestMu AI's Agent Testing at the live endpoint, upload the document the crew was built from, and let the platform generate and score the adversarial conversations nobody on the team would think to type. The Agent Testing platform getting started guide covers connecting an endpoint and reading your first readiness verdict.