AI Agent Observability: Tools, Tracing, and Best Practices

There is now a standard shape for an agent trace. OpenTelemetry describes it as a top-level invoke_agent span with child chat spans for each model call and execute_tool spans for each tool invocation.

The conventions are recent and still settling, which explains most of the confusion in this area. Teams instrument something, they instrument it differently from each other, and few can answer the question that matters after an incident: why did the agent do that.

This article covers what belongs in an agent trace, how evals and tracing divide the work between them, which seven tools store and query those traces, and the practices that keep an agent debuggable once it is live.

Agent observability is the practice of instrumenting an AI agent so that a completed run can be reconstructed from the outside: the model calls it made, the tools it invoked and with what arguments, what those tools returned, how control moved between agents, and what the whole thing cost in tokens and wall time.

The shift from ordinary application observability is in the unit of interest. A web service trace answers what happened to a request. An agent trace has to answer why a decision was taken, because the code path was chosen at runtime by a model rather than written in advance by a developer.

For the wider practice across AI systems generally, the guide to AI observability and its benefits covers the foundations this builds on.

The two terms get used interchangeably and they are not the same scope. The difference decides what you can debug.

If you only capture the model calls, an agent that chose a reasonable-looking but wrong tool produces a trace in which every individual completion looks fine. The fault is in the sequence, and the sequence is what the agent layer adds.

The narrower discipline is still worth understanding on its own terms, because the model-call signals it defines are the leaves of every agent trace. The practical guide to LLM observability covers those pillars and what to monitor at the call level, and its rundown of LLM observability tools weighs the platforms that store that data on licence terms and portability.

Start from the OpenTelemetry GenAI conventions rather than inventing a schema, because a convention that is still moving is easier to follow than to retrofit. The conventions define the agent span, the tool span, and the attributes that identify an agent.

invoke_agent {gen_ai.agent.name}        the whole task
  ├── chat {model}                       a model call
  ├── execute_tool {gen_ai.tool.name}    a tool the agent ran
  │     arguments, result, error, duration
  ├── chat {model}                       the model reading that result
  └── invoke_agent {sub-agent}           work delegated onward

attributes worth setting on the agent span:
  gen_ai.agent.id        stable identity across runs
  gen_ai.agent.name      human-readable, appears in the span name
  gen_ai.agent.version   so a regression can be tied to a change

Beyond the span tree, three things are worth capturing because they are the ones you will wish you had during an incident.

• Tool arguments and results, not just tool names - knowing the agent called a refund tool is far less useful than knowing which amount it passed.
• The decision context - what the agent had in context when it chose, since a wrong choice made on incomplete input is a retrieval bug rather than a reasoning one.
• A terminal outcome marker - an explicit record of whether the task ended in the action it was meant to produce, which is what lets you measure completion rather than absence of errors.

Those attribute names are not invented here. gen_ai.agent.id, gen_ai.agent.name, and gen_ai.agent.version all appear in the OpenTelemetry GenAI attribute registry, alongside the tool attributes below.

Instrumenting this takes less code than most teams expect. The example below emits the full span tree with the plain Node SDK, with the model and tool calls stubbed so the shape stays reproducible.

const { NodeTracerProvider, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-node');
const { SpanStatusCode, trace } = require('@opentelemetry/api');

const tracer = trace.getTracer('refund-agent');

await tracer.startActiveSpan('invoke_agent order-desk', async (agentSpan) => {
  agentSpan.setAttributes({
    'gen_ai.operation.name': 'invoke_agent',
    'gen_ai.agent.name': 'order-desk',
    'gen_ai.agent.id': 'agt_3d90b1',
    'gen_ai.agent.version': '2026.08.3',
  });

  // the tool span is the one that earns its keep: arguments AND result
  await tracer.startActiveSpan('execute_tool issue_refund', async (s) => {
    s.setAttributes({
      'gen_ai.operation.name': 'execute_tool',
      'gen_ai.tool.name': 'issue_refund',
      'gen_ai.tool.call.arguments': JSON.stringify({ order_id: 'A-4471', amount_cents: 12999 }),
      'gen_ai.tool.call.result': JSON.stringify({ status: 'ok', refund_id: 'rf_9912' }),
    });
    s.end();
  });

  // terminal outcome marker: lets you measure completion, not absence of errors
  agentSpan.setAttribute('agent.outcome', 'refund_issued');
  agentSpan.setStatus({ code: SpanStatusCode.OK });
  agentSpan.end();
});

Running that against OpenTelemetry SDK 2.10.0 produces the tree below. This is the actual console output, not an illustration.

trace_id = f89aabe32fa32cd626764de27ee2ccee
invoke_agent order-desk  [1.1ms]
     gen_ai.agent.id = "agt_3d90b1"
     gen_ai.agent.version = "2026.08.3"
     agent.outcome = "refund_issued"
  └─ chat gpt-4o  [0.2ms]
       gen_ai.request.model = "gpt-4o"
       gen_ai.usage.input_tokens = 412
       gen_ai.usage.output_tokens = 38
       gen_ai.response.finish_reasons = ["tool_calls"]
  └─ execute_tool issue_refund  [0.1ms]
       gen_ai.tool.name = "issue_refund"
       gen_ai.tool.call.arguments = "{\"order_id\":\"A-4471\",\"amount_cents\":12999}"
       gen_ai.tool.call.result = "{\"status\":\"ok\",\"refund_id\":\"rf_9912\"}"
  └─ chat gpt-4o  [0.0ms]
       gen_ai.request.model = "gpt-4o"
       gen_ai.usage.input_tokens = 96
       gen_ai.usage.output_tokens = 24
       gen_ai.response.finish_reasons = ["stop"]

Read the refund amount in that tool span. 12999 cents is the fact an incident review needs, and it is the one a trace that records only tool names throws away.

The terminal outcome marker earns its place twice over. An agent that records what it did is easier to verify later, which is the same property that determines how much of its behaviour any test harness can confirm.

Note: Execution records only become useful once something aggregates them across runs. TestMu AI turns per-run results into trends, flakiness signal, and root cause leads. Try it free.

Teams often adopt one and treat it as covering the other. They sit at different points in the lifecycle and fail in opposite ways.

A team with only evals ships confidently against a case set that drifts further from production every week. A team with only tracing learns about every problem from a customer. The agent eval framework guide covers the eval side in depth, and the roundup of AI agent evaluation tools compares what is available to run them.

The value is not in running both. It is in letting each one feed the other, which most teams never wire up.

• Traces become eval cases - when a production run goes wrong, the trace already contains the inputs and the context. Promote it into the eval set so the next change is measured against it, and the same incident cannot recur silently.
• Eval failures tell you what to instrument - if an eval fails and the trace cannot show why, that is a gap in instrumentation rather than a gap in the agent. Add the attribute and the next failure explains itself.
• Production distribution reweights the eval set - trace data shows which intents actually arrive and in what proportion, which is the only honest basis for deciding what the eval suite should overweight.
• Agent version ties both together - tagging traces and eval runs with the same version attribute is what turns two separate dashboards into one answer about whether a release helped.

The fourth point is the cheapest and the most often skipped. Without a shared version identifier, a rise in production failures cannot be attributed to the change that caused it.

Every claim below was checked against the vendor's own live documentation while writing this section. Four questions decided the order.

• Does it record the whole tree - model calls, tool calls with their arguments and results, and handoffs, rather than model calls alone.
• Is the data OpenTelemetry-shaped - which decides how much rework it costs to move to a different backend in a year.
• Is there a self-hostable option - the deciding factor for teams that cannot send prompt and tool payloads to a vendor.
• Does it close the loop into evaluation - whether a recorded production run can become a test case without being retyped.

One note on the ordering. TestMu AI appears last because it answers a different question from the other six, not because it scores worse on the same one. Its scope is stated plainly in that entry.

Langfuse frames tracing as structured logs of every request, capturing the exact prompt sent, the model's response, token usage, latency, and any tools or retrieval steps in between. That last clause is the part that matters for agents, because the retrieval and tool steps are where agent-specific failures live. It also accepts OpenTelemetry traces directly, on a native OTLP endpoint.

Its documentation states that Langfuse is open source and can be self-hosted using Docker, running the same codebase that powers its cloud offering, and that it depends only on open source components so it can be deployed locally, on cloud infrastructure, or on premises.

• Pick it when data residency rules out shipping prompt and tool payloads to a vendor, since the self-hosted deployment is the same code as the managed one.
• Know the limit - the documentation is explicit that some add-on features require a license key, so self-hosting is not automatically the full feature set.

Phoenix describes itself as an open-source AI observability platform for experimentation, evaluation, and troubleshooting, and it is explicit that its tracing works by instrumenting an application's runtime with OpenTelemetry.

That single design decision is why it ranks this high against the portability criterion. Traces emitted through OpenTelemetry instrumentation are readable by any OTLP-compatible backend, so the instrumentation work survives a later change of vendor.

• Pick it when you want the instrumentation you write this quarter to still be worth something if the backend decision is revisited.
• Know the limit - open-source Phoenix and the commercial Arize platform are different products, so check which one a given capability belongs to before planning around it.

LangSmith positions traces as the record of what agents did in production, used to debug failures, monitor quality, and build the datasets you evaluate against. That framing maps directly onto the loop described earlier in this article, where a bad production run becomes the next eval case.

It is worth correcting a common assumption here. Its documentation lists integrations spanning OpenAI, Anthropic, CrewAI, Vercel AI SDK, and Pydantic AI, so it is not restricted to teams building on LangChain.

• Pick it when the eval set is the artifact you care about most and you want promoting a trace into it to be a routine action rather than a project.
• Know the limit - it is a hosted product, so teams with strict residency requirements should confirm the deployment options available on their plan before committing.

Check the product name before citing this one in a design document. What many teams still call Datadog LLM Observability now sits under Agent Observability in its documentation, which describes monitoring, troubleshooting, and evaluating LLM-powered applications, with each request represented as a trace.

Its documentation notes that a trace can represent an individual LLM inference, including tokens, error information, and latency, or a predetermined LLM workflow. The pull here is not the agent features in isolation but adjacency: agent traces land beside the infrastructure and application telemetry the team already watches.

• Pick it when the on-call engineer who will be paged about an agent already lives in Datadog dashboards, because correlation across the stack beats a better standalone trace viewer.
• Know the limit - consolidating on one vendor is exactly what the portability criterion warns about, so weigh the convenience against the exit cost.

Braintrust captures inputs, outputs, model parameters, latency, token usage, and metadata for every request, and assigns a type to each span. One of those types is an eval root span that wraps a task span for your application code.

That is a more meaningful detail than it first appears. When a logged production run and a test case are the same shape of object, moving between them stops being an export step.

• Pick it when the team's real problem is that evaluation and production monitoring currently live in two systems that disagree.
• Know the limit - the span typing is its own model rather than the OpenTelemetry vocabulary, so plan for a translation layer if portability is a hard requirement.

Helicone takes the gateway route. Its quickstart promises a first logged request in under two minutes, achieved by pointing the standard OpenAI SDK at an OpenAI-compatible gateway that fronts access to more than 100 models across providers including OpenAI, Anthropic, Vertex, and Groq.

Changing a base URL is a far smaller commitment than writing span instrumentation, which makes this the cheapest way to answer whether anyone will actually look at the traces before investing in the full tree.

• Pick it when you need cost and usage visibility this week and cannot get instrumentation onto the sprint.
• Know the limit - a gateway sees model calls passing through it. Tool executions that never touch the gateway are invisible to it, which is precisely the agent-shaped gap this article opened with.

The six tools above watch an agent serving live traffic. Test Insights watches the suite that is supposed to catch problems before that traffic arrives, aggregating execution records across builds, time, browser and device configurations, teams, and projects.

What it turns that history into is trend dashboards, flakiness and stability signal, error categorization, and agentic root cause analysis that correlates network, console, and framework logs to localize a likely cause.

• Pick it when the question is whether the eval and regression suites gating your agent releases are themselves trustworthy, which no production tracer can answer.
• Know the limit - it is observability over testing, not over production traffic. It does not run tests, does not decide pass or fail, and its root cause output is a lead to verify rather than a verdict.
• Know the second limit - every trend it draws inherits the quality of the upstream instrumentation. Weak assertions produce a green trend that means very little.

Adding agents changes the shape of the problem rather than its size. One logical request stops being a line and becomes a tree, often executed partly in parallel.

• Trace context must survive every handoff - if the identifier is not propagated when one agent delegates to another, you are left with several partial stories and no way to join them.
• Attribution replaces error rate - the useful question stops being how many runs failed and becomes which agent's decision caused the outcome, which only a joined trace can answer.
• Concurrency introduces write ordering - two agents acting in parallel can touch the same record, and without timestamps and identity on each tool call the last writer is invisible.
• Cost accounting needs a per-agent split - a rising bill is unactionable until it is attributed to the agent and the tool that produced it.
• Rollout has to be per agent - releasing one specialist at a time keeps the blast radius small and makes a regression attributable, which a whole-system deploy does not.

Deployment practice follows from that. Ship one agent at a time behind a flag, keep the previous version routable so a bad release can be reversed without a rebuild, and hold the eval suite as the gate on the way in. The methodology for that gate is covered in the guide to multi agent testing, and the routing patterns these deployments rest on are in the piece on agentic AI orchestration.

Instrumenting everything at once produces a large bill and a dashboard nobody reads. This order gets a usable signal quickly.

• Emit the agent span and tool spans using the OpenTelemetry names, so your data is portable if you change backend later.
• Add a stable agent id and version attribute before you add anything else, because they are what make every later comparison possible.
• Record tool arguments and results, with secrets and personal data redacted at the point of capture rather than in the backend.
• Mark terminal outcomes so completion is measurable, not inferred from the absence of an exception.
• Sample deliberately, keeping every failed and every adversarial-looking run while sampling the successful ones.
• Only then build dashboards, and build them around drift signals rather than raw volume.

On redaction, treat the trace as a system that will be read by more people than the agent's own logs. Tool arguments are exactly where account numbers and personal data end up.

The rollout order above gets traces flowing. These five practices are about what goes wrong afterwards, once the instrumentation exists and somebody has to live with it.

• Alert on behavioural drift, not error rate - watch for a shift in which tools get called, a rise in runs ending without a terminal action, growth in steps per task, and cost per completed task. An agent that fails loudly is the easy case; the expensive one succeeds while doing the wrong thing, and no error-rate alert will ever fire for it.
• Keep content attributes bounded - OpenTelemetry's own guidance notes that captured prompt and completion content can be large, and that many backends render it as raw JSON. Decide a size cap and a sampling rule for message content before the first bill, not after it.
• Attribute cost per agent and per tool - the GenAI conventions define a gen_ai.client.token.usage metric that can be filtered by token type. Without that split a rising bill is a number nobody can act on; with it, one chatty sub-agent becomes visible in an afternoon.
• Promote failed traces on a cadence, not on outrage - put a recurring slot in the sprint where the worst production traces of the week become eval cases. Teams that only promote traces after an incident build a suite shaped entirely by their most embarrassing failures.
• Measure what you could not verify - track the share of behaviour your checks could not confirm, and treat that number as a property of the agent rather than of the harness.

That last practice is the one most teams have no number for, and it is where instrumentation quality stops being an abstraction. TestMu AI's Agent Assurance reports it directly as an assurance gap, the percentage of criteria a run could not check, sitting beside the pass rate instead of being folded into it.

Measured on TestMu AI's own reference agents, on suites of the same shape, the gap tracks how much each agent records about itself.

These are TestMu AI's own reference agents rather than customer data, and they were built to illustrate the mechanism rather than to benchmark anything. Read them for the direction, which is steep: adding an audit log of tool calls moved unverifiable criteria from 60% to 21% on the same suite shape.

The point generalizes past any one product. Observability is not only how you debug an agent after it misbehaves, it is what decides how much of its behaviour anyone can ever prove. The conversational side of that grading is covered on the AI agent testing platform page.

Start by adding two attributes: a stable agent id and a version on every trace you already emit. Without them no comparison across releases is possible, and with them most of the other analysis becomes available later without re-instrumenting.

Then wire the loop in one direction before both: take the next production run that goes wrong, promote it into your eval set, and gate the next release on it. For background on where this sits in a wider QA practice, see the guide to agentic quality assurance, and for the pre-release half of the loop, end to end agent testing.

To put run history behind a dashboard rather than a spreadsheet, create a free TestMu AI account and start with the analytics dashboard documentation.