AI Monitoring: What to Track, Thresholds and Blind Spots

What Is AI Monitoring?

AI monitoring is the continuous measurement of AI systems in production, such as predictive models, LLM apps, RAG pipelines and agents, against a baseline, with an alert when a signal drifts. It covers the service (latency, errors, throughput), the spend (tokens, cost) and the behaviour (output quality, input drift, actions taken), and teams also call it AI system monitoring.

Search results for the term mix two meanings. One is monitoring AI systems; the other is AIOps, which applies machine learning to monitoring ordinary IT such as servers and networks.

For some systems monitoring will be a legal duty. Under Article 72 of Regulation (EU) 2024/1689, the EU AI Act, providers of high-risk AI systems must "establish and document a post-market monitoring system" in proportion to the nature of the AI technologies and the risks of the system. Under Article 113, in the consolidated text as at 27 July 2026 (read on 2026-10-20), the high-risk sections of Chapter III (classification, requirements and the obligations of providers and deployers) apply from 2 December 2027 for systems listed in Annex III and from 2 August 2028 for those covered by Annex I.

An AI monitor reads these layers, cheapest to watch first:

• Service - request rate, latency percentiles, error and timeout rates, the same signals as any web service.
• Usage and cost - tokens in and out per request, spend per workload, and how often requests reach a context or output limit.
• Behaviour - sampled output quality, safety flags, shifts in what users ask and, for agents, which tools they call.
• Outcome - whether the task succeeded, which needs labels or a check of the system the AI changed, and often arrives late or never.

How Is AI Monitoring Different From AI Observability?

In the OpenTelemetry observability primer, observability is what helps a team answer the question "Why is this happening?" about a running system. AI monitoring has a smaller remit: it charts signals picked before launch and pages someone when one of them leaves the range agreed for it.

Each needs the other: a monitor says that a given minute went wrong, and the traces say which prompt version, model or tool caused it. The AI observability guide covers how to instrument a system so those traces exist.

Signals to Monitor for Each Kind of AI System

The signals depend on what the system produces and on how long it takes to learn whether that output was right.

Microsoft's model monitoring documentation for Azure Machine Learning, dated 27 January 2026, draws the same line. Its data drift and data quality signals compare production inputs with training data or recent production data, its prediction drift signal compares outputs with validation, labeled test or recent production data, and its model performance signals compare outputs with "collected ground truth data", which the page marks as required.

• Watch proxies until labels arrive - drift and data quality signals move within minutes of a pipeline break, while accuracy can lag by weeks.
• Split every signal by workload and version - a chat feature and a code feature on one model have different normal ranges, as the trace numbers below show.
• Record the change history next to the charts - model, prompt, retrieval index and tool releases on the same timeline make a drift alert point at its cause; LLM prompt monitoring covers tying a change in replies to the prompt version behind it.
• Watch what users send - shifts in prompt length, language or topic mix are input drift, which can arrive with no change on your side, so chart them per workload.
• Read feedback week over week - thumbs-down rate and hand-offs to a person cost nothing to collect but come mostly from unhappy users, so compare weeks instead of reading a single day.

For LLM apps, generative AI monitoring adds checks on the content of prompts and replies, which latency and token metrics cannot see; LLM observability covers which checks to run, how much to sample and how to alert on the scores.

AI Performance Monitoring: Latency, Throughput, Cost and Quality

AI performance monitoring measures how fast, how much and at what cost an AI system serves requests, and how often the result is usable. Compute each metric per workload, model and version, since a blended number hides the workload that is failing.

Latency gets two numbers because inference has two phases. The Splitwise paper, whose data is the trace used on this page, describes "a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics". Time to first token tracks the first phase and time per output token the second.

If you instrument with OpenTelemetry, request duration and token counts map to gen_ai.client.operation.duration and gen_ai.client.token.usage, marked Development in semantic conventions v1.41.0; v1.42.0 deprecated every gen_ai.* metric and moved the GenAI conventions to their own repository, whose main branch has since replaced the token metric. The OpenTelemetry LLM metrics taxonomy lists the other GenAI metric names with their units and status.

The trace shows why blended dashboards mislead. In the conversation file, the median request generated 129 tokens from a 1,020-token context; in the code file, the median request generated 13 tokens from 1,469, according to the script output in the next section.

• Volume - conversation traffic never fell below 191 requests a minute, while code traffic had 12 empty minutes and peaked at 632.
• Cost driver - set against the length of its reply, the median code request carries far more context than the median conversation request, so a change in output-token price moves conversation costs far more relative to their total.
• Latency driver - long code prompts load the prompt phase, long conversation replies the generation phase, so each workload needs its own latency baseline.

How to Set AI Monitoring Alert Thresholds From a Baseline

To compare common alert rules, a Node script with no dependencies, run on 2026-10-20, computed baselines and alert counts on both files of the Azure LLM inference trace 2023, which Microsoft publishes under a CC-BY 4.0 licence. Each file has three columns: timestamp, context tokens and generated tokens.

Azure LLM inference trace 2023, public sample

workload  requests  full minutes  empty  per minute
                                          p50  min  max
conv         19366            58      0    326  191  507
code          8819            57     12    111    0  632

tokens per request   p50    p95    p99    max
conv context        1020   4083   4142  14050
conv generated       129    451    601   1000
code context        1469   7315   7436   7437
code generated        13     90    252   1899

ceilings (requests at the observed maximum)
conv context      max 14050  at max    1  within 2%     1
conv generated    max  1000  at max   11  within 2%    14
code context      max  7437  at max   18  within 2%   444
code generated    max  1899  at max    1  within 2%     1

// Monitor 2: a minute's p95 generated tokens over the p99
// of the requests in the trailing WINDOW minutes
function tokenRule(by) {
  const fired = [];
  for (let m = WINDOW; m < by.length; m++) {
    const pool = by.slice(m - WINDOW, m).flat()
      .map((r) => r.g);
    const now = by[m].map((r) => r.g);
    if (now.length < 20 || pool.length < 100) continue;
    if (pct(now, 0.95) > pct(pool, 0.99)) fired.push(m);
  }
  return fired;
}

// Monitor 2b: same comparison against a frozen baseline,
// the p99 of the requests in minutes 0-29, never updated
const FROZEN = 30;
function frozenRule(by) {
  const pool = by.slice(0, FROZEN).flat().map((r) => r.g);
  const th = pct(pool, 0.99);
  const fired = [];
  for (let m = FROZEN; m < by.length; m++) {
    const now = by[m].map((r) => r.g);
    if (now.length < 20) continue;
    if (pct(now, 0.95) > th) fired.push(m);
  }
  return fired;
}

// Monitor 3: a minute's request count over mean + 3 sd
// of the counts in the trailing WINDOW minutes
function rateRule(by) {
  const fired = [];
  for (let m = WINDOW; m < by.length; m++) {
    const counts = by.slice(m - WINDOW, m)
      .map((x) => x.length);
    const limit = mean(counts) + 3 * sd(counts);
    if (by[m].length > limit) fired.push(m);
  }
  return fired;
}

static rule: one request over mean + 3 sd (whole hour)
workload  threshold  requests over  share   minutes hit
conv            700             52  0.27%            37
code            207            124  1.41%            36
one-sided normal tail above mean + 3 sd: 0.13%

pages in the clean hour (no incident in the trace)
token: minute p95 over baseline p99 generated tokens
rate: minute requests over baseline mean + 3 sd
workload  checked  token rolling  token frozen  rate
conv           43              0             0     1
code           42              0             0     0
rolling = trailing 15 minutes; frozen = minutes 0-29
checked = minutes after the 15-minute warm-up;
the frozen rule checks minute 30 onward only
conv rate rule fired at minute 22

synthetic incidents on the conv workload (invented)
minutes firing that were quiet in the clean hour
incident                 token roll  token frozen  rate
A output doubled from m40         2            18     0
  token roll minutes: 40-41
  token frozen minutes: 40-57
B retries x3 in m45-47            0             0     1
  rate minutes: 45
C 30 wrong records in m50         0             0     0
  rows marked wrong: 30; shape unchanged

AI Monitoring Blind Spots: Failures That Raise No Alert

A monitor sees the shape of traffic, not its meaning. Incident C labelled 30 replies as changing the wrong customer record while leaving their timing and length as they were, the way a wrong write looks from outside, so every rule stayed quiet by construction.

Content is often missing too. The README of the Azure LLM inference trace states: "Due to customer privacy requirements (e.g., GDPR), we do not have visibility into the content of the prompts." It also dates the collection to 11 November 2023, while every timestamp in both files falls on 16 November 2023, so check a dataset's own records before trusting its description.

• What monitors catch - slow, failing, expensive, truncated or drifting traffic, and sampled replies that fail a check.
• What monitors miss - a fluent reply that is wrong, a write to the wrong record, an action that should never have run but finished cleanly, and any failure on an input that has not arrived yet.

For agents that act, the gap is widest, since their output is a change in another system. AI agent monitoring covers what to watch on agent runs in production, while confirming that a run changed the right thing takes a read of the system it changed, ideally before release.

AI Monitoring Tools by Category

Tools differ most in which layer they read. The AI observability tools and AI agent observability tools roundups compare named products.

Whatever the category, check OpenTelemetry support first. The OpenTelemetry Collector README says it "offers a vendor-agnostic implementation on how to receive, process and export telemetry data", so instrumentation written once can feed more than one backend.

Testing Agent Actions That Monitoring Misses, With Agent Assurance

Monitoring reports what production traffic looked like after users sent it. For an agent that edits files, calls APIs or changes records, the check that incident C needed belongs before release, against a system where a wrong write costs nothing.

TestMu AI's Agent Assurance runs that kind of check before release; it neither observes live traffic nor intervenes in a running agent. Rook CLI, which you run from a terminal, is pre-alpha and publicly installable, and you direct it at your own agent to run a suite of scenarios.

npm install -g @testmuai/rook
rook --version

Builds exist for macOS and Linux, and 64-bit Windows through npm or WSL, with a minimum of Node.js 20 for the npm package; for macOS and Linux the Rook CLI installation guide also describes Homebrew and a checksum-verifying shell installer.

• Scenarios written for this agent - Rook CLI reads the agent's code, or a PRD, spec or API description if you cannot share code, and drafts scenarios that each carry acceptance criteria; functional and adversarial classes are generated by default.
• How it reaches the agent - a profile tells Rook CLI whether to call an HTTP endpoint, run a command or use an MCP tool, and the scenarios execute against staging or inside a CI job.
• What counts as evidence - edits to files below the paths named in the profile, artifacts the run produced, the calls made set against the agent's declared tools and its must-not-call list, and, for a case like incident C, a lookup of the record through a read-only MCP tool that you supply and approve.
• Verdicts - a criterion can only end Pass, Fail or Unable to Verify; that third result is reported apart and never counted for or against the pass rate.

Some criteria are scored by model judges, yet an agent saying it made a change never counts as proof: a criterion about a changed record is decided by reading that record through a read-only tool you supply, and without such a tool it is never a pass (it comes back Unable to Verify, or the scenario is skipped). Most eval and observability tools score what your agent said and recorded. Agent Assurance checks what the run changed, and reports what it could not verify.

Writes made during a run stay in place, so aim runs at a staging copy of the systems the agent touches. The Agent Assurance CI/CD guide warns that "A process exit code of zero is not an agent-quality gate", so a pipeline should gate on the run's report, for example the output of rook report --json.

For conversational phone agents, TestMu AI's Agent Testing can score recorded production calls: the phone agent testing docs describe uploading batches of them and scoring them with the same metrics as test calls.

A Starting Plan for Monitoring One AI Workload

Start with the busiest workload: export a week of request logs, compute p50, p95 and p99 for latency, context tokens, generated tokens and requests per minute, then apply the baseline-to-alert steps above to that frozen week. If those metrics land in Prometheus, its guide to histograms and summaries notes that quantiles "are estimated" and that a classic histogram's error "is limited by the width of the bucket the quantile is located in", so place bucket boundaries close to the p95 and p99 you alert on.

Where an agent in the system writes to records or files, add a pre-release check of those writes. How Rook CLI connects to an agent is described in the Agent Assurance overview, and npm install -g @testmuai/rook installs the CLI.