← All guides · Agent observability and evaluation

How to Detect Quality Drift in a Production AI Agent

Part 7 of Agent observability and evaluation · 1,181 words · ~5 min read · published 2026-10-03 · updated 2026-10-03 · Markdown variant

How to detect quality drift in a production AI agent: baseline aggregate signals, alert on a diff against the baseline, and separate a prompt, model or tool-version change from a shift in traffic mix.

Part of the AI Agent Observability and the Production Evaluation Playbook guide.

Quality drift in a production AI agent is detected by comparing aggregate signals from live traffic against a baseline you recorded earlier, and then attributing any gap to a version change or a traffic change. The pillar covers what to collect. This article covers what to do with the numbers over time. It is reasoned design as of October 2026, not a published standard, and the "drift" phrasing here is our own label for the problem. Search demand is validated for the broader topic of AI agent observability, not for this wording.

How do you baseline an aggregate quality signal?

Baseline a signal by freezing its value, its definition and the versions that produced it at a moment when you believe the agent was behaving acceptably. The baseline is a record, not a feeling, and it has three parts:

• The number: the aggregate you will compare against, such as the rate of a flagged outcome, computed over a stated window.
• The definition: what counts as a flagged outcome, how the denominator is chosen, and which sampling rule applied.
• The versions: the prompt version, model ID and tool-schema hash that were live. The prompt-management-and-versioning resource treats these as three independently changing parts and advises emitting all three on every trace span, so the baseline can name them.

The signals themselves, and how biased each one is, are covered in the live-feedback article. Judge-scored samples belong in the same record, as described in judge screening in production. If the judge or rubric changes, the baseline resets, because the metric is now a different measurement.

No resource in the corpus gives a window length or a minimum sample, as of October 2026. Pick them from your own traffic volume, and write the choice down beside the number.

What should a drift alert compare?

A drift alert should compare the current aggregate with the recorded baseline for the same definition and the same slice of traffic, and fire on the difference, not on an absolute level. An absolute level has no meaning until you know what the agent did when it was healthy.

Three rules keep the alert honest:

1. Alert on a diff against a named baseline. Store the baseline next to the alert rule so a reader sees both numbers.
2. Check the count before the ratio. A rate computed over very few interactions is noise. Suppress or annotate the alert when the denominator is small.
3. Alert on the slice, not only the total. A blended total can stay flat while one request type gets worse and another improves.

The agent-observability resource describes online monitoring as alerting on anomalous patterns in the live stream, and says stored traces feed evaluation datasets. It does not specify a statistical method or a threshold, and this article does not add one. For the reliability signals such as errors and latency, see observability for reliability. For spend, cost telemetry is the place.

How do you tell a version change from a traffic-mix change?

Tell them apart by splitting the moved metric along two axes, version and request type, and seeing which split makes the movement disappear. If the movement vanishes inside each request type, the traffic mix explains it. If it persists inside every type and lines up with a version boundary, the version explains it.

The third row matters because the three version fields are not the whole environment. A tool can return different data with an unchanged schema. The evaluating-ai-agents resource says online evaluation on real interactions detects distribution shift, which is the traffic half of this table, and that public benchmark scores often diverge from production, so teams should evaluate on their own task distribution.

How do you investigate once an alert fires?

Investigate in a fixed order: confirm the denominator, find the version boundary, split by request type, then read traces. The order is cheap-first, so most false alarms end at step one or two.

1. Confirm the denominator. Did the count of interactions change, or the sampling rule?
2. Find the boundary. Line the metric up against deploy times and the composite version on each trace. If you cannot, the missing field is the finding.
3. Split by request type. Use whatever categories your traces already carry.
4. Read a sample of traces from each side of the change. Flagged ones first, plus a random few, so you see what the flags miss.
5. Decide. Revert, fix forward, or accept the change and re-baseline.

If the version is the cause, the revert path is in rollout and rollback, which also covers why the composite should move together. If a confirmed failure should never recur, turn it into a fixture.

Re-baselining is a decision and should be logged. A baseline that silently follows the metric will never alert.

What does ChangeGamer do that resembles baseline-then-diff?

ChangeGamer's build compares two sources of the 402 payment contract against one canonical key list and fails on any difference, which is the baseline-then-diff pattern applied to a contract instead of a quality metric. In changegamer/src/data/contract-402-lockstep.ts, the function assertKeysMatch (lines 46-80) runs a set check first, throwing "unexpected key(s) not in canonical contract" or "missing key(s) from canonical contract" (lines 51-68), then a positional order check that throws "key order mismatch at position" (lines 70-79). Each error message includes the expected canonical set.

The analogy is partial. That check is deterministic and exact, so any difference is a failure. Agent quality signals are noisy aggregates, so a diff needs a count check and a human judgment about whether it is real. The transferable parts are narrower: the canonical record is stored explicitly, the comparison is mechanical, and the failure message states both sides. ChangeGamer operates no production agent, and nothing here is a measurement of one.

What the corpus does not tell you

The corpus gives no drift threshold, window length, sample size or statistical test for agent quality, as of October 2026, and this article has not supplied any. The shape of the method, a recorded baseline, a version-tagged trace and a split before a conclusion, is reasoned from the resources above and from general practice. Treat it as a hypothesis to validate against your own traffic, and record the first false alarm you get, because it tells you more about your thresholds than any external number.

#agents #observability #evaluation #drift #monitoring #production