← All guides · Agent reliability in production

How to Build an Incident Response Runbook for AI Agent Failures

Part 8 of Agent reliability in production · 1,146 words · ~5 min read · published 2026-08-29 · updated 2026-08-29 · Markdown variant

An operator playbook for the moment an AI agent fails in production: a triage step to classify the failure fast, trace freezing before rollback destroys the evidence, first-response depth on the four failure classes, and a blameless post-mortem that feeds back into guardrails and eval.

Part of the Agent Guardrails and the AI Agent Reliability Playbook guide.

The four minutes after an AI agent starts failing decide whether the incident costs a rollback or a much larger bill. The agent reliability in production pillar names four failure classes and their first responses in one short section; this article works the layer underneath that section — the triage step that tells you which class you are actually in, the trace-freezing habit that has to happen before any recovery action, and the post-mortem structure that turns an incident into a permanent fix rather than a story.

How do you triage an AI agent incident in the first few minutes?

Triage an AI agent incident by checking three fast signals before deciding on a response, because the four failure classes point to different fixes and applying the wrong one wastes the response window a real fix needed. Check spend velocity first — a sudden, sustained spike in token or dollar burn per minute against session or daily baselines points to runaway cost regardless of cause. Check error and output shape second — a rising failure rate on one specific tool, or every call to one dependency timing out, points to a tool or dependency outage, while a shift in task success rate or output distribution with no matching dependency error points to a prompt or behavioral regression from a recent release. Check for compromise signals last — credentials appearing in a context window, tool calls to scopes the agent has never touched, or data leaving through an unexpected channel all point to a security incident and override every other classification, since its containment steps take priority regardless of what else is going wrong.

These checks need no dashboard built specifically for triage — they reuse the trace and cost telemetry already collected per run, read against a recent baseline rather than a fixed threshold set months earlier. A team that skips triage and defaults to "roll it back" on every incident will occasionally revert a version that was never the problem, while a genuine outage or live compromise keeps running unaddressed.

Why should you freeze the trace before doing anything else?

You should freeze the trace before doing anything else because the two most common recovery actions — a rollback and a restart — can each destroy the evidence a post-mortem depends on. A rollback swaps the running version, so any trace context tied to the failing version's specific behavior stops accumulating the moment the swap happens; a restart clears in-memory state and can leave a trace store's last few spans incomplete or unflushed. Either action taken before the trace is exported turns "we have the full record of what the agent did" into "we have most of it, and the gap is exactly where the interesting behavior happened."

The habit itself is small: export or snapshot the trace tree for the incident window — every LLM call, tool call, and sub-agent hop under the run's trace ID — to storage independent of the production system, before triggering rollback, restart, or credential revocation. This costs a few minutes and belongs in the runbook as a named first step, not something a responder remembers only if time allows once the fire looks out. For a security incident specifically, storing traces independent of the production system matters doubly, since a compromised system cannot be trusted to log its own investigation.

What is the first response for each AI agent failure class?

The first response to an AI agent incident depends entirely on which of four classes it falls into, and each class has exactly one first move worth committing to before investigating further.

Runaway cost is the one class where stopping the bleed always outranks understanding it — a kill switch costs nothing extra to trigger and can be reversed once the cause is clear, while a runaway agent left running "to see what happens" only adds to the bill. The circuit-breaker response to a tool outage is a first move, not a full design; the internal mechanics a production circuit breaker needs are a deeper topic this runbook layer does not cover. A security incident's three-part response is likewise a first move only — threat classification, forensics, and containment beyond disabling the agent are a deeper discipline this article does not extend into.

What should a blameless post-mortem produce?

A blameless post-mortem should produce one of three concrete, shippable artifacts, not just a written account of what happened, because a narrative alone does not stop the same failure from recurring. Approach every incident as evidence that some check did not exist yet — an eval case, a guardrail rule, or an alert threshold — rather than as a person's mistake, and end the review by shipping whichever of the three actually closes the gap:

• An eval case covering the specific input or trajectory that triggered the incident, appended to the CI suite so a future release fails that check loudly in CI evaluation instead of failing silently in production.
• A new guardrail rule, added at whichever checkpoint — input, output, or action — would have caught the bad behavior before it executed; see agent guardrails for reliability for how those three checkpoints and their human-approval gate actually work.
• A new alert threshold, tuned against the triage signal that should have fired sooner — spend velocity, error rate, or output-distribution drift — so the next occurrence of the same class surfaces before a user notices rather than after.

A post-mortem that produces none of these three has not closed the loop, even if it correctly explains the incident. The frozen trace from the section above makes this review possible at all — a post-mortem run against an incomplete or overwritten trace can only guess at what happened, not verify it. Triage, trace freezing, and a post-mortem with a shipped artifact together separate a team that recovers from an agent incident from one that only survives it and waits for the next one; the full ten-dimension production checklist this runbook layer sits inside is in shipping AI agents to production.

#agents #reliability #incident-response #post-mortem #production #observability