← All guides · Agent reliability in production

The AI Agent Production Reliability Checklist

Part 12 of Agent reliability in production · 2,021 words · ~9 min read · published 2026-08-31 · updated 2026-08-31 · Markdown variant

A go/no-go checklist that turns the agent reliability pillar's twelve-discipline closing list into checkable gates — the specific artifact, header, or trace field that proves each one holds, with a link to whichever sibling article owns its mechanics.

Part of the Agent Guardrails and the AI Agent Reliability Playbook guide.

The agent reliability in production pillar closes with thirteen checklist bullets mapped onto twelve reliability disciplines — one terse line per discipline, stating what production-readiness requires without stating how a reviewer confirms it actually holds. This article turns each bullet into a go/no-go gate: the specific artifact, header, field, or test result that proves the condition is met, plus a link to whichever sibling article in this cluster owns that discipline's mechanics in depth. It does not re-derive any of those eleven siblings' content — a gate names the evidence a reviewer checks for, not the how-to behind producing it.

From checklist bullet to go/no-go gate

A checklist bullet and a go/no-go gate answer different questions about the same claim. A bullet like "guardrails are in place" or "the eval suite gates every promotion" names an intended state a team can assert without a reviewer ever confirming it held; a gate instead names the exact fact a reviewer checks directly — does this schema field exist, does this header actually get read, did this trace attribute actually get written on the last real run. The twelve gates below follow the pillar's own discipline order, since several depend on the ones before them: the rollout gate presumes a candidate already cleared the CI-evaluation gate, not the other way around, and a runbook's incident-response gate presumes the observability gate is already producing the trace data a post-mortem needs.

Is every model output contractually valid before anything executes?

A model's output is contractually valid once a schema physically rules out the shapes a downstream call would otherwise have to guess about — the first two gates confirm that guarantee actually holds in a test, not just that a provider offers the capability.

• Gate 1 — tool-calling contracts. Evidence: a schema audit walks every declared tool and confirms additionalProperties is set to false with no field left off the required list, and a deliberately malformed test call — one naming a tool the schema never declared — gets rejected before the handler executes, not discovered later from a crash log. Mechanics: tool-calling contracts for AI agents.
• Gate 2 — structured outputs. Evidence: the code path parsing a schema-constrained reply branches on stop_reason for max_tokens or refusal before it ever calls a JSON parser, since a 200 response alone does not rule out a truncated or refused object underneath it. Mechanics: structured outputs vs tool calling.

No-go: either gate fails if the only call on record is a clean, well-formed one — a contract nobody has tested against a malformed name or a truncated reply has not actually been verified, only assumed.

Can a retry or a crash-recovery replay duplicate a side effect?

Retries and idempotency get separate gates because a call can retry correctly on schedule and still duplicate a side effect if the key deduplicating it is built wrong.

• Gate 3 — retries. Evidence: the backoff implementation never returns a computed wait shorter than the value present in a live 429's own Retry-After header, confirmable by forcing that response in a test environment and checking the actual delay against it rather than trusting the backoff formula on paper. Mechanics: how to make AI agent retries idempotent.
• Gate 4 — idempotency. Evidence: a key-derivation audit confirms the value feeding a side-effecting call's dedup field comes from the workflow's own identifiers rather than from a timestamp function or a randomly generated identifier, verified in practice by replaying the identical step twice against a test double and counting only one real effect across both tries. Mechanics: how to make AI agent retries idempotent.

No-go: a retry path only ever exercised against a mocked success response has proven neither gate — both need a forced failure, and idempotency needs a forced duplicate attempt on top of that.

Does a stuck step get cut off before it stalls the whole run?

A stuck step should hit its own boundary and fail before it quietly spends the time budget every step after it was counting on.

• Gate 5 — timeouts. Evidence: the timeout attached to a given step comes from that step's own measured tail latency — inference calls, external API calls, and nested agent invocations don't share one number — and the sum of every step's retry-plus-timeout allowance still fits inside whatever SLA the calling context imposes, checked as a running chain total rather than per step in isolation. Mechanics: how to set timeouts for AI agent tool calls.
• Gate 6 — durable execution. Evidence: for any task that can outlive one process's lifetime, the orchestration code contains no direct clock read, no direct random-number call, and no direct LLM or API call — each of those instead runs inside a step or activity the engine records once and replays from, confirmable by killing the process mid-run and checking that whatever it does next picks up where the recorded log left off rather than re-triggering an action already taken. Mechanics: when do AI agents need durable execution.

No-go: a task that plainly crosses a process-lifetime boundary — a pending human approval, a multi-hour chain — but still runs on plain retry-and-timeout logic with no checkpoint log fails gate 6 outright, no matter how well gate 5 is sized.

Guardrails and CI evaluation: the two gates before a release ships

A release earns the right to reach a canary only after its behavior has been checked against policy and against a task suite, in that order.

• Gate 7 — guardrails. Evidence: input, output, and action guardrails each fire independently of one another, and a check that comes back inconclusive — no confident allow, no confident block — is treated as a block or an escalation on review rather than let through by default. Mechanics: how to design guardrails for AI agent reliability.
• Gate 8 — evaluation in CI. Evidence: the eval report scoring a release candidate carries a task-success-rate, a tool-call-accuracy, a cost-per-task, and a pass^k consistency figure side by side, not a lone pass/fail result from one run, and the CI pipeline itself runs mocked and cassette-replayed tests on every commit while a separate, small live-smoke suite runs on a nightly schedule and never blocks a pull request. Mechanics: how to evaluate AI agents in CI.

No-go: an eval report built from a single run's pass/fail, or a CI job that still calls a live model on every commit instead of replaying a stored exchange, has not cleared gate 8 regardless of how the numbers look.

Is there a record of the run afterward, and does a bad release come back out cleanly?

A release that ships without both of these gates leaves a team debugging blind if something goes wrong, and rolling back by hand if it does.

• Gate 9 — observability. Evidence: pulling one real trace shows every span holding four things at once — a computed dollar cost, its own duration next to the run's running total, a payload with personal data already stripped out, and whichever exception actually broke that step recorded on that step alone rather than rolled into one generic top-level error. Mechanics: what should an AI agent's observability system capture.
• Gate 10 — rollout. Evidence: a new version's canary carries somewhere between 5% and 10% of production traffic, an automatic rollback trigger watches for a falling task-success rate or a shifted output distribution rather than crashes alone, and the previous version keeps running through the whole canary window, which is what turns a rollback into swapping a pointer instead of standing up a new build. Mechanics: how to roll out a new AI agent version safely.

No-go: a rollback trigger wired only to exceptions and 5xx responses misses the failure gate 10 mainly exists to catch — a version that never throws an error but answers worse than the one it replaced.

Incident response and circuit breakers: when a gate fails anyway

Every gate above can pass in review and an agent can still fail once real traffic hits it, which is exactly what the last two gates are for.

• Gate 11 — incident response. Evidence: a written runbook exists that a responder can open and act on for each of four failure classes — a kill switch for runaway cost, a rollback for prompt or behavioral regression, a circuit breaker for a tool or dependency outage, credential revocation for a security incident — and exporting the trace store is the literal first line of that runbook, positioned ahead of a version swap or a restart so a post-mortem still has the spans it needs afterward. Mechanics: how to build an incident response runbook for AI agent failures.
• Gate 12 — circuit breakers. Evidence: a failure counter exists separately for every distinct thing the agent depends on — one per tool, one per model provider, one per retrieval index — rather than a single counter shared across an entire run, and a fallback behavior was chosen in writing, ahead of time, for whichever dependency is most likely to be the one that goes down first. Mechanics: how to design a circuit breaker for AI agents.

No-go: a runbook that lives only as an unwritten incident-channel habit, or a single breaker wrapped around an entire agent run instead of one per dependency, fails its gate regardless of how well-intentioned the team behind it is.

The twelve-gate summary table

None of these twelve gates amounts to much reviewed on its own — a schema field, a response header, a trace attribute, a traffic percentage. What makes the full review worth running rather than skimming is that the twelve are not actually independent once real traffic is involved: the same call whose schema gets audited in gate 1 is the one whose retry needs the dedup key checked in gate 4, whose slowest realistic case sets the number checked in gate 5, and whose eventual failure is exactly what gates 11 and 12 exist to have a pre-written answer for. Treating this as twelve unrelated checkboxes misses how often one bad production request works its way through most of them in a single run. See the agent reliability in production pillar for how these twelve disciplines fit together as a single stack, and each of the eleven sibling articles linked above for the mechanics behind its own gate.

#agents #reliability #production #checklist #operations