← All guides · Agent reliability in production

Agent Guardrails and the AI Agent Reliability Playbook

Pillar guide · 5,776 words · ~26 min read · published 2026-08-25 · updated 2026-09-04 · Markdown variant

Agent guardrails plus the eleven other disciplines that make an AI agent reliable in production: tool calling, retries, durable execution and rollout.

What are agent guardrails, and what does "reliable" mean for an AI agent?

Agent guardrails are runtime checks — on input, on the model's output, and immediately before a tool call fires — that enforce policy regardless of what the underlying model happens to produce on a given call. They are one of twelve disciplines that make an AI agent reliable: predictable and recoverable under the failure conditions that are certain to occur — malformed model output, a rate-limited API, a crashed process, a bad code deploy — not merely working on a demo run. Chat-completion reliability and agent reliability are different disciplines. A single-turn chatbot fails softly: a bad response is annoying but bounded. An agent fails compoundingly: a hallucinated tool call executes against a real system, a missed retry duplicates a database write, a crash mid-workflow loses hours of progress, and a bad deploy propagates to every user before anyone notices. Reliability engineering for agents is the set of mechanical safeguards that convert those compounding failures into bounded, recoverable, observable ones.

This guide treats reliability as a stack of twelve disciplines, roughly in the order a request travels through an agent and then through its release pipeline:

1. Tool-calling contracts — does the model reliably call the right tool with valid arguments?
2. Structured outputs — when you need a schema-valid final answer instead of a tool call, does the provider guarantee it?
3. Retries and rate limits — how does the agent behave when a provider or tool returns a transient error?
4. Idempotency — do retries and crash-recovery replays avoid duplicating side effects?
5. Timeouts — how long does a step wait before the system gives up and reacts?
6. Durable execution — does a long-running task survive a crash or restart?
7. Guardrails — does the agent's behavior stay within policy regardless of what the model outputs?
8. Evaluation in CI — is reliability measured and gated before every change ships?
9. Observability — can you reconstruct what an agent did and why, after the fact?
10. Rollout — does a new version reach production gradually, with automatic rollback?
11. Incident response — does the team have a pre-written response for each failure class?
12. Circuit breakers and degraded mode — does the system fail fast and gracefully instead of stalling or cascading?

None of these is exotic engineering; each has a standard pattern documented in the reference corpus this guide draws from. What changes production outcomes is doing all twelve deliberately instead of discovering the gaps one incident at a time.

Why do tool-calling contracts matter more than prompt engineering?

Tool-calling contracts matter more than prompt engineering because a single malformed tool call turns an otherwise-working agent step into an error-handling problem downstream, and no amount of prompt polish eliminates that risk the way schema enforcement does. As of August 2026, every major provider offers some form of constrained decoding that physically restricts which tokens the model can emit for a tool call or a structured final answer, rather than merely asking nicely: OpenAI's strict mode requires every object to set additionalProperties: false and every property to appear in required; Anthropic supports strict: true on individual tool definitions plus a separate Structured Outputs mechanism (output_config) for the final reply; Google Gemini forces compliance through ANY-mode function calls or a responseSchema/responseJsonSchema config; and self-hosted models push the same guarantee down to token sampling via grammars such as GBNF (llama.cpp), XGrammar (the default backend for vLLM, SGLang, TensorRT-LLM, and MLC-LLM), or Outlines.

Known failure modes and fixes

Even with constrained decoding enabled, real failures still recur in predictable shapes: a hallucinated tool name outside the declared set, a missing required argument, extraneous fields, unparsable JSON when strict mode isn't enabled, wrong field types, over-calling or under-calling tools, and out-of-order parallel calls with unstated dependencies. Each pairs with a known fix — validate the returned tool name against your schema before executing, enforce required/additionalProperties: false, strip unknown keys defensively, wrap parsing in try/catch with a retry-and-repair loop that sends the validation error back to the model as a correction request, prefer enums and const values over free-text fields, and set disable_parallel_tool_use: true (Anthropic) or force sequential tool_choice when call order matters. The Berkeley Function Calling Leaderboard (BFCL), maintained by the Gorilla team at UC Berkeley, is the standard benchmark for this failure class, scoring both single and simultaneous tool calls via AST comparison; its fourth iteration (current as of August 2026) folds in multi-step agentic tasks rather than testing isolated calls only. The full failure-mode table and mitigation reference lives in reliable tool calling and structured outputs.

The cross-cutting mitigation that matters most in practice: keep schemas shallow with a minimal set of required fields. Every optional field is a reliability risk, because it is one more place the model can omit, mistype, or hallucinate a value — a lesson that shows up just as often in MCP tool handlers as in first-party function calling (see common MCP server failure modes for the crash-recovery and malformed-call-handling side of this same problem at the protocol layer).

When should you use structured outputs instead of tool calling?

Use structured outputs when you need the model's own final answer in machine-parseable form and use tool calling when you need the model to invoke an external function — the two mechanisms are orthogonal and can be required together or separately. Tool calling produces an intermediate instruction for your application to execute (a named function plus arguments); structured outputs constrain the model's concluding response itself into a schema-valid object, with no function invoked. A classification task that just needs {"category": "billing", "confidence": 0.92} back from the model wants structured outputs, not a tool call; an agent that needs to actually query a database or send an email wants a tool call.

The provider mechanisms differ in what schema subset they support, which matters when you design a contract meant to survive a provider swap:

Two failure modes catch teams even under a strict-mode guarantee. First, truncation and refusal bypass the schema: a response cut off by stop_reason: "max_tokens" or an explicit safety refusal is not schema-compliant even though strict mode was enabled, so always check the stop reason as a first-class error branch rather than assuming a 200 response means a valid object. Second, first-request latency rises from grammar compilation the first time a new schema is used — mitigate with a warm-up call and by avoiding frequent schema churn. OpenAI now treats plain JSON mode (json_object) as legacy precisely because it only asks for syntactically valid JSON with no schema enforced; prefer json_schema with strict: true wherever the provider supports it. The full per-provider schema-subset comparison is in structured outputs and JSON mode.

How should an agent handle retries and rate limits?

An agent should treat a 429 (Too Many Requests) as a signal to honor the provider's Retry-After header and wait that long before retrying — never retry immediately, and never treat rate limits as a bug to route around rather than a budget to respect. Provider limits stack across at least three windows: requests per minute (RPM), tokens per minute (TPM, sometimes split into separate input/output limits as Anthropic does), and a daily ceiling (TPD/RPD) that overrides any remaining per-minute headroom. Both OpenAI and Anthropic loosen these ceilings as account spend and age grow through numbered usage tiers, but exact thresholds shift over time — check your provider dashboard rather than hard-coding a number, and request a tier increase ahead of need rather than reacting to production 429s.

The standard backoff pattern is exponential backoff with full jitter — randomizing the wait uniformly between zero and an exponentially growing cap, while always enforcing the provider's mandated Retry-After as a hard floor:

MAX_RETRIES = 7
BASE_DELAY  = 1      # seconds
CAP_DELAY   = 60     # seconds

def call_with_backoff(request):
    for attempt in range(MAX_RETRIES):
        response = send(request)
        if response.status != 429:
            return response
        floor = int(response.headers.get("Retry-After", 0))
        ceiling = min(CAP_DELAY, BASE_DELAY * 2 ** attempt)
        wait = max(floor, random_uniform(0, ceiling))
        sleep(wait)
    raise RateLimitExceeded("max retries reached")

Full jitter matters at fleet scale: without it, many retrying clients synchronize their backoff schedules and hammer the API in the same instant, turning a transient rate limit into a retry storm. Before ever reaching a 429, well-behaved agents self-throttle proactively — track a local token budget from the *-remaining response headers, cap concurrent in-flight requests with a semaphore, and pre-estimate token cost to hold back requests that would exceed remaining headroom. Not every error deserves a retry: distinguish transient failures (network timeout, 429, 5xx) that should retry with backoff from terminal failures (400 bad request, business-logic error) that should escalate immediately — retrying a malformed request forever just delays the failure. For high-volume, non-interactive workloads — evals, bulk summarization, embedding a large corpus — both major providers offer a batch API on a separate rate-limit pool at roughly half the per-token price with same-day turnaround; it is the wrong tool for a latency-sensitive interactive agent but the right one for the CI evaluation suite discussed later in this guide. See handling rate limits and retries for the full header reference and tier-advancement details.

Why does idempotency matter more for agents than for a simple API call?

Idempotency matters more for agents than for a stateless API call because an agent retries far more often — after rate limits, after transient tool errors, and after crash recovery — and every one of those retries risks re-executing a side effect that already happened. A crash can leave a step mid-flight, and a naive retry (or a durable-execution replay, discussed below) may attempt that step a second time. If the step is a database write, a payment charge, or an outbound email, a duplicate attempt is a duplicate side effect — a double charge, a double email, a corrupted counter.

The standard fix is a stable idempotency key derived from the workflow run ID and step index — never from wall-clock time or a freshly generated random value, both of which change on every retry and therefore cannot deduplicate anything. Pass that key to the downstream service (most payment, email, and messaging APIs accept an idempotency-key header) so a duplicate call becomes a no-op instead of a duplicate effect. This single pattern is the difference between "the agent retried and it worked" and "the agent retried and now the customer has two invoices," and it applies whether the retry originates from your own backoff loop, from a durable execution engine's crash recovery, or from an MCP client resending a tool call after a timeout.

What timeout should an agent use for a tool call or LLM step?

Every step in an agent's execution — an LLM call, a tool call, a sub-agent hop — needs an explicit timeout tied to what a reasonable worst case looks like for that specific operation, not a single global value copied across every step; no universal number exists because the right timeout depends entirely on what the step does. A timeout that is too short cancels legitimate slow operations (a large document extraction, a complex database query) and converts them into retries that compound latency; a timeout that is too long lets one stuck dependency stall an entire agent run and burn budget while nobody notices.

Two design choices compound with timeouts and are worth setting deliberately rather than defaulting:

• Bound the retry budget in wall-clock time, not just attempt count. A capped exponential backoff with several retries can still take a minute or more to exhaust; if the caller (a user-facing request, an upstream orchestrator) has its own SLA, the agent's total retry-plus-timeout budget must fit inside it, or the caller times out first and the retries become wasted work.
• Treat a timeout as a distinct outcome from an error. A step that times out has an unknown final state — the underlying operation may have completed on the far side even though the response never arrived. This is exactly the scenario idempotency keys exist for: a timed-out payment call retried with the same key is safe to resend even though you cannot be sure whether the first attempt succeeded.

Durable execution engines (next section) make per-step timeout policy — maximum attempts, initial interval, backoff multiplier, and a non-retryable error list — a first-class configuration rather than something reimplemented ad hoc in each workflow, as detailed in durable execution for long-running agents.

What is durable execution and when do you actually need it?

Durable execution is the programming model that lets a long-running agent workflow survive a crash, a restart, or a wait for human approval by persisting every meaningful step to a log before moving on, so that on resume the engine reconstructs state from stored records instead of re-executing real-world actions a second time. You need it for any agent task that can outlive a single process lifetime — a workflow that pauses for human approval, that runs for minutes to hours across multiple tool calls, or that must survive a server restart without losing progress. You do not need it for a short-lived, single-turn agent call that completes within one request handler in seconds; that case still needs ordinary retry and timeout handling, just not a checkpoint log.

Two constraints make durable execution work correctly, and violating either one is the most common way teams misuse it:

• Workflow code must be deterministic. Because resuming re-runs the orchestration function, that function must produce the same sequence of commands given the same recorded inputs. Non-deterministic operations — clock reads, randomness, direct LLM or API calls — cannot live inside the workflow logic itself; they belong in a separately recorded unit (called an activity, step, or handler depending on the engine) that executes outside the replay path and is injected from its stored result on resume.
• Side-effecting calls need the idempotency key described above, because a crash can cause a step that was mid-flight to be attempted again during replay.

Current engines fall into four architectural shapes, and the right choice usually follows existing platform commitment rather than an abstract feature comparison:

A durably parked workflow — one suspended on an external signal such as a human approval or a webhook — consumes no compute while waiting and restores full state the instant the signal arrives. This is architecturally distinct from a polling loop or a sleeping thread, and it is the mechanism that makes human-in-the-loop approval gates (see guardrails, below) actually durable rather than a fragile in-memory wait with a timeout attached.

How do guardrails keep an agent's behavior reliable, not just safe?

Guardrails keep an agent's behavior reliable by enforcing policy at runtime regardless of what the underlying model happens to output on a given call, which matters for reliability specifically because model behavior is inherently variable across calls even at fixed settings. In reliability terms — as distinct from the security-hardening question of defending against an adversarial attacker, which belongs to a different discipline — the guardrail question is simpler: does the agent behave correctly and predictably, and does a tool call that would produce an incorrect or unapproved action get stopped before it executes rather than after?

Guardrails apply at three checkpoints, each catching a different reliability failure:

• Input guardrails, before the model sees a request — catch malformed or off-policy input before it can produce a malformed downstream action.
• Output guardrails, after the model replies but before the caller sees it — catch schema or format violations and groundedness failures (does the answer stay within the intended scope?).
• Action guardrails, immediately before a tool call fires — catch calls outside an allowlist, calls exceeding defined parameter ranges, and irreversible or high-stakes operations that should require human approval rather than executing autonomously.

No single check catches everything a model might produce, so the practical pattern layers cheap, fast checks (regex and deterministic validators for format and known patterns) underneath slower, more thorough ones (a classifier for known categories, an LLM-as-judge for nuance), and defaults to blocking or escalating when a check is uncertain rather than defaulting to allow. Every guardrail verdict — allowed or blocked, which rule fired, what score it produced — should be logged against the run's trace ID, both to debug false positives that degrade usability and false negatives that provide false confidence, and to feed the observability discipline covered next. A single tool call should touch at most two of three properties — processing untrusted input, accessing a sensitive system, or changing external state — never all three at once; a call that would touch all three (say, acting on instructions found in a scraped web page to modify production data) is exactly the shape of action that belongs behind a human-approval gate rather than autonomous execution. The full checkpoint taxonomy and tooling landscape is in guardrails and safety filters for agents.

How do you evaluate whether an agent is reliable enough to ship?

Evaluating agent reliability requires scoring multi-step trajectories against your own task distribution, not a single-turn response or a public benchmark alone, because a model that scores well on a chat benchmark can still fail badly as an agent if it cannot recover from tool errors, maintain state across steps, or complete long-horizon tasks consistently. Three properties distinguish agent evaluation from ordinary LLM evaluation:

• Multi-step trajectories, not single-turn correctness. Errors compound across steps, so partial-credit metrics — did the agent complete 8 of 10 sub-steps correctly? — carry more signal than a binary pass/fail on the final outcome alone.
• Reliability across repeated runs, not a single trial. The same prompt can produce different tool-call sequences across runs, so a single pass/fail measurement is misleading. Use pass@k (at least one of k trials succeeds — a ceiling-capability metric) and pass^k (all k trials succeed — a consistency metric, used as tau-bench's primary metric) rather than reporting one run as the answer.
• Cost and latency as first-class metrics alongside accuracy. An agent that succeeds but costs substantially more or takes substantially longer than a simpler baseline is not actually ready to ship; token spend and wall-clock time per task belong in the same eval report as success rate.

Public benchmarks — SWE-bench for software-engineering tasks, GAIA for cross-tool generalist reasoning, BFCL for tool-call correctness, WebArena for web navigation, and others — are useful proxies but reliably diverge from production performance, so build an eval suite against your own task distribution and guard against benchmark contamination rather than shipping on leaderboard position alone. For tool-call correctness specifically, exact-match or AST-based comparison against a reference answer is more reliable than LLM-as-judge, which carries known biases (position bias, verbosity bias, self-preference when a model judges its own family's output) — reserve LLM-as-judge for open-ended subjective quality and use ground-truth matching for anything with a checkable structure. See evaluating AI agents for the full benchmark reference table.

Testing agents in CI without flaky, expensive builds

Testing an agent in CI resolves the tension between non-deterministic models and deterministic pipelines by testing different layers at different levels rather than trying to make every test reproducible end to end. A three-layer pyramid works in practice: Layer 1 is ordinary deterministic unit tests of the code around the model — tool functions, parsers, schema validators, retry logic — with the LLM client mocked entirely, run on every commit. Layer 2 uses cassette/VCR-style recorded fixtures: the first run hits the real API and serializes the exchange to a committed file; every subsequent run replays it, fast and network-independent, re-recorded only when prompts or schemas change. Layer 3 is a small, hand-curated set of live smoke tests against the real model, gated on a separate nightly or pre-release CI job rather than per-commit, because they are inherently slower and flakier and should never block a PR.

Two techniques worth calling out because they are commonly misused: setting temperature=0 reduces variance but does not guarantee bit-for-bit identical outputs across runs (floating-point non-associativity from GPU batching and MoE routing still produces different tokens in different batch contexts), so never rely on it as a substitute for mocking or cassette replay; and snapshot testing of tool-call sequences — asserting on the structure and argument values of the expected tool-call trajectory, not on free-text reasoning — surfaces unintended trajectory changes in CI before they reach production. For probabilistic Layer 3 tests, use pass@k thresholds (e.g., pass@5 with at least 4 successes) rather than a single run, and quarantine flaky tests into the nightly suite rather than masking them with automatic retries. See testing AI agents in CI for the tooling reference (VCR.py, pytest-recording, promptfoo, DeepEval).

What should an agent's observability system capture?

An agent's observability system should capture a full trace tree — every LLM call, tool call, retrieval step, and sub-agent invocation as a nested span under one stable trace ID — because flat, timestamped log lines cannot answer "why did the agent do that?" for a non-deterministic, branching run that may fan out into dozens of calls. The OpenTelemetry GenAI semantic conventions (formed by the GenAI SIG in April 2024, still labeled Development status as of July 2026 but already adopted by leading observability vendors) give this a vendor-neutral vocabulary: a trace is one complete agent run identified by a stable trace_id propagated across all child operations including sub-agents, and a span is one discrete operation — an LLM call, a tool call, a sub-agent hop — nested to form the full tree.

At minimum, capture per span: token usage converted into cost, latency for the step and for the overall trace, tool-call inputs and outputs (with PII redacted before logging), and the original exception attached to whichever span failed. Tooling splits between framework-agnostic OTel platforms — Langfuse, Arize Phoenix, OpenLLMetry, Logfire — and tracing built directly into a framework such as LangChain's LangSmith or the OpenAI Agents SDK's built-in trace processor. Stored spans double as raw material for two different feedback loops: sampling a slice of production traces builds evaluation datasets for judge- or metric-based scoring (closing the loop with the evaluation discipline above), and watching the same stream in real time flags anomalous behavior — a spike in tool errors, a cost outlier, a latency regression — as it happens rather than after a user complains. See agent observability and tracing for the OTel GenAI attribute reference and tooling landscape.

How do you roll out a new agent version without breaking production?

Roll out a new agent version by gating the promotion behind your own task-distribution eval suite first, then releasing to a small canary slice of traffic with automatic rollback triggered by metric regression — not only by crashes. Shadow mode, where the new version runs and logs its outputs without serving them to real users, removes rollout risk entirely during initial validation, because you can compare the new version's behavior against the current one on live traffic before any user is exposed to a difference. A canary of roughly 5–10% of traffic then surfaces regressions that shadow mode's non-live conditions might miss, with automated rollback triggers watching for behavioral regression — a lower task success rate, a changed output distribution, a cost spike — rather than only outright errors.

Rollback itself needs to be a fast, tested operation, not an improvised one during an incident:

• Version code, prompt template, model pin, and tool contracts together under one composite tag, so a rollback restores a known-good combination unambiguously rather than reverting one component while leaving others mismatched.
• Keep the previous version runnable for at least one canary period so rollback is a one-command, minutes-not-rebuild operation.
• Test rollback in drills, not only during real incidents — a rollback procedure nobody has exercised outside a crisis is a rollback procedure that fails when it matters most.

What happens when an agent fails in production, and how should the team respond?

When an agent fails in production, the team's response should already be written down as a runbook per failure class before go-live, because deciding how to react while an incident is active costs both time and judgment that a pre-written plan preserves. Four failure classes cover most real incidents and each has a standard first response: runaway cost (hard per-session budget plus a kill switch that halts the agent rather than letting spend continue while someone investigates), prompt or behavioral regression (rollback to the previous known-good version plus traffic redirect, using the rollback procedure above), tool or dependency outage (a circuit breaker plus a degraded-mode fallback, covered next), and security incident (revoke credentials, disable the agent, and preserve traces for investigation independent of the production system that might otherwise be part of the compromise).

Two disciplines make every incident in every class resolvable rather than a mystery: preserve traces on incident — the full trace tree from observability, kept accessible for the investigation window even if the production system itself is degraded or rolled back — and run a blameless post-mortem after every production incident, treating the incident as a signal that a guardrail, an eval case, or a monitoring alert was missing rather than as an individual's mistake. The output of a good post-mortem is concrete: a new eval case added to the CI suite that would have caught the regression, a new guardrail rule, or a new alert threshold — closing the loop back into the evaluation and observability disciplines above so the same failure class cannot recur silently. The full ten-dimension ship gate, including rollout and rollback mechanics, is in shipping AI agents to production.

Why do circuit breakers and degraded mode matter for agent reliability?

Circuit breakers and degraded mode matter because a permanently degraded dependency — a downed tool API, an exhausted daily rate limit, a database outage — should make an agent fail fast and predictably rather than retry forever and stall every request that touches it. A circuit breaker tracks the failure rate of a dependency and, once it crosses a threshold, stops sending requests to it for a cooldown period, returning an immediate failure (or falling back to a degraded response) instead of letting every caller independently retry and time out against a service that is not coming back soon. This is the natural complement to the retry-with-backoff pattern described earlier: backoff handles a single transient blip, while a circuit breaker handles a dependency that is down long enough that retrying is actively harmful — it burns budget, delays the caller, and can itself contribute to keeping the downstream service overloaded.

Degraded mode is the fallback behavior a circuit breaker triggers: an agent that cannot reach its primary retrieval index falls back to a cached or smaller-scope answer with an explicit disclosure rather than fabricating one; an agent that cannot reach a preferred model routes to a secondary provider through a gateway rather than failing the whole request; an agent whose reranking stage is down serves unreranked first-stage results rather than blocking. Designing degraded mode deliberately — deciding in advance what "good enough" looks like when a dependency is unavailable — is what separates a system that gracefully loses some quality under partial outage from one that cascades a single dependency's failure into a total outage for every user.

A production reliability checklist

Before calling an agent production-ready, it should survive this list, organized by the twelve disciplines above:

• Tool schemas use strict/constrained decoding where the provider supports it; unknown tool names and malformed arguments are rejected before execution
• Structured-output calls check the stop reason, not just the HTTP status, because truncation and refusal both bypass the schema guarantee
• All model and tool calls implement exponential backoff with full jitter, honoring Retry-After as a hard floor
• Every side-effecting tool call carries an idempotency key derived from run ID and step index
• Every step has an explicit timeout sized to its own worst case, with a wall-clock budget for the full retry sequence
• Tasks spanning more than one process lifetime run on a durable execution engine with deterministic workflow code
• Input, output, and action guardrails are layered independently of the model, with fail-closed behavior on uncertain verdicts
• An offline eval suite scored against your own task distribution gates every promotion in CI, with reliability measured across multiple trials
• Deterministic Layer 1/2 tests run on every commit; probabilistic Layer 3 smoke tests run nightly, never blocking a PR
• Every run emits a full trace tree under one trace ID with per-span cost, latency, and redacted payloads
• New versions ship behind a canary with automatic rollback on behavioral regression, and rollback is drilled, not just documented
• A written runbook exists per failure class (cost, regression, outage, security) with traces preserved and a blameless post-mortem after every incident
• Circuit breakers protect against permanently degraded dependencies, with a deliberately designed degraded-mode fallback

None of these disciplines is exotic in isolation — retries, idempotency keys, canaries, and CI gates are established engineering practice well outside AI. What is specific to agents is that a single request can touch all twelve at once: a tool call that needs a valid schema, a retry budget, an idempotency key, a timeout, a durable checkpoint if it runs long, a guardrail before it fires, an eval case that would catch a regression in it, a trace that explains it after the fact, a canary that limits its blast radius, a runbook if it fails, and a circuit breaker if its dependency goes down. Teams that treat reliability as this full stack, rather than as "retry a few times and hope," are the ones whose agents survive contact with real traffic.

The full agent-reliability cluster

Twelve sub-articles make up the agent-reliability cluster, each going deeper on one discipline than a single pillar reasonably can, and the cluster is complete as of August 2026:

• How to make AI agent tool calling reliable turns this pillar's tool-calling section into per-model schema-enforcement mechanics with runnable examples, a full catalog of the ways a validated call still goes wrong, and where BFCL's AST-based scoring actually draws the line on a correct call.
• Structured outputs vs tool calling turns the summary schema-subset table above into a framework for picking one mechanism over the other, with runnable schema code for every provider and a warm-up-call fix for the compilation delay a brand-new grammar pays on its very first call.
• How to make AI agent retries idempotent goes past the call_with_backoff example above into which errors are even worth retrying and deriving a dedup key that survives a crash mid-call, not just a rate limit.
• When do AI agents need durable execution expands the four-row situation table above into all ten vendors and frameworks it maps to, and the constraint that keeps a replayed workflow from re-running a real-world action a second time.
• How to design guardrails for AI agent reliability expands the three-checkpoint list above into a layered pipeline of cheap deterministic checks under slower judgment calls, and the specific untrusted-input/sensitive-system/state-change combination that should pull a human into the loop before a tool call fires.
• How to evaluate AI agents in CI expands the benchmark and CI-pyramid discussion above into the public-benchmark landscape as a diverging proxy and the --record-mode=none cassette-replay mechanics that keep per-commit CI honest without hitting a live model.
• How to roll out a new AI agent version safely goes past the canary-and-composite-tag summary above into where prompts should actually live, four ways to number a version, and how much traffic and patience an honest A/B test actually needs.
• How to build an incident response runbook for AI agent failures expands the four-failure-class table above into a three-signal triage step and the habit of exporting a trace before either recovery action gets a chance to overwrite it.
• How to set timeouts for AI agent tool calls expands the pillar's single paragraph into three angles it has no room for: sizing each step type on its own worst case, reserving a shrinking slice of one overall budget as a chain runs, and passing whatever time remains down into a nested call.
• How to design a circuit breaker for AI agents expands the one-paragraph mention above into the breaker's three-state machine — letting traffic through, cutting it off, then testing recovery with a handful of probes — and a fallback response designed in advance for each dependency type rather than improvised during an outage.
• What should an AI agent's observability system capture expands the one-paragraph trace/span mention above into the gen_ai.* wire vocabulary OpenTelemetry assigns to each field, the concrete list of what belongs on a span besides its timing, and a redaction rule for prompt and completion text specifically.
• The AI agent production reliability checklist turns the thirteen-bullet checklist above into a go/no-go gate per discipline, naming the specific evidence each one needs and the sibling article that owns its mechanics.

#agents #reliability #production #tool-calling #observability #evaluation #durable-execution