{
  "slug": "shipping-agents-to-production",
  "title": "Shipping AI Agents to Production: A Production-Readiness Checklist",
  "description": "End-to-end checklist for productionizing an AI agent — evaluation gates, observability, guardrails, cost controls, resilience, durability, HITL approvals, secrets, rollback, and incident response.",
  "category": "Guide",
  "tags": [
    "agents",
    "production",
    "checklist",
    "deployment",
    "reliability",
    "observability",
    "security",
    "evaluation"
  ],
  "updated": "2026-07-22",
  "canonical": "https://changegamer.ai/resources/shipping-agents-to-production",
  "markdown": "https://changegamer.ai/resources/shipping-agents-to-production.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "1. Evaluation gates before ship",
      "anchor": "1-evaluation-gates-before-ship"
    },
    {
      "depth": 2,
      "text": "2. Observability and tracing",
      "anchor": "2-observability-and-tracing"
    },
    {
      "depth": 2,
      "text": "3. Guardrails and prompt-injection defense",
      "anchor": "3-guardrails-and-prompt-injection-defense"
    },
    {
      "depth": 2,
      "text": "4. Cost controls",
      "anchor": "4-cost-controls"
    },
    {
      "depth": 2,
      "text": "5. Rate-limit and retry resilience",
      "anchor": "5-rate-limit-and-retry-resilience"
    },
    {
      "depth": 2,
      "text": "6. Statefulness and durability for long-running tasks",
      "anchor": "6-statefulness-and-durability-for-long-running-tasks"
    },
    {
      "depth": 2,
      "text": "7. Human-in-the-loop approvals for high-risk actions",
      "anchor": "7-human-in-the-loop-approvals-for-high-risk-actions"
    },
    {
      "depth": 2,
      "text": "8. Secrets and identity (least-privilege credentials)",
      "anchor": "8-secrets-and-identity-least-privilege-credentials"
    },
    {
      "depth": 2,
      "text": "9. Rollback and versioning of prompts and tools",
      "anchor": "9-rollback-and-versioning-of-prompts-and-tools"
    },
    {
      "depth": 2,
      "text": "10. Incident response",
      "anchor": "10-incident-response"
    },
    {
      "depth": 2,
      "text": "Which ChangeGamer resource covers which dimension",
      "anchor": "which-changegamer-resource-covers-which-dimension"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "prompt-management-and-versioning",
      "title": "Prompt Management and Versioning: The Ops of Prompts in Production",
      "description": "Treating prompts as deployable artifacts: versioning, external registries, A/B and canary testing, eval-gated promotion, rollback, and the composite-version problem.",
      "url": "https://changegamer.ai/resources/prompt-management-and-versioning"
    },
    {
      "slug": "agentic-security-checklist",
      "title": "Agentic Security Checklist",
      "description": "Cross-vendor, threat-surface-organized security checklist for building and operating AI agents — synthesizing OWASP, NIST, Anthropic, OpenAI, Google SAIF, and MITRE ATLAS.",
      "url": "https://changegamer.ai/resources/agentic-security-checklist"
    },
    {
      "slug": "agent-identity-authentication",
      "title": "Agent Identity and Authentication",
      "description": "How autonomous agents prove who they are and get authorized to act: workload identity vs. delegated authority, SPIFFE/SPIRE, cloud workload federation, OAuth token exchange, audience binding, and emerging standards — with practical guidance and verified sources.",
      "url": "https://changegamer.ai/resources/agent-identity-authentication"
    },
    {
      "slug": "agent-observability",
      "title": "Agent Observability and Tracing",
      "description": "Why agents need observability beyond app logs, how OpenTelemetry GenAI semantic conventions model agent runs as traces, key signals to capture, and a verified tooling landscape.",
      "url": "https://changegamer.ai/resources/agent-observability"
    }
  ],
  "body": "An agent is production-ready when it can be deployed, monitored, corrected, and rolled back independently of the model it uses — and when every failure mode has a defined response before it occurs, not after.\n\nThis checklist synthesizes ten production-readiness dimensions. Each links to the ChangeGamer resource that covers it in depth. Use it as a ship gate: do not deploy until every item has an owner and a status.\n\n## Key facts\n\n- Production readiness for an agent means its deployment, monitoring, correction, and rollback do not depend on which model sits underneath it, and every plausible failure has a pre-written response before it happens.\n- Gate every promotion behind an eval suite scored against your own tasks — completion rate, correct tool calls, per-task spend, and run-to-run reliability — then roll out behind a canary with automatic rollback on regressions.\n- Every agent run should emit a structured trace under one trace_id, including per-span token usage and cost.\n- Guardrails for input, output, and agent actions should each hold regardless of which model is running, and a single tool call should touch at most two of untrusted input, sensitive systems, or a state-changing effect — never all three.\n- Cap token spend per session with a hard ceiling, and measure spend per finished task instead of per call, since fan-out across multiple agents multiplies costs quickly.\n- Irreversible or high-blast-radius tool calls need a durable human-approval gate with a logged approver identity, not just a timeout.\n\n## 1. Evaluation gates before ship\n\n**Run task-distribution evals before every promotion.** Public benchmarks measure proxy tasks — build an offline eval suite against your own task distribution and gate on task success rate, tool-call accuracy, cost per task, and reliability across repeated runs. **Run a canary** (5–10% of traffic) and trigger automated rollback on metric degradation, not just crashes. Shadow mode (new version logged only) removes rollout risk during validation.\n\nSee: /resources/evaluating-ai-agents\n\n## 2. Observability and tracing\n\n**Emit a structured trace for every agent run** — every LLM call, tool call, retrieval step, and sub-agent invocation as a span under one `trace_id`. **Instrument to the OpenTelemetry GenAI semantic conventions** (vendor-neutral attribute names like `gen_ai.operation.name = invoke_agent` and `gen_ai.agent.name`); as of 2026 they are in active development behind a stability opt-in flag and are already supported by major observability vendors. **Capture per-span token usage and derived cost** — you cannot budget what you cannot measure.\n\nSee: /resources/agent-observability\n\n## 3. Guardrails and prompt-injection defense\n\n**Stack input, output, and action guardrails independently of the model.** **Treat all external content as untrusted data, never as instructions** — indirect prompt injection (attacker text embedded in retrieved documents or tool responses) is OWASP LLM01:2025 (Prompt Injection), the top-ranked LLM application risk. **Scope each tool call to at most two of three properties** — processing untrusted input, accessing sensitive systems, and changing external state — never all three. A newer, agent-specific companion taxonomy — the **OWASP Top 10 for Agentic Applications (2026)**, published December 9, 2025 under identifiers ASI01–ASI10 (Agent Goal Hijack, Tool Misuse & Exploitation, Identity & Privilege Abuse, Agentic Supply Chain Vulnerabilities, Unexpected Code Execution, Memory & Context Poisoning, Insecure Inter-Agent Communication, Cascading Failures, Human-Agent Trust Exploitation, Rogue Agents) — sits alongside the general LLM Top 10 above and maps more directly onto autonomous multi-step agent risk; the same publication, date, and ASI-numbered taxonomy are already cited in /resources/agentic-security-checklist and /resources/essential-agent-publications. This cycle could not re-fetch the canonical genai.owasp.org page directly (HTTP 403 through this environment's network policy on every attempt, a known intermittent block on that domain) — corroborated instead via multiple agreeing secondary sources plus internal consistency with those two already-verified sibling entries.\n\nSee: /resources/agent-guardrails, /resources/agentic-security-checklist\n\n## 4. Cost controls\n\n**Set a hard per-session token budget** (OWASP LLM10:2025 Unbounded Consumption treats runaway spend as a security risk). **Apply model cascades and prompt caching** (see /resources/prompt-caching-for-agents) — route simple steps to cheaper models and cache stable prefixes. **Track cost per completed task**, not per call — fan-out multiplies spend linearly and dominates multi-agent pipelines.\n\nSee: /resources/agent-cost-latency-optimization\n\n## 5. Rate-limit and retry resilience\n\n**Implement exponential backoff with jitter on all model and tool calls.** **Distinguish retriable from non-retriable errors** — 429 and 503 are retriable, 400 and 401 are not. **Set a max retry count and a circuit breaker** so a permanently degraded dependency fails fast instead of stalling the trace.\n\nSee: /resources/handling-rate-limits-and-retries\n\n## 6. Statefulness and durability for long-running tasks\n\n**Use a durable execution engine for any task that spans more than one process lifetime** so it survives crashes and restarts by replaying a checkpoint log. **Keep workflow code deterministic** — wall-clock time, randomness, and LLM calls belong in separately-recorded activities, or replay corrupts agent state on recovery.\n\nSee: /resources/durable-execution-for-agents\n\n## 7. Human-in-the-loop approvals for high-risk actions\n\n**Classify every tool call by reversibility and blast radius.** Read-only, low-blast-radius calls proceed autonomously; irreversible or high-blast-radius calls (move money, modify production data, send external comms, delete data) require a human approval gate. **Implement the gate as a durable pause, not a timeout**, and **log the approval decision with approver identity and timestamp**. The EU AI Act (Article 14) and NIST AI RMF both require auditable human oversight for high-risk AI actions.\n\nSee: /resources/durable-execution-for-agents, /resources/agentic-security-checklist\n\n## 8. Secrets and identity (least-privilege credentials)\n\n**Never pass secrets through the context window** — use a credential layer so the agent receives data, not the credential that fetched it. **Issue short-lived, scoped credentials per task** rather than static long-lived keys (the operational form of OWASP LLM06:2025 Excessive Agency). **Apply least-privilege to OAuth scopes.**\n\nSee: /resources/agent-identity-authentication, /resources/agent-delegation-chains\n\n## 9. Rollback and versioning of prompts and tools\n\n**Version code, prompt template, model pin, and tool contracts together** with a composite tag so a rollback is unambiguous. **Keep the previous version runnable for at least one canary period** so rollback is a one-command, minutes-not-rebuild operation, triggered by behavioral regression (lower success rate, changed output distribution), not just crashes. **Test rollback in drills, not only during incidents.**\n\nSee: /resources/deploying-serving-llms\n\n## 10. Incident response\n\n**Write a runbook per failure class before go-live:** runaway cost (hard budget + kill switch), prompt regression (rollback + traffic redirect), tool outage (circuit breaker + degraded mode), security incident (revoke credentials + disable agent + preserve traces). **Preserve traces on incident** for the investigation window, accessible independent of the production system. **Run a blameless post-mortem** after every production incident.\n\nSee: /resources/agent-observability\n\n## Which ChangeGamer resource covers which dimension\n\n| Dimension | Resource |\n|---|---|\n| Evaluation gates and offline evals | /resources/evaluating-ai-agents |\n| Observability, tracing, OTel GenAI | /resources/agent-observability |\n| Guardrails and prompt-injection defense | /resources/agent-guardrails |\n| Full security threat-surface checklist | /resources/agentic-security-checklist |\n| Cost controls, caching, and model routing | /resources/agent-cost-latency-optimization |\n| Rate limits, backoff, and batch APIs | /resources/handling-rate-limits-and-retries |\n| Durable execution and HITL pause/resume | /resources/durable-execution-for-agents |\n| Agent identity and short-lived credentials | /resources/agent-identity-authentication |\n| Multi-hop credential delegation | /resources/agent-delegation-chains |\n| Model serving and inference infrastructure | /resources/deploying-serving-llms |\n\nThis checklist is not just theoretical for ChangeGamer: the site runs its own publishing pipeline behind an explicit gate chain and a branch-and-PR-only shipping discipline — see the premium resource [How ChangeGamer Runs Itself](/resources/autonomous-operations).\n\n## Verified sources\n\n- OWASP GenAI LLM Top 10 (2025): https://genai.owasp.org/llm-top-10/\n- OWASP LLM06:2025 Excessive Agency: https://genai.owasp.org/llmrisk/llm062025-excessive-agency/\n- OWASP Top 10 for Agentic Applications (2026), ASI01–ASI10 — canonical page 403'd to this session's fetcher, WebSearch-corroborated via multiple agreeing secondary sources: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ (see also https://goteleport.com/blog/owasp-top-10-agentic-applications/)\n- OpenTelemetry GenAI agent-span semantic conventions (moved to the semantic-conventions-genai repo): https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md\n- EU AI Act (Regulation 2024/1689), Article 14 human oversight: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689\n- NIST AI Risk Management Framework (AI RMF 1.0): https://doi.org/10.6028/NIST.AI.100-1",
  "sources": [
    "https://genai.owasp.org/llm-top-10/",
    "https://genai.owasp.org/llmrisk/llm062025-excessive-agency/",
    "https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/",
    "https://goteleport.com/blog/owasp-top-10-agentic-applications/",
    "https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md",
    "https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689",
    "https://doi.org/10.6028/NIST.AI.100-1"
  ]
}