← All guides · Agent reliability in production

How to Roll Out a New AI Agent Version Safely

Part 7 of Agent reliability in production · 1,756 words · ~8 min read · published 2026-08-29 · updated 2026-08-29 · Markdown variant

An operator playbook for shipping a new agent version without breaking production: in-repo vs. registry prompt storage, a version-numbering comparison, the six-step promotion flow, A/B-test mechanics, the composite-version trace fields, and a rollback drill.

Part of the Agent Guardrails and the AI Agent Reliability Playbook guide.

Rollout gets one short section inside the agent reliability in production pillar — shadow mode, a canary slice, a composite version tag, rollback drills — squeezed in next to incident response and circuit breakers as three of a dozen disciplines the pillar surveys briefly. This article stays inside that single discipline and works through what a brief survey has no room for: where a production agent should actually store its prompts, how to number a version so a rollback points at something unambiguous, the concrete six-step pipeline a candidate travels through before it serves a real user, how to run an A/B test without mistaking model noise for a real win, and the exact trace fields a composite version tag needs to carry.

Where should a production agent store its prompts?

A production agent should store its prompts in the code repository when engineering owns every prompt change and already deploys often, or in an external registry when prompt iteration needs to outpace the code deploy cycle. In-repo storage is the faster option to start with: changes flow through the same pull-request review as any other code, and the full history is just Git history. Its cost shows up the first time a non-engineer needs to fix a wording problem at 6pm — that fix waits on a full code deploy, because the prompt is compiled into the same artifact as everything else.

An external registry inverts that tradeoff. Prompts are fetched at inference time by a name and a version label — something like my-agent-system:production — so a product manager, a safety reviewer, or a domain expert can push an update by reassigning that label, with no code deploy and no engineer in the loop. Each version is stored immutably once committed, so nothing is silently overwritten. The cost is a new runtime dependency the agent now depends on to boot, and the risk that prompt and code drift out of sync if nobody enforces which prompt version a given code release actually expects. As of August 2026, representative tooling spans integrated LLMOps suites that bundle prompts with evals and tracing (LangSmith Prompt Hub, Langfuse, Agenta, MLflow Prompt Registry), a prompt-management-first product (PromptLayer), and plain in-repo storage backed by an automated eval harness in CI — see prompt management and versioning for the full selection criteria.

How should you version a prompt, model, or tool contract?

An agent version should be numbered with one of four schemes, chosen for what a rollback needs to point at unambiguously, and every committed version should be treated as immutable rather than edited in place.

The environment-label scheme is the one that actually changes how a promotion works in practice: code always fetches prompt:production, so promoting a new version becomes reassigning that label rather than shipping a new build. A rollback under this scheme is the same operation run in reverse — point the label back at the prior immutable version — which is why it can be a one-command action instead of a redeploy.

What is the six-step flow for promoting a new agent version?

A new agent version should clear six sequential gates before it reaches full production traffic, not skip straight from authoring to a full release:

1. Author and commit — create a new immutable version and review it through the registry or a pull request.
2. Offline eval gate — run the candidate against an eval suite before it sees any live traffic, gating on task success rate, output-format compliance, and safety checks (see evaluating AI agents).
3. Safety and injection review — check whether the new version opens any system-prompt structure that indirect prompt injection could exploit, and confirm existing guardrails still hold against it.
4. Shadow or canary — route a small slice of traffic to the candidate, either logging its outputs invisibly (shadow) or serving a limited set of real users with rollback triggers already armed (canary).
5. Promote — flip the production label to the new version; with a registry, this needs no redeploy at all.
6. Retain the previous version — leave the prior release deployed and callable through an entire canary cycle, so undoing the switch is a matter of pointing traffic back rather than resurrecting a build.

Skipping step 2 or step 3 to save time is the single most common way teams end up doing an emergency rollback later — a candidate that never ran against a real eval suite or a safety review carries whatever regression those gates would have caught straight into a live canary, where it is discovered by users instead of by a check that runs in seconds.

How do you A/B test a challenger agent version?

An A/B test for a challenger agent version routes a modest slice of traffic — 5% to 20% — to the new candidate while holding each individual user's variant assignment steady for the entire test window, rather than reassigning them call by call. Sticky assignment matters because a user bouncing between the incumbent and the challenger from one request to the next makes any behavioral difference between them impossible to attribute cleanly to the version rather than to normal turn-to-turn variance.

Measure the challenger against the same metrics an eval suite already tracks — accuracy on the task, correct tool-call rate, latency, and per-task spend — plus any user-satisfaction signal available. The discipline that most teams skip is sample-size patience: language-model output carries enough run-to-run variance that a handful of early results in the challenger's favor is not evidence of a real improvement, only a plausible one. Wait until the sample is large enough to separate a genuine effect from noise before calling a winner, and tie every variant's identity into the same tracing pipeline that records everything else about the run, so each span can be attributed back to the exact prompt version that produced it.

Why does a rollback need a composite tag, not just a prompt version?

A rollback needs a composite tag because a production agent has at least three independently changing parts — the prompt template, the model pin, and the tool contracts — and a behavior change can originate from any one of them or from how they interact together. Reverting only the prompt while the model has already moved to a newer pin can fail to reproduce the exact behavior that shipped before, because the same prompt run against a different model is not the same system.

The fix is to name and log the full combination, not just the piece that appears to have caused a problem:

• Emit prompt_version, model_id, and tool_schema_hash on every trace span, so any regression can be traced back to the exact combination that produced it.
• Tag every release with one composite label that pins all three together, rather than versioning them independently.
• Roll back the composite as a unit — never the prompt alone against an already-newer model — since a partial rollback can leave two of the three components mismatched in a way nobody tested.

This is a different failure mode from a bad canary metric: a composite mismatch can pass every individual check on its own components while still producing behavior nobody has actually validated as a combination, which is exactly why the trace fields above need to exist independently of whatever eval or canary dashboard a team is watching.

What should a pre-deploy checklist confirm?

A pre-deploy checklist should confirm that every gate in the promotion flow above has actually been cleared, not assumed:

• The candidate version is committed as immutable, in source control or a registry.
• The offline eval suite has run and its pass thresholds are met.
• A safety review has confirmed no new injection surface and that guardrails still apply.
• A canary or shadow plan is defined, naming the traffic percentage, the rollback-trigger metric, and the observation window.
• The previous version is confirmed runnable, not just assumed to still work.
• A composite version tag covering prompt, model, and tool schema is recorded.
• Tracing is confirmed to emit the prompt version on every span before the candidate takes any live traffic.

Letting a candidate run invisibly first — its outputs logged against real inputs but never actually delivered to a user — is what catches most of that checklist's gaps for free, before a canary ever puts a live person in front of the new version. Once a canary starts serving real traffic, watch its dashboards for a softening completion rate, a tool-call pattern that looks different from the incumbent's, or spend per task drifting upward — none of which necessarily throws an error, so a monitor built only to catch crashes will miss all three.

How fast should a rollback actually be?

A rollback should take a team minutes, executed as a rehearsed action, not improvised for the first time while an incident is already underway. Leaving the outgoing release deployed and callable for a full canary cycle is what buys that speed — undoing a bad promotion becomes swinging a label or a traffic weight back, never rebuilding from an old commit. Speed alone does not make a rollback trustworthy, though; only running it counts. Schedule the rollback as a drill against a healthy, live-serving version on a normal week, note exactly who holds the access and how long the swing actually takes end to end, and fix whatever friction that drill surfaces — a stale credential, a permission nobody renewed, a step that quietly depends on one specific person — long before a real regression forces the team to discover it live, under pressure, with users already affected.

Rollback and rollout sit at opposite ends of the same discipline: shipping a new agent version and undoing that decision are the same muscle exercised in two directions. See agent reliability in production for the eleven remaining disciplines that surround this one — guardrails, evaluation, observability, and the rest of the twelve-part stack — and prompt management and versioning for the full promotion-flow, A/B-testing, and composite-version reference this article draws on.

#agents #reliability #rollout #versioning #prompt-management #production