← All guides · Support agents in production

How to Test an AI Support Agent With Historical Tickets: Build a Replay Set

Part 12 of Support agents in production · 1,601 words · ~7 min read · published 2026-10-25 · updated 2026-10-25 · Markdown variant

How to test an AI support agent with historical tickets: pick resolved tickets per request type including escalations and refusals, redact them, assert on behaviour not wording, grade deterministically first, bridge to live traffic with shadow mode, and gate releases.

Part of the AI Customer Support Agent in Production: Escalation, Policy, Tickets and Measurement guide.

To test an AI support agent with historical tickets, build a replay set from your own resolved tickets, include the cases that should end in escalation or refusal, assert on behaviour instead of wording, and make a clean run a condition of every release. The AI customer support agent in production pillar previews this topic in its launch section; the detail is here.

Reading key: where a sentence cites a reference, the claim comes from it, and the remaining design choices are the author's inference. ChangeGamer runs no support agent as of October 2026. The corpus gives no replay-set pass-rate bar, set size or trial count; its only pass-rate or trial-count example is a CI pass@k rule (pass@5 with at least 4 passes, in testing AI agents), which does not transfer here, so none is stated below.

Why replay historical tickets before every change?

Replay historical tickets because the shipping agents to production checklist (updated 2026-08-13) says to gate every promotion on an offline eval suite scored against your own task distribution, since public benchmarks measure proxy tasks. The evaluating AI agents reference (updated 2026-09-27) calls offline evaluation on a fixed dataset fast and reproducible, and says public scores typically diverge from production performance.

Support raises the stakes: the customer support agents reference (updated 2026-09-13) notes that the agent often reaches a paying customer before a human has seen the exchange. Run the full set at launch and on every change to the composite version, the single tag that pins code, prompt, model and tool contracts together. As inference, add the policy text to that tag.

Which resolved tickets belong in the replay set?

Select tickets per request type, and include in each type the cases whose right outcome is an escalation or a refusal rather than a helpful answer. The classes below use the escalation categories from the customer-support-agents reference; the grouping is inference.

Two selection rules, both inference. Weight by consequence, not volume, because a set sampled in proportion to traffic is mostly routine answers and the rare refund-boundary case goes missing. Treat a ticket a human resolved after an escalation as an escalation fixture: the expected result is the handoff, not the human's eventual answer. A request type with no fixtures is untested.

Turning tickets into safe fixtures

Redact each ticket before saving it as a fixture, then rebuild whatever the agent would have looked up as mock tool responses, so a replay never touches a real customer record. Detection placement and seeded-value verification are in redacting PII from AI agent traces; the same logic applies to tickets.

Ticket-specific points, all inference: reuse one placeholder per name or order ID across messages and tool stubs so the case still joins up; review free text and attachments by hand; stub write tools to record the call without executing it. The evaluating-ai-agents reference says an eval must sandbox or mock tool calls consistently. Whether tickets may serve as test data depends on consent, which the corpus does not cover, and on lawful basis and purpose limitation, which Data Privacy and PII for Agents covers only in general terms.

What should each replay case assert?

Each case should assert three behaviours: the outcome is policy-correct, the escalation decision and its trigger category are right, and no unauthorized action was taken. Exact wording is not asserted.

An illustrative fixture shape:

{
  "id": "refund-over-limit-017",
  "messages": ["<redacted>"],
  "tool_stubs": { "lookup_order": { "status": "delivered" } },
  "expect": { "outcome": "escalated", "trigger": "policy_boundary" },
  "forbidden_tools": ["issue_refund"],
  "policy_version": "2026-10"
}

Wording assertions fail on harmless rephrasing and teach the team to ignore red runs (inference). Label the expected outcome from current written policy, signed off by the policy owner; a copied human reply imports whatever policy applied that day.

Grading: deterministic checks first, a judge second

Run the deterministic assertions first and give a model-as-judge only the prose they cannot check. The evaluating-ai-agents reference calls exact-match the most reliable method for tool calls, free of verbosity and position bias, and reserves LLM-as-judge for open-ended quality. As inference, a case that fails a hard assertion fails whatever the judge says. Judge design is in LLM-as-judge screening in production; CI wiring is in evaluating AI agents in CI.

Run each case several times: the same reference says one prompt can yield different tool sequences and recommends pass^k, where all k trials must succeed. As inference, an escalation or refusal case passes only if every trial passes; the corpus gives no k for a replay set (its CI example uses k=5). This jq check assumes your runner writes one JSON line per trial, with id, trial, tool_calls, forbidden, escalated, trigger and expected_escalation:

jq -r '. as $r
  | select(any($r.tool_calls[]; . as $c | $r.forbidden | index($c))
      or $r.escalated != ($r.expected_escalation != null)
      or ($r.expected_escalation != null and $r.trigger != $r.expected_escalation))
  | "FAIL " + .id + " trial " + (.trial | tostring)' results.jsonl | tee failures.txt
test ! -s failures.txt   # non-zero exit when any trial failed

Shadow mode as the bridge to live traffic

Shadow mode is the stage after replay: the new version receives live contacts while its output is only logged, so it meets traffic a historical set cannot contain. The shipping checklist says logging only the new version removes rollout risk during validation, and recommends a canary at 5 to 10 percent of traffic. Ordering the three stages below is inference.

Contacts where the shadow draft and the human outcome disagree become replay candidates after redaction. Shadow-run mechanics are in support agent rules-based bot migration; here shadow mode is only a release stage. Post-launch measurement is in support agent quality measurement.

Keeping the set fresh as policies change

Tie each fixture's expected outcome to a written policy version, and relabel fixtures when the policy changes instead of editing the agent to match old answers. Every rule below is inference.

• Record the policy version and date on each fixture.
• On a policy change, relabel that type's fixtures from the new text and confirm the current agent fails them before you edit the prompt; the red-then-green order is in the incident-to-eval-fixture loop.
• Add corrections from the human review queue and incident fixtures as new cases tagged by source; retire fixtures for withdrawn procedures, with a reason.
• Keep a held-out slice you never tune against; the evaluating reference prefers held-out sets for final assessment, to guard against contamination.
• Re-sample each type from tickets resolved since the last refresh, and date the refresh.

How do you gate a release on the replay set?

Make a green replay run a required check on every change to the composite version, with no tolerance for failed hard assertions and a rule for the graded rest written before you see results. The shipping checklist lists task success, tool-call accuracy, cost per task and repeat-run reliability as gate metrics, and says nothing ships until each item has an owner and a status.

As inference, the gate has three parts:

1. Block on any unauthorized tool call, missed required escalation or wrong trigger category.
2. Compare judge-graded prose with the previous version on the same set, and set a bar only after measuring your own baseline; the corpus gives no replay-set pass rate.
3. Report cost and latency per case beside success, as the evaluating reference advises.

A replay gate checks known cases before release; online measurement and human review cover the rest.

What does ChangeGamer verify about its own gate?

ChangeGamer gates on shipped output rather than intent. As of 2026-10-25 the header of changegamer/scripts/verify-invariants.mjs reads "Checks dist/ artifacts only (no src imports), so it validates what ships, not what was intended.", and .github/workflows/ci.yml runs it after the build on every pull request. It fails the run if a built article is under 900 words. Verify it with npm run build && node scripts/verify-invariants.mjs in changegamer/.

A replay set likewise asserts what the agent did, not what the prompt meant. ChangeGamer runs no support agent, holds no tickets and has no replay set, so this is an analogy, not evidence about support quality. The full launch order is in the pillar.

#customer-support #testing #replay #evaluation #fixtures #shadow-mode #agents #production