{
  "slug": "testing-ai-agents",
  "title": "Testing AI Agents in CI",
  "description": "How to write deterministic, fast, CI-friendly tests for non-deterministic agents: the three-layer test pyramid, LLM mocking, cassette/VCR-style replay, snapshot testing of tool-call trajectories, pass@k thresholds, and verified tooling.",
  "category": "Guide",
  "tags": [
    "testing",
    "ci",
    "agents",
    "mocking",
    "determinism",
    "tool-calling",
    "pytest"
  ],
  "updated": "2026-07-19",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/testing-ai-agents",
  "markdown": "https://changegamer.ai/resources/testing-ai-agents.md",
  "outline": [
    {
      "depth": 2,
      "text": "The three-layer test pyramid for agents",
      "anchor": "the-three-layer-test-pyramid-for-agents"
    },
    {
      "depth": 3,
      "text": "Layer 1 — deterministic unit tests (run on every commit)",
      "anchor": "layer-1-deterministic-unit-tests-run-on-every-commit"
    },
    {
      "depth": 3,
      "text": "Layer 2 — recorded/replayed LLM interactions (run on every commit)",
      "anchor": "layer-2-recorded-replayed-llm-interactions-run-on-every-commit"
    },
    {
      "depth": 3,
      "text": "Layer 3 — live smoke / eval tests (nightly or pre-release, NOT per-commit)",
      "anchor": "layer-3-live-smoke-eval-tests-nightly-or-pre-release-not-per-commit"
    },
    {
      "depth": 2,
      "text": "Key techniques",
      "anchor": "key-techniques"
    },
    {
      "depth": 2,
      "text": "Handling flakiness",
      "anchor": "handling-flakiness"
    },
    {
      "depth": 2,
      "text": "Verified tooling",
      "anchor": "verified-tooling"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "choosing-an-llm-for-agents",
      "title": "How to Choose an LLM for Agentic Tasks",
      "description": "A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.",
      "url": "https://changegamer.ai/resources/choosing-an-llm-for-agents"
    },
    {
      "slug": "reliable-tool-calling",
      "title": "Reliable Tool Calling and Structured Outputs",
      "description": "How providers guarantee schema-valid tool calls and structured output — mechanisms, failure modes, and mitigations — for production agent builders.",
      "url": "https://changegamer.ai/resources/reliable-tool-calling"
    },
    {
      "slug": "streaming-for-agents",
      "title": "Streaming Responses for Agents",
      "description": "Transport formats, provider event schemas, and practical concerns for consuming streamed LLM responses in production agents: SSE mechanics, OpenAI (Chat Completions and Responses API) and Anthropic event formats, partial-JSON tool-call parsing, backpressure, cancellation, and gateway proxying.",
      "url": "https://changegamer.ai/resources/streaming-for-agents"
    },
    {
      "slug": "synthetic-data-generation",
      "title": "Synthetic Data Generation for Agent Training",
      "description": "How to build agentic training corpora without human annotation at scale: the three generation patterns (distillation, self-play, environment rollout), open pipelines (distilabel, AgentInstruct, APIGen-MT, TOUCAN), quality filtering, and the model-collapse risk.",
      "url": "https://changegamer.ai/resources/synthetic-data-generation"
    }
  ],
  "furtherReading": [
    {
      "slug": "testing-mcp-servers-in-ci",
      "title": "How to Test an MCP Server in CI",
      "description": "The implementation mechanics below the three-layer test pyramid: what a mocked MCP transport actually replaces, what a Streamable HTTP cassette contains, a concrete CI job/trigger shape, and how to catch spec-version drift before it reaches production.",
      "url": "https://changegamer.ai/articles/testing-mcp-servers-in-ci"
    },
    {
      "slug": "mcp-server-failure-modes",
      "title": "Common MCP Server Failure Modes and How to Fix Them",
      "description": "A runtime playbook for the two MCP server failure modes with no dedicated deep-dive elsewhere: unrecoverable state after a mid-call crash, and malformed or hallucinated tool calls that reach the handler despite upstream validation.",
      "url": "https://changegamer.ai/articles/mcp-server-failure-modes"
    }
  ],
  "body": "The core tension: agents are non-deterministic, but CI pipelines need tests that are deterministic, fast, cheap, and reliable. You can get both — by being deliberate about which layer of the stack you test at which level.\n\n## The three-layer test pyramid for agents\n\n### Layer 1 — deterministic unit tests (run on every commit)\n\nTest the code *around* the model: tool functions, parsers, prompt-template renderers, schema validators, retry logic, and output-format coercers. Mock or stub the LLM client entirely. These tests are ordinary unit tests — no API calls, no network, fast and free. They catch the majority of regressions because most bugs live in the glue, not the model.\n\n### Layer 2 — recorded/replayed LLM interactions (run on every commit)\n\nUse cassette/VCR-style fixtures: the first time a test runs, it hits the real API and serialises the full HTTP exchange to a YAML file. Every subsequent run replays that cassette instead of making a live call — fast, free, and network-independent. Commit cassettes to version control. Re-record only when prompt templates or schema change. In CI, pass `--record-mode=none` (pytest-recording's actual flag — not `--vcr-record`, which belongs to the separate, unrelated `pytest-vcr` package) so a missing cassette is a test failure, not a live API call.\n\n**Important**: scrub credentials and sensitive headers from cassettes before committing. VCR.py and pytest-recording both support `filter_headers` and `filter_post_data_parameters` for this.\n\n### Layer 3 — live smoke / eval tests (nightly or pre-release, NOT per-commit)\n\nA small, hand-curated set of end-to-end tasks run against the real model. Gate these on a separate CI job (nightly, or triggered manually for releases). They are expensive and inherently flaky — keep the set small and treat failures as signals, not hard blockers per PR.\n\n## Key techniques\n\n**Mocking the LLM client** — use `unittest.mock.patch` or a dependency-injection seam to replace the model client with a fixture that returns a pre-canned structured response. This is the cheapest form of Layer 1 testing.\n\n**temperature=0 and seeds** — setting temperature to zero and a fixed seed reduces variance but does not guarantee bit-for-bit identical outputs across runs. Floating-point non-associativity from GPU batching and MoE routing means the same prompt can yield different tokens in different batch contexts. Never rely on temperature=0 as a substitute for proper mocking or cassette replay.\n\n**Snapshot testing of tool-call sequences** — store the expected sequence of tool calls (names + arguments) as a snapshot. Assert on structure and argument values, not on the free-text reasoning. A tool-call diff in CI surfaces unintended trajectory changes before they reach production. Cross-link: /resources/reliable-tool-calling.\n\n**Structured-output assertions** — if your agent emits JSON or a Pydantic schema, assert against the schema and the key field values, not against the exact prose. This tolerates benign rephrasing while catching real regressions.\n\n**LLM-as-judge in tests** — a second model grades the output against a rubric. Useful for Layer 3 smoke tests but carries its own flakiness: the judge model can disagree with itself across runs. Treat judge scores as soft signals; set wide pass/fail thresholds and aggregate over multiple runs. See /resources/evaluating-ai-agents for benchmark-grade eval methodology.\n\n## Handling flakiness\n\n- **Separate suites** — keep deterministic (Layers 1–2) and probabilistic (Layer 3) suites in distinct test files and CI jobs. Never let a non-deterministic test gate a PR.\n- **pass@k thresholds** — for probabilistic tests, run k trials and assert that at least m succeed (e.g., pass@5 with m=4). This is more honest than a single run and absorbs natural variance without hiding real regressions.\n- **Retries vs quarantine** — automatic retries mask real failures; prefer quarantining a flaky test into the nightly suite until its failure mode is understood.\n- **Cost controls** — set token-budget limits per test job. Tag each Layer 3 job with expected cost and alert when actual cost drifts more than 20%. Cross-links: /resources/agent-cost-latency-optimization, /resources/agent-observability (traces from production runs can seed new cassettes and test cases).\n\n## Verified tooling\n\n**VCR.py** (`vcrpy`) — Python HTTP record/replay. Intercepts HTTP at the library level; serialises to YAML cassettes. Works with any HTTP-based LLM SDK.\n\n**pytest-recording** — a pytest plugin wrapping VCR.py. Adds `--record-mode` CLI option and `@pytest.mark.vcr` decorator. Maintained by kiwicom on GitHub.\n\n**promptfoo** — YAML-driven test runner for prompts and agents with native CI/CD integration (GitHub Action available). Supports structured assertions (`is-json`, `contains-json`, `llm-rubric`), cost/latency thresholds, and red-teaming. MIT-licensed; acquired by OpenAI in March 2026 but remains open-source.\n\n**DeepEval** — pytest-based LLM evaluation framework. `assert_test()` and `deepeval test run` plug directly into CI pipelines; supports parallel execution via `-n` flag. Maintained by Confident AI.\n\n## Verified sources\n\n- VCR.py repo (kevin1024/vcrpy): https://github.com/kevin1024/vcrpy\n- VCR.py docs: https://vcrpy.readthedocs.io/en/latest/\n- pytest-recording repo (kiwicom): https://github.com/kiwicom/pytest-recording\n- pytest-recording on PyPI: https://pypi.org/project/pytest-recording/\n- promptfoo CI/CD integration docs: https://www.promptfoo.dev/docs/integrations/ci-cd/\n- promptfoo GitHub: https://github.com/promptfoo/promptfoo\n- DeepEval unit testing in CI/CD: https://deepeval.com/docs/evaluation-unit-testing-in-ci-cd\n- DeepEval GitHub (confident-ai): https://github.com/confident-ai/deepeval\n- temperature=0 non-determinism explained: https://www.zansara.dev/posts/2026-03-24-temp-0-llm/",
  "sources": [
    "https://github.com/kevin1024/vcrpy",
    "https://vcrpy.readthedocs.io/en/latest/",
    "https://github.com/kiwicom/pytest-recording",
    "https://pypi.org/project/pytest-recording/",
    "https://www.promptfoo.dev/docs/integrations/ci-cd/",
    "https://github.com/promptfoo/promptfoo",
    "https://deepeval.com/docs/evaluation-unit-testing-in-ci-cd",
    "https://github.com/confident-ai/deepeval",
    "https://www.zansara.dev/posts/2026-03-24-temp-0-llm/"
  ]
}