{ "slug": "testing-mcp-servers-in-ci", "title": "How to Test an MCP Server in CI", "description": "The implementation mechanics below the three-layer test pyramid: what a mocked MCP transport actually replaces, what a Streamable HTTP cassette contains, a concrete CI job/trigger shape, and how to catch spec-version drift before it reaches production.", "kind": "sub", "order": 4, "target_query": "how to test an MCP server", "secondary_queries": [ "mocking the MCP transport in unit tests", "recording MCP tool call cassettes", "MCP server CI pipeline nightly smoke tests", "catching MCP spec drift with tests" ], "tags": [ "mcp", "testing", "ci", "agents", "tool-calling" ], "published": "2026-08-09", "updated": "2026-08-09", "words": 1337, "premium": false, "license": "https://changegamer.ai/license.xml", "canonical": "https://changegamer.ai/articles/testing-mcp-servers-in-ci", "markdown": "https://changegamer.ai/articles/testing-mcp-servers-in-ci.md", "takeaways": [ "An MCP server's Layer 1 unit tests mock the transport, not the model — most servers don't call an LLM at all, so the seam you replace is the JSON-RPC channel your handlers sit behind, not an LLM client the way generic agent testing does.", "Cassette-style record/replay is confirmed to work for a Streamable HTTP MCP server, because every message on that transport is literally an HTTP POST to one endpoint; there is no confirmed equivalent for stdio's raw stdin/stdout exchange, so treat \"record once, replay in CI, scrub credentials\" as a pattern to adapt for stdio, not a library that already does it.", "Layer 3 live smoke tests belong on a separate CI job definition, not just a separate test file — a nightly or pre-release schedule with its own cost budget, distinct from the job that gates every push and pull request.", "An MCP SDK version bump is itself a legitimate reason to re-run the full pyramid, deliberately: a Layer 1 or Layer 2 test that asserts the negotiated `protocolVersion` explicitly is what turns a breaking version change into a caught test failure instead of a silent production break." ], "outline": [ { "depth": 2, "text": "What a mocked transport actually replaces", "anchor": "what-a-mocked-transport-actually-replaces", "url": "https://changegamer.ai/articles/testing-mcp-servers-in-ci#what-a-mocked-transport-actually-replaces" }, { "depth": 2, "text": "What's actually inside an MCP cassette", "anchor": "what-s-actually-inside-an-mcp-cassette", "url": "https://changegamer.ai/articles/testing-mcp-servers-in-ci#what-s-actually-inside-an-mcp-cassette" }, { "depth": 2, "text": "A concrete CI job shape", "anchor": "a-concrete-ci-job-shape", "url": "https://changegamer.ai/articles/testing-mcp-servers-in-ci#a-concrete-ci-job-shape" }, { "depth": 2, "text": "Catching spec drift with a test, not a production incident", "anchor": "catching-spec-drift-with-a-test-not-a-production-incident", "url": "https://changegamer.ai/articles/testing-mcp-servers-in-ci#catching-spec-drift-with-a-test-not-a-production-incident" } ], "faq": [ { "question": "Do I mock the LLM or the transport when unit testing an MCP server?", "answer": "The transport. Generic agent Layer 1 testing mocks the LLM client because the code under test calls a model. An MCP server, in most cases, does not call a model at all — it responds to calls from a client that has one. What you replace at Layer 1 is the JSON-RPC channel: either call your registered tool handler directly with a constructed arguments object, bypassing the wire format entirely, or connect your server to an in-process substitute for stdio/Streamable HTTP so the real request-handling code runs without a subprocess or a socket." }, { "question": "Can I use VCR.py or pytest-recording to test an MCP server?", "answer": "Yes, but confirm the transport first. Both tools intercept HTTP at the library level and serialize the exchange to a cassette file, replaying it on later runs. A Streamable HTTP MCP server's messages are ordinary HTTP POST requests to a single endpoint, so an HTTP-level interceptor can record and replay a full session — initialize, discovery, one or more tool calls — the same way it would any other HTTP-based API. A stdio server exchanges JSON-RPC over stdin/stdout with no HTTP layer for these tools to intercept, so nothing in the corpus confirms they work there; apply the same discipline manually (serialize the request/response pairs yourself, replay them in a test, scrub anything sensitive) rather than assuming a drop-in library exists for that transport." }, { "question": "How often should Layer 3 live smoke tests run against real clients?", "answer": "Nightly or pre-release, never on every commit. Live tests against real target clients are expensive and inherently flaky, so gating a pull request on them turns an unrelated intermittent failure into a blocked merge. Run them on their own CI job with a schedule trigger, keep the test set small and hand-curated, tag the job with an expected cost, and alert if actual spend drifts materially from that baseline — a small suite that starts calling more tools, or a client that got slower, should surface as a cost anomaly before it surfaces as a surprise bill." } ], "body": "Everything you need to know about *what* to test on an MCP server — the three-layer pyramid, MCP Inspector as a manual first check, snapshot-testing tool-call trajectories, and BFCL as the underlying cross-provider reliability ceiling — is already covered in [MCP server in production](/articles/mcp-server-in-production). This article does not redefine those layers. It answers the question that comes right after: once you've decided to build a unit layer, a cassette layer, and a live layer, what does each one actually look like in code and in a CI config for an MCP server specifically, as opposed to an LLM-calling agent generally?\n\n## What a mocked transport actually replaces\n\nLayer 1 for an MCP server means mocking or stubbing the transport, and the transport is the part worth being precise about, because it is not the same seam generic agent testing mocks. Testing an agent that calls a model means mocking the LLM client — you replace the thing that talks to the API and returns canned completions ([testing AI agents in CI](/resources/testing-ai-agents)). Testing an MCP server means something different: in most servers, nothing in the code under test calls a model at all. The server's job is to receive a JSON-RPC request over stdio or Streamable HTTP, run a handler, and return a result — the model lives on the client's side of the connection, not the server's.\n\nTwo things are worth testing separately at this layer, and neither requires a live JSON-RPC round trip:\n\n- **The input validator, on its own.** Every tool declares an `inputSchema` ([building an MCP server](/resources/building-mcp-servers)); feed the validator malformed argument objects — missing required fields, wrong types, extra properties if your schema forbids them — and assert it rejects each one with the specific violation you expect, before any handler runs. This is the layer that turns a `tools/call` with bad arguments into a clean, typed error instead of a handler crash.\n- **The handler, called directly.** Call your registered tool function as a plain function with a constructed arguments object, skipping the wire format entirely, and assert on the returned content array — its shape, its `type` field, the actual values inside it. This is the fastest and most direct test you can write against an MCP server, and it catches the majority of real regressions because most bugs live in handler logic, not in the protocol layer around it.\n\nIf you want to exercise the actual JSON-RPC handling code rather than just the handler function underneath it, connect your server object to an in-process substitute for the transport instead of standing up stdio or a network listener — the same request/response cycle a real client would trigger, without a subprocess or a socket. Either approach keeps Layer 1 free of network calls, which is the property that makes it safe to run on every commit.\n\n## What's actually inside an MCP cassette\n\nA cassette for an MCP exchange is a serialized sequence of JSON-RPC request/response pairs — `initialize`, `tools/list`, one or more `tools/call` exchanges, matched on method name and parameters — replayed instead of a live session on every run after the first recording. This is the same pattern general agent testing uses for LLM API calls ([testing AI agents in CI](/resources/testing-ai-agents)), applied to the wire your server actually speaks.\n\nWhere the pattern is directly confirmed to work: **Streamable HTTP**. Every message on that transport is an HTTP POST to a single endpoint, and record/replay tools like VCR.py and pytest-recording intercept HTTP at the library level — so a Streamable HTTP MCP session records and replays the same way any other HTTP-based API exchange would, with the same credential scrubbing (`filter_headers`, `filter_post_data_parameters`) applying before you commit the cassette to version control.\n\nWhere it is not confirmed: **stdio**. That transport exchanges newline-delimited JSON-RPC over stdin/stdout with no HTTP layer for these tools to intercept, and nothing in the corpus states that VCR.py-style libraries record a child-process exchange. If your server runs over stdio, apply the same discipline by hand — serialize the request/response pairs your test client sends and receives, replay them from a fixture file, scrub anything sensitive — rather than reaching for an HTTP-recording library and assuming it works on a transport it was not built for.\n\nRe-record a cassette on two triggers, not one: when a tool's schema or description changes (the pillar's baseline rule), and when you deliberately bump your MCP SDK version. A stale cassette that silently stops matching a newer SDK's request shape is a test that passes for the wrong reason — in CI, pass `--record-mode=none` (pytest-recording's actual flag) so a missing or mismatched cassette fails the test outright instead of falling back to a live call.\n\n## A concrete CI job shape\n\nMap the three layers onto CI triggers, not just test files, so a flaky live test can never block a merge. A workable split:\n\n```yaml\n# .github/workflows/mcp-server.yml\non:\n push:\n pull_request:\n schedule:\n - cron: '0 3 * * *' # nightly\n workflow_dispatch: # pre-release, manual trigger\n\njobs:\n unit: # Layers 1 + 2 — every push and PR\n if: github.event_name == 'push' || github.event_name == 'pull_request'\n steps:\n - run: pytest tests/unit tests/cassette --record-mode=none\n\n smoke: # Layer 3 — nightly or pre-release only\n if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'\n steps:\n - run: pytest tests/smoke --maxfail=1\n```\n\nThe `unit` job runs deterministic and cassette-backed tests on every commit — fast, free, no network — and is the job that gates a pull request. The `smoke` job runs only on a schedule or a manual pre-release trigger, exercises real target clients, and should never gate a merge on its own; treat its failures as signals to investigate, tag it with an expected token/cost budget, and alert if actual spend drifts materially, since a handful of hand-curated live calls run nightly should have a predictable cost. If a smoke test starts failing intermittently for reasons you haven't root-caused, quarantine it out of the gating path rather than retrying it — a retry that eventually passes hides the failure instead of explaining it.\n\n## Catching spec drift with a test, not a production incident\n\nThe pillar's versioning discipline — pin your SDK, check `protocolVersion` explicitly, treat any given spec snapshot as something to reverify rather than something settled — tells you what practice to follow ([MCP server in production](/articles/mcp-server-in-production)). What it doesn't spell out is how that practice becomes a test that runs in CI rather than a habit someone forgets under deadline pressure. Two concrete tests do most of the work:\n\n- **Assert the negotiated `protocolVersion` explicitly, in both Layer 1 and Layer 2.** During `initialize`, the server responds with the `protocolVersion` it negotiated, and that value governs the rest of the session ([building an MCP server](/resources/building-mcp-servers)). A Layer 1 test against your in-process transport substitute, and a Layer 2 test replaying a cassette, should each check the returned value matches what you expect — not just that `initialize` didn't throw. A mismatch that would otherwise negotiate down and \"succeed\" silently becomes a visible test failure instead.\n- **Make an SDK version bump its own CI event, not a side effect of an unrelated dependency update.** Re-run the full pyramid — not just the unit job — whenever you deliberately bump the MCP SDK version, the same discipline Layer 2 already applies to a cassette when a tool's schema or description changes ([testing AI agents in CI](/resources/testing-ai-agents)). A validator or handler that behaved one way against an older SDK build can behave differently against a newer one without any of your own code changing, and testing the bump on purpose is what catches that before a client does.\n\nNone of this replaces the client-facing reliability question. Whether the model calling your server gets the arguments right in the first place is a different, provider-level problem, and BFCL remains the reference benchmark for it across vendors ([reliable tool calling](/resources/reliable-tool-calling)) — your test suite controls what your server does with a call it receives, not whether the model on the other end constructs that call correctly.", "cluster": { "id": "mcp-in-practice", "title": "MCP in practice", "description": "How to build, ship and run an MCP server in production — transport, auth, tool design, versioning, testing, distribution, observability, cost and failure modes.", "status": "complete", "pillar": { "slug": "mcp-server-in-production", "title": "MCP Server in Production: How to Build, Ship and Run One", "description": "The operator playbook for taking an MCP server past the quickstart: transport choice, OAuth 2.1 auth, tool design, versioning against a moving spec, testing across clients, distribution, observability, cost and the failure modes that show up once real clients connect.", "kind": "pillar", "order": 0, "html": "https://changegamer.ai/articles/mcp-server-in-production", "markdown": "https://changegamer.ai/articles/mcp-server-in-production.md", "json": "https://changegamer.ai/api/articles/mcp-server-in-production.json" }, "articles": [ { "slug": "mcp-stdio-vs-streamable-http", "title": "stdio vs. Streamable HTTP for MCP Servers: A Decision Framework", "description": "Which MCP transport to build against and why: the single-client-vs-shared decision rule, how state works without a session handshake under the 2026-07-28 spec, the auth-model switching cost, and what actually breaks migrating off HTTP+SSE.", "kind": "sub", "order": 1, "html": "https://changegamer.ai/articles/mcp-stdio-vs-streamable-http", "markdown": "https://changegamer.ai/articles/mcp-stdio-vs-streamable-http.md", "json": "https://changegamer.ai/api/articles/mcp-stdio-vs-streamable-http.json" }, { "slug": "mcp-oauth-implementation", "title": "How to Implement OAuth 2.1 for an MCP Server", "description": "A wire-level implementation walkthrough for OAuth 2.1 on a remote MCP server: what the discovery documents actually contain, CIMD vs. Dynamic Client Registration in your server code, per-SEP detail from the 2026-07-28 hardening set, and token-validation mechanics.", "kind": "sub", "order": 2, "html": "https://changegamer.ai/articles/mcp-oauth-implementation", "markdown": "https://changegamer.ai/articles/mcp-oauth-implementation.md", "json": "https://changegamer.ai/api/articles/mcp-oauth-implementation.json" }, { "slug": "mcp-tool-description-injection", "title": "Defending MCP Clients Against Tool Description and Output Injection", "description": "Two distinct MCP injection surfaces — a tool description at connect-time and a tool's return value at call-time — and the client-side architectural patterns (Dual LLM, Action-Selector, Context-Minimization) that contain each one.", "kind": "sub", "order": 3, "html": "https://changegamer.ai/articles/mcp-tool-description-injection", "markdown": "https://changegamer.ai/articles/mcp-tool-description-injection.md", "json": "https://changegamer.ai/api/articles/mcp-tool-description-injection.json" }, { "slug": "testing-mcp-servers-in-ci", "title": "How to Test an MCP Server in CI", "description": "The implementation mechanics below the three-layer test pyramid: what a mocked MCP transport actually replaces, what a Streamable HTTP cassette contains, a concrete CI job/trigger shape, and how to catch spec-version drift before it reaches production.", "kind": "sub", "order": 4, "html": "https://changegamer.ai/articles/testing-mcp-servers-in-ci", "markdown": "https://changegamer.ai/articles/testing-mcp-servers-in-ci.md", "json": "https://changegamer.ai/api/articles/testing-mcp-servers-in-ci.json" }, { "slug": "mcp-server-versioning-and-spec-migration", "title": "MCP Server Versioning and Spec Migration: An Operator Playbook", "description": "A migration runbook for MCP server operators: feature-detecting via capabilities instead of hard protocolVersion branching, a dual-version fleet rollout with rollback triggers, a compatibility shim for legacy clients still sending initialize, and a deprecation calendar built off the 12-month SEP-2577 floor.", "kind": "sub", "order": 5, "html": "https://changegamer.ai/articles/mcp-server-versioning-and-spec-migration", "markdown": "https://changegamer.ai/articles/mcp-server-versioning-and-spec-migration.md", "json": "https://changegamer.ai/api/articles/mcp-server-versioning-and-spec-migration.json" }, { "slug": "mcp-server-observability-opentelemetry", "title": "MCP Server Observability with OpenTelemetry: Spans, Metrics, and Trace Correlation", "description": "Instrumenting an MCP server past the pillar's baseline: what to put on a tool-call span beyond gen_ai.tool.name, what replaces the deprecated Logging primitive in practice, per-tool-name latency and error-rate metrics, and how a trace ID actually survives the agent-to-upstream-API hop.", "kind": "sub", "order": 6, "html": "https://changegamer.ai/articles/mcp-server-observability-opentelemetry", "markdown": "https://changegamer.ai/articles/mcp-server-observability-opentelemetry.md", "json": "https://changegamer.ai/api/articles/mcp-server-observability-opentelemetry.json" }, { "slug": "mcp-server-registry-publishing-playbook", "title": "How to Publish an MCP Server to the Official Registry", "description": "A step-by-step walkthrough of the mcp-publisher CLI and the server.json manifest for publishing an MCP server to registry.modelcontextprotocol.io, how to republish after a version bump, and how the registry relates to aggregators, marketplaces, and direct distribution.", "kind": "sub", "order": 7, "html": "https://changegamer.ai/articles/mcp-server-registry-publishing-playbook", "markdown": "https://changegamer.ai/articles/mcp-server-registry-publishing-playbook.md", "json": "https://changegamer.ai/api/articles/mcp-server-registry-publishing-playbook.json" }, { "slug": "mcp-server-cost-optimization", "title": "MCP Server Cost Optimization: Toolset Size, Caching Hints, and Fan-Out", "description": "How the token cost of an MCP server's tool list, the 2026-07-28 spec's ttlMs/cacheScope caching hints, fan-out from callers you do not control, and per-tool-name cost visibility each shape what a production MCP server actually costs to run.", "kind": "sub", "order": 8, "html": "https://changegamer.ai/articles/mcp-server-cost-optimization", "markdown": "https://changegamer.ai/articles/mcp-server-cost-optimization.md", "json": "https://changegamer.ai/api/articles/mcp-server-cost-optimization.json" }, { "slug": "mcp-server-failure-modes", "title": "Common MCP Server Failure Modes and How to Fix Them", "description": "A runtime playbook for the two MCP server failure modes with no dedicated deep-dive elsewhere: unrecoverable state after a mid-call crash, and malformed or hallucinated tool calls that reach the handler despite upstream validation.", "kind": "sub", "order": 9, "html": "https://changegamer.ai/articles/mcp-server-failure-modes", "markdown": "https://changegamer.ai/articles/mcp-server-failure-modes.md", "json": "https://changegamer.ai/api/articles/mcp-server-failure-modes.json" }, { "slug": "mcp-resources-and-prompts-vs-tools", "title": "MCP Tools vs Resources vs Prompts: How to Choose the Right Primitive", "description": "A decision procedure for MCP's three server-side primitives — who controls each one, a worked example of what it costs to expose a Resource as a Tool by mistake, and how Sampling and Elicitation fit as the client-side counterparts.", "kind": "sub", "order": 10, "html": "https://changegamer.ai/articles/mcp-resources-and-prompts-vs-tools", "markdown": "https://changegamer.ai/articles/mcp-resources-and-prompts-vs-tools.md", "json": "https://changegamer.ai/api/articles/mcp-resources-and-prompts-vs-tools.json" }, { "slug": "mcp-server-production-launch-checklist", "title": "The MCP Server Production Launch Checklist", "description": "A phase-by-phase go/no-go checklist for launching an MCP server: checkable gate conditions for transport and auth, tool design, cross-client testing, publish readiness, observability, and ongoing operation — with links to the mechanics each gate depends on.", "kind": "sub", "order": 11, "html": "https://changegamer.ai/articles/mcp-server-production-launch-checklist", "markdown": "https://changegamer.ai/articles/mcp-server-production-launch-checklist.md", "json": "https://changegamer.ai/api/articles/mcp-server-production-launch-checklist.json" }, { "slug": "mcp-enterprise-sso-id-jag", "title": "Zero-Touch Enterprise Authorization for MCP Servers: ID-JAG and SEP-990", "description": "How Enterprise-Managed Authorization (SEP-990) removes the per-server OAuth consent screen for MCP servers: the ID-JAG grant mechanism, its RFC 8693/7523 building blocks, named launch adopters as of August 2026, and how it layers on top of standard OAuth 2.1 rather than replacing it.", "kind": "sub", "order": 12, "html": "https://changegamer.ai/articles/mcp-enterprise-sso-id-jag", "markdown": "https://changegamer.ai/articles/mcp-enterprise-sso-id-jag.md", "json": "https://changegamer.ai/api/articles/mcp-enterprise-sso-id-jag.json" } ] }, "navigation": { "pillar": { "slug": "mcp-server-in-production", "title": "MCP Server in Production: How to Build, Ship and Run One", "description": "The operator playbook for taking an MCP server past the quickstart: transport choice, OAuth 2.1 auth, tool design, versioning against a moving spec, testing across clients, distribution, observability, cost and the failure modes that show up once real clients connect.", "kind": "pillar", "order": 0, "html": "https://changegamer.ai/articles/mcp-server-in-production", "markdown": "https://changegamer.ai/articles/mcp-server-in-production.md", "json": "https://changegamer.ai/api/articles/mcp-server-in-production.json" }, "previous": { "slug": "mcp-tool-description-injection", "title": "Defending MCP Clients Against Tool Description and Output Injection", "description": "Two distinct MCP injection surfaces — a tool description at connect-time and a tool's return value at call-time — and the client-side architectural patterns (Dual LLM, Action-Selector, Context-Minimization) that contain each one.", "kind": "sub", "order": 3, "html": "https://changegamer.ai/articles/mcp-tool-description-injection", "markdown": "https://changegamer.ai/articles/mcp-tool-description-injection.md", "json": "https://changegamer.ai/api/articles/mcp-tool-description-injection.json" }, "next": { "slug": "mcp-server-versioning-and-spec-migration", "title": "MCP Server Versioning and Spec Migration: An Operator Playbook", "description": "A migration runbook for MCP server operators: feature-detecting via capabilities instead of hard protocolVersion branching, a dual-version fleet rollout with rollback triggers, a compatibility shim for legacy clients still sending initialize, and a deprecation calendar built off the 12-month SEP-2577 floor.", "kind": "sub", "order": 5, "html": "https://changegamer.ai/articles/mcp-server-versioning-and-spec-migration", "markdown": "https://changegamer.ai/articles/mcp-server-versioning-and-spec-migration.md", "json": "https://changegamer.ai/api/articles/mcp-server-versioning-and-spec-migration.json" } }, "resources": [ { "slug": "testing-ai-agents", "html": "https://changegamer.ai/resources/testing-ai-agents", "markdown": "https://changegamer.ai/resources/testing-ai-agents.md", "json": "https://changegamer.ai/api/resources/testing-ai-agents.json" }, { "slug": "building-mcp-servers", "html": "https://changegamer.ai/resources/building-mcp-servers", "markdown": "https://changegamer.ai/resources/building-mcp-servers.md", "json": "https://changegamer.ai/api/resources/building-mcp-servers.json" }, { "slug": "reliable-tool-calling", "html": "https://changegamer.ai/resources/reliable-tool-calling", "markdown": "https://changegamer.ai/resources/reliable-tool-calling.md", "json": "https://changegamer.ai/api/resources/reliable-tool-calling.json" } ] }