Guides

How to make a site work for AI agents: llms.txt, crawler access, structured data, licensing and machine-payable access, with a 30-day plan.

The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.

A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.

How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.

What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.

Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.

Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.

A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.

Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.

How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.

Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.

How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.

Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.

A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

The operator playbook for the x402 protocol and selling content, APIs and tools to AI agents that discover, evaluate and pay on their own.

Why checkout built for a person watching a screen is unusable by an AI agent, and what a checkout flow that actually completes for a machine buyer looks like — 402 + API key versus native x402.

Why a prose pricing page cannot be evaluated by an AI agent, what fields a machine-readable offer catalog needs, and how to keep it in lockstep with your human pricing page and your 402 body.

A decision framework for choosing between ACP, AP2, and x402 (plus the self-hosted 402 gate) — sorted by who your buyer actually is, what you are selling, and what is live versus waitlisted as of July 2026.

A build guide for sellers who have already decided x402 is the right rail: the 402 response shape, the wallet/facilitator/network choices, the verify-then-settle retry flow, exact vs. upto pricing, and how to ship it dormant until you are ready to go live.

How to design a system that mints and delivers API keys to agent and software buyers with minimal human friction: trigger models, storage, delivery, key format, tiering, rotation and revocation — illustrated with ChangeGamer's own Stripe-webhook mechanism.

The axes that actually distinguish one pricing tier from another for a machine buyer — rate limits, content scope, deliverables and license grant — and how ChangeGamer structures its own four tiers around deliverable and license, not gated content.

The seller-side counterpart to agent spend controls — how an API operator reads an inbound agent's spend ceiling before granting access, which payment protocols actually prove that ceiling, how to revoke access, and what "trust" operationally means for a seller when no portable agent-reputation standard exists yet.

What happens on the seller side when an autonomous agent's purchase needs to be reversed or is disputed — API-key refund mechanics, why x402 settlement cannot be undone, what card-token revocation does and does not prove, and what to log before you reverse anything.

The packaging decisions behind selling a content corpus as a dataset product — export format, the free-sample/gated-full split, a per-record metadata schema, a corpus version number, and which of three delivery mechanisms to use — grounded in ChangeGamer's own three real export formats.

How an AI agent finds out a paid API or resource exists before it ever reads a price: llms.txt, the JSON API index, MCP registries, incidental 402 discovery, x402 auto-listing, and what .well-known does and does not cover.

How a seller of APIs, content, or tools to AI agents spots and mitigates abuse once access is already granted — key sharing, over-scope scraping, spend-ceiling circumvention, spoofed identity, and rate-limit evasion patterns specific to autonomous agents.

The revenue-layer fields and queries a seller adds on top of a general traffic log — authorized-vs-settled, revenue per rail, revenue per tier, and how to avoid double-counting a webhook retry as two sales.

MCP authentication with OAuth 2.1, plus transport, tool design, versioning, testing, distribution and observability for a production MCP server.

Which MCP transport to build against and why: the single-client-vs-shared decision rule, how state works without a session handshake under the 2026-07-28 spec, the auth-model switching cost, and what actually breaks migrating off HTTP+SSE.

A wire-level implementation walkthrough for OAuth 2.1 on a remote MCP server: what the discovery documents actually contain, CIMD vs. Dynamic Client Registration in your server code, per-SEP detail from the 2026-07-28 hardening set, and token-validation mechanics.

Two distinct MCP injection surfaces — a tool description at connect-time and a tool's return value at call-time — and the client-side architectural patterns (Dual LLM, Action-Selector, Context-Minimization) that contain each one.

The implementation mechanics below the three-layer test pyramid: what a mocked MCP transport actually replaces, what a Streamable HTTP cassette contains, a concrete CI job/trigger shape, and how to catch spec-version drift before it reaches production.

A migration runbook for MCP server operators: feature-detecting via capabilities instead of hard protocolVersion branching, a dual-version fleet rollout with rollback triggers, a compatibility shim for legacy clients still sending initialize, and a deprecation calendar built off the 12-month SEP-2577 floor.

Instrumenting an MCP server past the pillar's baseline: what to put on a tool-call span beyond gen_ai.tool.name, what replaces the deprecated Logging primitive in practice, per-tool-name latency and error-rate metrics, and how a trace ID actually survives the agent-to-upstream-API hop.

A step-by-step walkthrough of the mcp-publisher CLI and the server.json manifest for publishing an MCP server to registry.modelcontextprotocol.io, how to republish after a version bump, and how the registry relates to aggregators, marketplaces, and direct distribution.

How the token cost of an MCP server's tool list, the 2026-07-28 spec's ttlMs/cacheScope caching hints, fan-out from callers you do not control, and per-tool-name cost visibility each shape what a production MCP server actually costs to run.

A runtime playbook for the two MCP server failure modes with no dedicated deep-dive elsewhere: unrecoverable state after a mid-call crash, and malformed or hallucinated tool calls that reach the handler despite upstream validation.

A decision procedure for MCP's three server-side primitives — who controls each one, a worked example of what it costs to expose a Resource as a Tool by mistake, and how Sampling and Elicitation fit as the client-side counterparts.

A phase-by-phase go/no-go checklist for launching an MCP server: checkable gate conditions for transport and auth, tool design, cross-client testing, publish readiness, observability, and ongoing operation — with links to the mechanics each gate depends on.

How Enterprise-Managed Authorization (SEP-990) removes the per-server OAuth consent screen for MCP servers: the ID-JAG grant mechanism, its RFC 8693/7523 building blocks, named launch adopters as of August 2026, and how it layers on top of standard OAuth 2.1 rather than replacing it.

The operator playbook for agentic RAG in production: ingestion, chunking, hybrid retrieval, reranking, evaluation, freshness and cost.

The six properties that separate a production RAG ingestion pipeline from a demo script: tested extraction, idempotent writes, incremental updates, deletion propagation, durable execution, and metadata captured at ingest time.

Chunking decisions that actually move retrieval quality: structural boundaries before fixed windows, parent-document expansion, special handling for tables and code, overlap trade-offs, and tuning against recall@k instead of blog defaults.

Embedding selection as an operations problem: the criteria that dominate total cost of ownership, the never-mix-spaces invariant, reindex migrations with dual indexes, and where quantization fits once cost shows up.

Why hybrid retrieval is the production default rather than an upgrade: complementary failure modes of lexical and dense search, reciprocal rank fusion versus weighted scoring, parameter choices, and how filtering interacts with fusion.

Reranking as a budget decision: why first-stage ranking misorders good evidence, when cross-encoder reranking pays for itself, how to pick candidate depth at the knee, gating by query difficulty, and deduplicating after fusion.

The evaluation harness that keeps RAG changeable: golden-set construction, retrieval metrics separated from generation metrics, LLM-as-judge screening with human acceptance, CI regression gates, and the logged-query flywheel.

Freshness as an engineered property: per-source staleness contracts, document versioning and tombstones, effective-date filtering, deletion propagation with reconciliation backstops, and the sync-lag metrics that predict stale answers before users report them.

How to enforce access control inside retrieval for multi-tenant RAG: authorization as mandatory pre-filters derived from the authenticated principal, tenant isolation mechanics, permission-negative testing, and the post-filter trap that produces empty answers.

When AI agents consume retrieval as a tool rather than a pipeline stage: narrow tool contracts, hard budgets on iterative retrieval, query decomposition, provenance-carrying results, and keeping trust boundaries inside the retrieval path.

RAG cost and latency as engineered budgets: where the money actually goes, caching layers and their hit-rate economics, routing queries to right-sized models, bounding retrieval fan-out, the hidden lines (reindex migrations, eval compute), and p95 discipline.

An operator runbook for the four RAG failure classes with no dedicated deep-dive elsewhere: retrieval miss, context overload, injection via content at ingestion time, and silent quality degradation — symptom, first diagnostic, and fix for each.

A decision framework for choosing graph-structured retrieval over standard vector RAG: which query types GraphRAG actually wins, what building a knowledge graph costs, named implementations, and hybrid vector-plus-graph patterns.

A worked decision guide for the four real alternatives to retrieval-augmented generation: including knowledge directly, querying structured data with text-to-SQL, fine-tuning for behavior change, and graph-based retrieval for entity relationships.

Agent guardrails plus the eleven other disciplines that make an AI agent reliable in production: tool calling, retries, durable execution and rollout.

An operator playbook for tool-calling contracts: per-provider strict-mode config (OpenAI, Anthropic, Gemini, self-hosted grammars), the full failure-mode-to-fix table, what BFCL actually measures, and how to stop parallel tool calls from breaking a dependency chain.

A decision framework for structured outputs versus tool calling in AI agents, with runnable JSON Schema examples for OpenAI, Anthropic, Gemini, vLLM/SGLang, and llama.cpp, plus mitigation code for truncation, refusal, and grammar-compilation latency.

A deep-dive on retrying agent tool calls safely: the transient-vs-terminal decision, why an agent side effect can fire before a failure signal reaches the caller, idempotency-key mechanics (run ID + step index), the unknown-outcome edge case, and where idempotency keys do not reach.

A deep-dive on durable execution for AI agents: the persisted event log, the replay-determinism constraint, the four architectural shapes mapped across ten engines and frameworks, and a decision framework for when a durable execution engine is worth adding at all.

An operator playbook for reliability guardrails: the three checkpoints (input, output, action), layering cheap checks under slow ones with a fail-closed default, the two-of-three-properties rule for when a tool call needs human approval, and logging every verdict against the run trace ID.

An operator playbook for gating an AI agent release in CI: why agent eval needs trajectory-level scoring across the tasks it actually runs, how public benchmarks diverge as proxies, ground-truth vs LLM-as-judge tool-call scoring, and the three-layer test pyramid that keeps CI fast and non-flaky.

An operator playbook for shipping a new agent version without breaking production: in-repo vs. registry prompt storage, a version-numbering comparison, the six-step promotion flow, A/B-test mechanics, the composite-version trace fields, and a rollback drill.

An operator playbook for the moment an AI agent fails in production: a triage step to classify the failure fast, trace freezing before rollback destroys the evidence, first-response depth on the four failure classes, and a blameless post-mortem that feeds back into guardrails and eval.

A deep-dive on timeout and deadline design for AI agents: sizing LLM-call, tool-call, and sub-agent-hop timeouts differently, allocating a wall-clock budget across a multi-step chain, and propagating a remaining-deadline value from parent to child calls.

A deep-dive on the circuit breaker pattern for AI agents: the Closed/Open/Half-Open state machine with a worked open-source example, where to place a breaker in an agent's call path, and degraded-mode fallback design as its own discipline per dependency type.

An operator playbook for instrumenting an AI agent: the trace/span model behind a run, the OpenTelemetry GenAI attributes that name each field, the fields worth capturing per span, and what to redact before any of it gets logged.

A go/no-go checklist that turns the agent reliability pillar's twelve-discipline closing list into checkable gates — the specific artifact, header, or trace field that proves each one holds, with a link to whichever sibling article owns its mechanics.

Credential hygiene, prompt-injection defense in depth, sandboxing choices for code execution, supply-chain provenance, least privilege, audit trails, incident response, and rate/abuse controls — eight operator-side defenses against an adversarial actor or a compromised dependency, not against ordinary load or failure.

Why single-agent credential issuance is not the whole secrets problem: auditing which tool servers and MCP connectors hold credentials on an agent's behalf, treating provider-side prompt caches as a disclosure surface, and the named frameworks — OWASP's Secrets Management Cheat Sheet, Twelve-Factor config, and the OWASP GenAI project — that govern the rest.

A decision framework for matching Action-Selector, Plan-Then-Execute, Dual LLM, and the other named architectural patterns to a task's actual blast radius, including when to compose two patterns together and when none of them is worth the overhead.

A two-axis framework — trust in the code's source crossed with the blast radius of a successful escape — for picking an isolation layer, choosing among six hosted sandbox APIs, and hardening the harness around whichever one you pick.

An operational playbook for three separate trust-boundary gates — package-install time, model-load time, and MCP-server-connect time — that turns SBOM and attestation formats into checks a pipeline can actually run, plus a fail-closed default for the dependency that carries neither.

How a least-privilege grant actually gets enforced once an agent is running — the eight named checkpoints, five verdicts, and fail-closed contract in Microsoft's draft Agent Control Specification (ACS), and what happens when the policy engine that enforces the boundary breaks.

A field-by-field forensic playbook for an AI agent audit log: why each of the checklist's seven required fields matters for reconstruction, a worked incident walkthrough, and the real tension between OpenTelemetry's redact-by-default tracing and full audit logging.

Why the pillar's cut-credential, stop-instance, export-evidence order is structurally forced rather than a tidy convention, a concrete answer for who holds standing revocation authority, a pre-incident drill for the evidence-export path, and what changes when the compromised credential belongs to a third-party tool server instead of the agent itself.

Enforcement mechanics for the two ceilings an agent operator should set before production: where a per-credential tool-call counter has to live to stay correct under concurrent calls, where a spend ceiling gets checked in the tool-call loop, and how to reject out-of-scope tool-call arguments with canonicalization rather than a naive prefix match.

A three-state decision procedure — valid manifest, invalid signature, absent manifest — for what an AI agent's ingestion pipeline should do differently with a web image, an email attachment, or a retrieved document, plus a checklist for wiring a C2PA reader library into that pipeline as a gate before content reaches the model.

The two-layer model an autonomous agent needs to pass before any credential-custody or permission question even applies: a cryptographic workload identity proving what it is (SPIFFE/SPIRE, cloud workload identity federation) and a separate delegated-authority grant proving it may act on a human's or org's behalf (OAuth scopes, RFC 8693 token exchange, RFC 8707 audience binding).

Why an AI agent expands PII exposure past a bounded API call — large ingested context, external tool calls, persistent memory and logs, provider training risk — and the containment controls, provider data-handling terms, and GDPR/EU AI Act/CCPA compliance boundary that follow from it.

A go/no-go checklist that turns the agent security operations pillar's eight disciplines, plus content provenance, agent identity, and data privacy, into checkable gates — the specific inventory row, test result, or logged decision that proves each one holds, with a link to whichever sibling article owns its mechanics.

AI agent observability and evaluation once an agent is already live: tracing spans, retrieval-attribution logs, judge-based screening vs. human acceptance, online metrics, cost telemetry, and incident postmortems that feed evaluation fixtures.

How trace ID propagation actually works across a multi-agent handoff, why handoff and delegation need different trace shapes, and why an orchestrator-level trace can hide a failed leg of a fan-out.

An operator playbook for screening live AI agent output with an LLM judge: a confidence/stakes routing architecture to a human queue, continuous live-traffic rubric design, and per-bias mitigations for position, verbosity, and self-preference.

How to track AI agent costs in production: version your price table, separate cached from uncached tokens, keep failed runs in the books, attribute shared costs, reconcile against the invoice, and alert on rate of spend.

How to turn an AI agent incident into an evaluation test case: freeze the trace, redact it, minimize it to the failing decision, label the expected outcome with a trusted oracle, prove it fails then passes, and retire it later.

How to log RAG retrieval in production so a wrong agent answer can be debugged: per-retrieval fields, what reached the context window versus what the model cited, privacy by hashes and IDs, and retention and sampling choices.

How to measure AI agent quality from live user feedback: why explicit ratings are sparse and biased, how re-asks, abandonment and escalations mislead, and how to join each signal to a trace and route it to review.

How to detect quality drift in a production AI agent: baseline aggregate signals, alert on a diff against the baseline, and separate a prompt, model or tool-version change from a shift in traffic mix.

How to sample and retain production AI agent traces: head vs tail sampling, keep-all-errors plus a random baseline, whole-trace decisions for multi-agent runs, retention tiers and redaction before the clock starts.

How to keep trace data when an AI agent crashes mid-run: write a small run-start record outside the trace buffer, detect orphans, count them as unknown outcomes, and force-decide on shutdown.

How to choose an LLM observability platform: decide on OTel-native ingestion, self-host versus cloud, export portability, redaction hooks and retention support before comparing vendors.

How to redact PII from AI agent traces in practice: where the redactor sits in the pipeline, what to do per span field, how to test it with seeded fake PII, and how to clean up after a leak.