#observability

2 resources and 22 guides tagged #observability on ChangeGamer.

Agent Observability and Tracing · Guide
Why agents need observability beyond app logs, how OpenTelemetry GenAI semantic conventions model agent runs as traces, key signals to capture, and a verified tooling landscape.
Shipping AI Agents to Production: A Production-Readiness Checklist · Guide
End-to-end checklist for productionizing an AI agent — evaluation gates, observability, guardrails, cost controls, resilience, durability, HITL approvals, secrets, rollback, and incident response.

Guides

Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions · guide
Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
MCP Authentication and the Production MCP Server Playbook · pillar guide
MCP authentication with OAuth 2.1, plus transport, tool design, versioning, testing, distribution and observability for a production MCP server.
MCP Server Observability with OpenTelemetry: Spans, Metrics, and Trace Correlation · guide
Instrumenting an MCP server past the pillar's baseline: what to put on a tool-call span beyond gen_ai.tool.name, what replaces the deprecated Logging primitive in practice, per-tool-name latency and error-rate metrics, and how a trace ID actually survives the agent-to-upstream-API hop.
MCP Server Cost Optimization: Toolset Size, Caching Hints, and Fan-Out · guide
How the token cost of an MCP server's tool list, the 2026-07-28 spec's ttlMs/cacheScope caching hints, fan-out from callers you do not control, and per-tool-name cost visibility each shape what a production MCP server actually costs to run.
How to Keep a RAG Index Fresh (Staleness Bounds, Not Vibes) · guide
Freshness as an engineered property: per-source staleness contracts, document versioning and tombstones, effective-date filtering, deletion propagation with reconciliation backstops, and the sync-lag metrics that predict stale answers before users report them.
Common RAG Failure Modes and How to Fix Them · guide
An operator runbook for the four RAG failure classes with no dedicated deep-dive elsewhere: retrieval miss, context overload, injection via content at ingestion time, and silent quality degradation — symptom, first diagnostic, and fix for each.
Agent Guardrails and the AI Agent Reliability Playbook · pillar guide
Agent guardrails plus the eleven other disciplines that make an AI agent reliable in production: tool calling, retries, durable execution and rollout.
How to Design Guardrails for AI Agent Reliability · guide
An operator playbook for reliability guardrails: the three checkpoints (input, output, action), layering cheap checks under slow ones with a fail-closed default, the two-of-three-properties rule for when a tool call needs human approval, and logging every verdict against the run trace ID.
How to Build an Incident Response Runbook for AI Agent Failures · guide
An operator playbook for the moment an AI agent fails in production: a triage step to classify the failure fast, trace freezing before rollback destroys the evidence, first-response depth on the four failure classes, and a blameless post-mortem that feeds back into guardrails and eval.
What Should an AI Agent's Observability System Capture? · guide
An operator playbook for instrumenting an AI agent: the trace/span model behind a run, the OpenTelemetry GenAI attributes that name each field, the fields worth capturing per span, and what to redact before any of it gets logged.
How to Build an Audit Trail for an AI Agent · guide
A field-by-field forensic playbook for an AI agent audit log: why each of the checklist's seven required fields matters for reconstruction, a worked incident walkthrough, and the real tension between OpenTelemetry's redact-by-default tracing and full audit logging.
AI Agent Observability and the Production Evaluation Playbook · pillar guide
AI agent observability and evaluation once an agent is already live: tracing spans, retrieval-attribution logs, judge-based screening vs. human acceptance, online metrics, cost telemetry, and incident postmortems that feed evaluation fixtures.
Distributed Tracing for Multi-Agent AI Systems · guide
How trace ID propagation actually works across a multi-agent handoff, why handoff and delegation need different trace shapes, and why an orchestrator-level trace can hide a failed leg of a fan-out.
Designing an LLM-as-Judge Pipeline for Production AI Agents · guide
An operator playbook for screening live AI agent output with an LLM judge: a confidence/stakes routing architecture to a human queue, continuous live-traffic rubric design, and per-bias mitigations for position, verbosity, and self-preference.
How to Track AI Agent Costs in Production · guide
How to track AI agent costs in production: version your price table, separate cached from uncached tokens, keep failed runs in the books, attribute shared costs, reconcile against the invoice, and alert on rate of spend.
How to Log RAG Retrieval in Production for Debugging Agent Answers · guide
How to log RAG retrieval in production so a wrong agent answer can be debugged: per-retrieval fields, what reached the context window versus what the model cited, privacy by hashes and IDs, and retention and sampling choices.
How to Measure AI Agent Quality from Live User Feedback · guide
How to measure AI agent quality from live user feedback: why explicit ratings are sparse and biased, how re-asks, abandonment and escalations mislead, and how to join each signal to a trace and route it to review.
How to Detect Quality Drift in a Production AI Agent · guide
How to detect quality drift in a production AI agent: baseline aggregate signals, alert on a diff against the baseline, and separate a prompt, model or tool-version change from a shift in traffic mix.
How to Sample and Retain Production AI Agent Traces · guide
How to sample and retain production AI agent traces: head vs tail sampling, keep-all-errors plus a random baseline, whole-trace decisions for multi-agent runs, retention tiers and redaction before the clock starts.
How to Keep Trace Data When an AI Agent Crashes Mid-Run · guide
How to keep trace data when an AI agent crashes mid-run: write a small run-start record outside the trace buffer, detect orphans, count them as unknown outcomes, and force-decide on shutdown.
How to Choose an LLM Observability Platform for AI Agents · guide
How to choose an LLM observability platform: decide on OTel-native ingestion, self-host versus cloud, export portability, redaction hooks and retention support before comparing vendors.
How to Redact PII From AI Agent Traces: Placement, Testing and Cleanup · guide
How to redact PII from AI agent traces in practice: where the redactor sits in the pipeline, what to do per span field, how to test it with seeded fake PII, and how to clean up after a leak.