{
  "slug": "agent-observability",
  "title": "Agent Observability and Tracing",
  "description": "Why agents need observability beyond app logs, how OpenTelemetry GenAI semantic conventions model agent runs as traces, key signals to capture, and a verified tooling landscape.",
  "category": "Guide",
  "tags": [
    "observability",
    "tracing",
    "opentelemetry",
    "agents",
    "debugging",
    "evaluation"
  ],
  "updated": "2026-07-18",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/agent-observability",
  "markdown": "https://changegamer.ai/resources/agent-observability.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "The span/trace model maps naturally to agent runs",
      "anchor": "the-span-trace-model-maps-naturally-to-agent-runs"
    },
    {
      "depth": 2,
      "text": "OpenTelemetry GenAI semantic conventions",
      "anchor": "opentelemetry-genai-semantic-conventions"
    },
    {
      "depth": 2,
      "text": "Key signals to capture per agent run",
      "anchor": "key-signals-to-capture-per-agent-run"
    },
    {
      "depth": 2,
      "text": "Tooling landscape",
      "anchor": "tooling-landscape"
    },
    {
      "depth": 3,
      "text": "Open-source / vendor-neutral",
      "anchor": "open-source-vendor-neutral"
    },
    {
      "depth": 3,
      "text": "Framework-native",
      "anchor": "framework-native"
    },
    {
      "depth": 2,
      "text": "Evals + observability connection",
      "anchor": "evals-observability-connection"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "shipping-agents-to-production",
      "title": "Shipping AI Agents to Production: A Production-Readiness Checklist",
      "description": "End-to-end checklist for productionizing an AI agent — evaluation gates, observability, guardrails, cost controls, resilience, durability, HITL approvals, secrets, rollback, and incident response.",
      "url": "https://changegamer.ai/resources/shipping-agents-to-production"
    },
    {
      "slug": "choosing-an-llm-for-agents",
      "title": "How to Choose an LLM for Agentic Tasks",
      "description": "A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.",
      "url": "https://changegamer.ai/resources/choosing-an-llm-for-agents"
    },
    {
      "slug": "prompt-management-and-versioning",
      "title": "Prompt Management and Versioning: The Ops of Prompts in Production",
      "description": "Treating prompts as deployable artifacts: versioning, external registries, A/B and canary testing, eval-gated promotion, rollback, and the composite-version problem.",
      "url": "https://changegamer.ai/resources/prompt-management-and-versioning"
    },
    {
      "slug": "rag-retrieval-for-agents",
      "title": "RAG and Retrieval for Agents",
      "description": "End-to-end practitioner reference for Retrieval-Augmented Generation: pipeline stages, chunking strategies, dense/sparse/hybrid retrieval, reranking, agentic retrieval patterns, quality failure modes, and evaluation — with verified sources for every named technique.",
      "url": "https://changegamer.ai/resources/rag-retrieval-for-agents"
    }
  ],
  "furtherReading": [
    {
      "slug": "mcp-server-observability-opentelemetry",
      "title": "MCP Server Observability with OpenTelemetry: Spans, Metrics, and Trace Correlation",
      "description": "Instrumenting an MCP server past the pillar's baseline: what to put on a tool-call span beyond gen_ai.tool.name, what replaces the deprecated Logging primitive in practice, per-tool-name latency and error-rate metrics, and how a trace ID actually survives the agent-to-upstream-API hop.",
      "url": "https://changegamer.ai/articles/mcp-server-observability-opentelemetry"
    },
    {
      "slug": "mcp-server-cost-optimization",
      "title": "MCP Server Cost Optimization: Toolset Size, Caching Hints, and Fan-Out",
      "description": "How the token cost of an MCP server's tool list, the 2026-07-28 spec's ttlMs/cacheScope caching hints, fan-out from callers you do not control, and per-tool-name cost visibility each shape what a production MCP server actually costs to run.",
      "url": "https://changegamer.ai/articles/mcp-server-cost-optimization"
    }
  ],
  "body": "Standard application logging — a flat stream of timestamped lines — cannot answer \"why did the agent do that?\" Agent runs are non-deterministic, multi-step, and branching. A single run may spawn dozens of LLM calls, tool calls, and sub-agent delegations. Without structured tracing, debugging is guesswork and cost attribution is impossible.\n\n## Key facts\n\n- Flat, timestamped log lines can't explain a non-deterministic, branching agent run that may fan out into dozens of model, tool, and sub-agent calls, so without structured tracing, engineers are stuck guessing and can't attribute cost.\n- A trace bundles an entire run under one stable ID that threads through every child operation, while spans mark each individual step — an LLM call, tool call, sub-agent hop, or retrieval — nested into a full tree.\n- The GenAI SIG's attribute and span vocabulary is still labeled Development rather than final, yet leading observability vendors have already adopted `gen_ai.*` naming for spans, events, and metrics covering model calls, agent workflows, prompts/completions, and usage histograms.\n- Solid instrumentation records the whole call tree, converts token counts into cost per span, times each step plus the overall run, keeps redacted tool payloads, and attaches exceptions to whichever span failed — with a single ID linking it all together.\n- Options split between framework-agnostic OTel platforms — Langfuse, Phoenix, OpenLLMetry, Logfire — and tracing built directly into LangChain's LangSmith or the OpenAI Agents SDK.\n- Because stored spans double as raw material, sampling a slice of live runs builds datasets for judge- or metric-based evaluation, while watching the same stream in real time flags unusual behavior as it happens.\n\n## The span/trace model maps naturally to agent runs\n\nOpenTelemetry's span/trace model fits agents well:\n\n- **Trace** = one complete agent run, identified by a stable `trace_id` propagated across all child operations, including sub-agents. The same `trace_id` threading through every span is what enables cross-agent debugging.\n- **Span** = one discrete operation: an LLM call, a tool call, a sub-agent invocation, a retrieval step. Spans are nested (parent/child) to form the full trace tree.\n\nThis maps directly to what agent builders need: a tree view of every decision, the inputs and outputs at each node, latency per step, and a single ID to correlate across services.\n\n## OpenTelemetry GenAI semantic conventions\n\nThe OpenTelemetry GenAI SIG (formed April 2024) defines vendor-neutral attribute names, span types, events, and metrics for LLM and agent workloads. As of July 2026 the conventions still have **Development** status (formerly called experimental) — the attribute names carry a stability opt-in flag (`gen_ai_latest_experimental`) and may still change, but major observability vendors already support them. The spec itself has since moved out of the main `open-telemetry/semantic-conventions` repo into a dedicated `open-telemetry/semantic-conventions-genai` repo, and the old opentelemetry.io GenAI pages now just point readers there.\n\nCoverage breaks into four areas:\n\n- **Client spans** — LLM calls and retrieval steps (`gen_ai.*` attributes: model name, token usage, finish reason).\n- **Agent spans** — agent invocations and workflows; each tool call, LLM step, and retrieval becomes a child span.\n- **Events** — prompt and completion bodies captured as span events (off by default for PII safety).\n- **Metrics** — token usage and latency histograms.\n\nKey `gen_ai.*` attributes: `gen_ai.provider.name` (the identifier for which model/vendor emitted the span — e.g. `openai`, `anthropic`, `aws.bedrock`; this replaced the older `gen_ai.system` name, which is now deprecated though many instrumentation libraries still emit it during the transition), `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.tool.name`.\n\n## Key signals to capture per agent run\n\n- Full trace tree: every LLM call, tool call, and sub-agent invocation as a span.\n- Per-span token usage and derived cost (input + output tokens × model price).\n- Latency per span and end-to-end trace duration.\n- Tool-call inputs and outputs (redact PII before logging).\n- Errors and retries with the original exception attached to the failing span.\n- A stable `trace_id` propagated into all sub-agent calls (see /resources/agentic-security-checklist, section 11 — logging and auditability).\n\n## Tooling landscape\n\n### Open-source / vendor-neutral\n\n- **Langfuse** — open-source LLM engineering platform with an OTLP ingestion endpoint (`/api/public/otel`); accepts OTel traces and aims to comply with GenAI semantic conventions. Self-host or cloud. Source: github.com/langfuse/langfuse.\n- **Arize Phoenix** — open-source observability and evaluation platform; OTel-native, accepts traces over OTLP, auto-instruments LangChain, LlamaIndex, OpenAI, Anthropic, and others via OpenInference. Runs fully local (no API key required). Source: github.com/Arize-ai/phoenix.\n- **OpenLLMetry (Traceloop)** — OTel instrumentations for LLM providers and vector DBs; a Traceloop SDK wrapper emits standard OTel data you route to any OTel-compatible backend (Langfuse, Datadog, Grafana Tempo, etc.). Apache 2.0. Source: github.com/traceloop/openllmetry.\n- **Logfire (Pydantic)** — OTel-based observability platform with first-class Python and Pydantic AI integration; tracks token usage, cost, and tool calls; ships built-in inside Pydantic AI (see /resources/agent-frameworks-compared). Source: github.com/pydantic/logfire.\n\n### Framework-native\n\n- **LangSmith (LangChain)** — paired tracing and evaluation SaaS for LangChain and LangGraph; supports full end-to-end OTel ingestion so you can route spans to LangSmith and other backends simultaneously. Paid product; free tier available.\n- **OpenAI Agents SDK tracing** — built-in trace processor that captures agent runs, handoffs, and tool calls; exports to the OpenAI Traces dashboard by default. Custom `TracingProcessor` implementations let you redirect spans to any OTel-compatible backend.\n\n## Evals + observability connection\n\nTraces are the raw material for both offline eval and online monitoring. Stored traces feed evaluation datasets (sample a slice of production runs → score with an LLM judge or deterministic metric). Online monitoring alerts on anomalous patterns in the live trace stream. See /resources/evaluating-ai-agents for eval methodology and /resources/agent-frameworks-compared for framework-native tracing details. For a coarser-grained but real example of run-level audit trails, the premium resource [How ChangeGamer Runs Itself](/resources/autonomous-operations) describes the append-only JOURNAL this site's own agent cycles log per run (objective, outcome, verdict).\n\n## Verified sources\n\n- OTel GenAI semantic conventions — agent and framework spans (Development status; moved from opentelemetry.io into the dedicated semantic-conventions-genai repo): https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md\n- OTel GenAI semantic conventions — generative client AI spans: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md\n- OTel blog — Inside the LLM Call: GenAI Observability (2026): https://opentelemetry.io/blog/2026/genai-observability/\n- OTel blog — AI Agent Observability: Evolving Standards and Best Practices (2025): https://opentelemetry.io/blog/2025/ai-agent-observability/\n- Langfuse OTel integration docs: https://langfuse.com/integrations/native/opentelemetry\n- Langfuse GitHub: https://github.com/langfuse/langfuse\n- Arize Phoenix docs: https://arize.com/docs/phoenix\n- Arize Phoenix GitHub: https://github.com/Arize-ai/phoenix\n- OpenLLMetry GitHub (Traceloop): https://github.com/traceloop/openllmetry\n- Logfire (Pydantic) AI observability docs: https://logfire.pydantic.dev/docs/ai-observability/\n- Logfire GitHub: https://github.com/pydantic/logfire\n- LangSmith OTel support announcement: https://www.langchain.com/blog/end-to-end-opentelemetry-langsmith\n- OpenAI Agents SDK — Tracing docs: https://openai.github.io/openai-agents-python/tracing/",
  "sources": [
    "https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md",
    "https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md",
    "https://opentelemetry.io/blog/2026/genai-observability/",
    "https://opentelemetry.io/blog/2025/ai-agent-observability/",
    "https://langfuse.com/integrations/native/opentelemetry",
    "https://github.com/langfuse/langfuse",
    "https://arize.com/docs/phoenix",
    "https://github.com/Arize-ai/phoenix",
    "https://github.com/traceloop/openllmetry",
    "https://logfire.pydantic.dev/docs/ai-observability/",
    "https://github.com/pydantic/logfire",
    "https://www.langchain.com/blog/end-to-end-opentelemetry-langsmith",
    "https://openai.github.io/openai-agents-python/tracing/"
  ]
}