{
  "slug": "streaming-for-agents",
  "title": "Streaming Responses for Agents",
  "description": "Transport formats, provider event schemas, and practical concerns for consuming streamed LLM responses in production agents: SSE mechanics, OpenAI (Chat Completions and Responses API) and Anthropic event formats, partial-JSON tool-call parsing, backpressure, cancellation, and gateway proxying.",
  "category": "Guide",
  "tags": [
    "streaming",
    "sse",
    "server-sent-events",
    "openai",
    "anthropic",
    "gemini",
    "tool-calling",
    "latency",
    "agents"
  ],
  "updated": "2026-07-18",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/streaming-for-agents",
  "markdown": "https://changegamer.ai/resources/streaming-for-agents.md",
  "outline": [
    {
      "depth": 2,
      "text": "Transport: Server-Sent Events (SSE)",
      "anchor": "transport-server-sent-events-sse"
    },
    {
      "depth": 2,
      "text": "OpenAI Chat Completions streaming",
      "anchor": "openai-chat-completions-streaming"
    },
    {
      "depth": 2,
      "text": "OpenAI Responses API streaming (newer, recommended for new agent projects)",
      "anchor": "openai-responses-api-streaming-newer-recommended-for-new-agent-projects"
    },
    {
      "depth": 2,
      "text": "Anthropic Messages streaming",
      "anchor": "anthropic-messages-streaming"
    },
    {
      "depth": 2,
      "text": "Google Gemini streaming",
      "anchor": "google-gemini-streaming"
    },
    {
      "depth": 2,
      "text": "Streaming tool calls and structured outputs",
      "anchor": "streaming-tool-calls-and-structured-outputs"
    },
    {
      "depth": 2,
      "text": "Practical concerns for agent builders",
      "anchor": "practical-concerns-for-agent-builders"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "prompt-caching-for-agents",
      "title": "Prompt Caching for AI Agents",
      "description": "Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.",
      "url": "https://changegamer.ai/resources/prompt-caching-for-agents"
    },
    {
      "slug": "computer-use-browser-automation",
      "title": "Computer Use and Browser Automation for Agents",
      "description": "Two-layer reference: vendor computer-use APIs (Anthropic, OpenAI CUA, Google Gemini) that translate screenshots to actions, and the open harnesses (Playwright MCP, browser-use, Stagehand, Skyvern) that execute those actions — with loop mechanics, reliability tradeoffs, and security gates.",
      "url": "https://changegamer.ai/resources/computer-use-browser-automation"
    },
    {
      "slug": "handling-rate-limits-and-retries",
      "title": "Handling LLM Rate Limits (HTTP 429) and Retries for Agents",
      "description": "A practical reference for agent builders: what a 429 means, how to read provider rate-limit headers, exponential backoff with jitter, client-side throttling, and when to use a batch API.",
      "url": "https://changegamer.ai/resources/handling-rate-limits-and-retries"
    },
    {
      "slug": "choosing-an-llm-for-agents",
      "title": "How to Choose an LLM for Agentic Tasks",
      "description": "A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.",
      "url": "https://changegamer.ai/resources/choosing-an-llm-for-agents"
    }
  ],
  "furtherReading": [
    {
      "slug": "mcp-server-failure-modes",
      "title": "Common MCP Server Failure Modes and How to Fix Them",
      "description": "A runtime playbook for the two MCP server failure modes with no dedicated deep-dive elsewhere: unrecoverable state after a mid-call crash, and malformed or hallucinated tool calls that reach the handler despite upstream validation.",
      "url": "https://changegamer.ai/articles/mcp-server-failure-modes"
    },
    {
      "slug": "mcp-tool-description-injection",
      "title": "Defending MCP Clients Against Tool Description and Output Injection",
      "description": "Two distinct MCP injection surfaces — a tool description at connect-time and a tool's return value at call-time — and the client-side architectural patterns (Dual LLM, Action-Selector, Context-Minimization) that contain each one.",
      "url": "https://changegamer.ai/articles/mcp-tool-description-injection"
    }
  ],
  "body": "Streaming lets an agent start processing model output before the full response is complete. For agent builders the three payoffs are: (1) **time-to-first-token (TTFT)** — perceived latency drops because the pipeline can act on early output; (2) **early cancellation** — if the first few tokens reveal a hallucination or wrong tool, the request can be aborted before paying for the full generation; (3) **incremental parsing** — tool-call arguments and structured outputs arrive as partial JSON that can be validated and acted on progressively. See /resources/agent-cost-latency-optimization for the latency framing.\n\n## Transport: Server-Sent Events (SSE)\n\nAll three major providers (OpenAI, Anthropic, Google Gemini) stream over HTTP using **Server-Sent Events** (SSE), the W3C/WHATWG standard for unidirectional server-to-client push over a plain HTTP connection. The wire format is `Content-Type: text/event-stream`; each event is one or more `data:` lines terminated by a blank line. Named events use an `event:` field before the `data:` field.\n\nSSE works over HTTP/1.1 (chunked transfer encoding) and HTTP/2 (a single stream). WebSockets are used for **bidirectional** real-time protocols (e.g., OpenAI Realtime API for voice); pure generation streaming uses SSE, not WebSockets.\n\n## OpenAI Chat Completions streaming\n\nSet `\"stream\": true` in the request body. The response is a sequence of `data:` SSE lines, each carrying a JSON object of type `chat.completion.chunk`. Each chunk has:\n\n- `choices[].delta` — incremental content fragment. On the first chunk, `delta.role` is `\"assistant\"`. Subsequent chunks carry `delta.content` (text fragment) or `delta.tool_calls` (partial tool-call data).\n- `choices[].finish_reason` — `null` during the stream; `\"stop\"`, `\"tool_calls\"`, or another terminal value on the final content chunk.\n\nThe stream ends with `data: [DONE]` — a sentinel that is not valid JSON and signals the consumer to close the connection.\n\n**Tool calls in OpenAI streaming**: `delta.tool_calls` is a list indexed by position. The first delta for a call includes `id`, `type: \"function\"`, and `function.name`. Subsequent deltas carry only `function.arguments` as a *partial JSON string fragment*. The consumer must concatenate all `function.arguments` fragments across deltas, then parse the complete string as JSON after `finish_reason: \"tool_calls\"` is received. See /resources/reliable-tool-calling for schema-validation strategies on the parsed result.\n\n## OpenAI Responses API streaming (newer, recommended for new agent projects)\n\nOpenAI now positions the newer **Responses API** (`/v1/responses`) as the primitive for new agentic projects — Chat Completions stays fully supported, but new capabilities land in Responses first (reported, as of July 2026, per OpenAI's own developer-docs page titles and independent coverage surfaced via WebSearch; direct WebFetch to platform.openai.com and developers.openai.com both returned HTTP 403 this session, so this claim is WebSearch-only, not primary-fetched). Its streaming shape differs structurally from Chat Completions: instead of untyped `chat.completion.chunk` deltas ended by a `[DONE]` sentinel, each SSE event carries a distinct `type` field, closer to Anthropic's scheme — `response.created`, `response.output_item.added`, `response.output_text.delta` (`delta` field for text fragments), `response.output_text.done`, `response.output_item.done`, and `response.completed` (or `response.failed`/`error`). Tool-call arguments stream as `response.function_call_arguments.delta` events keyed by `output_index`, accumulated the same way as Chat Completions' `tool_calls.function.arguments` fragments, then finalized on `response.function_call_arguments.done` or `response.completed`.\n\n## Anthropic Messages streaming\n\nSet `\"stream\": true` in the request body to `/v1/messages`. Events use both SSE `event:` name fields and a `type` field inside the JSON `data:` payload. The ordered event flow is:\n\n1. `message_start` — contains a `Message` object with empty `content`.\n2. For each content block: `content_block_start` → one or more `content_block_delta` events → `content_block_stop`. Each block has an `index` matching its position in the final message.\n3. One or more `message_delta` events — top-level message metadata updates (e.g., cumulative `usage` token counts).\n4. `message_stop` — stream is complete.\n\nAdditional `ping` events may appear anywhere. Error events can arrive mid-stream (e.g., `overloaded_error`); consumers must handle unknown event types gracefully.\n\n**Delta types inside `content_block_delta`:**\n\n- `text_delta` — `delta.text` carries a text fragment.\n- `input_json_delta` — `delta.partial_json` carries a partial JSON string fragment for a `tool_use` block's `input` field. Accumulate fragments across deltas and parse the complete string at `content_block_stop`. Current models emit one complete key-value pair per emission, so gaps between events are normal.\n- `thinking_delta` — reasoning tokens when extended thinking is enabled.\n\n## Google Gemini streaming\n\nUse `streamGenerateContent` instead of `generateContent`. With the REST API add `?alt=sse` to receive SSE-formatted output. Each SSE `data:` event carries a complete `GenerateContentResponse` JSON object; incremental text arrives in `candidates[0].content.parts[0].text`. There is no separate `[DONE]` sentinel — the stream ends when the HTTP response body closes. Function-call arguments in streaming follow the same accumulate-then-parse pattern as other providers.\n\n## Streaming tool calls and structured outputs\n\nRegardless of provider, function-call arguments arrive as **partial JSON string fragments**. Two handling strategies:\n\n- **Accumulate-then-parse** (simplest): collect all fragments into a buffer; parse the complete JSON string once the block or stream terminates. Safe for all schema shapes.\n- **Streaming/partial JSON parser**: libraries such as `partial-json` (npm) or Pydantic's partial JSON parsing mode can deserialize incomplete JSON incrementally, enabling early field access before the stream ends. Useful for long structured outputs where upstream steps can act on early fields.\n\nFor validation and schema-enforcement concerns once the full arguments are available, see /resources/reliable-tool-calling.\n\n## Practical concerns for agent builders\n\n**Backpressure and buffering** — if your consumer processes chunks slower than the provider emits them, buffers grow. Size-bound your buffer and apply flow control; for gateway deployments see /resources/ai-gateways-llm-routing.\n\n**Cancellation / abort** — send an HTTP request abort (e.g., `AbortController` in browser or Node.js, `httpx` cancel in Python) to stop generation early. The provider stops decoding; you pay only for tokens generated up to the abort. Ensure your agent loop handles a partial-response state cleanly.\n\n**Error handling mid-stream** — an error event or a dropped TCP connection mid-stream leaves your state machine with a partially assembled response. Track which content blocks received `content_block_stop` (Anthropic) or whether `finish_reason` was set (OpenAI) before treating the response as complete.\n\n**Token accounting** — `usage` fields in streaming responses (OpenAI `stream_options: {\"include_usage\": true}`; Anthropic `message_delta.usage`) are cumulative, not per-chunk. Read the final value, not a running sum of chunk values.\n\n**Proxying through a gateway** — if you proxy streamed responses through an AI gateway or middleware, ensure the proxy flushes `data:` lines immediately rather than buffering the full response body. A buffering proxy negates all TTFT benefits. See /resources/ai-gateways-llm-routing for gateway selection criteria.\n\n## Verified sources\n\n- WHATWG HTML Living Standard — Server-sent events: https://html.spec.whatwg.org/multipage/server-sent-events.html\n- MDN Web Docs — Using server-sent events: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events\n- Anthropic Messages streaming (event types, input_json_delta, tool use): https://platform.claude.com/docs/en/build-with-claude/streaming\n- OpenAI ChatCompletionChunk type (delta fields, tool_calls.function.arguments; re-fetched live 2026-07-18 via raw.githubusercontent.com, confirmed current): https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_chunk.py\n- OpenAI Responses API streaming events (WebSearch-fallback only, 4+ agreeing sources — developers.openai.com and platform.openai.com doc-page titles, the OpenAI community forum's \"simple guide to events\" thread, and vLLM's OpenAI-compatible streaming_events implementation; platform.openai.com/developers.openai.com both 403'd to direct WebFetch this session): https://platform.openai.com/docs/api-reference/responses-streaming/response/output_text/delta\n- Google Gemini streaming (streamGenerateContent, GenerateContentResponse, alt=sse): https://ai.google.dev/api/generate-content\n- Google Gemini cookbook — Streaming REST quickstart: https://github.com/google-gemini/cookbook/blob/main/quickstarts/rest/Streaming_REST.ipynb",
  "sources": [
    "https://html.spec.whatwg.org/multipage/server-sent-events.html",
    "https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events",
    "https://platform.claude.com/docs/en/build-with-claude/streaming",
    "https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_chunk.py",
    "https://platform.openai.com/docs/api-reference/responses-streaming/response/output_text/delta",
    "https://ai.google.dev/api/generate-content",
    "https://github.com/google-gemini/cookbook/blob/main/quickstarts/rest/Streaming_REST.ipynb"
  ]
}