{
  "slug": "handling-rate-limits-and-retries",
  "title": "Handling LLM Rate Limits (HTTP 429) and Retries for Agents",
  "description": "A practical reference for agent builders: what a 429 means, how to read provider rate-limit headers, exponential backoff with jitter, client-side throttling, and when to use a batch API.",
  "category": "Guide",
  "tags": [
    "rate-limits",
    "http-429",
    "retry",
    "backoff",
    "throttling",
    "openai",
    "anthropic",
    "agents",
    "reliability"
  ],
  "updated": "2026-07-15",
  "canonical": "https://changegamer.ai/resources/handling-rate-limits-and-retries",
  "markdown": "https://changegamer.ai/resources/handling-rate-limits-and-retries.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "What the rate-limit headers tell you",
      "anchor": "what-the-rate-limit-headers-tell-you"
    },
    {
      "depth": 2,
      "text": "RPM, TPM, and TPD — what each limit covers",
      "anchor": "rpm-tpm-and-tpd-what-each-limit-covers"
    },
    {
      "depth": 2,
      "text": "Usage tiers and how limits advance",
      "anchor": "usage-tiers-and-how-limits-advance"
    },
    {
      "depth": 2,
      "text": "Exponential backoff with full jitter",
      "anchor": "exponential-backoff-with-full-jitter"
    },
    {
      "depth": 2,
      "text": "Throttling client-side before you hit 429",
      "anchor": "throttling-client-side-before-you-hit-429"
    },
    {
      "depth": 2,
      "text": "When to use a batch API instead",
      "anchor": "when-to-use-a-batch-api-instead"
    },
    {
      "depth": 2,
      "text": "Provider fallback via a gateway",
      "anchor": "provider-fallback-via-a-gateway"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "computer-use-browser-automation",
      "title": "Computer Use and Browser Automation for Agents",
      "description": "Two-layer reference: vendor computer-use APIs (Anthropic, OpenAI CUA, Google Gemini) that translate screenshots to actions, and the open harnesses (Playwright MCP, browser-use, Stagehand, Skyvern) that execute those actions — with loop mechanics, reliability tradeoffs, and security gates.",
      "url": "https://changegamer.ai/resources/computer-use-browser-automation"
    },
    {
      "slug": "prompt-caching-for-agents",
      "title": "Prompt Caching for AI Agents",
      "description": "Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.",
      "url": "https://changegamer.ai/resources/prompt-caching-for-agents"
    },
    {
      "slug": "streaming-for-agents",
      "title": "Streaming Responses for Agents",
      "description": "Transport formats, provider event schemas, and practical concerns for consuming streamed LLM responses in production agents: SSE mechanics, OpenAI (Chat Completions and Responses API) and Anthropic event formats, partial-JSON tool-call parsing, backpressure, cancellation, and gateway proxying.",
      "url": "https://changegamer.ai/resources/streaming-for-agents"
    },
    {
      "slug": "durable-execution-for-agents",
      "title": "Durable Execution for Long-Running Agents",
      "description": "Vendor-neutral reference on durable execution: event logs, replay determinism, idempotency, retries, and human-in-the-loop pause/resume — plus a cross-vendor survey and tradeoffs guide for Temporal, Restate, DBOS, Inngest, Step Functions, Azure Durable Functions, Cloudflare Workflows, GCP Workflows, LangGraph, and OpenAI Agents SDK.",
      "url": "https://changegamer.ai/resources/durable-execution-for-agents"
    }
  ],
  "body": "When an LLM provider returns HTTP 429 (Too Many Requests), the calling agent has exceeded one or more rate limits for the current window. **The first thing to do: read the `Retry-After` header (or its provider-specific equivalent) and wait that many seconds before retrying. Do not retry immediately.** Most providers also return per-dimension headers on every successful response, letting you throttle proactively rather than reactively.\n\n## Key facts\n\n- On a 429, the immediate move is to honor the `Retry-After` (or equivalent) header instead of retrying right away, and providers also expose per-dimension headroom on every success so agents can throttle before ever hitting a limit.\n- Limits actually stack across at least three windows — raw request-per-minute counts, combined or separately tracked input/output token-per-minute budgets, and a daily ceiling that overrides any remaining per-minute headroom.\n- Both providers loosen these ceilings as usage and spend grow, through numbered tiers that advance on account age/cumulative spend or purchased credit — but exact thresholds shift over time, so the practical move is to check the dashboard and request headroom ahead of need rather than react to production errors.\n- Randomizing retry delay across the full range up to an exponentially growing cap — while never going below whatever wait time the provider mandated — keeps a fleet of retrying clients from all hammering the API at once.\n- Before ever reaching a 429, agents can self-throttle by tracking a local budget from the remaining-headroom headers, capping concurrent in-flight calls, or pre-estimating token cost to hold back requests that would blow the budget.\n- Because it draws from a separate quota pool and runs at half the per-token price with same-day turnaround, the batch API suits offline, non-interactive workloads like evals or bulk summarization rather than latency-sensitive agent calls.\n\n## What the rate-limit headers tell you\n\n| Header | OpenAI | Anthropic |\n|---|---|---|\n| Requests remaining | `x-ratelimit-remaining-requests` | `anthropic-ratelimit-requests-remaining` |\n| Tokens remaining | `x-ratelimit-remaining-tokens` | `anthropic-ratelimit-tokens-remaining` |\n| Input/output tokens remaining | — | `anthropic-ratelimit-input-tokens-remaining` / `...-output-tokens-remaining` |\n| Limit (requests / tokens) | `x-ratelimit-limit-requests` / `...-tokens` | `anthropic-ratelimit-requests-limit` / `...-tokens-limit` |\n| Reset time | `x-ratelimit-reset-requests` / `...-tokens` | `anthropic-ratelimit-requests-reset` / `...-tokens-reset` (RFC 3339) |\n| Wait before retry | `Retry-After` (on 429) | `retry-after` (on 429) |\n\nGoogle Gemini exposes `x-ratelimit-remaining` and `retry-after` on the same pattern. Anthropic tracks input and output tokens in separate ITPM and OTPM limits; cached input tokens (cache reads) generally do not count toward ITPM, so prompt caching raises effective throughput without a tier change (see /resources/prompt-caching-for-agents).\n\n## RPM, TPM, and TPD — what each limit covers\n\n**RPM (requests per minute)** — total HTTP calls regardless of payload size; high-frequency, low-token bursts hit this first. The window is typically a rolling 60 seconds.\n\n**TPM (tokens per minute)** — input + output tokens per rolling minute. Long contexts or large system prompts burn TPM quickly even at low RPM. Anthropic enforces separate ITPM and OTPM; OpenAI combines them.\n\n**TPD / RPD (per day)** — a daily ceiling across all per-minute windows; hitting it means waiting for the reset regardless of per-minute headroom. Often the binding constraint for lower tiers running overnight workloads.\n\n## Usage tiers and how limits advance\n\nBoth major providers raise your limits as you spend more. OpenAI starts every new project on a Free usage tier (low limits, some models unsupported) and uses numbered usage tiers (Tier 1–5 above it) that advance automatically as cumulative spend and account age cross thresholds; Anthropic uses numbered tiers gated by cumulative credit purchase plus a monthly-invoicing tier with negotiated limits. The exact spend thresholds and per-model RPM/TPM/ITPM numbers change over time and per model — read them from your provider dashboard or the official rate-limit docs rather than hard-coding them. The practical rule at every tier: design for the limit you have today, and request an increase before you need it, not after 429s appear in production.\n\n## Exponential backoff with full jitter\n\nFull jitter — randomizing the delay uniformly between zero and the exponential ceiling — spreads retrying agents across time and prevents synchronized retry storms. Always enforce the provider's `Retry-After` value as a hard floor.\n\n```\nMAX_RETRIES = 7\nBASE_DELAY  = 1      # seconds\nCAP_DELAY   = 60     # seconds\n\ndef call_with_backoff(request):\n    for attempt in range(MAX_RETRIES):\n        response = send(request)\n        if response.status != 429:\n            return response\n        floor = int(response.headers.get(\"Retry-After\", 0))   # provider-mandated floor\n        ceiling = min(CAP_DELAY, BASE_DELAY * 2 ** attempt)\n        wait = max(floor, random_uniform(0, ceiling))         # full jitter\n        sleep(wait)\n    raise RateLimitExceeded(\"max retries reached\")\n```\n\nThe ceiling grows exponentially (1s, 2s, 4s, 8s…) but is capped; the `Retry-After` floor ensures you never retry before the provider permits. Source: aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/.\n\n## Throttling client-side before you hit 429\n\n- **Token bucket** — maintain a local token count from the `*-remaining` headers; subtract estimated cost before each request and sleep until the reset timestamp if it would go negative.\n- **Concurrency cap** — bound in-flight requests to a fraction of RPM with a semaphore or async queue.\n- **Estimated token pre-check** — tokenize the prompt before sending and queue requests that would exceed remaining headroom.\n\n## When to use a batch API instead\n\nFor non-urgent, non-interactive work, both major providers offer a batch API that runs on a separate rate-limit pool (so it does not consume your standard RPM/TPM) and costs **50% less per token**, with results delivered within 24 hours:\n\n- **OpenAI Batch API** — submit a JSONL file of many requests; 50% discount; separate pool. Source: platform.openai.com/docs/guides/batch.\n- **Anthropic Message Batches API** — submit up to 100,000 requests per batch; 50% off input and output. Source: platform.claude.com/docs/en/api/rate-limits.\n\nUse a batch API for evals, embedding large corpora, offline summarization, and synthetic-data generation — not for interactive agents that need a result in seconds.\n\n## Provider fallback via a gateway\n\nWhen a provider is at capacity, an LLM gateway can route to a secondary provider instead of retrying the same endpoint — useful for daily-limit exhaustion or regional outages. See /resources/ai-gateways-llm-routing. For workflow-level retries and idempotency, see /resources/durable-execution-for-agents; for malformed-output retries, see /resources/reliable-tool-calling.\n\n## Verified sources\n\n- OpenAI rate limits guide: https://platform.openai.com/docs/guides/rate-limits\n- OpenAI Batch API guide: https://platform.openai.com/docs/guides/batch\n- Anthropic rate limits reference: https://platform.claude.com/docs/en/api/rate-limits\n- AWS: Exponential Backoff and Jitter: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/",
  "sources": [
    "https://platform.openai.com/docs/guides/rate-limits",
    "https://platform.openai.com/docs/guides/batch",
    "https://platform.claude.com/docs/en/api/rate-limits",
    "https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/"
  ]
}