← All resources

Handling LLM Rate Limits (HTTP 429) and Retries for Agents

Guide · updated 2026-07-15 · Markdown variant

A practical reference for agent builders: what a 429 means, how to read provider rate-limit headers, exponential backoff with jitter, client-side throttling, and when to use a batch API.

When an LLM provider returns HTTP 429 (Too Many Requests), the calling agent has exceeded one or more rate limits for the current window. The first thing to do: read the Retry-After header (or its provider-specific equivalent) and wait that many seconds before retrying. Do not retry immediately. Most providers also return per-dimension headers on every successful response, letting you throttle proactively rather than reactively.

Key facts

• On a 429, the immediate move is to honor the Retry-After (or equivalent) header instead of retrying right away, and providers also expose per-dimension headroom on every success so agents can throttle before ever hitting a limit.
• Limits actually stack across at least three windows — raw request-per-minute counts, combined or separately tracked input/output token-per-minute budgets, and a daily ceiling that overrides any remaining per-minute headroom.
• Both providers loosen these ceilings as usage and spend grow, through numbered tiers that advance on account age/cumulative spend or purchased credit — but exact thresholds shift over time, so the practical move is to check the dashboard and request headroom ahead of need rather than react to production errors.
• Randomizing retry delay across the full range up to an exponentially growing cap — while never going below whatever wait time the provider mandated — keeps a fleet of retrying clients from all hammering the API at once.
• Before ever reaching a 429, agents can self-throttle by tracking a local budget from the remaining-headroom headers, capping concurrent in-flight calls, or pre-estimating token cost to hold back requests that would blow the budget.
• Because it draws from a separate quota pool and runs at half the per-token price with same-day turnaround, the batch API suits offline, non-interactive workloads like evals or bulk summarization rather than latency-sensitive agent calls.

What the rate-limit headers tell you

Google Gemini exposes x-ratelimit-remaining and retry-after on the same pattern. Anthropic tracks input and output tokens in separate ITPM and OTPM limits; cached input tokens (cache reads) generally do not count toward ITPM, so prompt caching raises effective throughput without a tier change (see /resources/prompt-caching-for-agents).

RPM, TPM, and TPD — what each limit covers

RPM (requests per minute) — total HTTP calls regardless of payload size; high-frequency, low-token bursts hit this first. The window is typically a rolling 60 seconds.

TPM (tokens per minute) — input + output tokens per rolling minute. Long contexts or large system prompts burn TPM quickly even at low RPM. Anthropic enforces separate ITPM and OTPM; OpenAI combines them.

TPD / RPD (per day) — a daily ceiling across all per-minute windows; hitting it means waiting for the reset regardless of per-minute headroom. Often the binding constraint for lower tiers running overnight workloads.

Usage tiers and how limits advance

Both major providers raise your limits as you spend more. OpenAI starts every new project on a Free usage tier (low limits, some models unsupported) and uses numbered usage tiers (Tier 1–5 above it) that advance automatically as cumulative spend and account age cross thresholds; Anthropic uses numbered tiers gated by cumulative credit purchase plus a monthly-invoicing tier with negotiated limits. The exact spend thresholds and per-model RPM/TPM/ITPM numbers change over time and per model — read them from your provider dashboard or the official rate-limit docs rather than hard-coding them. The practical rule at every tier: design for the limit you have today, and request an increase before you need it, not after 429s appear in production.

Exponential backoff with full jitter

Full jitter — randomizing the delay uniformly between zero and the exponential ceiling — spreads retrying agents across time and prevents synchronized retry storms. Always enforce the provider's Retry-After value as a hard floor.

MAX_RETRIES = 7
BASE_DELAY  = 1      # seconds
CAP_DELAY   = 60     # seconds

def call_with_backoff(request):
    for attempt in range(MAX_RETRIES):
        response = send(request)
        if response.status != 429:
            return response
        floor = int(response.headers.get("Retry-After", 0))   # provider-mandated floor
        ceiling = min(CAP_DELAY, BASE_DELAY * 2 ** attempt)
        wait = max(floor, random_uniform(0, ceiling))         # full jitter
        sleep(wait)
    raise RateLimitExceeded("max retries reached")

The ceiling grows exponentially (1s, 2s, 4s, 8s…) but is capped; the Retry-After floor ensures you never retry before the provider permits. Source: aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/.

Throttling client-side before you hit 429

• Token bucket — maintain a local token count from the *-remaining headers; subtract estimated cost before each request and sleep until the reset timestamp if it would go negative.
• Concurrency cap — bound in-flight requests to a fraction of RPM with a semaphore or async queue.
• Estimated token pre-check — tokenize the prompt before sending and queue requests that would exceed remaining headroom.

When to use a batch API instead

For non-urgent, non-interactive work, both major providers offer a batch API that runs on a separate rate-limit pool (so it does not consume your standard RPM/TPM) and costs 50% less per token, with results delivered within 24 hours:

• OpenAI Batch API — submit a JSONL file of many requests; 50% discount; separate pool. Source: platform.openai.com/docs/guides/batch.
• Anthropic Message Batches API — submit up to 100,000 requests per batch; 50% off input and output. Source: platform.claude.com/docs/en/api/rate-limits.

Use a batch API for evals, embedding large corpora, offline summarization, and synthetic-data generation — not for interactive agents that need a result in seconds.

Provider fallback via a gateway

When a provider is at capacity, an LLM gateway can route to a secondary provider instead of retrying the same endpoint — useful for daily-limit exhaustion or regional outages. See /resources/ai-gateways-llm-routing. For workflow-level retries and idempotency, see /resources/durable-execution-for-agents; for malformed-output retries, see /resources/reliable-tool-calling.

Verified sources

• OpenAI rate limits guide: https://platform.openai.com/docs/guides/rate-limits
• OpenAI Batch API guide: https://platform.openai.com/docs/guides/batch
• Anthropic rate limits reference: https://platform.claude.com/docs/en/api/rate-limits
• AWS: Exponential Backoff and Jitter: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

#rate-limits #http-429 #retry #backoff #throttling #openai #anthropic #agents #reliability

Category: Guide