Context Window Management: Budgets, Compaction and Caching
How to work within finite context windows: budget allocation across instructions, evidence and history; truncation versus summarization versus retrieval offload; prefix stability for prompt caching; and attention-quality caveats of very long contexts.
Every model reads a finite context window, and everything an agent needs — instructions, conversation history, retrieved evidence, tool results — competes for the same space. Context-window management is the discipline of allocating that space deliberately instead of discovering its limits through failures.
Allocate the budget by role
Treat the window as partitioned: stable instructions and tool definitions first (they change rarely), then working memory (task state, plan), then retrieved evidence, then recent history. Fixed allocations per tier make behavior predictable — when evidence grows, history shrinks on a rule rather than on vibes. Teams without explicit budgets get the classic failure mode: a long session quietly pushes the system prompt's constraints out of effective attention and the agent drifts.
Offloading beats squeezing
Three mechanisms move information out of the window without losing it:
- Retrieval: store knowledge outside the window and fetch relevant slices on demand — RAG is context-window management at corpus scale (see /resources/rag-retrieval-for-agents)
- Summarization and compaction: periodically compress older history into shorter state summaries; keep the summary faithful to decisions and open threads, not just transcripts
- Structured state files: persist task progress to an external store the agent reads and writes deliberately, rather than re-narrating it every turn (agent memory & context)
Truncation — dropping oldest messages — is the fallback, not the strategy: it silently amputates constraints and commitments made earlier in the session.
Long-context caveats
Very large windows invite just-include-everything designs. Two cautions temper that: attention quality across very long contexts degrades non-uniformly — models reliably use information at the beginning and end better than material buried mid-context — and cost scales with length whether or not buried content helps. Effective context engineering keeps relevant material close to where generation happens (prompt-context-engineering).
Prefix stability makes caching pay
Provider-level prompt caching charges reduced rates for repeated prefixes, which rewards one structural choice: put volatile content last. System prompt, tool schemas and standing instructions form the stable prefix; per-turn user content, retrieved evidence and history append after it. Reordering content between turns invalidates cache prefixes and multiplies cost — prompt caching for agents covers the mechanics, and response-level reuse (agent response caching) compounds savings for repeated questions.
Sizing inputs honestly
Estimate tokens before sending: English prose runs roughly four characters per token, so word counts convert to approximate token budgets with a ~1.33 multiplier. Leave headroom below the stated maximum for the output you expect back, since output tokens share the same window. Chunking strategy determines how much evidence a retrieval step must fetch for adequate coverage — tighter, better-targeted chunks mean fewer tokens per answered question (see /resources/chunking-strategies-for-rag).