{
  "slug": "agent-response-caching",
  "title": "Application-Level Response Caching for AI Agents",
  "description": "How to implement exact-match and semantic caching in your agent application to eliminate redundant LLM calls, with threshold guidance, invalidation strategies, and a decision matrix for when semantic caching is unsafe.",
  "category": "Guide",
  "tags": [
    "caching",
    "semantic-cache",
    "cost",
    "latency",
    "optimization",
    "agents",
    "embeddings"
  ],
  "updated": "2026-06-25",
  "canonical": "https://changegamer.ai/resources/agent-response-caching",
  "markdown": "https://changegamer.ai/resources/agent-response-caching.md",
  "outline": [
    {
      "depth": 2,
      "text": "Exact-match caching",
      "anchor": "exact-match-caching"
    },
    {
      "depth": 2,
      "text": "Semantic caching",
      "anchor": "semantic-caching"
    },
    {
      "depth": 2,
      "text": "Cache type comparison",
      "anchor": "cache-type-comparison"
    },
    {
      "depth": 2,
      "text": "Similarity threshold tuning",
      "anchor": "similarity-threshold-tuning"
    },
    {
      "depth": 2,
      "text": "When NOT to semantic-cache",
      "anchor": "when-not-to-semantic-cache"
    },
    {
      "depth": 2,
      "text": "Invalidation and staleness",
      "anchor": "invalidation-and-staleness"
    },
    {
      "depth": 2,
      "text": "Tooling",
      "anchor": "tooling"
    },
    {
      "depth": 2,
      "text": "Relationship to other caching layers",
      "anchor": "relationship-to-other-caching-layers"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "agent-cost-latency-optimization",
      "title": "Agent Cost and Latency Optimization",
      "description": "Practitioner reference for reducing the cost and latency of production AI agents: the compounding model, token-level levers (caching, pruning), request-level levers (Batch API, parallelism), model-level levers (routing, reasoning-effort controls), and architecture-level levers (step reduction, semantic caching, code offloading).",
      "url": "https://changegamer.ai/resources/agent-cost-latency-optimization"
    },
    {
      "slug": "prompt-caching-for-agents",
      "title": "Prompt Caching for AI Agents",
      "description": "Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.",
      "url": "https://changegamer.ai/resources/prompt-caching-for-agents"
    },
    {
      "slug": "choosing-an-llm-for-agents",
      "title": "How to Choose an LLM for Agentic Tasks",
      "description": "A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.",
      "url": "https://changegamer.ai/resources/choosing-an-llm-for-agents"
    },
    {
      "slug": "chunking-strategies-for-rag",
      "title": "Chunking Strategies for RAG",
      "description": "Practitioner reference for chunking documents before embedding: fixed-size, recursive, semantic, late chunking, and contextual retrieval — with a strategy comparison table, chunk-size and overlap tradeoffs, code/table/Markdown handling, embedding model context limits, and evaluation methods.",
      "url": "https://changegamer.ai/resources/chunking-strategies-for-rag"
    }
  ],
  "body": "Application-level response caching intercepts a request **before it reaches the model provider** and returns a stored response — eliminating the inference call entirely. This is distinct from provider-side prefix caching (see /resources/prompt-caching-for-agents), which only discounts repeated input tokens while still running inference. Application caching is code you own and deploy; it can cut cost and latency to near-zero for cache-eligible queries.\n\n## Exact-match caching\n\nHash the full request (model + messages + temperature + parameters) with a deterministic function such as SHA-256, and store the response under that key. A subsequent identical request hits the cache without a model call. Lookup is O(1) (~1–3 ms). False-positive risk: effectively zero. False-negative risk: any character change is a miss. Useful when your agent issues identical sub-task calls across sessions (e.g., the same review prompt on the same unchanged file).\n\n## Semantic caching\n\nEmbed the incoming query, run a vector similarity search against previously cached query embeddings, and if the nearest neighbour exceeds a cosine-similarity threshold, return the cached response. A miss calls the model and stores the response plus its embedding. Lookup is roughly single-digit milliseconds versus hundreds-to-thousands for a live call. For the embedding layer see /resources/embeddings-vector-search.\n\n## Cache type comparison\n\n| Cache type | Mechanism | Eliminates the call? | False-positive risk |\n|---|---|---|---|\n| **Exact-match** | SHA-256 hash of the full request | Yes, for identical requests | None |\n| **Semantic** | Embed query → similarity search → return if above threshold | Yes, for near-duplicate queries | Low–medium (threshold-dependent) |\n| **Provider prefix caching** | Provider reuses KV state on shared input prefix | No — input-token discount only | None (provider-managed) |\n\n## Similarity threshold tuning\n\nThe threshold is the single most consequential parameter. Start high — around **0.95 cosine similarity** — and only lower it after backtesting on thousands of real queries while precision stays high. A low threshold (e.g. 0.85) will serve answers about one topic to questions about a superficially similar but different one. Domain-specific terminology needs higher thresholds than general FAQ content. Example: \"Show DDA revenue by channel\" and \"Show GA4 revenue by channel\" can score ~0.96 yet reference different attribution models — a naive threshold would serve the wrong answer. Source: portkey.ai/blog/semantic-caching-thresholds/.\n\n## When NOT to semantic-cache\n\n- **Personalized responses** — answers that depend on user identity, session state, or account data. Serving user A's answer to user B is a correctness bug and a possible privacy violation.\n- **Temporal / real-time queries** — \"current price?\", \"in stock?\", \"what happened today?\" — a hit returns a stale answer.\n- **Tool-calling / action-taking agents** — \"check my email\" and \"send an email\" can score above 0.90 while requiring opposite actions. Caching action responses is unsafe without intent canonicalization.\n- **Multi-step trajectories with state dependencies** — a cached step computed against a now-changed world makes every downstream step act on stale premises.\n- **Safety-critical outputs** — medical, legal, financial. A wrong cached answer propagates indefinitely rather than decaying.\n\n## Invalidation and staleness\n\n- **TTL** bounds staleness but does not guarantee correctness. Use tiered TTLs: short (minutes) for volatile facts, longer (hours–days) for stable reference content; add jitter to avoid expiry storms. A hallucinated response cached before you detect it is served on every match until the TTL expires — build a force-invalidation-by-key path.\n- **Event-driven invalidation** — when underlying data changes, proactively invalidate affected entries.\n- **Version keys** — append a content version token to the cache key so document updates bust the cache automatically.\n\n## Tooling\n\n- **GPTCache** (github.com/zilliztech/GPTCache, MIT) — a popular open-source Python library for semantic LLM response caching; pluggable embedding models, vector stores, and cache backends; integrates with LangChain and LlamaIndex.\n- **Redis + vector search** — a two-layer pattern: exact-match via hash key, then semantic via a Redis vector index. LangChain ships `RedisSemanticCache` as a drop-in backend.\n- **LangChain built-in caches** — `InMemoryCache` and `SQLiteCache` (exact-match) for dev/low-volume; `RedisSemanticCache` for semantic. Set via `set_llm_cache()`.\n- **Gateway-level semantic caching** — Portkey and the LiteLLM proxy implement semantic caching at the gateway, applying it across providers with no app-level embedding code. Cloudflare AI Gateway supports exact-match caching today; semantic caching is listed as planned. See /resources/ai-gateways-llm-routing.\n\n## Relationship to other caching layers\n\nThese layers stack and are complementary: (1) application exact-match cache, (2) application semantic cache, (3) provider prefix caching (/resources/prompt-caching-for-agents — discounts input tokens; inference still runs). For the broader cost stack see /resources/agent-cost-latency-optimization.\n\n## Verified sources\n\n- GPTCache GitHub (Zilliz, MIT): https://github.com/zilliztech/GPTCache\n- LangChain RedisSemanticCache reference: https://reference.langchain.com/python/langchain-redis/cache/RedisSemanticCache\n- Portkey semantic caching docs: https://portkey.ai/docs/product/ai-gateway/cache-simple-and-semantic\n- Portkey semantic-caching thresholds: https://portkey.ai/blog/semantic-caching-thresholds/\n- Cloudflare AI Gateway caching docs (exact-match; semantic planned): https://developers.cloudflare.com/ai-gateway/features/caching/",
  "sources": [
    "https://github.com/zilliztech/GPTCache",
    "https://reference.langchain.com/python/langchain-redis/cache/RedisSemanticCache",
    "https://portkey.ai/docs/product/ai-gateway/cache-simple-and-semantic",
    "https://portkey.ai/blog/semantic-caching-thresholds/",
    "https://developers.cloudflare.com/ai-gateway/features/caching/"
  ]
}