Application-Level Response Caching for AI Agents
How to implement exact-match and semantic caching in your agent application to eliminate redundant LLM calls, with threshold guidance, invalidation strategies, and a decision matrix for when semantic caching is unsafe.
Application-level response caching intercepts a request before it reaches the model provider and returns a stored response — eliminating the inference call entirely. This is distinct from provider-side prefix caching (see /resources/prompt-caching-for-agents), which only discounts repeated input tokens while still running inference. Application caching is code you own and deploy; it can cut cost and latency to near-zero for cache-eligible queries.
Exact-match caching
Hash the full request (model + messages + temperature + parameters) with a deterministic function such as SHA-256, and store the response under that key. A subsequent identical request hits the cache without a model call. Lookup is O(1) (~1–3 ms). False-positive risk: effectively zero. False-negative risk: any character change is a miss. Useful when your agent issues identical sub-task calls across sessions (e.g., the same review prompt on the same unchanged file).
Semantic caching
Embed the incoming query, run a vector similarity search against previously cached query embeddings, and if the nearest neighbour exceeds a cosine-similarity threshold, return the cached response. A miss calls the model and stores the response plus its embedding. Lookup is roughly single-digit milliseconds versus hundreds-to-thousands for a live call. For the embedding layer see /resources/embeddings-vector-search.
Cache type comparison
| Cache type | Mechanism | Eliminates the call? | False-positive risk |
|---|---|---|---|
| Exact-match | SHA-256 hash of the full request | Yes, for identical requests | None |
| Semantic | Embed query → similarity search → return if above threshold | Yes, for near-duplicate queries | Low–medium (threshold-dependent) |
| Provider prefix caching | Provider reuses KV state on shared input prefix | No — input-token discount only | None (provider-managed) |
Similarity threshold tuning
The threshold is the single most consequential parameter. Start high — around 0.95 cosine similarity — and only lower it after backtesting on thousands of real queries while precision stays high. A low threshold (e.g. 0.85) will serve answers about one topic to questions about a superficially similar but different one. Domain-specific terminology needs higher thresholds than general FAQ content. Example: "Show DDA revenue by channel" and "Show GA4 revenue by channel" can score ~0.96 yet reference different attribution models — a naive threshold would serve the wrong answer. Source: portkey.ai/blog/semantic-caching-thresholds/.
When NOT to semantic-cache
- Personalized responses — answers that depend on user identity, session state, or account data. Serving user A's answer to user B is a correctness bug and a possible privacy violation.
- Temporal / real-time queries — "current price?", "in stock?", "what happened today?" — a hit returns a stale answer.
- Tool-calling / action-taking agents — "check my email" and "send an email" can score above 0.90 while requiring opposite actions. Caching action responses is unsafe without intent canonicalization.
- Multi-step trajectories with state dependencies — a cached step computed against a now-changed world makes every downstream step act on stale premises.
- Safety-critical outputs — medical, legal, financial. A wrong cached answer propagates indefinitely rather than decaying.
Invalidation and staleness
- TTL bounds staleness but does not guarantee correctness. Use tiered TTLs: short (minutes) for volatile facts, longer (hours–days) for stable reference content; add jitter to avoid expiry storms. A hallucinated response cached before you detect it is served on every match until the TTL expires — build a force-invalidation-by-key path.
- Event-driven invalidation — when underlying data changes, proactively invalidate affected entries.
- Version keys — append a content version token to the cache key so document updates bust the cache automatically.
Tooling
- GPTCache (github.com/zilliztech/GPTCache, MIT) — a popular open-source Python library for semantic LLM response caching; pluggable embedding models, vector stores, and cache backends; integrates with LangChain and LlamaIndex.
- Redis + vector search — a two-layer pattern: exact-match via hash key, then semantic via a Redis vector index. LangChain ships
RedisSemanticCacheas a drop-in backend. - LangChain built-in caches —
InMemoryCacheandSQLiteCache(exact-match) for dev/low-volume;RedisSemanticCachefor semantic. Set viaset_llm_cache(). - Gateway-level semantic caching — Portkey and the LiteLLM proxy implement semantic caching at the gateway, applying it across providers with no app-level embedding code. Cloudflare AI Gateway supports exact-match caching today; semantic caching is listed as planned. See /resources/ai-gateways-llm-routing.
Relationship to other caching layers
These layers stack and are complementary: (1) application exact-match cache, (2) application semantic cache, (3) provider prefix caching (/resources/prompt-caching-for-agents — discounts input tokens; inference still runs). For the broader cost stack see /resources/agent-cost-latency-optimization.
Verified sources
- GPTCache GitHub (Zilliz, MIT): https://github.com/zilliztech/GPTCache
- LangChain RedisSemanticCache reference: https://reference.langchain.com/python/langchain-redis/cache/RedisSemanticCache
- Portkey semantic caching docs: https://portkey.ai/docs/product/ai-gateway/cache-simple-and-semantic
- Portkey semantic-caching thresholds: https://portkey.ai/blog/semantic-caching-thresholds/
- Cloudflare AI Gateway caching docs (exact-match; semantic planned): https://developers.cloudflare.com/ai-gateway/features/caching/