# Application-Level Response Caching for AI Agents

> How to implement exact-match and semantic caching in your agent application to eliminate redundant LLM calls, with threshold guidance, invalidation strategies, and a decision matrix for when semantic caching is unsafe.

Category: Guide · Updated: 2026-06-25 · Tags: caching, semantic-cache, cost, latency, optimization, agents, embeddings
Canonical: https://changegamer.ai/resources/agent-response-caching
Variants: [HTML](https://changegamer.ai/resources/agent-response-caching) · [JSON](https://changegamer.ai/api/resources/agent-response-caching.json)
License: https://changegamer.ai/license.xml · Access: free

Application-level response caching intercepts a request **before it reaches the model provider** and returns a stored response — eliminating the inference call entirely. This is distinct from provider-side prefix caching (see /resources/prompt-caching-for-agents), which only discounts repeated input tokens while still running inference. Application caching is code you own and deploy; it can cut cost and latency to near-zero for cache-eligible queries.

## Exact-match caching

Hash the full request (model + messages + temperature + parameters) with a deterministic function such as SHA-256, and store the response under that key. A subsequent identical request hits the cache without a model call. Lookup is O(1) (~1–3 ms). False-positive risk: effectively zero. False-negative risk: any character change is a miss. Useful when your agent issues identical sub-task calls across sessions (e.g., the same review prompt on the same unchanged file).

## Semantic caching

Embed the incoming query, run a vector similarity search against previously cached query embeddings, and if the nearest neighbour exceeds a cosine-similarity threshold, return the cached response. A miss calls the model and stores the response plus its embedding. Lookup is roughly single-digit milliseconds versus hundreds-to-thousands for a live call. For the embedding layer see /resources/embeddings-vector-search.

## Cache type comparison

| Cache type | Mechanism | Eliminates the call? | False-positive risk |
|---|---|---|---|
| **Exact-match** | SHA-256 hash of the full request | Yes, for identical requests | None |
| **Semantic** | Embed query → similarity search → return if above threshold | Yes, for near-duplicate queries | Low–medium (threshold-dependent) |
| **Provider prefix caching** | Provider reuses KV state on shared input prefix | No — input-token discount only | None (provider-managed) |

## Similarity threshold tuning

The threshold is the single most consequential parameter. Start high — around **0.95 cosine similarity** — and only lower it after backtesting on thousands of real queries while precision stays high. A low threshold (e.g. 0.85) will serve answers about one topic to questions about a superficially similar but different one. Domain-specific terminology needs higher thresholds than general FAQ content. Example: "Show DDA revenue by channel" and "Show GA4 revenue by channel" can score ~0.96 yet reference different attribution models — a naive threshold would serve the wrong answer. Source: portkey.ai/blog/semantic-caching-thresholds/.

## When NOT to semantic-cache

- **Personalized responses** — answers that depend on user identity, session state, or account data. Serving user A's answer to user B is a correctness bug and a possible privacy violation.
- **Temporal / real-time queries** — "current price?", "in stock?", "what happened today?" — a hit returns a stale answer.
- **Tool-calling / action-taking agents** — "check my email" and "send an email" can score above 0.90 while requiring opposite actions. Caching action responses is unsafe without intent canonicalization.
- **Multi-step trajectories with state dependencies** — a cached step computed against a now-changed world makes every downstream step act on stale premises.
- **Safety-critical outputs** — medical, legal, financial. A wrong cached answer propagates indefinitely rather than decaying.

## Invalidation and staleness

- **TTL** bounds staleness but does not guarantee correctness. Use tiered TTLs: short (minutes) for volatile facts, longer (hours–days) for stable reference content; add jitter to avoid expiry storms. A hallucinated response cached before you detect it is served on every match until the TTL expires — build a force-invalidation-by-key path.
- **Event-driven invalidation** — when underlying data changes, proactively invalidate affected entries.
- **Version keys** — append a content version token to the cache key so document updates bust the cache automatically.

## Tooling

- **GPTCache** (github.com/zilliztech/GPTCache, MIT) — a popular open-source Python library for semantic LLM response caching; pluggable embedding models, vector stores, and cache backends; integrates with LangChain and LlamaIndex.
- **Redis + vector search** — a two-layer pattern: exact-match via hash key, then semantic via a Redis vector index. LangChain ships `RedisSemanticCache` as a drop-in backend.
- **LangChain built-in caches** — `InMemoryCache` and `SQLiteCache` (exact-match) for dev/low-volume; `RedisSemanticCache` for semantic. Set via `set_llm_cache()`.
- **Gateway-level semantic caching** — Portkey and the LiteLLM proxy implement semantic caching at the gateway, applying it across providers with no app-level embedding code. Cloudflare AI Gateway supports exact-match caching today; semantic caching is listed as planned. See /resources/ai-gateways-llm-routing.

## Relationship to other caching layers

These layers stack and are complementary: (1) application exact-match cache, (2) application semantic cache, (3) provider prefix caching (/resources/prompt-caching-for-agents — discounts input tokens; inference still runs). For the broader cost stack see /resources/agent-cost-latency-optimization.

## Verified sources

- GPTCache GitHub (Zilliz, MIT): https://github.com/zilliztech/GPTCache
- LangChain RedisSemanticCache reference: https://reference.langchain.com/python/langchain-redis/cache/RedisSemanticCache
- Portkey semantic caching docs: https://portkey.ai/docs/product/ai-gateway/cache-simple-and-semantic
- Portkey semantic-caching thresholds: https://portkey.ai/blog/semantic-caching-thresholds/
- Cloudflare AI Gateway caching docs (exact-match; semantic planned): https://developers.cloudflare.com/ai-gateway/features/caching/

---

## Related resources

- [Agent Cost and Latency Optimization](https://changegamer.ai/resources/agent-cost-latency-optimization.md): Practitioner reference for reducing the cost and latency of production AI agents: the compounding model, token-level levers (caching, pruning), request-level levers (Batch API, parallelism), model-level levers (routing, reasoning-effort controls), and architecture-level levers (step reduction, semantic caching, code offloading).
- [Prompt Caching for AI Agents](https://changegamer.ai/resources/prompt-caching-for-agents.md): Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.
- [How to Choose an LLM for Agentic Tasks](https://changegamer.ai/resources/choosing-an-llm-for-agents.md): A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.
- [Chunking Strategies for RAG](https://changegamer.ai/resources/chunking-strategies-for-rag.md): Practitioner reference for chunking documents before embedding: fixed-size, recursive, semantic, late chunking, and contextual retrieval — with a strategy comparison table, chunk-size and overlap tradeoffs, code/table/Markdown handling, embedding model context limits, and evaluation methods.

---

Index of all resources: https://changegamer.ai/llms.txt · Full corpus: https://changegamer.ai/llms-full.txt · Corpus data (NDJSON): https://changegamer.ai/api/corpus.jsonl · Offers: https://changegamer.ai/api/pricing.json