Semantic Cache

Traditional caching stores responses keyed by exact input matches: the exact same HTTP request or SQL query returns the cached result. This works well for deterministic systems with predictable repeated queries. LLM applications have a different problem: the same question can be asked in thousands of different ways, all deserving the same answer. "What is the capital of France?" and "Tell me the capital city of France" are semantically identical but textually different - a traditional cache would miss the second query and call the LLM again.

Semantic caching solves this by caching on semantic meaning rather than exact text. When a query arrives, it is embedded into a vector representation. The vector store is searched for similar cached queries. If a sufficiently similar cached query exists (similarity above a configured threshold), the cached response is returned without calling the LLM. If no similar query exists, the LLM is called, the response is returned, and both the query embedding and the response are cached for future use.

The implementation depends on a vector database (typically the same infrastructure used for RAG retrieval) and an embedding model that produces consistent representations of semantic meaning. The similarity threshold is a critical configuration: too high, and only near-identical queries hit the cache; too low, and semantically different queries incorrectly share responses.

Semantic caching is particularly valuable for several LLM application patterns. Customer support chatbots receive many variations of the same common questions - FAQs, product policies, troubleshooting steps. Code documentation tools receive similar queries from many developers working on the same codebase. Internal knowledge base systems have predictable query distributions centred on the organisation's common topics. In these applications, cache hit rates of 30-60% are achievable, with proportional reductions in LLM API costs and response latency.

Implementation considerations include cache invalidation (when source content changes, cached responses based on that content should be invalidated), personalisation (responses that depend on user identity should not be shared across users), and freshness (time-sensitive responses should have TTLs). GPTCache and LangChain's caching integrations provide ready-made semantic caching layers for common LLM frameworks.

For very high-traffic production applications, semantic caching can provide significant cost savings: LLM API calls at frontier model prices cost orders of magnitude more than cached vector similarity lookups, so even a 20% cache hit rate on a high-volume application can meaningfully reduce operational costs.