KV-Cache

When a language model generates text, it processes the entire conversation from the beginning every time it produces a new token. In the raw, unoptimised version, generating a 500-token response would require 500 full passes through the model - each pass processing the prompt plus all previously generated tokens. This would be extraordinarily slow.

The KV-Cache is the solution. During the attention computation, every token in the sequence produces two sets of vectors: key vectors and value vectors. These are what other tokens attend to when computing their contextualised representations. Crucially, for tokens that have already been processed, these vectors never change - they depend only on the token itself and everything before it, not on what comes after. So rather than recomputing them on every generation step, the model simply stores them in a cache and reuses them.

With a KV-Cache in place, generating each new token only requires computing the keys and values for that one new token, then running attention between it and all the cached keys and values from previous tokens. Instead of scaling linearly with sequence length on every step, generation costs a roughly constant amount per new token.

The trade-off is memory. KV-Cache entries accumulate as the conversation grows, and for long contexts or large batch sizes they can consume enormous amounts of GPU memory. This is one of the primary engineering constraints in deploying large language models at scale - the attention computation and the KV-Cache are the two dominant consumers of GPU memory, and balancing them against each other is a major focus of inference optimisation work.

Techniques like grouped query attention (which shares KV caches across multiple attention heads rather than maintaining a separate cache per head) and paged attention (which manages KV cache memory using virtual paging, like an operating system manages RAM) were both invented specifically to make KV-Cache more memory-efficient at scale.