Context Window Management

Every language model has a context window limit - a maximum number of tokens it can process at once. For many real-world tasks, the content you want to work with exceeds this limit. A 100-page legal contract. A full codebase. A months-long conversation history. Context window management is the collection of techniques for handling these situations intelligently.

The simplest approach is truncation: just cut off text that exceeds the limit. This is often poor practice because the most important information might be in the truncated section. A smarter version is sliding window processing: break the long document into overlapping chunks, process each chunk with sufficient overlap that context carries across boundaries, then combine the results.

Summarisation-based approaches compress past context rather than discarding it. Instead of keeping the raw text of a long conversation, periodically summarise the earlier portion and replace it with the summary, freeing up context space for new content while preserving key information in compressed form. This is how many long-running AI assistants maintain apparent memory of earlier conversation despite fixed context windows.

Retrieval-based approaches (which connect to RAG systems) avoid the problem by not loading all content into context at once. Instead, they retrieve only the most relevant portions based on the current query. For a 1,000-page technical manual, you never load the whole thing - you retrieve the relevant sections based on what the user is asking.

Hierarchical methods process documents at multiple levels of granularity: scan a whole document at a high level to build an index, then load specific sections when their detail is needed. For structured documents like code or legal contracts, this mirrors how humans actually navigate long documents.