What is a context window?

The context window is the maximum number of tokens — prompt plus generated output — a language model can attend to at once. Everything the model "knows" about your request lives inside it; anything pushed out of the window is simply gone.

Why bigger is not free

Attention cost grows with context, and the KV cache grows linearly with it — a 128k window can need more memory for cache than the model's weights. Long-context models also show "lost in the middle" behaviour: recall is strong at the start and end of the window and weaker in the middle.

[ system prompt ][ retrieved docs ][ chat history ][ user turn ][ ← generation → ]
|------------------------ context window (e.g. 128k tokens) ----------------------|

Practical sizing

• 1 token ≈ 4 characters ≈ 0.75 English words.
• 8k is plenty for chat; 32k+ matters for long documents and codebases.
• For large corpora, retrieval (RAG) beats stuffing everything into a huge window — it is cheaper and more accurate.

The trade-off: a larger window buys convenience and fewer retrieval failures, at the cost of latency, money (you pay per token of context) and the recall dip in the middle.

Sources

• Context windows - Claude Platform Docs (docs.anthropic.com) — "The 'context window' refers to all the text a language model can reference when generating a response, including the response itself."
• What is a context window? | IBM (ibm.com) — "The context window (or “context length”) of a large language model (LLM) is the amount of text, in tokens , that the model can consider or “remember” at any one time."