What is a context window?

The context window is the maximum number of tokens — prompt plus generated output — a language model can attend to at once. Everything the model "knows" about your request lives inside it; anything pushed out of the window is simply gone.

Why bigger is not free

Attention cost grows with context, and the KV cache grows linearly with it — a 128k window can need more memory for cache than the model's weights. Long-context models also show "lost in the middle" behaviour: recall is strong at the start and end of the window and weaker in the middle.

[ system prompt ][ retrieved docs ][ chat history ][ user turn ][ ← generation → ]
|------------------------ context window (e.g. 128k tokens) ----------------------|

Practical sizing

• 1 token ≈ 4 characters ≈ 0.75 English words.
• 8k is plenty for chat; 32k+ matters for long documents and codebases.
• For large corpora, retrieval (RAG) beats stuffing everything into a huge window — it is cheaper and more accurate.

The trade-off: a larger window buys convenience and fewer retrieval failures, at the cost of latency, money (you pay per token of context) and the recall dip in the middle.