Context window

How it works

The context window is the maximum amount of text, measured in tokens, that a language model can consider at once, spanning the system prompt, the conversation history, any retrieved documents, and the model's own response. A token is roughly three-quarters of a word. If the total exceeds the window, something has to be dropped or summarized, because the model physically cannot attend to text outside it. Modern models range from a few thousand tokens to hundreds of thousands or more.

When it matters

The context window matters for anything that stuffs a lot of text into a prompt: long documents, big RAG result sets, long agent conversations, or many tool outputs. It sets a hard ceiling on how much grounding you can supply per call and drives cost, since most providers bill per token. Larger windows help, but relevant retrieval usually beats brute-forcing everything into context, both for quality and for price.

Common misconceptions

• "Bigger context always means better answers." Models can lose track of information in the middle of a very long context; curation still matters.
• "A token is a word." A token is a sub-word unit, roughly 0.75 words on average in English.
• "The window is just the input." It also has to hold the output, so leave room for the response.