The context window: the model's short-term memory, and its hard limit
Everything a model 'knows' in a request has to fit in its context window. Understanding that budget — what fills it, what it costs, where it fails — is core to building with LLMs.
A language model has no memory between calls. Everything it can use to answer — the instructions, the retrieved documents, the conversation so far, your question — has to be packed into a single request. The context window is the size of that container, measured in tokens, and it shapes almost every design decision you'll make.
What fills the window
The window isn't just your question. It's the system prompt, tool definitions, any retrieved context (RAG), the running chat history, and the user's latest message — and it must also leave room for the model's response. Input and output share the same budget.
Bigger isn't automatically better
- Cost and latency — you pay for every token in the window, so stuffing it full of 'just in case' context is a tax on every call.
- Lost in the middle — models attend best to the start and end of a long context; facts buried in the middle can be effectively ignored.
- Relevance beats volume — a few well-chosen tokens often beat a giant dump, which is why good retrieval matters more than a bigger window.
Managing the budget
Long conversations and big documents will outgrow any window, so you manage it: retrieve only what's relevant, summarise or truncate old turns, and treat the window as a scarce resource to curate rather than a bucket to fill. A larger window buys you room, not a reason to stop being deliberate.
The context window is the model's entire world for one request. What you leave out matters as much as what you put in.