What is a context window in an LLM?

Every large language model has a limit on how much text it can take into account at one time. That limit is its context window. It's why a long conversation eventually starts to forget its beginning, why you can't paste an entire codebase into a single prompt, and why agentic tools like Claude Code sometimes pause to summarise what they've done so far.

What is a context window?

A context window is the maximum amount of text, measured in tokens, that a model can process in a single request. Everything the model uses to produce its answer has to fit inside it: the instructions, the conversation, any documents, and the answer itself.

Context windows vary widely between models, from tens of thousands of tokens to a million or more on some recent models. Since a token is about three quarters of an English word, even a modest window holds a lot of text, but long documents, big codebases and extended agent sessions fill them surprisingly fast.

What counts toward the context window

• The system prompt: instructions that set the model's role and rules.
• The conversation history: every earlier user message and model reply you send back.
• Documents, files and retrieved passages you include for reference.
• Tool definitions, and the results of any tools the model has called.
• The model's own output, including any reasoning or thinking tokens, which must fit in the space that's left.

The last point catches people out. If your input fills almost the whole window, there's little room left for the answer. Output also has its own separate cap, often called max tokens, which is usually much smaller than the context window.

Why models don't remember past requests

LLM APIs are stateless: the model keeps nothing between calls. A chat only feels continuous because the application resends the whole conversation with every new message. That's why a conversation's context grows with each turn, and why a long chat eventually hits the window's limit.

It also means every turn re-sends, and pays for, all the earlier turns. A conversation's tenth message costs far more than its first, even if the message itself is short.

What happens when the context window fills up

If a request is larger than the model's context window, the API rejects it with an error. Applications avoid that by managing context before it gets too large, usually in one of three ways:

1. Truncation: drop the oldest messages and keep the most recent ones.
2. Summarisation: replace older parts of the conversation with a short summary. Claude Code does this when a session grows long, and you can trigger it yourself with its /compact command.
3. Retrieval: store documents outside the prompt and include only the passages relevant to the current question.

A bigger context window isn't free

Large context windows are powerful, but filling them has costs:

• Price. You pay for every input token on every request. Sending 100,000 tokens of context each turn costs 100,000 input tokens each turn.
• Speed. More input takes longer to process, so responses start later.
• Focus. Models can give less weight to details buried in the middle of very long inputs. Putting key instructions at the start or end, and leaving out what isn't needed, often improves answers.

How caching helps

Much of a long context is repeated from one request to the next: the same system prompt, the same documents, the same earlier turns. Prompt caching lets the provider reuse that repeated part, billing it at a small fraction of the normal input price. For long conversations and agents, it's the single biggest lever on cost.

Making the most of a context window

1. Send only what the model needs for this request, not everything you have.
2. Keep stable content, such as the system prompt and reference documents, at the start so it can be cached.
3. Summarise or trim long histories rather than resending them in full.
4. Leave room for the answer, and set a sensible max tokens value.
5. Watch usage: every response reports how many input tokens it used, and your dashboard shows it per request.

Choosing a model by context window

A larger window is only worth paying for when your task needs it, such as analysing long documents or running long agent sessions. For short, self-contained requests, a model with a smaller window and a lower price per token is usually the better choice. With every model behind one API, you can use a long-context model only where it helps; the models page lists what's available.

Common questions

Is the context window the same as memory?

Not quite. The context window only covers what's included in the current request. Anything a chat app appears to remember between sessions is stored by the application and added back into the context when needed; the model itself retains nothing.

Does a bigger context window make a model better?

A bigger window lets a model take in more at once, but it doesn't make its answers smarter, and filling it costs more. For most tasks, a well-chosen, focused context produces better results than a very large one.

In short

The context window is the maximum number of tokens a model can consider in one request, covering everything from instructions and history to its own answer. Because APIs resend the full context each time, bigger contexts cost more and run slower, so the best results come from sending what matters, caching what repeats, and trimming what doesn't.