Blog/What is a context window?
Definition

What is a context window?

By Samuel Seidel · Published September 9, 2026

A context window is the maximum number of tokens a model can attend to in a single request: the prompt, any system instructions, whatever conversation history gets resent alongside it, and the reply the model generates, all counted together against one limit. A model advertised with a "128K context window" can handle up to 128,000 tokens across that entire combined input and output; anything beyond that either doesn't fit or has to be trimmed before it's sent.

Why the limit exists

The size of a context window is a design choice made when a model is trained, not something adjustable afterward without retraining or fine-tuning. It's set by the model's architecture and by how much memory and compute the attention mechanism can afford to spend relating every token to every other token in the window, a cost that grows faster than linearly with window size. A model trained for a longer window costs more to train and to run at that length, which is why window sizes vary widely across models instead of settling on one standard number.

What counts against it

Every token that passes through a request counts, not just the newest message a user types. In a multi-turn conversation, most applications resend the full prior transcript as part of each new request, so the window fills with accumulated history as the conversation goes on. Retrieved documents inserted into a prompt (as in retrieval-augmented generation, covered in the RAG vs agents explainer), system instructions and tool definitions all count as input tokens against the same limit.

What happens when you run out

A model doesn't make room on its own once its context window is full. The API usually rejects a request that exceeds the limit, so the application in front of the model has to manage it: trimming the oldest messages, summarizing earlier turns, or retrieving only the relevant part of a long document instead of sending all of it. Whatever gets dropped is gone from the model's view; nothing remembers it once it falls outside the window.

Context window size versus what it costs to use

A larger context window is a ceiling, not a target. Using more of it means processing more input tokens on every request, which takes longer and, on models billed per token, costs more per request.

It also means more memory for that request's KV cache during generation, because the cache grows with the number of tokens in play. The glossary entry on KV cache explains the mechanics, along with prefill and decode, the two processing phases a request's tokens pass through.

Where GPUwerk fits

How much context a model can serve on a DGX Spark, and how many concurrent conversations fit in its 128 GB of unified memory at a given context length, depends on the model's own context window and on how much memory is left for KV cache after the weights are loaded. GPUwerk's benchmarks page and vLLM doc report measured numbers for specific models rather than theoretical maximums.

Related pages

First top-up: pay $20, get $30 in credit

See what a Spark actually handles.

Measured context and throughput numbers, not vendor claims.

A dedicated DGX Spark in Europe, North America or East Asia, $0.79/hour, billed per minute. $30 of credit is about 37 hours of compute.

Create account