What is paged attention?
Paged attention stores the KV cache in fixed-size, non-contiguous blocks instead of one long contiguous chunk of memory per sequence. Each block holds the keys and values for a fixed number of tokens, and a lookup table maps each sequence to its blocks, wherever they sit in memory. The idea comes from virtual memory paging in operating systems, where a page table lets a process treat scattered physical pages as one contiguous address space.
The problem it fixes
Before paged attention, an inference engine had to guess how long a sequence's cache might grow and reserve a contiguous block of memory upfront to cover the worst case, usually the model's full context window. Most requests never use anywhere near that much, so the gap between what was reserved and what was used sat idle. Sequences of different lengths also fragmented the remaining memory, leaving gaps too small for the next allocation even when total free memory was enough. Between over-reservation and fragmentation, an engine could run out of memory for new requests while a large share of allocated memory sat empty.
How block-based allocation changes that
With paged attention, memory is divided into fixed-size blocks up front, and a sequence's cache is built up block by block as it generates tokens, not reserved all at once. A sequence holds only the blocks it currently needs, and a finished sequence's blocks go back into a pool any other sequence can use, in any order. The same indirection enables sharing: two sequences with an identical prefix, such as the same system prompt, can point to the same physical blocks for that prefix instead of duplicating them.
What it buys in practice
With less memory lost to over-reservation and fragmentation, an engine can hold more concurrent sequences in the same GPU memory, and that is what raises serving throughput under load. The attention math does not change: the computed numbers are identical, only where the key and value tensors live in memory differs. vLLM introduced the approach, and it is now close to standard in engines built for high-concurrency inference, alongside the techniques in the batching in LLM inference post.
Why it matters on a DGX Spark
A Spark's 128 GB of unified memory holds the model weights and the KV cache for every active request, with no separate pool to overflow into. Every gigabyte held for cache the model never fills is a gigabyte not serving another conversation or a longer context. Paged allocation is one reason vLLM packs more concurrent requests into a fixed memory budget than a naive contiguous scheme; the choosing an inference engine guide covers how engines differ here, and the benchmarks page has measured throughput under concurrent load.