What is grouped-query attention?
Grouped-query attention (GQA) is an attention design in which several query heads share one key head and one value head, instead of every query head having its own. In standard multi-head attention, a model with 32 attention heads computes and stores 32 separate sets of keys and values. With GQA, those 32 query heads might be split into 8 groups of 4, and each group shares a single key/value head, so only 8 sets of keys and values get computed and cached rather than 32.
Why the number of key/value heads matters
The KV cache stores a key and value tensor per attention head, per token, per layer, and its size scales directly with how many distinct key/value heads a model has. Query heads don't add to that cost, because only keys and values are cached across decode steps. Cutting distinct key/value heads from 32 to 8 while keeping all 32 query heads shrinks the cache by a factor of 4 and leaves the query heads, and the capacity they give the model, untouched.
Where it sits between two extremes
Standard multi-head attention gives every query head its own key/value head, maximizing representational flexibility at the cost of a large cache. Multi-query attention, an earlier alternative, goes to the opposite extreme: one shared key/value head for all query heads, minimizing cache size but limiting how differently each head can attend to the sequence. Grouped-query attention is a tunable middle: more groups keep more of multi-head attention's flexibility, fewer groups keep more of multi-query attention's memory savings. Most current open-weight models use a small number of groups, giving up a little representational capacity for a large cut in cache size.
What it costs, if anything
Because it reduces capacity in the key/value projections, GQA could in principle hurt output quality compared with full multi-head attention at the same parameter count. In practice, models trained with GQA from scratch (not converted afterward) show little measurable quality difference from multi-head attention at the same size, which is why it is now the default in most current LLM architectures.
It is a different kind of technique from flash attention, which changes how the attention computation is scheduled on a GPU without changing the model's architecture, and from paged attention, which changes how the resulting cache is laid out in memory; GQA changes how much there is to cache in the first place.
Why this matters for serving on a Spark
A smaller KV cache per token means more concurrent sequences or longer contexts fit in the same amount of GPU memory. On a DGX Spark, where the KV cache shares the 128 GB unified memory pool with the model weights instead of a separate VRAM allocation, a GQA model leaves noticeably more of that pool for cache than an equivalent multi-head model. Most current open-weight models on the GPUwerk models page already use GQA, which is part of why they reach the concurrency levels on the benchmarks page.