KV cache size by model: how much memory 8k, 32k and 128k tokens of context need, November 2026

KV cache size per token is 2 x layers x KV heads x head dimension x bytes, counted only over layers that keep a full cache. By that formula, applied to each model's config.json on 7 November 2026, 131,072 tokens at FP16 cost 40 GiB on Llama 3.3 70B, 12 GiB on Qwen3-Coder-30B-A3B, 8.1 GiB on Qwen3.8-27B, 4.5 GiB on gpt-oss-120b and about 0.7 GiB on DeepSeek-V4-Flash. FP8 or llama.cpp's q8_0 roughly halves those numbers with little or no measurable quality loss in published tests; q4_0 on the keys is where quality drops.

• Formula: KV bytes = 2 x full-attention layers x KV heads x head_dim x bytes per value x tokens x concurrent sequences.
• Classic GQA models (Llama 3.3 70B, Devstral 2, Qwen3-235B) need about 1.5 to 2.75 GiB per 8k tokens.
• Hybrid models (gpt-oss, Gemma 4, Qwen3.6/3.8, Nemotron 3) grow by 0.05 to 0.6 GiB per 8k tokens, plus a fixed per-sequence state.
• q8_0 KV in llama.cpp: perplexity 6.2322 to 6.2344 (PR #7412). q4_0 on K and V: 6.4384.
• A 32 GB RTX 5090 fits Qwen3.6-35B-A3B at 4-bit with its full 262k context; a 128 GB DGX Spark holds about 12 full-length 128k sequences of gpt-oss-120b.

How do you calculate KV cache size?

Multiply 2 (one key and one value) by the number of layers, the KV heads per layer, the head dimension and the bytes per stored value; that gives bytes per token, which you then multiply by context length and concurrent sequences. The inputs are in each model's config.json as num_hidden_layers, num_key_value_heads and head_dim (or hidden_size divided by num_attention_heads when head_dim is missing).

Worked for Llama 3.3 70B (80 layers, 8 KV heads, head_dim 128, FP16):

2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes per token (320 KiB)
x 131,072 tokens            = 42,949,672,960 bytes = 40.0 GiB per sequence
x 4 users at 128k           = 160 GiB

GPUwerk's own vLLM log for this model on a DGX Spark agrees: at --gpu-memory-utilization 0.85 it reported 207,120 KV tokens, and 207,120 x 327,680 bytes is 63.2 GiB, which plus 37 GiB of AWQ weights lands 3.2 GiB under the 103.4 GiB budget. The memory calculator shows that cross-check in full and lets you enter your own numbers. The glossary defines KV cache if the term is new.

The formula breaks for most models released in 2026, because they keep a full cache in only some layers. The table below handles each layout separately.

KV cache per token for 22 current models

Hybrid and latent-attention models now need 5 to 60 times less KV memory per token than a Llama-style model of similar size.

GiB per sequence (1 GiB = 2^30 bytes); 8k = 8,192, 32k = 32,768, 128k = 131,072 tokens. My calculation from each model's config.json on Hugging Face, read 7 November 2026 (examples: gpt-oss-120b, Qwen3.8-27B, Gemma 4 31B, Nemotron 3 Super, GLM-5.3-Flash, DeepSeek-V4-Flash). "Fixed per sequence" is the sliding-window cache at FP16 or the recurrent state, with Gated DeltaNet, Mamba-2 and KDA states assumed FP32 (the Qwen and Nemotron configs set mamba_ssm_dtype / mamba_ssm_cache_dtype to float32; the KDA dtype is my assumption). The FP8 column halves attention KV and leaves recurrent state at full size. *Qwen3-235B-A22B is 40,960 tokens native; 128k needs YaRN. MLA and DeepSeek rows exclude the small sparse-attention indexer cache. Gemma 4 global layers count K and V separately; see the next section.

Why do 2026 models need so much less KV cache?

Because only a fraction of their layers grow a cache with context, and the rest hold a fixed-size window or state.

Sliding window plus full attention

gpt-oss alternates sliding_attention and full_attention in its layer_types list with sliding_window: 128, so 18 of 36 layers in the 120b need only 128 tokens of cache each: 4.5 MiB per sequence, whatever the context. Gemma 4 uses a 1,024-token window and a 5:1 ratio, which makes the window part larger (800 MiB on the 31B) but the growing part small. Gemma 4 also sets attention_k_eq_v: true; the ComfyUI fix for the 31B (merged 22 October 2026) notes that none of its 10 global layers ships a separate value projection. An engine that stores that tensor once would need half my Gemma figures; I counted K and V separately because I have not confirmed which engines do. Xiaomi's MiMo-V2.6-Flash card confirms 39 sliding-window layers at 128 tokens and 9 global layers.

Linear attention and Mamba hybrids

The Qwen3.8-27B card (the Qwen3.8-27B on DGX Spark page covers running it) describes 16 blocks of three Gated DeltaNet layers and one Gated Attention layer, so 16 of 64 layers keep a cache. Each DeltaNet layer holds a fixed state (48 value heads x 128 x 128 in FP32, about 3 MiB) instead. Nemotron 3 Super's hybrid_override_pattern has 8 attention layers ("*") among 40 Mamba-2 and 40 MoE layers, so its growing cache is 8,192 bytes per token, 40 times smaller than Llama 3.3 70B. The catch is the fixed state per sequence (162 MiB for Nemotron 3 Super), which matters more than context length once you run dozens of short sessions.

Latent attention (MLA) and compression

DeepSeek-style MLA caches one compressed latent per layer (kv_lora_rank 512 plus a 64-dimension RoPE key) instead of per-head keys and values, so 128 attention heads cost the same as one. DeepSeek V4 adds compression along the sequence. The Hugging Face write-up says compressed sparse attention pools every 4 tokens into one entry and heavily compressed attention pools 128. DeepSeek-V4-Flash's config lists 21 layers at ratio 4 and 20 at ratio 128 with one shared 512-dimension KV head, which I read as 5,536 bytes per token at BF16. That is 7.9% of V3.2's 70,272, close to the 7% the DeepSeek-V4 report gives for 1M tokens; the report also stores most dimensions in FP8, which roughly halves it again.

Does FP8 or q8_0 KV cache hurt quality?

At 8 bits, no measurable loss in the published llama.cpp and NVIDIA tests; at 4 bits, the keys are the problem.

PPL = wikitext perplexity, KLD = KL divergence against the FP16 reference, both lower is better. The PR author found the K cache more sensitive than V. In the same discussion, sanmai's Qwen2.5-7B run (21 May 2026) showed q4_0 keys collapsing (KLD 5.49, top-token agreement 11.8%) while q4_0 values alone cost little (KLD 0.004).

Two practical points. llama.cpp's q8_0 is 8.5 bits per value and q4_0 is 4.5 (the PR lists the bits), so q8_0 saves 47%, not 50%. In vLLM, --kv-cache-dtype fp8 runs uncalibrated with scales of 1.0 unless you calibrate with llm-compressor, and the quantized KV cache docs make no accuracy claim of their own. The vLLM DGX Spark writeup (1 June 2026) left FP8 KV off in its measured runs and warns of a performance cost on the Spark. My rule: q8_0 or FP8 for both K and V when you need the room; never quantize keys below 8 bits.

How much context fits on a 32 GB RTX 5090?

A 4-bit MoE model with a hybrid cache gets its whole native context; a 4-bit dense 27B to 31B model gets 150k to 220k tokens at FP16.

Weights: GGUF file sizes in the unsloth and ggml-org repositories on Hugging Face, read 7 November 2026. "Left for KV" assumes 30 GiB usable after the CUDA context and compute buffers, which is my assumption; check nvidia-smi on your own card. Token counts are my arithmetic from the table above after subtracting the fixed state.

For one person on a 32 GB card, I would run Qwen3.6-35B-A3B at 4-bit with the full 262k window and FP16 KV, because it fits without touching cache precision. The RTX 5090 model picks cover speed and quality per use case; the RTX 5090 rental page has the hardware details.

How much context fits on a 128 GB DGX Spark?

About 121.6 GiB is addressable, so after weights the cache has 30 to 85 GiB, enough for a team on hybrid models and two or three long sessions on Llama-style ones.

• gpt-oss-120b (60.8 GiB MXFP4): about 57 GiB left after 4 GiB overhead, which is 1.65 million tokens at 36,864 bytes, or 12 sequences at the full 131,072.
• GLM-4.5-Air Q4_K_M (68.0 GiB): about 49.6 GiB left, 283,000 tokens, two full 128k sessions.
• Qwen3-235B-A22B UD-Q2_K_XL (82.7 GiB): about 34.9 GiB left, 195,000 tokens.
• Nemotron 3 Super UD-Q4_K_XL (78.0 GiB): about 39.6 GiB left; 20 users at 128k need 20 x (1.0 GiB + 0.16 GiB) = 23 GiB.

The 121.6 GiB and 4 GiB overhead follow GPUwerk's vLLM doc and memory calculator; weight sizes are the Hugging Face file sizes; the rest is my arithmetic. On a Spark, decode speed falls as the cache fills, which the long-context guide measures from published runs.

Where the formula and the engine disagree

Engines allocate cache differently from the arithmetic, in four common ways.

• vLLM preallocates. It turns whatever --gpu-memory-utilization leaves after weights into cache blocks at startup, so the log's "KV cache tokens" figure is a budget, not usage. Its hybrid KV cache manager keeps only the window for sliding-window layers and pads Mamba state to the attention page size, which can waste memory on some models; the docs still call the feature early-stage.
• llama.cpp windows are on by default. The llama-server README lists --swa-full as off by default; turning it on allocates a full-size cache for sliding-window layers, which on Gemma 4 31B would add about 50 x 16 KiB per token.
• Parallel slots multiply. -np 4 with -c 131072 reserves cache for four sequences unless the unified buffer (--kv-unified) shares it.
• Draft models and MTP heads add their own small cache; see speculative decoding with MTP and EAGLE-3.

Verdict: read layer_types or the hybrid pattern in config.json before you size anything. A 2026 hybrid model at 128k usually needs less cache than a 2024 dense model at 16k, and the parameter count tells you nothing about which one you have.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• config.json, openai/gpt-oss-120b, Hugging Face, 2025-08-26
• config.json, Qwen/Qwen3.8-27B, Hugging Face
• config.json, deepseek-ai/DeepSeek-V4-Flash, Hugging Face, 2026-06-22
• config.json, google/gemma-4-31B, Hugging Face, 2026-07-15
• config.json, nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16, Hugging Face, 2026-08-25
• config.json, zai-org/GLM-5.3-Flash, Hugging Face, 2026-10-15
• Qwen3.8-27B model card, Qwen on Hugging Face
• MiMo-V2.6-Flash-RL model card, Xiaomi on Hugging Face
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv
• DeepSeek-V4: a million-token context that agents can actually use, Hugging Face blog, 2026-04-24
• Fix Gemma 4 31B global attention (k_eq_v, 4 global KV heads), PR #17220, ComfyUI on GitHub, 2026-10-22
• CUDA: quantized KV cache FA support, PR #7412, llama.cpp on GitHub, 2024-05-20
• Asymmetric KV q8/q4 cache to preserve precision while reducing memory usage, discussion #23470, llama.cpp on GitHub, 2026-09-12
• llama-server README, llama.cpp on GitHub
• Hybrid KV cache manager, vLLM docs
• Quantized KV cache, vLLM docs
• vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation, vLLM blog, 2026-06-01
• Optimizing Inference for Long Context and Large Batch Sizes with NVFP4 KV Cache, NVIDIA Technical Blog, 2025-12-08