DGX Spark long context: how decode speed and prefill time change from 4k to 128k tokens

On a DGX Spark, long context costs some models almost nothing and others most of their speed. SparkBench reports Qwen3-Coder-Next decoding at 58.9 tok/s after a 4k fill and 55.0 tok/s after 100k, while Qwen3.8-27B with speculative decoding drops from 38.1 to 10.7 tok/s (vLLM and SGLang runs, July to October 2026). Prefill also slows with depth: llama.cpp processes gpt-oss-120b prompts at 2,444 tok/s on an empty context and 1,567 tok/s at 32k, so by my calculation a 100k-token prompt takes about 76 seconds before the first token. At 128k, the BF16 KV cache ranges from 1 GiB (Nemotron-3-Super-120B) to 40 GiB (Llama 3.3 70B).

Single-stream decode in tok/s after a fixed context fill, one GB10 node, read 7 November 2026. Layer counts are from each model's config.json (linked in the KV section). SparkBench does not explain why the DeepSeek row is faster at 50k than at 4k, and does not list engine versions for most rows. SparkBench lists no speculative draft for the Gemma row; other Gemma 4 reports on this site, with MTP, are higher.

How much does decode slow down at long context?

Between 7% and 72% from 4k to 100k in the SparkBench rows above. The pattern I see, which is my reading and not SparkBench's: the two rows without a speculative draft barely move, and every row with MTP, DFlash or DSpark loses half or more of its speed by 100k.

The mechanism behind the plain rows is bandwidth. Each new token reads the active weights once and then the entire KV cache, so the cache adds bytes to every step. llama.cpp's own DGX Spark llama-bench results (build 7941, February 2026) show it with full-attention and sliding-window models:

Token generation in tok/s at each context depth, from the llama.cpp repository's DGX Spark page (system info dated 5 February 2026). Drop percentages are my arithmetic.

A worked check. gpt-oss-120b reads about 3 GB of weights per token (about 5B active parameters at MXFP4, per the decode arithmetic on the benchmarks page). Its KV cache costs 36 KiB per token (18 full-attention layers x 8 KV heads x 64 dims x 2 for K and V x 2 bytes), so at 32,768 tokens it is 1.2 GB. Reads per token rise from 3 GB to 4.2 GB, predicting 58.72 x 3 / 4.2 = 41.9 tok/s. llama.cpp measured 42.76. Qwen3-Coder-30B-A3B keeps 96 KiB per token in all 48 layers, so its F16 cache at 32k is 3.2 GB, about the same as its roughly 3 to 3.5 GB of Q8_0 active weights. The prediction is a near halving, 29 to 32 tok/s; measured, 30.21.

Speculative rows break this arithmetic because they verify several drafted tokens per weight read. My guess, untested, is that both the draft acceptance rate and the attention cost per verified token get worse deep in a long context, which removes most of the gain. If you run long contexts, benchmark with speculation on and off at your real fill.

How long does prefill take for a 100k-token prompt?

About 45 seconds to 4 minutes on one Spark for the models llama.cpp benchmarked, by my calculation below. Long prompts hurt more than long answers here, because prefill rate falls as the context grows.

llama.cpp's results give prefill (pp2048) at five depths. Per-token time rises almost linearly with depth, since each new token attends to all the tokens before it. Fitting t(d) = a + b x d through the 0 and 32k points (it predicts the 16k point within 3%) and summing over the prompt gives total time T(N) = a x N + b x N² / 2.

Measured columns in tok/s from llama.cpp's DGX Spark page, build 7941. Calculated columns are my extrapolation beyond the measured 32k, single request, no prefix cache. For gpt-oss-120b, a = 1/2,444 s and b = 7.0 x 10^-9 s per token per token.

The extrapolation holds up against a longer measured run. On the same model family in vLLM 0.27.1, the GPUwerk benchmarks page records Qwen3-Coder-30B-A3B (AWQ) filling its full 262,144-token context in 447 seconds, an average of 586 tok/s. The llama.cpp fit predicts 514 seconds for that length at Q8_0. Same order, and vLLM a little faster.

Two multi-node reports for scale. DeepSeek-V4-Flash on two Sparks with vLLM 0.21.1 and FP8 KV took 2.4 minutes to the first token at a 240K fill and 16.6 minutes at 980K, while decode held 39.4 and 30.4 tok/s (NVIDIA forum, 4 June 2026; more on that model in the DeepSeek-V4-Flash notes). On an eight-Spark DeepSeek V4.1 Flash cluster, jameslacroix reports that with 100K coding contexts, going from 4 to 8 concurrent requests raised median time to first token from 64 to 114 seconds: long prefills block other users even when slots are free.

How much KV cache memory does each model need at 128k?

From 1 GiB to 40 GiB at BF16 for one 131,072-token sequence, depending on how many layers keep a full-length cache. The formula is bytes per token = 2 (K and V) x full-attention layers x KV heads x head dimension x bytes per value. Hybrid models keep a cache only on their attention layers; their linear-attention layers hold a fixed-size state instead.

My calculation from the config.json values, per sequence; 128k = 131,072 tokens. Gemma 4's config sets attention_k_eq_v, which may let an engine store one tensor instead of two on global layers; I counted both, so treat its row as an upper bound. The sliding-window caches are fixed in size: about 4.5 MiB for gpt-oss-120b and 200 MiB for Gemma at BF16. Qwen3.6-35B-A3B's 30 linear-attention layers hold about 31 MB of BF16 state per sequence regardless of length (30 x 32 value heads x 128 x 128 x 2 bytes, my estimate). gpt-oss-120b's native maximum is 131,072 tokens.

Against 128 GB of unified memory, none of these is a capacity problem for one user; Llama 3.3 70B at 40 GiB plus about 40 GB of 4-bit weights is the only one that takes a large share. The cache matters for concurrency (multiply by users) and, as shown above, for decode speed. The memory calculator does the same sum for other models and context lengths, and the KV cache size guide covers MLA models and the quality effect of 8-bit and 4-bit caches.

Which settings help with long context?

An FP8 KV cache, chunked prefill, prefix caching and a context limit sized to the job. None changes the attention arithmetic; each removes waste around it.

• FP8 KV cache. It halves both the cache's memory and the bytes each decode step reads. In vLLM it is --kv-cache-dtype fp8; the vLLM quantized KV cache page notes that without calibration all scales are 1.0, recommends calibrating with llm-compressor, and offers --kv-cache-dtype-skip-layers to keep sensitive layers such as sliding-window attention at native precision. Five of the six SparkBench rows above use it; the DeepSeek row lists its KV type as auto. llama.cpp takes -ctk q8_0 -ctv q8_0; the server README lists f16 as the default for both.
• Chunked prefill. vLLM's tuning guide says it is on by default in V1. --max-num-batched-tokens sets the trade: above 8,192 for throughput and time to first token, about 2,048 for smoother inter-token latency when other users are decoding.
• Prefix caching. If many questions hit the same long document, the second one skips the prefill. Prompt layout and the vLLM and llama.cpp switches are in the prompt caching guide.
• A realistic context limit. Set --max-model-len (vLLM) or -c (llama.cpp) to what you will send. In llama.cpp, -np sets the number of slots sharing that context, and --swa-full (off by default) would allocate a full-length cache for sliding-window layers, which you rarely want.

# vLLM: one long-context user, FP8 KV, prefix cache on
vllm serve /workspace/models/qwen3-coder-next-nvfp4 \
  --max-model-len 131072 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.85

# llama.cpp: one slot with the full 128k, 8-bit KV
llama-server -m model.gguf -c 131072 -np 1 -fa on -ctk q8_0 -ctv q8_0

Container and build steps are in the vLLM guide and the llama.cpp guide.

Which model should you pick for long documents?

A hybrid-attention mixture-of-experts model, without speculative decoding if the context is very long. On one Spark I would start with Qwen3-Coder-Next for code and long technical documents (55 tok/s at 100k in SparkBench, 3 GiB of BF16 cache at 128k) and with Nemotron-3-Super-120B for contract review and extraction, where its 1 GiB cache leaves room for several long sequences at once. I would avoid dense full-attention models for this job: Llama 3.3 70B needs 40 GiB of cache at 128k and, being dense, decodes slowly before the cache adds anything.

Budget the wait honestly. Even a favorable model spends one to two minutes on a fresh 100k prompt on one Spark, so long-context work suits batch review and repeated questions over a cached document better than interactive chat with new documents each time. If one Spark's 128 GB is too small for the model itself, the guide to large MoE models on several Sparks covers the memory math.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• SparkBench leaderboard, SparkBench, 2026-10-18
• Qwen3-Coder-Next on SparkBench, SparkBench, 2026-07-10
• Qwen3.6-35B-A3B on SparkBench, SparkBench, 2026-08-04
• Qwen3.8-27B on SparkBench, SparkBench, 2026-09-10
• Gemma-4-26B-A4B-IT on SparkBench, SparkBench, 2026-07-10
• Qwen3.8-Flash-Next FP4-Local on SparkBench, SparkBench, 2026-10-19
• DeepSeek V4 Flash 0731 (SparkInfer) on SparkBench, SparkBench, 2026-08-23
• DGX Spark benchmark results (llama-bench), ggml-org/llama.cpp on GitHub, 2026-02-05
• DeepSeek V4 Flash with 1M context on 2x DGX Spark, NVIDIA Developer Forums, 2026-06-04
• Quantized KV Cache, vLLM documentation, 2026-11-02
• Optimization and Tuning, vLLM documentation
• llama.cpp server README, ggml-org/llama.cpp on GitHub
• Qwen/Qwen3.6-35B-A3B config.json, Hugging Face
• Qwen/Qwen3.8-27B config.json, Hugging Face
• Qwen/Qwen3-Coder-Next config.json, Hugging Face
• Qwen/Qwen3-Coder-30B-A3B-Instruct config.json, Hugging Face
• openai/gpt-oss-120b config.json, Hugging Face
• unsloth/Llama-3.3-70B-Instruct config.json, Hugging Face
• nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 config.json, Hugging Face
• google/gemma-4-26B-A4B config.json, Hugging Face
• DeepSeek V4.1 Flash on 8x DGX Spark: TP8/EP8 concurrency results, NVIDIA Developer Forums, 2026-10-31