DGX Spark quantization speed: the same models in NVFP4, MXFP4, FP8, AWQ and GGUF

On a DGX Spark in November 2026, NVFP4 is the fastest format on vLLM and SGLang once the engine picks a native FP4 kernel, and slower than FP8 or AWQ when it falls back. For Qwen3-Coder-30B-A3B, published single-stream decode is 30.6 to 31.7 tok/s in BF16, 50.5 to 55.0 in FP8 and 65.0 to 66.0 in NVFP4 (patched vLLM and SGLang builds, February 2026), 61.1 at Q8_0 on llama.cpp, and 80.9 in AWQ 4-bit on vLLM 0.27.1 (GPUwerk, September 2026). For the dense Qwen3.8-27B, NVFP4 generates 29 to 34% faster than FP8 on vLLM 0.27.1.

One GB10 each. The forum runs used driver 590 and NVIDIA's 26.01 container as a base; "patched nightly" is the author's vLLM-next and SGLang-next builds after fixing CUTLASS, FlashInfer and sgl-kernel for sm_121, and the post does not give prompt lengths. The GPUwerk run is 128 tokens in and 512 out. Bits per weight for Q8_0 is my arithmetic from llama-bench's 30.25 GiB for 30.53B parameters. Read 7 November 2026.

Why does the format change decode speed?

Because decode reads every active weight once per token, so fewer bytes per weight means more tokens, until the kernel cannot keep up. The cleanest published pair is Qwen3.8-27B on vLLM 0.27.1 with identical flags and 16 concurrent prompts: the FP8 checkpoint is 30.9 GB and the Unsloth NVFP4 one 23.4 GB, a ratio of 1.32 by my arithmetic, and NVFP4 generated 1.29 to 1.34 times as many tokens.

vllm bench serve, random dataset, --ignore-eos, Qwen/Qwen3.8-27B-FP8 (block-128 E4M3) against unsloth/Qwen3.8-27B-NVFP4 (4-bit MLP, 8-bit attention, FP8 KV cache), from the NVIDIA forum, 15 August 2026.

The match is not always that close. For Qwen3-Coder-30B-A3B, BF16 to FP8 halves the weight bytes but bought 1.65 to 1.73 times the speed, and FP8 to NVFP4 cuts bytes by almost half again but bought 1.25 to 1.29 times. My reading is that a 3B-active model spends a growing share of each step on attention, routing and layers that stay in higher precision, so the weight savings stop dominating. The format's label also hides which layers it covers: one owner took a GPTQ-INT4 build of Qwen3.5-122B-A10B whose attention and shared experts were still BF16, swapped those layers for FP8, and went from 15 to 21.5 tok/s (NVIDIA forum, March 2026).

When is NVFP4 slower than FP8 or AWQ?

Whenever the engine does not find a native FP4 kernel for sm_121 and falls back to a dequantizing or emulated path. Every case below holds the model fixed and changes only the format or the kernel:

Single-stream decode unless the column says otherwise. The Qwen3-VL row ran on a two-Spark cluster; the others on one GB10.

The trend over the year is clear: AWQ beat NVFP4 by 32% in December 2025, patched builds had NVFP4 ahead of FP8 by February 2026, and vLLM 0.27.1 showed NVFP4 29 to 34% ahead in August. The checkpoint's name does not tell you which side of that line your build is on. The startup log does.

# which GEMM and MoE kernels did vLLM pick?
vllm serve nvidia/Qwen3-30B-A3B-NVFP4 --max-model-len 32768 2>&1 | tee serve.log
grep -iE "flashinfer|cutlass|marlin|emulat|fallback" serve.log

# then measure one user, same lengths for every format you compare
vllm bench serve --model nvidia/Qwen3-30B-A3B-NVFP4 --dataset-name random \
  --random-input-len 1000 --random-output-len 1000 --num-prompts 8 --max-concurrency 1

Marlin or emulation in that log means you are paying FP4's quality cost without its speed. Container and flag choices are on the vLLM page.

Which format is fastest on each engine?

NVFP4 on vLLM, SGLang and TensorRT-LLM when the kernels are present; the model's native file on llama.cpp. What the published runs show per engine:

TensorRT-LLM figures from NVIDIA's blog (2,048 in, 128 out, batch 1, 24 October 2025) and LMSYS (13 October 2025). gpt-oss-120b MXFP4 from llama.cpp build 7941's batched bench (512 in, 32 out); the Q4_K_M file from Jeff Geerling's run on build 7129 (pp512, tg128), in ai-benchmarks issue 34.

The TensorRT-LLM pair fits the bandwidth arithmetic: an 8B model is about 4.5 GB in NVFP4 and 8 GB in FP8, ceilings of 60 and 34 tok/s at 273 GB/s by my calculation, and both runs reach 60 to 64% of their ceiling. On llama.cpp, native NVFP4 matrix kernels merged on 28 April 2026 raised Nemotron-3-Super-120B's pp2048 from 506 to 611 tok/s on a Spark and left generation at 12.0 tok/s, per Spark results posted in PR 22196. vLLM decodes the same NVFP4 model at about 24 tok/s, per the PSA thread in the table above. Engine setup is in the planned SGLang and TensorRT-LLM guides and the engine comparison.

Does the format still matter with many users?

For dense models yes; for 3B-active mixture-of-experts models it fades from about 8 concurrent requests. The Qwen3.8-27B gap was measured at 16 concurrent requests, and its NVFP4 build used about 5.6% of the KV cache against about 15% for FP8 on the decode-heavy run, so it also leaves more room for users. A Qwen3.6 sweep with MTP compared a 35 GB FP8 build with a 22 GB PrismaQuant one:

Aggregate tok/s, steady-state runs, vLLM with MTP 3, NVIDIA forum, 22 April 2026. The post does not name the model size; a 35 GB FP8 build at 80 tok/s fits the 35B-A3B model.

Long contexts flatten the differences further. On SparkBench's agent benchmark at about a 50,000-token fill, Ornith 1.0 35B (3B active) ran 33.7 tok/s in FP8 and 33.0 in NVFP4 on vLLM, and 28.1 at Q8_0 on llama.cpp (SparkBench, July 2026). My reading is that at that depth attention over the cache, not the expert weights, sets the pace. Per-user speeds under load are in the planned concurrent users guide.

What does the faster format cost in quality?

On vendor checkpoints the published loss is usually under a point, per the model cards compiled in NVFP4 vs MXFP4 vs FP8; on this chip the bigger risk is a wrong kernel. Three Spark-specific data points: the author of the February FP4 test scored BF16 and FP8 builds of Qwen3-Coder-30B-A3B at 78 to 80% on the author's own math test and NVFP4 at 72 to 74%; the fastest FP8 result in that test, 60.1 tok/s, produced 0% because the block-FP8 Triton kernel returned garbage on sm_121; and llama.cpp measured Nemotron-3-Super perplexity at 4.6237 for Q4_K, 4.6283 for NVFP4 with 8-bit activations and 4.6577 with 4-bit activations on the tensor-core path. A reply on the forum warned that some FP4 kernels written for other Blackwell chips run on sm_121 without errors and produce garbage, mostly on mixture-of-experts paths (October 2026).

The formats themselves are explained on the quantization reference, and the quantized vs full precision guide covers testing on your own prompts. Read a few answers from any new build before you benchmark it.

My recommendation on a Spark today: use the model's native format if it has one, which means MXFP4 for gpt-oss. Otherwise use an NVFP4 checkpoint from the model vendor or NVIDIA on a current vLLM or SGLang build, after confirming the FP4 kernel in the log; that is about a third faster than FP8 on dense models. Use AWQ when no NVFP4 build has an accuracy table, FP8 when the model fits and you cannot afford to test quality, and Q8_0 or Q4_K_M GGUF when one person uses llama.cpp.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• FP4 on DGX Spark: Why It Doesn't Scale Like You'd Expect, NVIDIA Developer Forums, 2026-02-10
• Qwen3.8-27B on DGX Spark using vllm: NVFP4 vs FP8 performance, NVIDIA Developer Forums, 2026-08-15
• Measured inference benchmarks on a single DGX Spark (Ollama, llama.cpp, vLLM), NVIDIA Developer Forums, 2026-08-10
• Benchmarks for Qwen3.6 FP8 vs PrismQuant, NVIDIA Developer Forums, 2026-04-22
• PSA: State of FP4/NVFP4 Support for DGX Spark in VLLM, NVIDIA Developer Forums, 2025-12-01
• PSA: State of FP4/NVFP4 Support for DGX Spark in VLLM, post 131 (CUTLASS K=64 tiles), NVIDIA Developer Forums, 2026-03-29
• Qwen3.5-122B-A10B on single Spark: 15 to 21.5 tok/s with hybrid GPTQ-INT4 + FP8 dense layers, NVIDIA Developer Forums, 2026-03-18
• NVFP4 Is a Trap on GB10: FP8 Wins by 32% (vLLM + SGLang Tested), ai-muninn, 2026-05-06
• DGX Spark benchmark results (llama-bench and llama-batched-bench), ggml-org/llama.cpp on GitHub, 2026-02-05
• ggml-cuda: Blackwell native NVFP4 support (PR 22196, Spark results in comments), ggml-org/llama.cpp on GitHub, 2026-04-30
• Benchmark Dell Pro Max with GB10, ai-benchmarks issue 34, Jeff Geerling on GitHub, 2025-11-26
• How NVIDIA DGX Spark's Performance Enables Intensive AI Tasks, NVIDIA Technical Blog, 2025-10-24
• NVIDIA DGX Spark In-Depth Review, LMSYS, 2025-10-13
• Ornith 1.0 35B on SparkBench, SparkBench, 2026-07-10
• DGX Spark GB10: Magistral 24B BF16 ~4.85 tok/s, GEMV decode (reply on NVFP4 and FP8), NVIDIA Developer Forums, 2026-10-26