Gemma 4 26B-A4B on a DGX Spark

Gemma 4 26B-A4B is reported at 52 tok/s single-stream on one DGX Spark (vLLM 0.19, NVFP4, ai-muninn, April 2026), and 108 tok/s with multi-token prediction (MTP, ai-muninn, July 2026). The 31B dense sibling manages 6.9 tok/s on the same hardware in community figures ai-muninn cites. GPUwerk has not run any Gemma 4 model on its own fleet, so every Gemma 4 speed on this page is a third-party report with its source and date.

Variants

The Hugging Face model card (checked 2026-10-11) lists the licence as Apache 2.0 and five sizes. Three matter for a single Spark:

Licence, variant list and input types from the model card, checked 2026-10-11. Parameter counts and expert counts are also from the card. The card also lists E2B and a 12B Unified model, not covered here. The 26B-A4B accepts text and image input but not audio; its context length on the card is 256K tokens (128K for E4B). For image prompts see the multimodal vision models guide; GPUwerk has not tested Gemma 4 image input.

Why the 31B dense model is about 7x slower

Decode speed on a Spark is limited by memory bandwidth, 273 GB/s, divided by the bytes read per token (the formula on the benchmarks page). At 4-bit, roughly half a byte per parameter:

tokens/sec ceiling ≈ 273 GB/s ÷ active bytes per token

# 26B-A4B:   3.8B active x ~0.5 byte = ~1.9 GB  -> ceiling ~144 tok/s
# 31B dense: 30.7B active x ~0.5 byte = ~15.4 GB -> ceiling ~18 tok/s
# reported: 52 and 6.9 tok/s, about 36% and 39% of those ceilings

The ceilings are rough (they ignore quantisation scales and layers kept at higher precision). The point is the ratio: 30.7 divided by 3.8 is about 8, and the reported speeds differ by 7.5x (52 against 6.9). Both figures are NVFP4 and both appear on the same ai-muninn page, though the 31B number is one it cites from community benchmarks on the NVIDIA forums rather than its own run. The same cited table has the 31B at 3.7 tok/s in BF16 and 10.6 in AWQ int4: a smaller format helps, but no format brings a dense 31B near the MoE. GPUwerk saw the same dense-against-sparse pattern in its own runs (vLLM 0.27.1, AWQ 4-bit, 2 September 2026): Qwen3-Coder 30B-A3B measured 80.9 tok/s single-stream, Llama 3.3 70B measured 6.0.

Memory fit

The NVFP4 checkpoint used in the ai-muninn run (bg-digitalservices/Gemma-4-26B-A4B-it-NVFP4) is 16.5 GB on disk, as reported on that page. vLLM reports about 121.6 GiB of a Spark’s memory as usable on GPUwerk nodes, so the weights take roughly a seventh of it. That run used --gpu-memory-utilization 0.85 and a 131,072-token context (half the 256K the card lists) with an FP8 KV cache.

By arithmetic (not reported): the 26B-A4B at 16-bit would be about 50 GB (25.2B x 2 bytes), and the 31B at 4-bit about 15.4 GB before scales. Both fit one Spark. Memory is not the constraint with Gemma 4 here; bandwidth is.

Run it on a rented Spark

GPUwerk has no tested Gemma 4 command, so nothing below is verified on its fleet. The April ai-muninn run used Docker, the image vllm/vllm-openai:gemma4-cu130, a community NVFP4 checkpoint and a patched gemma4.py mounted over vLLM’s own. Its July 2026 update says NVIDIA’s official weights, nvidia/Gemma-4-26B-A4B-NVFP4 (shipped 2026-04-30), now run on stock vLLM without the community quant or the patch. Docker does not run inside a GPUwerk instance (see the vLLM doc), so the Docker command cannot be pasted as is. The serving flags from that run:

--quantization modelopt
--kv-cache-dtype fp8
--moe-backend marlin     # page: without it, the FP4 MoE path on SM121 / GB10 produces garbage
--gpu-memory-utilization 0.85
--max-model-len 131072
--reasoning-parser gemma4
--enable-auto-tool-choice --tool-call-parser pythonic

On a GPUwerk instance, install vLLM natively as described in the vLLM doc and pass these flags to vllm serve with the official NVIDIA checkpoint. Whether the official weights on a current vLLM still need --moe-backend marlin is not stated in the July update, so check the output before trusting it. Native MTP arrived in vLLM 0.21.0 (released 2026-05-15) per ai-muninn, which says the target and drafter pairing matters for its 108 tok/s. For the choice between engines see choosing an inference engine. A Spark costs $0.79/hour, billed per minute; see pricing.

Reported speeds

All figures are third-party reports, checked 2026-10-11. None is a GPUwerk measurement.

The three vLLM 26B-A4B rows disagree and GPUwerk has not reconciled them. The forum poster says MTP took vLLM single-stream on Gemma-4-26B-A4B-NVFP4 from 30.3 to 54.9 tok/s (+81%), while ai-muninn reports 52 tok/s without MTP and 108 with it; checkpoints, vLLM versions and prompts differ between the sources. ai-muninn also cites vLLM as about 30% faster than Ollama on the same model. Sources: ai-muninn FP4 post, ai-muninn 2026 guide, NVIDIA developer forum harness, model card. In the forum harness Ollama’s aggregate barely moved from one request to eight (49.6 to 51.9 tok/s) while vLLM’s rose to 303.6.

FAQ