Speculative decoding speedup: MTP vs EAGLE-3 vs DFlash on RTX 5090 and DGX Spark, November 2026

For one user on local hardware, speculative decoding speeds up decode by about 1.5 to 2.5 times in reports from May to August 2026: llama.cpp's MTP took Qwen3.6-27B from 63.7 to 105.5 tok/s on an RTX 5090, and vLLM's MTP took Gemma 4 26B-A4B from 30.3 to 54.9 tok/s on a DGX Spark. Use the model's own MTP head when it has one, EAGLE-3 when it does not, and switch speculation off for heavily batched serving, where published tests show it falling below 1x.

How does speculative decoding speed up one user?

A cheap drafter proposes several tokens and the big model checks them all in one forward pass, so each read of the weights can yield more than one token.

Single-user decode is limited by memory bandwidth, not compute: every token means reading the active weights once (the glossary has the arithmetic). Checking four tokens in one pass costs little more than generating one, so if three of them are accepted you got four tokens for about one weight read. With standard rejection sampling the output is still the target model's own distribution; vLLM calls this lossless up to hardware numerical precision and tests greedy-output equality with speculation on and off.

The gain depends on how many drafts survive. A DGX Spark run of Gemma-4-26B-A4B-NVFP4 on vLLM 0.26.1rc1, posted on the NVIDIA developer forum on 10 August 2026, logged per-position acceptance of 0.84, 0.60, 0.39 and 0.27. By my arithmetic, that gives:

tokens per verify step = 1 + sum of per-position acceptance
2 draft tokens: 1 + 0.84 + 0.60               = 2.44
4 draft tokens: 1 + 0.84 + 0.60 + 0.39 + 0.27 = 3.10
positions 3 and 4 add 0.66 tokens but double the drafting and verify width

The poster found 2 tokens beat 4 and measured 30.3 to 54.9 tok/s single-stream, +81%. That pattern repeats in almost every report below: n=2 or n=3 wins on local hardware.

Which 2026 models ship MTP heads?

Most large 2026 releases do, which makes MTP the default choice rather than an add-on.

Read from each model's config.json and file list on Hugging Face, 7 November 2026; notes from the MiMo-V2.6-Flash and Nemotron 3 Super model cards. Some MTP heads ship in a separate GGUF, for example unsloth's Qwen3.6 MTP repositories. vLLM's docs say Gemma 4 uses separate "assistant" checkpoints with "method": "mtp".

What speedups are published on RTX 5090, DGX Spark and datacenter GPUs?

On an RTX 5090 and a DGX Spark, MTP roughly doubles single-user decode on dense models; on datacenter GPUs the research numbers are higher but come from tuned setups.

Single-stream decode in tok/s unless stated. The DEV post compares llama.cpp with MTP against stock Ollama, so part of its gain is the engine. The DGX Spark Qwen3.6-27B baseline is the PR's approximate figure; its wall time fell from 201.1 s to 83.8 s. Every number in this table is a third-party report.

Two patterns stand out. Dense models gain most, because each verify pass reads all their weights anyway. MoE models gain less per pass in theory, since each drafted token can route to different experts, yet the 35B-A3B MoE reports above are still large; the PR's Qwen3.6-35B-A3B sweep on a 6 GB laptop GPU went from 22.92 to 29.39 tok/s at n=2. For more Spark numbers by model, see the best models for DGX Spark and the Gemma 4 26B-A4B page, which lists a third MTP figure (108 tok/s) that disagrees with the forum run.

When does speculative decoding make things slower?

At high concurrency, with low acceptance, and with long drafts on hard-to-predict text.

• Batching. In the DFlash paper's SGLang tests on one B200, DFlash on Qwen3-8B Math500 fell from 5.1x at concurrency 1 to 2.8x at 32, and EAGLE-3 with a 60-token draft fell to 0.5 to 0.6x on Llama-3.1-8B. The TensorRT-LLM docs say speedups appear only at low batch sizes and there is no way to switch speculation off dynamically.
• Low acceptance. An Apple M4 Max user in the unsloth thread measured 32.8% acceptance with MTP and dropped from 7.98 to 4.23 tok/s. Fixstars saw EAGLE-3 with 3 draft tokens accept 19.8% and lose 30% of throughput on gpt-oss-120b.
• Prefill. The llama.cpp PR's RTX 3090 test showed prompt processing falling from 1,315 to 665 tok/s with MTP on, a cost for long-prompt agent work.
• Parallel slots. llama.cpp MTP refuses -np above 1 ("MTP currently supports only n_parallel=1" at merge time), so it is a single-user feature there.
• Memory. The MTP layer and its own KV cache cost about 2.5 GiB of VRAM in the PR's 3090 test; the author estimates under 10% of total. On a 32 GB card, budget for it when sizing context (see KV cache size by model).

For a shared endpoint, the vLLM docs place model-based methods in "medium-to-low QPS" territory. GPUwerk's own vLLM 0.27.1 runs on a DGX Spark reached 3,235 tok/s total at 256 concurrent requests on Qwen3-Coder-30B-A3B without any speculation, and the capacity planning post shows how per-user speed falls as that batch grows; that is the regime where a drafter mostly adds work.

Which flags turn it on in each engine?

One flag or one JSON blob per engine; the method names differ.

# llama.cpp (merged 16 May 2026): model with an MTP head, one slot
llama-server -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
  --spec-type draft-mtp --spec-draft-n-max 2 -np 1

# vLLM: MTP (the draft model can usually be omitted)
vllm serve <model> --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

# vLLM: EAGLE-3 or a separate draft model
vllm serve <model> --speculative-config '{"method": "eagle3", "model": "<eagle3-head>", "num_speculative_tokens": 3}'

# SGLang: DFlash, from the LMSYS launch command
python3 -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path modal-labs/Qwen3.5-397B-A17B-DFlash \
  --speculative-dflash-block-size 8 --tp-size 8

llama.cpp's server README lists draft-simple, draft-eagle3, draft-mtp, draft-dflash, draft-dspark and five n-gram types for --spec-type, with --spec-draft-n-max defaulting to 3; the old --draft-max flag is removed. SGLang uses --speculative-algorithm with EAGLE3, EAGLE (also covering NEXTN/MTP), DFLASH, STANDALONE or NGRAM. TensorRT-LLM takes a decoding_type of MTP, Eagle3, DFlash, PARD, DraftTarget or SA, and deprecates its NGram mode for removal in January 2027. The llama.cpp on DGX Spark doc has the full MTP command for a Spark, and the engine comparison covers which engine to pick first.

MTP, EAGLE-3 or DFlash: which should you use?

MTP if the model has it, EAGLE-3 if it does not, and DFlash when a drafter exists for your model and engine.

MTP heads are trained with the model and need no extra download, and their acceptance in the table above (72 to 80% overall in the llama.cpp tests, 84% at the first draft position on the Spark) is at or above the best EAGLE-3 figure (70.6%). EAGLE-3 is the route for gpt-oss and Llama 3.3, but the gpt-oss results are thin: +7% on a Spark at K=1 and +1.7% on an RTX PRO 6000, with longer drafts slower. DFlash has the best research numbers (6.08x on Qwen3-8B HumanEval with greedy decoding in its paper), and llama.cpp, SGLang and TensorRT-LLM all accept it, but it needs a trained drafter per target model.

My recommendation for one person on an RTX 5090 or a DGX Spark: run a dense 27B with its MTP head at n=2, check the acceptance rate the server logs on your own prompts, and keep it only if acceptance stays above about 60%. On a team endpoint with more than a handful of concurrent users, benchmark with speculation off first. A dense 27B at 105 tok/s with MTP on an RTX 5090 is faster for one user than most MoE models without it, which changes which model is worth renting for interactive work.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• llama + spec: MTP Support, PR #22673, llama.cpp on GitHub, 2026-05-16
• Stable MTP first release! (discussion #6), unsloth/Qwen3.6-27B-MTP-GGUF on Hugging Face, 2026-05-21
• Measured inference benchmarks on a single DGX Spark (Ollama, llama.cpp and vLLM), NVIDIA Developer Forums, 2026-08-10
• nvidia_qwen3.6-35b-a3b, SparkBench, 2026-08-04
• gptoss-spark, luka-loehr on GitHub
• 267 tok/s local inference on RTX 5090: llama.cpp MTP + Qwen3-35B-A3B MoE, DEV Community, 2026-05-18
• Accelerating Inference of gpt-oss-120b with EAGLE-3, Fixstars Tech Blog, 2026-03-09
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, arXiv, 2025-04-23
• DFlash: Block Diffusion for Flash Speculative Decoding, arXiv, 2026-05-28
• The next generation of speculative decoding: DFlash and Spec V2, LMSYS Org, 2026-06-15
• DeepSeek-V3 Technical Report, arXiv
• Speculative Decoding, vLLM docs, 2026-11-03
• Speculative Decoding, SGLang docs
• Speculative Decoding, TensorRT-LLM docs
• llama-server README, llama.cpp on GitHub
• MiMo-V2.6-Flash-RL model card, Xiaomi on Hugging Face
• NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 model card, NVIDIA on Hugging Face, 2026-03-11