DGX Spark benchmarks: what to actually expect

On one GPUwerk fleet node, Qwen3-Coder 30B-A3B decodes at 80.9 tok/s for a single request and Llama 3.3 70B at 6.0 tok/s (vLLM 0.27.1, AWQ 4-bit, 2 September 2026, raw logs linked below). Under load the 30B reaches 3,235 tok/s total throughput, prompt plus generated tokens, at 256 concurrent requests, and the 70B peaks at 485 at 128. Prefill on an 8,192-token prompt was 5,705 tok/s for the 30B and 391 tok/s for the 70B.

Third-party figures for gpt-oss-120b on the same chip run from 33.5 to 60.7 tok/s single-stream, nearly 2× apart. This page separates what GPUwerk measured from what others reported, lists the reasons the numbers differ, and gives a one-line formula for predicting decode speed. The labels follow the benchmark methodology: “GPUwerk-measured” means this fleet, this engine, this run, with a linked log; “reported” means a third party’s figure that GPUwerk has not reproduced.

Measured on the GPUwerk fleet

I took these two runs with vllm bench serve against vLLM 0.27.1 on one GB10 node of the GPUwerk fleet with nothing else on its GPU, on 2 September 2026. They were the two console presets at the time, which is why I could measure them rather than cite them. Qwen3 Coder 30B is still a console preset; Llama 3.3 70B was removed from the presets on 14 September 2026 because of the single-stream speed shown here.

Concurrency 1 is a separate run of 8 requests, 128 tokens in and 512 out, and shows generated tokens per second. The other rows are 512 tokens in, 256 out, three requests per concurrency slot, and show vLLM’s total token throughput, which counts prompt and generated tokens together. Generated tokens alone are a third of that figure in these runs: 1,078 tok/s for the 30B at 256 and 162 tok/s for the 70B at 128, per the logs. Zero failed requests at every level. Raw logs: Qwen3-Coder · Llama 3.3 70B.

The two models do not behave the same way past their knee, and the difference is the practical one. The mixture-of-experts model saturates: going from 128 to 256 concurrent buys 7.5 % more throughput and nearly doubles the median time per output token (122 to 221 ms), so there is little left to win, but nothing breaks. The dense 70B collapses: throughput falls from 485 to 300 tok/s, and median time to first token goes from 8 seconds to 40. Past its knee the dense model is worse at everything at once. On these two runs the knee is a cliff on the dense model and a plateau on the sparse one, so find yours with your own workload before you depend on it.

The single-stream column also answers whether these are chat models on this hardware. The 30B at 80.9 tok/s is comfortably faster than reading speed. The 70B at 6.0 tok/s is under the 10 tok/s line drawn in the formula section below, and quantization or flag tuning will not move it far, so on one Spark it is a batch model. Speculative decoding is the one technique that could change this, and GPUwerk has not tested it on this model. That is a dense-versus-sparse result, not a size one: the 30B reads about 3B parameters per token, the 70B reads all 70B.

Both figures also land where that arithmetic predicts, which is the best evidence I have for the formula. The 30B’s roughly 1.7 GB of activated weights gives a 160 tok/s ceiling and it measured 80.9, half of it. The 70B’s 37 GB gives 7.4 tok/s and it measured 6.0, four fifths of it. Both sit inside the 50–80 % band, at opposite ends.

Why published Spark numbers differ

The same model on the same GB10 gets quoted at anywhere from 33.5 to 60.7 tok/s. Seven things account for most of that, and a figure is only comparable to another when they match. Checked 11 October 2026; every source is linked in the table.

Sources are the linked pages, opened 11 October 2026. The Ollama (v0.12.6) and SGLang figures are from autumn 2025 and predate newer builds, so read them as a snapshot of that engine then. Dendro Logic counts generated tokens; the GPUwerk total-throughput figures also count prompt tokens. The per-factor effects marked as multiples are my arithmetic on the quoted figures.

Reported numbers by model

Everything in this section is someone else’s measurement, labelled with its engine, format, concurrency and date where the source gives one. Unless the concurrency column says otherwise, decode is single-stream. Checked 11 October 2026.

Forum harness rows: the post labels its vLLM runs 4-bit and its Ollama runs Q4; the Gemma run used NVIDIA’s NVFP4 checkpoint with MTP speculative decoding, which the author measured at 30.3 tok/s without MTP. The llama.cpp rows are the maintainers’ current DGX Spark table (llama-bench, pp2048 and tg32, build 7941); earlier figures in discussion #16578 from October and November 2025 are lower for gpt-oss-20b and Qwen3-Coder. The Mistral Small 4 vLLM figure is from the same NVIDIA forum thread as its SGLang results (23.4 tok/s single-stream without speculative decoding, 26.8 with EAGLE, March 2026); the vLLM recipe page covers how to serve it and gives no Spark figures. The first Nemotron-3-Super row comes from the vLLM project writeup, which calls its results recipe-specific; the second from the NVIDIA forum PSA thread. Qwen3-235B-A22B is a mixture-of-experts model with 235B total and 22B active parameters.

A faster DeepSeek figure is circulating but is not documented well enough for the table. A post in an NVIDIA forum thread on 30 August 2026 links a separate topic titled with 1,058 tok/s prefill and 52 tok/s decode at concurrency 1 on one Spark for DeepSeek V4 Flash 0731, and recommends EXL3 quants served with vLLM. The post itself gives no quant width, engine version or patch list for that figure, so I kept the documented 13 to 21 tok/s row.

The Qwen3-235B-A22B row shows what happens when a model does not fit. The UD-Q4_K_XL file is 134 GB, more than one Spark’s 128 GB, and on one node it ran at about 1.8 tok/s. The same forum report clustered two Sparks with llama.cpp’s RPC backend over TCP/IP, splitting layers manually across both nodes, and got roughly 12.5 tok/s, about 7× faster. That is a real multi-node result, but it used a workaround rather than native tensor parallelism, and the author estimated a native vLLM-plus-NCCL setup could be 2 to 3× faster still. It shows that clustering helps on models this size; it does not show the ceiling.

Two patterns are visible in the table. Mixture-of-experts models with a few billion active parameters (Qwen3-30B-A3B, Gemma 4 26B-A4B, gpt-oss-20b and gpt-oss-120b) decode at about 50 to 85 tok/s single-stream on short prompts. Dense models do not: Nemotron Super 49B is at 5.79 tok/s, and ai-muninn reports 7 tok/s for the dense Gemma 4 31B on the same page as its 52 tok/s for the 26B-A4B. The bandwidth arithmetic below predicts both. For choosing an engine, see choosing an inference engine; for what fits in memory, see which models fit in 128 GB.

What concurrency does

Single-stream reviews leave this out. Dendro Logic (run 21 April 2026, published 22 April) swept concurrency on a single Spark with a character-extraction workload: roughly 1,500 prompt tokens in and up to 400 out, structured JSON. Its figures count generated tokens.

From Dendro Logic, running nvcr.io/nvidia/vllm:26.03-py3 on a single GB10 with 128 GB.

Aggregate throughput on the 49B rises 120-fold going from one request to 256, because batched requests share the same weight reads. That is the number to plan batch work around: total work per hour is far higher than a single-stream figure suggests.

The fourth column is the cost. At concurrency 256 each request runs at 2.85 tok/s, far too slow for a chat window. The 120× is a throughput result, not a latency one, so it applies to overnight document processing, batch classification and agent fleets, not to making a chat interface feel faster. Somewhere in between is a concurrency level that keeps per-request speed tolerable while getting most of the aggregate gain; where it sits depends on your prompt shape, so find it with your own workload.

Ollama versus vLLM, four models

A separate single-Spark run, posted to the NVIDIA developer forum on 10 August 2026, tested the same four models on Ollama and on a vLLM 0.26.1 nightly, single-stream and at 8 concurrent requests, with one harness and 400 max output tokens. Ollama has the slower first token on all three models where both engines report one, and it gains almost nothing from concurrency.

From an NVIDIA developer forum post running the same harness across engines on a single DGX Spark (driver 580.173.02, CUDA 13.0). The post labels the Ollama runs Q4 and the vLLM runs 4-bit; its “8x” columns are the 8-concurrent results. Ollama was faster on Qwen3-30B-A3B single-stream (85.2 against 77.1), so the single-stream winner depends on the model.

The gap is in the fourth column, not the third: Ollama’s aggregate at 8 concurrent lands within 2.3 tok/s of its single-stream number on all four models, which the author reads as Ollama not batching those requests. vLLM’s aggregate is 1.9 to 5.5× its own single-stream figure on the same hardware. The author also notes that the concurrent figures moved by up to 30% between runs on identical configurations. For an endpoint with more than one user, the concurrent column matters more than the single-stream one.

Other hardware, same model

On gpt-oss-120b the Spark leads both neighbours in the reported runs below, with a bigger margin on prefill than on decode. All three rows are reported, none is GPUwerk-measured, and all are llama.cpp numbers from October 2025, the Spark’s first weeks on sale. The Spark’s own llama.cpp figure for this model has since risen to 58.7 tok/s decode and 2,444 tok/s prefill (build 7941, February 2026); I have no equally recent Strix Halo or Thor figure from the same harness, so compare within a row, not across rows. Checked 11 October 2026.

Strix Halo: Hardware Corner quotes the 38.55 against 34.13 pair, and 1,723 against 340 tok/s prefill (pp2048), from the initial llama.cpp results on GitHub; it did not measure them itself. The Register’s December 2025 comparison did not test gpt-oss-120b; it found the GB10 roughly 2 to 3× faster at prompt processing and AMD’s system holding a narrow single-batch decode lead with llama.cpp’s Vulkan backend on the models it tested. For short single-user decode, a Strix Halo is competitive. Thor: JetsonHacks used llama.cpp b6767 results from 15 October 2025, 32 generated tokens and a 2,048-token prompt.

Full side-by-sides: DGX Spark vs Strix Halo, DGX Spark vs Jetson AGX Thor, and for a datacenter reference point DGX Spark vs H100.

Prefill is the real story

Prefill, processing the prompt before the first token comes out, is compute-bound rather than bandwidth-bound, and it is where the GB10 looks strong. Reported gpt-oss-120b prefill runs from 1,169 tok/s on Ollama (October 2025) to 2,444 tok/s on llama.cpp build 7941 (February 2026). The vLLM writeup measured about 1,880 tok/s for Nemotron-3-Super on a 7,234-token prompt, and the August 2026 forum harness reports about 6,000 tok/s prefill, flat out to 27,000 tokens, without naming the model in that sentence.

That shapes which workloads suit the box. Long prompt, short answer, summarising a contract, extracting fields from a document, classifying a support ticket with a large system prompt, plays to prefill and finishes quickly. Short prompt, long answer, “write me an essay”, is pure decode and will feel ordinary.

The vLLM writeup puts time-to-first-token at 0.42 seconds for short prompts, rising to 3.85 seconds for a 7,234-token input. For retrieval-augmented setups stuffing a lot of context into every request, that is the latency you are budgeting for.

The two fleet runs land far apart on this, and the gap is larger than the decode gap. On an 8,192-token prompt the 30B prefills at 5,705 tok/s and the 70B at 391, a 15-fold difference against a 13-fold one in decode. And prefill rate is not constant in prompt length, because attention cost grows quadratically: the same 30B that does 5,705 tok/s at 8k drops to 586 tok/s filling its full 262,144-token context, which took 447 seconds for a single request. Budget time-to-first-token against the context length you will actually send, not against a short-prompt figure.

Predict decode speed in one line

Token generation is memory-bandwidth-bound. Every token requires reading the active weights out of memory once, so the ceiling is:

tokens/sec ≈ memory bandwidth ÷ active parameter bytes

# Spark: 273 GB/s
# gpt-oss-120b at MXFP4, ~5B active params ≈ 3 GB read per token
273 ÷ 3 ≈ 90 tok/s theoretical ceiling

Real engines land somewhere between 50% and 80% of that ceiling: the reported 58.7 tok/s on llama.cpp and 60.7 on vLLM are about two thirds of it. The formula is accurate enough to tell you, before downloading 60 GB, whether a model will feel usable. If the answer is under about 10 tok/s, it will not feel good in a chat window. To check whether a model fits in memory at all, use the memory calculator.

Note the term is active parameters, not total. That is the whole reason MoE models do well here, and why the dense Nemotron 49B at 5.79 tok/s and the 70B at 6.0 tok/s are slow: every token reads all their parameters. Speculative decoding is the main way around the ceiling, because it checks several drafted tokens per weight read; the forum harness saw it lift Gemma 4 26B-A4B from 30.3 to 54.9 tok/s.

Run your own

Your workload is the only benchmark that decides anything. Both engines ship a harness:

# vLLM, against a running server
vllm bench serve \
  --model /workspace/models/your-model \
  --num-prompts 128 \
  --request-rate 8

# llama.cpp, prompt processing and generation separately
llama-bench -m model.gguf -p 2048 -n 128

Sweep --request-rate or the concurrency setting rather than measuring a single point, because as the tables above show, one number tells you almost nothing about how the box behaves under load. Use prompt and output lengths that match what you actually send. The benchmark methodology shows how GPUwerk runs its own.

An hour on a rented GPUwerk Spark costs $0.79, billed per minute from prepaid credit. The Qwen3-Coder sweep in the first table took about 23 minutes of benchmark time, per its log.

Per-model results

Each model on this page also has its own run page, with the exact command and what it is good for. The model index lists all of them:

• gpt-oss-120b
• gpt-oss-20b
• Nemotron-3-Super-120B
• Nemotron Super 49B
• DeepSeek-V4-Flash
• Llama 3.3 70B
• Qwen3-Coder 30B-A3B
• Qwen3-235B-A22B
• GLM-5.3-Flash
• Gemma 4 26B-A4B

FAQ