DGX Spark benchmarks: what to actually expect
Most DGX Spark reviews measure one person typing one prompt, which is the workload the hardware is worst at. Here are published numbers from several sources, what they mean, and the arithmetic that predicts them before you rent anything.
Predict decode speed in one line
Token generation is memory-bandwidth-bound. Every token requires reading the active weights out of memory once, so the ceiling is:
tokens/sec ≈ memory bandwidth ÷ active parameter bytes # Spark: 273 GB/s # gpt-oss-120b at MXFP4, ~5B active params ≈ 3 GB read per token 273 ÷ 3 ≈ 90 tok/s theoretical ceiling
Real engines land somewhere between 50% and 80% of that ceiling, and the measured 60 tok/s below is right in that band. The formula is accurate enough to tell you, before downloading 60 GB, whether a model will feel usable. If the answer is under about 10 tok/s, it will not feel good in a chat window no matter what you tune.
Note the term is active parameters, not total. That is the whole reason MoE models do well here.
Single-stream numbers
One user, one request at a time. All figures are from published sources, linked.
| Model and format | Engine | Decode | Prefill |
|---|---|---|---|
| gpt-oss-120b, MXFP4 | llama.cpp | ~60.5 tok/s | ~1,956 tok/s |
| Nemotron-3-Super-120B-A12B, NVFP4 | vLLM | 22.7 to 23.7 tok/s | ~1,900 tok/s |
| Nemotron Super 49B v1.5, NVFP4 | vLLM | 5.79 tok/s | not reported |
| gpt-oss-120B, MXFP4 | vLLM | 33.5 tok/s | not reported |
| DeepSeek-V4-Flash, 2-bit experts | patched vLLM | 14 to 21 tok/s | ~753 tok/s |
Sources: gpt-oss-120b on llama.cpp reported by user eugr in the llama.cpp DGX Spark performance discussion. Nemotron-3-Super figures from the vLLM project's own DGX Spark writeup. Nemotron 49B and gpt-oss-120B under vLLM from Dendro Logic's concurrency benchmark. DeepSeek-V4-Flash from the model's Hugging Face discussion thread, using a patched fork rather than stock vLLM.
The spread between the two gpt-oss-120B rows, 60 tok/s on llama.cpp against 33.5 on vLLM, is not a measurement error. llama.cpp is optimised for exactly this case, one stream, minimum latency. vLLM carries scheduler and batching machinery that costs something when there is nothing to batch. Pick the engine to match the workload: llama.cpp or Ollama if it is you alone, vLLM the moment it is a shared endpoint.
The Nemotron 49B row at 5.79 tok/s is worth staring at. It is a dense 49B model, so every token reads all 49B parameters, and the bandwidth arithmetic above predicts roughly that. Dense models in that size class are slow on this hardware, and no amount of configuration fixes it.
What concurrency does
This is the number the reviews miss. Dendro Logic ran a real character-extraction workload, roughly 1,500 prompt tokens in and up to 400 out with structured JSON, and swept the concurrency level on a single Spark.
| Model | Concurrency | Aggregate throughput | Per-request speed |
|---|---|---|---|
| Nemotron Super 49B NVFP4 | 1 | 5.79 tok/s | 5.79 tok/s |
| Nemotron Super 49B NVFP4 | 256 | 695.1 tok/s | 2.85 tok/s |
| gpt-oss-120B MXFP4 | 1 | 33.5 tok/s | 33.6 tok/s |
| gpt-oss-120B MXFP4 | 256 | 862.8 tok/s | 3.62 tok/s |
From Dendro Logic, running nvcr.io/nvidia/vllm:26.03-py3 on a single GB10 with 128 GB.
Aggregate throughput on the 49B rises 120-fold going from one request to 256. That is the number to plan batch work around, and it is the honest case for this hardware: total work per hour is far higher than a single-stream review suggests, because batched requests share the same weight reads.
Read the fourth column before getting excited. At concurrency 256, each individual request crawls at 2.85 tok/s. Nobody would sit in a chat window at that speed. The 120× multiplier is real and it is a throughput result, not a latency one, so it applies to overnight document processing, batch classification and agent fleets, and not to making your chat interface feel faster. Somewhere in the middle, and the crossover depends on your prompt shape, is a concurrency level that keeps per-request speed tolerable while getting most of the aggregate gain. Find it with your own workload rather than assuming ours.
Prefill is the real story
Prefill, processing the prompt before the first token comes out, is compute-bound rather than bandwidth-bound, and it is where the GB10 looks strong: roughly 1,900 to 1,950 tokens per second across both the vLLM and llama.cpp measurements above.
That shapes which workloads suit the box. Long prompt, short answer, summarising a contract, extracting fields from a document, classifying a support ticket with a large system prompt, plays to prefill and finishes quickly. Short prompt, long answer, "write me an essay", is pure decode and will feel ordinary.
The vLLM writeup puts time-to-first-token at 0.42 seconds for short prompts, rising to 3.85 seconds for a 7,234-token input. For retrieval-augmented setups stuffing a lot of context into every request, that is the latency you are budgeting for.
Why our numbers might differ from yours. Kernel version matters more than you would expect on this platform, and contributors in the llama.cpp thread report meaningful differences across NVIDIA kernel releases and unified-memory settings. Quantization format, context length and --max-num-seqs all move the result. Treat every figure here as a reference point, not a specification.
Run your own
Your workload is the only benchmark that decides anything. Both engines ship a harness:
# vLLM, against a running server vllm bench serve \ --model /workspace/models/your-model \ --num-prompts 128 \ --request-rate 8 # llama.cpp, prompt processing and generation separately llama-bench -m model.gguf -p 2048 -n 128
Sweep --request-rate or the concurrency setting rather than measuring a single point, because as the table above shows, one number tells you almost nothing about how the box behaves under load. Use prompt and output lengths that match what you actually send.
An hour on a rented Spark costs less than the time spent arguing about someone else's benchmark. That is roughly the entire reason we rent them by the hour.