Run gpt-oss-120b on a DGX Spark: reported tok/s
gpt-oss-120b runs on one 128 GB Spark and is fast enough for a chat window: published measurements show 60.5 tok/s decode at MXFP4 under llama.cpp for a single user. Under vLLM it drops to 33.5 tok/s single-stream but scales to 862.8 tok/s aggregate once 256 requests share the box. This page has those numbers, two ways to run it, and what it suits.
Measured on one Spark
| Engine | Quantization | Decode, single user | Prefill |
|---|---|---|---|
| llama.cpp | MXFP4 | ~60.5 tok/s (advantage) | ~1,956 tok/s |
| vLLM | MXFP4 | 33.5 tok/s | not reported |
llama.cpp figures reported by user eugr in the llama.cpp DGX Spark performance discussion. vLLM single-stream and concurrency figures from Dendro Logic's concurrency benchmark, running nvcr.io/nvidia/vllm:26.03-py3 on a single GB10 with 128 GB. Neither source states an exact run date; both are quoted on GPUwerk's benchmarks page.
| Concurrency (vLLM) | Aggregate throughput | Per-request speed |
|---|---|---|
| 1 | 33.5 tok/s | 33.6 tok/s |
| 256 | 862.8 tok/s (advantage) | 3.62 tok/s |
From Dendro Logic's character-extraction workload, roughly 1,500 prompt tokens in and up to 400 out with structured JSON. At 256 concurrent, aggregate throughput is 25.8 times the single-user number, but each individual request crawls at 3.62 tok/s. That is a batch result, not a chat-latency one.
Memory footprint
GPUwerk has not measured the total MXFP4 weight size, so this page does not state one, and without it the KV-cache headroom against the Spark's 121.6 GiB of addressable memory cannot be stated either. What is known is the active side: gpt-oss-120b activates roughly 5B parameters per token, about 3 GB of weight reads (arithmetic on the benchmarks page, not a measurement). That is why decode lands near 60 tok/s instead of the low single digits a dense 120B model would give.
How to run it
Console preset. The console lists a GPT-OSS 120B preset: llama.cpp serving Unsloth's UD-Q4_K_XL GGUF as model gptoss-120b, with a 32,768-token context split across 2 parallel slots, on port 8888 and behind the instance's HTTPS endpoint. A deploy succeeds only on a node that has these weights staged; if none has, the deploy is refused with a message saying so and nothing is charged. GPUwerk has not yet measured this preset, and its console speed figure is a bandwidth estimate.
Yourself, for one user. On a base image (DGX OS or Ubuntu), build llama.cpp for GB10 and serve the GGUF directly, following the llama.cpp doc:
cd ~/llama.cpp/build ./bin/llama-server \ -hf <gpt-oss-120b-gguf-repo> \ --host 0.0.0.0 \ --port 30000
Replace the placeholder with a gpt-oss-120b MXFP4 GGUF repository; the other flags match the llama.cpp doc's serving example. Result: once the download and load finish, curl http://127.0.0.1:30000/v1/models on the instance lists the model.
Yourself, for a shared endpoint. Use vLLM's CUDA 13 container from the vLLM doc:
docker run -d --name vllm \
--gpus all --ipc=host --restart unless-stopped \
-p 8000:8000 \
-e HF_TOKEN="$HF_TOKEN" \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:cu130-nightly \
vllm serve openai/gpt-oss-120b \
--gpu-memory-utilization 0.85 \
--max-model-len 32768 \
--max-num-seqs 4
What it is good for on a Spark
Single-user, llama.cpp: 60.5 tok/s is comfortably above reading speed, so it works as a local coding assistant or chat interface for one person, with the caveat that llama.cpp's batching is weaker than vLLM's once a second person joins.
Shared endpoint, vLLM: the single-request speed drops to 33.5 tok/s, still usable for one person, but the value case is the 862.8 tok/s aggregate at 256 concurrent, which suits overnight batch classification, document extraction or an agent fleet where no individual request needs to feel instant.
Prefill at roughly 1,900 to 1,950 tok/s across both engines makes it a good fit for long-prompt, short-answer work: summarizing a contract or classifying a ticket against a large system prompt finishes quickly regardless of which engine serves the decode.
Try it
Rent a Spark at $0.79/hour, billed per minute against a prepaid balance (see the pricing page), and run either command above against your own prompts before trusting any number on this page. If you would rather have the endpoint set up and the flags tuned for you, GPUwerk's first engagement covers that.
Comparing with your own GPU or Mac? FitLocally's page for gpt-oss-120b lists which graphics cards, Macs and mini PCs fit it at each quantization, with estimated speeds. FitLocally is a separate site I publish; its DGX Spark figures come from these guides.
FAQ
What single-user speed does gpt-oss-120b get on a DGX Spark?
About 60.5 tok/s decode under llama.cpp, or 33.5 tok/s under vLLM. Both are single-stream at MXFP4 on one GB10, measured by third parties and cited on GPUwerk's benchmarks page.
Which engine should I use for gpt-oss-120b on a Spark?
llama.cpp for one user: it is roughly 80% faster single-stream. vLLM once more than one person shares the endpoint, since its aggregate throughput reaches 862.8 tok/s at 256 concurrent against 33.5 tok/s for one request.
Does gpt-oss-120b fit in 128 GB alongside other work?
GPUwerk has not measured or published its total MXFP4 weight footprint, so treat that as not measured rather than assume a number. Its active parameter count is roughly 5B, which is why decode speed is closer to a small dense model than a 120B one.