DGX Spark/Guides/llama.cpp vs vLLM vs SGLang on a DGX Spark
Serving

llama.cpp vs vLLM vs SGLang on DGX Spark: tok/s and flags

By Samuel Seidel · Published September 9, 2026 · Updated 24 October 2026

On one DGX Spark, vLLM decodes gpt-oss-120b at 60.7 tok/s single-stream and Ollama at 42.1 in the same forum harness. At 8 concurrent requests vLLM reaches 116 to 153 tok/s in total, while Ollama stays at 42.5. llama.cpp’s own DGX Spark table gives 58.7 tok/s for the same model. So for one user the engines are close, and for a team vLLM wins. This page gives the numbers, a decision table, and the build and startup commands for llama.cpp, vLLM and SGLang on GB10.

The numbers

Model Ollama, 1 user vLLM, 1 user Ollama, 8 users (total) vLLM, 8 users (total) llama.cpp, 1 user
gpt-oss-120b42.160.742.5116 to 153 (advantage)58.7 (MXFP4)
Qwen3-Coder-Next59.274.860.4289.0 (advantage)not in table
Gemma 4 26B-A4B + MTP49.654.951.9303.6 (advantage)not in table
Qwen3-30B-A3B85.277.187.5313.4 (advantage)not in table

Decode in tok/s on one Spark, reported, not GPUwerk-measured. Ollama and vLLM columns: one harness, 400 max output tokens, Ollama at Q4 and a vLLM 0.26.1 nightly at 4-bit (NVIDIA forum, 10 August 2026). The author notes the concurrent figures moved by up to 30% between runs. The post’s title names llama.cpp, but it reports no llama.cpp figures, so the llama.cpp column comes from the maintainers’ DGX Spark table (llama-bench tg32, build 7941, February 2026), a different harness. Read 24 October 2026. More rows are on the benchmarks page.

Time to first token in the same post, Ollama against vLLM: 215 against 104 ms on Qwen3-Coder-Next, 511 against 94 ms on Gemma-4-26B-A4B, and 449 against 131 ms on gpt-oss-120B.

llama.cpp was not in that table, but it batches too. In the maintainers' llama-batched-bench run on gpt-oss-120b (512-token prompts, 32 generated tokens), generation rose from 57.1 tok/s with one sequence to 159.2 with 8 and 353.3 with 32 (llama.cpp DGX Spark results, build 7941). That is a different harness from the forum post, so do not line the numbers up against vLLM's.

GPUwerk-measured vLLM scaling

On 2026-09-02 GPUwerk ran vLLM 0.27.1 with Qwen3-Coder 30B-A3B in AWQ 4-bit on one GB10 node. A single request decoded at 80.9 tok/s. At 256 concurrent requests the aggregate was 3,235 tok/s, with no failed requests at any level. The raw log is public, and the full tables are on the benchmarks page.

This is the only GPUwerk-measured engine data on this page. GPUwerk has not measured llama.cpp, Ollama or SGLang on a Spark, so every number for those engines above is a third-party report.

Which engine

Situation Pick Why
One user, or a coding agent on your own boxllama.cpp (or Ollama)Single-stream speed is close to vLLM (58.7 against 60.7 tok/s on gpt-oss-120b), it serves GGUF files, and it starts faster.
A team, or many concurrent requestsvLLMContinuous batching: 116 to 153 tok/s total at 8 users on gpt-oss-120b, against 42.5 for Ollama, which does not batch those requests.
Two SparksvLLM, tensor parallelThe vLLM recipes for models that need two GB10 nodes use TP=2 over NCCL. llama.cpp spans nodes only through RPC, which splits layers and has no tensor parallelism.
Fast restarts and model swapsllama.cppvLLM spends time on weight loading, torch.compile and CUDA graph capture before it serves. See load times below.
Heavy shared prefixesSGLang, after a testRadixAttention prefix caching can help, but there is no same-harness SGLang figure here, so measure it on your traffic.

Load times

vLLM starts slower. A vLLM log posted in an NVIDIA forum thread (1 March 2026, gpt-oss MXFP4 across two Sparks) shows 36.8 seconds to load weights, 50 seconds for model loading in total and 8 seconds of torch.compile. That run then failed during memory profiling, so it is not a full startup time; CUDA graph capture comes after it. eugr, who maintains a widely used Spark vLLM build, wrote that vLLM has “painfully slow startup times compared to llama.cpp” (NVIDIA forum, 3 January 2026). On the llama.cpp side, a 108.72 GB GLM-5.3-Flash file loaded and was ready in 85 seconds at a 262,144-token context (ai-muninn, 30 August 2026). No source measures both engines on the same model, so treat this as direction, not a ratio.

Two Sparks: vLLM, not llama.cpp RPC

llama.cpp can split a model across two Sparks with its RPC backend, but it is limited. In one NVIDIA forum report (17 December 2025), Qwen3-235B-A22B at UD-Q4_K_XL went from about 1.8 tok/s on one Spark to about 12.5 tok/s on two. It ran over TCP/IP rather than NCCL, split layers between nodes, needed -fit off to avoid a crash, and the author lists open llama.cpp issues for tensor parallelism over RPC. vLLM runs tensor parallelism over NCCL instead. For example, the vLLM recipe for DeepSeek-V4-Flash uses two GB10 nodes with TP=2, and a forum run of it reports about 44 tok/s single-stream (NVIDIA forum, May 2026). GPUwerk rents two cabled Sparks as a cluster for $1.79/hour.

The decision guide covers the throughput against latency tradeoff at a high level. The sections below are the commands.

llama.cpp: build once, serve GGUF

llama.cpp needs building from source on GB10; there's no prebuilt wheel that targets sm_121 correctly. Following llama.cpp's CUDA build instructions (NVIDIA's earlier DGX Spark playbook for llama.cpp is no longer listed, checked 11 October 2026):

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp && cd ~/llama.cpp
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121a-real
cmake --build build --config Release --target llama-server -j

The architecture flag is the detail people copy wrong: GB10 is Blackwell at sm_121, not the datacenter Blackwell parts, so a generic 90 (Hopper) or 120 (desktop Blackwell) from an unrelated guide will build clean and then underperform or miss the GPU path entirely. Use 121a-real. The build takes 5 to 10 minutes; llama-server lands in build/bin/.

Serving a GGUF checkpoint with -hf pulls it from Hugging Face on first run and caches it under ~/.cache/huggingface/hub:

./build/bin/llama-server \
  -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
  --host 0.0.0.0 \
  --port 30000

By default llama-server tries to fit the model's full context window with room for 4 concurrent requests, backing off automatically if that doesn't fit; set --ctx-size explicitly if you want a firm ceiling. Multi-modal checkpoints need --mmproj pointed at the matching projector unless -hf fetched it for you. See the llama.cpp docs page for the full walkthrough, including speculative decoding with an MTP draft head via --spec-type draft-mtp.

vLLM: container, not pip install

pip install vllm in a fresh venv on GB10 can look like it succeeded and still leave you with a server that won't start, or one that silently runs on CPU. The aarch64 PyPI resolution has historically pulled CPU-only PyTorch, and even with CUDA-enabled PyTorch in place, FlashInfer's JIT compilation for sm_121 needs CUDA 12.9 or newer. Use the CUDA 13 container instead, per vLLM's own DGX Spark writeup:

docker run -d --name vllm \
  --gpus all --ipc=host --restart unless-stopped \
  -p 8000:8000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:cu130-nightly \
  vllm serve Qwen/Qwen3.8-27B-FP8 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 32768 \
    --max-num-seqs 4

--ipc=host gives the container the host's shared memory segment, without which larger models fail on worker startup. Pin the cu130-nightly tag to a specific release once you've validated a build; it moves, and a restart weeks later can silently pick up a different one. --gpu-memory-utilization is a fraction of total memory, not free memory, so a value that worked yesterday can fail today if something else is resident, per the vLLM memory-math guide. --max-num-seqs around 4 favors single-user latency; raise it once you're serving a team and throughput across users matters more than any one person's speed.

NVIDIA also publishes a vLLM container on NGC, nvcr.io/nvidia/vllm:<release>-py3. Release 26.09 ships vLLM 0.29.0 on CUDA 13.4.1, and its release notes list DGX Spark under known issues: Nemotron Nano V3 and Super V3 NVFP4 models need --max-num-seqs 4 there (read 24 October 2026). Dendro Logic ran its Spark concurrency sweep on nvcr.io/nvidia/vllm:26.03-py3. Newer models often land first in community Spark builds such as eugr’s, so check the model’s recipe before you pick an image.

SGLang: the third option, less charted on this hardware

SGLang targets the same territory as vLLM: high-throughput serving with continuous batching, plus its own RadixAttention prefix-caching scheme. It's worth knowing about even though GPUwerk's own published benchmarks don't yet cover it on GB10 the way they cover vLLM and llama.cpp. Its Docker path looks structurally similar to vLLM's:

docker run -d --name sglang \
  --gpus all --ipc=host --restart unless-stopped \
  -p 30000:30000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3.8-27B-FP8 \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.85

--mem-fraction-static is SGLang's equivalent of vLLM's --gpu-memory-utilization, and the same caution applies: check what's already resident before trusting a fraction that worked on an empty box. Blackwell and sm_121 support in SGLang's release cadence moves independently of vLLM's, so confirm the image tag you pull actually targets GB10 correctly before relying on it. If in doubt, verify with a real completion request (see the checks below) rather than trusting a clean startup log.

Where SGLang earns a look over vLLM: in some published community comparisons, RadixAttention's prefix caching helps workloads with heavy shared-prefix traffic (a coding agent replaying the same system prompt across many requests) more than vLLM's own prefix caching does. That's a workload-shape question, not a hardware one, and it's worth testing against your own traffic pattern rather than assuming either engine wins in general. Install steps, flags and current versions are in SGLang on a DGX Spark, and NVIDIA’s own engine in TensorRT-LLM on a DGX Spark.

GB10 (sm_121) engine status as of October 2026

The GB10 is compute capability 12.1, and several FP4 paths were written for the datacenter parts first. What I could confirm on 2026-10-11:

Verifying any of the three the same way

All three speak the same /v1/chat/completions shape, so the same curl check works regardless of which you picked:

# confirm it's loaded and answering
curl http://localhost:<port>/v1/models

curl http://localhost:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<served-model-name>",
    "messages": [{"role": "user", "content": "Explain KV cache in two sentences."}]
  }'

Anything built against an OpenAI-compatible client (a coding agent, Open WebUI or a LiteLLM gateway) works unchanged against whichever port you picked. That portability is exactly why the choice between engines is safe to revisit later: swapping the backend behind a gateway doesn't touch anything downstream of it.

Which one, in practice

Book a first engagement if you want help benchmarking a specific model across engines before committing. For Ollama’s own install and settings on this box, see Ollama on a DGX Spark; Ollama vs vLLM compares the two engines on RTX cards, and NVIDIA NIM on a DGX Spark covers NVIDIA’s packaged containers.

FAQ

Is vLLM or llama.cpp faster on a DGX Spark?

For one user they are close. On gpt-oss-120b, llama.cpp build 7941 reports 58.7 tok/s and vLLM 0.26.1 reports 60.7 tok/s in a forum harness. With several users vLLM pulls ahead because it batches requests: 116 to 153 tok/s aggregate at 8 concurrent, against 42.5 for Ollama in the same harness.

How long does vLLM take to start on a DGX Spark?

Longer than llama.cpp. One reported two-node vLLM log for gpt-oss shows 37 seconds to load weights, 50 seconds for model loading and 8 seconds of torch.compile, before CUDA graph capture. A community vLLM maintainer calls its startup painfully slow compared with llama.cpp. llama.cpp loaded a 108.72 GB GLM-5.3-Flash file and was ready in 85 seconds in one report.

Which engine should I use on two DGX Sparks?

vLLM with tensor parallelism over NCCL. The vLLM recipes for large models such as DeepSeek-V4-Flash use two GB10 nodes with TP=2. llama.cpp can span two Sparks only through its RPC backend, which splits layers and has no tensor parallelism; one report got about 12.5 tok/s on Qwen3-235B-A22B that way.

Is there an official vLLM container for DGX Spark?

NVIDIA publishes a vLLM container on NGC, nvcr.io/nvidia/vllm. Release 26.09 ships vLLM 0.29.0 on CUDA 13.4.1, and its release notes cover DGX Spark in the known issues. The upstream vllm/vllm-openai CUDA 13 image is the other common choice. Do not pip install vLLM on GB10.

First top-up: pay $20, get $30 in credit

Test all three engines on the same hardware.

A dedicated Spark from $0.79/h, billed by the minute, with no shared tenants to skew the numbers.

Create account Read the vLLM guide