Docs/Serving
Serving

llama.cpp on a DGX Spark: build, serve, when to pick it

By Samuel Seidel · Updated September 4, 2026 · 5 min read

By the end of this page you have llama.cpp built for the GB10's sm_121 architecture and llama-server answering OpenAI-compatible requests on your DGX Spark. For one user at a time, llama.cpp is clearly faster than vLLM on this hardware; the benchmark numbers are below.

You need:

On this page
  1. Why llama.cpp over vLLM here
  2. Build with CUDA for GB10
  3. Serve a model
  4. Test the endpoint
  5. Speculative decoding with MTP
  6. Context length and next steps

Why llama.cpp over vLLM here

vLLM's continuous batching is built to serve many concurrent users; llama.cpp is built to serve one user fast. On the DGX Spark's 273 GB/s of memory bandwidth, that difference shows up directly in single-stream decode. User eugr's community benchmarks on the llama.cpp GitHub discussions measured gpt-oss-120b MXFP4 at roughly 60.5 tokens/second decode and 1,956 tokens/second prefill under llama.cpp, against roughly 33.5 tokens/second decode under vLLM on the same model and hardware. Full figures and more reference points are on the benchmarks page.

That is roughly an 80% single-stream speedup for the cost of a source build. The trade-off: llama.cpp batches less well once several people share the box, which is where vLLM takes over.

The rule of thumb: one interactive user, or a coding assistant on your own machine, run llama.cpp. A team sharing one endpoint, run vLLM.

Build with CUDA for GB10

These steps follow llama.cpp's CUDA build instructions, built for the GB10's sm_121. NVIDIA's earlier DGX Spark playbook for llama.cpp is no longer listed on build.nvidia.com (checked 11 October 2026). Run them in the browser terminal or over SSH.

  1. Install the build dependencies and clone the repository. You should end up in a ~/llama.cpp directory. On a GPUwerk instance only /workspace survives a rebuild (see the tenant environment), so clone into /workspace/llama.cpp instead if you want to keep the build.
sudo apt update
sudo apt install -y git clang cmake libcurl4-openssl-dev libssl-dev

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
  1. Configure CMake with CUDA and the GB10's sm_121 architecture, then build only the server target:
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121a-real
cmake --build build --config Release --target llama-server -j
  1. Wait 5 to 10 minutes. When the build finishes, you should see llama-server in build/bin/.

The architecture flag is the part people get wrong. GB10 is Blackwell at compute capability sm_121, not the datacenter Blackwell parts. A generic -DCMAKE_CUDA_ARCHITECTURES=90 (Hopper) or 120 (desktop Blackwell) copied from an unrelated guide will build without error and then either fail to use the GPU correctly or underperform. Use 121a-real.

Serve a model

llama.cpp loads GGUF checkpoints. With -hf it downloads the model from Hugging Face into ~/.cache/huggingface/hub the first time and reuses it after. NVIDIA's playbook uses Qwen3.6-35B-A3B, which fits in 128 GB with room for a long context. Start the server; the log should end with the server listening on port 30000:

cd ~/llama.cpp/build
./bin/llama-server \
  -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
  --host 0.0.0.0 \
  --port 30000

By default llama-server tries to fit the model's full context window with room for 4 concurrent requests, and scales down if that does not fit. To use another GGUF, pass --model /path/to/file.gguf instead of -hf. Multimodal checkpoints then also need --mmproj pointed at the matching projector file; -hf fetches it for you.

Port 30000 is public. Each GPUwerk instance has its own public IPv4 address, so --host 0.0.0.0 makes the server reachable from the internet. Add --api-key <a-long-random-string> to require a bearer key, or use --host 127.0.0.1 and reach it through an SSH tunnel.

Test the endpoint

Large GGUFs take a minute or more to load. The first command returns once the health check passes. The second should return JSON with a choices list containing the answer:

# wait for the server to be ready
timeout 900 bash -c 'until curl -sf http://127.0.0.1:30000/health > /dev/null 2>&1; do sleep 5; done'

# a real completion
curl -X POST http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL",
    "messages": [{"role": "user", "content": "Explain KV cache in two sentences."}],
    "max_tokens": 200
  }'

The endpoint speaks the same /v1/chat/completions API as vLLM, so any OpenAI client works unchanged. Point Open WebUI, Continue.dev or a LiteLLM gateway at http://<spark-host>:30000/v1, where <spark-host> is your instance's hostname, <name>.gpuwerk.com.

Speculative decoding with MTP

Models that ship an MTP (multi-token prediction) draft head, like the Qwen3.6-35B-A3B-MTP checkpoint above, can use it for speculative decoding: the server drafts several tokens ahead and verifies them in one pass.

./bin/llama-server \
  -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
  --host 0.0.0.0 \
  --port 30000 \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  -np 1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2

Check the model card first. Unsloth's card for this GGUF (checked 11 October 2026) uses a 2-token draft and one slot: MTP does not yet work with more than one parallel slot (-np) or with --mmproj. --spec-type draft-mtp works only with a checkpoint that ships MTP weights; on one that does not, the server fails to start instead of falling back. The preserve_thinking flag is unrelated to MTP: it keeps earlier reasoning blocks in the conversation history, which matters for agent workflows on Qwen's thinking models.

Context length and next steps

llama.cpp allocates the model's maximum supported context if memory allows; set it yourself with --ctx-size. For coding agents, NVIDIA's guidance is at least 32,768 tokens and preferably 100,000 or more, because tool calls and file contents use context quickly.

"If memory allows" is measured against the Spark's unified 128 GB, which the OS and SSH share. Leave a few gigabytes of headroom instead of letting --ctx-size default to the model's maximum. Watch free -g and keep "available" well above zero.

Once it runs, use it like any other OpenAI-compatible endpoint. Put it behind LiteLLM if several scripts or people will call it, or point an editor agent at it with the coding agents guide.

See alsovLLM: when you need to serve a team, not one user See alsoFull benchmark numbers across engines
First top-up: pay $20, get $30 in credit

Try it on real hardware, not a spec sheet.

Deploy a dedicated DGX Spark in Europe, North America or East Asia and build llama.cpp against the actual GB10 GPU.

Create account