llama.cpp on a DGX Spark: build, serve, when to pick it
By the end of this page you have llama.cpp built for the GB10's sm_121 architecture and llama-server answering OpenAI-compatible requests on your DGX Spark. For one user at a time, llama.cpp is clearly faster than vLLM on this hardware; the benchmark numbers are below.
You need:
- A running GPUwerk instance on a base image (DGX OS or Ubuntu 24.04). If you only want a llama.cpp endpoint and no build, deploy a console preset instead: GPT-OSS 120B, Qwen3.6 35B-A3B and Qwen3.8 Flash Next run on llama.cpp with the weights already staged.
- About 15 minutes for the build, plus the model download.
- No SSH key. GPUwerk configures the machine, and the console's browser terminal works without one, so you can run every command on this page there. Add your own key only if you want your own SSH client, scp or port forwarding.
Why llama.cpp over vLLM here
vLLM's continuous batching is built to serve many concurrent users; llama.cpp is built to serve one user fast. On the DGX Spark's 273 GB/s of memory bandwidth, that difference shows up directly in single-stream decode. User eugr's community benchmarks on the llama.cpp GitHub discussions measured gpt-oss-120b MXFP4 at roughly 60.5 tokens/second decode and 1,956 tokens/second prefill under llama.cpp, against roughly 33.5 tokens/second decode under vLLM on the same model and hardware. Full figures and more reference points are on the benchmarks page.
That is roughly an 80% single-stream speedup for the cost of a source build. The trade-off: llama.cpp batches less well once several people share the box, which is where vLLM takes over.
The rule of thumb: one interactive user, or a coding assistant on your own machine, run llama.cpp. A team sharing one endpoint, run vLLM.
Build with CUDA for GB10
These steps follow llama.cpp's CUDA build instructions, built for the GB10's sm_121. NVIDIA's earlier DGX Spark playbook for llama.cpp is no longer listed on build.nvidia.com (checked 11 October 2026). Run them in the browser terminal or over SSH.
- Install the build dependencies and clone the repository. You should end up in a
~/llama.cppdirectory. On a GPUwerk instance only/workspacesurvives a rebuild (see the tenant environment), so clone into/workspace/llama.cppinstead if you want to keep the build.
sudo apt update sudo apt install -y git clang cmake libcurl4-openssl-dev libssl-dev git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp cd ~/llama.cpp
- Configure CMake with CUDA and the GB10's
sm_121architecture, then build only the server target:
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON \ -DCMAKE_CUDA_ARCHITECTURES=121a-real cmake --build build --config Release --target llama-server -j
- Wait 5 to 10 minutes. When the build finishes, you should see
llama-serverinbuild/bin/.
The architecture flag is the part people get wrong. GB10 is Blackwell at compute capability sm_121, not the datacenter Blackwell parts. A generic -DCMAKE_CUDA_ARCHITECTURES=90 (Hopper) or 120 (desktop Blackwell) copied from an unrelated guide will build without error and then either fail to use the GPU correctly or underperform. Use 121a-real.
Serve a model
llama.cpp loads GGUF checkpoints. With -hf it downloads the model from Hugging Face into ~/.cache/huggingface/hub the first time and reuses it after. NVIDIA's playbook uses Qwen3.6-35B-A3B, which fits in 128 GB with room for a long context. Start the server; the log should end with the server listening on port 30000:
cd ~/llama.cpp/build ./bin/llama-server \ -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 \ --port 30000
By default llama-server tries to fit the model's full context window with room for 4 concurrent requests, and scales down if that does not fit. To use another GGUF, pass --model /path/to/file.gguf instead of -hf. Multimodal checkpoints then also need --mmproj pointed at the matching projector file; -hf fetches it for you.
Port 30000 is public. Each GPUwerk instance has its own public IPv4 address, so --host 0.0.0.0 makes the server reachable from the internet. Add --api-key <a-long-random-string> to require a bearer key, or use --host 127.0.0.1 and reach it through an SSH tunnel.
Test the endpoint
Large GGUFs take a minute or more to load. The first command returns once the health check passes. The second should return JSON with a choices list containing the answer:
# wait for the server to be ready timeout 900 bash -c 'until curl -sf http://127.0.0.1:30000/health > /dev/null 2>&1; do sleep 5; done' # a real completion curl -X POST http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL", "messages": [{"role": "user", "content": "Explain KV cache in two sentences."}], "max_tokens": 200 }'
The endpoint speaks the same /v1/chat/completions API as vLLM, so any OpenAI client works unchanged. Point Open WebUI, Continue.dev or a LiteLLM gateway at http://<spark-host>:30000/v1, where <spark-host> is your instance's hostname, <name>.gpuwerk.com.
Speculative decoding with MTP
Models that ship an MTP (multi-token prediction) draft head, like the Qwen3.6-35B-A3B-MTP checkpoint above, can use it for speculative decoding: the server drafts several tokens ahead and verifies them in one pass.
./bin/llama-server \
-hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
--host 0.0.0.0 \
--port 30000 \
--chat-template-kwargs '{"preserve_thinking": true}' \
-np 1 \
--spec-type draft-mtp \
--spec-draft-n-max 2
Check the model card first. Unsloth's card for this GGUF (checked 11 October 2026) uses a 2-token draft and one slot: MTP does not yet work with more than one parallel slot (-np) or with --mmproj. --spec-type draft-mtp works only with a checkpoint that ships MTP weights; on one that does not, the server fails to start instead of falling back. The preserve_thinking flag is unrelated to MTP: it keeps earlier reasoning blocks in the conversation history, which matters for agent workflows on Qwen's thinking models.
Context length and next steps
llama.cpp allocates the model's maximum supported context if memory allows; set it yourself with --ctx-size. For coding agents, NVIDIA's guidance is at least 32,768 tokens and preferably 100,000 or more, because tool calls and file contents use context quickly.
"If memory allows" is measured against the Spark's unified 128 GB, which the OS and SSH share. Leave a few gigabytes of headroom instead of letting --ctx-size default to the model's maximum. Watch free -g and keep "available" well above zero.
Once it runs, use it like any other OpenAI-compatible endpoint. Put it behind LiteLLM if several scripts or people will call it, or point an editor agent at it with the coding agents guide.