vLLM on a DGX Spark: install, flags and memory math

vLLM is the right engine once more than one person is using the box, because continuous batching means concurrent requests share forward passes instead of queueing behind each other. Getting it running on GB10 has one sharp edge that costs people an afternoon, so start with that.

The zero-install path

If you deploy one of our vLLM images from the console, Qwen3 Coder 30B or Llama 3.3 70B, none of the below applies to you. The weights are already staged on the node and bind-mounted read-only, vLLM starts with the container, and your OpenAI-compatible endpoint is live at https://<name>.gpuwerk.com/v1 as soon as the instance is running. Skip to verify and measure.

Everything else here is for the case where you picked a base image and want to serve a model we do not stage, which is the normal reason to be reading this page.

Why pip install vllm fails on GB10

The GB10 is Blackwell with compute capability sm_121 on an aarch64 host, and that combination breaks the obvious install two separate ways. On aarch64, the PyPI dependency resolution has historically pulled a CPU-only PyTorch build, so vLLM installs cleanly and then cannot see the GPU at all. Even with a CUDA-enabled PyTorch in place, sm_121 kernel support has to be present in the build you are running, and FlashInfer's just-in-time compilation for sm_121 needs CUDA 12.9 or newer, which is why a container pinned to CUDA 12.8 fails at load time rather than at install time.

The net effect is that pip install vllm in a fresh venv can look successful and still leave you with a server that will not start, or one that runs entirely on the CPU at a few tokens per second. Use a container built for CUDA 13.

Install: the CUDA 13 container

The vLLM project's own DGX Spark writeup recommends the CUDA 13 build of the official server image rather than a pip install. Docker is already configured with the NVIDIA runtime on DGX OS, so this is a single command:

docker run -d --name vllm \
  --gpus all --ipc=host --restart unless-stopped \
  -p 8000:8000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:cu130-nightly \
  vllm serve Qwen/Qwen3.8-27B-FP8 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 32768 \
    --max-num-seqs 4

Two details in there matter more than they look. --ipc=host gives the container the host's shared memory segment, without which vLLM's workers fail on larger models. Mounting ~/.cache/huggingface means a container restart does not re-download tens of gigabytes of weights, and on a GPUwerk instance you want that cache under /workspace so it survives a stop.

Memory math on unified memory

The Spark has 128 GB of memory shared between CPU and GPU, roughly 121.6 GiB of it addressable, and this changes how --gpu-memory-utilization behaves compared to a discrete card. It is a fraction of total memory, not of free memory, and vLLM refuses to start if the fraction you asked for exceeds what is currently free. On a box where something else is already resident, a value that worked yesterday can fail today.

Work backwards from three numbers: weights, KV cache, and headroom.

• Weights are roughly parameters × bytes per parameter. A 70B model at FP8 is about 70 GB, at 4-bit about 37 GB, and a 27B at 4-bit about 15 GB.
• KV cache is what serves concurrency, and it scales with context length times concurrent sequences. This is the number you are actually buying with the remaining memory.
• Headroom is activations, the CUDA context, and anything else on the node. Leave some.

Our own Llama 3.3 70B image runs at --gpu-memory-utilization 0.65, which is 79 GiB of 121.6: about 37 GiB of weights plus roughly 42 GiB of KV cache, chosen so it still starts on a node with 35 GiB already in use. On a machine dedicated to one model, push toward 0.85 and take the extra cache. Going much past 0.9 tends to trade a small concurrency gain for out-of-memory crashes under load.

The flags worth tuning

vLLM has a long argument list and four of them do most of the work.

--gpu-memory-utilization, covered above. Start at 0.85 on a dedicated box, lower it if the server will not start.

--max-model-len caps the context window. Setting it to the model's full advertised length is the most common reason a server refuses to boot: vLLM preallocates KV cache for it, and a 262K context on a 27B model wants more memory than you have. Set it to the longest prompt you actually send, then raise it if you hit the ceiling.

--max-num-seqs caps concurrent sequences in a batch. vLLM's DGX Spark guidance suggests keeping this low, around 4, for interactive single-user work, because a smaller batch means each request gets a larger share of memory bandwidth and feels faster. Raise it when you are serving a team and total throughput matters more than any one person's latency.

--served-model-name sets the string clients pass as model. Without it, the API name is the full Hugging Face path or the local directory, and every client config has to repeat it. Set a short alias and your gateway config stays readable.

Picking a quantization

On Blackwell, NVFP4 is the format worth preferring when a model ships in it. It is a 4-bit floating point format with hardware support on this generation, and NVIDIA publishes NVFP4 builds of the Nemotron family on Hugging Face. AWQ is the well-supported 4-bit integer alternative with much broader model coverage, and it is what you will find for most community releases. GGUF is a llama.cpp format: vLLM's support for it exists but is not the fast path, so if the only quantization you can find is GGUF, run it under llama.cpp or Ollama instead of fighting vLLM.

Rule of thumb for a 128 GB box: 4-bit is the default, FP8 is worth it when the model is small enough that you have memory to spare, and unquantized FP16 only makes sense below about 30B parameters.

Verify and measure

Confirm the model is loaded and answering before you point anything at it:

# what is served, and under what name
curl http://localhost:8000/v1/models

# a real completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.8-27B-FP8",
    "messages": [{"role": "user", "content": "Explain KV cache in two sentences."}]
  }'

For throughput numbers, use vLLM's own harness rather than timing curl by hand, since it separates prefill from decode:

docker exec -it vllm vllm bench serve \
  --model Qwen/Qwen3.8-27B-FP8 \
  --num-prompts 64 --request-rate 4

Watch two numbers. Decode throughput is bandwidth-bound and will be measured rather than spectacular on this hardware. Prefill is where the Spark looks good: vLLM measured roughly 1,900 tokens per second of prefill on a 120B NVFP4 Nemotron model on a Spark, against 22.7 to 23.7 tokens per second of decode. See the benchmark page for more reference points.

Keep it running

--restart unless-stopped in the docker run above covers crashes and reboots, which is most of what a systemd unit would give you. If you are running vLLM outside Docker, or you want it ordered after other services, a unit file is in the Qwen3 serving guide, which walks the same setup end to end with Open WebUI attached.