vLLM on a DGX Spark: install, flags and memory math

This page gets you a vLLM server answering OpenAI-compatible requests on a DGX Spark, with memory and context flags sized for GB10's unified memory. vLLM is the engine to use once several people share the box: continuous batching lets concurrent requests share forward passes instead of queueing. If you only need a running endpoint, the console's Qwen3 Coder 30B preset gives you one with no install.

The zero-install path

The console's vLLM preset is Qwen3 Coder 30B. Deploy it and none of the install below applies. The weights are already staged on the node and mounted read-only, vLLM starts with the instance, and the model is served as qwen3-coder-30b with a 32,768-token context. Your OpenAI-compatible endpoint is the HTTPS endpoint shown on the instance page (it looks like https://app-<instance-id>.gpuwerk.com) plus /v1, and it answers once the instance is running. Every request needs a GPUwerk API key with the inference scope from the account that owns the instance. Skip to verify and measure.

The rest of the page is for a base image where you serve a model GPUwerk does not stage, such as Llama 3.3 70B or any other Hugging Face checkpoint.

Why pip install vllm fails on GB10

The GB10 is Blackwell with compute capability sm_121 on an aarch64 host, and that combination breaks the obvious install two separate ways. On aarch64, the PyPI dependency resolution has historically pulled a CPU-only PyTorch build, so vLLM installs cleanly and then cannot see the GPU at all. Even with a CUDA-enabled PyTorch in place, sm_121 kernel support has to be present in the build you are running, and FlashInfer's just-in-time compilation for sm_121 needs CUDA 12.9 or newer, which is why a container pinned to CUDA 12.8 fails at load time rather than at install time.

The net effect is that pip install vllm in a fresh venv can look successful and still leave you with a server that will not start, or one that runs entirely on the CPU at a few tokens per second. Use a container built for CUDA 13.

Install: the CUDA 13 container

The vLLM project's DGX Spark writeup recommends the CUDA 13 build of the official server image rather than a pip install. On a DGX Spark of your own, Docker is already set up with the NVIDIA runtime, so this is one command:

docker run -d --name vllm \
  --gpus all --ipc=host --restart unless-stopped \
  -p 8000:8000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:cu130-nightly \
  vllm serve Qwen/Qwen3.8-27B-FP8 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 32768 \
    --max-num-seqs 4

Two details matter more than they look. --ipc=host gives the container the host's shared memory segment; without it, vLLM's workers fail on larger models. Mounting ~/.cache/huggingface means a container restart does not download tens of gigabytes of weights again. When the log shows the server listening on port 8000, continue to verify and measure.

Memory math on unified memory

The Spark has 128 GB of memory shared between CPU and GPU, roughly 121.6 GiB of it addressable, and this changes how --gpu-memory-utilization behaves compared to a discrete card. It is a fraction of total memory, not of free memory, and vLLM refuses to start if the fraction you asked for exceeds what is currently free. On a box where something else is already resident, a value that worked yesterday can fail today.

Work backwards from three numbers: weights, KV cache, and headroom.

• Weights are roughly parameters × bytes per parameter. A 70B model at FP8 is about 70 GB, at 4-bit about 37 GB, and a 27B at 4-bit about 15 GB.
• KV cache is what serves concurrency, and it scales with context length times concurrent sequences. This is the number you are actually buying with the remaining memory.
• Headroom is activations, the CUDA context, and anything else on the node. Leave some.

Measured numbers from GPUwerk's own Sparks, so you can calibrate. Llama 3.3 70B AWQ is about 37 GiB of weights. At --gpu-memory-utilization 0.85 on an otherwise empty GB10 that leaves a 207,120-token KV cache; at 0.90 it leaves 227,712. Qwen3-Coder 30B-A3B, being far smaller in weights, gets 924,912 tokens of cache at 0.85 and 994,064 at 0.90. That cache is what you are spending the remaining memory on, and it is what concurrency consumes.

One trap is worth naming, because it cost me a set of published benchmarks: --gpu-memory-utilization is a fraction of total memory, not of free memory, and vLLM refuses to start if the resulting figure exceeds what is currently free. On a box with 35 GiB already held by something else, 0.85 asks for 103 GiB, finds 46, and fails at startup. The fix is to empty the box, not to lower the number: a fraction chosen to fit around another workload quietly leaves your users with a third less cache. Going much past 0.9 trades a small concurrency gain for out-of-memory crashes under load.

The flags worth tuning

vLLM has a long argument list; four flags do most of the work.

--gpu-memory-utilization, covered above. Start at 0.85 on a dedicated box, lower it if the server will not start.

--max-model-len caps the context window. Setting it to the model's full advertised length is the most common reason a server refuses to boot: vLLM preallocates KV cache for it, and a 262K context on a 27B model wants more memory than you have. Set it to the longest prompt you actually send, then raise it if you hit the ceiling.

--max-num-seqs caps concurrent sequences in a batch. vLLM's DGX Spark guidance suggests keeping this low, around 4, for interactive single-user work, because a smaller batch means each request gets a larger share of memory bandwidth and feels faster. Raise it when you are serving a team and total throughput matters more than any one person's latency.

--served-model-name sets the string clients pass as model. Without it, the API name is the full Hugging Face path or the local directory, and every client config has to repeat it. Set a short alias and your gateway config stays readable.

Picking a quantization

On Blackwell, NVFP4 is the format worth preferring when a model ships in it. It is a 4-bit floating point format with hardware support on this generation, and NVIDIA publishes NVFP4 builds of the Nemotron family on Hugging Face. AWQ is the well-supported 4-bit integer alternative with much broader model coverage, and it is what you will find for most community releases. GGUF is a llama.cpp format: vLLM's support for it exists but is not the fast path, so if the only quantization you can find is GGUF, run it under llama.cpp instead of fighting vLLM.

Rule of thumb for a 128 GB box: 4-bit is the default, FP8 is worth it when the model is small enough that you have memory to spare, and unquantized FP16 only makes sense below about 30B parameters.

Verify and measure

For a console preset:

1. Create an API key in the console with only the inference scope.
2. Copy the HTTPS endpoint from the instance page.
3. List the models with the key as a bearer token. You should see JSON whose data list contains qwen3-coder-30b:

curl https://app-<instance-id>.gpuwerk.com/v1/models \
  -H "Authorization: Bearer $GPUWERK_API_KEY"

For an OpenAI-compatible SDK, use the HTTPS endpoint plus /v1 as the base URL and your GPUwerk key as its API key. The gateway checks ownership on every request, and a revoked key stops working immediately. Keep the key out of frontend JavaScript and URLs. Inside the instance (browser terminal or SSH), the preset also answers at http://localhost:8888/v1 without the gateway key.

For your own install on port 8000, confirm the model is loaded and answering before you point anything at it. The first call should list the model; the second should return JSON with the answer in choices:

# what is served, and under what name
curl http://localhost:8000/v1/models

# a real completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.8-27B-FP8",
    "messages": [{"role": "user", "content": "Explain KV cache in two sentences."}]
  }'

For throughput numbers, use vLLM's own harness rather than timing curl by hand, since it separates prefill from decode:

docker exec -it vllm vllm bench serve \
  --model Qwen/Qwen3.8-27B-FP8 \
  --num-prompts 64 --request-rate 4

Watch two numbers. Decode throughput is bandwidth-bound and modest on this hardware. Prefill is where the Spark does well: vLLM measured roughly 1,900 tokens per second of prefill on a 120B NVFP4 Nemotron model on a Spark, against 22.7 to 23.7 tokens per second of decode. See the benchmark page for more reference points.

Keep it running

On a Spark of your own, --restart unless-stopped in the docker run above covers crashes and reboots, which is most of what a systemd unit would give you. If you run vLLM outside Docker, or want it ordered after other services, the Qwen3 serving guide has a unit file and walks the same setup end to end with Open WebUI attached. On a GPUwerk instance, start the server from the startup hook /workspace/.gpuwerk-init, which runs on every container start; see the tenant environment.