Serve Qwen3 with vLLM on a DGX Spark

Ollama is the fast path to a first token. vLLM is what you run when the Spark stops being your experiment and starts being your team's LLM server: continuous batching, an OpenAI-compatible API, and one process that can hold several concurrent chat sessions without falling over. This is the setup we run in production.

Why vLLM instead of Ollama for team serving

Ollama is built around one request at a time feeling snappy on a single machine. vLLM is built around a queue: it batches incoming requests together on the GPU, which is exactly the workload shape of "five people on the team hitting the same endpoint." You lose a bit of Ollama's zero-config simplicity and gain proper concurrency, an API surface every OpenAI client already speaks, and quantization formats (AWQ, GPTQ) tuned for throughput rather than just disk size. If you're the only user, either works. If you're standing up a shared endpoint, use vLLM.

1, Install vLLM

DGX OS ships CUDA and the NVIDIA driver stack preinstalled, so this is a normal Python install, no driver wrangling.

# a clean venv keeps vLLM's pinned dependencies off your system Python
python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate

# install vLLM
pip install --upgrade pip
pip install vllm

# sanity check
python -c "import vllm; print(vllm.__version__)"

vLLM tracks CUDA and PyTorch releases closely, if pip install vllm ever fails to resolve on Arm, check the vLLM installation docs for the GB10 / aarch64-specific wheel or build instructions for your exact version.

2, Serve Qwen3-32B-AWQ

AWQ is a 4-bit weight quantization that keeps quality close to full precision while cutting the weight memory footprint to roughly a quarter of FP16, about 20 GB for a 32B model, a good default for a 128 GB unified-memory box where you'd still like headroom for KV cache and concurrent users.

# serve with an OpenAI-compatible API on the LAN
vllm serve Qwen/Qwen3-32B-AWQ \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 32768

First launch downloads the model from Hugging Face (tens of GB) and compiles CUDA graphs, so give it a few minutes before the endpoint answers. Once it's up:

# confirm the model is loaded
curl http://localhost:8000/v1/models

# send a chat completion, OpenAI SDK format
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-32B-AWQ",
    "messages": [{"role": "user", "content": "Explain KV cache in two sentences."}]
  }'

Any OpenAI-compatible client, the openai Python SDK, LangChain, editor plugins, internal scripts, works against this endpoint by pointing base_url at http://spark.local:8000/v1 with any non-empty API key string. Smaller team, tighter memory budget, or want a reasoning-tuned variant instead? Swap the model name for something like Qwen/Qwen3-14B-AWQ or check the Qwen org on Hugging Face for the current AWQ releases, the family gets updated regularly.

3, Connect Open WebUI as a chat frontend

Your team doesn't want to curl a JSON blob every time they have a question. Open WebUI gives you a ChatGPT-style interface that talks to any OpenAI-compatible endpoint, including the one you just started.

# run Open WebUI in Docker, pointed at the local vLLM endpoint
docker run -d \
  --name open-webui \
  --network host \
  -e OPENAI_API_BASE_URL=http://localhost:8000/v1 \
  -e OPENAI_API_KEY=sk-local \
  -v open-webui:/app/backend/data \
  --restart unless-stopped \
  ghcr.io/open-webui/open-webui:main

Open http://spark.local:8080, create the first account (it becomes the local admin), and Qwen3 should already show up as a selectable model, Open WebUI queries /v1/models on startup. No GPU work happens in the container; it's just a frontend, so --network host keeps things simple rather than fighting Docker's bridge networking for the port.

4, Keep it alive with systemd

A vllm serve command in a terminal dies the moment you close the SSH session. For anything you'd call "team infrastructure," run it as a service that starts on boot and restarts on crash.

# /etc/systemd/system/vllm-qwen3.service
sudo tee /etc/systemd/system/vllm-qwen3.service > /dev/null <<'EOF'
[Unit]
Description=vLLM serving Qwen3-32B-AWQ
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=youruser
WorkingDirectory=/home/youruser
Environment="PATH=/home/youruser/vllm-env/bin:/usr/bin:/usr/local/bin:/bin:/usr/local/sbin:/usr/sbin"
ExecStart=/home/youruser/vllm-env/bin/vllm serve Qwen/Qwen3-32B-AWQ --host 0.0.0.0 --port 8000 --max-model-len 32768
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# enable and start it
sudo systemctl daemon-reload
sudo systemctl enable --now vllm-qwen3

# tail logs while it warms up
journalctl -u vllm-qwen3 -f

Swap youruser and the venv path for your actual setup. From here, a reboot, a kernel update, a power blip, brings the model back up on its own instead of paging you at 2am.

Basic LAN security notes

The commands above bind vLLM to 0.0.0.0, which means anyone on your network segment can hit port 8000 with no authentication at all. That's fine for a quick test; it's not fine left running. Before you call this done:

• Firewall the port. Use ufw or your router's rules to restrict 8000 (and 8080 for Open WebUI) to known IPs or a VPN subnet, not the whole office LAN.
• Put a reverse proxy in front of it. Nginx or Caddy with basic auth or an API-key check turns an open endpoint into a gated one with a few lines of config, do this before sharing the URL outside your immediate team.
• Don't expose it to the internet directly. If you need remote access, put it behind a VPN (Tailscale and WireGuard both work well here) rather than port-forwarding 8000 on your router.
• Rely on Open WebUI's own accounts for the chat frontend: it has login and admin roles built in, but remember that doesn't protect the raw /v1 API if it's still reachable directly.

None of this is exotic; it's the same posture you'd take with any internal API that happens to also be a company knowledge source.

What to actually expect, performance-wise

The GB10's 273 GB/s of unified memory bandwidth is the number that matters here, and it's worth being honest about what it means: single-stream generation, one person, one prompt, is measured, not blistering, because token generation is bandwidth-bound. Where the Spark earns its keep is exactly what vLLM is built for: continuous batching means several concurrent users share the same forward passes, so aggregate throughput across a small team holds up far better than the single-stream number suggests. Don't buy this setup expecting a snappier single chat than a beefy consumer GPU. Buy it expecting to hold a 32B (or larger) model in memory at once and serve a handful of people off it concurrently, that's the actual trade this hardware makes.

Next steps

If you haven't done the OS-level setup yet, SSH hardening, static IP, choosing between Ollama and vLLM in the first place, start with our DGX Spark setup guide. This guide assumes that groundwork is already in place.