Docs
How to get a model serving on hardware you control, written by the people who keep the fleet running. Every command here is one we run ourselves.
Start here
From a key pair on your laptop to a shell on a Spark.
Create an SSH key and connect
ed25519 on macOS, Linux and Windows, the agent, a config alias, port forwarding, and fixes for the three errors everyone hits.
DGX Spark setup: unboxing to first token
DGX OS first boot, drivers, and serving your first model on hardware sitting on your own desk.
Which models fit in 128 GB
The weights-plus-cache arithmetic, why MoE models win on this hardware, and current Hugging Face repos that fit. With the ones that do not.
Serving models
Turning a box with weights on it into an endpoint other software can call.
vLLM: install, flags and memory math
Why pip install vllm fails on GB10, the CUDA 13 container that works, and how to size gpu-memory-utilization against unified memory.
Serve Qwen3 with vLLM end to end
The full walkthrough: install, serve, attach Open WebUI, and keep it alive with a systemd unit that survives reboots.
Benchmarks: what to actually expect
Published tokens per second across vLLM and llama.cpp, why prefill looks great and decode looks ordinary, and what concurrency really does.
Interfaces and gateways
The layer between your endpoint and the people who will use it.
Set up Open WebUI
A ChatGPT-style interface on your own hardware: Docker install, connecting it to vLLM, locking down signups, RAG over your documents, HTTPS.
Set up LiteLLM as your AI gateway
One OpenAI-compatible URL in front of everything, with per-user virtual keys, budgets, rate limits and fallback routing.
Coding agents against a local endpoint
Pointing editor assistants and agent frameworks at your own model instead of a metered API.
Training and tuning
When prompting has run out of road and the model needs to learn your data.
Fine-tune a model on your company's data
LoRA on 70B-class models fits in 128 GB of unified memory, with the training set staying on the node's local NVMe.
Serving a LoRA adapter with vLLM
Hot-swapping adapters on one base model, and when a merged checkpoint is the better answer.
Choosing hardware
Whether this machine is the right one, argued honestly.
What to know before buying a DGX Spark
Power, noise, networking two units, and total cost of ownership against a 5090 rig or the cloud.
DGX Spark vs Mac Studio
The Mac wins decode, the Spark wins prefill and CUDA portability. Updated for the M5 Ultra.
DGX Spark vs RTX 5090
32 GB of fast VRAM against 128 GB of capacity. Not a close fight on speed, and the whole question is whether your model fits.
Background reading
Longer arguments about why any of this is worth doing on your own hardware.
The best models you're not allowed to send your data to
Which large open models fit in 128 GB or 256 GB, and how far behind the frontier they really are.
Shadow AI and where your data actually goes
What staff paste into consumer chatbots, and what it costs when a regulator asks about it.
An AI employee needs an office. Give it one you own.
Why an agent that reads your inbox belongs on hardware you control.
Automate the boring 80%
Batch document processing on a flat-rate machine: the marginal token is free, so run the model over everything.
Something missing?
If you hit a wall these docs did not cover, tell us and we will write it up. The gaps our customers find are the ones worth filling first.
Deploy a Spark Request a guide