Which models fit in 128 GB

The Spark's selling point is capacity: 128 GB of unified memory holds models that will not load on a 32 GB consumer card at any price. The constraint is bandwidth, 273 GB/s, which decides how fast those models generate. Both numbers matter, and picking a model well means knowing which one you are up against.

The arithmetic

Two numbers decide whether a model runs, and you can work both out before downloading anything.

Weights are parameters multiplied by bytes per parameter. FP16 is 2 bytes, FP8 is 1, and the 4-bit formats are roughly half a byte plus a little for scales. So a 70B model is about 140 GB at FP16, 70 GB at FP8, and 37 GB at 4-bit. That single calculation rules out most of what you might be considering.

KV cache is everything left over, and it is what buys concurrency and context. It grows with context length times the number of sequences you serve at once, so a model that fits with 6 GB to spare technically loads and then refuses real work.

Against 121.6 GiB of addressable memory, budget it like this: weights, plus enough KV cache for the context you actually use, plus a few gigabytes of headroom for activations and the CUDA context. A model whose weights use more than about 75 GB is a single-user model on this box. Under 40 GB, you have room to serve a team.

Why MoE models win here

This is the most useful thing to understand about the Spark, and it is why the model list below skews the way it does.

Generating a token requires reading the active weights out of memory, so decode speed is set by bandwidth divided by active parameter bytes. A dense 70B model reads all 70B parameters for every token. A mixture-of-experts model with 120B total parameters but 12B active reads only the 12B, because the router picks a handful of experts per token and the rest sit idle in memory.

The consequence: on a machine with lots of memory and moderate bandwidth, a big MoE model is roughly the speed of a small dense model with the knowledge of a large one. That combination is exactly what 128 GB of unified memory at 273 GB/s is good at, and it is why the naming convention matters. When you see 30B-A3B, that is 30B total parameters with 3B active, and it will generate far faster than a dense 30B.

Read the two numbers separately. Total parameters tell you whether it loads. Active parameters tell you how fast it runs.

Quantization formats

Four formats cover almost everything you will encounter, and the right choice depends more on your engine than on quality.

• NVFP4 is a 4-bit floating point format with hardware support on Blackwell, which is what the GB10 is. When a model ships in NVFP4, prefer it. NVIDIA publishes NVFP4 builds of the whole Nemotron family.
• FP8 keeps more quality than 4-bit at double the memory. Worth it when the model is small enough that you have room.
• AWQ and GPTQ are 4-bit integer formats with the broadest community coverage, and they are the well-trodden path in vLLM.
• GGUF is llama.cpp's format, with the Q4_K_M and Q5_K_M variants as the usual quality-per-byte sweet spot. If GGUF is the only quantization available for a model you want, run it under llama.cpp or Ollama rather than forcing it through vLLM.

Going below 4-bit is where quality starts to visibly degrade on most models, and it is rarely the right trade when the alternative is a smaller model at 4-bit.

What fits, right now

Current repositories on Hugging Face, checked August 2026. Footprints are the weights only, approximate, and do not include KV cache. Verify against the repo's own file sizes before planning around a number.

If you would rather not choose, deploy one of our prebuilt vLLM images and the weights are already staged on the node: Qwen3 Coder 30B for code, Llama 3.3 70B for general work. No download, no quantization decision.

What does not fit

Worth saying plainly, because the current frontier of open weights has moved well past a single 128 GB box.

DeepSeek-V4-Pro at 1.7T parameters, Qwen3.8-2.4T-A95B, GLM-5.2 at 753B, and Nemotron-3-Ultra-550B at roughly 300 GB in NVFP4 are all out of reach on one unit, at any quantization worth running. These are multi-node models.

The interesting edge case is DeepSeek-V4-Flash, 304B total. It has been made to run on a single GB10 box, but by way of 2-bit expert planes and a patched vLLM fork, reported in the model's own discussion thread at roughly 14 tok/s on prose and 21 tok/s on code, with about 753 tok/s of prefill. That is a genuine achievement and a bad default: it is a research configuration, not something to build a service on. If you want a 300B-class model in production, use more than one machine.

Getting the weights

Use the Hugging Face CLI rather than git, since the repos are large and git-lfs will make you regret it:

pip install -U "huggingface_hub[cli]"

# gated repos need a token from huggingface.co/settings/tokens
hf auth login

# download into the workspace, which survives an instance stop
hf download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
  --local-dir /workspace/models/nemotron-nano-30b

Set HF_HOME=/workspace/.cache/huggingface in your shell profile so that anything pulling weights implicitly, vLLM included, caches to persistent storage instead of the container filesystem. Re-downloading 60 GB because the cache was on the wrong volume is the kind of mistake you make exactly once.

Then point vLLM at the local directory rather than the repo name, which skips the network entirely on restart:

vllm serve /workspace/models/nemotron-nano-30b \
  --served-model-name nemotron-nano \
  --gpu-memory-utilization 0.85 \
  --max-model-len 32768