Run 70B to 120B-class models in 128 GB of unified memory

A DGX Spark has 128 GB of memory shared by its CPU and GPU, so models that never fit on a 24 or 32 GB graphics card load in one piece. Fitting is not the same as fast. The memory moves 273 GB/s, and every generated token reads the model's active weights once. On one of my Sparks a dense Llama 3.3 70B generated 6.0 tokens per second for a single request. A mixture-of-experts model reads only a slice of its weights per token: gpt-oss-120b is reported at 58.7 tok/s on the same chip. Pick the model for the bandwidth, then rent a Spark for the hours you need.

First top-up: pay $20, get $30 in credit, about 37 hours on one Spark. Billed per minute from prepaid credit. Terminate when you're done and billing ends.

What fits, and how fast one request runs

"Reported" rows are other people's measurements, which I collected and checked against their sources on 11 October 2026 for the benchmarks page; I have not reproduced them. The two gpt-oss-120b rows show how far engine and workload move one model on one chip. Parameter counts for gpt-oss-120b are from its model card, via the memory calculator.

Why a dense 70B is slow and a 120B MoE is not

Generating a token means reading the active weights out of memory once, so the ceiling is the memory bandwidth divided by the bytes read per token:

tokens/s ceiling ≈ 273 GB/s ÷ GB of active weights read per token

Llama 3.3 70B, 4-bit:  about 37 GB per token  → about 7.4 tok/s   (measured 6.0)
gpt-oss-120b, MXFP4:   about 3 GB per token   → about 90 tok/s    (reported 58.7)

Real engines land between about half and four fifths of the ceiling. Under about 10 tok/s a model feels slow in a chat window, which is why a dense 70B on one Spark is a batch model: with 128 requests in flight my 70B run reached 485 tok/s in total, counting prompt and generated tokens, but each request slowed to 1.4 tok/s. The full concurrency curves are on the benchmarks page.

What I would run

• A 120B-class model you can talk to: gpt-oss-120b in MXFP4. The weights are 65.2 GB, which leaves room for a long context, and it reads about 5B parameters per token.
• Speed first: a Qwen3 mixture-of-experts model at 4 bits, such as Qwen3-30B-A3B or Qwen3.6-35B-A3B. These read about 3B parameters per token.
• A dense 70B: only for batch jobs, or when you need that exact model. Expect about 6 tok/s per request.
• Quantize to 4 bits. MXFP4, NVFP4, AWQ or Q4_K_M halve the bytes per token against FP8 and roughly double the ceiling. Quality costs differ by format; the quantization speed guide compares them on this chip.
• Over 128 GB: two linked Sparks, at $1.79 an hour for the two-node cluster when the console offers it. The Qwen3-235B report above went from about 1.8 tok/s on one Spark to about 12.5 on two. The two-node guide covers what pairing buys.

Run gpt-oss-120b with llama.cpp

For one user at a time, llama.cpp is the faster engine on this chip. Deploy DGX Spark 1× with the DGX OS image, open the browser terminal, build llama.cpp for the GB10 (5 to 10 minutes) and start the model. The build steps are the same as in my llama.cpp guide.

apt-get update && apt-get install -y build-essential cmake libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp /workspace/llama.cpp
cd /workspace/llama.cpp
cmake -B build -DGGML_CUDA=ON -DGGML_CURL=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real
cmake --build build --config Release --target llama-server -j

# downloads about 65 GB the first time
./build/bin/llama-server -hf ggml-org/gpt-oss-120b-GGUF --jinja \
  --host 127.0.0.1 --port 30000

When the log says the server is listening, measure your own speed from a second terminal. llama-server returns its timings with every answer:

curl -s http://127.0.0.1:30000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Explain memory bandwidth in three sentences."}],"max_tokens":300}' \
  | python3 -c 'import json,sys; t=json.load(sys.stdin)["timings"]; print(round(t["predicted_per_second"],1), "tok/s")'

The download runs on the clock, so start it first and read while it loads. To reach the server from your laptop, use an SSH tunnel with your own key, or start it with --host 0.0.0.0 --api-key and a long random key: each instance has its own public IPv4 address. For several users at once, vLLM batches better; the vLLM guide has the memory math for unified memory.

Limits to know first

• I have not measured gpt-oss-120b on my fleet. Its numbers above are reported, and your prompts will land somewhere else. Measure before you plan around a figure.
• Long prompts are slow on big dense models. My 70B run read an 8,192-token prompt at 391 tok/s, about 21 seconds before the first token.
• Quantization costs quality. A 4-bit model is not the full-precision one. Check answers on your own prompts.
• The Spark is ARM64 with an sm_121 GPU. Engines and Python packages need aarch64 builds for it, and a plain pip install vllm can install fine and then not start; the vLLM guide explains the trap.
• It runs in a container. No Docker inside, so the vendor containers in many guides do not run; build natively as above.
• No SLA and no backups. Model downloads outside /workspace are lost on a rebuild, and terminating deletes everything.

Price and where the Sparks are

$0.79 an hour for one Spark, billed per minute from prepaid credit: $6.32 for an eight-hour day. First top-up: pay $20, get $30 in credit. A stopped instance keeps its workspace and is billed at 75% of the hourly rate. Terminate when you're done and billing ends; that deletes the workspace. Prices exclude VAT where applicable.

My own Sparks are in a datacenter in Hradec Králové, Czech Republic. Partner Sparks, owned and hosted by partners at their own premises, are in the USA, Sweden, the Netherlands, Slovakia and Hong Kong; a partner has physical access to their machine. The console shows the country, and whether it is in the EEA, before you deploy. Every location offers the DGX OS and Ubuntu images used above.

Who runs this

GPUwerk is new. It is run by me, Samuel Seidel, through PRINT IT! SE, a company registered in Prague. As of 9 November 2026, 94 developers have started 374 instances since September. Live status is on status.gpuwerk.com, and the contact page reaches me directly if you want to check a model before you rent.

FAQ