How many users can a DGX Spark serve? Concurrency curves and a worked team sizing

One DGX Spark can stream about 52 requests at once at 15 tokens per second each with a mixture-of-experts model that activates 3B parameters, and cannot give even one user chat speed with a dense 70B. In GPUwerk's vLLM 0.27.1 run of 2 September 2026, Qwen3-Coder-30B-A3B (AWQ) delivered 37.3 tok/s per request at 8 concurrent, 18.4 at 32 and 13.1 at 64, while Llama 3.3 70B started at 6.0 tok/s for a single user. A chat user is only generating part of the time, so by the duty-cycle calculation below 52 streams cover roughly 160 people.

Generated tokens per second for each request, one GB10. GPUwerk rows: 1 request is the measured output rate; the other columns are 1,000 divided by the median time per output token in the raw logs (512 tokens in, 256 out), which is why the 8-request figure here (37.3) differs slightly from the 35.2 on the benchmarks page, which divides total output by 8. Dendro Logic reports per-sequence speed directly (about 1,500 prompt tokens, up to 400 out, JSON). Petronella reports aggregates only; I divided by the user count. Petronella's 32-request figure sits in the 32 column.

How fast is each user at 8, 32, 128 and 256 concurrent requests?

Each user keeps about half the single-stream speed at 8 concurrent requests and about a quarter at 32, on the mixture-of-experts model. The raw logs behind GPUwerk's DGX Spark benchmarks page also record time to first token, which the summary table there leaves out, and that is the number a person notices first.

GPUwerk-measured with vllm bench serve against vLLM 0.27.1 on one GB10 node with nothing else on its GPU, 2 September 2026: Qwen3-Coder log, Llama 3.3 70B log. The 1-request rows are a separate run with 128 tokens in and 512 out; every other row is 512 in and 256 out, three requests per concurrency slot, zero failed requests. Per-user tok/s is my arithmetic, 1,000 divided by the median time per output token.

Two things in this table do not show on the summary page. First, the p99 wait for the first token on the 30B stays under 3 seconds up to 64 concurrent and then grows fast: 7.1 seconds at 128 and 21.5 at 256. Second, the dense 70B already makes one request in a hundred wait 10 seconds for its first token at 8 concurrent, because every prefill competes with seven decodes that each read all 37 GB of weights.

What do other published concurrency sweeps show?

The same shape on every model: aggregate throughput rises 10 to 120 times from one request to the peak, and per-request speed falls toward 3 to 5 tok/s at the top. What differs is the starting speed, which is set by bytes read per token.

Aggregate tok/s for one Spark. The Qwen3.6 post does not name the model size; a 35 GB FP8 checkpoint decoding at 80 tok/s for one user fits the 35B-A3B mixture-of-experts model, not the dense 27B. The Nemotron Nano plateau from 8 requests onward looks like a cap on running sequences in that NIM, but Dendro Logic does not say. Read 7 November 2026.

Two smaller data points fill in dense models. Magistral Small 24B in BF16 on vLLM 0.20.0 ran at about 4.8 to 5 tok/s for one request and about 86 tok/s in total for 20, with each request at about 4.3 tok/s (NVIDIA forum, 20 October 2026). Qwen3.8-27B with 16 concurrent prompts on vLLM 0.27.1 had a mean time per output token of 114.7 ms in NVFP4 and 147.9 ms in FP8, so about 8.7 and 6.8 tok/s per user (NVIDIA forum, 15 August 2026).

Generated tokens or prompt plus generated: which number is quoted?

Most headline concurrency figures over about 1,500 tok/s on a Spark count prompt tokens, so check the counter before you compare. vllm bench serve prints both an "Output token throughput" and a "Total token throughput" line. In GPUwerk's runs, with 512 prompt tokens and 256 output tokens per request, the total is three times the output: 1,078 generated tok/s at 256 concurrent, or 3,235 counting prompt tokens.

llama.cpp's batched bench does the same thing under different names: S_TG is generated tokens and S is the whole job. For gpt-oss-120b at 32 parallel sequences with 512-token prompts, S_TG is 353 tok/s and S is 1,832 tok/s (llama.cpp DGX Spark page). Dendro Logic counts generated tokens. For sizing a chat service, generated tokens per user are the number that matters; prompt tokens belong to the prefill budget further down.

Where is the knee, and what happens past it?

Mixture-of-experts models flatten out past their knee; dense models fall off it. The 30B gained 7.5% in aggregate between 128 and 256 concurrent while its median time per output token almost doubled. Llama 3.3 70B dropped from 161.8 to 100.0 generated tok/s between 128 and 256, with median time to first token rising from 8 to 40 seconds. Dendro Logic found that both kinds fall eventually: gpt-oss-120b at 384 (771.2 against 862.8 at 256), which the author puts down to MoE routing overhead, and the dense Nemotron Super 49B at 320 (514.7 against 695.1).

The useful knee for interactive work comes much earlier than the throughput peak. The vLLM project's own Spark writeup runs Nemotron-3-Super with --max-num-seqs 4 and warns that above four decode streams the "per-token bandwidth tax can outweigh continuous-batching gains" for latency (vLLM blog, 1 June 2026). The curves above agree with that for a 12B-active model and disagree for a 3B-active one, which still gives each user 18 tok/s at 32.

How many people is that? A worked sizing

Pick a per-user speed, read the number of streams off the curve, then convert streams to people with the fraction of time each person is waiting on an answer. All the steps below are my calculation from the measured curves.

Step 1, the floor. I use 15 tok/s per user for comfortable streaming, the bar in the capacity planning guide, and 10 tok/s as the minimum the benchmarks page draws for a usable chat window.

Step 2, streams. Interpolating linearly between measured points: Qwen3-Coder-30B-A3B falls from 18.4 tok/s at 32 to 13.1 at 64, so 15 tok/s lands at about 52 streams and 10 tok/s at about 104. gpt-oss-120b in the March forum run reaches 15 tok/s at about 14 streams and 10 tok/s at 32; with Dendro Logic's 1,500-token prompts, at about 9 and 21. Dense Qwen3.8-27B in NVFP4 reaches 15 tok/s at about 15 streams. Llama 3.3 70B never reaches either floor.

Step 3, duty cycle. A person who asks something every 2 minutes and gets a 400-token answer at 15 tok/s keeps one stream busy for 400 / 15 + 1 second to first token = 27.7 seconds, or 23% of the time. Average concurrent streams = users x 0.23.

Step 4, headroom. Requests arrive at random, so size for peaks rather than the average. Treating arrivals as Poisson and keeping the 99th percentile under the stream count (average + 2.33 x square root of average ≤ streams): 52 streams allow an average of 37.7, which is 163 users.

My calculation: streams by linear interpolation between measured concurrency levels; users at a 23% duty cycle with a Poisson 99th-percentile margin. Longer chat histories than the 512-token test prompts lower per-user speed, so treat these as upper bounds for a given model.

Agents are different. A coding agent sends its next request as soon as the last one finishes, so its duty cycle is close to 100% and one agent is one stream. Its limit is usually prefill. Ten agents each sending a fresh 20,000-token prompt every 30 seconds need 6,667 prompt tok/s, more than the 5,705 tok/s Qwen3-Coder-30B-A3B prefilled on an 8,192-token prompt in the fleet run. Prefill does not grow with concurrency the way decode does: llama.cpp's gpt-oss-120b prefill on 512-token prompts levels off near 2,480 tok/s from 4 parallel sequences to 32. Repeated context has to come from the cache, which is what the prompt caching guide sets up.

What moves the number: context, KV cache and a second Spark

Longer contexts cut both speed and the number of sessions that fit; a second Spark adds capacity mostly at high concurrency. The fleet logs show how much KV cache vLLM allocated at --gpu-memory-utilization 0.85: 924,912 tokens for Qwen3-Coder-30B-A3B and 207,120 for Llama 3.3 70B. By my arithmetic that is 28 sessions at a full 32,768-token context for the 30B, or 113 at 8,192, against 6 and 25 for the 70B. Per-token cache sizes for other models are in KV cache size by model, and decode at 50k and 100k fills is in the long-context guide.

Speculative decoding helps the first user most. In the Qwen3.6 sweep, a 37% smaller checkpoint ran 45% faster at 2 concurrent but tied within noise from 8 upward, where the author's reading is that attention and KV reads, not weights, set the pace. Measured bandwidth, the resource every one of these streams shares, is compiled in DGX Spark memory bandwidth measured.

For a second machine, the March forum test on gpt-oss-120b is the cleanest comparison. At 32 concurrent: one Spark 318.7 tok/s, two Sparks with tensor parallelism 471.7, two independent Sparks behind a LiteLLM proxy with random routing 382.1. At 64: 315.2, 471.8 and 567.9. Tensor parallelism wins up to about 32 concurrent requests and independent replicas win above it. The two-node cluster guide covers the cabling and the tensor-parallel setup.

Which engine should a shared Spark run?

vLLM or SGLang, because they batch; Ollama does not. In the August 2026 forum test, Ollama gave gpt-oss-120b 42.1 tok/s alone and 42.5 tok/s in total for eight concurrent requests, while vLLM went from 60.7 to 116 to 153 (NVIDIA forum, 10 August 2026). llama.cpp batches across its --parallel slots, as the batched-bench rows show, but splits one context budget between them. The engine comparison has the flags, and the planned SGLang on DGX Spark guide covers its install. Per-user keys and limits are in team access.

My recommendation: for a team of up to about 150 people chatting, run a 3B-active mixture-of-experts model on vLLM with --max-num-seqs around 64 and prefix caching on, and watch the p99 time to first token rather than throughput. Keep dense models of 27B and up for batch jobs on this box. For anything above about 50 concurrent streams, plan for a second Spark as an independent replica.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• NVIDIA DGX Spark Concurrency Benchmark: 120x Throughput, Dendro Logic, 2026-04-22
• DGX Spark benchmark results (llama-bench and llama-batched-bench), ggml-org/llama.cpp on GitHub, 2026-02-05
• Measured inference benchmarks on a single DGX Spark (Ollama, llama.cpp, vLLM), NVIDIA Developer Forums, 2026-08-10
• Why does Cluster TP=2 beat 2x Solo + LiteLLM proxy at low/mid concurrency?, NVIDIA Developer Forums, 2026-03-27
• Benchmarks for Qwen3.6 FP8 vs PrismQuant, NVIDIA Developer Forums, 2026-04-22
• Qwen3.8-27B on DGX Spark using vllm: NVFP4 vs FP8 performance, NVIDIA Developer Forums, 2026-08-15
• DGX Spark firmware update: what we learned on 8 GB10 units, Petronella Technology Group, 2026-09-26
• NVIDIA DGX Spark In-Depth Review, LMSYS, 2025-10-13
• vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation, vLLM blog, 2026-06-01
• DGX Spark GB10: Magistral 24B BF16 ~4.85 tok/s, GEMV decode, NVIDIA Developer Forums, 2026-10-20