Batch vs interactive AI workloads: why the distinction matters for hardware choice
First decide whether your workload is interactive (a person watches tokens appear) or batch (a queue nobody watches in real time). Then judge models by the matching number: single-user decode speed for interactive work, aggregate throughput under concurrency for batch. The same model on the same GPU can be good at one job and mediocre at the other, because reading a long prompt and generating tokens one at a time stress hardware in opposite ways.
Two different jobs inside one request
Every generation has two phases. Prefill processes the input prompt: the model reads the whole thing at once and computes attention over it in parallel. That is compute-bound work, the kind a GPU's raw arithmetic throughput is good at. Decode generates the output one token at a time, and each new token needs another full pass reading the model's weights from memory. That makes decode bound by memory bandwidth (how fast the hardware can move those weights) rather than by arithmetic per second.
Those are different bottlenecks, and a piece of hardware that's strong on one axis isn't automatically strong on the other. A workload dominated by prefill (long documents summarized into short answers) cares about compute throughput. A workload dominated by decode (long conversational replies to short prompts) cares about memory bandwidth. The same GPU serving the same model can look completely different depending on which one you're measuring.
What this looks like on measured hardware
The pattern shows up directly in published Spark numbers. Nemotron-3-Super-120B at NVFP4 runs single-user decode at 22.7 to 23.7 tok/s but hits prefill close to 1,900 tok/s, per the vLLM project's DGX Spark writeup quoted on GPUwerk's Nemotron page. That makes it a strong fit for long prompts with short answers and a modest one for long, chatty back-and-forth. Llama 3.3 70B AWQ, measured by GPUwerk on 2 September 2026, decodes at only 6.0 tok/s for a single user, too slow for interactive chat on this hardware. Its aggregate throughput, though, peaks at 485 tok/s at 128 concurrent requests, which makes it a good batch tool. Both numbers have the same cause: it's a dense model, so every decode step reads all 70B parameters. One user pays that full cost per token, while many concurrent requests share each pass over the weights.
Qwen3-Coder 30B-A3B, a mixture-of-experts model, decodes at 80.9 tok/s single-user under vLLM and keeps scaling to 3,235 tok/s aggregate at 256 concurrent requests, per GPUwerk's measurement. An MoE architecture activates only a subset of its parameters per token, so it reads less weight per step than a dense model of comparable total size. That's why MoE models tend to be reasonable at both interactive and batch use, while a large dense model tends to be good at one and merely acceptable at the other.
Match the model to the phase your workload lives in
Before picking a model, work out which phase dominates your traffic. A chat assistant, a coding copilot, or anything where a person watches tokens appear is a decode-bound, interactive workload. Single-user decode speed decides whether it feels usable. A summarization pipeline, a document-classification job, or any queue of requests nobody watches in real time is closer to a prefill-heavy batch workload. There, optimize aggregate throughput under concurrency. A model with mediocre single-stream decode speed can still be the right choice if it scales well under load.
This is also why a headline tokens-per-second number can mislead if you don't know which phase it was measured in. A single-stream decode number tells you almost nothing about batch throughput, and an aggregate throughput number at high concurrency tells you almost nothing about what one person waiting on a reply will experience. Read both, and read them against the shape of your own traffic, not against each other.
One question decides which number to read
Ask: is a person watching this response generate in real time, or is this request one of many being processed without anyone waiting on it individually? If a person is watching, weight single-user decode speed heavily and treat a large dense model with caution unless its decode number is fast enough. If it's a batch job, weight aggregate throughput at realistic concurrency and don't rule out a model with a slow single-stream number, since that number was never the one your workload was going to hit.
GPUwerk's benchmarks page works through the bandwidth arithmetic behind these numbers in more depth, including why a spec sheet alone won't tell you which side of this line your workload falls on, and why measuring it on real hardware is usually faster than reasoning it out from a data sheet.