Blog/Batch vs interactive AI workloads and why it changes hardware choice
Explainer

Batch vs interactive AI workloads: why the distinction matters for hardware choice

By Samuel Seidel · September 9, 2026

First decide whether your workload is interactive (a person watches tokens appear) or batch (a queue nobody watches in real time). Then judge models by the matching number: single-user decode speed for interactive work, aggregate throughput under concurrency for batch. The same model on the same GPU can be good at one job and mediocre at the other, because reading a long prompt and generating tokens one at a time stress hardware in opposite ways.

Two different jobs inside one request

Every generation has two phases. Prefill processes the input prompt: the model reads the whole thing at once and computes attention over it in parallel. That is compute-bound work, the kind a GPU's raw arithmetic throughput is good at. Decode generates the output one token at a time, and each new token needs another full pass reading the model's weights from memory. That makes decode bound by memory bandwidth (how fast the hardware can move those weights) rather than by arithmetic per second.

Those are different bottlenecks, and a piece of hardware that's strong on one axis isn't automatically strong on the other. A workload dominated by prefill (long documents summarized into short answers) cares about compute throughput. A workload dominated by decode (long conversational replies to short prompts) cares about memory bandwidth. The same GPU serving the same model can look completely different depending on which one you're measuring.

What this looks like on measured hardware

The pattern shows up directly in published Spark numbers. Nemotron-3-Super-120B at NVFP4 runs single-user decode at 22.7 to 23.7 tok/s but hits prefill close to 1,900 tok/s, per the vLLM project's DGX Spark writeup quoted on GPUwerk's Nemotron page. That makes it a strong fit for long prompts with short answers and a modest one for long, chatty back-and-forth. Llama 3.3 70B AWQ, measured by GPUwerk on 2 September 2026, decodes at only 6.0 tok/s for a single user, too slow for interactive chat on this hardware. Its aggregate throughput, though, peaks at 485 tok/s at 128 concurrent requests, which makes it a good batch tool. Both numbers have the same cause: it's a dense model, so every decode step reads all 70B parameters. One user pays that full cost per token, while many concurrent requests share each pass over the weights.

Qwen3-Coder 30B-A3B, a mixture-of-experts model, decodes at 80.9 tok/s single-user under vLLM and keeps scaling to 3,235 tok/s aggregate at 256 concurrent requests, per GPUwerk's measurement. An MoE architecture activates only a subset of its parameters per token, so it reads less weight per step than a dense model of comparable total size. That's why MoE models tend to be reasonable at both interactive and batch use, while a large dense model tends to be good at one and merely acceptable at the other.

Match the model to the phase your workload lives in

Before picking a model, work out which phase dominates your traffic. A chat assistant, a coding copilot, or anything where a person watches tokens appear is a decode-bound, interactive workload. Single-user decode speed decides whether it feels usable. A summarization pipeline, a document-classification job, or any queue of requests nobody watches in real time is closer to a prefill-heavy batch workload. There, optimize aggregate throughput under concurrency. A model with mediocre single-stream decode speed can still be the right choice if it scales well under load.

This is also why a headline tokens-per-second number can mislead if you don't know which phase it was measured in. A single-stream decode number tells you almost nothing about batch throughput, and an aggregate throughput number at high concurrency tells you almost nothing about what one person waiting on a reply will experience. Read both, and read them against the shape of your own traffic, not against each other.

One question decides which number to read

Ask: is a person watching this response generate in real time, or is this request one of many being processed without anyone waiting on it individually? If a person is watching, weight single-user decode speed heavily and treat a large dense model with caution unless its decode number is fast enough. If it's a batch job, weight aggregate throughput at realistic concurrency and don't rule out a model with a slow single-stream number, since that number was never the one your workload was going to hit.

GPUwerk's benchmarks page works through the bandwidth arithmetic behind these numbers in more depth, including why a spec sheet alone won't tell you which side of this line your workload falls on, and why measuring it on real hardware is usually faster than reasoning it out from a data sheet.

Related pages

First top-up: pay $20, get $30 in credit

Measure your own workload before you commit to a model.

Deploy a model on a dedicated Spark and time your own prompts at the concurrency you expect.

A dedicated DGX Spark in Europe, North America or East Asia, $0.79/hour, billed per minute. $30 of credit is about 37 hours of compute.

Create account Benchmarks page