Blog/What is batching in LLM inference?
Definition

What is batching in LLM inference?

By Samuel Seidel · Published September 9, 2026

Batching groups multiple concurrent requests into a single forward pass through the model, so one read of the weights out of memory produces output tokens for all of them instead of just one. Token generation is memory-bandwidth-bound: the GPU spends most of its time moving weight bytes out of memory, not computing on them, and that read costs about the same whether it serves one request or several dozen. Batching turns the spare compute into aggregate throughput.

The bandwidth arithmetic behind it

Single-stream decode speed tops out at roughly memory bandwidth divided by the bytes of active parameters, in tokens per second; GPUwerk's benchmarks page works through the relationship in detail. With one request, most of the GPU's compute sits idle waiting on that memory read, because modern GPUs can do far more math per second than they can move bytes. Every request in a batch uses the same weights, so adding concurrent requests to the same forward pass puts that idle compute to work without another read of the weights.

Why it isn't free past a point

Larger batches still cost more: more compute per pass, and more KV cache memory, since every request in flight needs its own cache. Aggregate throughput rises with batch size, but not without limit, and per-request latency rises too, because each pass does more work before returning a token to any one request. At some point, often called the knee, more concurrency stops buying much throughput and mostly adds latency.

What happens past the knee depends on the model. In GPUwerk's measurements a mixture-of-experts model's throughput levels off past its knee, while a dense model's throughput can fall as concurrency keeps climbing; the benchmarks page has the measured numbers for both.

Why MoE and dense models batch differently

A mixture-of-experts model only activates a fraction of its total parameters per token, so its bandwidth cost per token is already lower than a dense model of similar total size, leaving more headroom before compute replaces bandwidth as the bottleneck. That's part of why the benchmarks show an MoE model leveling off instead of collapsing as batch size grows: it has further to go before it hits a compute ceiling. A dense model reads its full parameter count on every token regardless of batch size, so it runs into that ceiling sooner.

Batching and the DGX Spark

A Spark's unified memory holds the model weights and the KV cache for every request in the batch in one 128 GB pool, so batch size is limited by the memory left after the weights, not by a separate cache tier. See what KV cache is for how the cache grows with concurrency, and which models fit in 128 GB for sizing a model with enough headroom for the batch sizes your workload needs.

Related pages

First top-up: pay $20, get $30 in credit

Find the knee for your own model.

Run your model at rising concurrency on a dedicated Spark and see where throughput stops climbing.

A dedicated DGX Spark in Europe, North America or East Asia, $0.79/hour, billed per minute. $30 of credit is about 37 hours of compute.

Create account Read the benchmarks