Latency vs throughput in LLM inference
Latency is how long a single request takes from submission to response. Throughput is how many requests, or tokens, a system processes in a given period across all requests combined. Both describe speed, but they are independent: a system can have low latency and low throughput, or high throughput and high latency, depending on how it's built and configured.
Two different questions
Latency answers "how long until I get my answer?" for one request. Throughput answers "how much total work can this system get through?" across many requests over time, usually measured in tokens per second aggregated across all concurrent users, or requests handled per minute. An unloaded GPU running one request at a time can give that request excellent latency while overall throughput stays poor, because the spare capacity serves nobody else.
Why the two trade off against each other
The main way to raise throughput on a GPU is batching: processing multiple requests together so the hardware's parallel compute isn't sitting idle waiting on one request's next step. Batching raises total tokens per second across all users, but usually increases each request's latency: a request waits for the batch to fill, or shares GPU cycles with the others in it, instead of getting the hardware's full attention. A system tuned for maximum throughput (many users, cheaply) and one tuned for minimum latency (one user, fast) often need incompatible configuration choices.
Metrics that matter for each
For latency, the relevant numbers are time-to-first-token, the delay before any output appears, and inter-token latency, the gap between subsequent tokens once generation has started. For throughput, the relevant number is aggregate tokens per second across all concurrent requests the system is handling, or requests completed per unit time. A model card or benchmark that reports only one of these shows you only one side of the trade-off.
Which one to optimize for
A chat interface where a person watches the reply stream in needs low latency; a slow first token feels broken even if the system could serve a hundred other users just as well. An overnight job over a large batch of documents, where nobody watches any single request, needs high throughput and can let any one document take longer, because total completion time is what matters. Building for the wrong one wastes either GPU capacity or user patience.
Where GPUwerk fits
A dedicated Spark isn't shared with other tenants, so you set the batching and concurrency on the inference server yourself, tuned for latency, throughput or a balance, and no other customer's traffic shows up in your results. On a shared, multi-tenant API the provider makes those batching decisions for everyone at once. See private LLM hosting for how self-hosting affects both dimensions in practice.