Best GPU for running open-source LLMs privately: how to think about the choice
The best GPU for private LLM inference is the one whose memory fits the smallest model that does your task, with enough bandwidth for the people using it. For a 100-120B model serving a team, a 128 GB DGX Spark fits. For a small model and one user, a consumer card like an RTX 5090 is faster. An H100 is for teams that need both at heavy concurrency.
Capacity and bandwidth pull in opposite directions
A model has to fit in memory before it can run at all, so capacity decides which models are on the table. Bandwidth decides how fast the ones that fit generate text, because each token reads the active weights out of memory once. You can do the arithmetic before buying anything: tokens per second is roughly memory bandwidth divided by the bytes of active parameters read per token. GPUwerk's benchmarks page has measured numbers.
The catch: hardware with the most bandwidth (a high-end datacenter GPU) usually has less memory than the largest useful models need, and hardware with the most memory per dollar has modest bandwidth. Getting both means paying datacenter prices. So decide which constraint bites harder for you: fitting a big model at all, or making a model that already fits feel fast for one person typing.
Single-user chat and shared serving are different engineering problems
If you're the only user, you feel latency per response. That workload is bandwidth-bound and mostly idle: the GPU waits while you read and type, then works on one request. It wants fast single-stream generation more than compute headroom.
If the model serves a team, a support queue or a batch pipeline, the picture changes. Serving frameworks batch concurrent requests into the same forward pass, so the cost of reading weights from memory is shared across everyone's requests. Aggregate throughput climbs well past a single stream. A machine that feels merely adequate for one person in a chat window can serve many people at once.
This is the most common sizing mistake I see: someone benchmarks a card as its only user, finds it too slow, and rules it out for a team deployment, where the number that matters is aggregate throughput under concurrency.
Where each class of hardware fits
DGX Spark. 128 GB of unified memory puts it on the capacity side (the models guide lists what fits). It holds 100-120B-class models whole. Its memory bandwidth, around 273 GB/s, is modest by datacenter standards, but with a mixture-of-experts model's small active-parameter count it feels responsive. It suits a team that wants a large model resident on one machine without paying for datacenter-card time.
Consumer GPU (RTX 5090). The opposite end: high bandwidth and compute for the price, but 32 GB of VRAM rules out the largest open models however fast the card is. Pick it when the model you need fits and you want the fastest single-stream output at that size. DGX Spark vs RTX 5090 compares the two with numbers.
Datacenter card (H100). Strong on both capacity and bandwidth, which is why it costs what it does. It is the answer when you need large models and datacenter throughput under heavy concurrency, and have the budget. Most teams evaluating private LLM hosting don't need this tier; DGX Spark vs H100 shows where the line falls.
Mac Studio. A real alternative for the same reason as a Spark: lots of unified memory and enough bandwidth for MoE models. It suits individual developers already on Apple hardware more than teams building a shared endpoint, partly because the software for serving many concurrent users is less mature there than on CUDA. See DGX Spark vs Mac Studio for the direct comparison. Card by card: best LLMs for an RTX 5090 and best LLMs for 16 GB of VRAM; how to run an LLM locally covers the first install.
Decide in three steps
- Find the smallest model that does your task well enough. That sets the memory floor.
- Decide whether you serve one person or many. That tells you whether to optimize single-stream speed or aggregate throughput under concurrency.
- Only then compare specific cards, using the comparisons above rather than a generic "best GPU for LLMs" ranking.