RTX 5090 LLM benchmarks: tokens per second for 19 models, November 2026

An RTX 5090 generates about 69 to 79 tokens per second on a dense 27B to 31B model at 4-bit, 230 to 410 tok/s on 30B-class mixture-of-experts models with 3B to 4B active parameters, and 368 tok/s on gpt-oss-20b, according to llama.cpp and vLLM runs published between May and November 2026. Prompt processing on the same models runs at roughly 3,000 to 12,000 tok/s. GPUwerk rents an RTX 5090 but has not published its own runs, so every number below is reported by the source named in its row.

How many tokens per second does an RTX 5090 generate?

Single-stream decode on an RTX 5090 ranges from about 46 tok/s (a 117B model with experts in system RAM) to 410 tok/s (a 30B model with 3B active parameters), and the model's active size explains most of that spread. The table collects the first-hand RTX 5090 figures I could fetch and date, across five engines. The largest single source is the witcheer/rtx-5090-benchmarks dataset, a 326-row CSV of llama-bench and vLLM runs on one card from 28 May to 25 October 2026.

All rows are one RTX 5090 and one stream. "witcheer" rows come from benchmarks.csv in the Hugging Face dataset, read 7 November 2026; its notes cite llama.cpp builds b9653, b10338, b10371 and b10853 and vLLM 0.21.0 and 0.25.1 without pinning a build to every row, on CUDA 12.8 with 64 GB DDR5-5600 for the offload rows. tg128 is 128 generated tokens on an empty context. Atomic Chat used llama-bench -p 512,4096 -n 128 with an fp16 KV cache. The two offload rows keep some expert weights in system RAM and are not comparable with fully resident rows. The highlighted cell is the fastest single-stream figure in the table; the Ollama Qwen3.8 range is the fastest dense result, and both it and the 119.8 row use MTP drafting.

Why does decode speed track the 1,792 GB/s memory bandwidth?

Each generated token reads the active weights from VRAM once, so the ceiling is bandwidth divided by the bytes read per token, and dense models land at 70 to 75% of it. NVIDIA's GeForce comparison page lists the RTX 5090 at 1,792 GB/s. The worked numbers:

ceiling (tok/s) = 1,792 GB/s ÷ GB read per token

Qwen3.8-27B Q4_K_M, dense: file 15.92 GiB = 17.09 GB
  1,792 ÷ 17.09 = 104.9 tok/s ceiling, reported 78.9  → 75%
Gemma 4 31B Q4_K_M, dense: file 18.32 GB
  1,792 ÷ 18.32 =  97.8 tok/s ceiling, reported 68.5  → 70%
Qwen3 32B Q4_K_XL, dense: 18.64 GiB = 20.01 GB
  1,792 ÷ 20.01 =  89.6 tok/s ceiling, reported 61.4  → 69%

Qwen3.6-35B-A3B UD-Q4_K_M, MoE: 22.13 GB for about 35B params
  = 0.63 bytes/param; 3B active × 0.63 = about 1.9 GB per token
  1,792 ÷ 1.9   = about 940 tok/s ceiling, reported 271 → 29%

File sizes are from the witcheer CSV, Hardware Corner and Unsloth's GGUF repos; the arithmetic is mine. A dense file is read almost whole per token (only the input embedding lookup is skipped), so using the file size is close enough.

The MoE row is the one people misread. At about 2 GB per token the card finishes streaming weights so fast that routing, kernel launches and synchronization dominate, which is my reading of why 3B-active models reach only 30 to 40% of the bandwidth ceiling (gpt-oss-20b: 3.6B active per its model card, about 2.0 GB per token, 368 reported against roughly 900). Bandwidth still sets the ranking, just not the absolute speed.

Two rows sit above the ceiling, and that is the tell for speculative decoding. vLLM with the Qwen3.8-27B MTP head reached 119.8 tok/s, and Ollama 0.32.13 reported 124 to 149 tok/s on a 17 GB build. Saiyam Pathak reports that Ollama's qwen3.8 build ships its multi-token prediction head with drafting on by default (draft_num_predict 4), an inference he draws from the shipped config. One weight pass then verifies up to four drafted tokens. The sibling article on MTP, EAGLE-3 and draft models covers acceptance rates.

The same arithmetic explains the gap to a DGX Spark, whose 273 GB/s is 6.6 times less bandwidth (see the Spark specs). GPUwerk measured Qwen3-Coder-30B-A3B on a Spark at 80.9 tok/s with vLLM 0.27.1 and AWQ; ai.rs reports the same model at 230.8 tok/s on a 5090 with Ollama and Q5_K_M. Different engines and quants, so treat the 2.9 times gap as an illustration: small-active MoE models narrow the bandwidth gap because neither machine is bandwidth-bound on them. The full comparison is on DGX Spark vs RTX 5090.

How fast is prompt processing on an RTX 5090?

Prefill is compute-bound, so it runs 30 to 55 times faster than decode: about 3,000 to 3,900 tok/s on dense 27B to 31B models and 9,000 to 12,000 tok/s on 3B-active MoE models at short context. The witcheer dataset puts Qwen3.8-27B Q4_K_M at 3,881 tok/s (pp512) and Nemotron 3.5 Lightning at 11,698. Its own time-to-first-token estimate for a 16,384-token prompt is about 4.5 seconds on the dense Qwen3.6-27B and 1.5 seconds on Lightning.

Engines differ more on prefill than on decode. In Atomic Chat's ExLlamaV3 test on Gemma 4 12B, EXL3 at 4.0 bpw prefilled a 2,048-token prompt at 4,077 tok/s against 1,762 for GGUF Q4_K_M through llama-server, while GGUF decoded faster (139.4 against 113.1 tok/s). The llama-bench prefill figure for the same GGUF in Atomic Chat's other article is 8,659 tok/s at 4k, so server overhead and prompt length move the number by a factor of four or more. Compare prefill rows only within one source.

Does the speed drop with long context?

Yes, and how much depends on the attention design: older full-attention models lose most of their speed by 128k, while 2026 hybrid models keep most of it. Hardware Corner's March 2026 context sweep shows the contrast.

Tokens per second, Hardware Corner RTX 5090 page, "Updated Benchmarks: March 2026", read 7 November 2026. The page does not state the llama.cpp build for every row. Qwen3.5 35B also ran at 256k: 97.3 tok/s decode, 2,004 tok/s prefill.

The witcheer dataset measures the same thing as "held", decode at 32k divided by decode on an empty context: 94% for Nemotron 3.5 Lightning (Mamba hybrid), 93% for Qwen3.6-27B (Gated DeltaNet hybrid), 87% for Ornith-1.5-35B-A3B and 86% for Gemma 4 31B. For agent work, where contexts grow to 50k and beyond, the hybrid models are the ones whose short-context benchmark still means something. How much VRAM that context costs is worked out in the best LLMs for an RTX 5090.

Which engine is fastest on an RTX 5090?

For a dense model and one user without drafting, llama.cpp is a little ahead in the published runs; with MTP drafting, vLLM and Ollama pass it; for many users, vLLM and TensorRT-LLM win. On Qwen3.8-27B at 4-bit the single-stream figures are 78.9 tok/s on llama.cpp, 73.0 on TensorRT-LLM 1.3.0rc29, 68.5 on vLLM 0.31.0, 119.8 on vLLM 0.25.1 with MTP and 124 to 149 on Ollama 0.32.13. On the MoE Nemotron 3.5 Lightning the order flips: vLLM with NVFP4 decoded at 410.5 tok/s against 377.4 on llama.cpp.

Format matters as much as engine on Blackwell. ComputingForGeeks measured Qwen3-8B on TensorRT-LLM at 99.8 tok/s in BF16, 170.0 in FP8 and 222.7 in NVFP4 for one request, with GSM8K strict-match at 86.66%, 86.43% and 84.91%. NVIDIA's NVFP4 introduction says the format arrived with Blackwell's fifth-generation tensor cores, which the 5090 has; the quantization docs explain the formats.

vLLM is also the engine most likely to trip over 32 GB. kohei_wada's NVFP4 build of Qwen3.8-27B took about 23 GB after loading on vLLM 0.27.1, startup ran out of memory while profiling with CUDA graphs enabled, and eager mode crawled at 27.8 tok/s. If vLLM is your target, size the quant with a few GB to spare for graphs and activations. The vLLM setup notes and llama.cpp notes list the flags that control this.

What does an RTX 5090 do under concurrent load?

Batching multiplies throughput: the same card that decodes Qwen3.8-27B at about 70 tok/s for one user delivers 1,195 to 1,361 tok/s across 32 users. ComputingForGeeks ran nvidia/Qwen3.8-27B-NVFP4 (21.9 GB) with a 4,096-token limit per sequence.

Aggregate output tok/s, ComputingForGeeks, updated 13 October 2026, driver 595. The TensorRT-LLM BF16 run fell at 32 because its KV cache could not hold all 32 sequences at once.

Smaller models go further. The same author measured Qwen3-8B NVFP4 at 5,340 tok/s with 32 concurrent streams on TensorRT-LLM. CloudRift reports 4,570 tok/s sustained for Qwen3-Coder-30B-A3B AWQ on vLLM with an FP8 KV cache and 8k context, against 2,259 on an RTX 4090. The RTX 4090 vs RTX 5090 comparison has the generational numbers. For sizing a team endpoint, the aggregate column is the one to read; the GPU buyer's guide explains why.

How do I reproduce these numbers on my own card?

Run llama-bench with the same prompt, generation and depth settings as the source, and record the five settings that changed results above: build, quant file, context depth, drafting and power limit.

# llama.cpp: decode and prefill on empty and 32k-deep context
llama-bench -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 99 -fa 1 \
  -p 512,4096 -n 128 -d 0,32768 -r 3

# vLLM: single stream and 32 concurrent against a running server
vllm bench serve --model nvidia/Qwen3.8-27B-NVFP4 \
  --dataset-name random --random-input-len 512 --random-output-len 256 \
  --max-concurrency 32 --num-prompts 96

Write down the power limit too. In the llama.cpp CUDA thread, RodriMora's RTX 5090 decoded Llama 2 7B at 248.6 tok/s on its 600 W default and 245.0 at 400 W, while prefill dropped from 16,042 to 13,823 tok/s. A card capped by its owner or its BIOS will match other people's decode numbers and miss their prefill numbers.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• witcheer/rtx-5090-benchmarks (dataset and benchmarks.csv), Hugging Face, 2026-10-26
• Best Local LLMs for RTX 5090: Benchmarks, Atomic Chat, 2026-09-18
• EXL3 vs GGUF: Quality, Speed and VRAM, Atomic Chat, 2026-09-14
• TensorRT-LLM vs vLLM on RTX 5090: Tested, ComputingForGeeks, 2026-10-13
• NVFP4 vs FP8 vs BF16 on RTX 5090: TensorRT-LLM Benchmarks, ComputingForGeeks, 2026-10-13
• Performance of llama.cpp on Nvidia CUDA (discussion #15013), ggml-org/llama.cpp on GitHub, 2025-08-01
• RTX 5090 LLM Benchmark Results: 10K Tokens/sec Prompt Processing, 139K Context, Hardware Corner, 2025-11-06
• RTX 5090 LLM benchmarks (Tier 1 Enthusiast), Hardware Corner, 2026-03
• NVIDIA DGX Spark In-Depth Review, LMSYS Org, 2025-10-13
• Qwen3.8 27B Measured on an RTX 5090, and Why vLLM Didn't Work, DEV Community (kohei_wada), 2026-11-03
• Running Qwen3.8-27B on DGX Spark, Kubesimplify (Saiyam Pathak), 2026-08-17
• Run Qwen3-Coder 30B-A3B on an RTX 5090: 231 tok/s with Ollama, ai.rs, 2026-05-22
• llm-performance-tests, DeutscheKI on GitHub
• GPU Benchmarks for LLM Inference, CloudRift
• Compare GeForce graphics cards, NVIDIA
• Introducing NVFP4 for Efficient and Accurate Low-Precision Inference, NVIDIA Technical Blog, 2025-06-24
• Qwen3.8-27B-GGUF (file sizes), Unsloth on Hugging Face