Fine-tuning an LLM on an RTX 5090: what fits in 32 GB, how fast it trains, and what a run costs

On a 32 GB RTX 5090, 4-bit QLoRA trains models up to about 32B parameters (Unsloth lists a 26 GB minimum for 32B and 41 GB for 70B), 16-bit LoRA tops out around 9B (24 GB; a 14B needs 33 GB), and plain full fine-tuning with AdamW fits only 1 to 2B. For 24 million training tokens, an 8B LoRA run takes about 2.2 hours at torchtune's published RTX 4090 speed, under $2 of rented RTX 5090 time at $0.89 an hour. Anything at 70B needs a 128 GB DGX Spark or an 80 GB H100.

• QLoRA (4-bit): up to about 32B; 40B only with short sequences. gpt-oss-20b needs 14 GB.
• LoRA (16-bit): up to about 9B (24 GB); 11B is tight at 29 GB.
• Full fine-tune: about 2B with standard AdamW; torchtune fit 8B in 18.9 GiB with its single-device recipe.
• Software: PyTorch 2.7 or later with CUDA 12.8 wheels; Unsloth via cu128 or its Docker image; axolotl recommends PyTorch 2.11+ and CUDA 13.0 for Blackwell.

What is the largest model you can fine-tune on 32 GB?

A 32B model with QLoRA, a 9B model with 16-bit LoRA. The table is Unsloth's published minimum VRAM per model size, which the project calls "the absolute minimum"; the verdict column is mine, against the 5090's 32 GB.

Unsloth requirements page and its gpt-oss guide, read 7 November 2026. Unsloth's figures assume its own kernels and gradient checkpointing; plain Hugging Face training needs more. The 128 GB Spark version of this table is in the Spark fine-tuning guide.

The headroom matters more than the yes or no. Every gigabyte left after the weights goes to activations, which grow with sequence length and batch size. A 32B QLoRA run that fits at 512 tokens can run out of memory at 4,096.

QLoRA, LoRA or full fine-tuning on a 5090?

QLoRA for anything above 9B, 16-bit LoRA for 8B-class models, and full fine-tuning only for small models. Full fine-tuning with mixed precision and standard AdamW keeps roughly 16 bytes per parameter (BF16 weights and gradients, FP32 master weights and two optimizer states), so 32 GB divided by 16 bytes is 2B parameters before a single activation, by my arithmetic. Recipes that cut that number exist: torchtune's README reports a Llama 3.1 8B full fine-tune at 18.9 GiB peak on one RTX 4090. At that recipe's ratio of about 2.4 GiB per billion parameters, a 13B model would need roughly 31 GiB, so on a 5090 the extra 8 GB mostly buys longer sequences rather than a bigger model (my extrapolation).

The difference between LoRA and QLoRA on the same model is speed and fidelity. torchtune's 8B figures on an RTX 4090: LoRA at 16.2 GiB and 3,083 tokens per second, QLoRA at 7.4 GiB and 2,413. QLoRA uses less than half the memory and runs about 22% slower, because the 4-bit base weights are dequantized on every step. If the 16-bit model fits, train in 16-bit.

Does the RTX 5090 work with Unsloth, axolotl and TRL?

Yes, with current versions; most failures come from old wheels built without Blackwell (sm_120) kernels. PyTorch 2.7 was the first release with Blackwell support and pre-built CUDA 12.8 wheels, then marked as a prototype feature, and it shipped Triton 3.3 with Blackwell support for torch.compile. What each training stack says on 7 November 2026:

• Unsloth: its Blackwell guide covers RTX 5060 to 5090. Install with the cu128 PyTorch build, or use the unsloth/unsloth Docker image. Do not upgrade Triton on its own, because each torch release pins a Triton build that already supports Blackwell. xformers is optional and can be built with TORCH_CUDA_ARCH_LIST="12.0". The requirements page recommends CUDA toolkit 12.8 or later for Blackwell.
• Axolotl: the installation docs ask for Python 3.12+ and PyTorch 2.13.0, and for Blackwell, PyTorch 2.11 or later with CUDA 13.0, which "avoids first-launch JIT overhead" on RTX 50-series cards. Blackwell images are tagged cu130.
• TRL: no 5090-specific notes, since it runs on whatever PyTorch you install. Its memory guide lists the switches that matter on 32 GB: max_length, packing, padding_free, activation_offloading, PEFT and gradient checkpointing, which is on by default.
• bitsandbytes (the 4-bit kernels behind QLoRA): its 0.50.0 release notes describe faster 4-bit inference "across Turing through Blackwell"; the latest release is 0.50.2.

A minimal setup that follows Unsloth's Blackwell instructions, then a 16-bit LoRA run sized for 32 GB:

# Python 3.12 venv with the CUDA 12.8 PyTorch build (Unsloth Blackwell guide)
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -U vllm --torch-backend=cu128
uv pip install unsloth unsloth_zoo bitsandbytes

# train.py: 16-bit LoRA on an 8B model, sized for 32 GB
from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-3.1-8B-Instruct",
    max_seq_length=4096,
    load_in_4bit=False,          # True for QLoRA on 14B to 32B models
)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
    use_gradient_checkpointing="unsloth",
)
SFTTrainer(
    model=model, processing_class=tokenizer,
    train_dataset=load_dataset("json", data_files="train.jsonl")["train"],
    args=SFTConfig(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        max_length=4096, packing=True,
        num_train_epochs=2, learning_rate=1e-4, bf16=True,
        output_dir="ckpt", save_steps=200,
    ),
).train()

Watch nvidia-smi for the first few hundred steps; peak memory arrives with the longest packed batch, not the first one. To serve the adapter afterwards, the vLLM on RTX 5090 guide covers the flags.

How fast does an RTX 5090 train?

I found no published, like-for-like RTX 5090 training throughput that I could verify, so the numbers below are the nearest reported ones. Treat the RTX 4090 rows as a floor for the 5090.

torchtune numbers are measured over one full epoch; the project says development wound down in 2025 and it is no longer maintained, but the figures stand as published. NVIDIA's post of 24 October 2025 gives different peaks in its text (53,657.6 and 5,079.4 tokens/s for the same two runs) than in its table, without explaining why; I use the table values, the lower ones.

How much faster is a 5090 than a 4090 at training? Training is compute-bound, and Tom's Hardware's spec table lists 104.8 against 82.6 FP32 TFLOPS and 838 against 661 FP16 tensor TFLOPS, 27% more on both. If the gain carried over in full, the 8B LoRA row would land near 3,900 tokens per second on a 5090. That is my scaling estimate from specifications, not a measurement. NVIDIA's December 2025 post on Unsloth notes that models above 30B "often exceed the VRAM capacity of consumer GPUs", which matches the table above.

Worked example: 20,000 examples, how long and how much?

Two to three hours and under $3 for an 8B LoRA or QLoRA, about 11 hours and $10 for a 32B QLoRA. Assume 20,000 chat examples averaging 600 tokens, trained for two epochs: 20,000 x 600 x 2 = 24 million training tokens. Time is tokens divided by throughput; cost is time times the hourly rate.

My calculation. The 32B row assumes throughput falls in proportion to parameter count, a rough rule that ignores attention cost and is not a measurement. Add setup, evaluation and checkpoint upload time to every row. Hourly rates: GPUwerk's RTX 5090 and DGX Spark (GPUwerk publishes this page), and RunPod A100 SXM Community and Secure Cloud, all checked 7 November 2026.

The GPU bill for one run is rarely the expensive part of a fine-tune; the failed runs before the good one are. That favors a card you rent by the minute while you iterate on data and hyperparameters, then the same card for the final run. Other providers' RTX 5090 rates are in RTX 5090 cloud pricing.

How long can training sequences be on 32 GB?

For an 8B QLoRA, roughly 116,000 tokens by interpolation from Unsloth's published limits. Its benchmark page lists the maximum context for Llama 3.1 8B with 4-bit QLoRA on all linear layers, rank 32 and batch size 1: 78,475 tokens on 24 GB and 153,977 on 40 GB, against 5,789 and 12,264 for Hugging Face with FlashAttention 2. A straight line between the two gives 116,226 tokens at 32 GB, my interpolation. For Llama 3.3 70B the same page needs 48 GB to reach 12,106 tokens, another way of saying 70B does not belong on this card.

Long sequences cost time as well as memory, so packing short examples (TRL's packing=True, default "bfd" strategy) is usually a better use of a 4,096-token window than padding each example to it.

When does a 128 GB Spark or a rented H100 fit better?

When the job needs more than about 30 GB, or when you need it done fast at 70B. My rules of thumb:

• Up to 9B in 16-bit, up to 32B in QLoRA: RTX 5090. It is the cheapest fast card that fits, at $0.89 an hour on GPUwerk's RTX 5090 and from about $0.65 for non-interruptible capacity elsewhere.
• 16-bit LoRA on 14B to 40B, or 70B QLoRA overnight: a 128 GB DGX Spark. Unsloth's 76 GB for a 32B LoRA and 41 GB for 70B QLoRA both fit with room to spare; the 128 GB memory calculator checks other sizes. It trains slower per token than a datacenter GPU, but the Spark fine-tuning guide walks through a 32B LoRA run on one node.
• 70B in 16-bit, or anything you want back in two hours: rented H100s or A100s. At torchtune's published 8x A100 rate, a 70B LoRA on this example's 24 million tokens would take under two hours. The H100, H200 and B200 price comparison lists per-GPU rates, and the DGX Spark vs H100 page covers the memory and bandwidth trade.

My pick for one engineer adapting a model to company data: start with an 8B or 14B model on a rented RTX 5090, prove the dataset improves your evaluation, and only then pay for a 32B or 70B run. Most of the quality comes from the data, and the Spark vs RTX 5090 comparison is the place to decide whether you need the bigger memory at all.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• Unsloth requirements, Unsloth docs
• Training LLMs with Blackwell, RTX 50 series and Unsloth, Unsloth docs
• Unsloth benchmarks, Unsloth docs
• gpt-oss: how to run and fine-tune, Unsloth docs
• torchtune README: memory and training speed, PyTorch, GitHub
• How NVIDIA DGX Spark's performance enables intensive AI tasks, NVIDIA Technical Blog, 2025-10-24
• RTX AI Garage: fine-tuning with Unsloth and DGX Spark, NVIDIA Blog, 2025-12-15
• PyTorch 2.7 release, PyTorch, 2025-05-15
• Installation, Axolotl docs
• Reducing memory usage, Hugging Face TRL docs
• Releases, bitsandbytes, GitHub
• Nvidia GeForce RTX 5090 Founders Edition review, Tom's Hardware
• Pricing, RunPod