llama.cpp --n-cpu-moe and expert offload: running gpt-oss-120b and 235B MoE models on a 16 to 32 GB GPU

llama.cpp's --n-cpu-moe N keeps the expert weights of the first N layers in system RAM and everything else on the GPU, which lets a 24 to 32 GB card run gpt-oss-120b: reported speeds are about 22 tok/s on an RTX 4090 (--n-cpu-moe 28) and about 30 tok/s on an RTX 5090 (--n-cpu-moe 21). Speed is set by RAM bandwidth, not the GPU: gpt-oss-120b reads about 1.9 GB of experts per token, so dual-channel DDR5-6000 (96 GB/s) caps it near 50 tok/s, and a 235B-class model at 4-bit near 12. Plan on 64 GB of RAM for gpt-oss-120b with a 24 GB card and 96 to 192 GB for a 235B model, depending on the quant; figures checked 7 November 2026.

• Flags: --cpu-moe (all experts in RAM), --n-cpu-moe N (first N layers), or -ot "exps=CPU" for regex control.
• Speed rule: tok/s ceiling = RAM bandwidth / expert GB read from RAM per token; reported runs land at roughly half.
• gpt-oss-120b: 63.4 GB MXFP4 GGUF, 3.58B active expert parameters per token, about 1.9 GB read per token.
• RAM that matters: channels x MT/s x 8 bytes. Two DDR5-6000 channels are 96 GB/s; a quad-channel Threadripper platform doubles that.

How does MoE expert offload work in llama.cpp?

The routed experts, which hold most of a MoE model's bytes but are only partly used per token, go to system RAM; attention, shared experts, embeddings and the KV cache stay in VRAM.

A mixture-of-experts model activates a few experts per layer per token (the glossary explains active versus total parameters). In gpt-oss-120b's config.json, each of 36 layers picks 4 of 128 experts. The Doctor-Shotgun guide (30 January 2026) gives the logic: attention and shared FFNs run for every token, so they belong on the GPU, while each routed expert is read only when chosen, so the CPU can compute it from RAM at an acceptable cost.

This beats the old approach of splitting whole layers. In a Zenn write-up on an RTX 4090 (4 October 2025), gpt-oss-120b ran at 12.98 tok/s with 12 full layers on the GPU, 17.77 with all experts on the CPU (--cpu-moe, about 8.5 GB of VRAM) and 21.96 with --n-cpu-moe 28. CPU only was 9.41. A DEV Community test on an RTX 3090 that split layers instead of experts got 7.65 tok/s on the same model.

Prompt processing works differently. For batches above 32 tokens, llama.cpp copies the CPU-held weights to the GPU over PCIe and processes the batch there, the Doctor-Shotgun guide explains, which is why it recommends -b 4096 -ub 4096: a bigger batch spreads the copy over more tokens.

How fast is it? The bandwidth math

Per token, decode has to read every active expert held in RAM once, so RAM bandwidth divided by those bytes gives a hard ceiling.

gpt-oss-120b, from config.json:
  per expert: 3 x 2880 x 2880            = 24.9M parameters
  per token:  4 experts x 36 layers x 24.9M = 3.58B parameters
  at MXFP4 (4.25 bits)                   = 1.90 GB read per token

ceiling with all experts in RAM:
  DDR5-6000, 2 channels: 6000 MT/s x 8 B x 2 = 96 GB/s  -> 96 / 1.90 = 50 tok/s
  measured (carteakey, RTX 4070, same RAM):              about 28 tok/s, 56% of ceiling
  same RAM at 2000 MT/s (XMP off), 32 GB/s:              9.72 tok/s measured

The RAM speed test is the cleanest evidence. carteakey's write-up found the BIOS running DDR5-6000 at 2000 MT/s; enabling XMP took generation from about 10 to about 28 tok/s on the same 12 GB RTX 4070. A 12 GB card with fast RAM then matched the roughly 30 tok/s that Georgi Gerganov reported for an RTX 5090 with --n-cpu-moe 21 in the llama.cpp gpt-oss guide, because the 5090 held more experts but the CPU side still set the pace.

Decode ceilings in tok/s with every routed expert in RAM, by my calculation: layers x experts per token x 3 x hidden_size x moe_intermediate_size from each config.json, times bytes per parameter. Bytes per parameter is 4.25 bits for MXFP4 and, for the other GGUFs, file size divided by total parameters (3.0 to 5.3 bits), so treat those rows as estimates. File sizes from the ggml-org and unsloth repositories on Hugging Face, read 7 November 2026. Channel counts: AMD lists 2 channels for the Ryzen 9 9950X (DDR5-5600, 256 GB max) and 4 and 8 channels for the TRX50 and WRX90 Threadripper platforms. Real runs reach about 40 to 60% of these ceilings, and keeping some experts on the GPU raises them.

Measured bandwidth is lower than the label. A Level1Techs user with two 96 GB DDR5 kits on AM5 measured about 80 GB/s in AIDA, falling to about 75 GB/s at 4800 MT/s, against 77 to 80 GB/s theoretical.

What tok/s do people report for gpt-oss-120b and larger models?

gpt-oss-120b runs at about 20 to 30 tok/s on a single 12 to 32 GB card with fast DDR5; 235B-class models at about 8 to 10 tok/s on desktop platforms.

All third-party reports; none of these setups has been measured by GPUwerk. The DEV post shows only a month and day; it tests Gemma 4, released in April 2026, so I date it 2026. The Hardware Corner 3090 run is an outlier the article does not explain; a 59 GiB model in 64 GB of RAM leaves little room, which is my guess at the cause. The same article's RTX 5090 run with an F16-labelled GGUF reached only 8.1 to 9.6 tok/s.

The spread inside one model is large, and it tracks RAM and placement more than the GPU. If you get single digits on gpt-oss-120b with a recent build, check memory speed in the BIOS and free RAM before touching llama.cpp flags. For dense models there is no equivalent trick; offloading dense layers is what produced the 7.65 tok/s row.

How much system RAM do you need?

The GGUF size, minus what stays on the GPU, plus 8 to 16 GB for the operating system and buffers.

• gpt-oss-120b (63.4 GB, about 61 GB of it experts): with --n-cpu-moe 21 on a 32 GB card, about 21/36 of the experts, 35.5 GB, sit in RAM; with --n-cpu-moe 28 on 24 GB, about 47 GB. 64 GB works for both. On a 12 to 16 GB card most experts go to RAM: carteakey notes that --n-cpu-moe 36 loads about 60 GB and can crash a 64 GB system, so buy 96 GB. For 16 GB cards, the 16 GB model guide lists what fits without offload.
• GLM-4.5-Air: Q4_K_M is 73.0 GB and UD-Q2_K_XL 47.4 GB; 96 GB for the 4-bit file, 64 GB for the 2-bit one.
• Qwen3-235B-A22B: UD-Q2_K_XL is 88.8 GB and UD-Q4_K_XL 134.3 GB. With a 32 GB card holding about 25 GB of weights, 96 to 128 GB of RAM fits the 2-bit file; the 4-bit one needs 128 GB at the very least and 192 GB to be comfortable. Unsloth's Qwen3 guide uses -ot ".ffn_.*_exps.=CPU" for this model, which does much the same as --cpu-moe.

The RAM and GPU split are my arithmetic from the file sizes and layer counts. A desktop AM5 board tops out at 256 GB per AMD's 9950X specification, which is enough for anything up to a 4-bit 235B model, but its two channels cap speed. The KV cache table gives the VRAM the context needs on top, since the cache stays on the GPU.

Which commands should you start with?

Start with --cpu-moe to see that it loads, then switch to --n-cpu-moe and lower N until VRAM is nearly full.

# 1. all routed experts in RAM, rest on the GPU (about 8 GB VRAM for gpt-oss-120b)
llama-server -hf ggml-org/gpt-oss-120b-GGUF --cpu-moe -c 32768 --jinja -fa on

# 2. experts of the first N layers in RAM; lower N while VRAM allows
llama-server -hf ggml-org/gpt-oss-120b-GGUF --n-cpu-moe 21 -c 32768 --jinja \
  -fa on -b 4096 -ub 4096 --threads 10

# 3. the same idea with a regex: layers 0-30 fully on GPU 0, other experts on CPU
llama-server -m model.gguf -ngl 999 \
  -ot "blk\.([0-9]|[1-2][0-9]|30)\.=CUDA0,exps=CPU"

Command 2 follows the gpt-oss guide's RTX 5090 example; the batch sizes and thread count follow the Doctor-Shotgun and carteakey write-ups, and command 3 is the Doctor-Shotgun regex. Per the llama-server README, --n-cpu-moe counts from the first layers, earlier -ot rules take priority, and current builds have --fit on by default, which adjusts unset arguments to fit device memory (carteakey used it with --fit-ctx 32768 before settling on a faster static placement). Newer builds also list --moe-cache-mib, a GPU cache for CPU-held experts; I found no published numbers for it yet. Two more settings from the reports: pin threads to performance cores on Intel hybrid CPUs, and keep -np 1, since parallel slots split the speed (Zenn saw 12.21 and 13.01 tok/s per request at -np 2). Building llama.cpp itself is covered in the llama.cpp doc.

When is offload the wrong answer?

When the model has many active experts and you only have two memory channels, or when several people need it at once.

For gpt-oss-120b on a 24 to 32 GB card such as an RTX 5090 or a used RTX 3090, offload is a good deal: 20 to 30 tok/s for the price of 64 GB of DDR5. For Qwen3-235B or MiniMax-class models on a desktop board, the table above caps you at 11 to 34 tok/s before efficiency losses, and the reported 7.5 to 10 tok/s runs match that. A quad-channel workstation doubles the ceiling, and a unified-memory machine removes the split: gpt-oss-120b on a DGX Spark is reported near 60 tok/s with every weight in its 128 GB pool, as the gpt-oss-120b page lists, and the DGX Spark vs RTX 5090 and DGX Spark vs Strix Halo comparisons cover the trade-offs. Offload is also a one-user setup; with experts in RAM there is no headroom for batching.

My pick for one person with a 24 or 32 GB card: gpt-oss-120b with --n-cpu-moe, 64 GB of DDR5 at its rated speed (check XMP or EXPO first), and no further tuning until you have measured. If a Qwen3-235B-class model is the goal, budget for 128 GB on four channels, or skip the GPU split altogether.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• llama-server README, llama.cpp on GitHub
• Add --cpu-moe and --n-cpu-moe options, PR #15077, llama.cpp on GitHub, 2025-08-04
• guide: running gpt-oss with llama.cpp, discussion #15396, llama.cpp on GitHub, 2025-08-18
• Performant local mixture-of-experts CPU inference with GPU acceleration in llama.cpp, Hugging Face blog (Doctor-Shotgun), 2026-01-30
• Optimizing gpt-oss-120b speed on consumer hardware, carteakey.dev, 2026-03-13
• gpt-oss-120b をローカルで快適に使いたい, Zenn (iizuka kota), 2025-10-04
• GPT-OSS 120B: Offloading MoE Layers to CPU Boosts RTX 3090 and 5090 Performance, Hardware Corner, 2025-11-10
• vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM, DEV Community
• Full DeepSeek Q1 with the IK version of llama.cpp on AM5, page 5, Level1Techs Forums, 2025-08-23
• Qwen3: How to Run & Fine-tune, Unsloth docs
• config.json, openai/gpt-oss-120b, Hugging Face
• ggml-org/gpt-oss-120b-GGUF, Hugging Face
• unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF, Hugging Face
• AMD Ryzen 9 9950X specifications, AMD
• AMD Ryzen Threadripper processors, AMD