Best small language models under 10B: ranks, memory and speed, November 2026

The best small language model for most local setups in November 2026 is Qwen3.5-9B: Apache 2.0, a 5.68 GB Q4_K_M file, and about 1.07 GB of KV cache at a 32k context by my calculation, so it fits an 8 GB GPU. On Artificial Analysis's Progress Index (read 7 November 2026), K2 Horizon 3.7B leads the 4B-and-under class at 242, and K2 Horizon 7B, at 263, is the only sub-10B model in the top 12 of the 4B-to-40B class. For phones, Google reports Gemma 4 E2B at 52 to 57 tokens per second on the GPUs of two 2026 flagships.

Read 7 November 2026. KV figures are my calculations from each config.json at 16-bit precision, explained below.

Which small language models are current in November 2026?

Eight developers released general-purpose sub-10B open-weight models in 2026; Mistral's newest small model is still Ministral 3 (2512). Dates are the card's where given, otherwise the month the repository was created.

Sources: the linked Hugging Face cards and Liquid AI's blog, read 7 November 2026. K2 Horizon's 9B total comes from its untied 250,624-token embedding and output matrices: 2 x 250,624 x 4,096 = 2.05B parameters (my calculation from config.json).

Older names have not moved. Qwen3.6 and Qwen3.8 start at 27B. The newest small Phi is Phi-4-mini-flash-reasoning (June 2025, MIT, English only), Meta's are Llama 3.2 1B and 3B (25 September 2024), and the newest SmolLM is SmolLM3-3B (2025).

How do small models rank on independent benchmarks?

Artificial Analysis is the only independent ranking I found that covers current sub-10B models, and its Progress Index v1 puts K2 Horizon 3.7B first among open models of 4B parameters or fewer. Per the methodology page, the index (introduced October 2026) fits model capability and task difficulty on one item-response scale, anchored at 220 for o3-mini (high) and 320 for GPT-5.5 (xhigh). The newer Intelligence Index v5.0.1 has no score yet for most small models; the Qwen3.5 9B page shows N/A.

Source: Artificial Analysis tiny and small open model pages, read 7 November 2026; neither shows an update date and each shows only its top 12.

A 7B model above Gemma 4 31B on one scale is the headline. The gaps matter too: LFM3-1.7B is not listed yet, and Qwen3.5 4B and 9B and Gemma 4 E4B are unscored or below the 4B-to-40B page's visible cutoff of 246, so these pages cannot rank them against K2 Horizon.

Can you trust the scores on model cards?

Treat them as each vendor's best case: one model on one benchmark differs by up to 46 points between publishers. These scores are from each developer's own card.

Self-reported, read 7 November 2026; n/a means the card gives no score. IBM and NVIDIA write "GPQA" without naming the Diamond subset. Evaluation code, sampling and thinking settings differ between cards.

The comparison columns on these cards show the spread. Gemma 4 E4B gets 52.0 on LiveCodeBench v6 from Google, 71.00 from Liquid AI and 53.9 from OpenBMB, and its IFEval is 90.02 in Liquid's table but 44.4 in OpenBMB's. Qwen3.5-4B's IFBench is 59.2 from Qwen and 38.16 from Liquid. GPQA Diamond is the stable one: Qwen3.5-4B gets 76.2, 75.76 and 77.1 in three tables. Compare GPQA Diamond across cards, then run your own prompts; the guide to evaluating a model before switching covers building that test set.

How much memory does a sub-10B model need at Q4 and Q8?

Weights take 1.1 to 5.7 GB at Q4_K_M and 1.9 to 9.6 GB at Q8_0; the KV cache, 0.2 to 5.4 GB at a 32k context, is what separates them. Only full-attention layers store KV for every token:

KV bytes per token = 2 (K and V) x attention layers x KV heads x head_dim x 2 bytes (16-bit)

Granite 4.2 8B: 2 x 40 x 8 x 128 x 2 = 163,840 B  ->  x 32,768 tokens = 5.37 GB
Qwen3.5-9B:     2 x  8 x 4 x 256 x 2 =  32,768 B  ->  x 32,768 tokens = 1.07 GB
                (8 of its 32 layers are full attention; 24 are Gated DeltaNet)

It matches Liquid AI's figures: the LFM3 announcement gives 1.6 GB of KV at 128k for LFM3-1.7B, 4.3 GB for Qwen3.5-4B and 5.6 GB for MiniCPM5-2B, and the same arithmetic on their config.json files gives 1.61, 4.29 and 5.64 GB. Other layouts are in KV cache size by model.

File sizes (GB = 10^9 bytes) from the GGUF repositories of Unsloth (Qwen3.5, Gemma 4), IBM, IFM, Mistral, OpenBMB, NVIDIA and Liquid AI, read 7 November 2026. KV is my calculation; totals exclude compute buffers and Qwen3.5's 0.92 GB vision projector. *Gemma 4 shares KV across its last 18 (E4B) or 20 (E2B) layers and windows most others at 512 tokens; I counted the 4 or 3 full-attention layers that keep their own KV.

The split is architectural. Hybrids (Qwen3.5's DeltaNet, LFM3's convolutions, Nemotron's Mamba-2, Gemma 4's shared and windowed KV) stay under 1.1 GB at 32k; dense GQA models (Granite 4.2, Ministral 3, K2 Horizon, MiniCPM5) pay 1.4 to 5.4 GB. So K2-Horizon-3.7B needs almost 8 GB at 32k while Qwen3.5-9B needs 6.75 GB. llama.cpp's --cache-type-k q8_0 --cache-type-v q8_0 roughly halves the KV column; see the quantization guide for formats and building llama.cpp for flags.

How fast do small models run on phones, laptops and 8-16 GB GPUs?

Reported decode speeds run from 22 tokens per second for Gemma 4 E4B on a 2026 flagship phone to 182 for a 7B Q4_0 model on an RTX 5070 Ti. Google publishes phone numbers for Gemma 4; for GPUs and Macs the yardstick is llama.cpp's community scoreboard, Llama 2 7B at Q4_0 (3.56 GiB).

Read 7 November 2026. LiteRT-LM rows: 1,024 prefill and 256 decode tokens at a 2,048-token context, Google's own files (2.58 GB for E2B with mixed 2/4/8-bit weights, 3.65 GB for E4B). llama.cpp rows: llama-bench pp512 and tg128 with flash attention; the CUDA thread ran from 1 August 2025 to September 2026 and the M5 Max row was added on 25 August 2026. Reported figures, not reproduced by GPUwerk.

No such scoreboard exists yet for Qwen3.5 or Granite 4.2, so I scaled the Llama 2 7B results by bytes read per token, since decode at this size is memory-bound: tok/s = tok/s(Llama 2 7B) x 3.82 GB / (weight file + KV cache in use).

My estimates in tokens per second at Q4_K_M with a 16-bit KV cache, for example 93.46 x 3.82 / (5.68 + 1.07) = 53. Treat them as ceilings: DeltaNet layers use different kernels in llama.cpp.

On a short prompt the two tie. With 32k of context filled, the hybrid is about 1.6 times faster, because Granite reads 5.37 GB of cache per new token. Thinking costs wait time as well: Artificial Analysis measured 47.63 seconds to the first answer token for Qwen3.5 9B with reasoning on API providers and 1.05 seconds for the non-reasoning variant. The Qwen card turns thinking off with enable_thinking: False.

Which small models handle European languages?

Qwen3.5 claims the widest coverage, only Granite 4.2 lists Czech, and K2 Horizon, Nemotron 3 Nano 4B and Phi-4-mini-flash-reasoning are English-only:

• Qwen3.5: 201 languages and dialects; the 9B reports 81.2 on MMMLU and 72.6 on WMT24++.
• Gemma 4: pre-trained on 140+ languages, 35+ supported out of the box.
• Granite 4.2: 12 tested, including German, Spanish, French, Portuguese, Czech, Italian and Dutch.
• Ministral 3: 11, including French, Spanish, German, Italian, Portuguese and Dutch.
• LFM3-1.7B: 16, including French, German, Italian, Spanish, Portuguese, Polish and Russian.
• Apertus v1.5 8B: "a large variety of languages", no list, plus an EU AI Act public summary. MiniCPM5-2B covers English and Chinese.

No card reports Czech, Polish or Hungarian scores, so a 50-prompt test in your language decides more than any table here.

What do the licenses allow?

Twelve of the fourteen models here are Apache 2.0, with no revenue limit (Apertus adds a use policy). The exceptions are LFM3, whose LFM Open License v1.0 licenses commercial use only below 10 million US dollars of annual revenue (Section 5), and Nemotron 3 Nano 4B, whose NVIDIA Open Model License allows commercial use but ends your rights if you bypass its guardrails without a substitute. For EU companies, the Llama 3.2 license withholds rights only for its multimodal models, not the 1B and 3B text models. The open-weight model licenses article covers larger families.

Which small model should you pick?

For one model on an 8-16 GB GPU or a 16 GB Mac, I'd pick Qwen3.5-9B at Q4_K_M with thinking off for routine prompts. It tops the self-reported table on GPQA Diamond and MMLU-Pro, and its 1.07 GB cache at 32k fits an 8 GB card. For larger options on 16 GB, see the best LLMs for 16 GB of VRAM.

• Phone or browser app: Gemma 4 E2B, the only model here whose developer publishes phone speeds as text.
• Agents and tool calls in 1.5 GB: LFM3-1.7B if your company is under the revenue threshold, otherwise MiniCPM5-2B. Pair either with constrained decoding as in the structured output guide.
• Reasoning and code in English: K2-Horizon-7B. Budget 12 GB for a 32k context, or quantize the KV cache.
• Czech, German or Dutch business text: Granite 4.2 8B or Ministral 3 8B; keep context under 16k on an 8 GB card.
• One narrow task at volume: fine-tune a 2-4B model, as in the fine-tuning guide.

Skip Llama 3.2, Phi-4-mini and SmolLM3 for new work: SmolLM3-3B reports 41.7 on GPQA Diamond with thinking and Phi-4-mini-flash-reasoning 45.08, against 76.2 for Qwen3.5-4B. With 16 GB or more, a small MoE such as Gemma 4 26B-A4B reads only 3.8B active parameters per token, so its decode cost is close to a 4B dense model's. To test a shortlist at Q8 or BF16 first, GPUwerk's RTX 5090 rental has 32 GB, enough for Qwen3.5-9B in BF16 (17.92 GB) plus a long context.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• Tiny open source AI models (4B parameters or fewer), Artificial Analysis
• Small open source AI models (4B to 40B parameters), Artificial Analysis
• Artificial Analysis Progress Index methodology, Artificial Analysis
• Qwen3.5 9B: intelligence, performance and price analysis, Artificial Analysis
• LLM leaderboard: comparison of models, Artificial Analysis
• Qwen/Qwen3.5-9B model card, Hugging Face
• google/gemma-4-E4B-it model card, Hugging Face
• litert-community/gemma-4-E2B-it-litert-lm, Hugging Face
• litert-community/gemma-4-E4B-it-litert-lm, Hugging Face
• LiquidAI/LFM3-1.7B model card, Hugging Face
• LFM Open License v1.0, Liquid AI
• Introducing LFM3, Liquid AI, 2026-10-27
• openbmb/MiniCPM5-2B model card, Hugging Face
• ibm-granite/granite-4.2-8b model card, Hugging Face, 2026-08-25
• mistralai/Ministral-3-8B-Instruct-2512 model card, Hugging Face
• nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 model card, Hugging Face, 2026-03-16
• NVIDIA Open Model License Agreement, NVIDIA, 2025-10-24
• IFM/K2-Horizon-7B model card, Hugging Face, 2026-10-01
• IFM/K2-Horizon-3.7B model card, Hugging Face
• swiss-ai/Apertus-v1.5-8B model card, Hugging Face
• Performance of llama.cpp on Nvidia CUDA (discussion #15013), GitHub, ggml-org/llama.cpp
• Performance of llama.cpp on Apple Silicon M-series (discussion #4167), GitHub, ggml-org/llama.cpp
• unsloth/Qwen3.5-9B-GGUF files, Hugging Face
• ibm-granite/granite-4.2-8b-GGUF files, Hugging Face
• microsoft/Phi-4-mini-flash-reasoning model card, Hugging Face
• meta-llama/Llama-3.2-3B-Instruct model card, Hugging Face, 2024-09-25
• HuggingFaceTB/SmolLM3-3B model card, Hugging Face