Reasoning model token budget: thinking tokens per answer, seconds on local GPUs, and how to cap it

Open reasoning models spend roughly 660 to 8,600 thinking tokens before the first answer token on Artificial Analysis's speed-test prompts (my arithmetic from its 7 November 2026 data), and over 20,000 per problem on hard math. At local speeds that is the wait: 5,200 thinking tokens take about 82 seconds on an RTX 5090 running a dense 27B at 63.7 tok/s, and 136 seconds on a DGX Spark at 38.1 tok/s. Every current family has a switch, from enable_thinking: false and reasoning_effort to llama.cpp's --reasoning-budget and vLLM's thinking_token_budget, and published scores show the cost of using it is large on math and agentic coding and small on knowledge questions.

• Whole-index token use: 63.3M (GLM-5.3 low) to 174M (DeepSeek V4.1 Flash max) output tokens to run the Artificial Analysis Intelligence Index v5.0.1.
• Effort pays unevenly: DeepSeek V4.1 Flash max uses 30% more tokens than high for 4 index points; GLM-5.3 max uses 2.1 times low's tokens for 9 points.
• Time rule: seconds to first answer token is about thinking tokens divided by decode tok/s, plus prefill.
• Cheapest switch: --reasoning-budget 0 in llama.cpp, "reasoning_effort": "none" in vLLM, "think": false in Ollama.

How many tokens do reasoning models spend?

Between 63 and 174 million output tokens for a full Artificial Analysis index run, and from under a thousand to about nine thousand thinking tokens on a typical speed-test prompt.

Artificial Analysis model pages, read 7 November 2026; index v5.0.1 (methodology updated 3 November 2026) has nine evaluations, several of them agentic. Its gpt-oss-120b page describes output speed as measured end to end including reasoning tokens. The last column is my estimate: time to first answer token times output speed. It slightly overstates thinking because the time also includes queueing and prefill. Medians on these pages are 99.3M index tokens for the larger open-model class and 77.7M for Qwen3.8 27B's class.

The token totals are not per-question counts; the methodology runs some evaluations several times, so compare models with each other, not with your own prompts. The order is the useful part: Kimi K3.1 Pro scores highest while using fewer tokens than DeepSeek V4.1 Flash at max or MiMo-V2.6-Flash. For a single hard problem, OpenAI's gpt-oss model card gives the upper range: gpt-oss-20b uses "over 20k CoT tokens per problem on average" on AIME at high effort.

How long does thinking take on an RTX 5090 or a DGX Spark?

Divide thinking tokens by single-user decode speed: at 40 to 100 tok/s, a 5,000-token think takes one to two minutes before any answer appears.

My arithmetic: tokens divided by a reported single-user speed. Speeds: Qwen3.6-27B q4_0 on an RTX 5090 in llama.cpp without and with MTP (PR #22673, May 2026); gpt-oss-120b with --n-cpu-moe 21 on an RTX 5090 (llama.cpp gpt-oss guide, August 2025); gpt-oss-120b MXFP4 tg32 at depth 0 on a DGX Spark (llama.cpp DGX Spark benchmarks, build 7941); Qwen3.8-27B NVFP4 with a DFlash2 draft on SGLang at 4k (SparkBench, 10 September 2026); Qwen3.6-35B-A3B NVFP4 with MTP on vLLM at 4k (SparkBench, 4 August 2026). The row labels say where each token count comes from; the speeds belong to the column's model, so this is a time scale, not a per-model prediction.

The table is optimistic for long thinking, because decode slows as the chain of thought fills the context. The same llama.cpp page has gpt-oss-120b at 58.7 tok/s at depth 0, 49.5 at 16k and 42.8 at 32k; SparkBench has Qwen3.8-27B falling from 38.1 at 4k to 19.1 at 50k. Speculative decoding helps here more than anywhere, since reasoning text is long and predictable (see MTP and EAGLE-3 speedups). A long think also grows the KV cache; the KV cache table gives the memory per token.

On rented hardware the cost is small per answer and large per day: on a DGX Spark at $0.79 an hour, 5,200 tokens at 38.1 tok/s is 136 seconds, or $0.03, but a single user who sends 200 such prompts waits over seven hours in total.

How much quality do you lose with less thinking?

A lot on competition math, science and agentic coding; little on knowledge tests.

Vendor-reported scores. OpenAI's card is dated 5 August 2025; the DeepSeek card was read 7 November 2026.

The pattern is the same in both: the first step up (off to High for DeepSeek, low to medium for gpt-oss) buys most of the gain on reasoning-heavy tests, and the last step costs the most tokens for the least. DeepSeek-V4-Flash gains 21.3 points on HLE from non-think to High and 5.4 more from High to Max, while SWE Verified moves 4.9 and then 0.4. The Artificial Analysis pairs agree on the token side: DeepSeek V4.1 Flash max spends 40M more index tokens than high for 4 points, and its time to first answer token more than doubles. MMLU-style knowledge barely moves, so for retrieval, summaries and chat, thinking off is usually the right default.

How do you cap or switch off thinking for each model?

Each family has its own switch in the chat template or system prompt; the engines add a generic token cap on top.

DeepSeek V4 ships no Jinja chat template; its card selects the mode with an encode_messages(..., thinking_mode=...) helper, so check that your engine's DeepSeek V4 integration exposes it. Qwen3.8-27B also keeps earlier turns' thinking by default (preserve_thinking), which grows context in long agent sessions.

Which engine flags set a thinking budget?

llama.cpp and vLLM both have a hard token cap; Ollama has on, off and named levels only.

# llama.cpp: no thinking, a 2,048-token cap, or a template switch
llama-server -m model.gguf --reasoning-budget 0
llama-server -m model.gguf --reasoning-budget 2048 \
  --reasoning-budget-message "Time is up. Give the answer now."
llama-server -m model.gguf --chat-template-kwargs '{"enable_thinking": false}'

# vLLM: per request (OpenAI-compatible body)
{"model": "...", "messages": [...], "reasoning_effort": "low"}
{"model": "...", "messages": [...], "thinking_token_budget": 2048}
{"model": "...", "messages": [...], "chat_template_kwargs": {"enable_thinking": false}}

# Ollama: per request
{"model": "gpt-oss:120b", "messages": [...], "think": "low"}
{"model": "qwen3", "messages": [...], "think": false}

Details from the docs. In the llama-server README, --reasoning-budget takes -1 for unrestricted (the default), 0 to end thinking immediately, or a positive token count, and --reasoning-format decides whether thoughts land in reasoning_content. vLLM's reasoning outputs page (5 November 2026) needs --reasoning-parser (qwen3, deepseek_r1, gemma4, glm45 and others), counts thinking_token_budget from the reasoning start string set in --reasoning-config, maps reasoning_effort: none to thinking off, and offers --default-chat-template-kwargs for a server-wide default; reasoning now comes back in a field named reasoning, not reasoning_content. Ollama's thinking docs accept true, false or a level name, list gpt-oss levels as low, medium and high with medium the default, and document no token budget. For engine setup on a Spark, see the llama.cpp and vLLM docs.

A hard cap is blunter than an effort level. The model was trained to finish its own chain of thought; cutting it mid-sentence and forcing an answer is a different regime from asking for low effort, and I found no published accuracy table for truncated budgets on these models. Prefer the model's own effort setting, and use the token cap as a ceiling against runaway thinking.

Which thinking budget should you use locally?

Off for chat, retrieval and summaries; the lowest effort that passes your own test set for coding and analysis; high only for problems where you would wait minutes anyway.

For one person on an RTX 5090 or a DGX Spark, I would default Qwen3.8-27B or Qwen3.6 to enable_thinking: false, and gpt-oss-120b to Reasoning: medium, which the card shows recovers most of the AIME gain over low (80.0 against 50.4) at a fraction of high's length. Then set --reasoning-budget or thinking_token_budget to about 8,000 tokens as a guard, which caps the wait at about 1.3 to 4.5 minutes at the speeds above. Agent loops are the exception: tool-heavy tasks gained 10 points on SWE-bench Verified from medium to high for gpt-oss-120b, and an agent that retries a failed step burns more tokens than one that thought longer the first time. Measure on your own prompts; the DGX Spark model guide lists reported speeds to plug into the table above, and the RTX 5090 page has that card's rental details.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• Qwen3.8 27B: intelligence, performance and price analysis, Artificial Analysis
• DeepSeek V4.1 Flash 1017 (max), Artificial Analysis
• DeepSeek V4.1 Flash 1017 (high), Artificial Analysis
• GLM-5.3 (max), Artificial Analysis
• GLM-5.3 (low), Artificial Analysis
• Kimi K3.1 Pro, Artificial Analysis
• MiMo-V2.6-Flash, Artificial Analysis
• Nemotron 3.5 Super VL, Artificial Analysis
• gpt-oss-120b (high), Artificial Analysis
• Intelligence Index methodology, Artificial Analysis
• gpt-oss-120b & gpt-oss-20b Model Card, OpenAI (arXiv), 2025-08-08
• deepseek-ai/DeepSeek-V4-Flash model card, DeepSeek on Hugging Face
• Qwen3.8-27B model card, Qwen on Hugging Face
• NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 model card, NVIDIA on Hugging Face, 2026-03-11
• Reasoning Outputs, vLLM docs, 2026-11-05
• llama-server README, llama.cpp on GitHub
• Thinking, Ollama docs
• DGX Spark benchmarks (llama-bench, build 7941), llama.cpp on GitHub
• guide: running gpt-oss with llama.cpp, discussion #15396, llama.cpp on GitHub, 2025-08-18
• llama + spec: MTP Support, PR #22673, llama.cpp on GitHub, 2026-05-16
• radixark_qwen3.8-27b, SparkBench, 2026-09-10
• nvidia_qwen3.6-35b-a3b, SparkBench, 2026-08-04