How to run an LLM locally in November 2026: pick a model by memory, install a runner, chat, and measure speed

To run an LLM locally, find out how much GPU memory (or Mac unified memory) you have, pick a model whose 4-bit download is well under that, install a runner, and talk to it. On 7 November 2026 the shortest path on Windows, macOS and Linux is Ollama v0.40.15 (released 3 November 2026): one install command, then ollama run qwen3.5:9b for a 6.6 to 7.6 GB model that suits 8 to 16 GB cards, or ollama run qwen3.8:27b (18 GB) on a 24 or 32 GB card. LM Studio 0.4.28 is the graphical alternative, llama.cpp (build b12189) gives full control, and vLLM v0.33.0 is for serving many users on Linux.

Download sizes from the Ollama library pages for qwen3.5, gemma4, qwen3.8, qwen3.6 and gpt-oss, read 7 November 2026; ranges cover the GGUF and MLX builds. The picks are my starting points, not benchmark winners.

Step 1: how much memory does the model need?

The model file, plus the KV cache for your context length, plus 1 to 2 GB for the runtime. The file size is on the download page. The KV cache comes from the model's config.json: bytes per token = 2 (keys and values) × attention layers × KV heads × head dimension × bytes per value (2 at f16). Two worked examples:

• Qwen3.8-27B. Its config.json lists 64 layers, of which only 16 use full attention, with 4 KV heads of dimension 256. That is 2 × 16 × 4 × 256 × 2 = 65,536 bytes (64 KiB) per token, so 2 GiB at 32k tokens and 4 GiB at 64k. With the 18 GB file, 32k fits a 24 GB card with about 2 GB to spare.
• gpt-oss-20b. Its config.json has 12 full-attention layers (the other 12 keep only a 128-token window) with 8 KV heads of dimension 64: 24 KiB per token. A 32k context adds 768 MiB to the 14 GB file, which is why it squeezes onto 16 GB cards at short contexts and has room for 128k on a 32 GB card.

Ollama chooses a default context for you based on GPU memory: 4k tokens under 24 GiB, 32k from 24 to 48 GiB, and 256k above that, per its context-length page. Coding agents and web search need more than 4k; the same page recommends at least 64,000 tokens for them. For larger models, the memory calculator does this arithmetic for 128 GB, and the sibling guide to the best LLMs for 16 GB of VRAM goes model by model for the most common card size.

Step 2: which tool should you use?

Ollama unless you have a reason not to. All four below run the same open models; they differ in how much they decide for you.

Versions from each project's GitHub releases page or LM Studio's changelog, read 7 November 2026. LM Studio does not support Intel Macs, per its system requirements.

The trade-offs in more depth: LM Studio vs Ollama covers licensing and privacy, and Ollama vs vLLM covers what happens when more than one person sends requests (on one RTX 5090, 74 against 936 tok/s at 32 concurrent requests in a published test).

Step 3: install it

Use the vendor's own installer or package; each is one or two commands.

Ollama

On Linux, and on macOS from the terminal, the README's script; on Windows, the PowerShell line or the OllamaSetup.exe installer (no administrator rights needed, per the Windows docs, which ask for Windows 10 22H2 or newer and NVIDIA driver 551.61 or newer):

# Linux and macOS
curl -fsSL https://ollama.com/install.sh | sh

# Windows (PowerShell)
irm https://ollama.com/install.ps1 | iex

On Linux the script registers a systemd service; check it with sudo systemctl status ollama and read logs with journalctl -e -u ollama, as the Linux docs describe. On a Mac you can instead drag the app from the .dmg into Applications; Apple silicon gets GPU acceleration, Intel Macs run on the CPU only (macOS docs).

LM Studio

Download the desktop app from lmstudio.ai. For a server without a screen, LM Studio 0.4.0 added llmster, the same core as a daemon:

# Linux and macOS
curl -fsSL https://lmstudio.ai/install.sh | bash
# Windows (PowerShell)
irm https://lmstudio.ai/install.ps1 | iex

lms daemon up
lms get <model>
lms server start

llama.cpp

Package managers, per the project's install page, or a prebuilt zip from the releases page (Windows builds for CUDA 12, CUDA 13 and Vulkan):

brew install llama.cpp        # macOS or Linux with Homebrew
winget install llama.cpp      # Windows

llama-server -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL --port 8080

The -hf flag downloads a GGUF from Hugging Face (here a 35B-A3B model that suits a 32 GB card and is tight on 24 GB) and defaults to Q4_K_M if you leave the quant off. Set --port: the server README gives the default as 9931, not the 8080 older guides assume. Building with CUDA yourself is covered in the llama.cpp build guide.

vLLM

Linux only (WSL on Windows), in a fresh environment, per the install docs:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Step 4: send your first prompt

With Ollama, ollama run downloads the model on first use and opens a chat in the terminal; ollama ps then shows what is loaded.

ollama run qwen3.5:9b
ollama ps

Check the PROCESSOR column in ollama ps: it should say 100% GPU. A split such as 40%/60% CPU/GPU means the model or its context did not fit, and speed drops sharply; pick a smaller model or a shorter context. The CONTEXT column shows the context Ollama allocated.

Every runner here also speaks an HTTP API, which is what editors, scripts and chat UIs use. Ollama listens on 127.0.0.1:11434:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:9b",
  "messages": [{"role": "user", "content": "Explain a KV cache in two sentences."}],
  "stream": false
}'

The OpenAI-style route works too (http://localhost:11434/v1/chat/completions), as do LM Studio's on port 1234 and llama-server's on the port you set. That is how you point coding agents such as Continue, Cline or aider at a local model.

Step 5: connect a chat UI

LM Studio and llama-server have a chat window built in (llama-server's web UI is on by default at its own port). For Ollama, Open WebUI is the usual choice. The quick start gives a Docker command that reaches Ollama on the same machine:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  -e WEBUI_SECRET_KEY=your-secret-key \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 and create the first account, which becomes the admin. Without Docker, DATA_DIR=~/.open-webui uvx --python 3.12 open-webui@latest serve works; Open WebUI supports Python 3.11 and 3.12, not 3.13. Accounts, signup control, documents and HTTPS are in the Open WebUI setup guide.

Step 6: how fast is it, and is that right?

Measure tokens per second, then compare it with what the hardware allows. Ollama's chat API returns eval_count (tokens generated) and eval_duration (in nanoseconds) with every non-streamed response:

tok/s = eval_count / eval_duration × 1,000,000,000
e.g. 512 tokens / 6,400,000,000 ns = 80 tok/s

With llama.cpp, llama-bench -m model.gguf prints two numbers: pp512 (prompt processing over 512 tokens) and tg128 (generating 128 tokens), per the llama-bench README.

For a dense model, decode reads every weight once per token, so memory bandwidth divided by model size is the ceiling. By my calculation for an RTX 5090: 1,792 GB/s (per NVIDIA's spec table) ÷ 18 GB for qwen3.8:27b ≈ 100 tok/s. ComputingForGeeks reported 76.8 tok/s for Ollama on that card with Qwen3.8-27B at Q4_K_M and the MTP draft off in October 2026, about three quarters of the ceiling, which is normal. If you see a third of the ceiling, something spilled to the CPU. Mixture-of-experts models such as qwen3.6:35b-a3b read only their active experts per token (about 3B parameters for that model), so they decode much faster than their file size suggests.

What usually goes wrong?

Four problems cover most first-day failures.

• The model is slow and ollama ps shows CPU in PROCESSOR. The file plus context does not fit. Use a smaller quant, cap the context with OLLAMA_CONTEXT_LENGTH, or quantize the KV cache with OLLAMA_KV_CACHE_TYPE=q8_0.
• A coding agent forgets the start of the task. The context is still at the 4k default for cards under 24 GiB. Raise it to 64k if memory allows.
• Answers ignore your tools. The model or its template does not support tool calling; use a model whose model card lists tool support.
• The API is reachable from the network. Ollama binds 127.0.0.1 by default (Ollama FAQ); setting OLLAMA_HOST=0.0.0.0 exposes it, and the docs describe no API key for it, so use an SSH tunnel or a proxy with authentication instead.

My recommendation for a first evening: Ollama, qwen3.5:9b on a 16 GB card or qwen3.8:27b on 24 GB and up, Open WebUI in Docker, and one speed check against the bandwidth ceiling. Move to llama.cpp when you want a flag Ollama does not expose, and to vLLM when a second person starts using it. If the model you actually need is in the 65 to 81 GB row, try it on 128 GB of rented memory before buying a machine; the DGX Spark vs Mac Studio page compares the two common ways to own that much.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• Releases · ollama/ollama, GitHub, 2026-11-03
• ollama/ollama README, GitHub
• Linux, Ollama docs
• Windows, Ollama docs
• macOS, Ollama docs
• Context length, Ollama docs
• FAQ, Ollama docs
• Chat API, Ollama docs
• qwen3.5, Ollama library
• gemma4, Ollama library
• qwen3.8 tags, Ollama library
• qwen3.6 tags, Ollama library
• gpt-oss, Ollama library
• LM Studio changelog, LM Studio, 2026-10-31
• System requirements, LM Studio docs
• Introducing LM Studio 0.4.0, LM Studio, 2026-01-28
• Releases · ggml-org/llama.cpp, GitHub, 2026-11-07
• Install llama.cpp, GitHub
• llama-server README, GitHub
• llama-bench README, GitHub
• GPU installation, vLLM docs
• Quick start, Open WebUI docs
• Qwen/Qwen3.8-27B config.json, Hugging Face
• openai/gpt-oss-20b config.json, Hugging Face
• Compare GeForce graphics cards, NVIDIA
• Ollama vs vLLM vs llama.cpp: Speed Benchmarked, ComputingForGeeks, 2026-10-06