The best models you're not allowed to send your data to
The open-weight models are good now. Not "good for open", good. The awkward part is that the organisations with the most to gain from them are the ones whose lawyers will never let a prompt leave the building. That constraint used to mean settling for a 7B toy. It doesn't any more, and the reason is memory capacity.
How far behind is open, really?
About one release cycle. Epoch AI's open-vs-closed tracking puts the best open model roughly 8 ECI points behind the closed frontier, which they characterise as a gap of about four months, not a generation. Artificial Analysis tells a similar story: on their Intelligence Index the leading open model has repeatedly closed to within a handful of points of the top closed model before the closed labs ship again and reopen the gap.
Coding is where you can see it most concretely. Z.ai reports GLM-4.6 at 68.0% on SWE-bench Verified and 82.8% on LiveCodeBench v6; on their own CC-Bench head-to-head against Claude Sonnet 4 it won 48.6% of tasks, tied 9.5% and lost 41.9%. Cline's diff-edit measurements put GLM-4.6 at 94.9% against Sonnet 4.5's 96.2%. Those are not the numbers of a consolation prize.
So the interesting question stopped being "are open models good enough" and became "which of them can I actually hold in memory."
What fits, and where
The number that decides everything is total weights in memory, not active parameters. A Mixture-of-Experts model routes only a fraction of its parameters per token (that's why it's fast), but every expert has to be resident. Below, Q4-class quantization (roughly 4 bits per weight, the quality/size sweet spot most people ship) and file sizes as published by the quantizers themselves. Leave 10–20% headroom on top for KV cache and context.
| Model | Params (active) | ≈ Q4 weights | 128 GB | 256 GB |
|---|---|---|---|---|
| Qwen3 32B | 32B dense | ~20 GB (Ollama) | Yes, easily (advantage) | Yes (advantage) |
| DeepSeek-R1-Distill-Qwen-32B | 32B dense | ~20 GB (bartowski) | Yes, easily (advantage) | Yes (advantage) |
| Llama 3.3 70B / R1-Distill-Llama-70B | 70B dense | 42.5 GB (Unsloth Q4_K_M) | Yes (advantage) | Yes (advantage) |
| Llama 4 Scout | 109B (17B active) | ~67 GB Q4_K_M (Ollama) | Yes (advantage) | Yes (advantage) |
| GLM-4.5-Air | 106B (12B active) | ~67 GB Q4 | Yes (advantage) | Yes (advantage) |
| gpt-oss-120b | 117B (5.1B active) | ~60–65 GB, native MXFP4 (Ollama: 65 GB) | Yes, the flagship fit (advantage) | Yes (advantage) |
| Mistral Large 2411 | 123B dense | 73.2 GB Q4_K_M (bartowski) | Yes (advantage) | Yes (advantage) |
| Qwen3 235B-A22B | 235B (22B active) | ~132–143 GB Q4_K_M | No | Yes (advantage) |
| GLM-4.6 | 355B (32B active) | ~200 GB+ Q4 | No | Tight / lower quant |
| Llama 4 Maverick | 400B (17B active) | ~245 GB Q4_K_M (Ollama) | No | Tight |
| DeepSeek V3 / R1 (full) | 671B (37B active) | ~400 GB Q4 | No | No |
Two things jump out of that table. First: 128 GB is not an arbitrary number. It is the capacity class where the 100–120B tier stops being a datacentre problem: gpt-oss-120b, GLM-4.5-Air, Llama 4 Scout and Mistral Large all fit with room for context. A 24 GB consumer card holds none of them. Second: linking two nodes to 256 GB moves you up exactly one tier, into Qwen3 235B territory, which is where "close to frontier" becomes a defensible claim. The 256 GB is split across two machines, so the model has to be served with a runtime that supports distributed execution.
Capacity is not speed
The GB10's ~273 GB/s of memory bandwidth is what makes 128 GB affordable, and it's also the tradeoff. On llama.cpp's DGX Spark benchmark discussion, gpt-oss-120b generates in the ballpark of 35–60 tok/s depending on stack and settings, with prompt processing well over 1,000 tok/s. Dense 70B models are much slower per stream: bandwidth-bound, in single-digit tokens per second at FP8. The lesson is architectural: on this hardware, prefer MoE. A 120B MoE with ~5B active parameters gives you 120B-class knowledge at roughly small-model speed, which is the entire reason gpt-oss-120b is the default recommendation for a 128 GB box.
It also batches well. Concurrency benchmarks on the Spark show aggregate throughput climbing into the hundreds of tokens per second across many simultaneous requests, so a single box serving thirty people in a firm behaves very differently from one person watching a cursor blink.
What you actually give up
I rent this hardware out, and I'd still rather you hear it straight:
- The hardest reasoning. On the longest multi-step problems, the closed frontier is still ahead. If your task sits at the hardest end of what any model can do today, you will feel it.
- Agentic long-horizon work. Tool-calling reliability over dozens of steps is where closed models have invested most. Open models have closed a lot of this gap (see the SWE-bench numbers above), but "a lot" is not "all".
- Recency. You are running a snapshot. There is no silent upgrade next Tuesday. Some people consider that a feature: your evals stay valid, and your outputs don't change under you.
- Peripheral polish. Multimodality, voice, enormous context windows, and the surrounding tooling are generally rougher than an API's.
What you do not give up, in my experience, is the bulk of real enterprise work: summarisation, extraction, classification, drafting, retrieval-augmented Q&A over your own corpus, code assistance in a known codebase. For those, the 100B tier is not "nearly as good." It's fine.
When privacy simply outranks the last stretch of capability
| Sector | Why the API is closed to you | What runs locally instead |
|---|---|---|
| Legal | Privilege and confidentiality obligations don't survive a third-party processor in another jurisdiction | Discovery review, contract abstraction, precedent search over the firm's own matter archive |
| Healthcare | Patient data under GDPR Art. 9 special categories; hospital DPO signs off on nothing that leaves the estate | Clinical note summarisation, coding support, cohort search, trial screening |
| Finance | Supervisory rules, client confidentiality, and MNPI handling; DORA makes third-party dependency itself the risk | KYC narrative drafting, filings analysis, internal research assistants |
| R&D and industry | Unpublished results and process know-how are the whole asset; no contract clause makes leaking them recoverable | Lab notebook Q&A, patent landscaping, spec and defect triage |
| Public sector | Data-residency mandates and procurement rules that presuppose EU operation | Case handling, translation, records search |
In each of these, the honest comparison isn't "open model vs. frontier API." It's "open model vs. nothing," because the API was never a legal option. A locally-run 120B beats an approval process that ends in "no."
The run-it-yourself path
Start with the fast path, then graduate to the serving path when the box becomes a team resource.
# 1. the 128 GB flagship, MoE, so it's fast as well as big ollama run gpt-oss:120b "Summarise this clause and flag unusual terms." # 2. a dense comparison point, for quality A/B against your own documents ollama run llama3.3:70b # 3. serve it to the team, OpenAI-compatible, on your own LAN vllm serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000
Then point Open WebUI, your editor plugins, or your internal RAG service at that base URL. Nothing in that stack phones home. The whole path from "we can't use AI" to "we have an internal assistant" is a machine, an afternoon, and a firewall rule.
Two practical notes. Quantize to Q4-class and stop worrying about it: the quality delta versus Q8 is small next to the delta between model tiers, and the tier you can fit is what matters. And build your eval set from your own documents before you pick a model; published benchmarks tell you who's in the running, your corpus tells you who wins.
Test it on a rented Spark first
128 GB of unified memory is the capacity class that turns this argument from theory into a purchase order. A DGX Spark holds the 100–120B tier at Q4 on a desk, off a normal wall socket. Two linked Sparks give you 256 GB and put Qwen3 235B in reach.
Rent one in EU-Central, $0.79/hour billed per minute, to find out whether your workload fits. A rented Spark is GPUwerk's machine in GPUwerk's rack, assigned only to you, so test with public or synthetic documents; it does not keep data inside your building. When the answer is yes, the same OS, models and commands run on a GB10 machine of your own: a GB10 machine from GPUwerk's store that GPUwerk hosts, or hardware for your own site, quoted on the on-premise page.