Blog/On-premise LLM cost vs cloud API cost: how to actually compare them
Cost analysis

On-premise LLM cost vs cloud API cost: how to actually compare them

By Samuel Seidel · Published September 9, 2026 · 8 min read

Dedicated hardware is cheaper than a per-token API once your volume is high and steady enough to keep the machine busy; below that, the API wins. This page gives you the four-step method to find that crossover for your own traffic, and works it through with GPUwerk's $0.79/hour rate. A cloud API charges per token, so no traffic means no bill. A dedicated machine charges per hour, busy or idle, and that difference is why most vendor comparisons mislead.

Per token vs per hour: what you are really comparing

A per-token API bills you for what you generate. Idle time costs nothing, because you share the provider's fleet with everyone else calling the same endpoint. A dedicated machine works the other way: you pay a flat rate for every hour it is assigned to you, whether you send it one request an hour or run it flat out. So its cost per token falls as utilization rises, and climbs without limit as utilization falls toward zero.

Low, spiky volume favors metered billing. Steady, high volume favors flat-rate hardware. The crossover is a number you calculate from your own traffic.

The formula

Four steps:

  1. Find your dedicated hardware's sustained throughput. Use the aggregate tokens/second your model reaches at the concurrency your real traffic runs at, not the single-stream number. Batch-1 and high-concurrency throughput on the same machine can differ by more than an order of magnitude, so use a number measured at a concurrency close to your own.
  2. Convert to tokens per hour, then to cost per token. Tokens/second × 3,600 gives tokens/hour. Divide the hourly rate by that figure and you have a cost-per-token number for the dedicated machine, assuming it stays busy at that concurrency the entire hour.
  3. Compare that figure to the API's per-token price. Most API pricing is split between input and output tokens at different rates, so match the ratio of input to output tokens in your own workload rather than comparing a single blended number.
  4. Multiply both by your monthly volume, then check the utilization assumption. The dedicated-hardware number only holds if the machine is busy at that concurrency for the hours you're counting. If your real traffic arrives in bursts with long gaps, the effective cost per token on dedicated hardware is higher than step 2 suggested, because you're still paying for the idle hours.

Worked example, using GPUwerk's own numbers

Take a workload generating 500 million output tokens a month. gpt-oss-120b reaches 862.8 tok/s aggregate at 256 concurrent requests on a single DGX Spark, per Dendro Logic's concurrency benchmark cited on GPUwerk's benchmarks page. That's 862.8 × 3,600 = 3,106,080 tokens/hour if the node stays saturated at that concurrency continuously. 500,000,000 ÷ 3,106,080 ≈ 161 hours. At GPUwerk's published $0.79/hour on-demand rate, 161 × $0.79 = $127.19 before tax.

That figure assumes the node is busy at 256 concurrent requests for the full 161 hours, back to back, with enough traffic to keep the queue full. Real traffic arrives in bursts. A machine billed by the hour that sits half-idle loses ground fast, because every hour on the clock is billed whether the GPU is generating tokens or not. A per-token API has no idle time to bill.

Put your chosen API's price for 500 million tokens next to $127.19 and you have the comparison for that volume. For your own traffic, the crossover comes down to how busy you can keep the machine.

What tips the answer one way or the other

Run it on your own numbers

The quickest real answer comes from measuring your own model's throughput instead of borrowing someone else's benchmark. Rent a DGX Spark by the hour, run your workload at your real concurrency, and put the measured tokens/second into step 1. A few hours at $0.79/hour replaces the Dendro Logic figure above with a number that describes your traffic. Current per-token prices for open models across API providers, with a break-even worked from published throughput, are in open-model API prices.

Related pages

First top-up: pay $20, get $30 in credit

Measure your own throughput before you commit to either model.

Run your real workload on a dedicated Spark for a few hours and read off your own crossover point.

A dedicated DGX Spark in Europe, North America or East Asia, $0.79/hour, billed per minute. $30 of credit is about 37 hours of compute.

Create account See the OpenAI API comparison