On-premise LLM cost vs cloud API cost: how to actually compare them
Dedicated hardware is cheaper than a per-token API once your volume is high and steady enough to keep the machine busy; below that, the API wins. This page gives you the four-step method to find that crossover for your own traffic, and works it through with GPUwerk's $0.79/hour rate. A cloud API charges per token, so no traffic means no bill. A dedicated machine charges per hour, busy or idle, and that difference is why most vendor comparisons mislead.
Per token vs per hour: what you are really comparing
A per-token API bills you for what you generate. Idle time costs nothing, because you share the provider's fleet with everyone else calling the same endpoint. A dedicated machine works the other way: you pay a flat rate for every hour it is assigned to you, whether you send it one request an hour or run it flat out. So its cost per token falls as utilization rises, and climbs without limit as utilization falls toward zero.
Low, spiky volume favors metered billing. Steady, high volume favors flat-rate hardware. The crossover is a number you calculate from your own traffic.
The formula
Four steps:
- Find your dedicated hardware's sustained throughput. Use the aggregate tokens/second your model reaches at the concurrency your real traffic runs at, not the single-stream number. Batch-1 and high-concurrency throughput on the same machine can differ by more than an order of magnitude, so use a number measured at a concurrency close to your own.
- Convert to tokens per hour, then to cost per token. Tokens/second × 3,600 gives tokens/hour. Divide the hourly rate by that figure and you have a cost-per-token number for the dedicated machine, assuming it stays busy at that concurrency the entire hour.
- Compare that figure to the API's per-token price. Most API pricing is split between input and output tokens at different rates, so match the ratio of input to output tokens in your own workload rather than comparing a single blended number.
- Multiply both by your monthly volume, then check the utilization assumption. The dedicated-hardware number only holds if the machine is busy at that concurrency for the hours you're counting. If your real traffic arrives in bursts with long gaps, the effective cost per token on dedicated hardware is higher than step 2 suggested, because you're still paying for the idle hours.
Worked example, using GPUwerk's own numbers
Take a workload generating 500 million output tokens a month. gpt-oss-120b reaches 862.8 tok/s aggregate at 256 concurrent requests on a single DGX Spark, per Dendro Logic's concurrency benchmark cited on GPUwerk's benchmarks page. That's 862.8 × 3,600 = 3,106,080 tokens/hour if the node stays saturated at that concurrency continuously. 500,000,000 ÷ 3,106,080 ≈ 161 hours. At GPUwerk's published $0.79/hour on-demand rate, 161 × $0.79 = $127.19 before tax.
That figure assumes the node is busy at 256 concurrent requests for the full 161 hours, back to back, with enough traffic to keep the queue full. Real traffic arrives in bursts. A machine billed by the hour that sits half-idle loses ground fast, because every hour on the clock is billed whether the GPU is generating tokens or not. A per-token API has no idle time to bill.
Put your chosen API's price for 500 million tokens next to $127.19 and you have the comparison for that volume. For your own traffic, the crossover comes down to how busy you can keep the machine.
What tips the answer one way or the other
- Volume and steadiness push toward dedicated hardware. A continuous internal assistant, a batch document pipeline, or an always-on agent fleet keeps utilization high, the condition the worked example assumes.
- Low or spiky volume pushes toward a metered API. If your traffic is a handful of requests an hour with long silent gaps, you're paying for a mostly idle machine on the dedicated side, and the per-token API's "no traffic, no bill" property wins by default.
- Model choice changes the throughput number, not the method. A mixture-of-experts model like gpt-oss-120b reaches much higher aggregate throughput at high concurrency than a dense model of similar size, because a MoE model activates only a fraction of its parameters per token. Rerun step 1 with your own model's measured throughput rather than reusing the number above.
- Data location and who can read your prompts are separate questions from cost. They often decide the choice before cost does; see private LLM hosting for that side of the comparison.
Run it on your own numbers
The quickest real answer comes from measuring your own model's throughput instead of borrowing someone else's benchmark. Rent a DGX Spark by the hour, run your workload at your real concurrency, and put the measured tokens/second into step 1. A few hours at $0.79/hour replaces the Dendro Logic figure above with a number that describes your traffic. Current per-token prices for open models across API providers, with a break-even worked from published throughput, are in open-model API prices.