DGX Spark/Guides/Rate limiting and quotas for an internal LLM deployment
Docs analysis

Rate limiting and quotas for an internal LLM deployment

By Samuel Seidel · Published September 9, 2026

A raw DGX Spark has no concept of a user, so it has no per-user limits either. If one person or one script can saturate the GPU and slow everyone else down, you fix that at the application layer. This guide shows how: per-key limits in LiteLLM if you run it as a gateway, or request-rate limits in a reverse proxy if you don't.

Why the platform doesn't do this for you

GPUwerk bills the instance itself by the minute at $0.79/hour on demand, not the requests or tokens flowing through whatever you run inside it. The platform sees whether your container is running, not what happens inside it, so it has no per-user request throttling. Anything resembling a quota, a request limit, or a fair-use cap on a shared deployment has to live in your own serving layer.

The straightforward path: LiteLLM

If your serving stack already runs LiteLLM as a gateway in front of your model, rate limiting comes almost free: it's built into how you issue keys. Two settings do most of the work, set when a virtual key is created:

Issuing a scoped key with limits is a single API call to LiteLLM's key endpoint that names the models the key can reach and its rpm_limit. You can query spend and usage per key, team or model from LiteLLM's /spend/report endpoint or see them in its admin UI, so you know who is actually hitting a limit before you raise or lower it.

On a single shared machine, the constraint that matters usually isn't cost. It's contention: one person's heavy job degrading everyone else's latency. Rate limits per key are the practical lever for that, more than a hard dollar budget.

If you're not running LiteLLM

Without an LLM-aware gateway, you're limited to what a general reverse proxy understands, which is connections and requests, not tokens. Nginx's limit_req module or Caddy's rate-limit plugin can cap requests per second per client IP or API key in front of whatever inference server you're running, which stops a client from hammering the endpoint even without visibility into token counts. This is coarser than LiteLLM's per-key token accounting: one large request costs far more GPU time than a small one, and a request-count limit can't see the difference. It's a reasonable fallback if adding a full gateway isn't worth it for your setup, but if you expect real multi-user load, putting LiteLLM in front is worth the extra piece.

Some inference engines expose their own concurrency or queue-depth settings (a max batch size, a request queue limit) that act as a coarse throttle even without a gateway in front. Check your engine's documentation; this varies, so don't assume it's there.

Practical checklist

See the LiteLLM setup guide for the full key-issuance walkthrough, or a first engagement if you want help sizing limits for a specific team's usage.

First top-up: pay $20, get $30 in credit

One machine, keys that keep it fair.

Rent a dedicated Spark from $0.79/h, billed by the minute, and put LiteLLM in front to give every teammate their own limit.

Create account Read the LiteLLM guide