Rate limiting and quotas for an internal LLM deployment
A raw DGX Spark has no concept of a user, so it has no per-user limits either. If one person or one script can saturate the GPU and slow everyone else down, you fix that at the application layer. This guide shows how: per-key limits in LiteLLM if you run it as a gateway, or request-rate limits in a reverse proxy if you don't.
Why the platform doesn't do this for you
GPUwerk bills the instance itself by the minute at $0.79/hour on demand, not the requests or tokens flowing through whatever you run inside it. The platform sees whether your container is running, not what happens inside it, so it has no per-user request throttling. Anything resembling a quota, a request limit, or a fair-use cap on a shared deployment has to live in your own serving layer.
The straightforward path: LiteLLM
If your serving stack already runs LiteLLM as a gateway in front of your model, rate limiting comes almost free: it's built into how you issue keys. Two settings do most of the work, set when a virtual key is created:
rpm_limitandtpm_limitcap requests and tokens per minute for that specific key. A lowrpm_limiton an automated or batch key keeps a runaway loop from starving interactive users, while a human's key can stay generous.max_budgetwithbudget_durationis a spend cap over a time window. It matters mainly for keys that can reach a paid external provider; if everything routes to your own Spark, there's no per-token bill to cap.
Issuing a scoped key with limits is a single API call to LiteLLM's key endpoint that names the models the key can reach and its rpm_limit. You can query spend and usage per key, team or model from LiteLLM's /spend/report endpoint or see them in its admin UI, so you know who is actually hitting a limit before you raise or lower it.
On a single shared machine, the constraint that matters usually isn't cost. It's contention: one person's heavy job degrading everyone else's latency. Rate limits per key are the practical lever for that, more than a hard dollar budget.
If you're not running LiteLLM
Without an LLM-aware gateway, you're limited to what a general reverse proxy understands, which is connections and requests, not tokens. Nginx's limit_req module or Caddy's rate-limit plugin can cap requests per second per client IP or API key in front of whatever inference server you're running, which stops a client from hammering the endpoint even without visibility into token counts. This is coarser than LiteLLM's per-key token accounting: one large request costs far more GPU time than a small one, and a request-count limit can't see the difference. It's a reasonable fallback if adding a full gateway isn't worth it for your setup, but if you expect real multi-user load, putting LiteLLM in front is worth the extra piece.
Some inference engines expose their own concurrency or queue-depth settings (a max batch size, a request queue limit) that act as a coarse throttle even without a gateway in front. Check your engine's documentation; this varies, so don't assume it's there.
Practical checklist
- Don't expect the platform to rate-limit anything inside your container; it only meters instance uptime.
- If you're running LiteLLM, set
rpm_limitandtpm_limitper key at issuance: generous for humans, tight for automated callers. - Use LiteLLM's
/spend/reportor admin UI to see actual usage before tightening limits further. - Without LiteLLM, fall back to a reverse proxy's request-rate limiting, knowing it can't see token counts the way a gateway can.
- Treat contention, not cost, as the real reason to rate-limit on a single dedicated machine.
See the LiteLLM setup guide for the full key-issuance walkthrough, or a first engagement if you want help sizing limits for a specific team's usage.