What is an AI gateway?
An AI gateway is a proxy layer that sits between your applications and one or more language models, exposing a single API regardless of how many different models or providers are behind it. Instead of every application wiring itself directly to a specific model's SDK and API key, they all call the gateway, and the gateway handles routing, authentication, and usage tracking underneath.
The problem it solves
Once a team is using more than one model, whether that's a self-hosted model plus a cloud API as a fallback, or several cloud providers for different tasks, every application that calls those models directly ends up hardcoding provider-specific request formats, holding its own copy of API keys, and losing any central view of who's calling what. Add a second team, and the same wiring gets duplicated. An AI gateway removes the duplication by putting one layer in front of all of it, so applications talk to one endpoint and the gateway resolves where the request actually goes.
Unified API across providers and models
Different model providers expose different request and response shapes. The most common convention a gateway standardizes on is the OpenAI chat completions format, since enough tooling already speaks it that adopting it as the common interface avoids rewriting client code per provider. An application sends one request format to the gateway, and the gateway translates it to whatever the underlying model actually expects, whether that's a self-hosted vLLM endpoint or a cloud vendor's own API. Swapping which model answers a given request becomes a configuration change on the gateway, not a code change in every application that calls it.
Routing
Routing is the gateway deciding which model actually handles a given request. That can be as simple as always sending a named model to the same backend, or more involved: sending overflow traffic to a second model when the first is saturated, falling back to a cloud API when self-hosted capacity runs out, or splitting traffic between models by cost or by task. The routing logic lives in one place, and it can change without touching any application code.
Key management
Rather than distributing the real credentials for each underlying model to every application and developer that needs access, a gateway issues its own virtual keys. Each key can be scoped to specific models, given a budget, and revoked independently, without rotating the actual provider credentials behind it. That matters most once more than a handful of people or services need access, since a leaked or over-broad key becomes a much smaller problem when it only unlocks a gateway-issued scope rather than the underlying account.
Usage tracking
Because every request passes through the gateway, it's the natural place to log token counts, cost, latency, and which key or application made each call. Without a gateway, that visibility has to be built separately into every application, or reconstructed after the fact from provider billing dashboards that don't distinguish between your internal teams or projects.
Where GPUwerk fits
We run LiteLLM as an AI gateway in front of a Spark, giving you one OpenAI-compatible endpoint, per-user virtual keys with budgets, and an optional fallback to a cloud model if your own hardware is busy. The setup doc walks through configuring routing and keys end to end.