Set up LiteLLM as your AI gateway

A raw vLLM endpoint has no concept of who is calling it. Everyone shares one URL, nobody has a key of their own, and you cannot answer "which team burned the GPU on Tuesday". LiteLLM sits in front and gives you per-user keys, budgets, and one address that can route to your Spark or to a cloud model without the caller knowing which.

What it actually buys you

LiteLLM is a proxy that speaks the OpenAI API on the front and roughly a hundred provider APIs on the back. On a private setup the interesting parts are narrower than the marketing suggests, and there are four of them.

• One stable URL. Clients point at the gateway forever. Swap the model behind an alias, move it to a different Spark, add a second box, and nothing downstream changes.
• Keys per person or per service. Revoke one without rotating everyone's.
• Spend and usage attribution. Even when the marginal token is free on your own hardware, knowing which service is generating the load is how you plan capacity.
• Fallbacks. When the Spark is saturated or rebooting, requests can spill to a second instance or to a cloud provider rather than erroring.

If you have one model, one user and no compliance requirements, skip this page. The gateway earns its keep at the point where a second team starts calling your endpoint.

Pin the version first

Install with Docker Compose

Virtual keys, budgets and spend tracking all need Postgres, so the database is not optional once you want the features that justify running this. Use the litellm-database image, which includes the migration step.

# /workspace/litellm/docker-compose.yml
services:
  litellm:
    # pin a real version, not :latest
    image: docker.litellm.ai/berriai/litellm-database:v1.83.0
    restart: unless-stopped
    ports:
      - "127.0.0.1:4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    environment:
      DATABASE_URL: postgresql://litellm:${PG_PASSWORD}@db:5432/litellm
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
    command: --config /app/config.yaml
    depends_on:
      - db

  db:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${PG_PASSWORD}
      POSTGRES_DB: litellm
    volumes:
      - ./pgdata:/var/lib/postgresql/data

Generate the secrets into a .env beside it, and never commit that file:

cat > .env <<EOF
PG_PASSWORD=$(openssl rand -hex 16)
LITELLM_MASTER_KEY=sk-$(openssl rand -hex 24)
LITELLM_SALT_KEY=$(openssl rand -hex 24)
EOF
chmod 600 .env

The salt key encrypts stored provider credentials in the database. Changing it later makes every saved credential unreadable, so set it once and back it up with the same care as the credentials themselves.

Note the port binding: 127.0.0.1:4000:4000, not 4000:4000. The gateway holds every key you own, and there is no reason for it to listen on a public interface. Reach it over an SSH tunnel, or put a reverse proxy with TLS in front of it.

Write config.yaml

This is where your own models get names. The openai/ prefix is what tells LiteLLM to treat the backend as OpenAI-compatible, which is exactly what vLLM is:

# /workspace/litellm/config.yaml
model_list:
  # the model on your Spark
  - model_name: spark-qwen
    litellm_params:
      model: openai/qwen3-coder-30b
      api_base: http://localhost:8000/v1
      api_key: "none"

  # a second Spark, same alias: LiteLLM load-balances between them
  - model_name: spark-qwen
    litellm_params:
      model: openai/qwen3-coder-30b
      api_base: http://spark-2.internal:8000/v1
      api_key: "none"

  # an escape hatch for when the fleet is busy
  - model_name: cloud-backup
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  drop_params: true
  num_retries: 2

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Two aliases pointing at the same model_name is not a mistake. LiteLLM treats entries sharing a name as a pool and distributes requests across them, which is how a second Spark gets added with no client changes at all.

drop_params: true silently discards parameters the backend does not understand rather than returning a 400. That is what you want with mixed backends, since a client sending an OpenAI-specific field should not break against a local model. It also means a typo in a parameter name fails quietly, so turn it off while debugging.

Bring it up and check it loaded what you expect:

docker compose up -d
docker compose logs -f litellm

curl http://localhost:4000/v1/models \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

Issue virtual keys

The master key is an admin credential. It should live in your password manager and never in an application config. What applications get is a virtual key, minted against the master key:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["spark-qwen"],
    "metadata": {"team": "support-bot"},
    "max_budget": 50,
    "budget_duration": "30d",
    "rpm_limit": 60
  }'

The response contains a sk- key scoped to exactly what you listed. This one can call spark-qwen and nothing else, so a compromised support bot cannot start spending against your cloud provider. Revoke it with /key/delete and no other caller notices.

Give every service its own key. The cost is one API call, and the benefit shows up the first time you need to answer what a key was used for, or turn one off in a hurry.

Budgets and rate limits

On your own hardware the meaningful limit is not dollars, it is contention: one enthusiastic batch job can make the shared chat interface unusable for everyone else. Rate limits are the useful control, and they apply per key.

rpm_limit and tpm_limit cap requests and tokens per minute. Set rpm_limit low on anything automated and leave interactive users generous, which keeps a runaway agent loop from starving the humans. max_budget with budget_duration matters mostly for keys that can reach a paid provider, where it is a genuine spend cap rather than a fairness knob.

Spend per key, per team and per model is queryable from the /spend/report endpoint, and the admin UI at http://localhost:4000/ui renders the same data if you would rather click than curl.

Fallbacks and hybrid routing

The honest reason most private deployments end up with a cloud key configured: a single Spark is one box, and one box reboots. Fallbacks make that a latency event rather than an outage.

litellm_settings:
  fallbacks:
    - spark-qwen: ["cloud-backup"]
  # separate list for "prompt was too long for the local model"
  context_window_fallbacks:
    - spark-qwen: ["cloud-backup"]
  num_retries: 2

Be deliberate about this one. A fallback to a cloud provider means that under exactly the conditions you did not plan for, prompts you intended to keep on your own hardware leave the building. If the whole point of the Spark is that data does not go to a third party, set the fallback to a second local instance and let requests fail when both are down. That is a policy decision, not a configuration detail, and it is worth writing down next to the config.

Point your clients at it

Everything that speaks OpenAI works unchanged, with the base URL swapped and a virtual key in place of an OpenAI one:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-your-virtual-key",
)

resp = client.chat.completions.create(
    model="spark-qwen",
    messages=[{"role": "user", "content": "Summarise this ticket."}],
)

For Open WebUI, set OPENAI_API_BASE_URLS to http://localhost:4000/v1 and OPENAI_API_KEYS to a virtual key. It will list whatever that key is allowed to see, so the model picker becomes a per-key view of your fleet without any change on the Open WebUI side.