Set up LiteLLM as your AI gateway

By the end of this page you have a LiteLLM gateway in front of your Spark: one OpenAI-compatible URL, a virtual key per user or service, rate limits and budgets per key, and optional fallback to a second Spark or a cloud model. A raw vLLM endpoint cannot tell callers apart, so you cannot answer "which team used the GPU on Tuesday". GPUwerk's console presets already require an API key with the inference scope, but that is one key for the account. LiteLLM adds separate keys and budgets for your own clients.

What it actually buys you

LiteLLM is a proxy that speaks the OpenAI API to clients and about a hundred provider APIs to backends. On a private setup, four features matter:

• One stable URL. Clients point at the gateway forever. Swap the model behind an alias, move it to a different Spark, add a second box, and nothing downstream changes.
• Keys per person or per service. Revoke one without rotating everyone's.
• Spend and usage attribution. Even when the marginal token is free on your own hardware, knowing which service is generating the load is how you plan capacity.
• Fallbacks. When the Spark is saturated or rebooting, requests can spill to a second instance or to a cloud provider rather than erroring.

If you have one model, one user and no compliance requirements, skip this page. The gateway earns its keep at the point where a second team starts calling your endpoint.

Pin the version first

Install with Docker Compose

Virtual keys, budgets and spend tracking all need Postgres, so the database is part of the setup. Use the litellm-database image, which runs the migrations for you. Create this file on the gateway host:

# /workspace/litellm/docker-compose.yml
services:
  litellm:
    # pin a real version, not :latest
    image: docker.litellm.ai/berriai/litellm-database:v1.83.0
    restart: unless-stopped
    ports:
      - "127.0.0.1:4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    environment:
      DATABASE_URL: postgresql://litellm:${PG_PASSWORD}@db:5432/litellm
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
    command: --config /app/config.yaml
    depends_on:
      - db

  db:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${PG_PASSWORD}
      POSTGRES_DB: litellm
    volumes:
      - ./pgdata:/var/lib/postgresql/data

Generate the secrets into a .env beside it, and never commit that file:

cat > .env <<EOF
PG_PASSWORD=$(openssl rand -hex 16)
LITELLM_MASTER_KEY=sk-$(openssl rand -hex 24)
LITELLM_SALT_KEY=$(openssl rand -hex 24)
EOF
chmod 600 .env

The salt key encrypts stored provider credentials in the database. Changing it later makes every saved credential unreadable, so set it once and back it up with the same care as the credentials themselves.

Note the port binding: 127.0.0.1:4000:4000, not 4000:4000. The gateway holds every key you own, so it should not listen on a public interface. Reach it over an SSH tunnel, or put a reverse proxy with TLS in front of it.

Write config.yaml

This is where your own models get names. The openai/ prefix tells LiteLLM the backend speaks the OpenAI API, which vLLM and llama.cpp both do. Set api_base to wherever the engine answers from the gateway host:

• Console preset: the preset's HTTPS endpoint from the console plus /v1 (for example https://app-<instance-id>.gpuwerk.com/v1), with model: openai/qwen3-coder-30b and api_key: os.environ/GPUWERK_API_KEY holding your inference-scope key.
• Your own engine: http://localhost:8000/v1 when LiteLLM runs on the same machine, as in the sample below; otherwise the instance's hostname, http://<name>.gpuwerk.com:8000/v1, with the engine started with --api-key.

# /workspace/litellm/config.yaml
model_list:
  # the model on your Spark
  - model_name: spark-qwen
    litellm_params:
      model: openai/qwen3-coder-30b
      api_base: http://localhost:8000/v1
      api_key: "none"

  # a second Spark, same alias: LiteLLM load-balances between them
  - model_name: spark-qwen
    litellm_params:
      model: openai/qwen3-coder-30b
      api_base: http://spark-2.internal:8000/v1
      api_key: "none"

  # an escape hatch for when the fleet is busy
  - model_name: cloud-backup
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  drop_params: true
  num_retries: 2

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Two aliases pointing at the same model_name is not a mistake. LiteLLM treats entries sharing a name as a pool and distributes requests across them, which is how a second Spark gets added with no client changes at all.

drop_params: true silently discards parameters the backend does not understand rather than returning a 400. That is what you want with mixed backends, since a client sending an OpenAI-specific field should not break against a local model. It also means a typo in a parameter name fails quietly, so turn it off while debugging.

Start the stack and check it loaded what you expect. The logs should end with the proxy listening on port 4000, and the /v1/models response should list spark-qwen and cloud-backup:

docker compose up -d
docker compose logs -f litellm

curl http://localhost:4000/v1/models \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

Issue virtual keys

The master key is an admin credential. Keep it in your password manager, never in an application config. Applications get a virtual key, created with the master key:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["spark-qwen"],
    "metadata": {"team": "support-bot"},
    "max_budget": 50,
    "budget_duration": "30d",
    "rpm_limit": 60
  }'

The response contains an sk- key limited to what you listed. This one can call spark-qwen and nothing else, so a compromised support bot cannot spend against your cloud provider. Revoke it with /key/delete and no other caller notices.

Give every service its own key. It costs one API call, and it pays off the first time you need to know what a key was used for, or turn one off in a hurry.

Budgets and rate limits

On a Spark you rent at a fixed hourly rate, the limit that matters is contention, not per-token cost. One busy batch job can make the shared chat interface unusable for everyone else. Rate limits are the control for that, and they apply per key.

rpm_limit and tpm_limit cap requests and tokens per minute. Set rpm_limit low on anything automated and leave interactive users generous, which keeps a runaway agent loop from starving the humans. max_budget with budget_duration matters mostly for keys that can reach a paid provider, where it is a genuine spend cap rather than a fairness knob.

Query spend per key, team and model from the /spend/report endpoint. The admin UI at http://localhost:4000/ui shows the same data.

Fallbacks and hybrid routing

A single Spark is one machine, and one machine reboots. Fallbacks turn that into a slower response instead of an outage.

litellm_settings:
  fallbacks:
    - spark-qwen: ["cloud-backup"]
  # separate list for "prompt was too long for the local model"
  context_window_fallbacks:
    - spark-qwen: ["cloud-backup"]
  num_retries: 2

Decide this one on purpose. With a cloud fallback, prompts you meant to keep on your Spark go to a third party exactly when something unplanned happens. If keeping data away from third parties is the reason you run the Spark, set the fallback to a second Spark and let requests fail when both are down. That is a policy decision, so write it down next to the config.

Point your clients at it

Any OpenAI client works with two changes: the gateway's base URL and a virtual key in place of an OpenAI key. This call should return a completion from spark-qwen:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-your-virtual-key",
)

resp = client.chat.completions.create(
    model="spark-qwen",
    messages=[{"role": "user", "content": "Summarise this ticket."}],
)

For Open WebUI, set OPENAI_API_BASE_URLS to http://localhost:4000/v1 and OPENAI_API_KEYS to a virtual key. The model picker then shows exactly the models that key may use, with no other change in Open WebUI.