Setting up a status page for your inference service

The Prometheus alerts from setting up Prometheus alerts tell you when something's wrong. They don't tell your callers anything, since the alert fires into a channel they're not in. A status page is the part of an incident that faces outward: one URL someone can check before opening a support ticket, and one place you point people to instead of answering the same "is it down for you too" message from three different people at once.

Run the checker somewhere that doesn't share a failure domain with the Spark

The entire value of a status page is that it stays up through the outage it's reporting on. That rules out hosting it on the node it's watching, and it's worth being deliberate about the checker too, not just the page: if the machine running the uptime check and the Spark being checked are behind the same router, a local network outage looks identical to the Spark being down. An external monitoring service, or even a small VM at a different provider, avoids this. GPUwerk's own status page runs this way, at status.gpuwerk.com, independent of any single customer node.

Check the real request path, not just a ping

A TCP ping to port 443 confirms the box is reachable. It doesn't confirm vLLM is actually serving completions, since a wedged process, an OOM'd worker, or a model that failed to load can all leave the port open while every real request times out or errors. A useful check calls the endpoint itself:

curl -s -o /dev/null -w "%{http_code}" \
  -X POST https://api.yourdomain.com/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.1-8b","messages":[{"role":"user","content":"ping"}],"max_tokens":1}'

A checker that runs this on a schedule and expects a 200 catches the failures a plain port check misses, at the cost of a token or two of GPU time per check. Running a cheap health endpoint on a short interval and a real completion check on a longer one, a minute versus five or ten minutes, covers both without spending inference time you don't need to.

A minimal external checker with a cron job and a static page

If you'd rather not depend on a third-party status page product, the whole thing is a scheduled script and a static HTML file pushed somewhere that isn't the Spark. A cron job on a small separate host:

# /etc/cron.d/status-check, runs every minute on a host that isn't the Spark
* * * * * root /usr/local/bin/check-inference.sh >> /var/log/status-check.log 2>&1

# /usr/local/bin/check-inference.sh
#!/bin/bash
CODE=$(curl -s -o /dev/null -w "%{http_code}" -m 10 \
  -X POST https://api.yourdomain.com/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.1-8b","messages":[{"role":"user","content":"ping"}],"max_tokens":1}')

if [ "$CODE" = "200" ]; then
  STATUS="operational"
else
  STATUS="down"
fi

cat > /var/www/status/data.json <<EOF
{"status": "$STATUS", "checked_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"}
EOF

A static page fetching data.json and rendering a colored dot is enough for a single endpoint. Once you have more than one node or more than one thing worth showing separately, a dedicated status page product handles history, incident notes, and subscriber notifications with less maintenance than growing this script into one yourself.

Require a few consecutive failures before you page anyone

A single dropped request, a slow cold start on a node using cold-start serving, or one bad network hop shouldn't flip the status page to red and send an alert. Most checkers support a failure threshold, two or three consecutive bad checks before the status actually changes, which filters out the noise without meaningfully slowing down detection of a real outage that lasts more than a couple of minutes.

FAQ