What is model serving?
Model serving turns a trained model into a running service other software can call: an API that accepts requests, runs inference and returns results reliably, under real load, for many users at once. A script that loads a model and runs it once on one input is not serving. Serving is what makes a model usable in production instead of only on someone's laptop.
What a one-off inference script doesn't handle
A script that loads a model and runs it on one input is fine for testing. It has no answer for ten requests arriving at once, a request landing while the previous one is still running, a crashed process, or a model nobody has called in an hour holding resources idle. That is not a flaw in the script; it was never its job. Model serving is the layer on top that handles concurrency, reliability and day-to-day operation.
What a serving stack actually does
An inference server handles the model-level parts: loading weights, managing the KV cache, and running continuous batching so concurrent requests share GPU passes instead of queueing one behind another. Around it, a serving deployment usually adds:
- Routing: sending each request to an available model instance, possibly across several GPUs or nodes.
- Autoscaling: adding or removing capacity as request volume changes.
- Health checks: detecting a stuck or crashed instance and routing around it instead of piling up failed requests.
Each is a separate piece of infrastructure. Skip one and you usually find out from an outage under load, not during the demo.
Serving decisions that shape everything downstream
Synchronous or streamed responses, how batching is scheduled and how many model replicas run at once all trade off against each other, and each choice moves time to first token and steady-state throughput differently. A setup tuned for a few large batch jobs looks very different from one tuned for many small interactive chats, even on identical hardware, because the two workloads stress compute and memory bandwidth in opposite ways.
Where this fits with self-hosted GPU capacity
On a self-hosted node, serving sits between raw GPU capacity and the application calling the LLM, and it takes more of the operational effort than choosing the model or the hardware. If you deploy one of GPUwerk's vLLM images on a Spark, vLLM starts with the container and serves an OpenAI-compatible endpoint once the instance is running; routing across nodes, scaling and health checks remain yours to design. The benchmarks page covers what a single DGX Spark can sustain under concurrent load, and what an inference server does covers the specific software layer that continuous batching and request scheduling run on.