Blog/What is an inference server?
Definition

What is an inference server?

By Samuel Seidel · Published September 9, 2026

An inference server is the software that loads a trained model into GPU or CPU memory and exposes it through an API, so applications send requests and get generated output back without touching the model's internals. It does the work between a raw model file and a usable endpoint: loading weights, allocating memory, batching concurrent requests, and returning results in a consistent format.

What it actually does

Running a model isn't just loading weights and calling a forward pass once per request. An inference server manages a queue of incoming requests, groups compatible ones into batches to use GPU compute efficiently, allocates and frees memory for each request's growing output, and usually streams tokens back as they're generated instead of waiting for the full response. Most also wrap the model in a standard API, commonly an OpenAI-compatible API, so client applications can call it the same way they'd call a hosted provider.

Where it sits in the stack

The model (its weights and architecture) is separate from the inference server that runs it. The same open-weight model can be served by different inference engines, and each performs differently on the same hardware. Above the inference server, an application or an AI gateway often handles routing, authentication, and request logging, treating the inference server as the backend that actually does the computation.

Why the choice of engine matters

Different inference engines make different tradeoffs in memory management, batching strategy, and supported model formats, and those tradeoffs translate directly into latency and throughput differences on the same hardware and model. A server tuned for high concurrent throughput can handle a single interactive request worse than one built for interactive use. Published numbers come from someone else's configuration; the reliable way to know what to expect is to benchmark your model and workload on the engine you plan to run. See benchmarking your own workload for how to run that comparison.

Running your own inference server

Self-hosting a model means running the inference server yourself, on your own or dedicated hardware, instead of sending requests to someone else's endpoint. You choose the engine, its configuration and what data reaches it; in exchange, you manage the deployment.

A GPUwerk Spark is a dedicated DGX Spark with 128 GB of unified memory, on which you run whichever engine you choose; the vLLM guide covers the GB10 install path. It costs $0.79/hour for one Spark or $1.79/hour for a linked pair, billed per minute, and a stopped instance is held for you at 75% of the hourly rate. See the hardware page for full specifications.

Related pages

First top-up: pay $20, get $30 in credit

Run your own inference server on a dedicated node.

One dedicated Spark, $0.79/hour, deployed in minutes in Europe, North America or East Asia.

Create account