DGX Spark/Guides/Batch processing large inference jobs on a rented Spark
Throughput

Batch processing large inference jobs on a rented Spark

By Samuel Seidel · Published September 9, 2026

This page shows how to configure and run a large batch job (bulk document processing, dataset labeling, a fixed prompt over a large pile of inputs) on a Spark you rent by the hour. Batch work needs different settings from an interactive chat endpoint. For the conceptual difference, see the batch vs. interactive inference post; this is the hands-on version.

Why batch settings differ from serving settings

An interactive endpoint optimizes for latency on each request: one user is waiting on the response right now. A batch job optimizes for total throughput across the whole job. Nobody is watching any single request, so the engine can queue many requests and process them together, keeping the GPU busy at the cost of any one item's latency. The practical consequence: batch jobs should run with higher concurrency and larger effective batch sizes than you'd ever use for a chat endpoint, and the metric that matters shifts from time-to-first-token to total job completion time and tokens processed per hour.

Running a batch job with vLLM

vLLM's offline batch inference API (LLM.generate() on a list of prompts, rather than the OpenAI-compatible server) is the most efficient path for a large fixed job, since it skips the HTTP layer and lets vLLM's continuous batching schedule the whole list at once:

from vllm import LLM, SamplingParams

llm = LLM(model="/path/to/model", gpu_memory_utilization=0.9)
params = SamplingParams(temperature=0, max_tokens=512)

prompts = [build_prompt(row) for row in dataset]
outputs = llm.generate(prompts, params)

Set gpu_memory_utilization higher than you would for a mixed-use box (see the memory management guide): a dedicated batch run doesn't need headroom for other resident models. If your job is large enough to not fit in memory as a single Python list of prompts, chunk it and call generate() per chunk rather than holding everything at once. vLLM's internal batching handles concurrency within each chunk, so you don't need to throttle manually beyond that.

If you'd rather use the OpenAI-compatible HTTP server (useful if your pipeline already speaks that protocol, or if the job runs from a language other than Python), raise --max-num-seqs above the default and send requests concurrently from the client. The server-side scheduler needs enough in-flight requests to batch effectively; a client sending one request at a time defeats continuous batching whatever the server settings.

Sizing the job against the hourly rate

Renting a Spark at $0.79/hour (or $1.79/hour for the two-node cluster, per pricing) makes the cost model straightforward for a batch job: total cost is hours run × hourly rate, so the only variable worth optimizing is job duration (tokens processed per hour). Two changes typically move that number more than anything else: enabling prompt caching if your job shares a fixed instruction template across every item (a common case in labeling and extraction jobs), and choosing the smallest model that still clears the accuracy bar for the task. If a 27B model finishes the job in a third of the time a 70B model takes, it is often the better economic choice.

Before committing a large job, run a timed sample of 50 to 100 representative items and extrapolate. Duration on real data rarely matches a back-of-envelope estimate from average token counts, because real inputs vary in length far more than synthetic test prompts do.

When the job doesn't fit one node

A batch job that is large in item count, not per-item complexity, generally doesn't need the two-node cluster. Running it longer on one node usually costs less than paying $1.79/hour for the pair: batch jobs are throughput-bound rather than latency-bound and don't benefit from the cluster's networking the way a single very large model does. The cluster is the right call when the model itself is too large for one node's memory, not when the queue is simply long; see the memory management guide for that distinction.

Checkpointing and resumability

A job running for hours on a rented instance should write results incrementally rather than holding everything in memory until the end. A crash partway through then costs only the unfinished items, and you can watch progress and catch a systematic formatting problem early instead of after the full run. Write completed items to disk (or to your target destination) as they finish, and track which input rows have been processed so a restarted job skips completed work instead of reprocessing it.

Practical checklist

See batch vs. interactive inference for the conceptual framing, or private AI for data labeling and annotation for the labeling-specific case.

First top-up: pay $20, get $30 in credit

Run the job by the hour, not by the token.

Rent a dedicated Spark from $0.79/h, billed by the minute, and tune vLLM for throughput on a machine nobody else shares.

Create account Read the vLLM guide