TensorRT-LLM on a DGX Spark: NVIDIA's container, trtllm-serve with NVFP4, and speed against vLLM and SGLang

TensorRT-LLM runs on a DGX Spark in NVIDIA's release container: the DGX Spark playbook (last updated 12 August 2026) pins nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 for one Spark and 1.3.0rc5 for two, while the newest release on 7 November 2026 is 1.4.0rc3 and the newest stable one is 1.2.1. trtllm-serve now loads Hugging Face NVFP4 checkpoints directly with no engine build, and NVIDIA reports 38.65 tok/s decode and 10,257 tok/s prefill for Llama 3.1 8B at NVFP4 on a Spark. An independent single-request test put it level with vLLM and SGLang (30.94 against 31.80 and 31.66 tok/s, July 2026), so it pays off only for models on NVIDIA's validated list.

Download sizes are the compressed arm64 layer totals I added up from the nvcr.io manifests on 7 November 2026. The build.nvidia.com/spark/trt-llm address now redirects to the TensorRT-LLM GitHub repository and the playbook is no longer in the build.nvidia.com/spark index; it lives on in NVIDIA's dgx-spark-playbooks repository.

Which TensorRT-LLM container does NVIDIA publish for the Spark?

The general release image, not a Spark-specific one: every recent tag of nvcr.io/nvidia/tensorrt-llm/release is a multi-arch manifest with an arm64 build. The playbook's choice is conservative. 1.3.0rc13 is six months and 20 release candidates behind 1.4.0rc3, and NVIDIA uses an even older 1.3.0rc5 for its two-node recipe. The registry also carries a single-architecture tag called spark-single-gpu-dev; NVIDIA does not document it, so I would not build on it.

Release dates from GitHub; architectures from the nvcr.io manifests and the NGC catalog entry, read 7 November 2026. The container docs point at 1.4.0rc3.

The release notes since spring read like a GB10 bring-up log: NCCL symmetric memory and window buffers disabled on GB10 (1.3.0rc19 and rc20), CUTLASS MoE GEMM tiles filtered by device shared memory on SM121 (rc23), SM121 MLA cache reuse (rc22), FlashInfer GDN prefill on SM120 and SM121 (rc29), and in 1.4.0rc3 new "DGX Spark agentic and real-text speculative decoding perf cases" in NVIDIA's own tests. That is a reason to try a newer tag than the playbook's if your model is new.

How do you serve an NVFP4 checkpoint with trtllm-serve?

Pull the image, check that tensorrt_llm imports, then start trtllm-serve with a small YAML file. These are the playbook's default-path commands for Llama 3.1 8B at NVFP4, on a Spark you own with Docker:

export TRTLLM_IMAGE="nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13"
export HF_TOKEN="<YOUR_HUGGINGFACE_TOKEN>"
export MODEL_HANDLE="nvidia/Llama-3.1-8B-Instruct-FP4"
mkdir -p "$HOME/.cache/huggingface"

docker run --rm --gpus all "$TRTLLM_IMAGE" \
  python -c "import tensorrt_llm; print(tensorrt_llm.__version__)"

docker run --name trtllm-server --rm -it \
  --gpus all --ipc host --network host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -e MODEL_HANDLE="$MODEL_HANDLE" -e HF_TOKEN="$HF_TOKEN" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$TRTLLM_IMAGE" \
  bash -c '
    hf download "$MODEL_HANDLE" &&
    cat > /tmp/extra-llm-api-config.yml <<EOF
print_iter_log: false
kv_cache_config:
  dtype: "auto"
  free_gpu_memory_fraction: 0.9
cuda_graph_config:
  enable_padding: true
disable_overlap_scheduler: true
EOF
    trtllm-serve "$MODEL_HANDLE" \
      --host 0.0.0.0 --port 8355 \
      --max_batch_size 64 --trust_remote_code \
      --extra_llm_api_options /tmp/extra-llm-api-config.yml
  '

# second terminal
curl -s http://localhost:8355/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "nvidia/Llama-3.1-8B-Instruct-FP4",
       "messages": [{"role": "user", "content": "Explain tensor parallelism in two sentences."}],
       "max_tokens": 64}'

Four details the playbook does not spell out. First, the Hugging Face repo has been renamed: nvidia/Llama-3.1-8B-Instruct-FP4 now redirects to nvidia/Llama-3.1-8B-Instruct-NVFP4. Second, --network host plus --host 0.0.0.0 publishes port 8355 on every interface of the Spark; use --host 127.0.0.1 if the API should stay on the box. Third, free_gpu_memory_fraction is not vLLM's setting under another name: the trtllm-serve reference defines it as the share of free memory reserved for KV cache after weights and buffers are allocated, so 0.9 on an otherwise empty Spark gives most of what is left to the cache. Fourth, newer releases accept --config as the preferred name for --extra_llm_api_options.

For gpt-oss-20b or gpt-oss-120b the playbook adds a step inside the container, before hf download: set TIKTOKEN_ENCODINGS_BASE=/tmp/harmony-reqs and wget the o200k_base and cl100k_base tiktoken files into it. If the box runs out of memory while loading weights, its troubleshooting table says to set TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1. Multi-node serving uses mpirun, trtllm-llmapi-launch and --tp_size 2 across two Sparks; the cabling is covered in the two-Spark cluster guide.

Which models does it support on a Spark, and what breaks?

NVIDIA validates 17 checkpoints, mostly its own FP8 and NVFP4 exports. The list: Nemotron-3-Nano-Omni-30B-A3B in BF16, FP8 and NVFP4; Nemotron-3-Super-120B-A12B NVFP4; gpt-oss-20b and 120b in MXFP4; Llama 3.1 8B in FP8 and NVFP4; Llama 3.3 70B, Qwen3-32B, Qwen3-30B-A3B and Llama-4-Scout in NVFP4; Qwen3 8B and 14B, Phi-4-multimodal and Phi-4-reasoning-plus in FP8 or NVFP4; and Qwen3-235B-A22B NVFP4 across two Sparks. The playbook's "agent-ready" Qwen3.6-35B-A3B NVFP4 recipe is still marked pending validation. Outside the list, expect work:

Known issues are quoted from the TensorRT-LLM release notes; forum rows are user reports. Status as read on 7 November 2026.

How fast is TensorRT-LLM on a Spark next to vLLM and SGLang?

For one request, no faster at equal precision; NVIDIA's headline numbers come mostly from NVFP4 reading fewer bytes. GPUwerk has not measured TensorRT-LLM; its own vLLM 0.27.1 runs are on the benchmarks page.

Reported figures. NVIDIA's table used 2,048 input and 128 output tokens at batch size 1. Kubesimplify used 75 input and 256 output tokens, one request at a time, driver 580.159.03. The forum user measured with aiperf and did not state concurrency in text, so I read "tps" as throughput under load rather than single-stream.

Is the NVFP4 lead an engine effect? By my calculation, mostly not. Decode reads every weight once per token, and the NVFP4 Llama checkpoint is 6.03 GB on disk per its file list, with lm_head left unquantized per its hf_quant_config.json:

embedding table   = 128,256 vocab x 4,096 hidden x 2 bytes  = 1.05 GB (one row read per token)
bytes per token   = 6.03 GB - 1.05 GB                        = 4.98 GB
NVFP4 ceiling     = 273 GB/s / 4.98 GB                       = 54.8 tok/s
TensorRT-LLM      = 38.65 tok/s                              = 71% of ceiling

FP8 weights       = about 6.98 B params x 1 byte + 1.05 GB BF16 lm_head = about 8.0 GB
FP8 ceiling       = 273 / 8.0                                = about 34 tok/s
SGLang FP8        = 20.5 tok/s                               = about 60% of ceiling

Vocabulary and hidden size are from the checkpoint's config.json (untied embeddings). The two engines land at 60 and 71 percent of their own ceilings, the 50 to 80 percent band the benchmarks page shows for vLLM. The format nearly doubled the speed; the engine added about ten points of efficiency, measured a year ago, in different setups. Prefill is compute-bound and favors NVFP4 too. Which format is fastest per engine is the subject of the planned quantization speed guide; accuracy is in NVFP4 vs MXFP4 vs FP8.

How long does it take to build and start?

There is no build any more, and startup is middling. The current trtllm-serve reference lists --backend as deprecated, with pytorch the only option, so the old ahead-of-time engine compile is gone from the serving path. Measured times to ready:

• Qwen2.5-3B, one Spark: about 80 seconds on TensorRT-LLM 1.2.1, against 156 seconds for vLLM 0.25.1 and about 82 for SGLang (Kubesimplify).
• gpt-oss-120b, two Sparks: 5 to 6 minutes on TensorRT-LLM, against about 12 for vLLM and 3 to 4 for SGLang, by mark440 in the two-Spark forum thread (December 2025).
• First run of the playbook: NVIDIA estimates 60 minutes including the container and model downloads, and rates the risk medium.

The download is the real fixed cost: 19.5 GB for the playbook's image before any weights.

Can you run TensorRT-LLM on a rented Spark without Docker?

Yes, through pip, though it is the least documented route in this guide. A GPUwerk instance is a Docker container with no Docker socket (see the tenant environment), so none of the docker run commands above work there. NVIDIA's wheel index does carry cp312 aarch64 wheels for 1.2.1, 1.3.0rc13 and 1.4.0rc3, so Python 3.12 is required. The pip install page (tested on Ubuntu 24.04) lists these prerequisites and commands for the stable wheel:

# prerequisites from the docs: CUDA Toolkit 13.1 installed and CUDA_HOME set
pip3 install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu130
apt-get -y install libopenmpi-dev        # root inside the instance, no sudo needed

pip3 install --ignore-installed pip setuptools wheel && pip3 install tensorrt_llm

# keep pip from swapping your torch for another build
echo "torch==$(python3 -c 'import torch; print(torch.__version__)')" > /tmp/torch-constraint.txt
pip3 install tensorrt_llm -c /tmp/torch-constraint.txt

The docs say the PyPI wheel is built against PyTorch 2.10.0, and that cuda-compat-13-1 "may be required depending on your system's NVIDIA GPU driver version". Inside an instance you are root, so apt and pip work; install the CUDA 13.1 toolkit first and keep the venv under /workspace so a rebuild does not wipe it. I found no public report of the wheel on GB10 outside NVIDIA's container, so budget time for the toolkit and OpenMPI setup. To match the playbook, pin tensorrt_llm==1.3.0rc13 and add --extra-index-url https://pypi.nvidia.com (my adaptation; the docs show only the stable install). The sibling guide to SGLang on a Spark has a pip route with fewer prerequisites.

When does the extra setup pay off?

When your model is one of NVIDIA's validated checkpoints and you will stay on NVIDIA's stack anyway. Then TensorRT-LLM gives you NVIDIA-tuned NVFP4 kernels, a container NVIDIA tests on GB10, and the engine NVIDIA itself used for four of the six rows in its Spark launch table. It also starts faster than vLLM. For a packaged, licensed alternative, the NIM guide covers NVIDIA's microservices.

For anything else on one Spark I would not bother. The published single-request numbers are level with vLLM and SGLang at equal precision, community model support arrives in vLLM and SGLang first (Qwen3-Next and GLM threads above), and you inherit release-candidate pinning: the playbook's own tags are six and eight months old. My order for a new model is vLLM or SGLang first, TensorRT-LLM if the model is on NVIDIA's list and you need the last few points of efficiency; the engine comparison has the broader trade-offs.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• Serve LLMs with NVIDIA TensorRT-LLM (playbook-trt-llm README), NVIDIA dgx-spark-playbooks on GitHub, 2026-08-12
• Releases · NVIDIA/TensorRT-LLM, GitHub, 2026-11-06
• trtllm-serve, TensorRT-LLM docs
• Installing on Linux via pip, TensorRT-LLM docs
• Pre-built release container images on NGC, TensorRT-LLM docs
• TensorRT-LLM release container, NVIDIA NGC Catalog
• tensorrt-llm wheel index, NVIDIA PyPI
• How NVIDIA DGX Spark's Performance Enables Intensive AI Tasks, NVIDIA Technical Blog, 2025-10-24
• Local LLM Inference Engines on DGX Spark, Explained, Kubesimplify, 2026-07-17
• Trtllm vs vllm performance w/ gpt-oss-120b, NVIDIA Developer Forums, 2026-01-19
• Qwen3-Next-80B NVFP4 and FP8 cannot be served via trtllm-serve on DGX Spark GB10 (TRT-LLM 1.3.0rc7), NVIDIA Developer Forums, 2026-03-15
• GLM-4.7-NVFP4 (NOT Flash) served with TRT-LLM on 2x DGX Spark, NVIDIA Developer Forums, 2026-01-25
• Setting up vLLM, SGLang or TensorRT on two DGX Sparks, NVIDIA Developer Forums, 2025-12-02
• NVIDIA DGX Spark In-Depth Review, LMSYS, 2025-10-13
• nvidia/Llama-3.1-8B-Instruct-NVFP4 files and hf_quant_config.json, Hugging Face