SGLang on a DGX Spark: install on aarch64 and sm_121, the flags that matter, and how fast it is next to vLLM

SGLang runs on a DGX Spark from its stock release: v0.5.23, published 2 November 2026, ships CUDA 13 Docker images for arm64 (lmsysorg/sglang:v0.5.23, 16.1 GB on Docker Hub) and aarch64 wheels on PyPI that pin PyTorch 2.14.1. On GB10, two flags decide whether it stays up: --mem-fraction-static 0.80 rather than 0.85, because DGX OS's earlyoom kills the scheduler at about 8 GB free, and --attention-backend flashinfer. Reported single-stream speed is level with vLLM on the same BF16 checkpoint (31.66 against 31.80 tok/s, July 2026) and was ahead on gpt-oss-120b in one December 2025 test (52 against 36 tok/s).

Tags and sizes read from Docker Hub and the nvcr.io registry on 7 November 2026. The engine comparison at a high level is in llama.cpp vs vLLM vs SGLang on a Spark; this page is the SGLang setup itself.

Which SGLang image runs on a DGX Spark?

Any current release tag: SGLang's images have been CUDA 13 by default since v0.5.11, and the release tags are multi-arch. The install docs say CUDA 12 images are no longer published and that v0.5.19-cu129 was the last CUDA 12 tag. What still trips people up is the old Spark-only tag from launch week.

Docker Hub sizes as listed on its tag pages; the NGC size is the compressed arm64 layer total I added up from the nvcr.io manifest. NVIDIA's 26.10 notes list flashinfer 0.7.0.post1, flash-attention 2.8.3.post1 and torch 2.15.0a0 in that image, and one known issue: no MTP for Nemotron-3-Super models.

The spark tag is the one LMSYS used for its October 2025 review. SGLang's tracking issue 11658 explains that it came from a personal development branch with temporary workarounds, among them FP8 CUTLASS kernels disabled on sm_121a and a PyTorch nightly. It has not been rebuilt since.

NVIDIA's SGLang playbook (last updated 31 July 2026) runs the container like this. I changed three things: a pinned tag instead of latest-cu130, 0.80 instead of 0.85 for the memory fraction (reason below), and the port bound to localhost.

export HF_TOKEN=""                # only needed for gated models
export MODEL_HANDLE="Qwen/Qwen3-8B"
docker pull lmsysorg/sglang:v0.5.23

docker run -d --name sglang-server \
  --gpus all --ipc host --cap-add SYS_NICE \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -p 127.0.0.1:30000:30000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v "$HOME/.cache/huggingface/hub:/root/.cache/huggingface/hub" \
  lmsysorg/sglang:v0.5.23 \
  sglang serve --model-path "$MODEL_HANDLE" \
    --host 0.0.0.0 --port 30000 \
    --context-length 32768 \
    --mem-fraction-static 0.80 \
    --attention-backend flashinfer \
    --enable-cache-report

timeout 900 bash -c 'until curl -sf http://localhost:30000/health > /dev/null; do sleep 10; done'

The playbook budgets 10 to 15 minutes for the first start of Qwen3-8B, most of it download and CUDA graph capture. For NVIDIA's NVFP4 checkpoints (nvidia/Qwen3-32B-FP4, nvidia/Llama-3.3-70B-Instruct-FP4 and others in its validated list) add --quantization modelopt_fp4. Docker and image pinning in general are covered in the sibling guide to Docker and NGC containers on a Spark.

Can you pip install SGLang on a Spark or a rented instance?

Yes: since the aarch64 wheels landed, pip is a first-class route, and on a rented Spark it is the only one. PyPI carries sglang 0.5.23 as manylinux_2_34 wheels for x86_64 and aarch64, pinned to torch 2.14.1, and PyPI's torch 2.14.1 aarch64 wheel pulls the CUDA 13.0.3 runtime libraries as pip packages, the same CUDA minor version DGX OS 7.6.0 ships. The compiled kernels now come from the sglang-kernel package (0.4.10 on PyPI, also with aarch64 wheels); the old sgl-kernel name stopped at 0.3.21 in January 2026.

A GPUwerk instance is a Docker container on a dedicated Spark with no Docker socket (see the tenant environment), so the docker run above does not work there; this does. Only /workspace survives a rebuild, so the environment and the model cache go there:

pip install uv
uv venv --python 3.12 /workspace/sglang-venv
source /workspace/sglang-venv/bin/activate
uv pip install --prerelease=allow "sglang==0.5.23"

python -c "import sglang, torch; print(sglang.__version__, torch.__version__, torch.version.cuda, torch.cuda.get_device_capability())"
# expect: 0.5.23 2.14.1 13.0 (12, 1)

export HF_HOME=/workspace/hf
nohup sglang serve --model-path Qwen/Qwen3-8B \
  --host 127.0.0.1 --port 30000 \
  --mem-fraction-static 0.80 --attention-backend flashinfer \
  --context-length 32768 >> /workspace/sglang.log 2>&1 &

The --prerelease=allow flag is from SGLang's docs: some dependencies only publish pre-releases, and uv older than 0.12.0 otherwise falls back to SGLang 0.5.9. Bind to 127.0.0.1 and use an SSH tunnel, or pass --api-key, because every instance has a public address. To start it after each restart, put the HF_HOME line and the serve command in /workspace/.gpuwerk-init, calling the venv's binary directly as /workspace/sglang-venv/bin/sglang, since the hook runs under /bin/sh without your activated venv.

The last pip failure on GB10 I could find is issue 18203 (February 2026): "Could not load any common_ops library!", a prebuilt sgl-kernel 0.3.21 wheel built against a different PyTorch ABI than the user's torch 2.10. I found no report either way for 0.5.23 on GB10, so let SGLang pick its own torch in a fresh venv.

Which flags matter on 128 GB of unified memory?

The memory fraction first, because the GPU pool and the operating system share the same 128 GB. SGLang's server arguments define --mem-fraction-static as the share used for model weights plus the KV cache pool; left unset it is computed from detected memory, or 0.88 if detection fails.

The Qwen3.8-27B cookbook has the measurement that matters: at 0.85 on a Spark, 15 of 48 configurations had the scheduler killed with exit code -15 and no traceback, because 0.85 leaves about 8 GB for the OS, exactly the threshold where DGX OS's earlyoom sends SIGTERM (journalctl -u earlyoom shows it). At 0.80 all 48 served. By my arithmetic, on the roughly 121.7 GB of unified memory a Spark exposes (the figure reported in issue 36941), 0.80 is a static pool of about 97 GB:

pool            = 0.80 x 121.7 GB                        = 97.4 GB
gpt-oss-120b    = MXFP4 weights, about 65 GB              -> ~32 GB left for KV
KV per token    = 2 x 18 full-attention layers x 8 KV heads x 64 dims x 2 bytes
                = 36,864 bytes (BF16; the other 18 layers use a 128-token window)
tokens of KV    = 32 GB / 36 KiB                          = roughly 0.9 million

Layer counts are from gpt-oss-120b's config.json (36 layers, 18 of them full attention, 8 KV heads of dimension 64). The 65 GB is my estimate for the checkpoint plus runtime buffers. The reverse also bites: Kubesimplify saw SGLang reserve about 80 GiB of KV cache for a 3B model at 0.75, memory nothing else on the box can use.

From the SGLang server arguments page, the Qwen3.8-27B cookbook and NVIDIA's playbook, read 7 November 2026. Which 4-bit format to pick is on the quantization page.

If memory pressure appears while you are inside the fraction, NVIDIA's troubleshooting tab blames page cache that unified memory does not hand back on its own, and gives sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'. It also warns that nvidia-smi's memory query fields read N/A on a Spark; watch MemAvailable in /proc/meminfo instead.

How fast is SGLang on a Spark compared with vLLM?

Level with vLLM for one request at the same precision, sometimes ahead with speculative decoding; I found no published same-setup multi-user comparison on one Spark. GPUwerk has not measured SGLang; its only engine measurements are vLLM 0.27.1 runs on the benchmarks page.

All reported by third parties. Only the Kubesimplify and Kompas rows test both engines in one setup. The Kompas run used a ThinkStation PGX (GB10), a nightly SGLang image, two patched files and CUDA graphs off. Issue 36131 is its author's AIPerf A/B with 2,000 new input tokens and 1,000 output on top of the shared prefix.

Read the gpt-oss rows with their dates: by August 2026 a forum test put vLLM 0.26.1 at 60.7 tok/s on the same model. For one request every engine moves the same bytes through 273 GB/s, and Kubesimplify's three-engine BF16 run finished within 0.86 tok/s. SGLang's structural edge is RadixAttention prefix reuse, which none of these rows isolate.

How do you turn on speculative decoding?

With --speculative-algorithm and a draft source; it is the one setting that moves single-user speed past the bandwidth line. LMSYS reported up to a 2x end-to-end speedup with EAGLE3 on Llama 3.1 8B on a Spark. Its command, moved to the current sglang serve CLI:

sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --quantization fp8 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \
  --speculative-num-steps 5 --speculative-eagle-topk 8 \
  --speculative-num-draft-tokens 32 \
  --mem-fraction-static 0.80 --host 127.0.0.1 --port 30000

LMSYS also set SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 for that draft. For checkpoints with a built-in MTP head, such as Qwen3.8-27B, the cookbook uses --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, and notes that MTP on the FlashInfer backend needs a FlashInfer newer than 0.6.15.post1, otherwise switch to --attention-backend triton. DFlash drafters have been in SGLang since April 2026 (the Kompas gist cites PR 22077). How the methods compare is in the sibling article on MTP, EAGLE-3 and DFlash.

How do you run SGLang across two Sparks?

Tensor parallel with --tp 2 --nnodes 2, one process per Spark, rendezvous on the ConnectX-7 address. The command follows SGLang's server-arguments example; cabling and NCCL setup are in the two-Spark cluster guide.

# Pin both to the 200GbE interface if the Sparks also have a slower NIC
export NCCL_SOCKET_IFNAME=<qsfp-interface> GLOO_SOCKET_IFNAME=<qsfp-interface>

# Spark 2 (rank 1) first
python -m sglang.launch_server --model-path <model> --tp 2 \
  --dist-init-addr <rank0-qsfp-ip>:50000 --nnodes 2 --node-rank 1

# Spark 1 (rank 0) within a few seconds
python -m sglang.launch_server --model-path <model> --tp 2 \
  --dist-init-addr <rank0-qsfp-ip>:50000 --nnodes 2 --node-rank 0 \
  --host 0.0.0.0 --port 30000 --mem-fraction-static 0.85

The rank 1 first order, the interface pinning and the 0.85 figure come from SGLang's Qwen3.8-Flash-Next cookbook, which ran TP=2 across two Sparks and measured 8 to 12 GiB of host memory free per node under load. It also traces a cross-node CUDA-graph deadlock to NCCL 2.28.x; check the startup line "sglang is using nccl==" for 2.29.7 or newer. Its DGX Spark cells prefilled 100k tokens at 2,400 to 2,840 tok/s. Which large models need two or more Sparks is in the multi-Spark MoE guide.

What errors do people hit, and what fixes them?

Mostly memory and stale builds.

Status as read on 7 November 2026. The sysctl values (vm.min_free_kbytes = 5242880 and lower dirty ratios) come from a user in issue 36941 whose box survived the event afterwards, losing only the worker; NVIDIA has not published them.

When should you pick SGLang over vLLM on a Spark?

When requests share long prefixes, or when the model's SGLang recipe is newer than its vLLM one. Agents that resend a large system prompt and tool schema, RAG with fixed instructions, and multi-turn chat all hit RadixAttention; SGLang also starts quicker (about 82 seconds against vLLM's 156 in Kubesimplify's 3B test, and 3 to 4 minutes against about 12 for gpt-oss-120b on two Sparks in the forum thread). For plain chat with unique prompts I would not switch for speed alone, since single-stream decode is a wash.

My pick for one Spark serving coding agents: lmsysorg/sglang:v0.5.23, --mem-fraction-static 0.80, FlashInfer attention, an FP8 KV cache and the model's MTP head if it has one. Keep a vLLM setup from the vLLM install guide next to it and run both on your own traffic for an hour. For the NVIDIA-tuned alternative, see the sibling guide to TensorRT-LLM on a Spark.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• Releases · sgl-project/sglang, GitHub, 2026-11-02
• Install SGLang, SGLang docs
• Server Arguments, SGLang docs
• Serve LLMs with SGLang: Instructions, NVIDIA build.nvidia.com
• Serve LLMs with SGLang: Overview, NVIDIA build.nvidia.com, 2026-07-31
• Serve LLMs with SGLang: Troubleshooting, NVIDIA build.nvidia.com
• lmsysorg/sglang tags, Docker Hub
• SGLang Release 26.10, NVIDIA Docs
• Qwen3.8-27B cookbook, SGLang docs
• Qwen3.8-Flash-Next cookbook, SGLang docs
• openai/gpt-oss-120b config.json, Hugging Face
• NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference, LMSYS, 2025-10-13
• Optimizing GPT-OSS on NVIDIA DGX Spark: Getting the Most Out of Your Spark, LMSYS, 2025-11-03
• Local LLM Inference Engines on DGX Spark, Explained, Kubesimplify, 2026-07-17
• Setting up vLLM, SGLang or TensorRT on two DGX Sparks, NVIDIA Developer Forums, 2025-12-02
• SGLang + DFlash on DGX Spark (Qwen3-Coder-Next NVFP4), GitHub Gist (Kompas), 2026-04-15
• [Feature] NVIDIA DGX Spark (GB10, sm_121a) Support Tracking (issue 11658), GitHub, 2025-10-15
• [Bug] DGX Spark: Could not load any common_ops library (issue 18203), GitHub, 2026-02-03
• [GB10] Long prefill (>40k tokens) exhausts unified memory (issue 36941), GitHub, 2026-08-29
• [Performance] Unified-cache default flip regresses long-prefix decode throughput on Spark and Thor (issue 36131), GitHub, 2026-08-24
• sglang release files (JSON API), PyPI