Docker and NGC containers on a DGX Spark: what DGX OS ships, arm64 images, a compose file and memory limits

A DGX Spark you own comes ready for GPU containers: DGX OS 7.6.0 (14 September 2026) ships Docker Engine 29.6.2 and NVIDIA Container Toolkit 1.20.0, preconfigured, so docker run --gpus all works on first boot. Every NGC framework image I checked on 7 November 2026 (PyTorch, vLLM and SGLang 26.10, TensorRT-LLM 1.4.0rc3) is multi-arch with a linux/arm64 build, and NVIDIA dropped the separate -igpu tags after 26.02. The trap is memory: Docker's --memory limit does not count GPU allocations on unified memory, so cap the inference engine instead. A rented GPUwerk Spark is itself a container with no Docker socket, so this guide is for owned machines.

Read 7 November 2026. Partner GB10 machines (ASUS, Dell, Lenovo and others) get the same stack later; check your maker's notes.

What does DGX OS install for containers?

Docker Engine plus the NVIDIA Container Toolkit, configured as Docker's GPU runtime. NVIDIA's container runtime page says the toolkit "is preinstalled and configured on DGX Spark systems" and that GPU devices are exposed through OCI hooks when you pass --gpus. The versions have moved with each DGX OS 7 release, per the DGX OS 7 notes:

From the DGX OS 7 release notes, read 7 November 2026. The toolkit's own GitHub releases show 1.20.1 published 19 September 2026. "Not listed" means that release's notes name no new version. Versions on a machine that installs OTA updates can be newer than its DGX OS ISO.

Check your own machine, and drop the need for sudo, with the commands from NVIDIA's guide:

docker --version
nvidia-ctk --version
cat /etc/docker/daemon.json          # should name the nvidia runtime
sudo usermod -aG docker $USER && newgrp docker
docker run --rm --gpus=all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi

If a container fails to see the GPU, NVIDIA's playbooks give sudo nvidia-ctk runtime configure --runtime=docker and a Docker restart. One more variant exists in the wild: SGLang's Qwen3.8-27B cookbook reports GB10 hosts where "docker GPU access is CDI-only" because no nvidia runtime is registered, and uses --device nvidia.com/gpu=all instead of --gpus all. nvidia-smi's memory columns read "Not Supported" on a Spark, which NVIDIA's known issues page calls expected for an integrated GPU.

How do you run an NGC container with GPU access?

Log in to nvcr.io once with an NGC personal API key, then use --gpus all and the shared-memory flags the framework images expect. The Spark NGC page says the key must include the NGC Catalog service, the username is literally $oauthtoken, and DGX Spark needs no serial-number registration before NIM pulls:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

docker run --gpus all -it --rm --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v "$PWD:/workspace" -w /workspace \
  nvcr.io/nvidia/pytorch:26.10-py3

--ipc=host gives PyTorch the shared memory it uses between processes; vLLM's Docker docs accept --shm-size as the alternative. The ulimit flags come from NVIDIA's TensorRT-LLM and SGLang playbooks. Most framework images need no login at all; NIM microservices do.

Two things in NVIDIA's own pages are stale. The Spark NGC page still pulls nvcr.io/nvidia/pytorch:24.08-py3, a CUDA 12.6 image from before NVIDIA's containers were optimized for Blackwell (that started with 25.01, per the PyTorch release notes); use 26.10. And the same notes say NVIDIA stopped producing standalone iGPU containers with 26.03, so the -igpu tags people used in 2025 end at 26.02. Which image to pick per framework is in the sibling guides to PyTorch and SGLang on a Spark; NIM's licensing is in the NIM guide.

Which images have arm64 builds?

All the major inference images do; the ones that bite are stale tags and application containers. I checked the registry manifests directly; this is the result on 7 November 2026:

Architectures from the nvcr.io, Docker Hub and ghcr.io manifests. NGC sizes are my sums of the compressed arm64 layers; Docker Hub sizes are as listed on its tag pages. The cu130-nightly row matters because older guides, including some on this site, recommend that tag: vLLM's 0.33 release notes say its images are now CUDA 13.0 only, so the plain release tag on Docker Hub replaces it.

To check any image yourself before pulling 15 GB, use the command from Docker's reference and look for a linux/arm64 platform line:

docker buildx imagetools inspect nvcr.io/nvidia/vllm:26.10-py3
docker buildx imagetools inspect vllm/vllm-openai:v0.33.0 | grep -A1 Platform

A compose file for vLLM plus Open WebUI

One GPU service and one CPU service, both bound to localhost. The GPU block follows Docker's Compose GPU guide (capabilities: [gpu] is mandatory). The vLLM service uses NVIDIA's 26.10 image with the model from NVIDIA's vLLM playbook (updated 19 October 2026), and the playbook's pattern of --entrypoint vllm followed by serve. Save as compose.yaml next to a .env file holding HF_TOKEN and VLLM_API_KEY:

services:
  vllm:
    image: nvcr.io/nvidia/vllm:26.10-py3
    entrypoint: ["vllm"]
    command: ["serve", "nvidia/Qwen3.8-27B-NVFP4",
              "--served-model-name", "qwen3.8-27b",
              "--gpu-memory-utilization", "0.7",
              "--max-model-len", "32768",
              "--api-key", "${VLLM_API_KEY}"]
    environment:
      - HF_TOKEN=${HF_TOKEN}
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
      - ${HOME}/.cache/vllm:/root/.cache/vllm
      - ${HOME}/.cache/flashinfer:/root/.cache/flashinfer
    ports:
      - "127.0.0.1:8000:8000"
    ipc: host
    ulimits:
      memlock: -1
      stack: 67108864
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    depends_on: [vllm]
    environment:
      - OPENAI_API_BASE_URL=http://vllm:8000/v1
      - OPENAI_API_KEY=${VLLM_API_KEY}
      - ENABLE_OLLAMA_API=false
    volumes:
      - open-webui:/app/backend/data
    ports:
      - "127.0.0.1:8080:8080"
    restart: unless-stopped

volumes:
  open-webui:

docker compose up -d
docker compose logs -f vllm          # first start downloads weights and compiles kernels
curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"

Notes on the choices. --gpu-memory-utilization 0.7 is NVIDIA's own advice in the vLLM 26.10 release notes, which warn that vllm serve otherwise allocates "effectively --gpu-memory-utilization≈1.0" and runs out of memory on unified-memory systems; the same notes say 26.10 ships prebuilt Qwen3.8 kernel and autotuning caches for the 27B on DGX Spark. NVIDIA's full recipe adds speculative decoding; this is the minimal version. The Open WebUI variables are from its environment reference, which marks OPENAI_API_BASE_URL as a ConfigVar: after the first start it is stored in Open WebUI's database, and later edits in the compose file are ignored, so change it in the admin settings instead. Accounts, HTTPS and signups are in the Open WebUI guide; when one compose file stops being enough, see running several services on a Spark.

How do memory limits work with unified memory?

Badly, at the container level. GPU allocations come out of the same 128 GB as everything else and are not charged to the container's cgroup. In the forum thread "DGX Spark becomes unresponsive instead of throwing CUDA OOM", an owner reported in April 2026 that allocating 10.7 GB on the GPU raised memory.current by only 377 MB, that a training container started with a 100 GB --memory limit still froze the machine, and that wrapping docker run in a systemd-run scope did not help because Docker places the container in its own cgroup. Those are one user's measurements; I found no NVIDIA statement either way.

So budget memory inside the engine. By my calculation for the compose file above:

visible unified memory          ~121.7 GB
vLLM pool at 0.7 (of total)       85.2 GB
  Qwen3.8-27B NVFP4 weights      -21.9 GB   (size on disk, per SGLang's cookbook)
  KV cache and activations        ~63 GB
KV per token (BF16)             2 x 16 full-attention layers x 4 KV heads x 256 dims x 2 bytes = 64 KiB
=> roughly 1 million tokens, about 30 full 32k conversations
left for DGX OS, Open WebUI, page cache  ~36 GB

Layer and head counts are from Qwen3.8-27B's config.json: 64 layers, of which 16 use full attention with 4 KV heads of dimension 256; the other 48 are linear attention with a small fixed state that I left out. The activation share is a guess, so treat the token count as an upper bound. The vLLM side of this arithmetic, including why the fraction counts total rather than free memory, is in the vLLM install guide.

Two host settings help when something still runs away. NVIDIA's known-issues page gives sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches' for page cache that unified memory does not release, and SGLang's cookbook notes that DGX OS runs earlyoom, which sends SIGTERM at about 8 GB free; check journalctl -u earlyoom when a container dies without a log line.

What are the multi-arch pitfalls?

x86-only images, x86-only tools and old tags. The exec format error is the classic symptom: Docker's multi-platform docs explain that containers share the host kernel, so a linux/amd64 image cannot run on an arm64 host without emulation. Examples from Spark owners:

• NVIDIA's RAG blueprint: the NIMs and Milvus ran, but the ingestor-server and rag-server containers exited with exec format error. NVIDIA replied in February 2026 that these tools "are not currently supported on ARM architecture" and are on the roadmap (forum thread 354990).
• The NGC CLI: an owner following the user guide in October 2025 downloaded an x86-64 binary (thread 348543); the guide now points at the ARM64 Linux tab.
• Stale tags that do have arm64 builds, such as lmsysorg/sglang:spark and vllm/vllm-openai:cu130-nightly, which miss months of GB10 fixes.

Building your own images is easiest on the Spark itself, natively. Building arm64 images on an x86 laptop works through docker buildx build --platform linux/arm64, which uses QEMU emulation, and Docker warns that emulation "can be much slower than native builds"; for images that compile CUDA code, that can mean hours. Also check CUDA versions: an image built on CUDA 13.4, like the NGC 26.10 set, runs on the Spark's 580 driver under CUDA minor-version compatibility, but code that JIT-compiles newer PTX may not.

Does any of this work on a rented Spark?

No, and that is by design. A rented instance is a Docker container on a dedicated Spark with the GPU passed through; the tenant environment page lists no Docker socket and no privileged mode, so docker run, docker compose and NGC or NIM images cannot start inside it. What does work there is the pip route for each engine: the sibling guides for SGLang and PyTorch cover it, and vLLM has a native install path. My advice: if your workflow is a stack of NGC containers and a compose file, you want a Spark of your own; if it is one engine serving one model, a rental is the faster way to find out whether the model fits. If you would rather manage project containers from a laptop, see AI Workbench on a DGX Spark.

FAQ

Sources

Every figure above is attributed to its source in the sentence or table where it appears. Pages were read on 7 November 2026 unless a date says otherwise.

• DGX OS 7 Release Notes, NVIDIA Docs, 2026-10-27
• NVIDIA Container Runtime for Docker (DGX Spark User Guide), NVIDIA Docs
• NGC (DGX Spark User Guide), NVIDIA Docs
• Known Issues (DGX Spark User Guide), NVIDIA Docs
• DGX Spark Release Notes, NVIDIA Docs
• Releases · NVIDIA/nvidia-container-toolkit, GitHub, 2026-09-19
• vLLM Release 26.10, NVIDIA Docs
• PyTorch Release 26.10, NVIDIA Docs
• Serve LLMs with vLLM (playbook-vllm README), NVIDIA dgx-spark-playbooks on GitHub, 2026-10-19
• Enable GPU access with Docker Compose, Docker Docs
• Multi-platform builds, Docker Docs
• docker buildx imagetools inspect, Docker Docs
• Using Docker (vLLM deployment docs), vLLM docs
• Environment Variable Configuration, Open WebUI docs
• vllm/vllm-openai tags, Docker Hub
• Releases · vllm-project/vllm, GitHub, 2026-11-02
• DGX Spark becomes unresponsive (zombie) instead of throwing CUDA OOM, NVIDIA Developer Forums, 2025-12-06
• RAG Blueprint on DGX Spark (ARM64 / GB10): ingestor-server / rag-server fail with exec format error, NVIDIA Developer Forums, 2025-12-17
• Ngc-cli wrong architecture?, NVIDIA Developer Forums, 2025-10-21
• Qwen3.8-27B cookbook, SGLang docs
• Qwen/Qwen3.8-27B config.json, Hugging Face