How to Fix NVIDIA Docker GPU Passthrough for Local LLMs on Linux

Configuring NVIDIA Container Toolkit and Docker Runtime Hooks

Nothing kills a deployment faster than watching nvidia-smi work fine on your bare-metal Ubuntu host while docker run --gpus all crashes immediately with could not select device driver "" with capabilities: [[gpu]]. Docker’s default runc runtime has no idea what a GPU is out of the box. It cannot see /dev/nvidia0.

To bridge the host kernel driver into container namespaces on Ubuntu or Debian, install the official NVIDIA Container Toolkit and register the OCI runtime hook:

# 1. Add NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# 2. Install toolkit packages
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

# 3. Configure Docker runtime and restart daemon
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU visibility inside a minimal CUDA container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

1. Why the Host Driver Alone Is Never Enough

Docker containers cannot access host PCIe hardware by default. While CPU cores and system RAM are sliced directly by standard Linux kernel cgroups, NVIDIA GPUs depend on proprietary user-space shared libraries (libcuda.so, libnvidia-ml.so) that must match the exact kernel module version running on the host.

If you try to bake the NVIDIA driver directly inside a Dockerfile, the container will break the moment the host kernel updates. Instead, libnvidia-container intercepts container creation via an OCI prestart hook and bind-mounts the host’s /dev/nvidia* character devices and user-space driver binaries straight into the container filesystem at launch time:

1. Host NVIDIA Driver: Installed on the bare-metal Linux OS (nvidia-smi must return a valid driver table on the host before touching Docker). If PyTorch still fails to detect the card on the host itself, follow our torch.cuda.is_available() returns False diagnostic guide.
2. NVIDIA Container Runtime Hook (libnvidia-container): Injects /dev/nvidia0, /dev/nvidiactl, and /dev/nvidia-uvm into container namespaces.
3. Docker Daemon Configuration: /etc/docker/daemon.json must register nvidia as an available OCI runtime. When nvidia-ctk runtime configure runs, it writes that JSON block automatically. Always restart docker.service afterward, or the daemon will keep using the stale in-memory config.

2. Running Local LLMs (Ollama & vLLM) in Docker

Once the OCI hook is active, launch local LLM serving containers with explicit GPU device flags.

Ollama with Full GPU Passthrough

docker run -d \
  --name ollama \
  --gpus all \
  -v ollama_models:/root/.ollama \
  -p 11434:11434 \
  --restart unless-stopped \
  ollama/ollama

Verify that Ollama offloads model layers to VRAM rather than falling back to system DDR5 RAM:

docker exec -it ollama ollama run deepseek-r1:8b

vLLM Production Container

For multi-user batched inference (see our Ollama vs vLLM local inference comparison for concurrency trade-offs), pass --ipc=host alongside --gpus all:

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --ipc=host \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-Coder-7B-Instruct

Do not skip --ipc=host. Without it, Docker restricts /dev/shm to 64 MB, and PyTorch’s NCCL shared-memory tensor rings crash midway through model initialization.

3. Docker Compose GPU Configuration (v2 Format)

Stop copying deprecated version: '3.8' snippets that rely on runtime: nvidia. Modern Docker Compose GPU support uses the deploy.resources.reservations.devices block:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama-gpu
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama_data:

If you run a multi-GPU Linux server (for example, dual RTX 4090s) and want to pin a container to a specific physical card, replace count: all with device_ids: ['0'].

4. Field Notes: Fixing the 4 nastiest GPU Container Traps

1. Rootless Docker or Snap Docker Ignoring /etc/docker/daemon.json

I lost an hour debugging a fresh Ubuntu 24.04 box where nvidia-ctk succeeded, /etc/docker/daemon.json looked right, and --gpus all still failed. The culprit was Ubuntu’s Snap-packaged Docker, which ignores /etc/docker/daemon.json entirely. Check which docker. If it points to /snap/bin/docker, remove the snap (sudo snap remove docker) and install docker-ce from Docker’s official apt repository.

2. “Failed to initialize NVML: Driver/library version mismatch”

This happens when Ubuntu’s unattended-upgrades silently updates linux-modules-nvidia in the background while your container is running. The running kernel module version no longer matches the user-space library mounted into the container. Stop all GPU containers, reload the kernel module (sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia && sudo modprobe nvidia_uvm), or reboot the host.

3. Container Exits Immediately with Code 137 During Weight Loading

If the container dies silently while loading .safetensors shards before cudaMalloc even fires, you hit the Linux cgroup host RAM limit, not a GPU VRAM error. Check dmesg -T | grep -i oom and follow our Docker Exit Code 137 OOM troubleshooting guide.

4. “CUDA Out of Memory inside container while host seems idle”

Another container or orphaned Python worker is holding an active CUDA context on the card. Run sudo fuser -v /dev/nvidia* on the host to list every PID attached to the GPU device nodes and kill any zombie processes before restarting your container.