To resolve RuntimeError: CUDA error: invalid device ordinal immediately, inspect how many devices PyTorch actually sees and reference them using zero-based virtual indices:

That stalls the pipeline. Execution halts immediately.

# Check how many visible GPUs PyTorch detects in your current shell
python3 -c "import torch; print(f'Available GPUs: {torch.cuda.device_count()}')"

If you set CUDA_VISIBLE_DEVICES=2,3, CUDA re-indexes physical GPU 2 to cuda:0 and physical GPU 3 to cuda:1. In that masked shell, requesting cuda:2 crashes immediately.

# Correct: Map to virtual ordinal 0 or 1
export CUDA_VISIBLE_DEVICES=2,3
python3 -c "import torch; x = torch.tensor([1.0]).to('cuda:0'); print('Success on:', torch.cuda.get_device_name(0))"

Root Cause Analysis

The exception RuntimeError: CUDA error: invalid device ordinal indicates that PyTorch requested a device index greater than or equal to the total count of visible CUDA devices.

The error trace typically looks like this:

Traceback (most recent call last):
  File "train.py", line 42, in <module>
    torch.cuda.set_device(args.gpu)
  File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 404, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace above might be incorrect.
Compiled with CUDA SDK 12.4.

Or when moving tensors directly to a device:

RuntimeError: CUDA error: invalid device ordinal when calling cudaGetDeviceCount(&count)

This failure stems from three primary architecture mismatches:

1. Virtual Index Remapping via CUDA_VISIBLE_DEVICES: When you mask GPUs in Linux, CUDA constructs a virtual ordinal space starting at index 0. If you assign CUDA_VISIBLE_DEVICES=3, your single available GPU is addressed as cuda:0. Code requesting cuda:3 fails because the visible device list length is 1.
2. Docker and Kubernetes Container Isolation: When running Docker with --gpus '"device=2,3"', the NVIDIA Container Runtime injects container-level isolation. Physical cards 2 and 3 appear inside the container as /dev/nvidia0 and /dev/nvidia1. Referencing host hardware indices inside the guest fails.
3. Global RANK vs Local LOCAL_RANK in Distributed Training: Multi-node frameworks (torchrun, Ray, SLURM) assign a global rank across all nodes (such as ranks 0 through 15 across two 8-GPU nodes). If code calls torch.cuda.set_device(int(os.environ["RANK"])) on node 2, rank 8 crashes because node 2 only has 8 local devices (indices 0 through 7).

Configuration Context	Physical Hardware IDs	Masking Variable	Valid PyTorch Ordinals	Invalid Ordinal Call
Default Shell	GPU 0, GPU 1	None	cuda:0, cuda:1	cuda:2
Selected Secondary GPU	GPU 0, GPU 1, GPU 2, GPU 3	CUDA_VISIBLE_DEVICES=2	cuda:0 (points to GPU 2)	cuda:2
Multi-GPU Sub-cluster	GPU 0 through GPU 7	CUDA_VISIBLE_DEVICES=4,5,6	cuda:0, cuda:1, cuda:2	cuda:4
Docker GPU Passthrough	Host GPU 1	--gpus '"device=1"'	cuda:0 inside container	cuda:1
Multi-Node torchrun	2 Nodes x 4 GPUs	LOCAL_RANK=0..3, RANK=0..7	cuda:0 through cuda:3	cuda:4

Step-by-Step Resolution

Step 1: Diagnose Physical vs. Visible Devices

First, inspect what the Linux host kernel sees versus what Python sees.

Check physical GPU hardware with nvidia-smi:

nvidia-smi --query-gpu=index,name,uuid,memory.total --format=csv

Example output:

index, name, uuid, memory.total [MiB]
0, NVIDIA RTX 4090, GPU-a1b2c3d4-e5f6-7890-1234-567890abcdef, 24564 MiB
1, NVIDIA RTX 4090, GPU-f6e5d4c3-b2a1-0987-6543-210fedcba987, 24564 MiB

Next, test Python device visibility in your active environment:

python3 -c "
import os, torch
print('CUDA_VISIBLE_DEVICES env:', os.environ.get('CUDA_VISIBLE_DEVICES'))
print('CUDA available:', torch.cuda.is_available())
print('Device count:', torch.cuda.device_count())
for i in range(torch.cuda.device_count()):
    print(f'  Index {i}: {torch.cuda.get_device_name(i)}')
"

If torch.cuda.device_count() prints 1, any target index above 0 triggers invalid device ordinal.

Step 2: Implement a Safe Device Selector in Code

Never hardcode GPU indices. Implement a fallback-aware device resolver that validates requested ordinals against runtime capacity:

import os
import torch
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("cuda_manager")

def select_torch_device(preferred_index: int = 0) -> torch.device:
    """Safely select a PyTorch CUDA device ordinal.
    
    Falls back to cuda:0 or CPU if the preferred index is invalid.
    """
    if not torch.cuda.is_available():
        logger.warning("CUDA unavailable. Falling back to CPU.")
        return torch.device("cpu")
    
    visible_count = torch.cuda.device_count()
    
    if visible_count == 0:
        logger.warning("No CUDA devices detected. Falling back to CPU.")
        return torch.device("cpu")
        
    if preferred_index < 0 or preferred_index >= visible_count:
        logger.warning(
            "Requested device index %d is out of bounds (visible count: %d). Defaulting to cuda:0.",
            preferred_index,
            visible_count
        )
        return torch.device("cuda:0")
        
    return torch.device(f"cuda:{preferred_index}")

# Test execution
device = select_torch_device(preferred_index=2)
x = torch.zeros((1024, 1024), device=device)
print(f"Allocated tensor on: {device} ({torch.cuda.get_device_name(device)})")

Step 3: Fix Docker Container GPU Arguments

When launching containers, developers often pass --gpus '"device=2"' and execute code containing device = "cuda:2".

Inside the container namespace, physical device 2 is mapped to index 0.

Launch the container by assigning device 2:

docker run --gpus '"device=2"' --rm -it pytorch/pytorch:2.5.0-cuda12.4-cudnn9-runtime \
  python3 -c "import torch; print(f'Inside container device: {torch.cuda.get_device_name(0)}')"

Notice that code inside the container references cuda:0, not cuda:2.

If your code strictly requires retaining host-matching numbers, expose all GPUs and isolate them inside via environment variables:

docker run --gpus all -e CUDA_VISIBLE_DEVICES=2 --rm -it pytorch/pytorch:2.5.0-cuda12.4-cudnn9-runtime \
  python3 -c "import torch; print('Visible count:', torch.cuda.device_count())"

Step 4: Correct Distributed Training Rank Assignment

In multi-GPU distributed data parallel (DDP) setups launched with torchrun, every process must bind to its local node rank, not the global cluster rank.

The fatal mistake:

# BROKEN: Global rank across all nodes
global_rank = int(os.environ["RANK"])
torch.cuda.set_device(global_rank) # Crashes on worker nodes where rank >= local GPUs

The verified fix:

import os
import torch
import torch.distributed as dist

def init_distributed():
    # Read LOCAL_RANK set by torchrun or SLURM
    local_rank = int(os.environ.get("LOCAL_RANK", 0))
    
    # Set device to local ordinal (0 to N-1 on the current machine)
    torch.cuda.set_device(local_rank)
    device = torch.device(f"cuda:{local_rank}")
    
    # Initialize process group
    dist.init_process_group(backend="nccl")
    
    print(f"[Node Local Rank {local_rank}] Bound to GPU: {torch.cuda.get_device_name(local_rank)}")
    return device

if __name__ == "__main__":
    device = init_distributed()

Launch with torchrun specifying the exact number of GPUs per node:

torchrun --nproc_per_node=2 train_ddp.py

Common Pitfalls & Edge Cases

Trap 1: Setting CUDA_VISIBLE_DEVICES After Importing PyTorch

CUDA initializes driver handles the moment torch.cuda is referenced. Modifying the environment variable in Python code after importing or calling CUDA functions fails silently:

# BROKEN ORDER
import torch

# This has zero effect if torch.cuda was touched
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
print(torch.cuda.device_count()) # Returns original count, not masked count!

To modify visible devices programmatically, set os.environ before importing PyTorch:

# CORRECT ORDER
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

import torch
print(torch.cuda.device_count()) # Returns 1, mapped to cuda:0

Trap 2: Using PCI Bus IDs vs. Driver Ordinals

nvidia-smi lists GPUs ordered by PCI bus ID by default, while PyTorch CUDA runtime orders them by compute capability or driver enumeration.

If physical card order conflicts, set:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1

This guarantees that index 0 in PyTorch maps to the exact physical card in PCI slot 0.

Trap 3: PyTorch Lightning and Accelerate Auto Mapping

Frameworks like Hugging Face Accelerate or PyTorch Lightning automate device allocation. If you pass device_map="auto" while simultaneously setting torch.cuda.set_device(), the framework clashes with manual ordinal bindings:

# Let Accelerate or Transformers manage mapping without manual set_device calls
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    device_map="auto" # Automatically calculates memory across visible virtual ordinals
)

Complete Verification Script

Save this script as check_cuda_ordinals.py to audit your system configuration before launching model runs:

#!/usr/bin/env python3
"""Diagnostic utility for CUDA device ordinals in PyTorch."""

import os
import sys
import torch

def audit():
    print("=" * 60)
    print("PyTorch CUDA Device Ordinal Audit")
    print("=" * 60)
    
    env_devices = os.environ.get("CUDA_VISIBLE_DEVICES")
    print(f"CUDA_VISIBLE_DEVICES : {env_devices if env_devices is not None else '<unset>'}")
    print(f"CUDA_DEVICE_ORDER    : {os.environ.get('CUDA_DEVICE_ORDER', '<unset>')}")
    print(f"PyTorch Version      : {torch.__version__}")
    print(f"CUDA Available       : {torch.cuda.is_available()}")
    
    if not torch.cuda.is_available():
        print("[-] CUDA is not available on this host. Exiting.")
        sys.exit(0)
        
    count = torch.cuda.device_count()
    print(f"Detected Device Count: {count}")
    print("-" * 60)
    
    for i in range(count):
        try:
            name = torch.cuda.get_device_name(i)
            props = torch.cuda.get_device_properties(i)
            total_mem_gb = props.total_memory / (1024 ** 3)
            print(f"[+] Virtual Ordinal 'cuda:{i}':")
            print(f"    Name      : {name}")
            print(f"    Memory    : {total_mem_gb:.2f} GB")
            print(f"    Capability: {props.major}.{props.minor}")
            
            # Test small allocation
            tensor = torch.zeros((100, 100), device=f"cuda:{i}")
            del tensor
            torch.cuda.empty_cache()
            print("    Allocation: OK")
        except RuntimeError as exc:
            print(f"[-] Virtual Ordinal 'cuda:{i}': FAILED ({exc})")

    # Test out of bounds ordinal to demonstrate expected behavior
    print("-" * 60)
    test_oob_index = count
    try:
        torch.cuda.set_device(test_oob_index)
        print(f"[-] Warning: cuda:{test_oob_index} did not raise an error.")
    except RuntimeError as exc:
        print(f"[+] Verified expected safety check on out-of-bounds ordinal 'cuda:{test_oob_index}':")
        print(f"    {exc}")

    print("=" * 60)
    print("All visible device ordinals verified successfully.")

if __name__ == "__main__":
    audit()

Run the audit script:

python3 check_cuda_ordinals.py

The script verifies every visible ordinal and confirms that PyTorch operates cleanly without device index collision.

Official Documentation & Primary References

• PyTorch Official CUDA Semantics Documentation: Memory management, device ordinals, asynchronous execution, and stream synchronization.
• PyTorch Autograd Mechanics: Computational graph lifecycle, in-place operation checks, and backward pass execution.
• PyTorch Distributed (c10d / DDP) Documentation: NCCL backend initialization, watchdog timeouts, and gradient reduction hooks.
• NVIDIA CUDA Toolkit & Driver Documentation: Official compute capability tables, cuBLAS/cuDNN status codes, and NVML driver management.

More in \"PyTorch & CUDA"

How to Fix 'RuntimeError: Ninja is required to load C++ extensions' in PyTorch on Linux

Fix PyTorch 'Ninja is required to load C++ extensions' on Linux. Install build dependencies, set MAX_JOBS to prevent OOM, and resolve NVCC mismatches.

How to Fix 'RuntimeError: Trying to backward through the graph a second time' in PyTorch on Linux

Fix PyTorch 'Trying to backward through the graph a second time' error on Linux. Understand autograd graph buffer freeing, loss accumulation, and retain_graph traps.

How to Fix 'RuntimeError: view size is not compatible with input tensor's size and stride' in PyTorch on Linux

Resolve PyTorch 'view size is not compatible with input tensor size and stride' on Linux. Understand tensor contiguity, reshape vs view, and memory overhead.