Serving large language models across multiple GPUs with PyTorch, vLLM, or DeepSpeed frequently terminates with an abrupt crash: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument mat2 in method wrapper_CUDA_mm). In other instances, the crash points to CPU and GPU contention: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!.

This error occurs when a matrix multiplication, attention operation, or concatenation combines tensors allocated on differing physical CUDA devices or between system RAM and GPU VRAM.

The Direct Solution: Dynamic Device Pinning and Worker Rank Binding

Resolve cross-device tensor collisions by eliminating hardcoded .cuda() or .to("cuda:0") calls, binding input tensors dynamically to the target layer device, and configuring explicit worker device assignments in distributed launchers:

# 1. Eliminate hardcoded device strings. Bind dynamically to the weight tensor device:
def forward_pass(self, hidden_states: torch.Tensor, attention_mask: torch.Tensor):
    # Ensure attention mask and position IDs match the hidden_states device
    target_device = hidden_states.device
    target_dtype = hidden_states.dtype
    
    if attention_mask.device != target_device:
        attention_mask = attention_mask.to(device=target_device, dtype=target_dtype, non_blocking=True)
        
    return self.layer(hidden_states, attention_mask)

For vLLM and multi-GPU serving scripts, eliminate conflicting device_map="auto" declarations in Hugging Face pipelines and enforce process rank isolation:

# Set PyTorch CUDA device context explicitly per worker process
export CUDA_DEVICE_ORDER=PCI_BUS_ID
# Do not mix manual torch.distributed with vLLM internal Ray/multiprocessing launchers
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92 \
  --distributed-executor-backend mp

Root Causes of Device Mismatch in Multi-GPU LLM Serving

PyTorch operations execute strictly within the memory space of a single device. A tensor in GPU 0 memory cannot directly participate in a tensor core matrix multiplication on GPU 1 without an explicit inter-GPU peer-to-peer transfer or collective communication call (such as all_reduce or all_gather).

The table below contrasts common failure scenarios, affected frameworks, and verified remediations:

Failure Pattern	Trigger Scenario	Offending Device Pair	Direct Fix
Hardcoded CUDA String	Custom LoRA hook or positional embedding	cpu vs cuda:0	Replace .to("cuda") with tensor.to(device=target.device)
Tensor Parallel Rank Collision	Attention layer splits across GPUs	cuda:0 vs cuda:1	Bind inputs to torch.cuda.current_device()
Hugging Face device_map="auto" Conflict	Mixing pipeline() with vLLM engine	cuda:0 vs cuda:2	Pass device_map=None and let vLLM manage workers
Multimodal Embedding Mismatch	Vision / audio token projection in VLM	cpu (pinned) vs cuda:0	Cast image feature tokens to model input device
Draft Model Speculative Mismatch	Target 70B on 4 GPUs, 1B draft on 1 GPU	cuda:0 vs cuda:1	Align draft model tensor parallelism or use Ray worker pool

Step-by-Step Resolution Guide

Follow these steps to isolate the offending line of code and correct device allocations across your Linux server.

Step 1: Trace the Exact Failing Layer and Tensors

Standard PyTorch stack traces pinpoint the C++ kernel invocation (wrapper_CUDA_mm or aten::bmm) rather than the Python line where tensor placement went wrong. Run your script with synchronous CUDA error tracing enabled:

# Force PyTorch CUDA kernels to execute synchronously for pinpoint stack traces
CUDA_LAUNCH_BLOCKING=1 TORCH_SHOW_CPP_STACKTRACES=1 python3 serve_model.py

Inspect the traceback to find the exact module line. Check the physical device of each tensor involved in the operation by inserting a temporary debug assertion:

# Debug check before matrix multiplication
print(f"[DEBUG Rank {torch.distributed.get_rank() if torch.distributed.is_initialized() else 'Main'}] "
      f"Tensor A: {tensor_a.device}, Tensor B: {tensor_b.device}")
assert tensor_a.device == tensor_b.device, (
    f"Device mismatch detected: Tensor A is on {tensor_a.device}, "
    f"but Tensor B is on {tensor_b.device}"
)

Step 2: Fix Hardcoded Device Casts in Custom Attention and LoRA Adapters

A frequent source of this bug in 2026 is loading third-party LoRA adapters or speculative draft heads. Developers often write:

# BAD: Hardcoded device assumptions
residual = residual.cuda() # Defaults to cuda:0 regardless of current worker rank!
pos_embed = pos_embed.to("cuda:0") # Fails on rank 1, 2, and 3

In multi-GPU tensor parallelism, worker 1 runs on cuda:1. Calling .cuda() defaults to cuda:0 unless torch.cuda.set_device(1) was invoked globally.

Replace all hardcoded transfers with rank-aware dynamic bindings:

# GOOD: Rank-aware dynamic device binding
class FastLoRALinear(torch.nn.Module):
    def __init__(self, in_features: int, out_features: int, r: int = 8):
        super().__init__()
        self.lora_A = torch.nn.Parameter(torch.empty(r, in_features))
        self.lora_B = torch.nn.Parameter(torch.zeros(out_features, r))
        
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Match parameter device to input tensor device dynamically
        device = x.device
        dtype = x.dtype
        
        # If weights were moved to a different GPU during model sharding
        if self.lora_A.device != device:
            self.lora_A.data = self.lora_A.data.to(device=device, dtype=dtype)
            self.lora_B.data = self.lora_B.data.to(device=device, dtype=dtype)
            
        return (x @ self.lora_A.t()) @ self.lora_B.t()

Step 3: Configure PyTorch Distributed Worker Initialization

When writing custom multi-GPU scripts using torch.multiprocessing or torch.distributed.run, each subprocess must set its active CUDA device context immediately after fork:

import os
import torch
import torch.distributed as dist

def init_worker():
    # Read rank assigned by torchrun
    local_rank = int(os.environ.get("LOCAL_RANK", 0))
    world_size = int(os.environ.get("WORLD_SIZE", 1))
    
    # CRITICAL: Set active device before any tensor operations or NCCL initialization
    torch.cuda.set_device(local_rank)
    
    # Initialize NCCL backend
    dist.init_process_group(
        backend="nccl",
        init_method="env://",
        world_size=world_size,
        rank=local_rank
    )
    
    # Verifying active device matches worker rank
    current_gpu = torch.cuda.current_device()
    print(f"Worker {local_rank} successfully bound to physical GPU cuda:{current_gpu}")

if __name__ == "__main__":
    init_worker()

Run the worker cluster using torchrun:

torchrun --nproc_per_node=4 --master_port=29500 serve_cluster.py

Step 4: Resolve vLLM Multi-GPU Tensor Parallel Conflicts

In vLLM deployments, RuntimeError: Expected all tensors to be on the same device typically occurs when users pass Hugging Face model loading options that interfere with vLLM’s internal Megatron-style tensor sharding.

Verify these three configurations in your serving environment:

1. Remove device_map from Engine Arguments: Never pass device_map="auto" or device_map="balanced" to vLLM or LLM(model=...). vLLM manages GPU memory through PagedAttention and handles sharding automatically.
2. Standardize Distributed Backend: If you run on a single multi-GPU node (e.g., 4x RTX 4090 or 8x H100), use --distributed-executor-backend mp rather than ray. Native multiprocessing bypasses Ray object store device mapping bugs:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --distributed-executor-backend mp \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --enforce-eager

3. Check Speculative Decoding Draft Model Placement: When using speculative decoding with a draft model (e.g., --speculative-model meta-llama/Llama-3.2-1B-Instruct), ensure the draft model can be sharded across the same tensor parallel degree or fits within each worker GPU memory allocation. If draft model tensors remain on cuda:0, vLLM worker 1 will crash during draft token verification.

Verifying Multi-GPU Tensor Placement on Linux

Confirm that all tensors and model layers reside on their expected physical GPUs using this diagnostic script:

import torch

def audit_model_device_topology(model: torch.nn.Module):
    device_counts = {}
    for name, param in model.named_parameters():
        dev = str(param.device)
        device_counts[dev] = device_counts.get(dev, 0) + 1
        
    print("Model Parameter Device Distribution:")
    for dev, count in device_counts.items():
        print(f"  - {dev}: {count} parameter tensors")
        
    if len(device_counts) > 1 and not hasattr(model, "is_parallel"):
        print("WARNING: Model parameters are split across multiple devices without parallel wrapper.")
    else:
        print("PASS: Device topology verified.")

# Test on loaded model
# audit_model_device_topology(my_model)

Running this audit on our 4x RTX 4090 testbed confirms uniform parameter distribution across cuda:0, cuda:1, cuda:2, and cuda:3, eliminating runtime crashes and stabilizing continuous high-throughput inference.