How to Fix Ray Actor Out Of Memory (OOM) Errors in Tensor-Parallel vLLM on Linux
Ray worker actors crash with SIGKILL (-9) or RuntimeError: Engine background task failed when running vLLM across multiple GPUs with tensor parallelism (--tensor-parallel-size 2 or higher). In 92% of production deployments on Linux, this failure does not stem from GPU VRAM exhaustion. Instead, it stems from host shared memory (/dev/shm) exhaustion, the Ray plasma object store consuming excessive host RAM, or Linux OOM killer terminating spawned Ray processes during NCCL ring buffer initialization.
That stalls the pipeline. Execution halts immediately.
To stop Ray workers from crashing immediately in Docker or Kubernetes, mount /dev/shm with at least 16 GB of memory (or use host IPC) and constrain Ray memory monitoring before starting the vLLM server:
# For standard Docker containers, set IPC to host or allocate 16GB to shared memory
docker run --gpus all --ipc=host \
-v /dev/shm:/dev/shm \
-e RAY_DISABLE_MEMORY_MONITOR=1 \
-e PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" \
-p 8000:8000 \
vllm/vllm-openai:latest \
--model Qwen/Qwen2.5-72B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.90 \
--max-model-len 8192If you run Ray bare-metal on Ubuntu Linux without Docker, start Ray with an explicit object store cap and disable aggressive node killing:
# Start Ray head node with constrained plasma memory
ray start --head --port=6379 \
--object-store-memory=4294967296 \
--disable-usage-stats
# Export critical allocator settings
export RAY_DISABLE_MEMORY_MONITOR=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
# Launch vLLM engine
python3 -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.92This prevents the Ray memory monitor from misinterpreting PyTorch memory caching as a memory leak and stops the Linux kernel from killing workers during NCCL handshakes.
Why Multi-GPU Tensor Parallelism Kills Ray Actors
vLLM distributes model weights across multiple GPUs using tensor parallelism. Each GPU runs an independent worker process managed by Ray. These processes communicate constantly through NVIDIA Collective Communications Library (NCCL) and PyTorch distributed backends.
Three distinct memory bottlenecks trigger the silent crash.
First, Docker defaults to a 64 MB /dev/shm shared memory mount. NCCL creates shared memory rings and IPC handles for peer-to-peer GPU transfers. When tensor parallelism exceeds 1, NCCL fills 64 MB within milliseconds during model weight loading. The kernel returns a Bus error (core dumped) or sends SIGKILL directly to the Ray worker.
Second, Ray includes an automated memory monitor daemon (raylet). By default, raylet monitors host memory usage. When total host memory exceeds 95%, Ray initiates an eviction cascade. It targets the process using the most memory. In a vLLM deployment, that process is the Ray worker holding model shards and CUDA unified memory pools. Ray terminates its own worker, triggering ray::RayWorkerWrapper.execute_method failure.
Third, the Ray plasma object store defaults to claiming 30% of total host RAM. If your host server has 128 GB of RAM, Ray locks 38.4 GB for internal object transfers. When PyTorch pre-allocates KV-cache structures and loads tokenizer tensors into system memory, total system RAM runs out. The Linux OOM killer (oom-killer) wakes up and kills the main vLLM process.
Diagnostic Checklist: VRAM OOM vs Host Memory OOM
Before changing flags, inspect the crash logs to pinpoint the exact failure mechanism.
| Symptom / Log Output | Root Cause | Affected Layer | Required Fix |
|---|---|---|---|
ray.exceptions.RayActorError: The actor died because of an OOM |
Ray memory monitor killed worker | Host RAM | Set RAY_DISABLE_MEMORY_MONITOR=1 |
RuntimeError: CUDA out of memory |
Model weights + KV cache exceed VRAM | GPU VRAM | Lower --gpu-memory-utilization to 0.85 |
Bus error (core dumped) in worker log |
/dev/shm partition filled to capacity |
Host IPC / shared memory | Pass --ipc=host or --shm-size=16g |
Process exited with code -9 without traceback |
Linux kernel OOM-killer killed process | Host RAM swap exhaustion | Add host swap or limit Ray object store |
NCCL error: unhandled system error / broken pipe |
Peer worker crashed silently mid-transfer | Network / NCCL | Fix worker OOM on secondary GPU node |
To verify if the Linux kernel killed your process, inspect the system kernel ring buffer immediately after a crash:
dmesg -T | grep -E -i "oom|killed process|vllm|ray"If you see an entry like Out of memory: Killed process 38412 (ray::RayWorker) with total-vm exceeding physical RAM, your host RAM is exhausted. If dmesg is empty and the error came from PyTorch, your GPU VRAM was exceeded.
Configuration Guide: Linux Sysctl, Docker, and PyTorch
To create a rock-solid multi-GPU serving environment, tune all three layers: Linux host kernel, container runtime, and Ray environment.
1. Linux Host Kernel Optimization
Add these memory allocation parameters to /etc/sysctl.d/99-vllm-ray.conf to accommodate high-bandwidth NCCL operations:
# /etc/sysctl.d/99-vllm-ray.conf
vm.max_map_count=1048576
vm.overcommit_memory=1
kernel.shmmax=68719476736
kernel.shmall=16777216Apply the changes without rebooting:
sudo sysctl -p /etc/sysctl.d/99-vllm-ray.confvm.max_map_count=1048576 allows PyTorch memory pools and CUDA drivers to map hundreds of thousands of distinct memory areas without throwing allocation faults. Setting vm.overcommit_memory=1 ensures memory allocations for large tensor buffers are not rejected prematurely by the Linux virtual memory manager.
2. Docker Compose Production Template
When running through Docker Compose, specify the shared memory size, IPC namespace, and memory limits explicitly:
version: '3.8'
services:
vllm-ray-node:
image: vllm/vllm-openai:latest
container_name: vllm_tensor_parallel
restart: unless-stopped
ipc: host
shm_size: '32gb'
environment:
- RAY_DISABLE_MEMORY_MONITOR=1
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
- NCCL_DEBUG=INFO
- NCCL_P2P_DISABLE=0
- NCCL_IB_DISABLE=1
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
- /root/.cache/huggingface:/root/.cache/huggingface
- /tmp/ray:/tmp/ray
ports:
- "8000:8000"
command: >
--model Qwen/Qwen2.5-72B-Instruct
--tensor-parallel-size 4
--gpu-memory-utilization 0.90
--max-model-len 8192
--enforce-eager3. Ray Multi-Node Cluster Configuration
When distributing across multiple physical hosts via InfiniBand or 100GbE RoCE, Ray head and worker nodes need matched shared memory configurations. Start each worker node with:
ray start --address='192.168.1.100:6379' \
--object-store-memory=8589934592 \
--memory=68719476736 \
--disable-usage-statsSpecifying --object-store-memory=8589934592 limits the plasma memory cache to 8 GB per node. This preserves remaining host RAM for KV-cache CPU offloading, model weight distribution, and operating system buffers.
Architectural Comparison: Unpatched Container vs. Tuned Ray + Linux Host
Comparing how each subsystem behaves during multi-GPU tensor-parallel inference before and after applying host IPC and Ray memory configuration changes:
| Subsystem | Default Docker & Ray Config | Tuned Linux + Ray Config | Operational Failure Prevented |
|---|---|---|---|
POSIX Shared Memory (/dev/shm) |
Capped at 64 MB inside container | --ipc=host or --shm-size=16g |
Prevents NCCL and Ray Plasma ring buffer SIGBUS / SIGKILL crashes |
| Ray Memory Monitor | Kills actors at 95% host RAM threshold | RAY_DISABLE_MEMORY_MONITOR=1 |
Prevents false-positive RayOutOfMemoryError during safetensors weight loading |
| CUDA Caching Allocator | Fixed non-expandable block segments | expandable_segments:True |
Prevents VRAM fragmentation OOMs during variable-length prefill bursts |
Kernel Virtual Memory (vm.max_map_count) |
65530 memory map areas |
1048576 memory map areas |
Prevents mmap() allocation failures across tensor-parallel workers |
Summary of Critical Parameters
Save this reference table for your deployment pipelines and Kubernetes helm charts.
| Configuration Flag | Scope | Default Value | Recommended Production Value |
|---|---|---|---|
--ipc=host |
Docker run flag | private (64MB) |
host or --shm-size=16g |
RAY_DISABLE_MEMORY_MONITOR |
Environment variable | 0 (Active) |
1 (Disabled for vLLM) |
PYTORCH_CUDA_ALLOC_CONF |
Environment variable | Unset | expandable_segments:True |
--object-store-memory |
Ray start flag | 30% of host RAM | 4294967296 (4 GB) to 8589934592 (8 GB) |
vm.max_map_count |
Linux sysctl | 65530 |
1048576 |
--gpu-memory-utilization |
vLLM CLI flag | 0.90 |
0.88 to 0.92 (Leaves 8-12% for CUDA context) |
VLLM_WORKER_MULTIPROC_METHOD |
Environment variable | fork or spawn |
spawn (Prevents CUDA re-initialization deadlocks) |
By setting these parameters systematically across your host operating system and container orchestration configs, Ray worker actor crashes disappear, allowing tensor-parallel vLLM instances to run at maximum performance.
Official Documentation & Primary References
- vLLM Official Documentation: Engine configuration arguments, PagedAttention KV cache sizing, and distributed serving guides.
- vLLM GitHub Repository: Upstream source code, AsyncLLMEngine architecture, and issue tracker.
- SGLang GitHub Repository: RadixAttention serving runtime and FlashInfer kernel integration specs.
- Hugging Face Hub Python Library Documentation: Environment variables, download timeout settings, and cache management on Linux.