How to Fix CUDA Illegal Memory Access Errors in vLLM PagedAttention on Linux
RuntimeError: CUDA error: an illegal memory access was encountered is a serious signal in a running vLLM worker, but it is not a license to reset every GPU on the host. CUDA calls execute asynchronously, so the Python frame reporting the error may be later than the kernel that actually failed.
An affected worker’s CUDA context may no longer be usable for subsequent operations. Restarting that worker after diagnosis may be appropriate. That is different from saying the entire GPU driver is permanently damaged. Start with an incident log, not a destructive cleanup.
Contain the Failure Without Killing Other GPU Jobs
Preserve the first traceback, vLLM/PyTorch/CUDA package versions, model ID, input shape, worker ID and device ID. Check which GPUs are in use. The following commands are read-only:
nvidia-smi
python3 -m pip show vllm torch triton
# If the service is managed by systemd, use its actual unit name:
# journalctl --user -u YOUR_VLLM_UNIT --since "30 minutes ago" --no-pagerDo not run sudo fuser -k /dev/nvidia*, nvidia-smi --gpu-reset, or a host-wide cache deletion as a first step. On a shared GPU node, they can disrupt training runs or unrelated inference jobs.
If you need to restart, identify and stop only the affected service according to its owner-approved maintenance procedure. Collect the first failing request or a safe minimal reproduction where privacy permits.
Reproduce on a Test Worker, Not the Production Endpoint
PyTorch documents that setting CUDA_LAUNCH_BLOCKING=1 makes CUDA calls synchronous so failures can be attributed more closely to the actual operation. vLLM recommends --enforce-eager to investigate errors around CUDA Graph replay. These settings are debugging aids, not universal fixes. Expect performance overhead.
# Run ONLY on a designated test GPU/host with the model already available.
# Replace MODEL_ID with your actual tested model and use a private port.
CUDA_LAUNCH_BLOCKING=1 VLLM_LOGGING_LEVEL=DEBUG \
vllm serve MODEL_ID \
--host 127.0.0.1 --port 8001 --enforce-eagerKeep model, driver, vLLM release, request lengths and batch conditions fixed between tests. Compare the same request with and without eager mode in separate fresh worker processes. Eager mode removing the symptom suggests a CUDA Graph interaction; it does not prove a pointer overflow.
After reproducing, remove debugging variables from the production environment. Do not leave CUDA_LAUNCH_BLOCKING=1 enabled for latency-sensitive serving.
Separate Observations from Suspected Causes
| Evidence from Logs or Reproduction | Hypothesis Worth Testing | Appropriate Investigation |
|---|---|---|
First failure near graph.replay() |
CUDA Graph capture/replay compatibility | Compare an isolated fresh worker with --enforce-eager |
| Error persists in eager mode | Attention kernel, extension or other CUDA operation | Reduce reproduction to a minimal shape; align package versions |
| Failure appears after upgrading vLLM/PyTorch/CUDA | Runtime or compiled-extension incompatibility | Compare installed versions with official compatibility notes |
| Failure appears only at particular batch lengths | Shape-dependent kernel bug or request-triggered edge case | Capture lengths, batch sizes, backend and reproducible input |
| NCCL error follows another worker’s illegal address | Possible secondary distributed failure | Identify first failing rank before tuning NCCL |
These are hypotheses, not one-to-one error-code diagnoses. In particular, arbitrary prompt lengths that are not multiples of the PagedAttention block size are normal; they do not by themselves demonstrate an out-of-bounds read. Do not claim a stale Triton cache, a RoPE overflow, or a specific CUDA Graph pointer fault without a versioned reproducer or upstream issue.
See vLLM troubleshooting and PyTorch CUDA semantics for the current diagnostic primitives.
Check the Build and Attention Backend
- Record
vllm --version,python3 -m pip show torch triton,nvidia-smi, and the container image digest. - Verify that driver and CUDA wheel combinations are supported by that vLLM release. Architecture-specific compiled kernels should match the GPU actually used.
- Check the attention backend selected by the running version rather than assuming
FLASH_ATTNis valid for every model and CUDA architecture. - If the bug reproduces only after a version change, reproduce on a clean, pinned test environment before updating the production stack.
- Clear only the affected worker’s compilation cache if logs and release notes indicate a stale artifact. Back up the path and confirm no other worker shares it. Never delete user-wide caches indiscriminately.
The TORCH_CUDA_ARCH_LIST environment variable controls builds of certain PyTorch CUDA extensions. Setting it on a command line does not magically repair a previously compiled binary. Likewise, --block-size 16 changes allocation granularity but does not guarantee a fix for illegal memory access.
Production Recovery Decision
| Situation | Next Action | Why |
|---|---|---|
| Isolated worker has an unrecoverable CUDA context | Drain its traffic and restart that worker in a maintenance-safe way | Avoid touching unrelated GPU jobs |
| Multiple workers report the same reproducible kernel failure | Pin a known-good environment or roll back the affected version | Focus on compatibility, not blind resets |
| Host-wide NVIDIA errors appear independently of vLLM | Escalate to the platform administrator with logs | Driver issues need host ownership and a separate procedure |
| Debug run with eager mode succeeds | Investigate upstream CUDA Graph issues and compare latency | Debug success is not a durable fix |
Do not expose the temporary test server on 0.0.0.0 without authentication and network controls. Prefer a private loopback bind for local diagnosis. Avoid --ipc=host and unnecessary device mounts as blanket Docker recommendations; select the minimum privileges needed by your documented runtime.
No production GPU workload, kernel module, NVIDIA service, or model was executed or reset for this technical correction. Example flags should be checked against the version actually installed.
Official References
- vLLM troubleshooting and CUDA Graph debugging
- vLLM configuration and optimization
- PyTorch CUDA semantics, asynchronous execution
- PyTorch CUDA environment variables
Technical review: 2026-11-03. The original host-wide GPU kill/reset recipe and unverified root-cause matrix were removed.