How to Fix 'CUDA Setup failed despite GPU being available' in bitsandbytes on Linux
To resolve CUDA Setup failed despite GPU being available or The installed version of bitsandbytes was compiled without GPU support on Linux, export the CUDA toolkit library path and reinstall the official pre-compiled multi-backend wheel:
That stalls the pipeline. Execution halts immediately.
# 1. Export CUDA library directories into dynamic linker path
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
# 2. Reinstall the latest bitsandbytes wheel supporting CUDA 11.x and 12.x
pip uninstall -y bitsandbytes
pip install --upgrade bitsandbytes
# 3. Verify CUDA detection immediately
python3 -m bitsandbytesIf the diagnostic report prints SUCCESS: CUDA detected and compiled with GPU support, your environment is repaired.
Why bitsandbytes Fails to Detect the GPU
The bitsandbytes library provides 8-bit optimizers and 4-bit NormalFloat4 (NF4) quantization for LLMs. Unlike standard PyTorch wheels that bundle their own CUDA runtime binaries (libcudart), bitsandbytes dynamically links against the host system’s CUDA driver and toolkit shared objects (libcuda.so and libcudart.so) at import time.
When the dynamic linker cannot find matching libraries, bitsandbytes falls back to CPU-only execution and outputs the following error trace:
===================================BUG REPORT===================================
/usr/local/lib/python3.12/site-packages/bitsandbytes/cuda_setup/main.py:166:
UserWarning: The installed version of bitsandbytes was compiled without GPU support.
CUDA Setup failed despite GPU being available. Inspect the following to identify the issue:
1. The installed version of bitsandbytes was compiled without GPU support.
2. CUDA driver not found or LD_LIBRARY_PATH not configured.
3. GPU compute capability not supported.
================================================================================
RuntimeError: CUDA Setup failed despite GPU being available.+-----------------------------------------------------------------------------------------+
| BITSANDBYTES DYNAMIC LINKING FAILURE ANATOMY |
+-----------------------------------------------------------------------------------------+
| Python Process: import bitsandbytes |
| | |
| v |
| Search LD_LIBRARY_PATH for: libcuda.so.1 & libcudart.so |
| | |
| +---> /usr/local/cuda/lib64 (MISSING OR NOT EXPORTED) |
| +---> /usr/lib/x86_64-linux-gnu (DRIVER STUB ONLY / MISSING SYMLINK) |
| | |
| v |
| [Dynamic Linker Fails] -> bitsandbytes activates libbitsandbytes_cpu.so fallback |
| [Kernel Call] ----------> RuntimeError: CUDA Setup failed |
+-----------------------------------------------------------------------------------------+Four distinct configuration bugs cause this crash:
- Unset
LD_LIBRARY_PATH: Python runs inside a virtual environment where system CUDA directories are not included in library paths. - Missing
libcuda.so.1in Docker: Headless container runtimes mounted withoutNVIDIA_DRIVER_CAPABILITIES=compute,utilityomit the driver interface. - Architecture Mismatch: Outdated wheels lack support for modern microarchitectures like Ada Lovelace or Hopper.
- PyTorch CUDA Mismatch: PyTorch was compiled with CUDA 12.4 while the system toolkit is CUDA 11.8.
Step-by-Step Diagnostic and Repair
Step 1: Run the Built-in Diagnostic Tool
Run the official diagnostic tool to inspect where the linker searches:
python3 -m bitsandbytesReview the printed paths. Look at the Found duplicate CUDA runtime and CUDA Version detected fields. If the output states CUDA not found, proceed to Step 2.
Step 2: Configure System CUDA Paths
Locate your CUDA installation on Linux. Typical locations are /usr/local/cuda, /usr/local/cuda-12.4, or /usr/local/cuda-12.1.
# Check if CUDA toolkit is installed
ls -d /usr/local/cuda*Add the paths to your shell profile (~/.bashrc or ~/.zshrc):
# Append to ~/.bashrc
export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
# Reload current shell environment
source ~/.bashrcVerify that the linker can now locate libcudart.so:
ldconfig -p | grep -E "libcudart|libcuda.so"Step 3: Fix Docker Missing Driver Symlinks
Inside Docker containers, the most frequent reason for this error is a missing libcuda.so.1 symlink in /usr/lib/x86_64-linux-gnu.
If running containerized workloads, check if the driver library exists:
ls -l /usr/lib/x86_64-linux-gnu/libcuda.so*If libcuda.so.1 is present but libcuda.so is missing, create the symbolic link:
sudo ln -s /usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/x86_64-linux-gnu/libcuda.so
sudo ldconfigWhen launching Docker containers, verify that the NVIDIA container toolkit passes the compute capability:
docker run --gpus all \
-e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
-e NVIDIA_VISIBLE_DEVICES=all \
--ipc=host \
your-ai-image:latestStep 4: Install the Modern Multi-Backend Wheel
Starting with bitsandbytes version 0.43.0 and newer, the library discontinued separate version-locked wheels in favor of a universal multi-backend binary. It automatically dispatches kernels matching your runtime CUDA version.
Install the latest release:
pip uninstall -y bitsandbytes
pip install --no-cache-dir "bitsandbytes>=0.43.3"If you operate on older systems with CUDA 11.8 or custom ROCm/AMD drivers, you can compile bitsandbytes from source directly:
# Compile bitsandbytes from source with CMake
git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git
cd bitsandbytes
pip install -r requirements.txt
cmake -DCOMPUTE_BACKEND=cuda -S . -B build
cmake --build build -j$(nproc)
pip install .Verifying 4-Bit Model Loading in PyTorch
Once the dynamic libraries are repaired, test model quantization in Python using Hugging Face Transformers.
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
# 1. Define 4-bit NormalFloat quantization configuration
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True
)
print(f"CUDA Available: {torch.cuda.is_available()}")
print(f"Active GPU: {torch.cuda.get_device_name(0)}")
# 2. Test 4-bit linear layer quantization
from bitsandbytes.nn import Linear4bit
test_layer = Linear4bit(256, 128, bias=False, compute_dtype=torch.bfloat16)
test_layer = test_layer.cuda()
test_input = torch.randn(2, 256, dtype=torch.bfloat16, device="cuda")
with torch.no_grad():
output = test_layer(test_input)
print(f"Quantized layer test passed! Output shape: {output.shape}")If the script executes without throwing warnings or segmentation faults, your environment is ready for production 4-bit fine-tuning and inference.
Troubleshooting Matrix
The following table summarizes common bitsandbytes symptoms and their direct resolutions:
| Symptom / Error Message | Root Cause | Direct Resolution |
|---|---|---|
UserWarning: compiled without GPU support |
Outdated package or dynamic linker path missing | Export LD_LIBRARY_PATH and upgrade to bitsandbytes>=0.43.3 |
libbitsandbytes_cuda*.so: cannot open shared object file |
System CUDA version mismatch | Compile from source via cmake -DCOMPUTE_BACKEND=cuda |
CUDA Setup failed in Docker container |
Missing driver capability flags | Add -e NVIDIA_DRIVER_CAPABILITIES=compute,utility to Docker CLI |
ValueError: bfloat16 compute_dtype unsupported |
GPU architecture lacks native BF16 units | Switch bnb_4bit_compute_dtype=torch.float16 for Turing/Pascal GPUs |
CUDA error: out of memory during quantization |
Tensor fragmentation during layer weights conversion | Set export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" |
If your GPU throws compute capability errors after resolving the CUDA setup, check our guide on fixing bfloat16 compute capability errors on Linux.
For broader memory allocator tuning during 4-bit model training, refer to our walkthrough on fixing PyTorch cuBLAS allocation failed errors.
Official Documentation & Primary References
- PyTorch Official CUDA Semantics Documentation: Memory management, device ordinals, asynchronous execution, and stream synchronization.
- PyTorch Autograd Mechanics: Computational graph lifecycle, in-place operation checks, and backward pass execution.
- PyTorch Distributed (c10d / DDP) Documentation: NCCL backend initialization, watchdog timeouts, and gradient reduction hooks.
- NVIDIA CUDA Toolkit & Driver Documentation: Official compute capability tables, cuBLAS/cuDNN status codes, and NVML driver management.