SGLang v0.5.22 Linux Migration Guide: Fixing Python 3.11, CUDA 13, and DP Attention Breaking Changes

Run pip install --upgrade sglang on an Ubuntu 22.04 LTS server using the default system Python 3.10, and the package manager will exit with code 0 while silently pinning your environment to sglang==0.5.19. Pip does not warn you. The moment you pass a v0.5.22 argument such as --attn-dp-size or --moe-runner-backend flashinfer_megamoe, sglang.launch_server aborts with error: unrecognized arguments.

Upgrading an SGLang RadixAttention serving cluster across the v0.5.20 through v0.5.22 release cycle introduces four breaking changes on Linux: a hard Requires-Python >= 3.11 metadata floor, the retirement of prebuilt CUDA 12 wheels (-cu12x), a structural split between --dp-size and --attn-dp-size, and the removal of legacy DeepEP waterfill flags. This guide covers the exact diagnosis and migration steps required to update systemd units and Docker deployments cleanly.

Summary of SGLang v0.5.20 to v0.5.22 Breaking Changes

Before modifying launch scripts, compare the runtime requirements and CLI flags across the v0.5.19 boundary as documented in the official SGLang GitHub release notes:

1. Breaking Change #1: Python 3.10 Silent Version Pinning

Ubuntu 22.04 LTS (jammy) ships /usr/bin/python3 at version 3.10.12. Because SGLang v0.5.20+ declares Requires-Python: >=3.11 in its PyPI package metadata, running pip install --upgrade sglang inside a Python 3.10 virtual environment does not raise a dependency conflict. Instead, pip’s resolver filters out every wheel from 0.5.20 onward and reports that 0.5.19 is already up to date.

Verify whether your environment was silently pinned by enforcing an explicit minimum version bound:

python3 -m pip install "sglang[all]>=0.5.22"

On Python 3.10, pip immediately reveals the constraint:

ERROR: Ignored the following versions that require a different python version: 0.5.20 Requires-Python >=3.11; 0.5.21 Requires-Python >=3.11; 0.5.22 Requires-Python >=3.11
ERROR: Could not find a version that satisfies the requirement sglang>=0.5.22 (from versions: 0.1.0, ..., 0.5.19)

Fix: Provision an Isolated Python 3.12 Runtime with uv

Never overwrite /usr/bin/python3 on Ubuntu 22.04, as apt and netplan depend on system Python 3.10 modules. Use Astral’s uv to provision a standalone Python 3.12 virtual environment alongside matching sgl-kernel wheels:

# Install uv if not already present on the host
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create an isolated Python 3.12 virtual environment
uv venv /opt/sglang-env --python 3.12
source /opt/sglang-env/bin/activate

# Install SGLang v0.5.22+ with all serving and kernel dependencies
uv pip install --upgrade "sglang[all]>=0.5.22"
python -c "import sglang; print('Loaded SGLang version:', sglang.__version__)"

2. Breaking Change #2: CUDA 12 Lane Retirement and CUDA 13 Baselines

In the SGLang release pipeline, v0.5.19 was the final milestone publishing -cu12x wheels and CUDA 12 container tags. Starting with v0.5.20 and v0.5.22, default wheels (sglang and sgl-kernel) and container images (lmsysorg/sglang:v0.5.22) are compiled against CUDA 13 and the updated PyTorch CUDA 13 runtime.

If your Linux host runs an older NVIDIA R535 or R550 data-center driver without CUDA 13 forward-compatibility libraries, starting the server triggers an immediate initialization failure or a FlashInfer JIT compilation error:

RuntimeError: CUDA error: system has unsupported display driver / cuda driver combination (cudaErrorInsufficientDriver)

Upgrade the host driver first. On Ubuntu 22.04 or 24.04 servers running Ampere (sm_80), Ada (sm_89), Hopper (sm_90), or Blackwell (sm_100) GPUs, install the R570+ driver branch or deploy the official lmsysorg/sglang:v0.5.22 container image, which bundles the user-space CUDA 13 toolkit libraries:

# Check host driver version (require >= 570.xx for native CUDA 13 runtime support)
nvidia-smi --query-gpu=driver_version,compute_cap --format=csv

# Pull the official CUDA 13 SGLang v0.5.22 container image
docker pull lmsysorg/sglang:v0.5.22

If you are locked to a legacy Pascal (sm_61) or Volta (sm_70) card that CUDA 13 drops at the compiler level, explicitly pin your environment to sglang==0.5.19 rather than pulling v0.5.22.

3. Breaking Change #3: --enable-dp-attention Replaced by --attn-dp-size

This is the most critical CLI breaking change for multi-GPU MoE deployments such as DeepSeek-V3, DeepSeek-R1, and DeepSeek-V4.

As detailed in our LLM VRAM & KV Cache Sizing Guide, Multi-Head Latent Attention (MLA) compresses the Key-Value cache into a single shared latent vector ($d_c = 512$, costing $68.63\text{ KiB/token}$ in BF16). Under standard Tensor Parallelism (--tp-size 8), that latent KV cache is replicated across all 8 GPUs because a single shared latent head cannot be partitioned across TP ranks without all-reduce overhead.

To eliminate latent cache duplication across an 8-GPU node ($T_p = 8$), SGLang introduced Data-Parallel Attention (DP Attention), which partitions incoming batch sequences across attention ranks while keeping the routed MoE expert weights sharded across all 8 GPUs:

$$\text{Effective KV Capacity}{\text{DP-Attn}} = N{\text{attn-dp}} \times M_{\text{KV per GPU}}$$

In v0.5.19 and earlier, engineers enabled this by combining --tp-size 8 --dp-size 8 --enable-dp-attention. That overloaded --dp-size to mean two completely different things depending on whether --enable-dp-attention was present.

In v0.5.22, the SGLang server argument parser strictly separates pure model replication from attention data parallelism:

• --dp-size <N>: Counts pure, independent model replicas. Setting --tp-size 8 --dp-size 2 now requires $8 \times 2 = 16$ physical GPUs.
• --attn-dp-size <N>: Partitions the attention layer across $N$ data-parallel shards within the existing --tp-size GPU group ($N$ must divide --tp-size evenly).
• --enable-waterfill: Replaces the removed --enable-deepep-waterfill flag to balance token dispatch loads across DP attention ranks before DeepEP all-to-all routing. That old flag is gone.

Diff: Migrating an 8-GPU Launch Script to v0.5.22

Remove --enable-dp-attention and --enable-deepep-waterfill from your launch scripts and replace them with --attn-dp-size and --enable-waterfill:

 python3 -m sglang.launch_server \
   --model-path deepseek-ai/DeepSeek-R1 \
   --tp-size 8 \
-  --dp-size 8 \
-  --enable-dp-attention \
-  --enable-deepep-waterfill \
+  --attn-dp-size 8 \
+  --enable-waterfill \
+  --moe-runner-backend flashinfer_megamoe \
   --trust-remote-code \
   --host 0.0.0.0 \
   --port 30000

Notice the addition of --moe-runner-backend flashinfer_megamoe, introduced in the October 2026 release cycle to fuse MoE expert dispatch and group GEMM execution via FlashInfer.

4. Pre-Upgrade Audit Script for Systemd and Docker Configs

Before restarting production services, run the following Python audit script against your deployment directory (/etc/systemd/system or your Kubernetes/Docker Compose manifests) to catch deprecated SGLang flags before they trigger a crash loop:

#!/usr/bin/env python3
"""
audit_sglang_flags.py - Scan Linux service files and scripts for SGLang v0.5.22 breaking flags.
"""
import re
import sys
from pathlib import Path

BREAKING_PATTERNS = [
    (
        re.compile(r"--enable-deepep-waterfill\b"),
        "REMOVED in v0.5.22: Replace '--enable-deepep-waterfill' with '--enable-waterfill'.",
    ),
    (
        re.compile(r"--enable-dp-attention\b"),
        "DEPRECATED in v0.5.22: Remove '--enable-dp-attention' and '--dp-size', and use '--attn-dp-size <N>' instead.",
    ),
    (
        re.compile(r"\bnum_tokens_per_bs\b"),
        "RENAMED in v0.5.22: Replace 'num_tokens_per_bs' with 'num_tokens_per_req' in speculative decoding configs.",
    ),
]

def scan_file(filepath: Path) -> int:
    try:
        content = filepath.read_text(encoding="utf-8", errors="ignore")
    except OSError:
        return 0

    hits = 0
    for line_no, line in enumerate(content.splitlines(), start=1):
        for pattern, advice in BREAKING_PATTERNS:
            if pattern.search(line):
                print(f"[FAIL] {filepath}:{line_no}: {advice}")
                print(f"       Line: {line.strip()}")
                hits += 1
    return hits

def main():
    target_dir = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(".")
    exts = {".service", ".sh", ".yaml", ".yml", ".py", ".env"}
    total_hits = 0

    for path in target_dir.rglob("*"):
        if path.is_file() and path.suffix in exts:
            total_hits += scan_file(path)

    if total_hits > 0:
        print(f"\nFound {total_hits} breaking SGLang v0.5.22 configuration issue(s).")
        sys.exit(1)
    print("All scanned files are compatible with SGLang v0.5.22+ CLI flags.")

if __name__ == "__main__":
    main()

Run the check against your deployment repository or /etc/systemd/system:

python3 audit_sglang_flags.py /etc/systemd/system

Official References & Release Documentation

• SGLang Official GitHub Repository (sgl-project/sglang): Source code for the SGLang runtime, sgl-kernel, and RadixAttention engine.
• SGLang Releases & Changelog (v0.5.20 - v0.5.22): Primary changelog documenting the Python 3.11 floor, CUDA 12 retirement, and --attn-dp-size refactor.
• SGLang Server Arguments Documentation: Official reference for --tp-size, --dp-size, --attn-dp-size, and MoE runner backends.
• FlashInfer GitHub Repository (flashinfer-ai/flashinfer): Kernel library powering the flashinfer_megamoe backend.