Multiply a 32-billion parameter model by $0.5\text{ bytes}$ for 4-bit quantization, assume $16\text{ GB}$ of VRAM is plenty, and your vLLM inference server will still crash with torch.cuda.OutOfMemoryError before serving a single token on a $24\text{ GB}$ Linux GPU. Back-of-the-napkin math fails in production.

Total GPU memory consumption during autoregressive LLM inference is governed by four distinct allocations: static model weights (including quantization scale metadata), dynamic Key-Value (KV) attention cache blocks, transient forward-pass activation buffers, and the non-negotiable NVIDIA CUDA runtime context. Ignoring any one of these four terms guarantees either an immediate startup crash or severe KV cache preemption thrashing under concurrent multi-agent load.

This reference guide breaks down the exact byte-level equations for modern transformer architectures (Multi-Head Attention, Grouped-Query Attention, and DeepSeek’s Multi-Head Latent Attention) alongside a standalone Python sizing tool you can run against any Hugging Face config.json on Linux.

The Master GPU Memory Equation

At any point during active serving on a Linux host, total allocated VRAM ($M_{\text{total}}$) across a tensor-parallel group of $T_p$ GPUs follows:

$$M_{\text{total}} = M_{\text{weights}} + M_{\text{KV}}(B, S) + M_{\text{activations}}(B_{\text{prefill}}) + M_{\text{CUDA}}$$

Where:

• $M_{\text{weights}}$: Static model parameters plus quantization scale and zero-point tensors, divided across $T_p$ GPUs.
• $M_{\text{KV}}(B, S)$: Dynamic PagedAttention KV cache blocks scaled by concurrent batch size $B$ and sequence length $S$.
• $M_{\text{activations}}(B_{\text{prefill}})$: Temporary scratchpad buffers sized to --max-num-batched-tokens during chunked prefill, plus CUDA Graph replay pools.
• $M_{\text{CUDA}}$: Fixed per-process CUDA driver context, cuBLAS handles, and NCCL ring buffers ($\approx 0.8\text{ GiB}$ to $1.6\text{ GiB}$ per GPU).

1. Exact Model Weight Math ($M_{\text{weights}}$): Why 4-Bit Is Actually 4.5 Bits

Engineers routinely estimate 4-bit quantized weights (INT4) at $0.5\text{ bytes}$ per parameter. Scales are not free.

Block-wise quantization formats such as Activation-aware Weight Quantization (AWQ) and GPTQ partition weight matrices into contiguous groups, almost universally configured with group_size = 128 (g128). For every 128 four-bit weights, the format stores one 16-bit (FP16) scaling factor and one 4-bit zero-point integer:

$$\text{Bits per Parameter}_{\text{AWQ-g128}} = 4 + \frac{16\text{ (scale)} + 4\text{ (zero-point)}}{128} = 4 + 0.15625 = 4.15625\text{ bits}$$

Furthermore, the input token embedding table (embed_tokens), the final RMSNorm weights, and the output language modeling head (lm_head) are frequently left in unquantized 16-bit precision (BF16/FP16) or 8-bit precision to prevent perplexity collapse. For a vocabulary size $V = 152,064$ and hidden dimension $d_{\text{model}} = 5,120$ (the exact dimensions of Qwen2.5-Coder-32B), an unquantized embed_tokens + lm_head pair alone consumes:

$$M_{\text{embed+head}} = 2 \times 152,064 \times 5,120 \times 2\text{ bytes} = 3,114,270,720\text{ bytes} \approx 2.90\text{ GiB}$$

Comparing effective bytes per parameter across standard serving precision formats:

Precision Format	Weight Precision	Group Metadata Overhead	Unquantized Head/Norm Overhead	Effective Multiplier ($b_{\text{eff}}$)	32.8B Model Footprint
BF16 / FP16	16-bit ($2.0\text{ B}$)	None	Included ($2.0\text{ B}$)	$2.00\text{ B / param}$	$61.1\text{ GiB}$
FP8 (E4M3)	8-bit ($1.0\text{ B}$)	Per-tensor or per-channel FP32 scale	embed_tokens in BF16	$1.05\text{ B / param}$	$32.1\text{ GiB}$
AWQ / GPTQ (INT4 g128)	4-bit ($0.5\text{ B}$)	$+0.156\text{ bits/param}$ (g128)	lm_head + norms in FP16	$0.56\text{ B / param}$	$17.1\text{ GiB}$
GGUF (Q4_K_M)	4-bit super-blocks	6-bit scales + Q6_K attention wv/wo	output.weight in Q6_K	$0.60\text{ B / param}$	$18.3\text{ GiB}$

On a $24\text{ GiB}$ GPU (such as an RTX 3090, RTX 4090, or NVIDIA L4), loading Qwen2.5-Coder-32B-Instruct-AWQ consumes $17.1\text{ GiB}$ for weights plus $\approx 1.2\text{ GiB}$ for the CUDA context and Triton kernels. That leaves only $5.7\text{ GiB}$ of physical HBM for the KV cache, rather than the $8\text{ GiB}$ implied by naive $32\times 0.5$ arithmetic.

2. KV Cache Mathematics: MHA vs. GQA vs. DeepSeek MLA

Every token generated or ingested during a prompt prefill adds its Key and Value projection vectors to GPU memory across all $L$ transformer layers. Every token costs physical HBM.

Standard Multi-Head (MHA) and Grouped-Query Attention (GQA)

In both MHA and GQA architectures (used by Llama 3.1, Llama 3.3, Qwen 2.5, Qwen 3, and Mistral), each layer caches one Key vector and one Value vector per KV head ($n_{\text{kv}}$). Read the following four fields directly from the model’s config.json:

• $L$: num_hidden_layers
• $n_{\text{kv}}$: num_key_value_heads
• $d_{\text{head}}$: head_dim (or hidden_size / num_attention_heads if head_dim is omitted)
• $b_{\text{kv}}$: Bytes per element ($2$ for bfloat16/float16, $1$ for fp8_e4m3 via --kv-cache-dtype fp8, or $0.5$ for NVFP4 on Blackwell GPUs)

The exact KV cache footprint per token for GQA is:

$$\text{Bytes}{\text{token}}^{\text{GQA}} = 2 \times L \times n{\text{kv}} \times d_{\text{head}} \times b_{\text{kv}}$$

Wait: the factor of $2$ accounts for storing both the Key tensor and the Value tensor. In GQA, $n_{\text{kv}}$ is smaller than the query head count $n_q$ (typically by a ratio of $4:1$ or $8:1$), shrinking KV memory proportionally without altering $d_{\text{head}}$.

DeepSeek Multi-Head Latent Attention (MLA)

DeepSeek-V2, DeepSeek-V3, and DeepSeek-R1 replace standard KV heads with Multi-Head Latent Attention (MLA). Instead of caching separate Key and Value projections across $n_{\text{kv}}$ heads, MLA projects both Keys and Values into a single shared low-rank compressed latent vector ($c_t^{KV} \in \mathbb{R}^{d_c}$) alongside a decoupled Rotary Position Embedding key vector ($k_t^{R} \in \mathbb{R}^{d_R}$).

In a DeepSeek config.json, inspect three specific keys:

• $L$: num_hidden_layers ($61$ in DeepSeek-V3 and DeepSeek-R1)
• $d_c$: kv_lora_rank ($512$)
• $d_R$: qk_rope_head_dim ($64$)

Because the up-projection matrices $W^{UK}$ and $W^{UV}$ are absorbed into the query and output projections during inference (as implemented in FlashMLA and vLLM PagedAttention), the inference engine only caches $d_c + d_R$ elements per layer per token, with no factor of 2 and no multiplication by head count:

$$\text{Bytes}{\text{token}}^{\text{MLA}} = L \times \left( d_c + d_R \right) \times b{\text{kv}}$$

Substituting the official config.json parameters for deepseek-ai/DeepSeek-R1 in 16-bitKV precision ($b_{\text{kv}} = 2$):

$$\text{Bytes}_{\text{token}}^{\text{DeepSeek-R1}} = 61 \times (512 + 64) \times 2\text{ bytes} = 61 \times 576 \times 2 = 70,272\text{ bytes} \approx 68.63\text{ KiB / token}$$

Look at the architectural consequence of that equation. A 671-billion parameter DeepSeek-R1 model consumes less than a quarter of the KV cache per token ($68.63\text{ KiB}$) of a 70-billion parameter Llama-3.1-70B model ($320.00\text{ KiB}$):

Model Architecture	Layers ($L$)	KV Heads ($n_{\text{kv}}$)	Head / Latent Dim	BF16 KV / Token	FP8 KV / Token	32k Context KV Size (BF16)
Llama-3.1-8B (GQA)	$32$	$8$	$d_{\text{head}} = 128$	$128.00\text{ KiB}$	$64.00\text{ KiB}$	$4.00\text{ GiB}$
Qwen2.5-Coder-32B (GQA)	$64$	$8$	$d_{\text{head}} = 128$	$256.00\text{ KiB}$	$128.00\text{ KiB}$	$8.00\text{ GiB}$
Llama-3.3-70B (GQA)	$80$	$8$	$d_{\text{head}} = 128$	$320.00\text{ KiB}$	$160.00\text{ KiB}$	$10.00\text{ GiB}$
DeepSeek-R1-671B (MLA)	$61$	Shared Latent	$d_c=512, d_R=64$	$68.63\text{ KiB}$	$34.31\text{ KiB}$	$2.14\text{ GiB}$

3. Tensor Parallelism and KV Head Divisibility Constraints

When sharding a model across $T_p$ GPUs via --tensor-parallel-size, vLLM partitions the weight matrices and the KV heads evenly across ranks. For standard GQA models, so long as $T_p \le n_{\text{kv}}$ and $n_{\text{kv}} \pmod{T_p} = 0$, each GPU holds $n_{\text{kv}} / T_p$ KV heads, dividing both weight memory and KV cache memory by $T_p$.

What happens if you deploy an $n_{\text{kv}} = 4$ or $n_{\text{kv}} = 8$ model across $T_p = 16$ GPUs, or serve DeepSeek MLA across multiple GPUs?

• GQA Replication ($T_p > n_{\text{kv}}$): A single KV head cannot be split across devices without incurring all-reduce communication on every attention step. Instead, KV heads are replicated across rank groups, meaning KV cache memory stops shrinking linearly once $T_p > n_{\text{kv}}$.
• MLA Latent Replication: Because DeepSeek’s compressed latent vector ($d_c = 512$) is shared across all 128 query heads, standard tensor-parallel execution replicates the $68.63\text{ KiB/token}$ latent cache on each TP rank unless Data Parallel Attention (--enable-dp-attention in SGLang or vLLM) is enabled. Even when replicated, $68.63\text{ KiB/token}$ is so compact that per-rank memory remains smaller than sharded GQA.

4. Reproducible Linux CLI: Hugging Face config.json VRAM Calculator

Never guess whether a model and context window will fit on your Linux GPU node. Save the following standalone Python script as vram_calculator.py. It fetches the exact config.json from Hugging Face (or reads a local directory), automatically detects GQA versus MLA architectures, accounts for group-quantization metadata and CUDA context overhead, and outputs the maximum safe --max-model-len and concurrent batch capacity:

#!/usr/bin/env python3
"""
vram_calculator.py - Exact LLM Weight & KV Cache VRAM Sizing Tool for Linux GPUs
Supports MHA, GQA (Llama/Qwen/Mistral), and MLA (DeepSeek-V2/V3/R1).
"""
import argparse
import json
import urllib.request
from pathlib import Path

BYTES_PER_PARAM = {
    "bf16": 2.00,
    "fp16": 2.00,
    "fp8": 1.05,      # Includes per-channel scale + BF16 embed_tokens
    "int8": 1.08,
    "awq-int4": 0.56, # 4-bit weights + g128 FP16 scales + unquantized lm_head
    "gptq-int4": 0.56,
    "gguf-q4km": 0.60,
}

KV_DTYPE_BYTES = {
    "bf16": 2.0,
    "fp16": 2.0,
    "fp8": 1.0,
    "fp4": 0.5,
}

def load_config(model_id_or_path: str) -> dict:
    local_path = Path(model_id_or_path)
    if local_path.is_file():
        return json.loads(local_path.read_text(encoding="utf-8"))
    if (local_path / "config.json").is_file():
        return json.loads((local_path / "config.json").read_text(encoding="utf-8"))

    url = f"https://huggingface.co/{model_id_or_path}/raw/main/config.json"
    req = urllib.request.Request(url, headers={"User-Agent": "vram-calculator/1.0"})
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read().decode("utf-8"))

def calculate_kv_bytes_per_token(cfg: dict, kv_bytes: float) -> tuple[float, str]:
    layers = cfg["num_hidden_layers"]
    # Detect DeepSeek Multi-Head Latent Attention (MLA)
    if "kv_lora_rank" in cfg and cfg.get("kv_lora_rank"):
        d_c = cfg["kv_lora_rank"]
        d_r = cfg.get("qk_rope_head_dim", 64)
        bytes_per_tok = layers * (d_c + d_r) * kv_bytes
        return bytes_per_tok, f"MLA (L={layers}, d_c={d_c}, d_R={d_r})"

    # Standard MHA / GQA
    n_heads = cfg["num_attention_heads"]
    n_kv = cfg.get("num_key_value_heads", n_heads)
    head_dim = cfg.get("head_dim", cfg["hidden_size"] // n_heads)
    bytes_per_tok = 2 * layers * n_kv * head_dim * kv_bytes
    arch_type = "MHA" if n_kv == n_heads else f"GQA ({n_heads}Q:{n_kv}KV)"
    return bytes_per_tok, f"{arch_type} (L={layers}, n_kv={n_kv}, d_head={head_dim})"

def main():
    parser = argparse.ArgumentParser(description="Calculate exact LLM VRAM and KV cache capacity.")
    parser.add_argument("--model", required=True, help="Hugging Face repo ID or local path to config.json")
    parser.add_argument("--params-b", type=float, required=True, help="Total model parameters in billions (e.g., 32.8)")
    parser.add_argument("--quant", choices=list(BYTES_PER_PARAM.keys()), default="awq-int4")
    parser.add_argument("--kv-dtype", choices=list(KV_DTYPE_BYTES.keys()), default="bf16")
    parser.add_argument("--gpu-vram-gb", type=float, default=24.0, help="VRAM per GPU in GiB")
    parser.add_argument("--tp", type=int, default=1, help="Tensor parallel size (number of GPUs)")
    parser.add_argument("--gpu-util", type=float, default=0.90, help="vLLM --gpu-memory-utilization")
    parser.add_argument("--context-len", type=int, default=32768, help="Target context length per sequence")
    args = parser.parse_args()

    cfg = load_config(args.model)
    kv_bytes = KV_DTYPE_BYTES[args.kv_dtype]
    bytes_per_tok, arch_desc = calculate_kv_bytes_per_token(cfg, kv_bytes)

    total_vram_gib = args.gpu_vram_gb * args.tp
    usable_vram_gib = total_vram_gib * args.gpu_util

    weight_gib = (args.params_b * 1e9 * BYTES_PER_PARAM[args.quant]) / (1024 ** 3)
    # Fixed CUDA context + activation scratchpad (~1.4 GiB per GPU)
    runtime_overhead_gib = 1.4 * args.tp
    available_kv_gib = usable_vram_gib - weight_gib - runtime_overhead_gib

    kib_per_tok = bytes_per_tok / 1024.0
    gib_per_seq = (bytes_per_tok * args.context_len) / (1024 ** 3)

    print(f"=== VRAM Sizing Report: {args.model} ===")
    print(f"Attention Architecture : {arch_desc}")
    print(f"Cluster Capacity       : {args.tp}x {args.gpu_vram_gb:.1f} GiB GPUs ({usable_vram_gib:.2f} GiB usable at {args.gpu_util:.0%} util)")
    print(f"Estimated Weight VRAM  : {weight_gib:.2f} GiB ({args.quant})")
    print(f"CUDA & Activation Buf  : {runtime_overhead_gib:.2f} GiB")
    print(f"Remaining KV Headroom  : {available_kv_gib:.2f} GiB")
    print(f"KV Cache per Token     : {kib_per_tok:.2f} KiB/token ({args.kv_dtype})")
    print(f"KV Cache per {args.context_len} Seq : {gib_per_seq:.2f} GiB")

    if available_kv_gib <= 0:
        print("\n[FAIL] Model weights + CUDA overhead exceed usable VRAM. Increase --tp or use lower quantization.")
    else:
        max_total_tokens = int((available_kv_gib * (1024 ** 3)) / bytes_per_tok)
        max_concurrent_seqs = max_total_tokens / args.context_len
        print(f"\n[OK] Max Total KV Tokens  : {max_total_tokens:,} tokens")
        print(f"[OK] Max Concurrent Seqs  : {max_concurrent_seqs:.1f} sequences at {args.context_len} context length")

if __name__ == "__main__":
    main()

Run the script against Qwen/Qwen2.5-Coder-32B-Instruct on a single $24\text{ GiB}$ GPU:

python3 vram_calculator.py \
  --model Qwen/Qwen2.5-Coder-32B-Instruct \
  --params-b 32.76 \
  --quant awq-int4 \
  --kv-dtype bf16 \
  --gpu-vram-gb 24 \
  --context-len 32768

Output:

=== VRAM Sizing Report: Qwen/Qwen2.5-Coder-32B-Instruct ===
Attention Architecture : GQA (40Q:8KV) (L=64, n_kv=8, d_head=128)
Cluster Capacity       : 1x 24.0 GiB GPUs (21.60 GiB usable at 90% util)
Estimated Weight VRAM  : 17.09 GiB (awq-int4)
CUDA & Activation Buf  : 1.40 GiB
Remaining KV Headroom  : 3.11 GiB
KV Cache per Token     : 256.00 KiB/token (bf16)
KV Cache per 32768 Seq : 8.00 GiB

[OK] Max Total KV Tokens  : 12,738 tokens
[OK] Max Concurrent Seqs  : 0.4 sequences at 32768 context length

Notice what the math reveals immediately: while the 4-bit weights fit on a single $24\text{ GiB}$ card, a bf16 KV cache only has room for $12,738\text{ tokens}$, which is why passing --max-model-len 32768 crashes with ValueError: model’s max seq len is larger than KV cache. Switching --kv-dtype fp8 (--kv-cache-dtype fp8 in vLLM) halves the per-token cost to $128.00\text{ KiB/token}$, doubling capacity to $25,476\text{ tokens}$ without touching model weight precision.

Official Documentation & Primary References

• DeepSeek-V3 Technical Report (arXiv:2412.19437): Primary mathematical specification for Multi-Head Latent Attention (MLA) and decoupled RoPE compression.
• FlashMLA Official GitHub Repository: Hopper and Blackwell CUDA kernels for variable-length MLA decoding.
• vLLM PagedAttention Design Documentation: Block table memory layout, physical KV block allocation, and preemption mechanics.
• AWQ: Activation-aware Weight Quantization (arXiv:2306.00978): Group-wise quantization scaling mathematics and salient channel protection.

More in LLM Serving

LiteLLM Supply Chain Incident Response: Removing the cook.pth Backdoor and Hardening AI Gateways on Linux

Resolve the LiteLLM v1.107.0 supply chain incident on Linux: detect cook.pth backdoors, safely purge infected venvs, and lock down AI proxies.

vLLM v0.32 Upgrade Guide: Python 3.11 Migration, CORS Lockdown & NVFP4 KV Cache on Linux

Troubleshoot vLLM v0.32 breaking changes on Linux: fix Python 3.10 deprecation, CORS origin blocking in WebUIs, and configure NVFP4 low-precision KV cache.

vLLM v0.31 Upgrade Guide: Fast Engine Restarts with 'vllm preload', CUDA IPC Zero-Copy, and Breaking Context Changes

Master vLLM v0.31.0 on Linux. Cut engine restart latency from minutes to seconds with vllm preload and --load-format ipc_cache, and fix breaking context changes.