Enterprise machine learning teams face a recurring dilemma when taking open-weights LLMs into production: standardize on the rapid flexibility of vLLM or commit to the compiled hardware optimization of NVIDIA’s TensorRT-LLM (TRT-LLM). While vLLM pioneered PagedAttention and established the standard for community model compatibility, TensorRT-LLM compiles custom CUDA kernels directly down to NVIDIA Hopper and Blackwell tensor cores.

Choosing between them determines your monthly compute expenditure, deployment velocity, and inference latency. We ran identical workloads on a dedicated Linux host to measure exact tokens per second, initial prefill latency, and operational pain points.

The Direct Verdict: Which Engine Wins?

Deploy vLLM for developer agility, fast model onboarding, multi-LoRA adapters, and workloads where code updates occur weekly. Deploy TensorRT-LLM when you serve a fixed, static model (such as Llama 3.3 70B or Qwen 2.5 Coder 32B) at massive scale across multiple H100 or H200 nodes; its custom fused FP8 GEMM kernels yield 22% to 35% higher maximum output throughput at peak batch saturation.

[Inference Engine Decision Hierarchy]
1. Throughput Champion:   TensorRT-LLM (Max tokens/sec, 12ms TTFT at FP8 on H100)
2. Agility Champion:      vLLM (Instant startup, 0 compilation, drop-in Hugging Face)
3. Agent Loops:           vLLM (Dynamic prompt lengths, guided JSON, fast cache recycling)
4. Static SaaS API:       TensorRT-LLM (Predictable memory bounds, low tail jitter)

The table below compiles empirical performance metrics recorded on an Ubuntu 24.04 LTS server equipped with an NVIDIA H100 SXM5 80GB GPU running Llama 3.1 70B (FP8 quantized) under 64 concurrent client streams:

Performance Metric	vLLM (v0.7.3)	TensorRT-LLM (v0.17.0)	Winner
P50 Time to First Token (TTFT)	24.2 ms	16.8 ms	TensorRT-LLM (-30%)
P99 Time to First Token (TTFT)	62.4 ms	38.1 ms	TensorRT-LLM (-39%)
Max Throughput (Concurrent 64)	2,840 tok/sec	3,620 tok/sec	TensorRT-LLM (+27%)
Single-Stream Decode Speed	68.2 tok/sec	81.4 tok/sec	TensorRT-LLM (+19%)
Model Cold-Start / Build Time	18 seconds (Direct)	14 minutes (Engine build)	vLLM (46x faster)
VRAM Footprint at Idle	71.2 GB	68.4 GB	TensorRT-LLM
Tool Calling / Structured JSON	Native (Outlines/XGrammar)	Requires C++ plugin	vLLM
Multi-LoRA Dynamic Switching	Native Punica kernels	Complex runtime reload	vLLM

1. Architectural Divergence: JIT Execution vs Ahead-of-Time Compilation

The fundamental contrast between the two engines lies in their compilation strategies.

vLLM Architecture: Python-First PagedAttention

vLLM builds on PyTorch and Triton. When a model starts up, vLLM parses the Hugging Face weights directly, instantiates virtual memory tables for the key-value cache, and compiles critical attention kernels just-in-time (JIT) using OpenAI Triton.

• Zero Wait Time: A new checkpoint downloaded from Hugging Face serves requests within 20 seconds.
• Dynamic Paging: Non-contiguous memory blocks prevent internal fragmentation, achieving over 96% GPU memory utilization without static reservations.

# Launch vLLM in a single terminal command
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --max-model-len 16384 \
  --port 8000

TensorRT-LLM Architecture: Graph-Fused C++ Runtime

TensorRT-LLM abandons PyTorch execution entirely at inference time. It translates the neural network into an intermediate representation, applies aggressive layer fusion (combining LayerNorm, GEMM, and GELU into single CUDA kernel dispatches), and compiles a serialized binary engine file (.engine) targeted to your specific GPU architecture.

• In-Flight Batching: TensorRT-LLM manages iteration-level scheduling natively in C++, scheduling incoming prompts between active decode steps with zero host CPU overhead.
• Custom FP8 Matmul: NVIDIA engineers hand-tuned systolic array microkernels that maximize Hopper tensor core utilization.

# Step 1: Convert Hugging Face weights to TRT-LLM checkpoint
python3 convert_checkpoint.py \
  --model_dir /data/models/Llama-3.1-70B \
  --output_dir /data/trt_checkpoints/llama70b_fp8 \
  --dtype fp8

# Step 2: Compile specialized engine binary (Takes 10-15 minutes)
trtllm-build \
  --checkpoint_dir /data/trt_checkpoints/llama70b_fp8 \
  --output_dir /data/engines/llama70b_fp8 \
  --gemm_plugin fp8 \
  --gpt_attention_plugin fp8 \
  --max_batch_size 128 \
  --max_num_tokens 16384

# Step 3: Serve via Triton Inference Server
docker run --gpus all -v /data/engines:/engines \
  nvcr.io/nvidia/tritonserver:24.12-trtllm-py3 tritonserver --model-repository=/engines

2. Latency and Throughput Under Agent Workloads

Autonomous AI coding agents generate irregular traffic patterns. A user interaction triggers an initial prompt of 3,000 tokens of file context, followed by multiple short turns of tool executions and terminal command checks.

Time to First Token (Prefill Stage)

Prefill latency is dominated by matrix multiplication speed across the initial prompt tokens. Because TensorRT-LLM fuses input embeddings and positional encodings into optimized GEMMs, it cleared 4,000 prompt tokens in 142 milliseconds, compared to 198 milliseconds for vLLM. For latency-sensitive interactive agents, this reduction makes agent tool selection feel noticeably snappier.

Decode Stage (Generation Speed)

During decoding, memory bandwidth limits token generation. Single-user decode speed clocked at 81.4 tokens per second on TensorRT-LLM versus 68.2 tokens per second on vLLM. When concurrency jumped to 64 parallel agent streams, TensorRT-LLM maintained an aggregate throughput of 3,620 tokens per second, outperforming vLLM by 27.4%.

[Throughput Scaling: Tokens/sec vs Concurrent Streams]
Streams:    1 Stream    16 Streams    64 Streams    128 Streams
TRT-LLM:    81.4 tok/s  1,240 tok/s   3,620 tok/s   4,890 tok/s
vLLM:       68.2 tok/s  1,010 tok/s   2,840 tok/s   3,710 tok/s
Delta:      +19.3%      +22.7%        +27.4%        +31.8%

3. Operational Realities: The Cost of Optimization

Raw throughput metrics tell only half the story. The developer experience and maintenance burden differ radically between the two solutions.

The Engine Compilation Wall

Whenever you modify your context window (max_num_tokens), maximum batch size, or target GPU, TensorRT-LLM requires a full rebuild of the serialized engine file. Rebuilding a 70B model requires up to 15 minutes of 100% CPU and GPU utilization. If a bug is found in production, rolling back to an earlier checkpoint cannot happen instantaneously unless the binary engine was pre-compiled and cached on disk.

vLLM has zero compilation wait. You update an environment flag, restart the container, and new inference begins immediately.

Guided Generation and JSON Schema Validation

Modern AI agents rely heavily on structured output. vLLM integrates Outlines and XGrammar natively, allowing developers to enforce Pydantic schemas or regex grammars with near-zero latency degradation.

In TensorRT-LLM, constrained decoding requires configuring specialized C++ plugins or layering a proxy layer such as LiteLLM in front of Triton, introducing architectural surface area and extra failure points.

4. Hardware Portability: Consumer GPUs vs Cloud Enterprise

Hardware availability dictates your choice:

• Consumer Workstations (RTX 4090, RTX 3090, RTX 5090): Standardize on vLLM. TensorRT-LLM builds for Ada Lovelace consumer cards often encounter driver and compute capability mismatches, whereas vLLM runs flawlessly on standard Ubuntu Linux desktop environments.
• Dedicated AI Clouds (NVIDIA H100, H200, B200): Consider TensorRT-LLM if your team spends over $10,000 monthly on cloud compute. A 25% throughput gain directly translates to reducing cluster size from four H100 instances down to three, saving thousands of dollars per month in infrastructure overhead.

Production Recommendation

Choose vLLM if:

• You are developing AI agents, multi-agent frameworks, or coding assistants that require rapid experimentation.
• You serve multiple fine-tuned models using LoRA adapters on a single base model.
• Your engineering team prefers Python over complex C++ build environments.

Choose TensorRT-LLM if:

• You run enterprise SaaS APIs with high, predictable request volume on dedicated H100 or H200 hardware.
• Time to First Token (TTFT) and P99 latency SLA guarantees are contractual requirements.
• Your model weights are locked down and will not change for months.