How to Enable Speculative Decoding in vLLM for 3x Faster Local Coding Agents

Key Takeaway: The 3x Speedup Flag in vLLM

Speculative decoding pairs a heavy target model with a lightweight draft model to predict multiple tokens per forward pass. Because source code contains predictable syntax, indentation, and standard library calls, draft acceptance rates on structured code routinely exceed 75%.

To enable speculative decoding in vLLM, pass the --speculative-model flag alongside --num-speculative-tokens 5:

# Production command to serve Qwen2.5-Coder-32B with a 1.5B draft model
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct \
    --speculative-model Qwen/Qwen2.5-Coder-1.5B-Instruct \
    --num-speculative-tokens 5 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 16384 \
    --port 8000

Single-stream generation throughput jumps by 2x to 3x while mathematical output remains lossless and identical to running Qwen2.5-Coder-32B-Instruct alone. Zero degradation. Zero hallucination increase.

+-----------------------------------------------------------------------+
|  STEP 1: Draft Model Generates K Speculative Tokens                   |
|  Qwen2.5-Coder-1.5B emits: ["def ", "calculate", "_total", "(items", ")"]|
|  Fast memory-bound draft passes                                       |
+-----------------------------------------------------------------------+
                                  |
                                  v
+-----------------------------------------------------------------------+
|  STEP 2: Target Model Verifies All K Tokens in ONE Forward Pass       |
|  Qwen2.5-Coder-32B checks probabilities concurrently in parallel      |
|  Single compute-bound batch verification pass                         |
+-----------------------------------------------------------------------+
                                  |
                                  v
+-----------------------------------------------------------------------+
|  RESULT: 4 of 5 Tokens Accepted in Parallel                           |
|  4 tokens emitted in the time of ~1.2 target forward passes           |
+-----------------------------------------------------------------------+

Speculative Decoding Trade-Offs Across Lookahead Depth ($K$)

When pairing Qwen2.5-Coder-32B-Instruct-AWQ with Qwen2.5-Coder-1.5B-Instruct on a single 24GB GPU (such as an RTX 4090) or 80GB A100, throughput scales with the speculative lookahead length $K$ up to the point where rejected draft tokens waste verification compute:

Setting K=5 delivers the sweet spot for Python and TypeScript files. Higher values like K=7 suffer diminishing returns because verification penalties outweigh speculative hits on branching algorithmic logic.

Why Code Generation Has Exceptionally High Acceptance Rates

Standard conversational English has high entropy. The draft model guesses words with 50% to 60% accuracy.

Code is different. Code has structural rigidity.

# The draft model easily guesses the next 12 tokens without error:
for item in shopping_cart.items():
    total_price += item.get_price()

Keywords (def, return, import), indentation blocks (4 spaces), common operators (+=, ==), and standard library methods (json.dumps, pathlib.Path) are trivial for a 1.5B parameter model to predict correctly. The 32B target model simply signs off on the draft tokens in parallel.

Hardware and VRAM Sizing Rules

The draft model and target model must reside on the same GPU instance, or share tensor parallel ranks.

Calculate your VRAM budget before launching:

Total Required VRAM = Target Weights + Draft Weights + KV Cache Space

For a single 24GB consumer GPU (RTX 3090 / 4090):

• Target model: Qwen/Qwen2.5-Coder-32B-Instruct-AWQ (18.2 GB)
• Draft model: Qwen/Qwen2.5-Coder-1.5B-Instruct (3.1 GB in FP16, or 1.1 GB in AWQ)
• Remaining for KV Cache: ~2.7 GB (Permits 8,192 context window with FP8 KV cache)

Run the single-GPU quantized speculative server:

vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ \
    --speculative-model Qwen/Qwen2.5-Coder-1.5B-Instruct \
    --num-speculative-tokens 4 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.95 \
    --max-model-len 8192

Common Traps and Failure Modes

1. Mismatched Tokenizers

The target model and draft model must share an identical vocabulary and tokenizer. Pairing Qwen2.5-Coder-32B with DeepSeek-R1-Distill-1.5B causes instant runtime crashes because token IDs map to different subwords. Always use models from the exact same architecture lineage.

2. Oversized Draft Models

Using an 8B or 14B model as a draft for a 32B model destroys throughput. The draft model forward pass takes too long, negating the verification speedup. Keep the draft parameter count under 10% of the target model.

3. High Temperature Degradation

Speculative decoding relies on greedy or low-temperature decoding (temperature <= 0.2). At temperature=0.8, token sampling distributions diverge, causing draft acceptance rates to plunge below 40%. For coding agents, keep temperature at 0.0 or 0.1.

Connecting Your Accelerated vLLM Server to Cursor and Cline

Once vLLM is running, configure your agent clients to point to the local OpenAI-compatible endpoint:

{
  "api_provider": "openai",
  "base_url": "http://127.0.0.1:8000/v1",
  "model_name": "Qwen/Qwen2.5-Coder-32B-Instruct",
  "temperature": 0.1,
  "max_tokens": 4096
}

Files that previously took 25 seconds to generate now stream in under 8 seconds. The agent finishes test suites and file rewrites before your attention wanders.