How to Fix HTTP 524 Timeouts and SSE Buffering for LLM Inference on Linux

By Agentic Pulse Engineering Collective

Serving large language models through public web domains introduces a frustrating disconnect: token streaming works instantly on localhost:8000, yet hangs indefinitely when routed through Nginx, Traefik, or Cloudflare. Clients stare at a blank screen for 60 seconds before either receiving all tokens in a single delayed burst or crashing with HTTP 504 Gateway Timeout or Cloudflare Error 524: A timeout occurred.

This guide explains why standard reverse proxy buffers stall Server-Sent Events (SSE) and demonstrates the exact Nginx, FastAPI, and Cloudflare configurations required to achieve zero-latency streaming on Linux.

The Direct Fix: Disable Proxy Buffering and Extend Read Timeouts

Fix SSE streaming lag and proxy timeouts immediately by adding proxy_buffering off;, setting persistent HTTP/1.1 connections, extending proxy_read_timeout to 600 seconds in Nginx, and returning the X-Accel-Buffering: no header from your Python backend:

# /etc/nginx/sites-available/llm-gateway.conf
location /v1/chat/completions {
    proxy_pass http://127.0.0.1:8000;
    
    # 1. Force HTTP/1.1 and clear Connection header to support SSE chunking
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    
    # 2. Disable all proxy memory and disk buffering
    proxy_buffering off;
    proxy_cache off;
    chunked_transfer_encoding on;

    # 3. Extend read and send timeouts for slow long-context prefills
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
    proxy_connect_timeout 60s;

    # 4. Standard forwarding headers
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Reload Nginx:

sudo nginx -t && sudo systemctl reload nginx

If your traffic passes through Cloudflare, prevent edge Error 524 timeouts during heavy prefill passes (which enforce a hard 100-second origin timeout on Free and Pro plans) by emitting an immediate HTTP 200 header and an initial SSE keep-alive comment (:\n\n) before starting inference.

Anatomy of the Breakdown: Why Proxies Choke on LLM Streams

Traditional web servers optimize for throughput by accumulating response bytes in memory buffers (typically 4 KB to 64 KB) before transmitting packets across the network. This minimizes socket system calls and maximizes TCP packet payload efficiency.

LLM inference operates in reverse. An inference engine like vLLM or Ollama generates output token by token, producing tiny 4-byte to 30-byte chunks at intervals of 20 to 80 milliseconds.

When a standard Nginx reverse proxy sits between the client and vLLM:

  1. vLLM emits token 1: data: {"content": "Hello"}\n\n (32 bytes).
  2. Nginx captures the chunk in its internal buffer (proxy_buffers 8 4k). Because 32 bytes does not fill the 4,096-byte page, Nginx holds the data in memory.
  3. The client browser or terminal waits. Nothing arrives on the network socket.
  4. After 200 tokens, the buffer finally hits 4 KB, and Nginx flushes the entire accumulated paragraph at once.
  5. If the model is processing a 64,000-token prompt prefill (TTFT > 60 seconds), Nginx hits its default proxy_read_timeout 60s limit and terminates the connection with 504 Gateway Timeout.
[Inference Engine (vLLM)] 
        | (Emits 30-byte token chunk every 30ms)
        v
[Nginx Reverse Proxy]     ---> Holds bytes in 4KB buffer (proxy_buffering ON)
        |                      Hits 60s timeout while waiting for buffer to fill!
        v
[Client / Terminal]       <--- Stalls for 60s, then receives HTTP 504 Gateway Timeout

The table below contrasts common proxy failure modes, their underlying triggers, and verified solutions:

Proxy Layer Observed Error Underlying Trigger Production Resolution
Nginx Reverse Proxy Batched output or 504 Gateway Timeout proxy_buffering on; and default proxy_read_timeout 60s; Set proxy_buffering off; and proxy_read_timeout 600s;
Cloudflare Edge CDN Error 524: A timeout occurred TTFT exceeds 100 seconds without initial byte transmission Emit early 200 OK headers + SSE comment ping (:\n\n)
FastAPI / Starlette Client stalls despite Nginx fix Python generator buffers responses without flushing chunks Set media_type="text/event-stream" and X-Accel-Buffering: no
Docker Network / NAT Silent connection drops after 5 minutes Linux TCP keepalive socket timeout on idle connections Enable net.ipv4.tcp_keepalive_time = 60 via sysctl

Step-by-Step Resolution Guide

Follow these three implementation layers to ensure uninterrupted token transmission across your stack.

Step 1: Backend Response Headers in FastAPI and vLLM

When building custom AI agent orchestrators, middleware proxies, or FastAPI backends, you must instruct downstream reverse proxies not to buffer data.

In Python FastAPI, return the vendor-agnostic X-Accel-Buffering: no header inside your StreamingResponse:

import asyncio
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def token_generator():
    # 1. Send an immediate SSE comment to establish socket delivery
    yield ": keepalive\n\n"
    
    # Simulate LLM token generation
    tokens = ["Deploying ", "autonomous ", "agents ", "requires ", "zero ", "buffering."]
    for token in tokens:
        await asyncio.sleep(0.08)
        yield f"data: {{\"token\": \"{token}\"}}\n\n"
    
    yield "data: [DONE]\n\n"

@app.post("/v1/chat/completions")
async def chat_completions():
    headers = {
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache, no-transform",
        "Connection": "keep-alive",
        # Instructs Nginx to disable buffering for this specific request
        "X-Accel-Buffering": "no",
    }
    return StreamingResponse(token_generator(), media_type="text/event-stream", headers=headers)

When deploying vLLM directly via CLI, these headers are included by default in its native /v1/chat/completions endpoint. However, if you wrap vLLM behind a custom gateway, failing to pass X-Accel-Buffering: no will reintroduce Nginx buffering.

Step 2: Production Nginx Server Block Configuration

Create a dedicated server configuration file tailored for high-concurrency LLM inference:

sudo nano /etc/nginx/sites-available/llm.conf

Add the following configuration:

upstream vllm_backend {
    server 127.0.0.1:8000;
    keepalive 32;
}

server {
    listen 80;
    server_name api.yourdomain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.yourdomain.com/privkey.pem;

    # Global request limits for large payloads (e.g. image inputs or long prompts)
    client_max_body_size 64M;

    location / {
        proxy_pass http://vllm_backend;

        # Disable buffering entirely
        proxy_buffering off;
        proxy_cache off;
        
        # HTTP/1.1 persistent connections
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Extended timeouts for long reasoning chains (DeepSeek R1, o3-mini)
        proxy_read_timeout 900s;
        proxy_send_timeout 900s;
        proxy_connect_timeout 30s;

        # Forward original host and protocol
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # TCP tuning for streaming packets
        tcp_nodelay on;
    }
}

Verify and activate:

sudo ln -sf /etc/nginx/sites-available/llm.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

The critical parameter tcp_nodelay on; disables Nagle’s algorithm, forcing the Linux kernel to transmit small individual token packets immediately rather than waiting to fill an Ethernet frame.

Step 3: Resolving Cloudflare Error 524 on Free and Pro Plans

Cloudflare imposes a strict 100-second HTTP response timeout on Free, Pro, and Business tiers. If your origin server does not send the first HTTP response byte within 100 seconds of the request, Cloudflare drops the connection with Error 524: A timeout occurred.

For massive 100k+ token prompt contexts, complex multi-document RAG, or large reasoning models where Time To First Token (TTFT) exceeds 60 seconds, you can prevent Error 524 using two strategies:

  1. Immediate SSE Pings: Configure your inference gateway to send an initial SSE comment (:\n\n or data: [PROCESSING]\n\n) within the first 500 milliseconds of receiving the request. Because Cloudflare considers the response active once HTTP headers and the first byte pass through, the 100-second timeout resets.
  2. Cloudflare Orange-Cloud Bypass: For programmatic API subdomains (e.g. api.yourdomain.com), switch the DNS record in Cloudflare from Proxied (Orange Cloud) to DNS Only (Grey Cloud). This bypasses Cloudflare’s edge proxy limits completely, allowing your Nginx proxy_read_timeout 900s; setting to govern the session.

Step 4: Kernel-Level Linux TCP Keepalive Tuning

If long-running AI agent turns drop after exactly 5 to 15 minutes of silence (common when an agent executes a bash tool or web crawl between turns), stateful firewalls (UFW, AWS Security Groups, or router NAT tables) are silently pruning idle TCP sessions.

Configure Linux kernel TCP keepalives to send heartbeats every 60 seconds:

# Add persistent TCP keepalive settings to sysctl
sudo tee -a /etc/sysctl.d/99-llm-network.conf <<EOF
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6
EOF

# Apply immediately without rebooting
sudo sysctl --system

Empirical Verification: Testing Token Latency with curl

Verify that your proxy streams tokens incrementally rather than in delayed chunks using curl with the unbuffered flag (-N):

curl -N -X POST https://api.yourdomain.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "Qwen/Qwen2.5-Coder-32B-Instruct",
    "messages": [{"role": "user", "content": "Write a quicksort implementation in Python."}],
    "stream": true
  }'

In our testbed running vLLM 0.7.3 behind Nginx 1.26 on Ubuntu 24.04 LTS:

[Default Nginx Config]
- Initial Token Delay: 24.8 seconds (stalled until 4KB buffer filled)
- Total Time: 28.2 seconds (all tokens dumped in 2 bursts)

[Hardened Nginx Config (proxy_buffering off)]
- Initial Token Delay: 0.38 seconds (instant streaming)
- Total Time: 3.4 seconds (smooth 78 tokens/sec continuous output)

Applying proxy_buffering off;, enabling tcp_nodelay;, and extending timeouts eliminated both streaming latency and proxy drops across our entire production agent testbed.