nginx streams your tokens fine. HAProxy holds them for 206ms.

7.8 relevance

Performance comparison of nginx vs HAProxy for SSE streaming, directly actionable for cloud infrastructure.

General dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For engineers building AI agent orchestration systems or serving LLM token streams through HAProxy, this undocumented buffering behavior adds 200ms+ latency to first-token delivery, directly degrading user-perceived responsiveness.

Key Takeaways

Benchmark your SSE proxy with real token-sized payloads (~60 bytes), not synthetic 1KB chunks—HAProxy's default config hides coalescing unless tested with the LLM stream profile.

nginx streams your tokens fine. HAProxy holds them for 206ms.

Summary

Measurements show HAProxy 3.4.4 buffers SSE token streams for 206ms, delivering frames in bursts of five (5.12 frames per read), while nginx 1.31.5, Caddy 2.11.4, and Traefik v3.7.13 pass tokens through at 50ms intervals indistinguishable from a direct connection. The coalescing is triggered by the small payload size (~60 bytes) of LLM token streams; growing frames to ~1.1KB reduces first-token latency to 53ms and frames-per-read to 1.46, revealing the issue is a buffer-fill threshold rather than a fixed timer. The commonly advised fix for SSE buffering—nginx's proxy_buffering off—has no effect because nginx was never the problem.

Author

Remdore