A dead Kubernetes node is detected in 3 seconds and keeps receiving traffic for 13

7.9 relevance

Detailed K8s node failure investigation with surprising results, highly actionable.

AI/ML dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For platform engineers managing Kubernetes clusters, this exposes that endpoint update latency—not node detection—is the dominant source of traffic loss during unplanned failures, and common tuning like externalTrafficPolicy: Local doesn't fix it.

Key Takeaways

Measure your cluster's endpoint propagation time after node failures and consider implementing readiness probes or external load balancer health checks that react faster than the default endpoint controller.

A dead Kubernetes node is detected in 3 seconds and keeps receiving traffic for 13

Summary

A hard power-off of a Kubernetes node on DigitalOcean is detected in ~3 seconds (not the default 40s grace period), but the endpoint controller takes another ~10 seconds to remove pods from the Service, causing all traffic loss within that gap. Switching to externalTrafficPolicy: Local shifts failures to an immediate burst rather than spreading them out, but does not reduce total failed requests—the endpoint propagation delay remains the bottleneck.

Author

Remdore