Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows

7.2 relevance

Netflix's rework of Conductor for massive scale offers direct insights for workflow orchestration at scale.

Startup infoq.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For platform engineers building multi-agent orchestration systems, this demonstrates how to scale stateful workflows beyond naive DAG executors by separating evaluation from execution and eliminating distributed lock contention using application-level reconciliation.

Key Takeaways

Apply Netflix's blueprint: separate workflow metadata from task state, load only decision-relevant tasks lazily, and reconcile pending/terminal task states without locks.

Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows

Summary

Netflix rebuilt Conductor 4.0 to handle 420 million monthly workflow executions, increasing task size from 2,500 to 30,000 per workflow while reducing p99 evaluation latency by 40%. The redesign separates workflow metadata from task data, loads only necessary task portions during evaluation, and removes locking by storing pending and terminal states separately, reconciled in the application layer. Failed lock attempts dropped from 2,700 per interval to zero, and indexing was decoupled via Kafka with Elasticsearch and Iceberg for long-term storage.

Author

Leela Kumili