Why we stopped using one Elasticsearch index per tenant and built our own

7.9 relevance

Open-sourced custom search engine replacing ES multi-tenant pattern; deep technical and open source relevance.

General dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For engineers building multi-tenant search at scale, this S3-backed architecture sidesteps Elasticsearch's cluster-state bottleneck and offers a template for horizontally scalable, low-ops search infrastructure without sacrificing Elasticsearch compatibility.

Key Takeaways

Evaluate S3-based shared storage and ownership routing (via etcd) to decouple tenant count from cluster state, using immutable segments to simplify caching and failover.

Why we stopped using one Elasticsearch index per tenant and built our own

Summary

A multi-tenant search provider abandoned one-index-per-tenant on Elasticsearch after cluster state issues from thousands of indexes. They built SeaSearch, an open-source, Go-based engine using Bluge, storing index data on S3 and routing via etcd — compute nodes hold no authoritative data, so failover is a map update. Immutable segments enable local caching with parallel warm-up, avoiding shard replication and allowing unbounded index counts.

Author

SeaCloud Labs