Multi-Tenant Control Plane Overload

One large tenant ('north-star-retail') has begun issuing a very high rate of configuration-change requests, saturating the shared control plane — the scheduler, config service, policy engine, and metadata DB that manage desired state for every tenant. The worker fleet that actually serves live tenant traffic remains healthy: this is the misleading symptom. A control-plane emergency is not the same thing as a customer-facing data-plane outage, even though both share the same front door. The business has stated the following invariants explicitly: existing data-plane traffic must remain available throughout; config convergence may slow temporarily, but metadata durability must never be lost — a config write may be delayed, it must never be silently dropped; one tenant must not starve unrelated tenants' configuration changes; and no single tenant may consume more than 30% of shared control-plane capacity, regardless of how much that tenant is currently generating. Decide how to contain the hot tenant's churn, protect everyone else's fairness, and bring the control plane back under its own capacity ceiling — without creating a second, self-inflicted failure domain in the data plane you were never asked to touch.

Difficulty

Senior Staff

Expected Time

~20 min

Category

Distributed Systems