Leader Election Storm

A production consensus cluster backs a service where writes and strong reads must go through a single leader at any given moment. One node's network link has started experiencing intermittent latency — the node itself is healthy, but its heartbeats are sporadically arriving late. The cluster is holding repeated, disruptive leadership elections, and every election is a window where writes and strong reads fail. The remaining fault-tolerance margin of the voting set matters for whatever you do next — the cluster must stay able to tolerate losing a node, not just look quiet right now. Diagnose why the cluster keeps re-electing a leader and bring it to a state that is durably stable, not merely quiet at this exact moment.

Difficulty

Staff

Expected Time

~15 min

Category

Distributed Systems