🔒 Paid
Platform-Wide Clock Skew Breaks Distributed Consensus
The distributed lock service is split-brained. The event-sourcing ledger is reordering writes. The rate limiter is false-throttling healthy traffic. Three services, no shared edges between them, all failing at once. Each looks like its own bug — until you trace all three back through the fleet nodes to a single silently-failing NTP sync job. Diagnose the shared upstream cause and resync the fleet's clocks, rather than chasing three unrelated-looking incidents at once.
- Difficulty
- Principal
- Expected Time
- ~25 min
- Category
- Distributed Systems
Skills
- Recognizing a single upstream cause behind multiple independent-looking symptoms
- Distinguishing a shared-dependency failure from three separate single-subsystem bugs
- Understanding how timestamp-ordered consensus degrades under clock skew
- Recognizing a plausible-but-insufficient per-subsystem remediation
- Verifying recovery across every affected subsystem, not just one
System Architecture
- NTP Time Source
- Distributed Lock Service
- Event-Sourcing Ledger
- Sliding-Window Rate Limiter
- Fleet Nodes
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents