Cascading Failure Across Shared Platform

Three independent products — Product A, Product B, and Product C — all run on the same shared platform: a Shared Gateway, a Shared Cache, Shared Identity, Shared Metadata, Shared Messaging with its Async Workers, and a Shared Database. Shared Identity's token-validation latency degrades first — the true root cause, before any product-level symptom appears. Product B's own aggressive retry policy then amplifies that degradation into a compounding load multiplier against the shared platform, making Product B's dashboard the loudest, reddest signal in the incident, even though Product B did not cause it and fully containing Product B alone does not make the platform safe. Separately and more quietly, Product A's connections — held open longer while waiting on slow identity validation — begin exhausting the Shared Database's connection pool. Five business invariants govern every decision: (1) a single product must not exhaust shared-platform capacity for every product; (2) critical identity correctness must be preserved — an unverifiable token must never be treated as valid merely to improve availability metrics; (3) shared metadata/database durability must not be traded away for short-term availability; (4) products may be intentionally degraded or isolated to protect the platform, and a safely isolated, not-yet-restored product is a legitimate outcome, not a failure; and (5) recovering the root dependency does not mean the backlog and connection pressure it caused have already cleared — both drain independently, on their own real-elapsed-time schedules, never instantly. Determine the true root failure domain, which product is amplifying the incident, which shared dependency to protect first, which product's traffic to isolate, the order in which to drain and recover, and the order in which to safely restore every product afterward — restoring before the platform has genuinely recovered retriggers the entire cascade.

Difficulty

Principal

Expected Time

~35 min

Category

Distributed Systems