🔒 Paid

Multi-Region Payment Platform Degradation

EU's regional path to the shared Fraud Provider has degraded — EU risk checks are failing or timing out at an elevated rate, while US's own calls to the same provider remain healthy. Independently, cross-region ledger replication from the US Ledger to the EU Ledger Replica has begun lagging behind the write stream. Shifting EU traffic onto US capacity relieves EU's payment availability, but it adds EU's volume on top of US's own native ledger/settlement load, risking a second, distinct failure domain in US capacity that did not exist before the mitigation. Three business invariants govern every decision here: (1) payment authorization must not create divergent ledger authority — a payment must never be authorized in a way that lets its ledger write land independently in more than one region at the same time; (2) mandatory fraud controls may not be bypassed under any capacity or latency pressure, for any transaction; (3) settlement delay may be tolerated within an authored bounded window — a growing-but-draining settlement backlog is an acceptable, bounded tradeoff, but an unbounded, climbing backlog is not. Decide how much EU traffic to shift, how to protect ledger write authority and mandatory fraud checks while doing so, and how to protect US ledger/settlement capacity from the pressure that traffic shifting itself creates.

Difficulty
Principal
Expected Time
~25 min
Category
Distributed Systems

Skills

  • Recognizing a mitigation that creates a second, self-inflicted failure domain
  • Protecting correctness invariants before shifting load across regions
  • Distinguishing a bounded, draining backlog from an unbounded one
  • Sequencing containment, correctness protection, and capacity protection correctly

System Architecture

  • Clients
  • Global Traffic Manager
  • US Payment API
  • US Risk Service
  • US Ledger
  • US Event Publisher
  • EU Payment API
  • EU Risk Service
  • EU Ledger / Replica
  • EU Event Publisher
  • Fraud Provider
  • Settlement Pipeline

This is a Pro scenario. Upgrade to Pro to unlock it.

Upgrade to ProBack to Incidents