Global Feature Rollout Incident

A global feature rollout has introduced a region-dependent correctness problem: the US, EU, and APAC Service Fleets can each independently diverge in how they interpret the new feature, and the rollout is writing customer records in a new, possibly-incompatible representation into one Shared Data Store. Five business invariants govern every decision here: (1) correctly-written existing customer state must remain readable after rollback — rollback is never automatically a repair; (2) a regional rollout may be halted independently of the others — a single region's decision never requires touching every region; (3) global rollback may not discard or hide already-written incompatible data — the historical count of incompatible writes must remain visible and must not be silently zeroed; (4) the Shared Data Store must not be left holding mutually incompatible active representations without an explicit compatibility strategy (dual-write, migration repair, or a read-time compatibility shim); (5) correctness outranks rollout velocity — a faster rollout is never worth a wider, unrepaired correctness gap. Decide which regions to halt or continue, whether to roll the feature back globally or fix forward with dual-write, how to resolve the already-written incompatible backlog (migration repair vs. a read-compatibility shim), and in what order — rolling back before repairing leaves a real, measurable gap that rolling back alone does not close.

Difficulty

Principal

Expected Time

~25 min

Category

Deployments & Reliability