API Migration Dual-Write Failure

A migration from Legacy Service/DB to New Service/DB is in partial rollout. The Migration Layer is dual-writing every request to both systems, but some writes succeed only on Legacy and some succeed only on New — Legacy and New have begun to diverge for a real, growing set of records, even though customer-facing request success rate stays high throughout. Reads are routed by rollout cohort, meaning some customers read from Legacy and some from New depending on which cohort their account was assigned to. Five business invariants govern containment: (1) there must be exactly one authoritative write source at any time — dual-write itself is the mechanism creating divergence; (2) acknowledged writes must not disappear during rollback; (3) rollback does not automatically repair data already written only to the new system — it only changes where new requests go, never what already happened to old ones; (4) read routing must not expose unresolved divergent state — a read must never silently omit an acknowledged write still sitting unreconciled on the other side; (5) migration safety outranks rollout velocity — increasing rollout while the system is still dual-writing and diverging is never justified by migration progress alone. Decide how to freeze write authority to a single source, how to reconcile the historical divergence that already exists before declaring the system safe, and whether the right final destination is a safe rollback to Legacy or a safe, completed migration onto New — both are legitimate outcomes provided the historical backlog has genuinely been resolved first.

Difficulty

Principal

Expected Time

~30 min

Category

Distributed Systems