Data Lineage Break During ETL Migration

A migration from ETL v1 to ETL v2 is in progress. Sources remain fully available, Warehouse writes keep succeeding, and Reports keep generating on schedule — but lineage is becoming inconsistent across the two ETL paths, and nothing about availability signals it. Four business invariants govern every decision here: (1) a report or metric must never silently mix v1-derived and v2-derived data within the same computed value; (2) migration cutover must be traceable and auditable per source/metric, not an opaque global flip; (3) stakeholders consuming a metric whose lineage is currently inconsistent must be able to know that through a visible untrusted/flagged state, never be silently served a wrong number with full confidence styling; (4) once a report has been served with inconsistent lineage, that exposure is a real, non-resetting historical consequence even after the lineage is later fixed. Decide how to manage the cutover so lineage stays traceable and consistent — cutting over per-source in controlled, auditable stages, or running both paths in parallel and reconciling before declaring a source cut over — whether specific metrics or reports should be flagged untrusted until their full lineage path completes migration, and whether tearing down the legacy path early is worth losing the only reconciliation and rollback reference for sources not yet verified.

Difficulty

Principal

Expected Time

~25 min

Category

Data Pipelines