Cross-Team Schema Contract Drift Breaks Downstream Pipeline
Orders Service just deployed a backward-incompatible change to its event schema, with no version negotiation. Billing, Analytics, and Fraud Detection all consume the exact same event stream — and each is now failing in a completely different way. Billing is silently dropping events it can't map, with no error spike to notice. Analytics isn't dropping anything; it's writing corrupted rows straight into the warehouse. Fraud Detection is the loudest of the three, crash-looping on deserialization failures. Fixing the producer's schema is necessary but not sufficient — it stops new damage, but repairs nothing already lost, corrupted, or queued. Triage across all three independently-owned pipelines and bring every one of them back, not just the one making the most noise.
- Difficulty
- Senior Staff
- Expected Time
- ~25 min
- Category
- Data Pipelines
Skills
- Recognizing that a single root cause can cascade into DIFFERENT failure modes per consumer
- Distinguishing silent data loss from silent data corruption from loud crash-looping
- Understanding that a producer-side compatibility fix stops new damage but never repairs existing damage
- Prioritizing remediation across multiple independently-owned pipelines rather than chasing the loudest alarm
System Architecture
- Orders Service (producer)
- Event Bus
- Billing Pipeline
- Analytics Warehouse
- Fraud Detection Pipeline
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents