Data Pipeline Freshness Collapse

The Stream Processor sitting between Ingestion and the Warehouse has begun falling behind real-time. Source Systems and Ingestion success both stay healthy throughout — this is not an ingestion outage, and the dashboard that only watches ingestion will never show anything wrong. Five business invariants govern every decision here: (1) Tier-1 operational datasets must remain within their authored 120-second freshness SLO at all times, even while the incident is active; (2) lower-priority Tier-2 analytical datasets may lag substantially during the incident — there is no freshness SLO on them, and protecting them is never a reason to risk Tier-1 or the Warehouse; (3) events may never be dropped merely to make lag metrics look better — discarding events to improve a dashboard is a permanent, irreversible correctness violation, not a shortcut, and it is tracked as a historical count that never resets once incurred; (4) Warehouse correctness and durability must be preserved — scaling the Stream Processor's concurrency without also protecting the Warehouse's write path (writer concurrency and checkpoint cadence) floods the Warehouse with more concurrent writers than it can safely absorb, which is the single most tempting and most harmful response to this incident; (5) a backlog is an acceptable way to hand this incident back only if it is bounded and demonstrably draining, never if it is frozen or still growing. Decide how to safely increase processing throughput without breaking the Warehouse, how to protect Tier-1 freshness without starving the processor entirely, and how much real elapsed drain time a given backlog genuinely needs before the incident can be called resolved.

Difficulty

Senior Staff

Expected Time

~25 min

Category

Data Pipelines