Event Platform Partial Outage

Three genuinely independent problems begin within the same narrow window: one broker partition class — the class backing Topic B and Topic C, not Topic A, whose partitions sit on a separate, currently-healthy partition class — becomes degraded; Consumer Group B develops growing lag against Topic B from a slow pattern inside its own processing path; and Consumer Group C begins retrying aggressively against its own, independently degraded downstream DB/API. Overall broker health reads ambiguous rather than cleanly down. You must diagnose and act on the broker problem, the consumer problem, and the downstream problem independently — this must never collapse into 'just add consumers': naively scaling Consumer Group B's concurrency while Consumer Group C is still hammering the shared downstream in unmanaged retry mode measurably worsens the already-degraded downstream dependency, even though Consumer Group B's own lag number will look great immediately afterward. Four business invariants govern every decision: (1) critical Topic A events must remain durable; (2) lower-priority Topic C events may be delayed; (3) no consumer group may be allowed to destabilize unrelated topics; (4) dropping events merely to reduce lag is not acceptable. If the degraded broker partition class is ever left unisolated under heavy batch load while Consumer Group C's retry storm is active for long enough, that contention bleeds into Topic A's own partition class and real, durable Topic A events are dropped — a historical loss that persists for the rest of the incident even after the live configuration is corrected, because fixing the configuration stops new drops but can never retroactively restore events that are already gone.

Difficulty

Senior Staff

Expected Time

~22 min

Category

Messaging & Streaming