🔒 Paid
Message Retry Storm
A downstream dependency your worker pool depends on has become partially unhealthy — a rising share of its requests are failing or timing out. Messages that fail to process re-enter the queue for redelivery, and every added worker pulls independently from the same broker. Diagnose why queue depth and retry volume keep climbing even though no single customer-facing request is failing outright, and bring the backlog back under control before it grows unrecoverably large. The downstream dependency's own operators have confirmed it is degraded, not fully down — some requests still succeed.
- Difficulty
- Senior
- Expected Time
- ~15 min
- Category
- Messaging & Streaming
Skills
- Diagnosing retry amplification in an asynchronous queue
- Distinguishing queue-health incidents from synchronous request failure
- Retry bound, backoff, and dead-letter queue design
- Recognizing when added worker capacity doesn't fix the problem
System Architecture
- Producer Service
- Message Broker
- Worker Pool
- Downstream Dependency
- Dead Letter Queue
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents