🔒 Paid
Orphaned Background Worker Queue Growth
A deploy just went out to the background worker fleet. Producers are publishing normally, but the job queue is growing without bound — active consumer count has dropped to zero. The worker pods are crash-looping on a bad environment variable baked into the new deploy. Fixing the deploy and draining the backlog that built up while workers were dead are two genuinely independent problems; you need to solve both.
- Difficulty
- Staff
- Expected Time
- ~16 min
- Category
- Messaging & Streaming
Skills
- Distinguishing a dead consumer from a slow or poisoned one
- Recognizing that a bad deploy and an accumulated backlog are two independent problems
- Reading zero active consumers as the root cause, not restart count alone
- Avoiding a plausible-but-incomplete single-track remediation
System Architecture
- Producer Service
- Job Queue
- Worker Pool
- Notification API
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents