🔒 Paid
Webhook Delivery Storm
A provider shared by a meaningful subset of your customers' webhook receivers has gone down, and deliveries to that subset are now failing or timing out. All customer destinations — healthy and affected alike — are served by the same dispatcher worker pool and retry queue. Every customer's webhook feed is business-critical to them individually, and destinations outside the affected subset have done nothing wrong. Diagnose why delivery is degrading even for customers whose endpoints are fully healthy, and bring the dispatcher back to a durable, sustainable state without leaving the platform unable to tell a transient blip from a genuine outage.
- Difficulty
- Staff
- Expected Time
- ~20 min
- Category
- API & Traffic
Skills
- Diagnosing shared-resource contention across independent destinations
- Scoping circuit breakers/isolation to the actual failing subset
- Recognizing when adding shared capacity doesn't contain a retry storm
- Weighing backoff and dead-lettering as partial versus structural fixes
System Architecture
- Event Source
- Webhook Dispatcher
- Affected Customer Endpoints
- Healthy Customer Endpoints
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents