Request Timeout Cascade
A dependency at the bottom of a four-hop call chain (Gateway, Service A, Service B, and the Dependency itself) has developed a latency blip — it is slow, not failing outright, and customers depend on the capability it provides. The Gateway is configured to retry a request when its own call times out. No request chain is being explicitly cancelled anywhere along the way when a caller gives up on it. In-flight request counts and connection utilization are climbing at every layer, and customer-facing latency is degrading with them. Diagnose where in the chain pressure is actually accumulating — and why it keeps accumulating even after a fix lands on one hop — before connection capacity runs out anywhere in the chain.
- Difficulty
- Staff
- Expected Time
- ~20 min
- Category
- API & Traffic
Skills
- Reasoning about timeout budgets across multiple chained services
- Recognizing retry-without-cancellation as a distinct accumulation mechanism from retry amplification against a single slow hop
- Distinguishing a re-proportioned, cancellation-aware fix from a dependency fenced off with a circuit breaker
- Identifying why fixing a single hop's timeout in isolation leaves upstream capacity oversized
System Architecture
- Client
- Gateway
- Service A
- Service B
- Dependency
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents