🔒 Paid

Request Timeout Cascade

A dependency at the bottom of a four-hop call chain (Gateway, Service A, Service B, and the Dependency itself) has developed a latency blip — it is slow, not failing outright, and customers depend on the capability it provides. The Gateway is configured to retry a request when its own call times out. No request chain is being explicitly cancelled anywhere along the way when a caller gives up on it. In-flight request counts and connection utilization are climbing at every layer, and customer-facing latency is degrading with them. Diagnose where in the chain pressure is actually accumulating — and why it keeps accumulating even after a fix lands on one hop — before connection capacity runs out anywhere in the chain.

Difficulty
Staff
Expected Time
~20 min
Category
API & Traffic

Skills

  • Reasoning about timeout budgets across multiple chained services
  • Recognizing retry-without-cancellation as a distinct accumulation mechanism from retry amplification against a single slow hop
  • Distinguishing a re-proportioned, cancellation-aware fix from a dependency fenced off with a circuit breaker
  • Identifying why fixing a single hop's timeout in isolation leaves upstream capacity oversized

System Architecture

  • Client
  • Gateway
  • Service A
  • Service B
  • Dependency

This is a Pro scenario. Upgrade to Pro to unlock it.

Upgrade to ProBack to Incidents