Cross-Service Retry Amplification

A customer request flows through Gateway, Service A, Service B, and Service C before reaching an External Provider. Each of these four layers was built independently and each already retries failed calls on its own. The External Provider has just begun a moderate degradation — elevated latency and a higher transient error rate, not an outage. Because every layer retries independently, the SAME original customer request can be re-issued by up to all four layers at once, and those retries compound multiplicatively down the chain rather than simply adding up. Business invariants: customer requests may retry at exactly one controlled ownership boundary — retrying at multiple layers simultaneously is never acceptable, however well-intentioned each individual layer's retry policy is. The External Provider must be protected from multiplicative amplification — its connection capacity is shared with every other customer of ours, not just this request path. Cancellation and timeout propagation should prevent zombie work: once a request has been abandoned or has failed past its owning layer's retry budget, every downstream layer still processing it on its behalf must be told to stop, not left running to completion for no one. Diagnose which layer should be the one retry owner, and verify that the other layers have actually stood down — a visible improvement at one layer while amplification simply continues through the others is not a resolution.

Difficulty

Senior Staff

Expected Time

~20 min

Category

Microservices & Dependencies