Search Platform Cascade

Ranking Service latency has risen moderately — it has not failed outright. The Query Coordinator now waits longer on every query for a ranking result, holding its connections to Index Cluster A, Index Cluster B, and the Feature Store open for the extended duration. This single, moderately-degraded internal dependency compounds through the Coordinator into three distinct, simultaneously growing downstream pressure points: Index Cluster A/B connection pressure, Feature Store QPS, and Search API tail latency. Core search and index data remain healthy throughout. Business rules for this incident: basic search results are business-critical and must remain available; advanced ranking and re-ranking are optional during an incident — degrading or disabling them is an acceptable, legitimate response, not a failure; search correctness must never be sacrificed by returning stale or unrelated content outside the allowed bound, even to make the system look fast or available; and feature enrichment may be degraded. Decide how much of the ranking/enrichment path to keep, how to protect Index Cluster A/B and the Feature Store from the Coordinator's own held-connection pressure, and how — and whether — to bring ranking back without re-triggering the same pressure you just relieved.

Difficulty

Senior Staff

Expected Time

~20 min

Category

Distributed Systems