ML Inference Cost vs Quality Incident

An upstream request-complexity classifier has degraded, so the Routing Layer's default policy sends far more traffic to the expensive Large Model than genuinely necessary — monthly inference spend has exploded, but routing-layer uptime and inference latency stay fully healthy throughout. This is a COST problem wearing a routing-decision disguise, not an availability problem. Four business invariants govern every decision here: (1) quality-critical request classes must not be silently downgraded to the Small Model without an explicit, bounded policy decision; (2) cost reduction must be achieved by smarter routing, not blanket quality sacrifice; (3) a quality regression affecting real user-facing decisions must be trackable, never silent; (4) model-serving availability is not the primary objective. Decide whether to repair the actual broken classifier (slower, durable) or apply an explicit quality-tiering policy on top of it as an immediate stopgap, whether to pin quality-critical traffic to the Large Model as defense-in-depth, and whether the Small Model's quality tier and response caching are worth their own real, ongoing cost.

Difficulty

Principal

Expected Time

~25 min

Category

Data Pipelines