Expensive Retry and Logging Amplification

A recent deploy raised retry frequency, logging verbosity, telemetry duplication, and payload logging size across the API and Worker. System health remains acceptable throughout — this is not an outage and the External Dependency itself is not meaningfully degraded. Instead, logging cost, telemetry ingestion cost, dependency request cost, and storage cost have risen dramatically. Five business invariants govern every decision here: (1) security/audit-required logs must remain intact; (2) operational debugging signal must remain sufficient; (3) retries must preserve correctness and bounded reliability; (4) duplicate and noisy logs may be reduced freely; (5) cost reduction must never blind operators or violate audit requirements. Decide where observability spend is genuinely valuable and where it is pure waste, which layer should own retries, and whether a bounded-cost partial fix is worth handing back versus pursuing the full cost floor.

Difficulty

Senior Staff

Expected Time

~20 min

Category

Deployments & Reliability