Multi-Service Memory Pressure Incident

A recent payload/data-shape change now causes the Aggregator to request and hold the full nested object from Dependency A, Dependency B, and Dependency C on every inbound request. Dependency A — the component with the least heap headroom in the fleet — is the first to visibly GC-thrash, but it is not necessarily where the problem actually lives. Customer requests may return reduced enrichment if necessary, but the service fleet must not enter restart/OOM churn. Any payload truncation may only occur within explicit correctness bounds — a response that is missing fields it is contractually required to carry is a correctness failure, not an acceptable tradeoff. Scaling the one service that happens to be failing loudest is not a durable fix if the actual upstream amplification is never addressed: diagnose which component is genuinely causing the memory pressure, apply a fix that reduces what is actually built and held in memory, and allow real elapsed time for GC and in-flight heap pressure to decay before declaring the incident resolved.

Difficulty

Senior Staff

Expected Time

~20 min

Category

Infrastructure & Capacity