Capacity Exhaustion During Peak Event
A scheduled peak sales event has driven a sharp, legitimate, fully-expected rise in traffic — this is demand you want, not an attack or an anomaly, and it cannot simply be blocked or rate-limited away wholesale. As pressure builds, cache effectiveness falls, the shared database approaches real saturation risk, the async backlog grows, and optional features degrade. Six business invariants govern every decision here: (1) critical purchase/transaction traffic must remain available as long as safely possible; (2) optional personalization/search/enrichment may be degraded or disabled; (3) the database must not be pushed into collapse; (4) accepted async work must remain durable; (5) traffic shedding must be policy-based, never arbitrary or random; (6) a bounded queue backlog is acceptable if it is demonstrably draining. The central judgment this scenario forces: what do you intentionally sacrifice to protect the business-critical path? Maximizing total throughput or aggregate success rate across every feature, rather than deliberately protecting checkout at the cost of optional features, is the wrong objective and must score worse, not better. Decide which traffic to prioritize, how to shed the rest responsibly, how to keep the database out of collapse, how to keep the async backlog bounded and draining rather than ignored, and whether to chase full recovery or hand back a genuinely safe degraded state.
- Difficulty
- Principal
- Expected Time
- ~30 min
- Category
- Infrastructure & Capacity
Skills
- Deliberately sacrificing optional capacity to protect a business-critical path
- Recognizing that raising a concurrency ceiling can transfer pressure downstream rather than remove it
- Distinguishing policy-based traffic shedding from blunt, proportional shedding
- Reading independent recovery-lag signals (cache warmup vs. DB connection pool recovery) rather than assuming one fix heals everything
System Architecture
- Clients
- Edge / CDN
- API Fleet
- Payment / Transaction Service
- Search / Personalization Service
- Cache
- Database
- Async Queue
- Async Workers
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents