Database Connection Saturation
- Category
- Databases & Storage
- Difficulty
- Senior
- Est. time
- ~15 min
A traffic spike exhausts the database connection pool while a batch job competes for the same capacity.
PostgresConnection PoolingReplicationCapacity PlanningBackpressure
- Category
- Messaging & Streaming
- Difficulty
- Intermediate
- Est. time
- ~12 min
A producer traffic spike and a batch of slow records push a Kafka consumer group into a critical backlog.
KafkaBackpressureConsumer GroupsIdempotencyObservability
- Category
- Microservices & Dependencies
- Difficulty
- Senior
- Est. time
- ~20 min
A degrading payment provider triggers a retry storm across checkout, payments, and settlement.
RetriesCircuit BreakersBackpressureKafkaPostgresIncident Response
Abuse-Prevention False Positive Storm
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Senior Staff
- Est. time
- ~18 min
A bad detection rule blocks real users while real abuse is also present — disabling protection fixes one problem by creating another.
Abuse PreventionRate LimitingFraud & Abuse ContainmentHistorical DamageFail-Open vs Fail-Closed
Analytics Warehouse Query Contention
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Senior Staff
- Est. time
- ~18 min
Ad-hoc BI/Data Science queries are starving a critical, deadline-bound scheduled report — scaling the warehouse alone will not fix it.
Query ContentionWorkload PrioritizationData WarehouseStakeholder TradeoffAdmission ControlCost Efficiency
API Gateway Saturation
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~16 min
The service fleet stays healthy and idle while the gateway in front of it saturates on its own per-request overhead — and scaling the backend does nothing about it.
API GatewayConnection ReuseRoutingCapacity PlanningAuthentication
API Migration Dual-Write Failure
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Principal
- Est. time
- ~30 min
Dual-write divergence grows silently behind a healthy dashboard — rollback alone never repairs it, and read routing can expose or lose acknowledged writes if sequenced wrong.
Migration SafetyDual-WriteData IntegrityReconciliationRollbackHistorical Damage
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~15 min
A legitimate traffic spike collapses the cache hit rate and saturates the database connection pool, and scaling the API fleet alone only makes it worse.
Traffic SpikeCachingConnection PoolingRate LimitingCapacity
API Version Sunset Breaks Silent Long-Tail Consumers
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Staff
- Est. time
- ~18 min
Sunsetting Orders API v1 breaks a long tail of internal consumers the team's own registry never knew existed.
API VersioningDeprecationConsumer DiscoveryBackward CompatibilityInternal Dependencies
Audit Log Integrity Failure
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Principal
- Est. time
- ~25 min
Audit events are missing, duplicated, and out of order while every availability dashboard stays green — publishing the next compliance report on schedule looks routine but silently certifies a picture the evidence doesn't support.
ComplianceAudit TrailEvidence IntegrityData GovernanceHistorical DamageImmutability
Authentication Service Degradation
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Staff
- Est. time
- ~18 min
The shared authentication service is degrading under load, and every dependent service feels the impact.
AuthenticationToken ValidationSecurityAvailability TradeoffsCapacity Planning
Authorization Policy Rollout Failure
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Principal
- Est. time
- ~25 min
A policy rollout causes selective false-denies while some caches across the fleet still serve the old policy version — fail-open fixes availability but creates permanent unauthorized access.
AuthorizationPolicy RolloutCache ConvergenceFail-Open vs Fail-ClosedHistorical DamageSecurity Correctness
Autoscaling Failure
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Senior
- Est. time
- ~14 min
A traffic spike overwhelms available capacity while the autoscaler either reacts too slowly or is structurally capped.
AutoscalingCapacity PlanningTraffic SpikesCost Tradeoffs
Backfill Overwhelms Production
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Senior Staff
- Est. time
- ~25 min
A historical backfill competes with live traffic for one shared database, and the tempting fixes are either to flood it or to kill it forever — neither is the safe answer.
Database ContentionBatch vs. OLTPCapacity BudgetingPressure TransferHistorical DamageRecovery Lag
Bad Configuration Rollout
🔒 Paid
- Category
- Deployments & Reliability
- Difficulty
- Intermediate
- Est. time
- ~10 min
A bad configuration push sets the API rate limit far below normal traffic, rejecting legitimate requests immediately.
Configuration ManagementFeature FlagsRollbackChange Management
- Category
- API & Traffic
- Difficulty
- Staff
- Est. time
- ~18 min
Automated traffic that looks almost exactly like real customers is rising fast, and a defense blunt enough to stop it will also catch genuine users.
Bot MitigationRate LimitingCAPTCHAFraud/AbuseEdge Security
Cache Eviction Storm
🔒 Paid
- Category
- Caching
- Difficulty
- Senior
- Est. time
- ~15 min
A deployment changed cached object size and key cardinality, causing eviction churn that more memory alone cannot durably fix.
CachingEviction PolicyCapacityCardinalityDatabase Protection
Cache Invalidation Across Regions
🔒 Paid
- Category
- Caching
- Difficulty
- Senior Staff
- Est. time
- ~25 min
The same invalidation stream is delayed in one region, duplicated in another, and healthy in a third — a global cache-disable fix overloads shared Origin far worse than a differentiated per-region response.
Multi-RegionCache InvalidationOrigin ProtectionBounded StalenessBlast-Radius IsolationRecovery Lag
- Category
- Caching
- Difficulty
- Intermediate
- Est. time
- ~10 min
A cache node fails and its keys redistribute, risking a database overload unless capacity or database protection is restored.
CachingDatabase ProtectionCapacityCache WarmingThundering Herd
Cache Revalidation Herd
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~15 min
A batch of popular cache entries shares one unjittered expiry, and every API instance races to refill the same keys from origin at once.
CachingCache InvalidationSingle-FlightTTLStale-While-RevalidateRetry Storms
- Category
- Caching
- Difficulty
- Senior
- Est. time
- ~15 min
A popular cache key expires under peak load, sending a thundering herd of duplicate requests straight to the origin database.
RedisConcurrencyTraffic SpikeThundering HerdTTL
Capacity Exhaustion During Peak Event
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Principal
- Est. time
- ~30 min
Legitimate peak-event traffic cascades from Edge/CDN through Cache, Database, and the Async Queue — protect checkout, sacrifice optional features deliberately, and never push the database into collapse.
Peak LoadCapacity PlanningTraffic TriagePressure TransferBusiness-Critical PrioritizationDegraded Mode
Cascading Failure Across Shared Platform
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Principal
- Est. time
- ~35 min
A shared-platform incident spanning three products and five shared dependency layers, where the loudest-complaining product is not the root cause and no single action resolves it.
Shared PlatformMulti-ProductRoot Cause AnalysisAmplificationPressure TransferHistorical DamageRecovery OrderingRestoration Sequencing
Cascading Service Failure
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~18 min
One deep service failure propagates backward through a synchronous call chain, making multiple healthy services look broken at once.
Circuit BreakersRoot Cause AnalysisMicroservicesResilienceBackpressure
Certificate Expiry Across Large Fleet
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Senior Staff
- Est. time
- ~25 min
A large certificate fleet spanning Modern, Critical, and Legacy cohorts approaches a shared expiry deadline — revoking the old certificate before Legacy's trust bundle is verified breaks it, even though the same eventual fix would have worked in the right order.
Certificate RotationDeadline RiskFleet OperationsSequencingHistorical DamageOperational Governance
Checkout, Payment & Inventory Redis Contention
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~18 min
Payment retries and Inventory latency both climb — the shared cause is a Redis connection pool neither service's own metrics point to directly.
RedisShared DependenciesCorrelationConnection PoolingCaching
Connection Pool Starvation
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Intermediate
- Est. time
- ~12 min
A deploy quietly shrinks the DB connection pool's max size — normal traffic alone is enough to exhaust it.
PostgresConnection PoolingConfiguration RegressionDeploysCapacity Planning
Consistency vs Availability Incident
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~20 min
A cross-region connectivity problem forces a real tradeoff between consistency and availability — and relaxing the wrong operation category turns healthy-looking availability into a silent correctness incident.
CAP TheoremConsistencyAvailabilityMulti-RegionQuorum
Consumer Rebalance Storm
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Staff
- Est. time
- ~15 min
A slow rolling deploy keeps triggering full consumer group rebalances, and the group never stabilizes long enough to drain its backlog.
KafkaConsumer GroupsRebalancingDeploymentsObservability
CORS Misconfiguration Blocks Partner Integrations
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Senior
- Est. time
- ~15 min
A CORS hardening change silently blocks legitimate partner API callers — the fix that caused the incident IS the incident.
CORSPartner IntegrationsSecurity HardeningAPI GatewayPreflight
Cron Job Overlap Double-Run
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Senior
- Est. time
- ~15 min
A nightly batch job's run duration outgrows its own schedule interval, so two instances run at once and double-write the same output rows — the fix is a missing concurrency guard, not more compute.
CronSchedulingConcurrency ControlIdempotencyBatch Jobs
Cross-Region Latency Spike
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~15 min
Inter-region latency has spiked while both regions stay individually healthy, and the candidate must decide which traffic can trade consistency for speed without touching payment authorization.
Cross-RegionConsistencyAvailabilityLatencyReplicationMulti-Region
Cross-Region Replication Cost Explosion
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Senior Staff
- Est. time
- ~20 min
A replication-policy change exploded cross-region cost while everything stays healthy — real savings exist, but one tempting lever quietly breaches a durability floor.
ReplicationCost OptimizationDurabilityDisaster RecoveryMulti-Region
Cross-Service Retry Amplification
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Senior Staff
- Est. time
- ~20 min
Four independently-configured retry layers are each re-issuing the same degraded-provider request, multiplying effective load far beyond original customer traffic.
RetriesCircuit BreakingTimeoutsBackpressureCancellationOwnership
Cross-Team Schema Contract Drift Breaks Downstream Pipeline
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Senior Staff
- Est. time
- ~25 min
One backward-incompatible schema change fans out into three downstream pipelines, each silently breaking in its own distinct way.
Schema CompatibilityEvent StreamingData ContractsTriageMulti-Consumer Fan-Out
Data Lineage Break During ETL Migration
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Principal
- Est. time
- ~25 min
An ETL v1-to-v2 migration silently mixes lineage across two computation paths while every availability metric stays perfectly healthy — flipping the cutover globally looks fast but risks corrupting every downstream report at once.
Data LineageETL MigrationData TrustAnalytics PlatformHistorical DamageCost Efficiency
Data Pipeline Freshness Collapse
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Senior Staff
- Est. time
- ~25 min
Stream processing falls behind while ingestion stays healthy, and the obvious fix — scaling the processor — floods the Warehouse with write contention unless it's protected first.
Stream ProcessingData FreshnessTiered SLOsPressure TransferHistorical DamageBacklog Drain
Data Retention Policy Misconfiguration
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Principal
- Est. time
- ~25 min
A misconfigured retention policy is wrongly deleting Financial Records while every availability dashboard stays green — pausing all deletion looks safe but breaches an unrelated data class's own deletion deadline.
ComplianceData RetentionLegal HoldData GovernanceHistorical DamagePolicy Correctness
Database Lock Contention
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Senior
- Est. time
- ~14 min
A batch job's long-held row locks on a hot table block normal writes behind it, piling up blocked transactions.
PostgresLockingConcurrencyBatch ProcessingCapacity Planning
Dead-Letter Queue Explosion
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Senior
- Est. time
- ~18 min
Events are failing parsing and piling into the DLQ while a reprocessor replays them concurrently — and discarding the backlog looks like a fix but is really irreversible data loss.
KafkaDead-Letter QueueReplayData IntegritySchema Compatibility
- Category
- Databases & Storage
- Difficulty
- Senior
- Est. time
- ~16 min
Two services acquiring database locks in inconsistent order are deadlocking each other on every concurrent checkout.
PostgreSQLDeadlocksTransactionsLockingConcurrency
Dependency Rate-Limit Exhaustion
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Senior
- Est. time
- ~15 min
A third-party provider's hard request quota is being exhausted before its rolling window resets, and retrying harder only makes it worse.
Rate LimitingThird-Party DependencyCachingRetry BehaviorQuota Management
Dependency Timeout Cascade
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Staff
- Est. time
- ~20 min
A deep dependency has slowed down, and misaligned timeouts across four hops are letting blocked connections cascade all the way up to the API.
TimeoutsCircuit BreakerRetriesThread PoolsCascading Failure
Deployment Regression
🔒 Paid
- Category
- Deployments & Reliability
- Difficulty
- Intermediate
- Est. time
- ~12 min
A new deploy ships a subtle regression; error rates and latency climb on affected endpoints shortly after rollout.
RollbackCanary DeploymentsRegressionIncident Response
Deployment Regression
🔒 Paid
- Category
- Deployments & Reliability
- Difficulty
- Senior
- Est. time
- ~18 min
A partial rollout's new version is regressing for the users routed to it, and a full rollback isn't the only defensible way to contain it.
Canary DeploymentsFeature FlagsRollbackTraffic ShiftingBlast Radius
Distributed Coordination Failure
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~18 min
A coordination service's ownership table has gone stale, causing some work items to be claimed by two workers at once and others to be claimed by none.
CoordinationSchedulingOwnershipFencingReconciliationData Integrity
Distributed Lock Failure
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~15 min
Lock leases are expiring while workers are still processing, letting two workers believe they own the same lock at once — and the obvious fix of lengthening the lease only hides the problem.
Distributed LockingFencing TokensLeasesConcurrencyData Integrity
Downstream Dependency Failure
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Senior
- Est. time
- ~12 min
A failing downstream service exhausts the calling service's own capacity because every blocked request ties up a thread waiting on a dead dependency.
Circuit BreakersResilienceFallbacksIsolationTimeouts
Duplicate Event Processing
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Staff
- Est. time
- ~20 min
Acknowledgement failures are causing redelivery, and duplicate order/payment side effects keep accumulating even while throughput looks healthy.
KafkaIdempotencyDeduplicationCorrectnessAt-Least-Once Delivery
Duplicate Request Processing
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~20 min
Client and gateway retries against a slow dependency are creating the same order more than once, even while retries look like they are succeeding.
IdempotencyHTTP RetriesCorrectnessRequest DeduplicationAPI Gateway
Emergency Capacity Reservation Decision
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Principal
- Est. time
- ~25 min
A major demand event is approaching and the reservation deadline is closing — decide how much expensive, non-refundable capacity to lock in, and where, before real demand ever arrives.
Capacity PlanningProactive DecisionForecast UncertaintyRegional ResilienceCost DisciplineDeadline Risk
Enterprise-Wide Dependency Deprecation
🔒 Paid
- Category
- Deployments & Reliability
- Difficulty
- Principal
- Est. time
- ~30 min
Four products sit at four different migration stages against a hard external deprecation deadline — every live dashboard reads healthy while marking everything 'migrated' without verification, or waiting because nothing looks broken yet, quietly becomes the real incident.
Migration GovernanceDeprecationDeadline RiskMulti-Team CoordinationVerificationTechnical DebtRollback Readiness
Event Platform Partial Outage
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Senior Staff
- Est. time
- ~22 min
A degraded broker partition class, a lagging consumer group, and a retry storm against a degraded downstream all hit at once — scaling consumers alone transfers pressure downstream instead of fixing anything.
KafkaEvent StreamingConsumer GroupsPressure TransferDurabilityRetry Ownership
Event Processing Backpressure
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Staff
- Est. time
- ~20 min
Producers are outpacing the processing layer, and scaling consumers to drain Kafka can quietly overload the database or the external API instead.
KafkaBackpressureConcurrencyCapacityRate Limiting
Expensive Retry and Logging Amplification
🔒 Paid
- Category
- Deployments & Reliability
- Difficulty
- Senior Staff
- Est. time
- ~20 min
A bad deploy quietly tripled observability and dependency-request cost with no outage — fixing it the cheap way by disabling audit logs looks great on cost but is a compliance trap.
Cost EfficiencyObservabilityRetry AmplificationLoggingTelemetryAudit LoggingInternal Tooling
Fan-Out Request Explosion
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Staff
- Est. time
- ~20 min
One of five parallel downstream calls degrades under a new fan-out, and relieving it safely means knowing which dependency is optional and which mitigation quietly pressures a core one.
Fan-OutAggregationTimeoutsPartial ResponsesCapacityCaching
Fraud Control Degradation During Payment Surge
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Principal
- Est. time
- ~22 min
A legitimate peak traffic event meets a degraded fraud-risk provider — bypassing fraud review looks like it fixes checkout, but it doesn't.
Fraud PreventionRisk PolicyPaymentsPeak TrafficHistorical DamageFail-Open vs Fail-Closed
Geo-DNS Failover Flapping
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~15 min
GeoDNS keeps failing traffic back and forth between two individually-healthy regions because its health check is too sensitive to transient latency blips.
DNSFailoverHealth ChecksHysteresisMulti-Region
Global Checkout Brownout
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Principal
- Est. time
- ~20 min
Three moderately degraded dependencies compose on the synchronous checkout path into an unusable incident — nothing is down, but checkout still fails.
CheckoutCritical PathLatency CompositionRetry AmplificationDegraded Mode
Global Feature Rollout Incident
🔒 Paid
- Category
- Deployments & Reliability
- Difficulty
- Principal
- Est. time
- ~25 min
A global feature rollout diverges by region and writes incompatible records into a shared data store — rollback stops new damage but never repairs what was already written.
Feature RolloutMulti-RegionData CompatibilityHistorical DamageSequencingCorrectness
Graceful Shutdown Drops In-Flight Requests
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Senior
- Est. time
- ~15 min
Rolling deploys drop a steady trickle of requests — pods are killed before their shutdown hook finishes draining in-flight work.
DeploysGraceful ShutdownKubernetesRequest DrainingRollout Strategy
Hot Database Partition
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Staff
- Est. time
- ~20 min
A handful of high-volume tenants are overloading one database partition while the rest of the cluster sits idle, and scaling broadly only makes it worse.
ShardingPartitioningMulti-TenancyRoutingCapacity
Hot Shard + Replica Lag + Failover Risk
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Principal
- Est. time
- ~22 min
A hot shard and its lagging replica interact with a tempting failover decision -- promoting too early strands acknowledged writes that reconciliation alone cannot undo instantly.
ShardingReplication LagFailover RiskData IntegrityPressure TransferHistorical Damage
Hot-Key Cache Overload
🔒 Paid
- Category
- Caching
- Difficulty
- Senior
- Est. time
- ~15 min
One popular key is overloading a single cache shard while the rest of the cluster stays idle, and adding more cache capacity elsewhere does not help.
CachingHot KeyShardingRequest CoalescingCapacity
Idempotency Failure
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~20 min
The idempotency store itself is degrading, and duplicate (or silently dropped) side effects keep accumulating even while throughput looks healthy.
IdempotencyDeduplicationCorrectnessFail-Open vs Fail-ClosedKey Scoping
Identity Platform Degradation
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Principal
- Est. time
- ~22 min
The MFA Provider degrades and Session Store pressure rises at the same time — a broad MFA bypass looks like it fixes everything, but it doesn't.
AuthenticationMFASession ManagementRisk-Based PolicyHistorical DamageFail-Open vs Fail-Closed
Kafka Partition Skew
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Staff
- Est. time
- ~18 min
A hot tenant key concentrates traffic on one Kafka partition, which falls behind even though aggregate consumer capacity looks fine.
KafkaPartitioningHot KeyScalingObservability
Large Payload Regression
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~15 min
A response shape change is returning full nested records instead of references, and scaling the API tier alone cannot shrink what each request now carries.
Payload SizeSerializationBandwidthResponse ShapeCapacity
Leader Election Storm
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~15 min
Intermittent network latency on one node is triggering repeated, disruptive leader elections — and the obvious 'raise the timeout' fix has its own real cost.
ConsensusLeader ElectionQuorumRaftNetwork Jitter
Ledger Reconciliation Divergence
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Principal
- Est. time
- ~25 min
A subset of financial events is missing, duplicated, or delayed between two ledgers while customer-facing traffic stays healthy — fixing the cause doesn't undo the backlog it already created.
Financial CorrectnessLedger ReconciliationHistorical DamageEvent StreamingSettlementData Integrity
Load Balancer Imbalance
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~16 min
Two of four healthy instances are carrying most of the load because of sticky-session pinning, and adding more instances alone won't relieve them.
Load BalancingSticky SessionsHealth ChecksTraffic RoutingCapacity
- Category
- Infrastructure & Capacity
- Difficulty
- Intermediate
- Est. time
- ~15 min
A deploy-introduced memory leak is climbing steadily toward an OOM crash, and the real fix can't ship live.
Memory ManagementObservabilityProcess SupervisionOOM
Message Retry Storm
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Senior
- Est. time
- ~15 min
A degraded downstream dependency triggers retry amplification that grows an async queue's backlog without bound.
QueuesRetriesBackpressureDead Letter QueueCircuit Breaker
Missing Database Index
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Intermediate
- Est. time
- ~12 min
A recently deployed query path is scanning the entire orders table as traffic climbs, and scaling the API layer alone only makes it worse.
PostgreSQLIndexingQuery PerformanceCapacityConnection Pooling
ML Inference Cost vs Quality Incident
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Principal
- Est. time
- ~25 min
Inference cost explodes as a degraded classifier misroutes traffic to the expensive model — routing everything to the cheap model looks free but silently sacrifices quality on traffic that must never be downgraded.
ML PlatformInference CostDecision QualityRouting PolicyCost EfficiencyHistorical Damage
Model Feature Freshness Degradation
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Principal
- Est. time
- ~25 min
Feature freshness degrades while model-serving uptime stays perfectly healthy — silently serving stale features to critical decisions looks free but is genuinely unsafe.
Feature FreshnessModel ServingDecision QualityML PlatformHistorical DamageCost Efficiency
Multi-AZ Storage Failure During Live Migration
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Staff
- Est. time
- ~20 min
AZ-A's volumes degrade mid-migration, and the migration controller's own retries cascade the overload onto the one AZ that was still healthy.
Multi-AZStorage MigrationCascading FailureRetry StormsCapacity
Multi-Region Payment Platform Degradation
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Principal
- Est. time
- ~25 min
EU dependency degradation and cross-region ledger lag interact with traffic shifting — fixing EU availability can quietly overload US ledger/settlement capacity instead.
Multi-RegionPaymentsLedger CorrectnessPressure TransferFraud ControlsSettlement
Multi-Service Memory Pressure Incident
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Senior Staff
- Est. time
- ~20 min
Dependency A is the first service to visibly GC-thrash, but the real memory-pressure source is upstream at the Aggregator — scaling the loud symptom only buys temporary, fading relief.
Memory PressureGCAggregationRoot Cause AnalysisRecovery LagPressure Transfer
Multi-Tenant Control Plane Overload
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Senior Staff
- Est. time
- ~20 min
The data plane looks fine while one tenant's unthrottled config churn quietly saturates the shared control plane underneath it.
Multi-TenancyControl PlaneFairnessCapacity IsolationBlast Radius
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~20 min
Both regions keep serving traffic normally during a partition — but ledger writes and profile writes carry very different consistency requirements, and conflating them is the trap.
Network PartitionConsistencyWrite RoutingMulti-RegionCAP Tradeoffs
Orphaned Background Worker Queue Growth
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Staff
- Est. time
- ~16 min
Worker pods crash-loop on a bad deploy and consumer count hits zero — queue depth grows unbounded until both the deploy and the backlog are fixed.
QueuesBackground WorkersDeploysConsumer LagCrash Loop
Out-of-Order Events
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Staff
- Est. time
- ~20 min
A routing change broke per-account event ordering, and throughput looking healthy doesn't mean the materialized state is trustworthy.
KafkaEvent OrderingPartitioningData IntegrityReconciliation
Overprovisioned Resilience During Low-Risk Period
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Staff
- Est. time
- ~15 min
The high-risk period has ended and the platform is still paying for peak-event resilience it no longer needs — reduce cost to the stated baseline without quietly dropping below the mandatory resilience floor.
Cost EfficiencyCapacity PlanningResilience FloorProactive ReviewMulti-AZInternal Cost Review
Partial Regional Dependency Failure
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Staff
- Est. time
- ~20 min
A shared dependency is healthy in one region, slow in another, and intermittently failing in a third — all at once — and a uniform global fix is not automatically the right call.
Multi-RegionCircuit BreakerFallbackDependency HealthTraffic Routing
Platform-Wide Clock Skew Breaks Distributed Consensus
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Principal
- Est. time
- ~25 min
Lock service, ledger, and rate limiter all break at once with no shared edge between them — the real cause is a silently-failing NTP sync job several hops upstream.
Clock SkewNTPConsensusCausal UnificationLeader ElectionEvent SourcingRate Limiting
Poison Message Loop
🔒 Paid
- Category
- Messaging & Streaming
- Difficulty
- Intermediate
- Est. time
- ~12 min
A malformed message loops forever in retry, blocking every healthy message queued behind it.
QueuesDead Letter QueueRetriesReliabilityIdempotency
Primary Database Failover
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Staff
- Est. time
- ~15 min
The primary database is failing and a replica must be promoted — but promoting it too aggressively risks stale writes or two nodes accepting writes at once.
PostgreSQLFailoverReplicationHigh AvailabilityData Integrity
Privacy Deletion Backlog
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Senior Staff
- Est. time
- ~22 min
Deletion requests are accepted successfully while downstream execution falls behind across three stores — marking a request complete before every store has actually finished creates a false completion that looks identical to a real one.
PrivacyData DeletionComplianceAuditabilityHistorical DamageExecution Completeness
Privilege Escalation via Stale Authorization Cache
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Principal
- Est. time
- ~25 min
A privilege revocation has not yet converged across the cache fleet — some caches still grant it while others correctly deny it, and fail-open fixes availability while actively granting unauthorized access.
AuthorizationPrivilege RevocationCache ConvergenceFail-Open vs Fail-ClosedHistorical DamageSecurity Correctness
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~20 min
A 5-node consensus group edges toward — and may cross — the threshold for safe quorum, and write availability can look perfectly healthy while writes are being accepted without a real majority.
ConsensusQuorumNode AvailabilityMembershipCorrectness
- Category
- API & Traffic
- Difficulty
- Intermediate
- Est. time
- ~12 min
Naive immediate retries on 429 responses are extending a payment provider's rate-limit penalty window instead of clearing it.
Rate LimitingBackoffRetriesThrottling
Rate-Limit Misconfiguration
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~16 min
A newly deployed rate-limit policy throttles legitimate tenants while letting one high-volume tenant dominate shared capacity — and an emergency bypass can hide the problem without fixing it.
Rate LimitingMulti-TenancyFairnessAPI GatewayQuotas
- Category
- Databases & Storage
- Difficulty
- Senior
- Est. time
- ~14 min
A write-heavy batch job pushes read replica lag into a critical backlog, serving stale data to consistency-sensitive consumers.
ReplicationConsistencyPostgresRead ScalingIdempotency
Regional Failover Causes Global Overload
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Principal
- Est. time
- ~20 min
Region A has failed and automatic failover is shifting its traffic into Region B and Region C — which are healthy, but not sized to absorb all of it without becoming a second failure.
Multi-RegionFailoverCapacity PlanningLoad SheddingBlast Radius
Regulatory Data-Residency Violation During Cross-Region Failover
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Principal
- Est. time
- ~22 min
An automatic cross-region failover protects availability but silently routes EU customer data through a US region — fail back now, stay failed over, or wait for a clean resolution, each with real consequences.
Data ResidencyGDPRCross-Region FailoverComplianceAvailability Tradeoffs
Replication Backlog
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~18 min
Write volume keeps outrunning replication throughput, and the backlog keeps growing — protecting consistency, read latency, and ingest throughput all pull in different directions.
ReplicationConsistencyThroughputCapacity PlanningBackpressure
Request Timeout Cascade
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Staff
- Est. time
- ~20 min
A slow dependency triggers Gateway retries that never cancel the original request, so abandoned work piles up across four hops even after any single timeout gets tightened.
TimeoutsRetriesCircuit BreakerRequest CancellationThread Pools
Retry Budget Exhaustion
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Intermediate
- Est. time
- ~10 min
A client service's unbudgeted per-call retries multiply load onto an already-degraded downstream dependency.
RetriesCircuit BreakersLoad AmplificationCapacity Planning
Runaway Cloud Spend During Healthy Traffic
🔒 Paid
- Category
- Infrastructure & Capacity
- Difficulty
- Senior Staff
- Est. time
- ~18 min
The system is fully healthy, but infrastructure spend is far higher than it needs to be — cut cost without quietly breaching the redundancy floor.
Cost OptimizationAutoscalingCapacity PlanningRedundancyProactive / No-OutageHistorical Damage
Schema Migration Under Live Traffic
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Principal
- Est. time
- ~20 min
Old and new app versions are writing incompatible row shapes to one shared database mid-migration, and rollback is not automatically safe.
Schema MigrationExpand/ContractRollbackBackfillData Integrity
Search Platform Cascade
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Senior Staff
- Est. time
- ~20 min
Ranking Service latency has risen moderately, and the Query Coordinator's held-open connections are compounding that into index-cluster, feature-store, and API-tail-latency pressure all at once.
SearchLatency CascadeDegraded ModeLoad SheddingPressure Transfer
Secret Leakage Containment Incident
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Principal
- Est. time
- ~20 min
A credential is found exposed in logs and a CI/CD artifact — doing nothing is invisible but unsafe, and revoking it broadly breaks production.
Secrets ManagementIncident ResponseContainment SequencingCredential RotationAuditabilityHistorical Damage
Service Discovery Failure
🔒 Paid
- Category
- Microservices & Dependencies
- Difficulty
- Senior
- Est. time
- ~15 min
The service registry starts handing out stale, unhealthy endpoints — and retrying harder only hits the same dead instances.
Service DiscoveryRetriesFallback RoutingHealth ChecksReliability
Service-to-Service Credential Rotation Failure
🔒 Paid
- Category
- Security & Identity
- Difficulty
- Senior Staff
- Est. time
- ~20 min
A service-to-service credential rotation goes wrong when the old credential is revoked before every workload cohort has adopted the new one — disabling authentication fixes availability but creates a permanent authentication-bypass violation.
Credential RotationAuthenticationSequencingBlast RadiusHistorical DamageSecurity Correctness
Session Affinity Loss After Cache Node Failover
🔒 Paid
- Category
- Caching
- Difficulty
- Staff
- Est. time
- ~15 min
A cache node failover leaves the load balancer routing against a stale topology, silently bouncing logged-in users between inconsistent session states.
CachingSession AffinityConsistent HashingFailoverCorrectness
Slow Query Regression
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Senior
- Est. time
- ~14 min
A deploy-induced query plan regression drops a hot query's index usage, pushing database CPU into a critical full-table-scan spiral.
PostgresQuery PlanningIndexingPerformanceCapacity Planning
Split-Brain Incident
🔒 Paid
- Category
- Distributed Systems
- Difficulty
- Staff
- Est. time
- ~20 min
Both regions may believe they hold write leadership during a network partition — and availability can look perfectly healthy on both sides while writes silently diverge.
Split BrainQuorumFencingConsistencyMulti-Region
Stale Feature-Flag Kill Switch
🔒 Paid
- Category
- Deployments & Reliability
- Difficulty
- Intermediate
- Est. time
- ~12 min
An unrelated, seemingly-safe flag flip wakes a long-dormant legacy pricing engine — reverting the flag that triggered it doesn't put it back to sleep.
Feature FlagsFlag HygieneDead CodeKill SwitchDormant Bug
Storage IOPS Saturation
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Senior
- Est. time
- ~12 min
A background compaction job is saturating storage IOPS, and the fastest customer-facing relief comes at the direct cost of the batch job's own progress.
StorageIOPSCapacityCheckpointingBatch Processing
Third-Party API Latency
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Intermediate
- Est. time
- ~12 min
An external shipping-rate vendor is slow, not down — every checkout request blocks on it until you insulate your own system.
Third-Party APIsTimeoutsCachingFallbacksResilience
- Category
- API & Traffic
- Difficulty
- Senior
- Est. time
- ~12 min
An entire client fleet's synchronized schedule fires at once, sending a massive instantaneous spike at the API gateway.
Traffic SpikesJitterLoad SheddingSynchronized ClientsRate Limiting
Training Pipeline Produces Corrupted Model Artifact
🔒 Paid
- Category
- Data Pipelines
- Difficulty
- Principal
- Est. time
- ~25 min
A corrupted training dataset produced a technically valid model artifact that is already serving — deleting the evidence to clean up looks fast but destroys the forensic record for good.
ML PlatformModel RegistryArtifact PromotionLineage & AuditabilityHistorical DamageRollback
Uneven Tenant Traffic
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Staff
- Est. time
- ~18 min
One tenant's traffic surge is consuming shared capacity and degrading every other tenant, and scaling the shared fleet alone never restores fairness.
Multi-TenancyFairnessRate LimitingCapacity PlanningTenant Isolation
Webhook Delivery Storm
🔒 Paid
- Category
- API & Traffic
- Difficulty
- Staff
- Est. time
- ~20 min
A shared downstream outage is making a subset of customer webhook endpoints unreachable, and uncontained retries against them are degrading delivery to everyone else too.
WebhooksRetriesCircuit BreakersMulti-TenancyBackpressure
Write Amplification from Misconfigured Storage Compaction
🔒 Paid
- Category
- Databases & Storage
- Difficulty
- Senior Staff
- Est. time
- ~20 min
Disk IOPS saturate under routine traffic — but the real cause is a compaction setting misconfigured three weeks earlier, not the traffic itself.
Storage EngineCompactionWrite AmplificationLSM TreeIOPSConfiguration Drift