🔒 Paid
Multi-AZ Storage Failure During Live Migration
Midway through a live storage-tier migration, Availability Zone A's volumes start throwing I/O errors. That alone wouldn't take anything down — but the migration controller's own retry logic keeps re-attempting the failed copy operations against AZ-C, the one zone that was still fully healthy and already serving live production traffic. AZ-C's connection pool starts saturating, and customer-facing latency and error rates climb through the shared storage router. Stop the cascade at its source and get live traffic safely off the degraded zone — without accelerating the very overload you're trying to relieve.
- Difficulty
- Staff
- Expected Time
- ~20 min
- Category
- Databases & Storage
Skills
- Recognizing a retry loop amplifying one failure into a second, distinct failure
- Distinguishing a hardware/infra degradation from a process that a restart can fix
- Sequencing remediation so a failover relieves load instead of doubling it
- Verifying recovery across both the source failure and its cascaded effect
System Architecture
- App Service
- Storage Router
- Storage AZ-A (migration source)
- Storage AZ-B (migration target)
- Storage AZ-C (remaining healthy replica)
- Migration Controller
This is a Pro scenario. Upgrade to Pro to unlock it.
Upgrade to ProBack to Incidents