During a coastal storm, a city’s emergency notification platform hit API and concurrent-execution throttles that delayed outbound alerts. Postmortem shows soft service quotas were never reviewed. Which reliability action best addresses the root cause?
Select an answer to reveal the explanation.
Short Explanation
Quotas are speed limits you only notice when the sirens are already blaring. Catalog the limits, raise them before hurricane season, and alarm on throttles so the next surge does not surprise you.
Full Explanation
During a coastal storm, an emergency notification platform hit API and concurrent-execution throttles that delayed outbound alerts because soft service quotas were never reviewed. Inventorying critical service quotas and account limits, requesting increases ahead of storm season, and alarming on throttle metrics before another surge addresses that root cause. That reliability action works because disaster-time scale is predictable enough to pre-clear quotas rather than discover limits while citizens need alerts. Ignoring quotas because AWS supposedly always raises them automatically during disasters without requests fails—many increases still require proactive asks. Deleting CloudWatch alarms related to throttling so dashboards stay green fails by hiding the signal that delayed alerts. Moving the platform to a single Region with no retry budget to simplify limits fails resilience and retry best practices. Exam caveat: quota increases are not instant and may need justification; also design retries with backoff so approved higher limits are not wasted on thundering herds. Operational check: list APIs used for notification fan-out, compare current quotas to last storm’s peak, submit increase requests, add throttle alarms with paging, and re-run a load drill that stays under the new ceilings.