A brittle synchronous chain across permitting, payments, and document services caused cascading outages when one dependency slowed. Which reliability redesign best reduces cascade risk?
Select an answer to reveal the explanation.
Short Explanation
When payments sneeze, permitting catches pneumonia—that is cascade failure. Put queues and bulkheads between the pieces so one slow clerk does not shut the whole permit counter.
Full Explanation
A brittle synchronous chain across permitting, payments, and document services caused cascading outages when one dependency slowed. Decoupling non-critical steps with queues or events and isolating failure domains via timeouts, backoff, and bulkheads reduces cascade risk so one slow dependency cannot take down the entire permitting path. That reliability redesign works because synchronous fan-out turns every dependency’s latency into the citizen’s latency and every outage into a shared outage. Adding more synchronous retries with no backoff and no circuit breaking across every hop fails by amplifying load on the struggling dependency. Running all three services in one process on a single EC2 instance fails availability even if it eliminates some network calls. Sharing one database connection pool string across every service for tighter coupling fails conceptually by concentrating failure and saturation. Exam caveat: payments authorization may remain synchronous with strict timeouts and circuit breakers; decoupling does not mean dropping financial integrity—classify what must be online versus eventual. Operational check: introduce timeouts/circuits on each synchronous edge, move document generation to a queue, inject latency in the document service during a drill, and confirm permitting submit still succeeds within SLO.