A busy records-search API is healthy but occasionally slow under load. An aggressive liveness httpGet treats latency as death and restarts Pods, causing a thundering herd. What design lesson applies?
Select an answer to reveal the explanation.
Short Explanation
Liveness is the ejector seat, not the speedometer. If you fire it just because the app is busy, you restart healthy Pods and stampede the survivors. Save liveness for truly stuck processes; let readiness pull traffic when the instance is overloaded.
Full Explanation
When a busy records-search API is healthy but occasionally slow under load, an aggressive liveness httpGet that treats latency as death restarts Pods and can cause a thundering herd. The design lesson is to keep liveness conservative—detect deadlocks or truly unrecoverable failure—and use readiness for traffic shedding under load. Municipal search APIs under burst should shed and slow, not suicide-restart en masse.
Why the other options fail by concept: Lowering successThreshold on liveness to 0 is nonsensical/invalid as a strategy to avoid restarts. Pointing every liveness probe at the kube-apiserver /livez from app Pods couples app restarts to control-plane reachability. Setting liveness failureThreshold equal to the Deployment’s replica count is an arbitrary coupling with no sound health meaning.
Exam caveat: Timeouts that are too aggressive convert slowness into failed probes. Readiness failures remove Endpoints without killing processes—prefer that under load spikes. Investigate app performance and resources (CPU throttling) rather than only tuning probes. startupProbe remains separate from this steady-state concern.
Operational check: Lengthen liveness timeout/thresholds to true-death detection, move load sensitivity into readiness, load-test records-search, and confirm restart counts stay flat while Ready may flap briefly. Watch Service backends and error budgets during the test.