A CPU alarm on a fleet flips between OK and ALARM every evaluation period, and on-call is paging on each flip. The metric really does bounce around the threshold. Deleting the alarm was proposed. What should they change so a brief bounce does not page?
Select an answer to reveal the explanation.
Short Explanation
The metric is wiggling across the line, and the alarm treats every wiggle as a fire. You ask for more datapoints over the line before it pages. Deleting the alarm removes the warning, and dropping the threshold makes the page permanent.
Full Explanation
Datapoints to alarm, set as M out of N periods, keeps a single noisy period from entering ALARM. That matches a metric that genuinely bounces around the threshold. Deleting the alarm stops the noise by stopping detection. A threshold under the lowest normal CPU stays in ALARM. Turning off detailed monitoring changes the period; it does not require a sustained breach. Caveat: a longer evaluation delays a real incident, so pick M and N that still page in time for this fleet. Operational check: the alarm should stay OK through a one-period spike and enter ALARM only after the sustained breach you configured. Set the alarm to something like 3 datapoints out of 5, tuned to how fast this fleet can actually be in trouble. One period over the line then stays inside OK. Deleting the alarm leaves the bounce invisible. A threshold under the normal minimum is a page that never clears. Detailed monitoring changes resolution and can make flaps more visible, which is the opposite of deleting the metric. Watch one noisy hour and confirm a single spike does not page.