A bank lowered alert thresholds until ordinary transactions paged, and real incidents are now buried in that noise. Which step should the team take?
Select an answer to reveal the explanation.
Short Explanation
An alarm set so sensitive that toast sets it off will not tell you about a real fire. You prove the rule fires on a known bad condition and clears when that condition ends, instead of paging on ordinary transactions. Lowering thresholds further, deleting the rule, or trusting the save keeps the noise.
Full Explanation
Once ordinary transactions page, the rule no longer distinguishes a known bad condition from normal work, so more of the same tuning cannot restore that contrast. Proving the rule fires on a known bad condition and clears when that condition ends is the step that makes the rule usable again, because a page then means the known condition and a clear state means the condition is over. Lowering every remaining threshold until the metrics that are still quiet page as well extends the noise to the metrics that still had meaning. Deleting the rule after the latest page and stopping watching the condition it was built to catch removes coverage instead of restoring a testable rule. Trusting the rule because the lowered thresholds were saved and pages are now arriving treats volume of pages as validation. Pages caused by ordinary transactions do not show that a known fault fires the rule or that recovery clears it. Exam caveat: a rule that pages on normal activity has not been validated, no matter how many pages it sends. Operational check: compare one known bad interval with one known healthy interval and confirm the rule pages only for the bad interval and clears afterward.