A payments API pages on-call whenever the error count is at least one in five minutes. Overnight, a single failed call pages them even though almost nobody is using the API. They want a page only when errors are a large share of requests. What should the alarm use?
Select an answer to reveal the explanation.
Short Explanation
One spilled drink in an empty diner is not a dinner rush. You alarm on errors divided by requests, so a quiet hour with a single failure stays quiet. Dropping the raw count threshold to zero makes the page even easier to trip.
Full Explanation
A raw error count treats one failure the same at midnight as at noon, so lowering its threshold from one to zero pages on every error, including the quiet-hour failure you want to ignore. A metric-math expression divides the error metric by the request metric and alarms on that ratio, matching your goal of paging only when errors are a large share of requests. An S3 bucket notification counts objects, not API results, and a Route 53 health check with no relationship to the API’s error metric does not compute the error rate. Caveat: guard the expression when requests are zero, or the ratio is undefined and the alarm can sit in insufficient data; FILL or a minimum request count belongs in the math. Use the same period and statistic for both metrics so the ratio does not compare different windows. Practical check: graph the expression over last night’s single error and confirm it stays under the threshold, then confirm a real error spike crosses it. Keep the raw error count on the dashboard to see that overnight failure without paging, and test with both a quiet hour and a replay of a bad deploy. An error rate of a few percent is a different page from a count of one.