An emergency management agency's AI resource-dispatch model underperforms specifically during unusual event types, even though its aggregate accuracy metrics look strong. How should the agency mitigate this reliability risk?

Aggregate accuracy metrics summarize performance across the full range of scenarios a model encounters, and because unusual event types are, by definition, a small share of total cases, poor performance on them can be mathematically diluted into an overall score that still looks strong. Directing mitigation that specifically targets edge-case underperformance means the agency evaluates and improves the model's behavior on unusual scenarios directly, rather than inferring safety from a number that mostly reflects routine cases. Concluding that no mitigation is needed because aggregate metrics look strong is precisely the trap this scenario describes: it treats an average as if it were evidence about every subgroup, when the disclosed pattern shows the opposite. Increasing routine, everyday-scenario testing reinforces performance on cases that are already handled well while leaving the actual weak spot, unusual events, unexamined. Retraining staff on dispatch procedures addresses human process rather than the model's demonstrated behavior gap, and it does nothing to correct why the model itself underperforms on unusual inputs. A scope caveat: edge-case mitigation may require gathering more training examples of unusual events specifically, which can be genuinely scarce data to collect. A concrete operational check is to break out performance metrics by event type, rather than reporting only an aggregate figure, so unusual-event performance is visible on its own.