A city AI strategy office is watching for bias drift in an automated code-enforcement prioritization tool as neighborhood demographics shift over time. How should the office treat bias monitoring for this tool?
Select an answer to reveal the explanation.
Short Explanation
Neighborhoods don't stay frozen in place, and neither should a fairness check. A tool trained on yesterday's demographics can quietly drift out of step as a city's population shifts, so watching for bias drift has to be a habit, not a one-time stamp of approval. Set it, forget it, and the tool ages badly without anyone noticing.
Full Explanation
Bias drift describes how a model's fairness properties can degrade over time as the population it operates on changes, even if the model itself was never retrained, because the real-world data it is scoring no longer resembles the data it was built or last validated against. For a code-enforcement prioritization tool operating in neighborhoods with shifting demographics, that means fairness has to be monitored continuously, with periodic re-evaluation of flagging rates across demographic groups. Treating fairness as a one-time check at launch assumes the population and context stay static, which is precisely the assumption that lets drift go undetected. Waiting for a formal discrimination complaint before checking makes bias monitoring reactive to harm that has already occurred, rather than a control that catches problems early. Assuming the vendor owns bias monitoring after signing ignores that the operating agency remains accountable for the tool's real-world impact on its own residents, regardless of who built it. Believing no monitoring is needed because the algorithm itself does not change misunderstands the mechanism: it is the data distribution shifting under a fixed model that causes drift, not the model's code changing. A scope caveat: monitoring cadence should be proportionate to how quickly a given neighborhood's demographics are actually changing. A concrete operational check is to compare quarterly flagging-rate breakdowns by neighborhood against the office's own equity baseline data.