A housing authority considers a black-box vendor scoring model for waitlist prioritization. What explainability concern should audit raise?
Select an answer to reveal the explanation.
Short Explanation
You can’t point to global feature importance on a vendor dashboard and call that a sufficient rationale for an individual waitlist appeal. Keeping input logs and vendor version numbers doesn’t fulfill transparency duties for a contested eligibility outcome. And aggregate accuracy or calibration metrics don’t give a household due-process notice of why it was ranked where it is; you still need a decision-level explanation residents can use to challenge the score.
Full Explanation
A housing authority using a black-box vendor model for waitlist prioritization makes high-impact eligibility and ranking decisions about residents. Audit should raise that such decisions need transparency sufficient for appeal, oversight, and fairness review. Applicants, hearing officers, and civic oversight bodies need enough explanation of features, weights, or decision rationales to challenge outcomes, detect disparate impact, and demonstrate that documented policy—not opaque vendor internals—drives priority.
Global feature importance from a vendor dashboard does not provide sufficient rationale for individual waitlist appeals; aggregate importance scores cannot explain why a particular applicant was ranked where they were or give them a basis to challenge that specific score. Retaining input logs and vendor version numbers does not fulfill transparency duties for contested eligibility outcomes; record-keeping is not the same as providing residents with a usable explanation or meaningful notice of the decision. Disclosing aggregate accuracy and calibration metrics does not address due-process notice for prioritized households; summary performance statistics cannot tell an individual household why it was prioritized or deprioritized.
Exam caveat: “explainability” on civic exams often means decision transparency proportionate to impact, not full disclosure of proprietary weights if equivalent human-readable rationale and appeal paths exist. Operational check: require model cards or decision notices residents can use, bias/fairness testing on protected classes relevant to housing policy, appeal workflows that capture override reasons, and contractual rights to sufficient explanation for oversight.