During evaluation of a housing eligibility model, unexplained trigger phrases produce sudden approval flips that normal cases do not. What concern should the team document?
Select an answer to reveal the explanation.
Short Explanation
Weird trigger phrases that flip approvals are a backdoor smell—document and dig. Don't chalk it up to expected sampling variance, seasonal concept drift, or a minor calibration error; those issues don't explain rare trigger flips.
Full Explanation
During evaluation of a housing eligibility model, unexplained trigger phrases that produce sudden approval flips not seen on normal cases are indicators of possible backdoor or trojan behavior in a civic model. Backdoors activate on rare trigger patterns while appearing accurate on ordinary inputs, which makes them dangerous in benefits and eligibility contexts where a hidden phrase could force approvals or denials. The team should document the triggers, reproduction steps, affected cohorts, and training-data hypotheses, then escalate for dataset and training-pipeline forensics—not dismiss the anomaly as quirkiness.
Expected sampling variance across held-out validation folds produces normal fluctuation in approval rates, but it does not create rare trigger phrases that flip approvals on isolated cases. Routine concept drift from seasonal demographic shifts changes the overall applicant distribution gradually; it does not explain sudden, phrase-specific approval flips on otherwise normal cases. A minor calibration error from imbalanced approval labels would shift scores broadly across the population, not produce isolated approvals tied to rare trigger phrases. These distractors fail because they describe broad statistical, demographic, or calibration effects rather than anomalous model behavior under rare trigger inputs.
Exam caveat: unexplained triggers are a concern warranting investigation; they are not conclusive proof of an implanted trojan until training provenance and replication confirm it. Rule out preprocessing bugs, encoding issues, and leakage from evaluation scripts. Distinguish backdoors from ordinary prompt sensitivity or brittle NLP—backdoor framing fits when a rare, specific pattern flips decisions while normal cases stay stable.
Operational check: build a red-team suite of suspected triggers and near-miss paraphrases; measure approval delta versus matched clean cases; freeze promotion of the housing model until triggers are explained or removed via clean retraining. Audit annotation and augmentation stages for opportunities to insert trigger patterns. Record findings in the model card so civic reviewers see the residual risk.