A parks department wants a vision model for trail-condition photos. What should audit check before accepting the pilot metrics?
Select an answer to reveal the explanation.
Short Explanation
Trail photos need a scoreboard everyone agreed on—overall training accuracy doesn't prove the model is safe for trail decisions. Lock labeling and pass/fail criteria to real maintenance decisions before calling the pilot a win.
Full Explanation
Before accepting pilot metrics for a parks department vision model on trail-condition photos, audit should check that defined performance metrics and acceptance criteria are tied to trail-safety and maintenance decisions. Photos drive work orders and hazard response; "looks good" is not a control. Labeling standards, class definitions (washout, debris, signage damage), precision/recall or cost-sensitive thresholds, and agreement on false-negative tolerance for safety-relevant classes must exist before declaring success.
Reporting overall accuracy on training photos using vendor-recommended thresholds lacks objectivity and reproducibility for trail-safety decisions, since training performance doesn't validate real-world hazard detection. Verifying image upload latency and encrypted storage for ranger field devices checks infrastructure and security, not whether the model's predictions are accurate or safe enough to drive maintenance decisions. Tracking average inference cost per batch against cloud budget targets addresses operational spending, not whether the model meets trail-safety and maintenance acceptance criteria.
Exam caveat: metrics must match the decision: a metric that looks strong overall can still fail on rare but high-severity trail hazards. Operational check: approve a labeled gold set with maintenance SME review, set pass/fail thresholds by hazard class, require error analysis on misses that would delay repairs, and lock criteria before vendor or internal teams tune to the test.