A transit AI control test relies only on design walkthroughs and collects no operating effectiveness sample. What distinction should the auditor apply?

Distinguishing design evaluation from operating effectiveness sampling for AI controls works because a walkthrough can show that monitoring, approval, or override controls are designed, while only timed samples of actual operation prove they worked through the period. Transit AI audits that stop at design leave open whether alerts were handled, reviews occurred, or exceptions were resolved in production. Both design and OE testing are needed for credible municipal assurance.

Treating design walkthrough evidence as operating effectiveness sampling for transit AI fails because a walkthrough only confirms the control was designed—it cannot stand in for a timed sample proving the control actually operated throughout the period. Relying on model validation reports in place of operating effectiveness sampling fails conceptually: a validation report evaluates model performance at a point in time, not whether the AI control actually ran correctly on every relevant transaction or alert across the audit period. Accepting transit telemetry dashboards as a substitute for formal operating effectiveness testing fails because a dashboard shows current system status, not an audited sample demonstrating the control operated effectively across the full testing period.

Exam caveat: OE sample size and period must match control frequency; a single walkthrough day cannot prove continuous monitoring effectiveness. Operational check: split the transit AI test plan into design procedures and OE sampling (populations, periods, sample sizes), execute both, and report separately so findings clearly state whether failures are design gaps or operating lapses.