Meridian's data science team wants to evaluate the ramp-safety computer-vision model using only offline metrics computed on a held-out image test set. What gap should the AI project manager point out in this evaluation plan?
Select an answer to reveal the explanation.
Short Explanation
A test set sitting on a hard drive can't tell you how the model handles rain, glare, or a camera bumped six inches off angle. A comprehensive evaluation plan pushes the model into shadow-mode against the real ramp before it ever makes a live call.
Full Explanation
A comprehensive CPMAI evaluation plan goes beyond offline metrics on a static held-out test set, especially for a computer-vision system operating in variable real-world conditions like an airport ramp. The gap to flag is the absence of live or shadow-mode validation against actual camera feeds — real lighting changes, weather, camera positioning drift, and unpredictable ramp activity that a curated offline test set may not fully represent. Running the model in shadow mode (observing its calls without acting on them yet) before full production rollout surfaces these real-world gaps while the safety-critical decision still rests with human oversight. Declaring offline metrics sufficient on their own ignores exactly this gap between curated test conditions and messy production reality, which is a common and serious evaluation-plan weakness for vision systems. Skipping evaluation and launching straight to production abandons CPMAI's Go/No-Go discipline entirely, which is especially risky for a system touching ground-crew safety. Replacing the offline test set with the original training data would be actively worse — evaluating a model on the same data it was trained on tells you almost nothing about generalization and is a basic evaluation-design error, not an improvement. The exam point: a strong evaluation plan blends offline and live/shadow validation, particularly for perception-based, safety-relevant models.