An auditor is reviewing the model validation report for a natural language processing model used in automated contract review. The validation report shows high performance on the vendor's internal test set but the deploying organization has not conducted its own independent testing. What is the MOST significant validation gap?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — a is correct because vendor test sets are composed of the vendor's own data, which may not represent the deploying organization's specific contract types, terminology, legal jurisdictions, or language complexity. The organization must independently test the model on its own representative contract corpus to validate that performance generalizes to their use case.
Full explanation below image
Full Explanation
A is correct because vendor test sets are composed of the vendor's own data, which may not represent the deploying organization's specific contract types, terminology, legal jurisdictions, or language complexity. The organization must independently test the model on its own representative contract corpus to validate that performance generalizes to their use case. Vendor performance metrics alone are insufficient for the deploying organization's due diligence. Internal audit review of the vendor's test set (B) and third-party auditing firm (C) are additional assurance activities but secondary to the fundamental gap of deployer-specific testing. Carbon footprint (D) is not a model performance validation concern.