A court AI audit sample spans a period when the model changed mid-window, muddying conclusions. What sampling consideration should the auditor apply?
Select an answer to reveal the explanation.
Short Explanation
You're pooling outputs across model versions just to boost sample size, and that confuses the story. You need periods homogeneous—or segmented—around model changes, not just matched to filing volumes. Prioritizing high-value cases won't untangle mixed model versions.
Full Explanation
Evaluating period selection and sample homogeneity when AI systems change mid-test works because mixing pre- and post-upgrade court AI outputs in one undifferentiated sample confounds attribution. Conclusions may blame or clear the wrong model version, masking regressions introduced by the change. Auditors should segment periods around deployment boundaries, or explicitly design comparative tests across versions with labeled strata.
Increasing sample size to improve precision while pooling outputs across model versions fails: versions can differ in prompts, weights, tools, or policies, and pooling them confounds attribution, so version homogeneity is an assumption to test, not a default. Drawing samples proportionally from filing volumes to reflect court workload mix fails conceptually by ignoring version boundaries and can still blend outputs from different systems, muddying period conclusions. Prioritizing high-value cases for manual review to target residual audit risk fails: targeting case value does not resolve the confounding created by mixing model versions, and it does not supply the version boundaries needed for homogeneous period design.
Exam caveat: if change logs are incomplete, first raise a change-management finding; sample segmentation depends on knowing when the court AI actually changed. Operational check: obtain model/version change tickets and timestamps, split the audit sample into homogeneous windows (or labeled comparative groups), analyze within segments, and disclose version boundaries in the court AI audit report so readers know which system the conclusions describe.