A school-district team periodically compares their custom autorater's scores against a sample independently rated by human staff, to confirm the autorater hasn't drifted from human judgment. What risk does this periodic calibration step address that building the autorater once and trusting it indefinitely would not?
Select an answer to reveal the explanation.
Short Explanation
An autorater built once and left alone is like a thermostat nobody recalibrates for years — it might still report a number, but you've lost any guarantee that number still means what it used to. Checking it against human judgment periodically is how the team catches that drift before it quietly skews every score downstream.
Full Explanation
Periodic calibration exists because an autorater's judgment can drift from human judgment over time as the agent's behavior evolves, as scoring criteria get refined, or as the kinds of transcripts being scored shift into territory the autorater wasn't originally tuned for, and none of that drift announces itself — the autorater keeps producing confident-looking numbers whether or not they still track what a human reviewer would actually conclude. Comparing a sample against independent human ratings is the mechanism that surfaces that silent divergence before it undermines trust in every downstream decision the autorater informs. Storage capacity is an infrastructure concern unrelated to whether the autorater's judgment still matches human judgment. Needing a more powerful underlying model is a capability question, not what a human-comparison calibration step is designed to detect. Software compatibility after an update is an integration issue, separate from whether the autorater's scoring logic still reflects human standards. Scope caveat: calibration only catches drift for the categories represented in the compared sample, so a sample that skips a scenario type leaves that category's drift undetected. Operational check: track the agreement rate between autorater and human scores over successive calibration rounds and confirm it hasn't been trending downward.