A housing authority has three caseworkers manually label housing-casework case notes with sentiment tags to train a triage model. Before trusting the labeled set, what should the team do to validate label quality across the three annotators?
Select an answer to reveal the explanation.
Short Explanation
Picture three referees calling the same play differently — until you compare their calls side by side, you don't actually know they disagree. Inter-annotator agreement does exactly that: it has all three caseworkers label an overlapping sample so you can measure where their sentiment tags line up and where they don't. The notes where they clash go to adjudication, which is what actually fixes the labels, not a volume count.
Full Explanation
Inter-annotator agreement quantifies how consistently independent labelers apply the same tag to the same input, and it's the standard way to validate a human-labeled training set before it feeds a model. Measuring agreement on a shared sample surfaces systematic disagreement — one caseworker calling frustration 'negative' while another calls it 'neutral' — and adjudicating those cases produces a reconciled label a model can actually learn from. A guideline review alone assumes training equals consistency, but it never confirms the outcome; people who read the same guideline still diverge in practice, which is precisely what agreement measurement is built to catch. Having one senior caseworker relabel everything trades a consistency problem for a single-rater bias problem and forfeits the diversity of judgment multiple annotators provide. A labeling volume threshold like one hundred notes says nothing about whether those notes were labeled consistently — quantity isn't a proxy for agreement. Scope caveat: agreement metrics tell you where labelers disagree, not which of them is right, so adjudication rules still need a tiebreaker. Operational check: compute a pairwise agreement score on a held-out sample and flag any pair scoring below your team's threshold for review.