A public-safety team uses SageMaker Ground Truth to have human reviewers label dashcam footage for a hazard-detection model. Different reviewers frequently label the same clip differently, suggesting the labeling guidance may be ambiguous. Which practice helps identify and address this before the labels are used for training?
Select an answer to reveal the explanation.
Short Explanation
When two graders can't agree on the same answer sheet, the problem is usually the question, not the graders. Measuring inter-rater agreement and refining ambiguous instructions where reviewers disagree fixes the root cause instead of papering over it.
Full Explanation
Mechanism: measuring inter-rater agreement across reviewers highlights specific clips or categories where disagreement clusters, which usually points to ambiguous labeling instructions rather than random reviewer error, letting the team refine the guidance and re-label the affected clips before that ambiguity gets baked into training labels the model would otherwise learn as ground truth. Why the wrong options fail by concept: assigning every clip to only one reviewer removes the very signal, disagreement between independent reviewers, that revealed the ambiguity in the first place, it doesn't fix inconsistent labeling, it just hides it behind a single unchecked opinion. Automatically accepting whichever label was submitted first treats arrival order as a proxy for correctness, which has no relationship to label quality and could just as easily lock in a mistaken interpretation. Discarding every clip with any disagreement throws away exactly the examples most likely to represent genuinely ambiguous or edge-case hazards, the training set ends up missing the hard cases the model most needs to learn from. Scope caveat: even after refining instructions, some level of legitimate disagreement may persist for genuinely ambiguous real-world scenarios, and the team should distinguish that from disagreement caused by unclear guidance. Operational check: after refining the instructions, re-measure inter-rater agreement on a sample of previously disputed clips to confirm agreement improved.