A clerk extraction pipeline reports 97% overall accuracy but fails badly on handwritten hardship affidavits. What does this illustrate?

A clerk extraction pipeline that reports 97% overall accuracy while failing badly on handwritten hardship affidavits illustrates that aggregate metrics may mask weak segments by document type or field. City forms are heterogeneous—typed permits, scanned IDs, handwritten hardship affidavits—and a single average can look excellent while a vulnerable class fails residents who need hardship relief the most.

Segment awareness works because operational risk concentrates in hard document types and critical fields (income, case numbers, signatures). Monitoring by type and field reveals where human review must stay dense even when the global score looks green.

Claiming overall accuracy guarantees uniform quality on every document type fails because averages hide tails. Saying handwritten forms never need separate monitoring fails because handwriting is a known weak segment for OCR/LLM extraction. Concluding that high overall scores mean human review can be removed everywhere fails because masked segments would then ship errors into eligibility decisions without clerks.

Exam caveat: celebrate overall metrics only alongside stratified breakdowns; never use a headline percentage alone to retire review. Operational check: split extraction accuracy by document type and key fields; if handwritten hardship affidavits lag, keep or increase human review on that segment despite a high overall score.