Evaluation

NCP-GENL · 14 questions

• City clerks must decide if an agenda summarizer is ready for go-live. Which evaluation approach is most appropriate?
• Housing referral replies are scored by an LLM-as-a-judge for helpfulness. What additional step keeps the process trustworthy?
• Tax FAQ validation perplexity drops sharply while BLEU on letter drafts barely moves. How should analysts read that?
• Public-health outbreak SMS candidates need clarity ranking and medical safety checks. Which layered evaluation fits?
• Error analysis shows a 311 bot fails when residents use slang street names missing from the gazetteer. What did the team diagnose?
• Permit assistants omit floodplain checks across a week of logs. What error-analysis action is appropriate?
• A court FAQ invents forms that look real. How should error taxonomy handle that versus outdated-but-real forms?
• A transit chatbot flips morning versus evening headways after daylight-saving changes. What should the team correlate?
• A county must compare ordinance QA quality and latency on on-prem DGX versus cloud GPUs. What approach is correct?
• A city moves ordinance QA from on-prem DGX to cloud GPUs. What keeps quality deltas attributable to the platform?
• Emergency model quality matches on cloud and DGX, but p95 latency meets SLA only on-prem. What should ops document?
• A housing AI program needs scalable assessment across automatic scores, judges, humans, and error reviews. Which design fits?
• Public works wants models blocked from promotion unless evaluation is complete. What CI practice fits?
• School-district LLM governance must stay scalable yet deep. Which evaluation cadence is appropriate?