Evaluation and Tuning

NCP-AAI · 27 questions

• City IT builds a gold set of 200 labeled 311 dialogues to score the agent weekly. What evaluation practice does this implement?
• Permit-answer accuracy must be measured against ordinance excerpts. Why is that benchmark choice better than generic chatbot leaderboards?
• Public-health agents must be scored on correct triage category and on refusing diagnosis. How should the evaluation pipeline be designed?
• A city RAG agent’s evaluation should distinguish bad retrieval from bad generation. What pipeline design achieves that?
• Nightly CI must run a frozen civic scenario suite before any agent prompt change merges. What evaluation practice is this?
• The same agent scores high on trash-pickup FAQs but fails on complex variance petitions. What evaluation insight should teams take?
• English and Spanish housing datasets show unequal agent success. How should performance be compared?
• Two retrieval corpora—municipal code versus website FAQ—are tested on the same permit questions. What is the evaluation goal?
• Seasonal storm datasets break a summer-tuned 311 agent. What does comparing performance across temporal slices reveal?
• Multi-agent and single-agent setups are candidates for rental-assistance completion. How should teams compare them fairly?
• Counter-staff assisted mode outperforms fully autonomous mode on appeal letters. What comparison should drive the operating choice?
• Residents can rate answers thumbs-up/down and must pick a reason code such as wrong hours, wrong fee, or unclear. Why collect feedback that way?
• Clerks file structured “agent erred” tickets that feed a weekly prompt-fix backlog. What evaluation-and-tuning practice is this?
• A transit agency clusters free-text rider complaints into themes for agent improvement. What should the team do first with that unstructured feedback?
• Feedback for a benefits-eligibility agent sometimes includes case numbers. What must happen before that feedback enters the training or eval store?
• A housing portal denies an application and explains why. Why require structured feedback right after that denial explanation?
• 311 leadership wants sub-two-second answers, but lowering max tokens truncates complete fee explanations. What trade-off should the tuning decision reflect?
• A city lowers temperature for ordinance quoting but keeps it higher for parks event brainstorming. What does that parameter choice recognize?
• GPU costs spike while the city runs off-peak permit batch jobs. How should tuning address the latency–cost–accuracy trade-off?
• A clerk agent sometimes invents unused ordinance IDs. Which tuning move best reduces that hallucination pattern?
• A larger model improves appeals accuracy but misses the city’s p95 latency SLO. What should the team do?
• Evaluation shows retrieval misses concentrated on alley easement documents. What is the best next optimization step?
• Confusion matrices show trash versus recycling tickets often misrouted. What should the team optimize first?
• Latency budgets fail only on multimodal photo-ticket turns. How should optimization proceed?
• Faithfulness scores drop after a new website scrape. Analysis points to noisy HTML boilerplate. What should the team do?
• Human review finds many answers that are factually correct but unhelpful. What optimization best improves civic usefulness?
• Week-over-week dashboards show Spanish-language task scores lagging English. What should longitudinal eval analysis drive?