Testing, Validation, and Troubleshooting

AIP-C01 · 22 questions

• A city wants a scorecard for its permitting assistant that reflects GenAI answer quality rather than classical ML classification accuracy alone. Which metric set best matches that goal?
• A county must choose between two foundation models for ordinance Q&A using a structured, repeatable comparison. Which approach best supports that decision?
• A transit disruption assistant needs continuous signal on whether riders find explanations helpful. Which evaluation mechanism best captures that user-centered feedback?
• A public-health team refuses to promote GenAI builds that regress hallucination rates. How should that policy be enforced in the delivery pipeline?
• A clerk office drafts hearing notices with a foundation model and needs scalable review that still catches high-stakes mistakes. Which assessment pattern best fits?
• A housing RAG team sees fluent answers that still miss the right policy sections. What evaluation design best isolates the failure?
• A water utility work-order agent must prove it finishes tasks and uses tools correctly, not merely that it returns text. Which evaluation focus is most appropriate?
• A 311 product manager must keep department stakeholders informed about GenAI answer quality without dumping raw CloudWatch logs. What practice best meets that need?
• Before a parks department upgrades the chatbot’s foundation model, release engineers want confidence that behavior did not silently drift. Which validation approach is most appropriate?
• A school district wants to try a new foundation model for a staff-facing assistant with limited blast radius. Which rollout pattern best fits?
• An emergency-management team must balance storm-season answer speed against answer quality for a resident briefing assistant. Which analysis should guide the configuration choice?
• A city regression suite must fail builds when ordinance answers stop including required citation formatting. What testing practice enforces that standard?
• A county needs subject-matter judgment on whether clerk-bot answers are factually correct. Which evaluation mechanism best supplies that signal?
• A transit RAG team already tracks relevance scores but riders still abandon lookups that feel slow. What should they add to retrieval quality evaluation?
• A city 311 bot truncates mid-answer after operators stuffed large case histories into each prompt. What troubleshooting focus best addresses the failure?
• A county sees spikes of Bedrock 4xx/5xx errors right after shipping a new client library for a permitting assistant. What should engineers do first to diagnose the integration issue?
• Transit FAQ quality dropped immediately after a prompt edit. How should the team troubleshoot the regression most effectively?
• After a public-health corpus migration, RAG answers pull off-topic SOPs even though the generator settings are unchanged. Which troubleshooting path best targets the likely cause?
• A clerk bot confuses fee schedules, and investigators suspect different environments are loading mismatched prompt templates. Which diagnostic approach best confirms that?
• A housing assistant cites outdated policy chapters even after authors publish updates in the source repository. What remediation best targets the retrieval pipeline?
• A water agent’s tool calls succeed, yet the final recommendation is wrong. What troubleshooting technique best reveals where the multi-step reasoning failed?
• A parks kiosk stream hangs until users give up, and distributed traces show gateway timeouts on long foundation-model completions. What should the team prioritize?