Implement generative AI quality assurance and observability

AI-300 · 30 questions

• A permits chatbot team needs a fair way to judge answer quality before each release. What evaluation asset should they build?
• A 311 evaluation harness must check whether answers stick to the city’s published service catalog. Which quality metric fits best?
• A public-health assistant answering vaccination-hours questions needs more than a single vague score. What evaluation approach fits?
• A housing chatbot receives free-text dialogs that might include harmful or self-harm content. What evaluation should GenAIOps configure?
• A transit agent pipeline must re-check quality whenever prompts or models change. What workflow should GenAIOps set up?
• A courts summarizer must be scored on whether required summary fields appear from docket excerpts. What test-dataset practice helps?
• A water-advisor bot’s answers become jumbled after a prompt edit even though some facts remain. What quality gate should block promotion?
• A library youth-services bot must align safety checks with age-appropriate content policies. What should GenAIOps configure?
• A parks recreation assistant must quote fees correctly from the fee schedule, not only sound fluent. What evaluation addition helps?
• Teachers find edge-case word problems that the school-district tutoring bot mishandles, and those cases are missing from eval. What should GenAIOps do?
• A fire-public-info agent answers evacuation-zone questions grounded in GIS bulletins. Which quality metric best checks that answers address the resident’s location-critical ask?
• A city contact center must not ship a municipal policy bot when safety evaluations show elevated jailbreak success. What release practice fits?
• A clerk’s generative forms helper must fail the release pipeline when Foundry evaluation metrics regress. How should GenAIOps wire that gate?
• A grants office needs Spanish and English prompts to share equivalent expected intents in the evaluation set. What dataset practice supports fair bilingual scoring?
• A sanitation chatbot’s fluency scores look fine while groundedness collapses on pickup-schedule facts. What should ops prioritize before promotion?
• A municipal AI review board requires evaluation evidence in every GenAI go-live packet. Which package meets that governance need?
• A 311 Foundry deployment just launched and product owners need live quality signals after go-live. Which capability should they enable?
• A transit assistant must meet rider-app SLOs for how quickly and how many requests it handles. Which observability focus fits?
• A public-health chatbot must stay inside clinic conversation budget limits. What should ops track?
• A housing agent fails eligibility checks when tool calls break, and support needs to debug production. What observability capability helps most?
• After a courts corpus refresh, continuous monitoring shows a spike in low-groundedness responses. What is the appropriate ops interpretation?
• A water-ops copilot dashboard shows p95 latency rising during morning SCADA report generation. Why does that percentile matter?
• A library chatbot cost report shows higher token burn after a verbose system prompt shipped last week. How should cost metrics guide the fix?
• A parks event bot needs support tickets linked to GenAI traces for faster triage. What logging practice enables that?
• A school-district tutoring service expects exam-week demand spikes and wants to avoid throttling. What should observability emphasize beforehand?
• A fire-district assistant’s traces show repeated retrieval timeouts against the policy index. What should ops conclude?
• A city AI CoE wants department product owners engaged with Foundry continuous monitoring. Which operational practice fits?
• A clerk’s office must know when Foundry token spend exceeds the monthly appropriation. What cost-control practice fits?
• A sanitation route advisor samples production traces and finds possible PII in logs. What privacy-aware observability step fits?
• A grants chatbot SLO pack must reflect both resident wait time and spend efficiency. Which combination belongs in the pack?