Optimize generative AI systems and model performance
AI-300 · 30 questions
- A permits RAG bot returns irrelevant code sections for resident questions. What optimization should engineers try first for retrieval quality?
- A 311 knowledge base of long PDFs retrieves noisy chunks that miss the right procedure. Which chunking change often improves civic RAG?
- A public-health RAG system struggles with clinical-guideline wording in the city’s corpus. What embedding approach fits domain accuracy?
- A housing policy assistant must find exact program names while still supporting semantic questions. Which retrieval approach fits?
- A transit disruption bot team wants evidence that a new retrieval strategy beats the current one. What evaluation approach fits?
- A courts RAG tool keeps citing weakly related opinions that confuse clerks. Engineers want fewer weakly grounded answers without rewriting the whole index. What should they tune first?
- A water-quality FAQ struggles with treatment-plant jargon in SOP PDFs even though chunking looks fine. What embedding change best improves domain retrieval?
- A library catalog assistant finds topical books well but often misses exact ISBN lookups residents paste into chat. What retrieval approach should GenAIOps add?
- A parks permits RAG team changes how ordinance PDFs are chunked and needs proof retrieval quality improved. What should they use?
- A school-district policy bot fails on multi-part parent questions that a single naive top-k query cannot cover. What retrieval strategy change fits best?
- A fire-code assistant often retrieves the wrong code sections despite decent chunking. Annotated pairs of queries and correct sections are available. What should the team do to improve lookup accuracy?
- A city grants portal RAG misses exact NOFO identifiers that applicants type precisely, even when semantic search finds related topics. What should GenAIOps implement?
- A clerk’s ordinance bot team wants a fair comparison of two rerankers before changing production RAG. What evaluation approach should they use?
- A sanitation SOP assistant’s answers often miss steps that span adjacent sections because chunks are too small. What retrieval tuning should the team try?
- A municipal open-data narrator must reliably retrieve ward names and local toponyms in resident questions. How should the team choose among embedding models?
- A 311 contact-center GenAIOps team claims a hybrid index beat pure vector search for resident FAQs. What should leadership require before accepting the change as proven?
- A permits office wants a city-tone drafting model customized on limited GPU budget without full foundation-model retraining. Which fine-tuning approach fits best?
- A 311 team lacks enough labeled resident chats to fine-tune an intent-routing helper. What data strategy should they use?
- A public-health team fine-tunes a vaccination-FAQ model and must catch quality regressions across checkpoints. What should they do after each training checkpoint?
- A housing authority finished sandbox evaluation of a fine-tuned eligibility explainer and is ready for residents. What lifecycle step should GenAIOps take next?
- A transit authority needs to customize a foundation model for agency acronyms and stop names without full-weight retraining. Which method best fits?
- A courts clerk team needs fine-tuning examples that resemble dockets but must not train on sealed real cases. What should they create?
- A water-utility fine-tuned assistant shows rising toxic-content flags on safety evaluations after a new checkpoint. What should ops do first?
- A library summarizer fine-tune is ready, but production still serves the prior model. What promotion sequence is correct?
- A parks department wants a copywriting model that follows approved park-board style guides rather than generic marketing tone. Which fine-tuning design fits?
- A school-district tutoring fine-tune uses synthetic math word problems that change over time. What should GenAIOps do with those datasets?
- A fire-district after-action assistant was fine-tuned and must prove it improved over the base foundation model before cutover. What comparison should ops run?
- A city GenAIOps runbook is cutting over to a new fine-tuned checkpoint that cleared eval and canary. What should happen to the old fine-tuned deployment?
- A grants office is choosing between continued pretraining and chat-style fine-tuning for NOFO-heavy drafting. How should they decide?
- A clerk’s office runs a fine-tuned forms helper and must decide whether customization still pays off. What monitoring loop should GenAIOps use?