Teachers find edge-case word problems that the school-district tutoring bot mishandles, and those cases are missing from eval. What should GenAIOps do?

Expanding the evaluation set after analyzing production misses and edge-case word problems is the correct GenAIOps response when teachers find tutoring failures absent from eval. Production misses are free signal: they reveal distribution shift, adversarial phrasings, and curriculum edge cases the original happy-path set never covered. Folding anonymized, labeled versions of those items into the school-district eval corpus hardens future releases and prevents the same word-problem failures from scoring green forever.

Releasing an updated tutoring prompt shaped by teacher feedback and production traces skips the eval gap; changing the prompt without adding the missed edge cases to the eval set means the fix cannot be verified and regressions teachers already see may persist. Tuning response guardrails for tutoring answers using teacher reports and trace review hides the failures instead of closing the eval gap; it lets the bot avoid hard items rather than learning to handle them, which is metric gaming and undermines integrity. Raising manual sampling of production chats to monitor tutoring quality trends over time is passive observation; it does not add the known missing edge cases to the eval set or remediate the specific word-problem failures teachers found.

Exam caveat: Expand carefully—deduplicate, anonymize student content, and avoid overfitting to a handful of viral failures without broader coverage. Track eval-set version IDs so historical scores remain interpretable after expansion.

Operational check: Maintain a intake path for teacher-reported misses; triage, label, and add representative items to the eval set; re-run the tutoring bot against the expanded set before the next promotion; monitor whether newly added edge cases pass after prompt or retrieval fixes; and publish eval-set changelog notes for curriculum owners.