A water-advisor bot’s answers become jumbled after a prompt edit even though some facts remain. What quality gate should block promotion?

Failing promotion when coherence scores drop after a prompt edit is the correct quality gate for a water-advisor bot whose answers become jumbled even if some facts remain. Coherence measures whether paragraphs are logically ordered and non-contradictory—critical when residents must follow boil-water or conservation steps in sequence. A prompt change that preserves scattered facts inside contradictory or disordered text still fails civic usability, so GenAIOps should block Foundry promotion until coherence recovers.

Failing promotion when groundedness scores drop after a prompt edit is the wrong gate because the bot's jumbled answers can still be fully grounded—supported facts don't ensure logical order. Failing promotion when fluency scores drop after a prompt edit is the wrong gate because fluency measures smooth reading, not logical order—a prompt edit can jumble paragraphs while keeping sentences fluent, so fluency drop won't catch the incoherence. Failing promotion when relevance scores drop after a prompt edit is the wrong gate because a jumbled answer can remain fully relevant to the water-advisor question—relevance doesn't capture whether the steps are logically ordered.

Exam caveat: Coherence metrics complement—not replace—groundedness and safety. A coherent but ungrounded water answer remains wrong; set multi-metric gates. Investigate whether the prompt edit conflicted with retrieval formatting or length limits that truncate mid-instruction.

Operational check: Compare coherence (and groundedness) before versus after the prompt edit on the water-advisor eval set; fail the release when coherence drops below threshold; manually read failing items for contradiction patterns; roll back to the last coherent prompt tag; and add regression cases that previously exposed the jumble.