A transit agent pipeline must re-check quality whenever prompts or models change. What workflow should GenAIOps set up?

Automating evaluation workflows on every prompt or model change using built-in and custom metrics is the correct GenAIOps workflow for a transit agent pipeline. Prompt or foundation-model edits can silently break rider answers about delays, fares, or accessibility. Continuous automated eval—platform built-ins plus transit-specific custom metrics—creates a quality gate that runs in CI or release pipelines so regressions are caught before the rider app serves them; reindexing grounding sources on prompt or model change using embedding updates and vector store refresh keeps retrieval fresh but is not an automated eval workflow on each prompt or model edit.

Rerunning manual review queues on prompt or model change using production sampling and annotator scoring fails automated change-triggered risk management; human sampling and scoring cannot scale as a quality gate at the moment of every prompt or model edit and lacks continuous built-in and custom metrics. Executing adversarial safety scans on prompt or model change using jailbreak tests and harm detectors misses the quality requirement; jailbreak and harm checks cover safety, not rider-answer correctness for delays, fares, or accessibility, and lack built-in and custom quality metrics. Reindexing grounding sources on prompt or model change using embedding updates and vector store refresh alone fails shift-left quality; refreshing embeddings keeps retrieval current but does not evaluate answer quality or catch regressions before riders see them.

Exam caveat: Automation frequency should match change velocity; also re-baseline metrics when the service catalog or GTFS feeds change, not only when prompts change. Built-in metrics alone may miss fare-quotation or stop-ID accuracy—custom metrics close that gap.

Operational check: Hook eval workflows to prompt and model deployment pipelines; include built-in quality metrics plus custom transit checks; block promotion on threshold breaches; publish trend dashboards for GenAIOps and transit ops; and verify a deliberate breaking prompt change fails the pipeline before fixing it.