Transit operations fine-tunes a rider-alert model for production. What evaluation practice should security require around the customization?

When transit operations fine-tunes a rider-alert model for production, security should require comparable evaluation gates before and after customization. Fine-tuning can improve domain fit while regressing safety, factuality, refusal behavior, or bias. Baseline-versus-candidate comparison on municipal test sets is how teams detect those regressions before riders receive incorrect or harmful alerts about service disruptions, safety incidents, or accessibility accommodations.

Skipping baseline testing because fine-tuning "always improves safety" is false and removes the only objective detection method for regressions. Evaluating only latency after go-live ignores content-safety and quality risk that matter more than milliseconds for public rider communications. Relying solely on vendor marketing claims substitutes sales language for municipal evidence and fails due diligence.

Exam caveat: Customization without paired evaluation is a governance failure, not an optional polish step. Distinguish performance metrics (latency, throughput) from safety and alignment metrics. Production cutover is too late for the first meaningful content-safety check on a transit GenAI system.

Operational check: Freeze a rider-alert evaluation suite covering disruption wording, emergency phrasing, accessibility language, and refusal or injection cases. Score the base model, score the fine-tuned candidate under the same harness, require documented pass or fail against transit thresholds, and retain artifacts for audit. Block promotion if safety or quality deltas regress beyond agreed tolerances.