A fire-district after-action assistant was fine-tuned and must prove it improved over the base foundation model before cutover. What comparison should ops run?

Comparing the fine-tuned model to the base foundation model on groundedness using the same civic evaluation set is the comparison ops should run before cutover of a fire-district after-action assistant. Fine-tuning is a claim of improvement until side-by-side scores on shared after-action prompts show better groundedness than the base model. Same-set comparison controls for prompt and label differences. Comparing the fine-tuned model to the base foundation model on fluency ratings using the same civic evaluation set rewards fluent wording without checking that answers are grounded in district records. Comparing the fine-tuned model to the base foundation model on groundedness using general support benchmarks measures performance on unrelated civic content rather than fire-district after-action records. Comparing the latest fine-tuned checkpoint to the prior checkpoint on groundedness using after-action training prompts checks progress within fine-tuning, not improvement over the base foundation model. Exam caveat: base-versus-fine-tuned comparisons should use the production-relevant quality metric named in the stem—here groundedness—on a fixed civic eval set. Fluency ratings, general-support benchmarks, and checkpoint-to-checkpoint scoring are not substitutes for that base-model comparison. Operational check: run the fine-tuned checkpoint and the base model on the identical after-action eval set, score groundedness and factual citation to district records, require a predefined lift before cutover, and archive the comparison report with the promotion ticket.