Housing chat win-rate versus the prior version drops right after a LoRA swap. What should production reliability practice trigger?
Select an answer to reveal the explanation.
Short Explanation
If the new housing LoRA loses head-to-head against last week’s build, that benchmark drop should open a rollback review—not a victory lap for ‘newer.’ Version scores are the gate, not the calendar.
Full Explanation
When housing chat win-rate versus the prior version drops right after a LoRA swap, production reliability practice should trigger an automatic rollback review gated by that version benchmark drop. Continuous benchmarking of deployed municipal models against prior versions exists specifically so “newer” is not automatically “better.” A head-to-head win-rate decline means the new housing LoRA is losing quality contests residents depend on—eligibility explanations, document checklists, or referral accuracy—and change control should open rollback or remediation before the weaker adapter hardens as default. Permanent lock-in of the worse LoRA because it is newer confuses calendar order with quality gates. Deletion of the prior version so comparisons are impossible removes the recovery baseline and turns a measurable regression into an unrecoverable one. Ignoring win-rate because LoRA adapters never affect quality is false: LoRA can improve or degrade task behavior depending on data and rank; municipal reliability assumes adapters can hurt. Exam caveat: define win-rate on a fixed housing eval set and statistical threshold so noise does not thrash rollbacks; also distinguish serving incidents from quality drops—both can justify review, but the trigger here is the benchmark signal. Operational check: keep prior and candidate LoRA versions scored on the same housing suite, automate a review ticket when win-rate drops past the gate, preserve the prior adapter for instant revert, and only clear production after the candidate beats or matches the baseline on agreed metrics.