Permit QA fine-tuning shows falling train loss while validation exact-match plateaus for three consecutive evals. What should the team do?
Select an answer to reveal the explanation.
Short Explanation
Train loss can keep smiling while the model stops getting better on real permits—that’s overfitting knocking. Early stopping on the validation exact-match plateau saves the checkpoint that actually helps residents.
Full Explanation
Permit QA fine-tuning that shows falling train loss while validation exact-match plateaus for three consecutive evals is a classic overfitting signal. The team should trigger early stopping based on the stalled validation exact-match and keep the best civic checkpoint rather than chasing train loss to zero. Validation exact-match reflects permit-field correctness the municipality actually cares about; when it stops improving, further epochs mostly memorize the training permits and waste GPU time.
Ignoring validation and keeping training until train loss hits zero fails by concept: train loss can keep falling while generalization collapses. Switching the loss to maximize validation perplexity deliberately fails because maximizing perplexity is anti-optimization—it worsens language-model fit instead of guiding a sensible stop. Deleting the validation set so the plateau cannot be observed fails because hiding the metric removes the only early-stop signal and invites silent overfit into production permit QA.
Exam caveat: define patience and metric direction explicitly (for example, stop after N evals with no exact-match improvement) so “plateau” is not a vague ops judgment call. Operational check: log train loss and validation exact-match every eval cadence, mark the best exact-match checkpoint, halt when three consecutive evals show no improvement, and promote the marked checkpoint—not the final overfitted weights—into the permit staging harness.