Faithfulness scores drop after a new website scrape. Analysis points to noisy HTML boilerplate. What should the team do?
Select an answer to reveal the explanation.
Short Explanation
Boilerplate is like serving a sandwich wrapped in the menu flyer—more text, worse meal. Clean the HTML junk from the scrape before you retune. Raising the score bar while leaving noise in place just grades a dirty kitchen.
Full Explanation
Eval regressions after corpus updates often trace to data-quality issues such as HTML boilerplate rather than model weights alone. When faithfulness scores drop after a new website scrape and analysis points to noisy HTML, the team should root-cause the regression to data quality and clean boilerplate from the municipal corpus before retuning the agent. Nav bars, footers, scripts, and cookie banners pollute chunks; garbage in the knowledge store produces unfaithful answers no matter how carefully prompts are polished afterward.
Celebrating the scrape because more bytes always mean better answers confuses volume with usable content and can worsen retrieval. Raising faithfulness thresholds without cleaning the corpus moves the goalposts while leaving noise in place. Blaming residents for asking harder questions that week externalizes a self-inflicted corpus defect and stalls remediation.
Exam caveat: faithfulness drops after scrapes usually mean clean the municipal corpus first; threshold theater and user-blame are not remediation. Operational check: compare pre/post-scrape chunks for boilerplate rate, strip nav/footer/script noise, re-index, re-run the faithfulness suite on the same question set, gate future scrapes on a boilerplate quality check, and only then consider prompt or model retunes if scores remain low.