A utility company indexes outage reports for search. The word outage appears in nearly every report, while the word transformer appears in only a small share of them. The team wants weights that shrink a term when it shows up across almost the whole collection. Which weighting does that?
Select an answer to reveal the explanation.
Short Explanation
A word that is shouted in every report stops being a clue, the way a chorus you hear in every song stops helping you name the song. TF-IDF downweights a term that appears in almost every outage report and leaves a rarer word such as transformer with more weight. The trap is raw term frequency, which only notices how often the word repeats inside one report.
Full Explanation
TF-IDF multiplies a term's frequency inside one document by an inverse-document-frequency factor that shrinks as the term appears in more documents. Outage, present in nearly every report, receives a small inverse-document-frequency factor, while transformer, present in only a small share, keeps more of its weight. That is the weighting the search index needs. Raw term frequency counts repetitions inside a single report and never looks at how widely the word is spread, so a universal word can outrank a specific one. An unweighted bag of words tallies tokens and treats those counts as equally informative, with no collection-wide discount. An averaged embedding can place similar reports near each other in a vector space, but it is not a scheme that downweights collection-wide terms inside a document vector of term weights. Exam caveat: the inverse-document-frequency factor is the piece that shrinks terms common to the collection; a high count inside one report can still matter, but it is scaled by how rare the term is across reports. Operational check: compare the stored weights of outage and transformer on the same report and confirm outage is discounted relative to its raw count while transformer is not.