IDF in a Spark text pipeline down-weights terms that:
Select an answer to reveal the explanation.
Short Explanation and Infographic
IDF reduces weight of ubiquitous tokens so rare discriminative terms shine.
Full explanation below image
Full Explanation
TF-IDF = term frequency adjusted by inverse document frequency.