In Spark MLlib, what is the correct pipeline to convert raw text documents to TF-IDF feature vectors?
Select an answer to reveal the explanation.
Short Explanation and Infographic
TF-IDF in Spark is a three-step recipe: split text into tokens (Tokenizer), count how often each word appears (CountVectorizer=TF), then downweight common words across documents (IDF). No single TFIDF class exists.
Full explanation below image
![Infographic 1024x768 showing TF-IDF pipeline in Spark: three connected boxes. Box 1 'Tokenizer': raw text 'the quick brown fox' → ['the', 'quick', 'brown', 'fox']. Box 2 'CountVectorizer (TF)': word list → sparse frequency vector. Box 3 'IDF': TF vector → TF-IDF weighted vector (common words like 'the' have lower weight shown). IDF formula displayed. CountVectorizer vs HashingTF options noted.](/media/databricks/dbml/DBML-q065-infographic.webp)
Full Explanation
Spark MLlib's TF-IDF pipeline requires separate components: Step 1 — Tokenizer(inputCol='text', outputCol='words') splits text on whitespace. For advanced tokenization use RegexTokenizer. Step 2 — CountVectorizer(inputCol='words', outputCol='tf_features', vocabSize=10000) computes term frequency vectors (TF), learning vocabulary from training data. Alternative: HashingTF(inputCol='words', outputCol='tf_features', numFeatures=10000) — faster, no vocabulary learning, but hash collisions. Step 3 — IDF(inputCol='tf_features', outputCol='features') learns inverse document frequencies from training data and applies: TF-IDF = TF × log((total docs + 1) / (docs containing term + 1)). The TFIDF class combining all steps does not exist in Spark MLlib. Word2Vec produces word embeddings (dense vectors), not TF-IDF sparse representations.