When a categorical feature has millions of unique values (e.g., URL paths), which Spark MLlib transformer handles encoding without creating millions of columns?
Select an answer to reveal the explanation.
Short Explanation and Infographic
FeatureHasher is the escape hatch for extremely high-cardinality features — it maps each value to a bucket via hashing, so you get a fixed-size sparse vector regardless of how many unique values exist.
Full explanation below image
Full Explanation
pyspark.ml.feature.FeatureHasher(inputCols=['url', 'category'], outputCol='features', numFeatures=1048576) applies the hashing trick: each feature value is hashed to an integer index in [0, numFeatures). The result is a SparseVector of size numFeatures where the hash position for each feature value gets a 1 (or the numeric value for numeric features). Advantages over OHE: fixed output size regardless of cardinality, handles unseen values at test time (they hash to some bucket), works well for high-dimensional sparse features. Disadvantage: hash collisions (two values mapping to the same bucket). OHE creates one column per unique value — with millions of values this is impractical. StringIndexer only assigns integer indices, not sparse vectors. CountVectorizer is for tokenized text, not arbitrary categorical features. numFeatures=2^20 ≈ 1M is a common choice for URL hashing.