In a Spark ML Pipeline, what is the correct order of transformers when converting a categorical string column to a numeric format suitable for tree-based models?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of it as a recipe: first translate strings to numbers (StringIndexer), then expand numbers into binary columns (OneHotEncoder), then combine all features into one vector (VectorAssembler). Order matters.
Full explanation below image
![Pipeline stage flow infographic 1024x768: three boxes in sequence with arrows: Box 1 'StringIndexer' (string 'cat/dog/bird' → numbers 0/1/2), Box 2 'OneHotEncoder' (numbers → sparse binary vectors [1,0,0],[0,1,0],[0,0,1]), Box 3 'VectorAssembler' (multiple feature columns merged into single vector). Color-coded columns showing transformation at each stage.](/media/databricks/dbml/DBML-q018-infographic.webp)
Full Explanation
The correct pipeline order for categorical encoding is: 1) StringIndexer — converts string labels ('cat', 'dog', 'bird') to integer indices (0.0, 1.0, 2.0) based on frequency. Each category gets a numeric index. Output is a double column. 2) OneHotEncoder — takes integer indices and converts to a sparse binary vector representation. Required because most Spark ML algorithms treat numeric features as ordinal (category 2 > category 1 > category 0), which is wrong. OHE eliminates this false ordering. Output is a Vector column. 3) VectorAssembler — combines all numeric and encoded feature columns into a single feature vector required by Spark ML estimators. Note: tree-based models (Decision Tree, Random Forest, GBT) can often use StringIndexer output directly without OHE because they split on categories, not ordinal values.