Why does Spark MLlib require all features to be assembled into a single Vector column before training?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Spark MLlib algorithms all expect a single column called 'features' containing a dense or sparse vector — it's an API contract. VectorAssembler is the standard tool to pack individual columns into that vector.
Full explanation below image
![Infographic 1024x768 showing VectorAssembler process: DataFrame with multiple columns (age: 25, income: 50000, clicks: 3, category: [1,0,0]) on left → VectorAssembler box in center → DataFrame with single 'features' column containing DenseVector([25, 50000, 3, 1, 0, 0]) on right. Then arrow to MLlib algorithm box (LogisticRegression, RandomForest options). Column packing visualization.](/media/databricks/dbml/DBML-q035-infographic.webp)
Full Explanation
Spark MLlib's design requires a single Vector-typed column (typically named 'features') because: 1) Uniformity — the ML algorithm API (fit, transform) is the same regardless of whether the model is logistic regression, random forest, or SVM. All see the same 'features' vector. 2) Dense vs Sparse efficiency — the Vector type handles both DenseVector and SparseVector transparently, enabling memory-efficient representation of high-dimensional sparse data (e.g., text features with thousands of dimensions). 3) Distributed processing — each row's feature vector can be processed independently by executors. VectorAssembler(inputCols=['age', 'income', 'clicks'], outputCol='features') packs individual columns into the required vector. The assembler does NOT do feature scaling (StandardScaler does). Spark DataFrames have no practical column number limit.