When applying PCA in Spark MLlib, what does the k parameter control?
Select an answer to reveal the explanation.
Short Explanation and Infographic
k in PCA is how many dimensions to keep — if you have 100 features and set k=10, PCA finds the 10 directions of maximum variance and projects all data down to those 10 components.
Full explanation below image
Full Explanation
pyspark.ml.feature.PCA(k=10, inputCol='features', outputCol='pca_features') — k is the number of principal components (output dimensions) to keep. PCA finds the k orthogonal directions (eigenvectors of the covariance matrix) that capture the most variance. The PCAModel stores the principal components and can explain the variance ratio (model.explainedVariance — a DenseVector of per-component variance ratios). Choosing k: look at the explained variance ratio and pick k where cumulative variance exceeds a threshold (e.g., 95%). Unlike scikit-learn (which has n_components_percentage like 0.95), Spark PCA requires an integer k. Workflow: StandardScaler → PCA → classifier. PCA is NOT iterative (not controlled by iterations). The variance threshold approach requires computing full PCA first then selecting k manually. k is not related to k-NN.