When training a LogisticRegression classifier in Spark MLlib on an imbalanced dataset (1% fraud, 99% non-fraud), which parameter adjusts for class imbalance during training?
Select an answer to reveal the explanation.
Short Explanation
Spark MLlib uses a weightCol approach — you add a weight column to your DataFrame (higher values for minority class), and LogisticRegression uses it to upweight minority class examples in the loss function.
Full Explanation
In Spark MLlib, LogisticRegression(weightCol='weight') uses per-sample weights to address class imbalance. Implementation: compute class weights (e.g., weight for minority class = n_majority / n_minority) and add as a column to the DataFrame. For a 1%/99% split: minority_weight = 99, majority_weight = 1. df = df.withColumn('weight', F.when(F.col('label')==1, 99.0).otherwise(1.0)). The LogisticRegression loss becomes: Σ w_i * loss(y_i, ŷ_i), making each fraud example worth 99x a non-fraud example in the gradient. balanceClasses=True doesn't exist in Spark MLlib (it's a Spark H2O parameter). classWeights=[99, 1] is not a valid constructor parameter. Spark MLlib does NOT automatically handle class imbalance. Alternative: oversample the minority class using df.sampleBy('label', fractions={0: 0.01, 1: 1.0}) to balance the dataset.