When training a LogisticRegression classifier in Spark MLlib on an imbalanced dataset (1% fraud, 99% non-fraud), which parameter adjusts for class imbalance during training?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Spark MLlib uses a weightCol approach — you add a weight column to your DataFrame (higher values for minority class), and LogisticRegression uses it to upweight minority class examples in the loss function.
Full explanation below image
Full Explanation
In Spark MLlib, LogisticRegression(weightCol='weight') uses per-sample weights to address class imbalance. Implementation: compute class weights (e.g., weight for minority class = n_majority / n_minority) and add as a column to the DataFrame. For a 1%/99% split: minority_weight = 99, majority_weight = 1. df = df.withColumn('weight', F.when(F.col('label')==1, 99.0).otherwise(1.0)). The LogisticRegression loss becomes: Σ w_i * loss(y_i, ŷ_i), making each fraud example worth 99x a non-fraud example in the gradient. balanceClasses=True doesn't exist in Spark MLlib (it's a Spark H2O parameter). classWeights=[99, 1] is not a valid constructor parameter. Spark MLlib does NOT automatically handle class imbalance. Alternative: oversample the minority class using df.sampleBy('label', fractions={0: 0.01, 1: 1.0}) to balance the dataset.