Domain 3: Model Training & Evaluation

Databricks Certified Machine Learning Associate · 63 questions

• In Spark MLlib, which class implements k-fold cross-validation for hyperparameter tuning?
• A model achieves 99% training accuracy but only 72% validation accuracy. Which problem does this describe, and what is the most appropriate remedy?
• A fraud detection model is trained on data where 99% of transactions are legitimate. The model predicts 'not fraud' for every transaction and achieves 99% accuracy. Why is accuracy a misleading metric here?
• When training an XGBoost model on Databricks, which runtime environment is recommended to avoid manually installing XGBoost?
• When using scikit-learn's Pipeline with crossvalscore() on Databricks, what is the key benefit of wrapping preprocessing and the estimator in a Pipeline?
• What is the role of SparkTrials when using Hyperopt for hyperparameter tuning on Databricks?
• Which Python package provides LightGBM integration optimized for distributed training on Databricks clusters?
• A regression model predicts house prices. The RMSE is $45,000 and the MAE is $28,000. Which statement correctly interprets these metrics?
• What does an MLflow model signature provide when a model is logged with mlflow.sklearn.logmodel(model, 'model', signature=signature)?
• When splitting an imbalanced dataset into train and test sets in Python, which approach ensures each split has the same class proportion?
• An ROC curve plots True Positive Rate (Recall) against False Positive Rate. What does an AUC of 0.5 indicate?
• When defining a Hyperopt search space, which distribution should you use for the learning rate parameter, and why?
• A binary classifier produces: TP=80, FP=20, FN=40, TN=860. What are the precision and recall?
• Which RandomForestClassifier hyperparameter most directly controls the tradeoff between individual tree diversity and individual tree accuracy?
• After fitting a scikit-learn GridSearchCV on Databricks, how do you access the best hyperparameter combination found?
• When training an XGBoost model with early stopping on Databricks, what does earlystoppingrounds=10 mean?
• For a multiclass classification problem with 5 classes, which Spark MLlib evaluator computes accuracy and is appropriate?
• In a logistic regression model with high-dimensional sparse features, which regularization technique tends to produce sparse model weights (setting unimportant features to zero)?
• What is the key advantage of TrainValidationSplit over CrossValidator in Spark MLlib when training on very large datasets?
• What does a SHAP (SHapley Additive exPlanations) value represent for a specific feature in a prediction?
• When training a LogisticRegression classifier in Spark MLlib on an imbalanced dataset (1% fraud, 99% non-fraud), which parameter adjusts for class imbalance during training?
• A data scientist applies StandardScaler to all data before using CrossValidator with 5 folds. What error does this introduce?
• Which scikit-learn algorithm is commonly used for unsupervised anomaly detection (finding unusual data points without labeled examples)?
• A learning curve shows training accuracy = 0.95 and validation accuracy = 0.94 with both plateauing as training set size increases. What does this indicate?
• What is the difference between Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression())]) and makepipeline(StandardScaler(), LogisticRegression()) in scikit-learn?
• In Hyperopt, what algorithm does tpe.suggest implement and why is it preferred over random search for hyperparameter tuning?
• How does Hyperopt with SparkTrials help hyperparameter tuning on Databricks?
• For highly imbalanced fraud detection, which metric set is MORE appropriate than plain accuracy?
• CrossValidator in Spark ML is used to:
• On a 2TB labeled dataset in Delta, which training approach is MORE practical?
• Feature importance from tree models helps practitioners:
• LightGBM is often chosen on Databricks because:
• ParamGridBuilder in Spark ML is used to:
• ROC-AUC of 0.5 for a binary classifier indicates:
• Gradient-boosted trees often outperform linear models on tabular data because:
• RegressionEvaluator with metricName rmse measures:
• Stratified splitting is important when:
• Early stopping in XGBoost/LightGBM helps by:
• TrainValidationSplit differs from CrossValidator mainly by:
• SHAP explanations can provide:
• Probability calibration matters when:
• IsolationForest is typically used for:
• A learning curve showing high train and val error as data grows suggests:
• Macro-F1 is preferred over micro-F1 when:
• L1 regularization tends to encourage:
• Compared to full grid search, random/TPE-style search often:
• PR-AUC is often preferred under class imbalance because:
• XGBoost's default handling of missing values is notable because:
• After training a probabilistic classifier, F1 can often improve by:
• LightGBM native categorical handling can reduce:
• Setting class weights in logistic regression helps:
• Hyperopt hp.choice vs hp.uniform differ in that:
• Setting earlystoppingrounds too aggressively small may:
• A golden evaluation set for regression gates should be:
• Bagging (e.g., Random Forest) vs boosting (e.g., GBM) differs in that:
• SMOTE-like oversampling should be applied carefully because:
• A validation curve vs model complexity that diverges (train↑ val↓) indicates:
• Optimizing log-loss while the business cares about top-decile precision requires:
• Early stopping should monitor a metric that:
• Excessive HPO parallelism can hurt when:
• Temperature scaling is a technique for:
• ChiSquareSelector in Spark ML is a:
• Evaluating metrics on data slices (region, segment) helps: