Domain 1: Machine Learning Fundamentals

SAS AI and Machine Learning · 47 questions

• A data scientist is training a gradient boosting model in SAS and notices the training error continues to decrease while validation error increases after a certain number of iterations. Which concept best describes this phenomenon?
• When performing k-fold cross-validation in SAS Model Studio, what is the primary purpose of holding out each fold in turn?
• A model trained on a small dataset achieves 99% training accuracy but only 62% test accuracy. Which remediation strategy would most directly address the bias-variance tradeoff issue present?
• In SAS, a confusion matrix shows: TP=80, FP=20, FN=10, TN=90. What is the model's precision?
• A data scientist needs to compare two classifiers when the class distribution is highly imbalanced (95% negative, 5% positive). Which metric is most appropriate?
• Which cross-validation strategy is most appropriate when the dataset contains a strong time dependency, such as stock price data?
• The AIC (Akaike Information Criterion) penalizes model complexity. Which of the following correctly describes the AIC formula used in SAS model selection?
• When constructing a train/test split in SAS, a data scientist sets the random seed before calling PROC SURVEYSELECT. What is the main purpose of setting the seed?
• A data scientist observes that a model performs well on training data but poorly on new data. They decide to apply L2 regularization. In the context of the bias-variance tradeoff, what effect does increasing the L2 penalty (lambda) have?
• A data scientist constructs a ROC curve for a binary classifier. The curve passes exactly through the point (0.0, 1.0). What does this indicate about the model?
• In stratified k-fold cross-validation for a classification problem, what makes it 'stratified'?
• A model selection process uses BIC (Bayesian Information Criterion) instead of AIC. What is the key difference between BIC and AIC in model selection?
• A regression model produces residuals that exhibit a funnel shape when plotted against fitted values. Which diagnostic does this indicate and what remediation is recommended?
• In a multi-class classification problem with 5 classes, a data scientist in SAS wants to use a One-vs-Rest (OVR) strategy with PROC HPLOGISTIC. How many binary models must be trained?
• When computing the F1 score for an imbalanced binary classification problem, which formula is correct?
• A data scientist observes that a gradient boosted model in SAS Viya has very low training error but high validation error, while a linear model has moderate error on both. The team wants to improve the gradient boosting model without collecting more data. Which technique is specifically designed to address this situation?
• A data scientist constructs a confusion matrix and finds the following values: TP=45, FP=5, FN=15, TN=135. What is the recall (sensitivity) of the model?
• A data scientist must choose between a model with AUC=0.82 and another with AUC=0.79. However, the business requires maximizing recall at a minimum precision of 0.70. Which approach best addresses this requirement?
• A data scientist is selecting between a model with 10 parameters (AIC=245) and one with 25 parameters (AIC=238). Which model should be selected and why?
• A model's learning curve shows that training error and validation error have both converged to a high value (approximately 0.35 RMSE) and neither decreases as more training data is added. What does this indicate?
• A data scientist uses PROC SURVEYSELECT in SAS to create a stratified random sample for model building, stratifying by a 'region' variable. Which option ensures proportional allocation across strata?
• In a binary classification problem, a data scientist reports that a model achieves 'lift of 3.5 at the first decile.' What does this mean?
• A SAS data scientist needs to compare the performance of PROC HPFOREST against PROC HPLOGISTIC using 5-fold cross-validation on the same data. Which approach correctly implements this comparison in SAS?
• A data scientist in SAS Viya wants to apply oversampling to a minority class before training PROC HPFOREST. Which SAS procedure generates a synthetic oversampled training dataset using SMOTE (Synthetic Minority Oversampling Technique)?
• A data scientist observes that increasing training set size improves validation accuracy but training accuracy stays fixed near 100%. This learning curve pattern is characteristic of which condition?
• Which regularization technique simultaneously performs variable selection by producing exactly zero coefficients for irrelevant features?
• A model achieves AUC = 0.50 on the test set. What does this indicate?
• When using k-fold cross-validation with k=5, how many times is each observation used for validation?
• A classification model on a 95% negative / 5% positive dataset predicts 'negative' for every observation. What is the F1 score for the positive class?
• In the context of model evaluation, what does 'calibration' refer to?
• A machine learning practitioner uses 5x2 cross-validation to compare two models. Which statistical test is specifically designed for this comparison procedure?
• A random forest model with 500 trees shows no improvement in OOB error beyond 200 trees. What does this demonstrate about random forests?
• A data scientist uses the Matthews Correlation Coefficient (MCC) instead of accuracy on an extremely imbalanced dataset where 98% of records are class 0. What is the key advantage of MCC in this situation?
• A model reliability diagram (calibration plot) shows that predicted probabilities near 0.8 correspond to observed event rates of only 0.6. What does this indicate about the model?
• Which cross-validation approach is specifically recommended for time series data where the relationship between features and target evolves over time?
• A team encodes the categorical variable 'state' (50 categories) by replacing each state with the average default rate computed on training data. What encoding method is this, and what is its primary risk?
• A data scientist uses Platt scaling to calibrate a gradient boosting model's output. What does Platt scaling specifically fit to the raw model scores?
• For a logistic regression credit risk model, Weight of Evidence (WOE) encoding is applied to a high-cardinality categorical variable. Why is WOE particularly suited for logistic regression?
• Log-loss penalizes confident wrong predictions much more heavily than uncertain wrong predictions. Which scenario produces the highest log-loss for a single observation?
• A practitioner computes Cohen's Kappa = 0.72 for a 3-class text classifier. What does this value indicate according to the Landis and Koch scale?
• A data scientist needs model-agnostic local explanations for individual predictions from a gradient boosting model. Which technique provides additive feature attributions that sum to the difference between the model's output and the global expected output?
• A credit risk team builds a logistic regression model and reports Gini = 0.62. How is the Gini coefficient derived from the AUC?
• A machine learning team trains a neural network and observes that training loss decreases smoothly but validation loss exhibits high variance across epochs. Which regularization technique specifically reduces variance by injecting Gaussian noise into the input layer during training?
• A researcher applies SMOTE (Synthetic Minority Over-sampling Technique) to address class imbalance before training. What synthetic minority samples does SMOTE generate?
• Information Criterion (AIC and BIC) are used to compare competing regression models. Which statement correctly differentiates AIC from BIC for model selection?
• A practitioner builds a stacking ensemble with logistic regression, gradient boosting, and random forest as base learners, and a ridge regression as the meta-learner. What is the critical data hygiene requirement for training the meta-learner?
• A machine learning team applies partial dependence plots (PDPs) to interpret a random forest model's relationship between 'customertenure' and predicted churn probability. The PDP shows a non-monotone U-shaped curve. What does this indicate?