A model deployed in SAS Model Manager is monitored using PROC ASSESS. The data scientist wants to track Kolmogorov-Smirnov (KS) statistic over monthly scoring runs. What does a significantly increasing KS statistic over time indicate?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — a significantly increasing KS distance between score distributions is a red flag for model drift and degraded performance. You're seeing the score distributions diverge from what the model expected, so its discriminatory power is slipping.
Full Explanation
In model monitoring, the KS statistic from PROC ASSESS measures the maximum difference between the cumulative distribution of predicted scores for events versus non-events. A declining KS over time indicates the model is losing its ability to separate classes. However, the question describes a significantly increasing KS distance between score distributions. If that means the divergence between training-time and current-time score distributions is increasing, it signals data drift and performance degradation, so the model's discriminatory power is degrading. The option saying the model's ability to separate the two classes is improving is wrong because it describes improvement, not degradation. The option saying the model is increasingly well-calibrated to the production population is wrong because it also describes improvement, not degradation. The option saying increasing KS statistic has no diagnostic meaning for model drift is wrong because KS is a primary monitoring metric.