The gap statistic is used to select the optimal number of clusters k. What is the reference distribution against which the observed log(WSS) is compared?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — b is correct because the gap statistic (Tibshirani et al., 2001) compares the observed log(WSS) at each k to the average log(WSS) across Monte Carlo datasets simulated from a uniform distribution spanning the data range. The optimal k maximizes the gap: E[log(WSS_reference)] - log(WSS_observed).
Full explanation below image
Full Explanation
B is correct because the gap statistic (Tibshirani et al., 2001) compares the observed log(WSS) at each k to the average log(WSS) across Monte Carlo datasets simulated from a uniform distribution spanning the data range. The optimal k maximizes the gap: E[log(WSS_reference)] - log(WSS_observed). A describes a different approach (comparing successive k values). C uses a single-cluster baseline, not what the gap statistic uses. D describes a different model-based approach.