The gap statistic is used to select the optimal number of clusters k. What is the reference distribution against which the observed log(WSS) is compared?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — you compare the observed log(WSS) to the average log(WSS) from multiple datasets drawn from a uniform distribution over the range of the observed data. That's what the gap statistic does, and the best k is the one that maximizes the gap.
Full Explanation
The correct reference is the average log(WSS) from multiple datasets drawn from a uniform distribution over the range of the observed data. The gap statistic (Tibshirani et al., 2001) compares observed log(WSS) at each k to the average log(WSS) across Monte Carlo datasets simulated from that uniform distribution. The optimal k maximizes E[log(WSS_reference)] - log(WSS_observed). The previous k-1 solution option compares successive k values, not a reference distribution. The single-cluster option uses a k=1 baseline, not the gap statistic. The Gaussian mixture option describes a different model-based approach.