A categorical feature has more than 50 distinct values (for example, a 'city' column with hundreds of possible cities). If this feature is one-hot encoded, what problem is likely to arise?
Select an answer to reveal the explanation.
Short Explanation
One-hot encoding gives every distinct category its own column, with a single 1 marking which one applies and 0s everywhere else. That's totally fine for a handful of categories, but once you're past 50, 100, or more distinct values, you end up with a feature vector that's mostly zeros stretched across a huge number of dimensions — sparse and high-dimensional, which wastes memory, slows things down, and can make it harder for the model to learn meaningful patterns from such a thin, spread-out signal. That's exactly why 'very sparse, high-dimensional representation' is the answer. It's not that the network literally can't accept it — plenty of models handle sparse inputs, it's just inefficient and often suboptimal. One-hot encoding never groups anything together — each category stays fully separate by design. And it doesn't produce random numeric labels; that's closer to label encoding, a different (and also flawed for neural nets) technique.
Full Explanation
One-hot encoding represents each category in a categorical feature as a binary vector with a length equal to the total number of distinct categories, where exactly one position is set to 1 (corresponding to that category) and every other position is 0. When the number of distinct categories is small (for example, 3 to 10), this produces compact, manageable vectors. However, as cardinality grows — say, past 50 distinct cities, product SKUs, or user IDs — the resulting one-hot vectors become extremely long, with the vast majority of positions equal to 0 for every single sample. This is described as a sparse, high-dimensional representation: sparse because almost every value is zero, and high-dimensional because the vector length scales linearly with the number of categories. This creates practical problems: it substantially increases memory usage and the number of input weights the model must learn (since each of those dimensions typically connects to a Dense layer with its own weights), it can slow down training, and it can make it harder for the model to learn useful patterns because the representation carries no inherent notion of similarity between categories (two cities that behave similarly get arbitrarily distant one-hot vectors, no closer to each other than to a totally unrelated city).
One-hot encoded features are not impossible to feed into a neural network; they can absolutely be used as direct input to Dense layers or other architectures. The issue is inefficiency and representational quality at high cardinality, not an outright technical incompatibility.
One-hot encoding does not group similar categories together in any way; every category is treated as entirely independent and equally distant from every other category by construction, since each gets its own dedicated, orthogonal dimension. There is no mechanism within one-hot encoding itself that captures similarity or relatedness between categories.
One-hot encoding also does not silently convert categories into random numeric labels; each category is deterministically mapped to a fixed position in the binary vector based on a category-to-index mapping established during preprocessing (commonly alphabetical or based on first appearance), not a random assignment.
The standard solution for high-cardinality categorical features in deep learning is to use an embedding layer instead of one-hot encoding: the embedding layer maps each category to a smaller, dense, learned real-valued vector, which is far more memory-efficient and, crucially, allows the model to learn meaningful notions of similarity between categories during training.