What does dimensionality reduction mean in machine learning practice?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: sometimes your dataset is a mile wide—hundreds of sensors, tokens, click features—and a lot of those axes are redundant noise. Dimensionality reduction squeezes the story into fewer coordinates that still carry the plot. Think of summarizing a huge map with a clean subway diagram: less clutter, same rides that matter. Boss wants models that train faster and don't drown in the curse of dimensionality—this is a go-to move (PCA, autoencoders, careful feature selection). Exam trap: adding features, training on fewer rows, or one-hot encoding are not “reduction.” Land the takeaway: fewer informative variables from many. When you see “main variables" or “components," think reduce dimensions. You've done the right thing learning this—it shows up constantly.
Full explanation below image
Full Explanation
Dimensionality reduction is the process of representing data with fewer variables while preserving as much relevant information as possible for modeling, analysis, or visualization. Approaches fall roughly into feature selection, which keeps a subset of the original columns, and feature extraction or projection, which creates new axes such as principal components, linear discriminants, or nonlinear embeddings. Common goals include mitigating the curse of dimensionality, reducing multicollinearity and noise, speeding training and inference, enabling two- or three-dimensional visualization, and sometimes improving generalization when many raw features are redundant or weakly informative relative to sample size.
The correct definition emphasizes obtaining a smaller set of principal or informative variables from a larger original feature set. Techniques include principal component analysis and singular value decomposition, truncated SVD on sparse text matrices, manifold methods such as t-SNE and UMAP that are mainly used for visualization, autoencoders that learn compressed latent codes, and filter, wrapper, or embedded selection methods that score or search original features. Choice depends on whether you need invertibility, supervised class separation, sparsity, computational budget, or interpretability of the resulting components for stakeholders.
Adding engineered interaction terms or polynomial expansions increases dimensionality rather than reducing it. Training on a smaller sample reduces the number of rows N, not the feature dimension d, so it is not dimensionality reduction. Encoding categoricals changes representation type and often increases d when one-hot indicators are used; that step answers a different pipeline need. These neighboring operations are valuable, but they do not match the definition asked on the exam.
Best practices include fitting reduction transforms only on training folds to avoid leakage into validation metrics, choosing the number of components via explained variance, cross-validation utility, or domain constraints, and remembering that aggressive reduction can discard rare but critical signals. Exam memory aid: reduction shrinks the feature axis; if an option grows features, shrinks only rows, or only rewrites categories, eliminate it. Connecting dimensionality reduction to the curse of dimensionality helps: fewer well-chosen dimensions can restore workable data density and more stable geometric relationships for many classical algorithms.