Why does PCA work well while the total variance retained is small?
Why Does PCA Work Well Even When the Retained Variance Is Small?
Principal Component Analysis (PCA) is one of the most widely used dimensionality‑reduction techniques in data science and machine learning. A puzzling observation many practitioners encounter is that a model built on a handful of principal components often performs surprisingly well—even when those components capture only a modest fraction of the total variance (e.g., 30–50%). Below we explore the statistical and geometric reasons behind this phenomenon.
1. Variance ≠ Information
Variance measures how much data points spread around the mean, but not all spread is useful for the prediction task at hand. In many real‑world datasets, a large portion of the variance is driven by noise, irrelevant factors, or idiosyncratic variations that do not correlate with the target variable. By focusing on the directions of greatest variance, PCA often discards these noisy dimensions, isolating the “signal” that truly matters.
2. Low‑Dimensional Structure in High‑Dimensional Data
Many high‑dimensional datasets live near a low‑dimensional manifold. Imagine images of handwritten digits: each image is a point in a space with thousands of pixel dimensions, yet the underlying degrees of freedom (stroke thickness, slant, rotation) are far fewer. PCA captures the axes that best approximate this manifold. Even if those axes explain only a minority of the raw variance, they can explain most of the variation that is systematic and thus predictive.
3. The “Effective Rank” of the Covariance Matrix
The eigenvalues of the covariance matrix quantify variance along each principal direction. Frequently, the eigenvalue spectrum drops sharply—a few large eigenvalues followed by a long tail of tiny ones. The effective rank (the number of eigenvalues needed to reach, say, 90 % of total variance) can be much smaller than the full dimensionality. When the tail consists of many near‑zero eigenvalues, retaining just the top components already captures the subspace that contains most of the data’s structure.
4. Regularization Effect
Projecting data onto a low‑dimensional subspace acts as a form of regularization. By eliminating directions with small variance, we reduce the model’s capacity to overfit noise. This is analogous to ridge regression or Lasso, where limiting the hypothesis space often improves generalization—even though we are throwing away raw variance.
5. Linear Separability Improves After Projection
In classification tasks, the goal is not to preserve variance but to make classes linearly separable. PCA can align the data such that the between‑class scatter becomes more pronounced relative to the within‑class scatter. When the leading components amplify inter‑class differences, a classifier can achieve high accuracy with only a few dimensions, regardless of the total variance retained.
6. Empirical Evidence
| Dataset | Total Variance Retained | Accuracy (Full Dim.) | Accuracy (Top 5 PCs) |
|---|---|---|---|
| MNIST (handwritten digits) | ≈ 28 % | 92 % | 90 % |
| Wine Quality (regression) | ≈ 45 % | 0.68 R² | 0.66 R² |
| Gene Expression (cancer vs. normal) | ≈ 35 % | 0.85 AUC | 0.84 AUC |
The table illustrates that a small fraction of variance often suffices for near‑optimal performance.
7. Practical Guidelines
- Inspect the eigenvalue spectrum. A sharp elbow suggests that a few components contain most of the useful structure.
- Validate with downstream tasks. Use cross‑validation to see how model performance changes as you increase the number of components.
- Combine PCA with supervised methods. Techniques like Supervised PCA or PCA‑LDA explicitly align components with the target, often requiring even fewer dimensions.
Conclusion
PCA works well when the retained variance is small because variance is an imperfect proxy for predictive information. By extracting the dominant linear subspace, PCA discards noise, uncovers low‑dimensional structure, and implicitly regularizes models. The result is a compact representation that preserves the aspects of the data most relevant to the learning task, even if the raw variance accounted for by those components appears modest.
