Why does PCA work well while the total variance retained is small?

Why Does PCA Work Well Even When the Retained Variance Is Small?

Principal Component Analysis (PCA) is a staple dimensionality‑reduction tool in the AI toolbox. Practitioners often observe that models built on a handful of principal components (PCs) can achieve high accuracy, even though those components capture only a modest fraction of the total variance in the original dataset. This apparent paradox can be explained by several intertwined concepts that go beyond the raw variance numbers.

1. Variance vs. Information

Variance is a measure of spread, not of relevance. In many high‑dimensional data sets, a large portion of the variance is driven by noise or irrelevant factors (e.g., measurement error, lighting variations in images, or random background pixels). PCA isolates directions with the greatest variance, but the most predictive directions for a specific task often lie among the top few components, while the remaining variance may be largely uninformative.

2. Low‑Dimensional Structure in High‑Dimensional Data

Real‑world data used in AI often lies near a low‑dimensional manifold. Even if that manifold is embedded in a space of thousands of dimensions, its intrinsic dimensionality may be only a few dozen. PCA’s job is to discover the axes that span this manifold. Once the manifold is captured, adding more components yields diminishing returns because they merely describe variations orthogonal to the manifold (i.e., noise).

3. Signal‑to‑Noise Ratio Improves with Projection

When data is projected onto the first k principal components, the signal that is common across many samples gets amplified relative to random fluctuations. This “denoising” effect often leads to better downstream performance, even though the projected data retains only, say, 20–30% of the original variance.

4. Class Separability Doesn't Correlate Linearly with Variance

Supervised learning cares about the separation between classes, not about how spread out the data is overall. A small set of PCs can align the classes more cleanly than the full high‑dimensional representation. For example, in image classification, variations due to background or pose can dominate total variance but be irrelevant for identifying the object; removing those dimensions can actually increase class separability.

5. Regularization Effect

Reducing dimensionality acts as an implicit regularizer. By limiting the number of features, we constrain the hypothesis space, reducing the risk of overfitting. This regularization often outweighs the loss of variance, especially when the training set is limited.

6. Empirical Evidence

  • MNIST Digits: Retaining just 30 PCs (≈ 15% variance) still yields > 95% classification accuracy with simple linear classifiers.
  • Text Embeddings: Applying PCA to transformer embeddings and keeping only 50 components (often < 10% variance) preserves most semantic relationships.
  • Genomics: In gene‑expression studies, a handful of PCs can capture disease‑related patterns while discarding experimental noise.

7. Practical Guidelines

  1. Plot the explained variance ratio and look for an “elbow,” but don’t stop there—evaluate downstream performance.
  2. Use cross‑validation to determine the optimal number of components for your specific AI task.
  3. Combine PCA with supervised dimensionality reduction (e.g., LDA, supervised autoencoders) when class information is available.
  4. Remember that a low retained variance is not a flaw; it often indicates effective noise removal.

Conclusion

PCA works well even when the total retained variance is small because variance is a crude proxy for predictive power. By focusing on the dominant axes of variation, PCA discards noisy, irrelevant dimensions, highlights the underlying low‑dimensional structure, and regularizes models. The result is a compact representation that often captures the essence of the data needed for AI tasks, proving that quality of captured variance matters far more than its quantity.