Why does PCA work well while the total variance retained is small?

Why Does PCA Work Well Even When the Retained Variance Is Small?

Principal Component Analysis (PCA) is often praised for its ability to simplify high‑dimensional data while preserving the most important structure. Yet, practitioners sometimes notice that a relatively low proportion of the total variance—say 30 % or 40 %—can still yield excellent downstream performance. How can a method that “throws away” most of the variance remain so effective? The answer lies in understanding what variance actually measures, how PCA aligns with the underlying signal, and why many learning tasks care more about information than raw variance.

1. Variance Is Not Synonymous With Information

Variance quantifies the spread of data points around their mean. In many real‑world datasets, a large chunk of that spread originates from noise, measurement error, or irrelevant factors:

• Noisy dimensions: Sensors often produce high‑frequency fluctuations that inflate variance without contributing useful patterns.
• Redundant features: Multiple variables can capture the same underlying phenomenon, spreading variance across many dimensions.
• Irrelevant variability: In image data, lighting changes can dominate pixel‑wise variance but are unimportant for object classification.

Because PCA ranks components by variance, the first few components tend to capture the directions that dominate the data distribution—including both signal and noise. When most of the variance is noise, discarding it can actually improve the signal‑to‑noise ratio.

2. PCA Maximizes Signal Alignment, Not Variance Preservation

Consider a dataset that lies near a low‑dimensional manifold embedded in a high‑dimensional space. The variance along directions orthogonal to that manifold is often tiny, yet those directions are crucial for describing the manifold’s shape. PCA’s eigenvectors are oriented to capture the directions of greatest variance, which, in many cases, align with the true underlying factors of variation:

• Latent structure: If the data are generated by a few latent variables, the principal components will span the subspace spanned by those latent variables, regardless of how much total variance they explain.
• Linear separability: For classification tasks, the decision boundary often depends on a subspace where the classes are most separable, not necessarily the subspace with the highest variance.

Thus, even when the retained variance is modest, the projected space can preserve the essential relationships needed for downstream models.

3. Downstream Algorithms Often Require Only the “Right” Directions

Many machine learning models (e.g., linear classifiers, k‑nearest neighbors, clustering algorithms) are invariant to rotations and scaling of the feature space. They primarily need:

1. Clear separation between classes or clusters.
2. Reduced dimensionality to avoid the curse of dimensionality.

PCA provides exactly that by delivering a compact, orthogonal basis that emphasizes the most discriminative directions. Even if the retained components explain only 30 % of the variance, they may already contain the discriminative cues that the classifier exploits.

4. Empirical Evidence: High Accuracy with Low Variance Retention

Numerous studies report that models trained on PCA‑reduced data achieve comparable—or sometimes superior—performance to those trained on the full feature set:

• Image recognition: Using only the first 50–100 principal components (often < 10 % of the total variance) can retain > 95 % of classification accuracy on datasets like MNIST.
• Text classification: Latent Semantic Analysis (LSA) applies truncated SVD (a PCA variant) and achieves strong results while preserving a small fraction of variance.
• Genomics: In high‑throughput gene expression data, a handful of PCs capture the biological signal, while the majority of variance stems from batch effects and noise.

5. Practical Guidelines for Using PCA Effectively

1. Inspect the scree plot: Look for an “elbow” where the eigenvalue drop stabilizes. The components before the elbow often contain the meaningful structure.
2. Validate downstream performance: Instead of targeting a specific variance threshold (e.g., 95 %), evaluate model accuracy as you vary the number of retained components.
3. Combine with domain knowledge: If you know certain features are crucial, ensure they are represented in the retained subspace (e.g., by augmenting PCA with supervised dimensionality reduction).
4. Consider whitening or scaling: Standardizing features before PCA prevents high‑variance but irrelevant dimensions from dominating the components.

Conclusion

PCA works well even when the retained variance is small because variance is a proxy for noise as well as signal. By focusing on the directions that capture the most systematic variation, PCA often isolates the underlying structure that matters for prediction, clustering, or visualization. The key is to remember that “explained variance” is a useful diagnostic, not a strict performance metric. In practice, the true test of PCA’s effectiveness is how well downstream models perform after the dimensionality reduction—not how much of the raw variance remains.