# Principle Component Analysis (PCA), Simply Explained

#72 Oct 13, 2025 Theory

https://www.aibutsimple.com/p/principle-component-analysis-pca-simply-explained

Dealing with high-dimensional data is one of the biggest challenges in modern data science and machine learning. Images, text embeddings, genomic data, or sensor streams used as data can often contain thousands of [features](https://en.wikipedia.org/wiki/Feature_(machine_learning)?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained) (input variables).

Such high-dimensional datasets carry a huge amount of information, but they also make computation more expensive, models harder to train, and results more difficult to interpret.

This is where dimensionality reduction comes in. The central idea of this concept is simple: can we represent the data with fewer variables while still retaining most of the important information?

One of the most powerful, widely used, and simple techniques for dimensionality reduction is Principal Component Analysis (PCA).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/85dcc2c2-0ae5-41aa-85ef-53bd812e2ad2/PCA_Process.png)

PCA is a linear mathematical method that transforms high-dimensional data into smaller-dimensional data with a set of new variables called principal components. These components are uncorrelated, and each component captures as much variance (spread) in the data as possible.

### Why Dimensionality Reduction?

Before we dive into PCA, let’s first understand why we even need to reduce the dimensionality of data.

1. Visualization: This is arguably the most popular use case. Being able to plot and analyze data is useful—but it’s impossible to visualize data directly beyond 3D. Reducing higher-dimensional data to 2D or 3D makes it possible to plot and understand patterns.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/75117321-02c1-4ad5-a6cd-bf2ce034a7fd/image.png)

1. Noise Reduction: High-dimensional features often include noise. PCA filters out low-variance (low information) directions in the data, keeping only the most informative parts.
2. Faster Computation: Many ML/AI algorithms scale poorly with an increasing number of features. Reducing the dimensions helps them speed up.
3. Avoiding Overfitting: Too many features can lead to models memorizing the noise rather than learning true patterns in the data.

Let’s walk through a simple use case of PCA. In facial recognition, each image may have thousands of pixels. But not all pixels are equally important.

PCA finds the directions of maximum variation (overall face shape, eye distance, etc.) and compresses the dataset into a smaller set of features that can still capture a person’s identity.

---

### Go from AI overwhelmed to AI savvy professional

[![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2c848121-9d72-4a63-bd9d-5e8362d2ff4c/5.jpg)](https://www.superhuman.ai/c/confirmation?magiclink_subscription)

AI will eliminate 300 million jobs in the next 5 years.

Yours doesn't have to be one of them.

Here's how to future-proof your career:

- Join the [Superhuman AI](https://www.superhuman.ai/c/confirmation?magiclink_subscription) newsletter - read by 1M+ professionals
- Learn AI skills in 3 mins a day
- Become the AI expert on your team

[Start learning AI now](https://www.superhuman.ai/c/confirmation?magiclink_subscription)

---

### How Does PCA Actually Work? (Variation and Information)

The core idea of PCA is based on variance. Variance measures the spread of a dataset. The key to representing the data with fewer variables while still retaining most of the important information is to look at the variance in the data.

Let’s look at an example. Below, we have the plot of two features, height and weight. The points form an elongated cloud that stretches diagonally:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a0590a01-27e3-4e42-ade1-a58a8b259731/pca_fig1_scatter.png)

Instead of describing each point by height and weight, we could rotate the axes so that one new axis lies along the longest stretch of the cloud.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/aac502fa-a26e-487a-ab75-93f1f77892bb/pca_fig4_projection_pc1.png)

This new axis captures the maximum variance in the data; it explains most of the differences between individuals. That axis is the first principal component (PC1).

The second axis, orthogonal to the first, captures the next largest variation (PC2).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d710e2cf-07f0-40de-8ea5-7995ee03f695/pca_fig2_axes.png)

We find that PC1 captures the most variation, so if we reduce this 2D data into 1D, collapsing the points onto PC1 is a better choice than collapsing the points onto PC2:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bd9a5bf1-c7d1-40de-b285-caca006bfcc0/pca_fig3_explained_variance.png)

In higher dimensions, PCA generalizes this idea: find orthogonal directions that maximize variance.

### Mathematical Foundation of PCA

Now, let’s go step by step through the mathematics of PCA:

Step 1: Standardizing the data

PCA is sensitive to scale. For example, if one feature is measured in kilograms and another in meters, the larger scale will dominate. Therefore, we first subtract the mean (μ) and divide by standard deviation (σ) to standardize the data (x*):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/86e879f3-0c78-4bea-9783-25d69cb3246d/image.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6ebda6b6-64d4-4d1b-9fcb-03d0b610d32c/PCA_Standardization.png)

Step 2: Covariance matrix

The covariance matrix tells us how features vary with respect to each other. For a dataset X (with n samples and d features), the covariance matrix is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e3930b88-b35b-4da6-98b6-cb0b34369086/image.png)

This is a d×d matrix where each entry shows how feature i and j vary together

Step 3: Eigenvalues and Eigenvectors

The key insight: the directions of maximum variance (principal components) in the data are given by the [eigenvectors](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained) of the covariance matrix, and the amount of variance is explained by its [eigenvalues](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0ba873be-c0fb-4c27-be66-261491183dd7/Copy_of_PC1.png)

Step 4: Sort and select

We sort eigenvalues from largest to smallest. The top k (k being the number of desired dimensions) eigenvectors corresponding to the largest eigenvalues make up the projection matrix Wk.

Step 5: Projection

Finally, we project the original data onto the new subspace (Z) by multiplying our dataset (X) by Wk:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6504f63d-e425-43c5-b407-e4d0ea18f763/image.png)

Z becomes the reduced dataset with k dimensions.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/37c47d44-2cb9-4c50-b600-4c148117af57/Reducing_Dimensions.png)

Let’s go through a worked example of PCA, going from 2D data to 1D data. Let’s say we have some sample data points of heights and weights.

| Height | Weight |
| --- | --- |
| 160 | 55 |
| 165 | 60 |
| 170 | 62 |
| 180 | 785 |

If we plot these points, we’ll see they form a diagonal cloud, as can be seen in the earlier images: taller people tend to weigh more.

- PCA finds the diagonal axis (PC1) that best explains the data.
- Projecting onto this axis reduces the dataset from 2D to 1D, keeping most information.

This is why PCA is often visualized as “rotating” the coordinate system.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained)

---

### Applications of PCA in Machine Learning

PCA is commonly used in data visualization, where high-dimensional embeddings (like word vectors or image features) are reduced to 2D for plotting. A good example is visualizing clusters of documents in NLP.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/56202680-4a8e-4563-ac38-6d154829fd23/Copy_of_Data_Visualization_High-D___2D_Scatter.png)

PCA is also used for noise reduction. We keep only top principal components and ignore those with tiny eigenvalues (low variance = mostly noise).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/87dfd930-8920-4cfe-91d5-3dec305e5b71/Noise_Reduction_PCA.png)

Compression is also a common use case for PCA. In image compression, PCA can represent images with fewer dimensions while maintaining or changing image quality.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ab603bd0-c84b-4602-89c2-8c9d31bb876e/Image_compression_PCA.png)

PCA is often used before applying algorithms like K-means clustering or SVM, making them faster and more accurate—PCA is used for preprocessing.

### Limitations of PCA

PCA has limitations; it is not always the best choice:

- Linearity: PCA assumes data is linearly correlated. Nonlinear patterns won’t be captured well.
- Scaling: Sensitive to feature scaling—standardization is essential.
- Interpretability: Principal components are combinations of features, sometimes hard to interpret.
- Global Method: PCA finds global variance directions, not local structures.

### Beyond PCA

There are a multitude of algorithms beyond PCA, including [t-SNE](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained) (t-distributed Stochastic Neighbor Embedding) and [UMAP](https://umap-learn.readthedocs.io/en/latest/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained) (Uniform Manifold Approximation and Projection).

T-SNE is great for visualization of local clusters, while UMAP preserves both global and local structure. These algorithms do provide more precision in nonlinear and complex scenarios, but as a tradeoff, they are more technically advanced and more computationally demanding.

All in all, Principal Component Analysis (PCA) is one of the most important tools in the machine learning toolkit.

From its mathematical roots in eigen decomposition to its widespread applications in visualization, compression, and preprocessing, PCA provides a powerful way to simplify high-dimensional data without losing essential information.

While newer nonlinear techniques like t-SNE and UMAP exist, PCA remains the default first step in exploring and understanding complex datasets.

As Karl Pearson, who first introduced PCA in 1901 put it: PCA helps us “see the forest for the trees,” that is, distill large, noisy datasets into their most informative directions.

---

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=principle-component-analysis-pca-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team