What Is Principal Component Analysis (PCA)? A Simple Guide

Updated · By Viviai

Principal component analysis (PCA) is a dimensionality reduction technique that transforms a large set of correlated variables into a smaller set of uncorrelated variables called principal components, while retaining most of the variation in the data.

The problem it solves

A dataset with many columns is hard to plot, slow to train on, and often contains redundant measurements. For example, height in centimeters and height in inches carry the same information. PCA looks for these overlaps and combines them so the data can be described with fewer numbers without throwing away the main patterns.

How it works in plain terms

  1. The data is centered so each feature has a mean of zero.
  2. PCA measures how features move together using the covariance matrix.
  3. It finds directions, called principal components, along which the data spreads out the most.
  4. The first component captures the largest variation, the second captures the next largest, and so on, with each component at a right angle to the others.

The components come from the eigenvectors of the covariance matrix, and their eigenvalues indicate how much variation each one holds.

What a principal component is

Term Meaning
Component A new variable built as a weighted mix of the original features
First component The direction that captures the most variation
Loading The weight of each original feature inside a component
Explained variance The share of total variation a component accounts for

Common uses

PCA is often used to compress features before modeling, reduce noise, detect outliers, and plot high-dimensional data in two or three dimensions. It is also a common preprocessing step before clustering or nearest-neighbor methods.

Honest limitations

PCA only captures linear relationships, and the new components can be harder to interpret because each mixes several original variables. The number of components to keep is a modeling choice, often guided by the explained variance.

FAQ

Does PCA remove features?

It does not delete columns outright. It creates new components from them, and you choose how many components to keep.

How many components should I keep?

A common approach is to keep enough components to cover a large share of the explained variance, such as the first few that together account for most of it.

Is PCA a type of machine learning?

It is an unsupervised statistical method used within machine learning pipelines, rather than a model that predicts labels.

Does PCA work on categorical data?

PCA is designed for numeric variables. Categories are usually encoded first, and other techniques may fit non-numeric data better.

Key takeaways

Continue with what is clustering and what are embeddings.

Sources

  1. https://en.wikipedia.org/wiki/Principal_component_analysis
  2. http://ufldl.stanford.edu/tutorial/unsupervised/PCAWhitening/
  3. https://www.cs.ox.ac.uk/people/varun.kanade/teaching/ML-MT2017/lectures/lecture17.pdf

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free