Issue #81 Theory Edwin Dong, Anurag Shinde
Dimensionality Reduction: t-SNE and UMAP
Free editionPRO edition
This week’s PRO issue is accompanied by a Colab notebook with a simple coding introduction to dimensionality reduction. The notebook features a multi-layer perceptron implementation in full Numpy, building the network from the ground up to understand exactly how the math works behind the scenes.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Modern machine learning models operate in extremely high-dimensional spaces.
A single image can contain tens of thousands of pixel values, word embeddings often have hundreds of dimensions, and the hidden layers of deep networks may contain thousands of learned features.
While these high-dimensional representations are powerful, they are also very difficult to interpret, visualize, and analyze directly.
Dimensionality reduction is the process of transforming high-dimensional data into a lower-dimensional space while preserving the most important structure in the data.
By mapping data into two or three dimensions, we can visually explore clusters, analyze class separability, detect outliers, and understand how machine learning models organize information internally.
While Principal Component Analysis (PCA) provides a linear approach to dimensionality reduction, many real-world datasets lie on nonlinear spaces.
To address this, two powerful nonlinear techniques, t-SNE (t-Distributed Stochastic Neighbor Embedding) and UMAP (Uniform Manifold Approximation and Projection), have become standard tools for modern data visualization.
Before we continue, let’s unpack some general dimensionality reduction terms we should have an understanding of:
Manifolds, knowing that they are complex spaces that locally look like regular Euclidean space. For instance, Earth’s surface is curved globally but is flat enough locally for maps. The key assumption is that real-world data is concentrated near a low-dimensional manifold.
Neighborhoods, a way in which data points are considered “close” or “similar” to each other. This can be within a radius, a hyperparameter k, or even probability-based, like in t-SNE.
Perplexity, a t-SNE parameter controlling how neighborhoods are defined. Low perplexity gives local neighborhoods, while high perplexity gives larger, more global neighborhoods.
KL divergence (Kullback–Leibler divergence), a way to measure how different two probability distributions are. You can read more about KL divergence in our previous issue.
This edition is for PRO members.
