Issue #112 Interp. Edwin Dong, Lalit Julapalli
Sparse Autoencoders (SAEs), Simply Explained
Blackbox. That’s a term that’s floated around AI research for the last few years.
From the moment you submit your query to the LLM, researchers have been investigating the journey your words take through the neural network in order to understand why a certain output comes out the other side.
Our current architectures are defined by matrix operations that correspond to weights, biases, and probabilities but are semantically hard to interpret.
Take a picture of a dog, for example: with a simple CNN, we’ll be able to determine whether an image depicts a dog or not, but it’s not immediately apparent from the output logits that a dog is what’s identified.
Over the years, research has moved from trying to analyze the outputs to inside the network’s layers. This research area is called Mechanistic Interpretability.
Researchers try to reverse engineer a way to get model weights to try and see what happens to models at the neuron level.
However, they faced a huge issue: superposition, which is when a network tries to identify more features than it has neurons, causing them to overlap in what their activation weights correspond to.
The attribute of when neurons represent multiple things at the same time is polysemanticity, which complicates mechanistic interpretability further.
Trying to decipher the tangled neuron activations is the job of the Sparse Autoencoder (SAE).
We’ll provide a visual breakdown of what Sparse Autoencoders are and how they’re used in mechanistic interpretability research.
