# The Lottery Ticket Hypothesis, Simply Explained

#92 Mar 9 Theory

https://www.aibutsimple.com/p/the-lottery-ticket-hypothesis-simply-explained

Deep neural networks are usually significantly over-parameterized. Modern models contain millions or billions of parameters, yet it has been routinely observed that large portions of them can be removed after training with little loss in performance.

But the natural follow-up question is deeper: Are these extra parameters just redundant, or were they necessary to discover the final solution?

In their landmark paper, “[The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://arxiv.org/abs/1803.03635?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained),” Jonathan Frankle and Michael Carbin propose a bold answer.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fa28533e-c494-4d5e-b86b-33b1d52c3e0a/Lottery_Ticket_Hypothesis_10__Size_Same_Performance.png)

Large networks contain small subnetworks at initialization that are already capable of training to full accuracy, but only if you find them.

They call these subnetworks winning tickets. These winning tickets are 10-20% of the dense network size but have comparable accuracy.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/238ec138-5366-44d9-9efe-24699a58545b/Lottery_Ticket_Hypothesis_Dense_vs_Sparse_Subnet__1_.png)

### The Core Hypothesis

Consider a neural network (θ0) with parameters randomly initialized.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ee6cf32-9cf0-4cea-ad1a-795b84621711/image.png)

Here, D is a probability distribution, typically Gaussian. After training the model with stochastic gradient descent (SGD) on some training data (x), we obtain:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cb4b78e0-8a17-48e4-9de3-19d9406d2cf7/image.png)

Now, let’s define a binary mask that selects a subset of weights. This is typically a matrix that, when multiplied with the model weight matrix, sets some weights to 0 and leaves the rest untouched.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/63736f06-74eb-4bec-a15b-ae4e8ea0da75/image.png)

The masked network is the element-wise product (⊙) of the binary mask and the network weights. This results in a sparse network.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3fabe9dd-255d-4d6b-a5c3-1fe3af59f31a/image.png)

The Lottery Ticket Hypothesis states that there is a sparse mask (m) such that:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/dd009050-9865-4d14-8f43-33bf759fd41b/image.png)

This means that after full training, the sparse model (m ⊙ θ0) has comparable accuracy and training speed with just a fraction of the parameters.

For this to be true, two things must hold. First, the subnetwork should be sparse. Second, the subnetwork must use the original initialization values.

That second condition is crucial.

### Why Initialization Is the Key

One of the paper’s most surprising discoveries is that if you take the same sparse structure (m), but reinitialize the weights randomly, it no longer trains to the same high accuracy.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/51e6fd8b-0372-4f29-bd60-968b351d0e51/image.png)

So, structure alone is insufficient. The winning ticket depends on:

- The exact subnetwork topology
- The exact initial weight values

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/68fda5e1-ae6d-494f-bd08-9756b61b7004/Lottery_Ticket_Hypothesis__Dense_Network__1_.png)

This suggests that random initialization is crucial in seeding many potential “good training trajectories,” and pruning reveals the lucky ones. This is the lottery metaphor.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained)

---

### How The Algorithm Works: Iterative Magnitude Pruning (IMP)

The authors do not magically find winning tickets. Instead, they discover them through an algorithm called iterative magnitude pruning.

Iterative magnitude pruning works by first initializing a dense network θ0.

Then, we iteratively:

1. Train the network to convergence to obtain (θT)
2. Prune a percentage (%p) of the smallest-magnitude weights by using a sparse mask (m)
3. Reset the remaining weights to their original values in (θ0)

We repeat this cycle until we achieve the desired sparsity, typically 10-20%.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c4082135-c675-4891-a787-9afa08a8ebac/Lottery_Ticket_Hypothesis_Iterative_Magnitude_Pruning.png)

Mathematically, the network gets more sparse by applying the pruning mask (mk) element-wise to the weights:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a2e994b1-a2f5-42fb-9d7e-82ac1cd1608b/image.png)

We then reset the remaining weights to the original dense initialization (which is one of the key reasons it works), then retrain the network, repeating the cycle.

Each iteration of the pruning algorithm increases sparsity while attempting to preserve accuracy and does so fairly well, according to the paper.

Although simple and effective, this procedure is computationally expensive. It requires retraining multiple times, but it systematically uncovers sparse subnetworks that retain the same level of performance.

### Experimental Results

The paper evaluates deep feedforward networks and convolutional networks using the IMP algorithm. They test:

- LeNet (on [MNIST](https://www.kaggle.com/datasets/hojjatk/mnist-dataset?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained) dataset)
- Conv-6 (on [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained) dataset)
- ResNet variants

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0746375a-f9b4-4bce-ba3f-8c6ca94cb2fd/Lottery_Ticket_Hypothesis__Tested_Architectures.png)

The findings are striking:

- Networks can be pruned to 10-20% of original size with no loss.
- Some networks can be pruned to <5% with only slight degradation.
- Random reinitialization destroys performance.
- Training a small dense network from scratch underperforms the winning ticket.

Another key finding is that when the sparse ticket is larger than the 10-20% target size, we can actually see an increase in performance, even greater than the original dense model itself.

This last point is quite important. A small network of equal parameter count does not match the winning ticket. So sparsity alone is not the explanation.

### Searching for the Optimal Network

The Lottery Ticket Hypothesis reframes how we think about large models.

Instead of realizing that over-parameterized networks generalize surprisingly well, they suggest large networks work because they contain many trainable sparse subnetworks.

In this view, the dense model acts as a search space over subnetworks.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ea633ab0-19e1-4d16-acf4-02791af927a3/Lottery_Ticket_Hypothesis__Small_Network_vs_Large_Networks__1_.png)

So, stochastic gradient descent does not create a dense solution from scratch. Rather, it helps identify and bring forth one of many already viable subnetworks.

The optimization landscape or manifold becomes more clear: the dense network initialization contains many local and global minima that minimize the loss.

Since a dense network has so many ticket subnetworks, there is a high probability that at least one of them (by luck) has its initial weights positioned close to these minima, within what we call the basin of attraction, guiding the network to converge at the minima.

### The Role of Training Dynamics

But why can’t we just train small networks directly? The experiments show that small, dense networks converge slower, achieve worse accuracy, and are harder to optimize.

The winning ticket, however, inherits the favorable gradient flow properties of its position inside the larger network.

In other words, large models may be easier to optimize, but not because they need all parameters, but because they contain substructures that make the optimization process easier.

### A Math Perspective on a Sparse Network

Let L(θ) be the training loss. The training trajectory is defined by the gradient descent update equation, found below.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c49e2c06-580a-4fea-b379-f68a678bd113/image.png)

The hypothesis implies that there exists a sparse mask (m) such that the restricted (sparse) parameter space still contains a path under SGD that reaches low loss.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6b1c7cec-7f2d-4623-8738-f4e5036a5693/image.png)

The winning tickets have a much easier descent into the local and global minima, requiring less training time.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1ff0add3-8b5d-4144-b27f-c1dac4dc4122/Lottery_Ticket_Hypothesis_SGD_Path_Comparison.png)

The full, dense model does not necessarily need its full dimensionality, but it simply increases the probability that a good subspace exists.

### Limitations

Despite its impact, the original paper had some constraints. First, experiments were on relatively small datasets, only on feedforward networks and CNNs. They did not explore this phenomenon with transformers or other architectures.

Additionally, large-scale ImageNet models were harder to prune, showing some difficulty extending this theorem to larger models. IMP also requires multiple full training cycles, which can be computationally expensive with all the retraining.

Finally, they did not find a way to obtain winning tickets before training, which would largely speed up the process.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/413cc833-85a7-4fe1-92cb-f35a0ac646c1/Lottery_Ticket_Hypothesis_Training_Cycle.png)

Later work introduced learning rate rewinding, early pruning techniques, and connections to neural tangent kernel theory.

But the core open question remains: can we identify winning tickets at initialization, without full training?

In today’s era of billion-parameter language models in AI, the Lottery Ticket Hypothesis is an important direction. Large foundation models often display surprising redundancy and compressibility.

Sparse fine-tuning methods and structured pruning techniques echo the idea that dense networks may merely host more efficient internal structures.

However, whether true winning tickets exist at a massive scale remains an active area of research. It is possible that scale changes the story or that winning tickets become easier to find as models grow larger.

### Conclusion

The Lottery Ticket Hypothesis changed how researchers think about model size.

Its key contributions:

- A precise, testable hypothesis
- A reproducible pruning methodology
- A new framing of over-parameterization

It shifted the conversation from “How can we compress trained networks?” to “Were smaller, similarly accurate networks already there all along?”

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/61ef4c73-e4c7-4817-8c57-3b4026e06584/Lottery_Ticket_Hypothesis_Conclusion_and_Parameter_Reduction.png)

This reframing changes how we think about architecture design, compression, and optimization.

It suggests that progress in deep learning may not always come from scaling alone but from understanding the sparse structures that make training possible. More than a pruning technique, the paper is a conceptual reorientation of deep learning theory.

If the hypothesis holds broadly, then the future of neural networks may not be about building massive systems but about understanding the sparse structures hidden inside them.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici
- Yassine Ennaji

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-lottery-ticket-hypothesis-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team