Issue #92 Theory Edwin Dong, Anurag Shinde
The Lottery Ticket Hypothesis, Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook that will intuitively show how a large neural network contains specific, smaller subnetworks, that can match the performance of the full network with a fraction of the size.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Deep neural networks are usually significantly over-parameterized. Modern models contain millions or billions of parameters, yet it has been routinely observed that large portions of them can be removed after training with little loss in performance.
But the natural follow-up question is deeper: Are these extra parameters just redundant, or were they necessary to discover the final solution?
In their landmark paper, “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,” Jonathan Frankle and Michael Carbin propose a bold answer.
Large networks contain small subnetworks at initialization that are already capable of training to full accuracy, but only if you find them.
They call these subnetworks winning tickets. These winning tickets are 10-20% of the dense network size but have comparable accuracy.
This edition is for PRO members.
