# Neural Scaling Laws, Simply Explained

#50 May 12, 2025 Training

https://www.aibutsimple.com/p/neural-scaling-laws-simply-explained-16c1

Neural scaling laws are mathematical relationships that show how a neural network’s performance scales, such as its test loss or error rate, when you increase the number of parameters (model size), the amount of training data, or the computational resources used.

These laws have become a blueprint for building modern deep learning models—primarily in the development of LLMs in natural language processing (NLP), but also in other areas like deep convolutional neural networks (CNNs) in computer vision.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/51ad56eb-4328-4e69-8ed8-8dbb3c1fd429/Copy_of_Copy_of_dbsdadas.png)

The neural scaling laws show that it is possible to approximate the loss of an ML system by analyzing two main variables:

- The model size (N), measured in the number of parameters. Parameters are weights and biases in a model, which is adjusted in training. The more parameters there are, the more it can “learn” and “remember” from a dataset.
- The dataset size (D), typically measured in tokens, pixels, or other fundamental units.

Additionally, the amount of computational resources, known as compute (C), is included in the analysis, as it is the resource that enables more parameters and more data. Compute is measured in the number of calculations performed over time.

- The metric for compute is FLOP/s, the number of floating-point operations a computer performs per second.

The compute is typically fixed; organizations have a given budget, and scaling laws can determine the ideal model and dataset size given that budget.

### Bigger Model or More Data?

Neural scaling laws indicate that performance in the form of test loss improves predictably, following a mathematical curve called a power law. This means doubling your model size might reduce errors by a consistent percentage, making developing efficient models easier.

- A power law is a type of relationship between two quantities where one quantity varies as a power of another.

When plotted on logarithmic scales for both axes (what we call a log-log plot), the graph of a power law becomes a straight line. This is the most common way to visualize power laws.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/84522b0d-0b36-456c-9a72-1c169764e752/download__4___1_.png)

To clearly observe the power laws’ scaling behavior in the neural scaling laws, you vary one of the variables (N, D, or C) while holding the others fixed. This isolates a one-dimensional power law, and it is how the loss appears linear on a log-log plot.

Power laws are very common across a wide range of fields in nature and in society. For instance, animals’ metabolic rates are proportional to an exponent of their body mass, demonstrating a clear power law.

- Some other examples include city populations and word frequencies in language. They are also seen in chemistry for analyzing rates of reactions.

### Timeline of Neural Scaling Laws

Neural scaling laws emerged as deep learning models grew in complexity and scale. Empirical evidence began with Hestness et al. in their 2017 study, “[Deep Learning Scaling is Predictable, Empirically](https://arxiv.org/pdf/1712.00409?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained)”.

They found that test loss (L) scales as a power law with dataset size (D), with a proportionality statement L∝D−α , where α varies by task (between 0.07 and 0.35 for tasks such as image classification and language modeling). This work established that scaling is predictable and dependent on the task at hand.

#### OpenAI’s Scaling Laws

In 2020, Kaplan et al. published a groundbreaking paper, “[Scaling Laws for Neural Language Models](https://arxiv.org/pdf/2001.08361?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained)”, on scaling laws for large language models, focusing on transformer-based architectures.

They observed that the cross-entropy loss scales as power laws with model size (L∝N−αN​), dataset size (L∝D−αD), and compute (L∝C−αC), with trends holding true over seven orders of magnitude.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b326b34f-b873-4f85-8e53-533771ff5f9f/image.png)

They also introduced something called the compute-efficient frontier, the optimal boundary with the best possible performance a model can achieve given a fixed amount of compute, a line that no model they produced could cross. It is seen in orange in the left-most graph.

Their findings suggested that larger models are more sample-efficient, and optimal compute usage involves training large models on modest data, stopping before convergence. They proposed a scaling ratio where model size scales faster than dataset size (N∝C0.73,D∝C0.27).

They suggest that by increasing the model size more with less data, they should have a more efficient model given a fixed amount of compute. This is why GPT-3 has a lot of parameters (175B) and not enough data (300B tokens)—roughly 1.7 tokens per parameter.

Kaplan and his team also introduced how to determine the number of FLOP/s (which is a measure of compute, C) a training run uses:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d0242fc4-4513-426c-999b-71fe02b7c54e/image.png)

#### Chinchilla Scaling Laws

The 2020 paper suggested an optimal scaling ratio where model size grows faster than dataset size, leading to GPT-3 with 175B parameters and 300B tokens.

Hoffmann et al. decided to challenge this view in their 2022 paper, “[Training Compute-Optimal Large Language Models](https://arxiv.org/pdf/2203.15556?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained)”, introducing what is known as the Chinchilla scaling laws.

By training over 400 transformer models ranging from 70 million to 16 billion parameters on 5 billion to 500 billion tokens, they found that existing large language models (LLMs) like GPT-3 (175B parameters), Gopher (280B), and Megatron-Turing NLG (530B) were undertrained—there was insufficient data relative to their size.

They proposed that for compute-optimal training, model size and training tokens (dataset size) should scale equally (with a 1:1 ratio), doubling both for every doubling of compute.

---

The mathematical formulation behind this 1:1 ratio is as follows. Hoffman and his team determined the combined loss L(N,D) to be a sum of the reciprocals of the model size and dataset size multiplied by constants and additional error terms.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2828d224-ec19-45cb-8f83-d756cc4834cf/image.png)

Here, A = 406.4, B = 410.7, α = 0.34, β = 0.28, L0​ =1.69, and E is an irreducible loss, since loss will never be zero.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/deaeaa50-d7ea-4751-87db-712dfe1761fe/image.png)

Since α≈β, the optimal model size and dataset size are approximately proportional to C0.5 , leading to a 1:1 scaling ratio.

For more math explanations, consider [upgrading](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained)!

---

For a fixed compute budget, the paper suggests that model performance improves more by using smaller models trained on more data rather than larger models trained on less data, like many of the largest LLMs that were popular at the time.

- They came up with a rule of thumb, which is approximately 20 tokens per parameter for decoder-only LLMs.

This led to the Chinchilla model (70B parameters, 1.4T tokens), which outperformed larger models using the same compute budget.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e1b8a929-8c03-43ab-b55c-66e1dbcb425f/image.png)

To calculate the optimal number of parameters and tokens needed for Chinchilla given a fixed compute target, the creators extrapolated the numbers with a 1:1 model size and dataset size scaling ratio. A visual is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3bc09d75-a115-4695-8c94-e5a25d303ea6/Copy_of_Copy_of_CV_illust__4_.png)

The Chinchilla scaling laws describe and analyze two limiting factors of model scaling:

- Variance-limited scaling, which is performance improvement that is limited by statistical variance. Small datasets limit performance due to statistical variance, akin to needing more samples for accurate statistical estimates.
- Resolution-limited scaling, in which performance is limited by the model’s ability to approximate the structure of the data. Large datasets require models complex enough to figure out the underlying data manifold—the high-dimensional structure of the data.

#### Other Scaling Laws

Other scaling laws have been introduced in later research, exploring scaling in other domains, such as [Zhai et al. (2022)](https://arxiv.org/pdf/2106.04560?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained) for vision transformers and [Frey et al. (2023)](https://www.nature.com/articles/s42256-023-00740-3?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained) for chemical models. Scaling laws span across various fields but have domain-specific exponents.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b3b9d652-f90b-41f6-ac80-0a8547abac44/image.png)

Scaling Vision Transformers up to 3 Billion Parameters

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-scaling-laws-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team