# Neural Architecture Search (NAS)

#36 Feb 3, 2025 Architecture

https://www.aibutsimple.com/p/neural-architecture-search-nas

Before starting this issue, it would be helpful to understand how neural networks work while understanding their architecture and the mathematics behind them. If you don’t already, we have issues detailing this below:

- [Neural Networks, Simply Explained (Beginner)](https://www.aibutsimple.com/p/neural-networks-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas)
- [Math Behind Multi-Layer Perceptrons (Intermediate)](https://www.aibutsimple.com/p/multilayer-perceptrons-math-behind-deep-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas)

---

Designing neural networks has always been a time-consuming and challenging process. For many years, researchers have experimented with various neural network architectures, adjusting the type of connections, tweaking the size and ordering of layers, and fine-tuning the parameters—all in search of better performance.

This process has always been based on trial and error, with experts tweaking layers and parameters based on intuition and experience. This works, but it can be improved, since at most, developing an effective architecture is simply an educated guess.

That being said, the standard method of finding model architectures has led to many of the most influential and useful neural network architectures, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformers, and Generative Adversarial Networks (GANs).

But finding the optimal architecture is an immense challenge, needing many years of research and millions of dollars in funding. The sheer complexity of designing models has led researchers to attempt to streamline this process.

In this issue, we’re going to explore one of the most exciting new waves of deep learning research: Neural Architecture Search (NAS). It is an algorithm that automates the search for the best-performing neural network architecture.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a14e76b3-e6c4-4357-bf2d-953735b12533/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png)

Before NAS, designing neural networks required months of testing different configurations. Now, NAS uses algorithms to explore thousands of designs automatically, saving time and often producing better results.

NAS is similar to a standard [hyperparameter tuning](https://www.aibutsimple.com/p/hyperparameter-tuning-deep-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas) process, however, it tunes the model architecture instead of fixing an architecture—doing so automatically using many different search approaches.

### Seeking impartial news? Meet 1440.

Every day, 3.5 million readers turn to [1440](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_impartial&_bhiiv=opp_89f562cf-cd1e-4c02-a0de-b73f7ef191cb_1b75ca79) for their factual news. We sift through 100+ sources to bring you a complete summary of politics, global events, business, and culture, all in a brief 5-minute email. Enjoy an impartial news experience.

[Join for free today!](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_impartial&_bhiiv=opp_89f562cf-cd1e-4c02-a0de-b73f7ef191cb_1b75ca79)

### Neural Architecture Search Process

---

#### Search Space (What should we search?)

The search space defines all possible neural network architectures the algorithm can explore. This can range from simple architectures to highly complex ones.

For example, NAS can take into account layer types (convolutional layers for images, recurrent layers for text), connections (skip/residual connections, standard connections), the number of input/output channels, the number of neurons per layer, and the number of layers, just to name a few.

This is similar to defining a parameter space in hyperparameter tuning, searching through configurations of these parameters to find the optimal one.

#### Search Strategy (How do we search?)

The search strategy is the type of algorithm that searches through the search space. Popular search strategies for NAS include:

- Reinforcement Learning (RL), where an agent proposes architectures and gets rewarded for good performance. It is the oldest method.
- Gradient-Based Optimization, using gradient-based optimizers such as gradient descent or Adam to find the best configuration.
- Evolutionary Algorithms that mimic [natural selection](https://en.wikipedia.org/wiki/Natural_selection?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas), mutating and combining the best-performing designs.

#### Performance Estimation (How good is the model?)

Performance estimation is the evaluation of a specific configuration, narrowing down the search space.

However, training a neural network from scratch takes hours or days. Since NAS has thousands of configurations to consider, to save time, NAS uses certain tricks like weight sharing and proxy tasks.

Weight sharing involves reusing parameters of already trained networks (instead of training each candidate from scratch).

- In weight sharing, a super-network is built, made up of all possible architectures in the search space. When a new child architecture (a specific configuration of layers) is tested, it inherits weights from the super-network.

Proxy tasks are simplified or scaled-down versions of a complex task, which can be done by using a small number of epochs or a smaller dataset—sort of like a “practice run” to estimate the performance of such a model.

- Proxy tasks trade off some accuracy for speed and efficiency.

The general structure of a NAS process is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/23667d5e-91d8-4f9a-9c3f-69b3506c48ae/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_300_px___1_.png)

### Applications

---

#### Computer Vision

A notable application of NAS is computer vision, where it has been used to design high-performing convolutional neural networks (CNNs).

These networks, optimized for tasks such as image classification (EfficientNet) and object detection (NAS-FPN), outperform manually designed architectures.

- EfficientNet, a family of convolutional neural networks, was designed using NAS and has shown superior performance on image classification tasks while being computationally more efficient than the next best model at the time.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/af8e1dd6-6039-4e99-94c9-bd55ed69441d/image.png)

Object Detection Example

#### Natural Language Processing (NLP)

NAS also shines in the field of NLP. NAS has automated the design of some recurrent and transformer-based models, which excel in tasks like translation, sentiment analysis, and text generation.

A heavily used example we see today is BERT (Bidirectional Encoder Representations from Transformers).

- While not entirely an NAS model, BERT incorporates elements of NAS in its pre-training phase, showing the strength of automated design.

### Papers

---

#### NAS with Reinforcement Learning (2017)

One of the earliest papers on NAS is one using reinforcement learning as its search strategy.

Paper: [Zoph & Le, "Neural Architecture Search with Reinforcement Learning"](https://arxiv.org/abs/1611.01578?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas).

In this paper, Zoph and his team used an RL agent (the "controller") to design convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

How did the process work? The controller generated network designs (placing layers, activations, etc.); each design was trained and evaluated, and then the controller updated its strategy based on accuracy (getting a higher score for better designs).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ddd8a168-84a3-4d43-ab5f-99f515f79911/image.png)

In this process, they achieved extremely high accuracy on image classification (CIFAR-10) but used 800 GPUs for 28 days—which is extremely expensive.

#### Efficient Neural Architecture Search (ENAS) (2018)

ENAS is the paper that introduced weight sharing into NAS. Instead of training every design from scratch, ENAS reused parameters across architectures by using “supernets”. They sample “subnets” (different architectures) within these supernets, where “subnets” inherit weights from the supernet.

Paper: [Pham et al., "Efficient Neural Architecture Search via Parameter Sharing"](https://arxiv.org/abs/1802.03268?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c61f5f88-823a-4a3b-a60f-21030ce66729/image.png)

They used graph theory methods (realizing NAS’s search space can be represented by a single directed acyclic graph) to efficiently search through the space by sharing parameters.

In some cases, they reduced computational costs by 1,000x, making NAS more accessible—even to smaller research teams with fewer resources.

From the paper itself, it mentions: “Importantly, in all of our experiments, for which we use a single Nvidia GTX 1080Ti GPU, the search for architectures takes less than 16 hours.”

#### DARTS: Differentiable Architecture Search (2019)

DARTS gave way to gradient-based optimization search methods, allowing the search space to be searched using optimizers such as gradient descent—something that could be solved with calculus.

Paper:[](https://arxiv.org/abs/1806.09055?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas)[Liu et al., "DARTS: Differentiable Architecture Search"](https://arxiv.org/abs/1806.09055?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas).

The team treated the architecture search as a continuous optimization problem.

They would assign weights to different layer types (e.g., convolution vs. pooling) then use gradient descent to adjust these weights.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0c1707b3-6599-4666-88ce-3c104d4c7d42/image.png)

Liu and his team reduced the search time to 1-4 days on a single GPU.

#### EfficientNet (2019)

Paper: [Tan & Le, "EfficientNet: Rethinking Model Scaling for CNNs"](https://arxiv.org/abs/1905.11946?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas).

EfficientNet aimed to solve the problem of the inefficiency of manually scaling networks (making them deeper/wider) arbitrarily.

They decided to use NAS to balance three scaling factors automatically: the depth (number of layers), the width (number of neurons per layer), and the resolution (input image size). EfficientNet uniformly scales these factors using fixed scaling coefficients.

Through this process, it ended up achieving higher accuracy with 10x fewer parameters than previous models.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/08d1c505-5a41-4e45-bebe-6e38093e45b5/image.png)

#### Once-for-All (2020)

Once-for-all (OFA) built on the “supernet” idea introduced by ENAS. The goal of OFA was to train a single neural network that can adapt to different devices (smartphones, servers, etc.).

Paper: [Cai et al., "Once-for-All: Train One Network and Specialize It for Efficient Deployment"](https://arxiv.org/abs/1908.09791?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas).

One of the ways they did this was to introduce progressive shrinking. The “supernet” is trained in a way that allows it to "shrink" into smaller sub-networks while maintaining high accuracy. This allows models of varying sizes to perform well without retraining.

In addition, the sub-networks can be activated based on hardware constraints (e.g., speed vs. battery life), moving towards more efficient AI deployment on devices like drones and wearables.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ad21f9a-5979-4972-bbf5-c3646f6fa3be/image.png)

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-architecture-search-nas)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team