# Hyperparameter Tuning for Deep Learning

#6 Jun 17, 2024 Training

https://www.aibutsimple.com/p/hyperparameter-tuning-deep-learning

Hyperparameter Tuning is the act of searching for the best values of hyperparameters to minimize some loss function.

If you’ve ever explored any Kaggle competitions, or just want a method to optimize your network further, this is one of the most popular methods.

- Hyperparameters are user-defined values, while parameters are altered during training
- For instance, a hyperparameter may be the learning rate, while a parameter is the weight or bias.

If you don’t know what most of these words mean, please check out a previous [issue](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=hyperparameter-tuning-for-deep-learning), where we delved into the process of learning.

There are many different approaches to searching for the best configuration of hyperparameters, and some of the most known ones are:

- Grid Search
- Random Search
- Bayesian optimization

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d49f6dc8-9050-430d-afd5-7968c96fa1be/image.png)

Popular hyperparameter tuning methods

There are also 2 main categories of these algorithms: Black-Box and Multi-Fidelity.

Black-box hyperparameter tuning treats the model as a black box, meaning it doesn’t factor any internal details of the model into account. It only considers the inputs (hyperparameters) and the outputs (performance metrics).

- It’s easy to implement since it doesn’t require knowledge of the model's internal structure.
- But it’s costly: there are high computational costs, especially with complex models.

Multi-fidelity hyperparameter tuning uses different levels of fidelity (or accuracy) to optimize hyperparameters.

Multi-fidelity hyperparameter optimization allocates more resources to promising configurations and stops evaluations of poorly performing ones early.

- It reduces the overall cost of hyperparameter tuning by using cheaper, approximate evaluations to guide the search.

In this issue, we will only go into black-box hyperparameter tuning methods, as they are more simple and would be best to learn first.

### Grid Search

---

In grid search, we try every possible configuration of hyperparameters, and take the configuration with the lowest loss (hyperparameter as input, loss as output, known as an objective function).

- So when we define values of certain parameters in a tabular format (where each row is a hyperparameter and there are multiple columns for values of each hyperparameter), it will look at every single possible configuration of them:

```
parameters ={
    'neurons': (10, 100),
    'activation':(0, 9),
    'optimizer':(0,7),
    'learning_rate':(0.01, 1),
    'batch_size':(200, 1000),
    'epochs':(20, 100)
}
```

This is how it searches for the optimal “set” of hyperparameters.

- Common hyperparameters include “batch_size”, “learning_rate”, etc.
- If you don’t know what these are, read this resource on optimizers

This is time consuming and resource-intensive, and the completion time explodes when there are many hyperparameter options.

- For instance, given some arbitrary hyperparameter value choices in this list: [3 choices, 5 choices, 7 choices, 3 choices]
  - This selection of values will have 3×5×7×3= 315 combinations

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/638f56b9-dcce-4c75-9e1e-43f2b012f1fa/image.png)

In many popular libraries for deep learning, they will attach hyperparameter tuning to cross-validation (we will discuss this in another issue) for evaluation.

So, on top of those prior combinations, it will perform K-Fold Cross Validation, which will multiply the time of complete runs by K.

- Using 5 fold CV, we will have 315×5 = 1575 times

Grid Search will train the model and evaluate it 1575 times in total for the example above.

- 1575 full runs is not a small number, this could take days on a decently powerful GPU.

Though Grid Search guarantees to find the best configuration at the end, it’s not preferable to other techniques unless the number of parameters to test is very small.

Let’s try to find a better method:

### Random Search

---

Random Search does a randomized search on the configuration space to pick some points (and in the end, take the best performing configuration).

The basic idea behind random search is simply that we can explore the hyper-parameter space more widely with Random Search (especially for the more important variables).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/21b52db2-3dd4-40b6-9d4b-f409f4f1b3f0/image.png)

Random search (right) will search a wider area of values

This will help us find the best configuration in fewer iterations. For example, see the image below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4bd8f3dc-3957-484c-b749-eb14ce41ce00/image.png)

Random search can (with luck) hit an area with very low loss in some of its first iterations, but it can also take longer.

- This is the added edge over grid search, as we can have early stopping to not search the entire parameter space.
- It could take as long as grid search or sometimes even longer, depending on the case, but for most cases, Random Search outperforms grid search.

However, even though Random Search performs better than Grid Search when the number of parameters increases, both these approaches are still computationally expensive and time consuming.

- For a detailed analysis of Random Search, please refer to this [paper](https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=hyperparameter-tuning-for-deep-learning).

So we need an even better method:

### Bayesian Optimization

Here’s some basic intuition about Bayesian Optimization:

Imagine you had to set the hyperparameters manually, like trying to brute force a combination lock.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/eb33071d-f4a1-4826-b0e0-1d0526dc5ed8/Copy_of_Copy_of_Add_a_heading.png)

Firstly, you would try a set of parameters, then look at the result, change one of the parameters, rerun, and compare the results, so that way you know whether you are going in the right direction.

Bayesian Optimization works in a similar fashion—the performance of your past hyperparameter configuration affects the future choices of hyperparameter configurations.

In comparison, Random Search and Grid Search do not take into account past performance when determining new hyperparameters to evaluate.

Bayesian optimization builds on an objective function, which is simply the loss function in deep learning. It takes a certain configuration of hyperparameters as input and outputs the loss.

- The objective function is unknown, as we don’t know which hyperparameters give the lowest loss
- It would be very costly to model the entire objective function.

It will then build a model on top of this function, and a common choice for this model is a Gaussian process, which is a probability model of the function's output for any given input.

- This is essentially a model trained on (hyperparameter, objective function score) pairs.

After modelling the function, we use an acquisition function to decide where to test next. This function helps us balance between trying new areas (exploration) and checking promising areas (exploitation).

- Popular acquisition functions include Expected Improvement (EI), Probability of Improvement (PI), and Upper Confidence Bound (UCB).

The optimization process is iterative. Initially, a few points are sampled randomly to get some data. These points are evaluated using the objective function.

Based on these evaluations, the Gaussian process model is updated to improve its predictions. The acquisition function then suggests the next point to evaluate.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c38c6f36-b5fa-4f1c-94dd-ab5226907665/image.png)

If you wish to learn more about this algorithm and probability, consult this resource: [https://www.investopedia.com/terms/p/posterior-probability.asp](https://www.investopedia.com/terms/p/posterior-probability.asp?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=hyperparameter-tuning-for-deep-learning)

This process of sampling, evaluating, updating the model, and selecting new points continues until a stopping criterion is met, such as a fixed number of iterations or a convergence threshold.

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=hyperparameter-tuning-for-deep-learning)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team