# Deep Learning Common Terms and Misconceptions

#21 Oct 14, 2024 Foundation

https://www.aibutsimple.com/p/supporter-only-deep-learning-common-terms-and-misconceptions

This week, we’ll be going over some commonly used terms in deep learning and machine learning, which will help you build a stronger understanding of deep learning processes and methods.

We’ll take the time to review each term individually, provide clear definitions, and explore use cases to see how they are applied.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f4a95779-a41c-4ce8-ba66-8ef9deb2788a/Copy_of_Copy_of_Copy_of_Copy_of_Input_to_Hidden__1_.png)

### Batch and Mini-Batch

---

Batches and mini-batches can be confusing as they are used interchangeably when referring to a subset of an entire dataset.

A batch is a subset of the training dataset used to train the neural network in one iteration.

A mini-batch is also a small subset of the training dataset (usually 32, 64, or 128 samples).

A dataset is split up into these “batches” (or mini-batches) after defining a set “batch size”, the number of samples in the batch.

- A smaller batch size can introduce some noise, which can help with convergence, while a larger batch size takes longer for the model to make an update.

The difference is made during the naming of gradient descent techniques.

- Gradient descent is used as an optimizer, an optimization technique that allows neural networks to “train”.
- Optimizers are algorithms used to change the parameter values of the neural network, such as weights and learning rate, to reduce losses.

When the batch size equals the total number of samples in the training dataset, it is called batch gradient descent.

On the other hand, mini-batch gradient descent updates the model through backpropagation every time a batch is processed, and since the batch size for mini-batch is less than the whole dataset, there are more updates than batch gradient descent.

- Mini-batch gradient descent also introduces some noise, taking from the benefits of stochastic gradient descent to escape plateaus and converge better.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/568d781d-3e4b-4095-b886-ef89b4db3a36/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__2_.png)

If you want to learn more about the different types of optimizers, feel free to check out this [issue](https://www.aibutsimple.com/p/optimizers-deep-learning-simple-explanation?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=supporter-only-deep-learning-common-terms-and-misconceptions).

### Hyperparameters and Parameters

---

Hyperparameters are the settings or configurations that the model doesn’t update after learning from the data.

Hyperparameters are set before training begins and are tuned using hyperparameter tuning methods like grid search or random search.

Some hyperparameters include learning rate, batch size, number of epochs, network architecture, and the choice of activation functions.

- Certain optimization algorithms like Adam and RMSprop have adaptive learning rates; they adjust the learning rate based on the parameters.