# Neural Networks: Learning Parameters (Weights, Biases, and Backprop)

#2 May 20, 2024 Training

https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop

Last week we discussed how a neural network works in very simple terms.

- (If you haven’t read it, you can find it [here](https://www.aibutsimple.com/p/neural-networks-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-learning-parameters-weights-biases-and-backprop))

Today, we’re going to add on to that concept.

So how do neural networks actually learn?

When data is fed through a neural network, its weights and biases affect how the data influences the final output of the model.

- The output of a neural network can vary for different cases. For instance, for classification problems, it may output a vector of probability values of each class.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/58364c58-6eb7-4416-b336-a6246db432d8/Add_a_heading__5_.png)

But how are these weights and biases actually tweaked?

We use something called an error gradient.

When we get the output from the neural network, we need a measure of how good it performed on its current iteration.

- This tells you when to stop training, and also the accuracy of the model (not accuracy, but accuracy as a general term).

This “measure” of performance is called a loss function.

- A common loss function for regression (estimating numerical values) is mean squared error, where the distances between the result and the prediction are squared and summed.
- A common loss function for classification is cross entropy loss (depending on the activation function)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4ddf5188-cd04-4102-875d-3ebac8ac730b/Add_a_heading__6_.png)

After getting the loss, we can use this data to perform backpropagation to update the weights and biases.

- Updating the weights and biases occurs when we pass data from the output back through the network.
- Backpropagation is the name we give this process.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5e2d3e2d-2309-4d18-9625-4926d889c13f/image.png)

What we want to know when fixing the current weight and bias (which is initially generated randomly/set to some value) is whether the current weight and bias values are too large or too small:

- Do we need to decrease or increase our current value to get a smaller error?
- How much do we need to decrease/increase our current value?

A loss function must be differentiable, as we need to take the partial derivative (the gradient) of a variable with respect to another to perform gradient descent.

- Specifically, the gradient we are looking for is derivatives of loss with respect to weights and biases.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/25846149-9322-48dd-aa94-7adffa429c92/Add_a_heading__7_.png)

Gradient descent is the process of converging towards a minimum of a loss function by using the gradient.

- This is intuitive, since if we plot the loss function on a graph, the gradient represents the line tangent to the current point of a model’s weight.
- As long as the gradient’s magnitude becomes smaller and smaller, the model is converging towards a minimum/maximum.
- Since we want to walk towards the minimum of the loss function, we subtract the gradient:
  - Looking at the example below, if the gradient is positive (positive slope), then the ball’s w coordinate will move towards the left, which is close to the minimum.
  - If the gradient were to be negative (negative slope), it would be added to the ball’s w coordinate and would move towards the right, getting closer to the minimum.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/48d41b86-1d19-40bd-b167-9e9e4493315e/grad_descent.png)

Credit goes to [https://stats385.github.io](https://stats385.github.io/assets/img?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-learning-parameters-weights-biases-and-backprop)

#### So a simple example of the learning process goes like this:

A neuron will take an input, multiply it by it’s weights, add a bias, and pass this through an activation function to produce an output.

- We repeat this process for all neurons for all layers to produce an output—which we can then compute the loss through a loss function.

After computing the loss, we can go ahead and start the backwards pass (backpropagation)

- Backpropagation is just using the chain rule to find the gradient
  - (since during the forwards pass, we use activation functions and more processing, so we need to find multiple partial derivatives and arrange them in a way to use chain rule)
  - If you don’t know what this is, feel free to read [this](https://math.libretexts.org/Bookshelves/Calculus/Calculus_(OpenStax)/14%3A_Differentiation_of_Functions_of_Several_Variables/14.05%3A_The_Chain_Rule_for_Multivariable_Functions?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-learning-parameters-weights-biases-and-backprop) guide.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3bbf5695-a214-40ce-ab7f-229b7f9cdc51/Add_a_heading__8_.png)

- So we get the gradient of the loss with respect to the weights and biases, and we can use this data to adjust them in the next step.

After backpropagation, we now have the gradient of the loss function, so we use this to update the weights and biases.

- For instance, a method of finding a new weight is to make it equal to the old weight minus the gradient multiplied by the learning rate.
  ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2e3d8830-bfbb-4cfb-b2b9-da73b2c1fb51/Screenshot_2024-05-13_164702.png)
  - The learning rate controls how far down we move down the loss function with respect to the weights (here’s a cool visualization):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d20a568a-3300-46af-9504-d040876bff89/image.png)

- After the adjustments, we pass data back into the input and repeat the process until we achieve the desired loss/amount of time.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fbc7e1ff-2857-407a-b58e-c832cb8816e1/image.png)

Epochs here represent a unit for the amount of time (one cycle)

And that’s just about it! Now you know the basics of how a neural network works! 🙂

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-learning-parameters-weights-biases-and-backprop)

Advertising, thoughts, suggestions, feedback? Please send to [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

---

That’s it for the second issue of AI, but simple. See you next week!

—AI, but simple team