# Neural Networks in Pytorch

#11 Jul 29, 2024 Foundation

https://www.aibutsimple.com/p/neural-networks-pytorch

In past issues, we’ve gone over the theory of neural networks, we’ve talked about the concepts of them, but we haven’t discussed how to implement them in real life.

So today, we’re going to make a ANN without any convolutional layers (used in CNNs) to classify some images.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cbc7d814-d826-443f-9df0-a514ee9d8632/image.png)

Convolutional layers are shown above

- Now, if you know anything about CNNs, you’ll know that they work great for image processing related tasks.
- To demonstrate the power of simple neural networks, we’ll take on the challenge of making an ANN image classifier with acceptable accuracy.

We’re going to implement the ANN in Python using Pytorch, along with some helper libraries like Numpy and Matplotlib.

Disclaimer: This issue will require some knowledge of Python. If you have no knowledge but still want to follow along, please consult a beginner Python tutorial before continuing.

But why choose PyTorch for ANNs?

Pytorch’s intuitive syntax and flexibility make it a favorite among researchers and developers. It’s backed by a large community and lots of support, which is one of the reasons it’s so popular.

We’re going to be training our model on the infamous Fashion-MNIST dataset, as it’s much more robust than the MNIST dataset.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d80464c7-f121-4229-bedc-1895392f36b8/image.png)

- The Fashion-MNIST Dataset consists of 60,000 training images and 10,000 testing images
- Each image in the dataset is a 28x28 pixel image
- Every image in the dataset belongs to one of ten classes of items:

| Label | Description |
| --- | --- |
| 0 | T-shirt/top |
| 1 | Pants |
| 2 | Pullover |
| 3 | Dress |
| 4 | Coat |
| 5 | Sandal |
| 6 | Shirt |
| 7 | Sneaker |
| 8 | Bag |
| 9 | Ankle boot |

The idea is to construct a neural network using a decent architecture and enough layers to learn patterns, hoping it will be able to classify images effectively.

By pairing fully connected layers with the ReLU activation function and gradually decreasing the number of neurons, we hope to achieve good performance on this image classification dataset.

- Keep in mind that the model we will be creating won’t be state-of-the-art or too complicated, so its performance won’t be the absolute best.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/02d56f8f-eb0e-4f61-bba8-ef513278c653/Copy_of_Copy_of_Add_a_heading__1_.png)

We want to go for an architecture like shown above

Let’s start with the imports and download the dataset.

```
import numpy as np # python numerical library
import matplotlib.pyplot as plt # graphing library
from collections import OrderedDict 

import torch # pytorch
import torch.nn as nn # neural network
import torch.optim as optim # optimizers
from torchvision import datasets, transforms # data

# Download training and testing data
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])

train = datasets.FashionMNIST('F_MNIST_data', download=True, train=True, transform=transform)

test = datasets.FashionMNIST('F_MNIST_data', download=True, train=False, transform=transform)
```

Then, let’s split our training dataset into 80% training and 20% validation.

- As a reminder, validation datasets are to evaluate performance and to do any hyperparameter tuning.

```
# split train set into training (80%) and validation set (20%)
train_num = len(train)
indices = list(range(train_num))
np.random.shuffle(indices)
split = int(np.floor(0.2 * train_num))
val_idx, train_idx = indices[:split], indices[split:]
len(val_idx), len(train_idx)
```

When we use Pytorch, we usually use data loaders, which are just tools to manage our data.

- They can make batches for training, shuffle the data, transform the data, and load data in parallel to speed up the training process.

```
# prepare dataloaders
train_sampler = torch.utils.data.sampler.SubsetRandomSampler(train_idx)
train_dl = torch.utils.data.DataLoader(train_ds, batch_size=64, sampler=train_sampler)

val_sampler = torch.utils.data.sampler.SubsetRandomSampler(val_idx)

val_dl = torch.utils.data.DataLoader(train_ds, batch_size=64, sampler=val_sampler)

test_dl = torch.utils.data.DataLoader(test_ds, batch_size=64, shuffle=True)
```

Let’s display a sample image and its label:

```
# Return a batch of images and labels
image, label = next(iter(train_dl))

# Print the size of one image
print(image[0].shape)

desc = ['T-shirt/top','Trouser','Pullover',
'Dress','Coat','Sandal','Shirt',
'Sneaker','Bag','Ankle Boot']

# Print the label
print(desc[label[0].item()])

# Display the image
plt.imshow(image[0].numpy().squeeze()); 
```

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b7323165-3eff-4e5f-ad03-55af594904d3/image.png)

Now here comes the important part: model building. We’re going to use the sequential neural network option since the model is quite simple, instead of using nn.module in Pytorch.

```
def network():
    model = nn.Sequential(OrderedDict([
        ('fc1',nn.Linear(784, 128)),
        ('relu1', nn.ReLU()),
        ('drop1', nn.Dropout(0.25)),                                       
        ('fc2', nn.Linear(128, 64)),
        ('relu2', nn.ReLU()),
        ('drop2', nn.Dropout(0.25)),                                       
        ('output', nn.Linear(64, 10)), # 10 classes
        ('logsoftmax', nn.LogSoftmax(dim=1))
    ]))

    # Use GPU if available
    device = 'cuda' if torch.cuda.is_available() else \
             'cpu'
    model = model.to(device)  

    # define the criterion and optimizer
    loss_fn = nn.NLLLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.003)

    return model, loss_fn, optimizer, device    

model, loss_fn, optimizer, device = network()
print(model)
```

We build this model using ReLU as our activation function, fully connected layers with decreasing amounts of neurons, and we use Softmax to transform the output into a probability distribution.

- In this case, we’re using Negative Log Likelihood Loss (NLLLoss) as our loss function, which is used along with LogSoftmax (which is literally the log of softmax, log(softmax(input))) for computational efficiency.
- In fact, we’re simply doing what Pytorch’s Cross Entropy Loss does, but manually.
- To read more about [LogSoftmax](https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-in-pytorch#torch.nn.LogSoftmax) and [NLLLoss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-in-pytorch#torch.nn.NLLLoss), consult the documentation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b4ba3f85-4454-4a9f-9150-213bdfad3a01/image.png)

We also use dropout to improve the performance of our model and reduce overfitting.

- In Pytorch, dropout randomly zeroes some of the elements of the input tensor during training with probability p (for instance, 0.25)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3eddefd4-8525-492c-8f4b-b9956461433c/Copy_of_Add_a_heading__2_-1.png)

Dropout is a regularization technique that “deactivates” certain nodes to make the model more robust

Since the code has become too large to type out in code blocks, we will display them using images.

Let’s write the main method used to train the neural network.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6b6b67b0-b4ae-4cd6-bcde-2dd6389e18e5/image.png)

This function takes in the model, the training dataset dataloader, the loss function, the optimizer, and the device (normally GPU) as parameters.

Above, on line 2, we set the model to training mode. Certain layers, like dropout needs this specification to take action.

- This is because when evaluating the model we want certain layers, like dropout to behave differently

For every image and label in the training dataset, we pass them into the model to produce an output (line 9), then backpropagate through the network and update the model’s weights.

We then add the current training example’s loss to the sum (on line 16) to divide over the number of samples to get the average loss (line 17).

We can use this function alongside a validation function to train the network.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8c76d7f3-1615-4e58-a1a2-b70d34930104/image.png)

This validation method takes in the model, the validation dataset dataloader, the loss function, and the device as parameters.

On line 2, we set the model to evaluation mode, which tells layers like dropout to deactivate.

We will disable the gradient calculation (line 5) for validating the model since we don’t need to use backpropagation; we’re not training the model. This will speed up the validation process a lot.

- We use the Python keyword with, which will re-enable the gradient calculation after the code block finishes execution

We load the images and labels, then feed them through the model for output (line 7).

We then calculate the losses and sum them to divide by the length of the validation dataset later (line 11-12).

We also calculate the validation accuracy by taking the highest probability prediction and seeing if our answer is right or wrong (then we divide it by the length of the validation dataset).

- Note that we call [torch.exp](https://pytorch.org/docs/stable/generated/torch.exp.html?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-in-pytorch)(test_outputs), which just returns the probability distribution in standard softmax, since we used log softmax instead (eln(softmax(test_outputs)) = softmax(test_outputs))

Let’s combine the training process and the validation process for the full neural network training process:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fb46401f-1315-4554-aab6-516d47ffcbcb/image.png)

In this function, if you don’t specify the number of epochs, we will default to 25 epochs.

For every epoch, we train the model on the train dataset, validate the model on the validation dataset, and then print out the losses of each epoch.

We will also plot the losses over each epoch to determine if our model is overfitting.

For those who want the code directly (copy and paste), you can access it [here](https://pastebin.com/8istxvmX?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-in-pytorch).

Finally, let’s call the train_validate function to train our model.

```
train_validate(model, loss_fn, optimizer, train_dl, val_dl, device)
```

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bc3554ca-3aee-4fd5-a80e-88186ddd9ef2/image.png)

It was trained for 25 epochs, this is only a portion of the training process

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/16f29246-c768-4954-a36e-f3fd79904de4/image.png)

We can see that there is not much overfitting in the model since its validation loss is decreasing gradually.

After plotting our loss curves, let’s try to predict a singular example and see if it classifies it correctly!

```
# Depends on platform, colab doesn't need it
%matplotlib inline 
# Higher quality images
%config InlineBackend.figure_format = 'retina'

# Retrieve the images and labels in the next batch
images, labels = next(iter(test_dl))
images, labels = images.to(device), labels.to(device)

# Choose some random image and its label
img, label = images[5], labels[5] 

# Convert 3D tensor to 2D matrix (image)
img = img.view(img.shape[0], -1)

# Calculate the class probabilities (softmax) for img
proba = torch.exp(model(img))

# Plot the image and probabilities
desc = ['T-shirt/top','Trouser','Pullover','Dress',
'Coat','Sandal','Shirt','Sneaker','Bag','Ankle Boot']

# Create a large plot with 2 subplots side by side
fig, (ax1, ax2) =  plt.subplots(figsize=(13, 6), nrows=1, ncols=2) 

ax1.axis('off') # no axes for the image

# Put image tensor on cpu, convert it to numpy array and remove extra useless dimension
ax1.imshow(images[5].cpu().numpy().squeeze())

ax1.set_title(desc[label.item()])
ax2.bar(range(10), proba.detach().cpu().numpy().squeeze())

ax2.set_xticks(range(10))
ax2.set_xticklabels(desc, size='small')
ax2.set_title('Predicted Probabilities')
plt.tight_layout()
```

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ba636b43-7014-4dae-be5d-1955339d37da/image.png)

The output should be an image and a bar chart of the probabilities (the image may be different based on the shuffling of the data).

As you can see, by using the Softmax activation function, we get a probability distribution rather than a prediction, but we usually pick the one with the maximum likelihood.

- So above, we would pick “pullover“ as our prediction

Finally, we can evaluate our model on the test dataset.

```
with torch.no_grad():
    batch_acc = []
    model.eval()
    for images, labels in test_dl:
        # put tensors on gpu
        images = images.to(device)
        labels = labels.to(device)
        # flatten images to batch_size x 784
        images = images.view(images.shape[0], -1)
        # make predictions and get probabilities
        proba = torch.exp(model(images))
        # take the class with highest probability
        _, pred_labels = proba.topk(1, dim=1)
        # compare actual labels and predicted labels
        result = pred_labels == \
                   labels.view(pred_labels.shape)
        # calculate accuracy
        acc = torch.mean(result.type(torch.FloatTensor))
        batch_acc.append(acc.item())
    else:
        mean = torch.mean(torch.tensor(batch_acc))*100
        print(f'Test Accuracy:{mean:.2f}%')
```

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7734b14a-9469-4019-b432-f552d63c5e58/image.png)

And just like that, we made a decently accurate (87.23% accuracy) image classifier without any convolutional layers!

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-networks-in-pytorch)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team
