# Deep Unsupervised Learning: Autoencoders #16 Sep 9, 2024 Foundation https://www.aibutsimple.com/p/deep-unsupervised-learning-autoencoders Autoencoders are a specific type of feedforward neural network where the model attempts to deconstruct the input and construct a similar output. They compress the input into a lower-dimensional version and then reconstruct the output from this representation. This aims to learn a compressed, yet informative, representation of the input data. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/13dabe22-3477-419c-8ecd-0f5c7bef3f04/Copy_of_Copy_of_Add_a_heading__1_-1.png) They are one of the foundational techniques of deep unsupervised learning (learning without labels). - They are mostly used as a form of dimensionality reduction (making it easier to visualize, process, or use in other machine learning tasks). - Some other common applications are anomaly detection, denoising, and generation. - Other networks used in deep unsupervised learning include the GAN, clustering nets, and more. Technically, autoencoders can also be considered “self-supervised” as they can generate their own labels. Let’s think back to the main ideas of autoencoders. On a very basic level, the autoencoder is just tasked with generating the same output from the code produced by the input. We call this version the “Undercomplete” autoencoder. This way, we can learn the intricacies of the data. The autoencoder will try to capture the most important features or patterns in the data by compressing it into a lower-dimensional form and then reconstructing it back. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/15ece583-c8b0-4fb5-9b34-dba6aeb4c10f/image.png) A loss function is used to train the model; it simply evaluates the difference between the true target and the generated image. If you know anything about dimensionality reduction in machine learning, you might have heard of PCA or ICA—these methods can perform linear transformations to the data. Autoencoders can perform larger scale non-linear dimensionality reduction, making it useful in its little niche. ### Autoencoder Architecture --- Let’s go over the architecture of an undercomplete autoencoder, which is the general substructure of many versions of autoencoders. An autoencoder consists of 3 components: the encoder, the code/bottleneck, and the decoder. The encoder compresses the input and produces the code, the decoder then reconstructs the input only using the output from the code. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/670924e3-d2e7-4d35-9df4-0ce217a36785/Copy_of_Copy_of_Add_a_heading__21_.png) Let’s explore the details of these components more deeply. Both the encoder and decoder are fully-connected feedforward neural networks, essentially just ANNs. The code is a single layer with the dimensionality of our choice. The number of nodes in the code layer (called the code size) is a hyperparameter that we set before training the autoencoder. - It is a module that contains the compressed knowledge representations and is the most important part of the network. - The compressed lower-dimensional version of the input is called the latent-space representation. Note that the decoder architecture is the mirror image of the encoder. This is not a requirement, but it’s typically the case. - The only requirement is that the input and the output must be the same size. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/476545bb-f269-4ce5-83cb-1381ab071dc0/Copy_of_Copy_of_Copy_of_Add_a_heading-1.png) Let’s talk about the hyperparameters used in autoencoders; specifically, there are 4 main ones. The first hyperparameter is the code size. This determines the number of nodes in the middle layer, and a smaller size results in more compression. Then, we have the number of layers we want to have. The autoencoder can be as deep as we like. In the above image, it shows only one layer of encoding and decoding, which is rarely the case. We can also tweak the number of nodes per layer. The number of nodes per layer defines the weights we use per layer. Usually, the number of nodes per layer decreases with each subsequent layer of the encoder and increases back in the decoder. We should also choose a suitable loss function. Two popular options are mean squared error and binary crossentropy. If the input values are in the interval [0, 1], then we typically use crossentropy, otherwise, we use MSE. Autoencoders are trained the same way as ANNs via backpropagation. - If you don’t quite understand what backpropagation is, we have an issue on it [here](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-unsupervised-learning-autoencoders). There are many other popular forms of autoencoders stemming from undercomplete autoencoders. One of the most used ones is the denoising autoencoder. ### Denoising Autoencoders --- Denoising autoencoders, as the name suggests, are autoencoders that remove noise from an image. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6616edeb-ae63-4d09-a4e8-c632e5d88619/Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__1_.png) One of the key differences between this autoencoder and the undercomplete autoencoder is that it does not use the “ground truth” image as its input. Instead, we add some noise (usually Gaussian) to the ground truth image and feed it into the autoencoder. The output is then compared to the actual ground truth image without noise. The denoising autoencoder gets rid of noise by learning a representation of the input where the noise can be filtered out. - While removing noise directly from the image seems difficult, the autoencoder does this by mapping the input data into a lower dimension (like in undercomplete autoencoders), which makes it easier to filter out noise. The loss function used in denoising autoencoders is usually L1 or L2 loss. ### Sparse Autoencoders --- Sparse autoencoders are similar to the undercomplete autoencoder, as the general structure is the same. While undercomplete autoencoders are regulated by tuning the size of the bottleneck, the sparse autoencoder is regulated by changing the number of nodes at each hidden layer. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/95fc7ae6-1e26-41a5-882a-f0246ef8c2fe/Copy_of_Copy_of_Add_a_heading__2_-1.png) Instead of designing a network that has a changing number of nodes at each layer (not yet seen), sparse autoencoders penalize the activations of certain neurons in the layers. In other words, the loss function has a term that calculates the number of neurons that have been activated and provides a penalty that is proportional to that. This penalty, named the sparsity function, prevents the neural network from activating too many neurons and works as a regularizer. This allows the sparse autoencoder to be more flexible than the undercomplete autoencoder, and perform better. --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-unsupervised-learning-autoencoders) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team