# Vision Transformers (ViTs) #32 Dec 30, 2024 Architecture https://www.aibutsimple.com/p/vision-transformers-vits The Transformer neural network architecture was initially developed to tackle machine translation problems but quickly became the top choice in various natural language processing (NLP) tasks due to its impressive performance. As the Transformer revolutionized NLP, researchers began to question whether its success could reach into other domains, such as computer vision. In the field of computer vision, the Convolutional Neural Network (CNN) has long been the dominant architecture. However, researchers started to wonder if it might be possible to do better by adapting Transformer models to image data. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f4a95779-a41c-4ce8-ba66-8ef9deb2788a/Copy_of_Copy_of_Copy_of_Copy_of_Input_to_Hidden__1_.png) This research paved the way to Vision Transformers (ViTs) (Dosovitskiy et al., 2021), a transformer model that was “re-engineered” for computer vision tasks. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c57f535d-c3fb-4084-806d-321b07203950/image.png) Essentially, a Vision Transformer (ViT) is just an adaptation of the Transformer architecture with some modifications for processing the input and generating the output. They extract patches from images and feed them into a Transformer encoder block to obtain a global representation (information about the entire image), which will finally be transformed for classification. - In a ViT, images are divided into fixed-size patches, and each patch is treated as a token, similar to words in NLP. Unlike CNNs, which rely on convolutional layers to extract features, ViTs use self-attention mechanisms to process images. Before continuing, it is recommended to have general knowledge of transformers and how they work. We have an issue detailing this below: - [Introduction to Transformers and the Attention Mechanism (Intermediate)](https://www.aibutsimple.com/p/transformers-and-the-attention-mechanism?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-transformers-vits) #### A Quick Review of Self-Attention --- ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/94326af5-3838-4d6f-b50a-d48dcf34c44f/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Input_to_Hidden.png) The attention mechanism is a fundamental part of transformer models that allows them to weigh the importance of different tokens (usually words or symbols) in an input sequence when making predictions. It assigns varying degrees of importance to different tokens to help the model focus on the most relevant portions of the data, like how certain words have more meaning in a sentence. The mechanism itself computes importance values between each pair of tokens; every token looks at every other token to decide which ones are most important to understanding the context. In ViTs, instead of using words for its input tokens, it uses image patches as the tokens instead, allowing the attention mechanism to capture relationships to better understand the image. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b06f88fc-a956-4c82-9699-5a012a65faf0/image.png) ### Architecture --- The structure of a ViT is made up of three key components: a stem, a body, and a head. The stem splits the input image into small, fixed-size patches (e.g., 16×16). These patches are flattened and treated as tokens, much like words in a sentence. A special “” token is added to represent the entire image. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1daded88-109e-499c-b6df-3fa739f132c2/56f7584f-746d-44cb-81f7-00224bfd42f5.png) Each patch is linearly embedded into a vector in a high-dimensional space, similar to how transformers embed words into high-dimensional space. Positional encodings are added to maintain the spatial structure of the image, since transformers do not capture each patch’s positional information. The body is based on the multilayer transformer encoder, passing the embedded image patches along with the embedded “” token through self-attention blocks with normalization in between layers, finally passing through an MLP to generate representations. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/de1f9b41-db86-44c7-be6a-b42d6ff2d94c/image.png) As you can see above, self-attention in the body works similarly to convolution when highlighting important features. In the head of the ViT, after passing the data through the Transformer layers, the final representation of the “” token is used to make predictions about the image class. This is done by feeding the representation through a normalization layer and a multi-layer perceptron (MLP) to produce the output label. In summary, a ViT consists of a stem that patchifies images, a body based on the multilayer transformer encoder, and a head that transforms the output label. ### Transformers or CNNs for Vision? --- For small datasets like Fashion-MNIST, ViTs do not outperform the ResNet, a common CNN architecture. Similar observations can be made even on the ImageNet dataset (1.2 million images). - This is due to the fact that Transformers lack some of the useful principles in convolution, such as translation invariance and locality. #### Translation Invariance Imagine looking at a cat in a picture. Whether the cat is in the center, the top left corner, or the bottom right, one could still recognize it as a cat, right? ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4c51084c-3c92-4fd4-a5ba-01e1be9ee5e8/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Input_to_Hidden__720_x_600_px___900_x_750_px___2_.png) This is an example of translation invariance—it is the ability to identify an object no matter where it appears in the image. Convolutional layers in CNNs are great at this because they apply the same filter across the whole image, making them naturally good at detecting patterns anywhere, regardless of their position. But Transformers, do not have this built-in ability. They process the image as a sequence of patches, treating each patch independently, and do not understand that the same features (like a cat's tail or nose) can appear in different positions. They need positional encoding to know where each patch belongs in the image. #### Locality In CNNs, each filter looks at a small region of the image at a time (e.g., a 3×3 grid). So, CNNs naturally focus on local patterns such as edges, textures, or small shapes before looking at the bigger picture. - This helps them build a detailed understanding of what’s happening in specific parts of the image. In comparison, Transformers look at the entire image at once and process each patch by attending to all the others in a self-attention mechanism. This gives them a global view, but they don’t automatically “zoom in” on local features. They have to “learn” local patterns over time through their attention mechanism, without the local focus that CNNs provide. Although in smaller datasets, the CNN will outperform the ViT, there still exists a need for ViTs as the scale of data and models is steadily increasing. When training larger models on larger datasets (e.g., 300 million images), ViTs outperform ResNets by a large margin in image classification, demonstrating the intrinsic superiority of Transformers in scalability. Transformers have now also become a game-changer in computer vision, much like they have been in revolutionizing NLP tasks. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-transformers-vits): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-transformers-vits) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team