Issue #32 Architecture Edwin Dong
Vision Transformers (ViTs)
Free editionPRO edition
The Transformer neural network architecture was initially developed to tackle machine translation problems but quickly became the top choice in various natural language processing (NLP) tasks due to its impressive performance.
As the Transformer revolutionized NLP, researchers began to question whether its success could reach into other domains, such as computer vision.
In the field of computer vision, the Convolutional Neural Network (CNN) has long been the dominant architecture. However, researchers started to wonder if it might be possible to do better by adapting Transformer models to image data.
Before the introduction of Vision Transformers (ViTs) in 2021, two major papers attempted to adapt transformers for image data.
In 2019, Ramachandran et al. proposed a plan for replacing convolution with self-attention for use in CNNs. However, its use of specialized patterns in attention makes it hard to scale up models on hardware accelerators.
Then, in 2020, Cordonnier et al. theoretically proved that self-attention can learn to behave similarly to convolution. In their paper, patches were taken from images as inputs, but the small patch size makes the model only applicable to image data with low resolutions.
This research paved the way to Vision Transformers (ViTs) (Dosovitskiy et al., 2021), a transformer model that was “re-engineered” for computer vision tasks.
Essentially, a Vision Transformer (ViT) is just an adaptation of the Transformer architecture with some modifications for processing the input and generating the output.
They extract patches from images and feed them into a Transformer encoder block to obtain a global representation (information about the entire image), which will finally be transformed for classification.
In a ViT, images are divided into fixed-size patches, and each patch is treated as a token, similar to words in NLP.
Unlike CNNs, which rely on convolutional layers to extract features, ViTs use self-attention mechanisms to process images.
Before continuing, it is recommended to have general knowledge of transformers and how they work. We have an issue detailing this below:
This edition is for PRO members.
