Alignment In Transformers, Simply Explained

Transformer models are the most widely used architecture in AI at the moment. Some of its most popular uses are Natural Language Processing (NLP) tasks and multimodal tasks such as Image Captioning.

Transformers are one of the key architectures that tackle the problem of alignment.

Well, what exactly is alignment? In NLP, if we look at language translation, the difference in order and context of words forms a sentence’s meaning—we must figure out which input words correspond to which target words.

• This process is essentially what alignment is: figuring out which part(s) of the input corresponds to which part(s) of the output.

In traditional models, alignment had to be solved as a separate task, but with the arrival of transformers and their “self-attention” mechanism, alignment became integrated within the model.