Issue #46 LLMs Edwin Dong
BERT: Bidirectional Encoder Representations From Transformers
Free editionPRO edition
The Bidirectional Encoder Representations From Transformers (BERT) model is a transformer model that fundamentally transformed the field of natural language processing (NLP), becoming a foundational model for a wide range of NLP tasks.
BERT was revealed in Google’s groundbreaking 2018 paper, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” which has been cited over 120,000 times.
BERT’s introduction incited the development of countless additional transformer-based models (such as RoBERTa, GPT-2, or T5), leading to a wave of innovations in NLP that continues to influence research and applications today.
BERT introduced a novel approach with its bidirectional transformers. Before BERT, most models processed text in one direction: either left-to-right or right-to-left.
Why Was BERT Needed?
One of the biggest challenges in NLP is the lack of enough task-specific data and labeled data.
There is an abundance of unlabeled and unorganized data on the web, but labeling and organizing this data is expensive and time-consuming.
With a medium-sized budget, we end up with only a few thousand or a few hundred thousand human-labeled training examples.
Unfortunately, in order to perform well, NLP models require much larger amounts of data—they see major improvements when trained on millions or billions of annotated training examples.
To help bridge this gap in data, researchers have developed various techniques for training general-purpose language models using unannotated text on the internet—this is known as pre-training.
These general-purpose pre-trained models can then be fine-tuned on smaller task-specific datasets, such as translation.
When BERT was released in 2018, it was a promising pre-trained general-purpose NLP model—achieving state-of-the-art results in a wide variety of NLP tasks.
As a transformer, BERT uses the self-attention mechanism to weigh the significance of each word based on its context—both the context to the left and to the right of the word.
This bidirectional processing is what makes BERT so good at understanding context, better than standard unidirectional transformers.
Want to learn more about transformers and how they work? Check out our past issue here.
This is an oversimplification, but think of it this way—BERT reads and then re-reads the sentence to gain a deeper understanding of each word’s meaning. It generates a left and right context instead of just a left context of standard transformers.
Let’s take this sentence: “The bank closed early today.” The word “bank” has an ambiguity; it can either mean the side of a river or a financial institution.
This edition is for PRO members.
