# BERT: Bidirectional Encoder Representations From Transformers #46 Apr 14, 2025 LLMs https://www.aibutsimple.com/p/bert-bidirectional-encoder-representations-from-transformers In Partnership With: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/287b2be7-dc3e-4d6a-9a1a-3b69bfa86833/image.png) The Bidirectional Encoder Representations From Transformers (BERT) model is a transformer model that fundamentally transformed the field of natural language processing (NLP), becoming a foundational model for a wide range of NLP tasks. BERT was revealed in Google’s groundbreaking 2018 paper, “[BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=bert-bidirectional-encoder-representations-from-transformers),” which has been cited over 120,000 times. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f32aa340-1125-4620-b4e7-fde680532471/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png) BERT’s introduction incited the development of countless additional transformer-based models (such as RoBERTa, GPT-2, or T5), leading to a wave of innovations in NLP that continues to influence research and applications today. BERT introduced a novel approach with its bidirectional transformers. Before BERT, most models processed text in one direction: either left-to-right or right-to-left. --- [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1f57e64b-9e37-4ce4-8abd-9b419243bcca/Beehiiv_Writer_Apr_10-18.png)](https://srv.buysellads.com/ads/long/x/T65RGODFTTTTTTTVSGYC5TTTTTTVIGXLKATTTTTT6TDXE7YTTTTTTLJYPRDCW2PEPJCWEJ7LKH26K2DIQQFH4L4UVVYE?_bhiiv=opp_e5203713-1df5-448b-96c8-490e32bf25cb_9d11e123) ### You’ve heard the hype. It’s time for results. For all the buzz around agentic AI, most companies still aren't seeing results. But that's about to change. See real agentic workflows in action, hear success stories from our beta testers, and learn how to align your IT and business teams. [Watch the product showcase now](https://srv.buysellads.com/ads/long/x/T65RGODFTTTTTTTVSGYC5TTTTTTVIGXLKATTTTTT6TDXE7YTTTTTTLJYPRDCW2PEPJCWEJ7LKH26K2DIQQFH4L4UVVYE?_bhiiv=opp_e5203713-1df5-448b-96c8-490e32bf25cb_9d11e123) --- As a transformer, BERT uses the self-attention mechanism to weigh the significance of each word based on its context—both the context to the left and to the right of the word. - This bidirectional processing is what makes BERT so good at understanding context, better than standard unidirectional transformers. - Want to learn more about transformers and how they work? Check out our past issue [here](https://www.aibutsimple.com/p/transformers-and-the-attention-mechanism?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=bert-bidirectional-encoder-representations-from-transformers). This is an oversimplification, but think of it this way—BERT reads and then re-reads the sentence to gain a deeper understanding of each word’s meaning. It generates a left and right context instead of just a left context of standard transformers. Let’s take this sentence: “The bank closed early today.” The word “bank” has an ambiguity; it can either mean the side of a river or a financial institution. If BERT only had a left-to-right sequential context, only knowing the relationship between “the” and “bank” is not enough to know which word it is referring to. However, since BERT has both left and right context, the word “closed” directly after “bank” allows the model to distinguish the word. It suggests that “bank” is referring to the financial institution. - It’s still possible that a unidirectional model might still make an educated guess based on prior training data, but it wouldn't be as accurate over time without the complete context. This deeper understanding, along with large-scale pre-training, is exactly why BERT achieved state-of-the-art results on various NLP tasks on its release. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b7e20ef9-3967-41c8-b4c6-d1acdbfc8e8d/sdasdasdasdasdasds.png) On top of its increased understanding of context, BERT popularized a technique called Masked Language Modelling (MLM) that further helps models understand left and right context in a sentence. This might sound familiar; after all, models such as GPT use masking to prevent cheating—predicting the next word by already seeing the next word. - This type of masking is not MLM. The masking that does not allow a transformer to see future tokens is called causal or autoregressive masking. - MLM only occurs in the pre-training phase and is not used to make predictions. This is different from GPT’s causal masking that occurs during both training and inference. In MLM, certain input tokens (15% of the total) are randomly masked at any position. Then, the model is tasked to predict them given the full context of the sentence. Here, words to the left and right of each masked word are taken into account. MLM is used together with a technique called Next Sentence Prediction (NSP), which involves predicting if two sentences are consecutive (if the second sentence logically follows the first). This helps BERT further understand sentence relationships. - 50% of the time, the second sentence is the actual next sentence (labeled as "IsNext"), and 50% of the time, it is a random sentence (labeled as "NotNext"). ## How Does BERT Work? Transformers consist of an encoder that reads the text input and a decoder that produces a prediction for the task. Since BERT’s goal is to generate a language representation model, it only needs the encoder portion. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f1c01cd8-7db4-4ced-9051-676fc6f77fe1/Copy_of_Copy_of_Copy_of_saduiagduih908hdiso.png) ### Pre-Training BERT is pre-trained on massive amounts of text data (such as Wikipedia and BooksCorpus), totaling about 3.3 billion words. During pre-training, BERT uses the previously mentioned two techniques: it performs MLM and NSP in parallel during this phase. #### Input Pre-Processing ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f6c87ec0-a122-4cf6-9990-ef9ef2361ff8/saduiagduih908hdiso.png) Since NSP must be performed, two sentences are fed to BERT at a time—tokenized with WordPiece tokenization. This process splits words into smaller pieces, such as turning “running” into “run” and “ning.” A [CLS] token is added to the input tokens at the beginning of the first sentence, and a [SEP] token is inserted at the end of each sentence. Then, segment embeddings are added to each token, indicating which sentence the token belongs to. Like other transformers, positional embeddings are added to indicate a token’s position in a sentence. #### Masked Language Modelling (MLM) Masking is applied to these sentences for MLM. We randomly mask out 15% of the words in the input. Out of these 15% selected for masking: - 80% of the tokens are actually replaced with the token [MASK]. - 10% of the tokens are replaced with a random token. - 10% of the tokens are left unchanged. We run the entire sequence through the BERT encoder, predicting only the masked words based on the context provided by the other non-masked words in the sequence. Keep in mind that the BERT loss function considers only the predictions for the masked tokens and ignores the predictions of the non-masked ones. - This results in a slower convergence, with other models such as GPT converging faster per token (since GPT uses all tokens for loss and updates). #### Next Sentence Prediction (NSP) BERT also uses NSP during pre-training to further understand relationships between sentences, creating a well-rounded language model. This can be relevant for tasks such as question answering, where sentence-to-sentence relationships are crucial. As stated previously, the model is fed input pairs of sentences, and it learns to predict if the second sentence corresponds to the first logically. Pairs of input sentences are selected so that in half of the cases, the second sentence follows the first, and in the other half, it does not. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/08684592-777e-4abb-8478-f14fccaba91a/Copy_of_saduiagduih908hdiso.png) To predict if the second sentence follows the first, the input sequence (after processing) is fed into the transformer to produce outputs for all tokens—each token ends up with its own vector representation. The vector corresponding to the [CLS] token is transformed using a simple classification layer (a fully connected layer in a model used to predict labels), and the labels are assigned using the Softmax activation function. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/80f27dc0-674d-4005-8032-c1959e38f760/Copy_of_saduiagduih90sdsd8hdiso.png) During training, BERT seeks to minimize the combined loss of both MLM and NSP, jointly learning from both tasks. Specifically, the total loss is the sum of the MLM loss and the NSP loss. ### Fine-Tuning After pre-training, BERT is adjusted—or fine-tuned—for specific tasks, like figuring out if a review is positive or identifying nouns within text sequences. With BERT’s extensive pretraining, we can use smaller, task-specific data, removing the need for a huge (and expensive) task-specific dataset. Transferring BERT’s prior experience (from its pre-training phase) to a new task is an example of what we call transfer learning. Transfer learning is when you use a model trained on one task as a starting point for another task. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a2d9b74c-478b-46d2-8e6b-33e2ca64391b/image.png) If we want to fine-tune BERT for our own uses, we can do so by simply adding specific layers on top of the base model. ### Worked Example: Question Answering For example, let’s say we want to use BERT for question answering. At its core, question answering is simply a prediction task: when given a question, output a correct answer. To fine-tune BERT for QA, we train it on specific examples where it's given a context paragraph, a question, and the correct answer (which is a span of text within the context). - This still follows the NSP two-sentence inputs: the question represents the first sentence, and the context represents the second. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ac2ee27b-0789-44cc-8e74-5f9d51df1c77/Copy_of_Copy_of_saduiagduih908hdiso__1_.png) The model learns to predict the starting and ending positions of the answer span, which are both indices over the input tokens. To do this, on top of BERT’s output, we add two trainable vectors (layers). One predicts the start token, and one predicts the end token. The model then outputs two probability distributions over all tokens: one for start and one for end. This shows exactly why BERT is so useful. For question answering, we can simply take the BERT base model and train it to learn just two extra vectors—it can then be used for an entirely different application. Additionally, during fine-tuning, most hyperparameters stay untouched, making BERT convenient and easy to use. Note that during fine-tuning, we still need to transform our input into the same format that was used for pre-training, such as adding [CLS] tokens and adding segment embeddings that distinguish the different sentences (sentence one or two). ### BERT vs. GPT Summary In summary, here are some similarities and differences between Google’s BERT and OpenAI’s GPT. BERT is bidirectional, meaning it can process text both left-to-right and right-to-left. This is different from GPT, which is autoregressive (predicts the next with the previous) and unidirectional, processing tokens in only one direction. BERT uses the encoder portion of the Transformer model, while GPT uses the decoder—BERT is a language understanding model, while GPT is a text generation model. BERT performs two unsupervised tasks together: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). GPT’s text generation is straightforward, committing to autoregressive language modeling. BERT is used in a variety of NLP tasks—such as Google Docs, Gmail, Google Search, and voice assistance—among many others. GPT is used in ChatGPT, code generation, writing tasks (articles, scripts, etc.), and many more text generation tasks. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=bert-bidirectional-encoder-representations-from-transformers): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=bert-bidirectional-encoder-representations-from-transformers) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=bert-bidirectional-encoder-representations-from-transformers) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team