# Language Models Are Few-Shot Learners: The Introduction of GPT-3 #40 Mar 3, 2025 LLMs https://www.aibutsimple.com/p/language-models-are-few-shot-learners-the-introduction-of-gpt-3-4099 One of the most significant milestones in machine learning (ML) and AI research is the publishing of OpenAI’s groundbreaking paper “[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=language-models-are-few-shot-learners-the-introduction-of-gpt-3)” by Brown et al. In that paper, the researchers presented GPT-3, a massive language model with 175 billion parameters, and showed that scaling up language models by a huge factor (size + training data) allows them to perform new, unseen tasks with zero training for that specific task. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2450fa49-638f-41c9-b326-de30223f0a4e/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png) With no gradient updates, users give the model a few examples (or even zero examples) in a text prompt, and it infers what to do. This is a technique popularly known as few-shot learning, an in-context learning method allowing the model to learn from context without fine-tuning on specific tasks. Through the researchers’ experiments, they showed that bigger models don’t just memorize—they generalize patterns from diverse text. But before getting into the vastly popular few-shot learning method, let’s start from the very beginning. How did they train GPT-3? ### How GPT-3 Works --- GPT-3 is a transformer-based large language model (LLM), excelling at NLP tasks such as text generation or translation. Transformers are great at handling text sequences by weighing the importance of different words through their self-attention mechanism. Transformers are autoregressive models, meaning that they use past data to predict future outputs. For instance, an LLM works by predicting every next word in a sentence sequentially—being influenced by past words. --- Want to learn more about transformers? Check out our past issues explaining the transformer architecture: - [Transformers and Multi-Head Attention, Mathematically Explained (Intermediate)](https://www.aibutsimple.com/p/transformers-and-multi-head-attention-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=language-models-are-few-shot-learners-the-introduction-of-gpt-3) - [Transformers, Mathematically Explained (Part 2) (Intermediate)](https://www.aibutsimple.com/p/transformers-and-multi-head-attention-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=language-models-are-few-shot-learners-the-introduction-of-gpt-3) - [Introduction to Transformers and the Attention Mechanism (Intermediate)](https://www.aibutsimple.com/p/transformers-and-the-attention-mechanism?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=language-models-are-few-shot-learners-the-introduction-of-gpt-3) --- GPT-3 uses a decoder-only architecture (like GPT-2), meaning it doesn’t need an encoder like other transformer models used for translation (such as BART). - GPT only uses the decoder because its primary function is to generate text from a sequence of tokens, and the decoder is well-suited for this process. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8df2eaaf-d6a7-4f89-9a7d-835c85f3dca7/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___2_.png) GPT-3 has 175 billion parameters (compared to GPT-2’s 1.5 billion) and 96 layers (compared to GPT-2’s 48). This is extremely large compared to any example we’ve seen so far. Consider a feedforward network with 300 input neurons, 12 hidden layers with 25 neurons each, and 10 neurons in the output. The weights matrices take up 300×25 + 11(25×25) + 25×10 = 14625 parameters. The bias vectors take up 12×25 + 10 = 310 parameters. The total sum of parameters is 14935. This is not even a tiny fraction of how many parameters GPT-3 has—in terms of scale, GPT-3 is huge. In the paper itself, 8 different sizes of the model were trained, the largest being what we refer to as “GPT-3”. The statistics are listed below: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f4be1e63-f313-485b-a521-a1259fc0df1b/image.png) GPT-3 is pretrained on a massive dataset; the training data totals to about 45 TB of text. The majority of its training data comes from the Common Crawl dataset, a dataset consisting of over 250 billion web pages and petabytes of data. GPT-3 uses a filtered version, around 410 billion tokens (the size can be roughly measured using 1 token = 0.75 words). GPT-3 uses data from various categories: books (fiction, non-fiction), websites (Reddit, Wikipedia, news), code (GitHub repositories), and scientific papers (arXiv, PubMed). Here’s the specific breakdown on the training data and their weighting in the training process: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9f3cf019-f5b7-4c64-a0b7-3f44d05e15e8/image.png) Most of GPT-3’s training is unsupervised learning pre-training, where the model is not given labels (for example, GPT-3 is not told if it is doing a translation or a coding task). The model is just fed raw text data from books, articles, and other sources as listed above. The model essentially creates its own labels by trying to predict the next word in a sequence. To train GPT-3, the OpenAI team spent around $12 million in cloud computing resources while using energy equivalent to over 1250 households’ annual electricity use. ### In-Context Learning --- One of the key features of the paper is the use of in-context learning, popularizing the technique within language models. In-context learning refers to the ability of a model to understand and perform a task based on examples provided within the input prompt. What’s surprising about this behavior is that an “in-context learner” can perform new tasks with minimal examples, without training for that specific task and without updates to model parameters. For instance, a model might understand how to unscramble letters to make words given multiple examples in the input. The model doesn’t need to be retrained on a specific task; it instead "adapts" from the context and examples. Let’s take the task of "Translating English to French": We feed it some examples: "bread → pain", "cheese → fromage". Then, we can give it a query “mint →”, where the model generates the answer (“menthe”) by recognizing the pattern. Some examples can also be found in the diagram below: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1613f219-82ff-4f61-a708-fafdb3c803f5/image.png) The researchers describe the “inner loop” as the in-context learning cycle, which occurs within each forward pass in each sequence. The “outer loop”, made up of many inner loops, is the model’s unsupervised pre-training that performs parameter updates through stochastic gradient descent (SGD). Within the realm of in-context learning, they also introduce zero-shot, one-shot, and few-shot learning, cases where the model solves tasks with no examples (zero-shot), one example (one-shot), or a few examples (few-shot). #### Few-Shot Learning Few-shot learning is a specific case of in-context learning that uses limited examples to solve varying tasks—such as learning to recognize an animal from a photo. - Few-shot learning, as described here for language models, is related to few-shot learning used in other contexts in ML. Both involve learning based on a broad distribution of tasks and then rapidly adapting to a new task. For instance, consider the following prompt: “Answer the question: Q: What is the capital of France? A: Paris” Q: What is the largest planet in our solar system? A: Jupiter Q: What is the world’s longest river? A: The Nile River Q: What is the currency of Japan? A:” The model is given three examples of question-answer pairs, showing it the kind of answers expected. Based on the provided examples, the model can infer how to respond to the prompt asking it what the currency of Japan is (“Japanese Yen”). The main advantage of few-shot is a major reduction in the need for task-specific data and a lower chance to learn an “overly specific” pattern from a fine-tuning dataset. A disadvantage of few-shot learning and adapting within the prompt itself is that results from this method have so far been much worse than state-of-the-art fine-tuned models. Additionally, a small amount of task-specific data is still required. #### One-Shot Learning One-shot learning is a case of in-context learning that uses just one example and a natural language description of a task. An example of one-shot learning is shown below, where the prompt starts with the description of the task, then one example, ending with a prompt. “Translate English to French: sea otter → loutre de mer cheese →” The model would output “fromage” as the correct translation. The reason to distinguish one-shot from few-shot is that it most closely matches the way in which some tasks are communicated to humans. #### Zero-Shot Learning Zero-shot learning is a setting of in-context learning where a model generates an answer given only a natural language description of the task with zero examples of the task being performed. - This method is the most convenient and has the greatest potential for robustness but is also the most challenging. Here’s an example of a zero-shot prompt: “Translate English to French: cheese →” The model, given no context, should also output “fromage” as the correct answer. ### Zero-Shot, One-Shot, Few-Shot Performance --- We compare the accuracy of zero-shot, one-shot, and few-shot learning in the graph below: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/29dc4569-d87a-4989-bab6-ffebc19d33d0/image.png) Here, we observe that larger models make increasingly efficient use of in-context information. In other words, with scale, large models “automatically” become better at in-context learning. - The steeper “in-context learning curves” for large models show an improved ability to learn a task from contextual information. ### No More Fine-Tuning --- ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/35ba51da-37e7-47ca-a77d-af4cb6216fc3/image.png) One of the biggest benefits of few-shot learning is removing the need for fine-tuning. Before GPT-3, most NLP systems required extensive fine-tuning on labeled datasets. The problem with fine-tuning is that it takes up computational resources and time while also requiring lots of labeled data, which is hard to come across (and very expensive). GPT-3 broke this pattern, showing the world it could generalize well to new tasks with no fine-tuning at all—performing tasks from translation to coding with just a few examples in the prompt. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=language-models-are-few-shot-learners-the-introduction-of-gpt-3): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50,000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=language-models-are-few-shot-learners-the-introduction-of-gpt-3) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=language-models-are-few-shot-learners-the-introduction-of-gpt-3) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team