Language Models Are Few-Shot Learners: The Introduction of GPT-3

One of the most significant milestones in machine learning (ML) and AI research is the publishing of OpenAI’s groundbreaking paper “Language Models are Few-Shot Learners” by Brown et al.

In that paper, the researchers presented GPT-3, a massive language model with 175 billion parameters, and showed that scaling up language models by a huge factor (size + training data) allows them to perform new, unseen tasks with zero training for that specific task.

With no gradient updates, users give the model a few examples (or even zero examples) in a text prompt, and it infers what to do.

This is a technique popularly known as few-shot learning, an in-context learning method allowing the model to learn from context without fine-tuning on specific tasks.

Through the researchers’ experiments, they showed that bigger models don’t just memorize—they generalize patterns from diverse text.

But before getting into the vastly popular few-shot learning method, let’s start from the very beginning. How did they train GPT-3?