# Self-Supervised Learning, Simply Explained #35 Jan 27, 2025 Training https://www.aibutsimple.com/p/self-supervised-learning-simply-explained Machine learning (ML) is divided into 3 main categories: supervised learning, unsupervised learning, and reinforcement learning. As a quick refresh, supervised learning involves training a model on labeled data, learning a mapping from input to output. An example of a supervised task is classification. Unsupervised learning trains the model on unlabeled data, learning an underlying structure or pattern without the guidance of labels, such as clustering data into groups. Reinforcement learning is a machine learning algorithm that allows an agent to learn environment behavior by trial and error using feedback and reward from its own actions and experiences. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3cad6856-d811-4472-b690-57787cf3c88e/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png) In addition to these three categories, in many of our issues, we discuss deep learning, a field of machine learning that uses neural networks to learn from input data. Tasks solved in deep learning are usually supervised—such as image classification, speech recognition, and translation. --- ### Learn AI in 5 minutes a day What’s the secret to staying ahead of the curve in the world of AI? Information. Luckily, you can join 1,000,000+ early adopters reading [The Rundown AI](https://subscribe.therundown.ai/?form=open) — the free newsletter that makes you smarter on AI with just a 5-minute read per day. [Sign up to start learning.](https://subscribe.therundown.ai/?form=open) --- If you want to learn more about the different types of ML, feel free to check out our previous issues going over them: - [Fundamentals of Reinforcement Learning (Beginner)](https://www.aibutsimple.com/p/fundamentals-reinforcement-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=self-supervised-learning-simply-explained) - [Supervised Machine Learning, Simplified (Beginner)](https://www.aibutsimple.com/p/supervised-machine-learning-simplified?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=self-supervised-learning-simply-explained) - [Unsupervised ML (Clustering) (Intermediate)](https://www.aibutsimple.com/p/unsupervised-machine-learning-clustering?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=self-supervised-learning-simply-explained) However, recently, there is a growing field in ML called self-supervised learning (SSL), which has been known for its efficiency and versatility. Self-supervised learning is a machine learning process where a model creates its own training labels from unlabeled data, allowing itself to learn one part of the input from another part of the input. It is also known as predictive or pretext learning. It is a subset of unsupervised learning. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/41a65d7f-60f8-44ef-b2c4-29b9d0f251ea/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png) Unlike supervised learning—where large, labeled datasets are required—SSL doesn’t need explicit labeling. Instead, it generates tasks, called pretext tasks, that help the model learn. During the training process, we first tackle an unsupervised problem, then transform it into a supervised problem by auto-generating the labels. - The fundamental idea for SSL is to create supervisory signals by making sense of the unlabeled data provided (an unsupervised process) on the first iteration. - Then, we can use supervised learning models and neural networks in a later stage to target a specific task. We can understand this better with an analogy: Imagine a child is given some pictures of animals, but each picture is missing a small part like a tail or an eye. The child doesn’t know exactly what animal they’re looking at, but they’re asked to predict what part is missing, based on the visible features such as color or shape. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1369e049-0a22-4317-864c-bdf2504ccac8/image.png) Over time, the child learns to recognize animals better by understanding the connections between parts of the whole picture. This is all done without giving the child any indication of what animal each image is—there are no labels. We can tie this to the real world. For instance, in natural language processing, if we have a few words, we can use self-supervised learning to complete the rest of the sentence. Similarly, in a video, we can predict past or future frames based on available video data. ### The SSL Process --- Like we discussed earlier, training a self-supervised model consists of two stages: The first one is for the model to learn to solve a pretext task (also known as a supervised task), such as predicting image rotations or filling in words. This portion of the process is called “pretext task pre-training”, which is the earliest phase of the SSL model. The goal of having a pretext task is to allow the model to learn important features or representations of the data (such as edges, shapes, or context). The goal is not actually the pretext task itself but to force the model to learn useful general representations of the data. This portion of the process is similar to generative models such as the Generative Adversarial Network (GAN). In GANs, the generators are used to generate realistic images for the discriminator, while the aim of self-supervised learning is to identify good features that can be used for a downstream task instead of fooling the discriminator. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b00b8a6d-d695-4f92-873a-68929561e0da/image.png) Then, we have the downstream task. The downstream task is the transfer of the pre-trained model’s knowledge to a specific and useful task. During this transfer, we fine-tune the model using labeled data (there still is a need for labeled data) specific to the downstream task, although now, far less than was once needed. Some examples of downstream tasks include object recognition, image classification, or even text generation. ##### Benefits of SSL: - Data Efficiency - Scalability - Performance & Robustness SSL solves the issue of dependence on labeled data. For many years, building ML models has been largely dependent on good quality labeled data, which can be very costly. Obtaining high-quality annotated data is a major bottleneck in many ML projects. - Labeling data is also expensive and time-consuming. SSL reduces the dependency on labeled datasets by using unlabeled data. Since unlabeled data is abundant (think of billions of images and videos online), SSL scales well. Models trained with SSL often perform extremely well on downstream tasks. SSL also encourages models to learn more generalized representations, making them better at handling variations and noise—useful for handling real-world data. ### Applications of SSL --- ##### Image Data Imagine you have a dataset of photos but no labels. An SSL model might learn from these images by learning to solve pretext tasks such as: - Predicting the correct rotation angle of images that were rotated randomly (e.g., 90°, 180°, 270°). - Rearranging pieces of an image that was broken into small sections like a puzzle. These pretext tasks help with later downstream tasks such as image classification, since, for instance, to predict the rotation of an image, the model must recognize edges and shapes while also capturing the textures and overall structure of the object. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0d779bf1-1bbb-48ae-bfac-e2250195176a/image.png) ##### Text Data Given unlabeled text data, an SSL process could remove random words from sentences and train a model to predict the missing words. For example, we can feed the model the sentence “I like my coffee __”, which the model can learn to predict “black” or “hot”. Here, a portion of the data is masked out, and the model predicts the missing part. This process of guessing the missing word is called masked input prediction, and it’s used in large language models like BERT or other chatbots for text. By learning the pretext task of guessing the word, the model learns contextual relationships between words, and it also learns how to fit words together in sentences (which tackles syntax and semantics). ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6af7cf26-1584-4dec-84c9-095a79fd99a6/image.png) ### Seeking impartial news? Meet 1440. Every day, 3.5 million readers turn to [1440](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_impartial&_bhiiv=opp_0b07521c-4157-4a00-abd6-c828eecc1e63_1b75ca79) for their factual news. We sift through 100+ sources to bring you a complete summary of politics, global events, business, and culture, all in a brief 5-minute email. Enjoy an impartial news experience. [Join for free today!](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_impartial&_bhiiv=opp_0b07521c-4157-4a00-abd6-c828eecc1e63_1b75ca79) --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=self-supervised-learning-simply-explained): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=self-supervised-learning-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team