Issue #35 Training Edwin Dong
Self-Supervised Learning, Simply Explained
Free editionPRO edition
Machine learning (ML) is divided into 3 main categories: supervised learning, unsupervised learning, and reinforcement learning.
As a quick refresh, supervised learning involves training a model on labeled data, learning a mapping from input to output. An example of a supervised task is classification.
Unsupervised learning trains the model on unlabeled data, learning an underlying structure or pattern without the guidance of labels, such as clustering data into groups.
Reinforcement learning is a machine learning algorithm that allows an agent to learn environment behavior by trial and error using feedback and reward from its own actions and experiences.
In addition to these three categories, in many of our issues, we discuss deep learning, a field of machine learning that uses neural networks to learn from input data. Tasks solved in deep learning are usually supervised—such as image classification, speech recognition, and translation.
If you want to learn more about the different types of ML, feel free to check out our previous issues going over them:
However, recently, there is a growing field in ML called self-supervised learning (SSL), which has been known for its efficiency and versatility.
Self-supervised learning is a machine learning process where a model creates its own training labels from unlabeled data, allowing itself to learn one part of the input from another part of the input.
It is also known as predictive or pretext learning. It is a subset of unsupervised learning.
Unlike supervised learning—where large, labeled datasets are required—SSL doesn’t need explicit labeling. Instead, it generates tasks, called pretext tasks, that help the model learn.
During the training process, we first tackle an unsupervised problem, then transform it into a supervised problem by auto-generating the labels.
The fundamental idea for SSL is to create supervisory signals by making sense of the unlabeled data provided (an unsupervised process) on the first iteration.
Then, we can use supervised learning models and neural networks in a later stage to target a specific task.
A difference between self-supervised learning and supervised learning is the fact that the data labels used as targets in every iteration are different.
This edition is for PRO members.
