Issue #53 Multimodal Edwin Dong
Multimodal Machine Learning, Simply Explained
Free editionPRO edition
Multimodal machine learning is a recent trending field of ML algorithms that learn and identify relationships from multimodal datasets—datasets that contain multiple types of data, such as images, text, or audio.
In the real world, multimodal learning is one that takes advantage of multiple senses, such as giving a speech (audio) and pointing to a visual. This allows for a more complete view and deeper comprehension of the situation.
The same goes for ML models—the more data and modalities (types of data) we feed into the model, the more information it has to infer from to obtain an accurate prediction.
Consequently, researchers started to pose the question: If humans use five senses to experience the world around them, capturing loads of information, why not do the same with ML models?
Currently, multimodal models can be found in many real-world applications, such as image captioning, emotion recognition, text-to-image generation, text-to-video generation, and much, much more. These applications combine unimodal data to form multimodal datasets.
In multimodal deep learning, the most common modalities are visual (like images or videos), text-based, or sound-based (music, voice, etc.).
Combinations of these common unimodal data types result in multimodal datasets, and the most popular combinations are:
Image + Text
Image + Audio
Image + Text + Audio
Text + Audio
There are other modalities that are less common but are still used for certain niche applications, such as depth sensor data, 3D data, and LiDAR sensor data. For instance, video + LiDAR + depth sensor data is used for self-driving cars.
Modalities in medical applications can include CT and X-ray scans or even electroencephalogram (EEG) data.
Multimodal ML thrives by combining different modalities, as a model can better understand its environment since some signals or signs exist only in certain modalities.
For example, let’s take the task of emotion recognition. Recognizing an emotion takes more than just looking at the face or the visual modality.
The tone and pitch of a person’s voice (audio modality) houses a huge amount of information about someone’s emotional state, which may not be visible through their facial expressions.
Unimodal models such as computer vision models or language models simply do not have the versatility and the capabilities multimodal models may have.
This edition is for PRO members.
