# Multimodal Machine Learning, Simply Explained #53 Jun 2, 2025 Multimodal https://www.aibutsimple.com/p/multimodal-machine-learning-simply-explained Multimodal machine learning is a recent trending field of ML algorithms that learn and identify relationships from multimodal datasets—datasets that contain multiple types of data, such as images, text, or audio. In the real world, multimodal learning is one that takes advantage of multiple senses, such as giving a speech (audio) and pointing to a visual. This allows for a more complete view and deeper comprehension of the situation. - The same goes for ML models—the more data and modalities (types of data) we feed into the model, the more information it has to infer from to obtain an accurate prediction. Consequently, researchers started to pose the question: If humans use five senses to experience the world around them, capturing loads of information, why not do the same with ML models? ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f12a24ee-7f10-469f-bf60-82963ab39989/image_-_Edited_-_Edited.png) Currently, multimodal models can be found in many real-world applications, such as image captioning, emotion recognition, text-to-image generation, text-to-video generation, and much, much more. These applications combine unimodal data to form multimodal datasets. --- Big thanks to our partners for keeping this newsletter free. If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you’ll enjoy. ### Hire an AI BDR & Get Qualified Meetings On Autopilot [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e7a120ce-5859-4779-a317-7e146065a792/Beehiiv_Asset_8.jpg)](https://artisan.co/talk-to-sales?utm_campaign=8593553-Beehiiv+2025&utm_source=sponsored_email&utm_term=XIAEVJHTGJ&_bhiiv=opp_745f0fe4-e64b-4e25-8352-2a20abf00afe_1becabb6) Outbound requires hours of manual work. Hire Ava who automates your entire outbound demand generation process, including: - Intent-Driven Lead Discovery Across Dozens of Sources - High Quality Emails with Human-Level Personalization - Follow-Up Management - Email Deliverability Management [Hire Ava to slash costs & boost productivity.](https://artisan.co/talk-to-sales?utm_campaign=8593553-Beehiiv+2025&utm_source=sponsored_email&utm_term=XIAEVJHTGJ&_bhiiv=opp_745f0fe4-e64b-4e25-8352-2a20abf00afe_1becabb6) --- Multimodal ML thrives by combining different modalities, as a model can better understand its environment since some signals or signs exist only in certain modalities. - For example, let’s take the task of emotion recognition. Recognizing an emotion takes more than just looking at the face or the visual modality. - The tone and pitch of a person’s voice (audio modality) houses a huge amount of information about someone’s emotional state, which may not be visible through their facial expressions. Unimodal models such as computer vision models or language models simply do not have the versatility and the capabilities multimodal models may have. ### Multimodal Learning Challenges Multimodal learning involves 5 key challenges: representation, fusion, alignment, translation, and co-learning. #### Representation Multimodal representation is the task of how to best represent and encode modalities, combining them to form a vector or tensor—this involves fusing the information into one space. The problem lies in the fact that feature extraction from multimodal/multi-input data in a way that takes advantage of the synergies between modalities is difficult while also having to not pay attention to redundant information. Additionally, one modality should not dominate, and only essential information should be taken. #### Fusion Multimodal fusion involves integrating information from two or more modalities to perform a prediction task or generate an output. The goal is to combine the strengths of each modality to achieve better performance than using any single modality alone. Fusing multimodal information is the core (and most researched part) of multimodal research, but it comes with many challenges. One challenge is the fact that different data types have different statistical properties and noise, making them hard to combine directly. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c90cd63e-94fb-4416-b6cd-575c5d53ed79/image.png) Fusing Audio and Video Modalities Together Theoretically, the large portion of the challenge is finding the most optimal fusion technique. Deciding when to fuse (early vs. late) and how to fuse is the topic of ongoing research. Some fusion techniques include simple operations such as concatenations or weighted sums, while more sophisticated techniques include attention mechanisms such as transformers or attention-based recurrent neural networks (RNNs). #### Alignment Alignment refers to the task of identifying direct relationships between different modalities; different modalities often describe the same event or concept but need to be lined up correctly spatially or temporally. What we mean by this is that when different modalities refer to a similar semantic concept, their representations must be similar or close together in a latent space. For example, the sentence “a tennis ball hit the ground,” an image of a tennis ball, and the audio signal of a ball hitting the ground should lie close together in the representation space. #### Translation/Mapping Translating is the task of converting or mapping one modality to another. A common application is text-to-image generators, where we translate an image (visual modality) to text (textual modality). In this process, the semantic meaning should still be retained. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ed9eba7-a10b-42a8-a3d0-7b23253eee30/image.png) Types of translation: Example-based vs. Generative Part of the difficulty of translations is the fact that they are open-ended and subjective—no perfect answer truly exists. This is especially true in generative tasks, making evaluation of translation quality difficult, which adds to the complexity of the task. #### Co-Learning Multimodal co-learning refers to the ability to transfer knowledge learned through one or more modalities to tasks involving another modality. Alternatively, learning from multiple modalities at once to improve the understanding of each is also a form of co-learning. Co-learning is especially important in cases of low-resource tasks and missing or noisy modalities. Resource-rich modalities can improve the performance of resource-poor ones, balancing resources to optimize performance. Translation may be used as a method of co-learning to transfer knowledge from one modality to another. ### Multimodal Model Architecture ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/672e38b1-ed74-4cc0-9aab-35a4bda466de/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__7_.png) Although multimodal networks range widely in terms of architecture, most models have a couple key components in common. First, they are typically a combination of multiple unimodal neural networks. For example, an audiovisual model might consist of two unimodal networks, one for visual data and one for audio data. These unimodal neural networks usually process their inputs separately in a process called encoding. After unimodal encoding takes place, the information extracted from each model must be fused together. Many fusion techniques have been proposed, ranging from simple concatenation to taking advantage of attention mechanisms. - If you don’t know what attention mechanisms are or simply want to brush up on the concept, check out our issue [here](https://www.aibutsimple.com/p/transformers-and-the-attention-mechanism?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-machine-learning-simply-explained). The process of data fusion is one of the most important factors in determining the success of a model. After fusion takes place, a final “decision” network takes the fused information and is trained on the end task, producing a prediction or relevant output. Simply put, multimodal architectures usually have 3 main sections: - Unimodal encoders that encode individual modalities. Usually, one for each input modality. - A fusion network that combines the features extracted from each input modality during the encoding phase. - A classifier that accepts the fused data and makes predictions. ### Multimodal Example: Image Captioning Multimodal models are commonly used in image captioning, where the goal is to generate short text descriptions for a given image. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/04fe55bf-b17a-4580-96d6-a4ec605bed2f/image.png) To caption images, models must overcome the translation challenge described previously by mapping visual representations into textual ones. For a model to translate visual modalities into text, it has to capture the semantics of a picture. The model is tasked to detect the key objects, key actions, and key characteristics of objects. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1ef72e97-721f-4cf8-8355-2b720acdcff1/image.png) For instance, in the right photo in the image above, the model identifies “a horse,” “two people,” and “hay” as the key objects, “carrying” as the key action, and “a large load” as the key characteristic. The model also identifies spatial relationships, noting that the “narrow shelf” sits under the “bunk bed.” However, the task of multimodal translation is open-ended. The caption “two men riding a horse carriage with hay” is also a valid caption. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-machine-learning-simply-explained): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-machine-learning-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-machine-learning-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team