# Multimodal Large Language Models (MLLMs), Simply Explained

#95 Mar 30 Multimodal

https://www.aibutsimple.com/p/multimodal-large-language-models-mllms-simply-explained

Hello from the AI, but simple team! If you enjoy our content and custom visuals, consider sharing this newsletter with others or [upgrading](http://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) so we can keep doing what we do.

Most machine learning systems are built around a single modality. A language model reads text, an image classifier looks at pictures, and a speech recognizer processes audio.

Each is impressive within its functionality, but none resembles how humans actually perceive and reason about the world. We simultaneously read, see, hear, and combine modes effortlessly to understand what is happening around us.

Multimodal Large Language Models (MLLMs) are an attempt to close that gap. By connecting powerful language models to visual encoders and other modality-specific components, these models can receive an image, a video clip, or an audio recording alongside text and reason across all of them in a unified way.

MLLMs now power multimodal functionalities across major LLM providers, such as Gemini, Claude, and GPT. [GPT-4V (Vision)](https://openai.com/index/gpt-4v-system-card/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) led to a prominent breakthrough in MLLMs, and a large ecosystem of open and closed models has emerged rapidly around the same idea.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/214878ba-c441-4e47-b8fe-a5d641641812/Multimodal_LLM__MLLMs_.png)

### The Problem with Single-Modality Models

Large Language Models (LLMs) have shown remarkable emergent capabilities like instruction following, multi-step reasoning, and few-shot generalization.

However, they are inherently blind to everything that isn't text. They cannot look at a chart and describe what it shows. They cannot tell you whether the object in a photograph matches the description in the caption.

They process the world as a stream of tokens and nothing else.

Visual models have the opposite limitation. Systems like [CLIP](https://openai.com/index/clip/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) can represent images in a rich embedding space aligned with language, but they are not reasoning engines.

They can tell you that an image is semantically similar to a phrase, but they cannot write a story about it, answer a follow-up question about it, or explain why a meme is funny.

MLLMs resolve this by treating the LLM as the reasoning brain and the visual encoder as the sensory organ, with a learned connector bridging the two.

### The Three-Part MLLM Architecture

A typical MLLM is built with three core modules that mimic how humans process information.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2419a618-8bdf-46a0-96ff-84f658f44312/MLLM_Architecture__Modality_Encoder___1_.png)

#### The Modality Encoder

The first section of the MLLM architecture is the modality encoder, representing the eye. It compresses raw visual input (pixels, spectrogram frames, or point clouds) into a compact feature representation.

Rather than training these encoders from scratch, most systems borrow pre-trained ones. [CLIP's](https://openai.com/index/clip/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) visual encoder is the most widely used because it has already been aligned to language through large-scale training on image-text pairs.

This makes the fine-tuning downstream task considerably easier.

More recent work uses higher-capacity encoders like [EVA-CLIP (1 billion parameters)](https://arxiv.org/abs/2303.15389?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) for stronger visual representations, and several systems process images at higher resolutions by dividing them into patches and encoding each separately.

#### The LLM

The second module is the LLM, which represents the brain. It brings world knowledge, reasoning, and language generation. Most MLLMs are built on top of publicly available LLMs such as [LLaMA](https://www.llama.com/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained), [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained), or [Qwen](https://qwen.ai/research?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) rather than being trained from scratch.

This is not just a practical shortcut: the pre-trained LLM carries enormous amounts of general knowledge that generalizes to visual reasoning tasks when properly connected to a visual encoder.

#### The Connector

The connector is the translator between the two. Since LLMs can only process tokens, the visual features must be converted into something “token-like” before being fed into the language model.

The two dominant approaches are projection-based connectors, which use a small MLP to project visual features into the LLM's embedding space, and query-based connectors like [Q-Former](https://arxiv.org/abs/2303.15105?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained), which use a set of learnable query tokens to compress visual information into a smaller number of representation vectors.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59cca0f9-9b78-4553-9c1a-dd6696658495/Multimodal_LLM__MLLM__Connector_Types.png)

A third approach, feature-level fusion, inserts cross-attention layers directly inside the LLM to let language features attend to visual features at every layer. This strategy is used in [Flamingo](https://arxiv.org/abs/2204.14198?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) and [CogVLM](https://arxiv.org/abs/2311.03079?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained).

Overall, from empirical research, the exact connector design matters less than the number of visual tokens and the input resolution.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Real-World Interview Questions and Cases (From Real Companies)
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break Into AI With PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained)

---

### The MLLM Training Process

Building a capable MLLM requires three distinct training stages, each with different data and different objectives.

#### Pre-training

Pre-training aligns the visual and language modalities. The model is trained on large-scale image-caption pairs, using datasets like [LAION-5B](https://laion.ai/blog/laion-5b/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) (5.85 billion image-text pairs scraped from the web), [CC-12M](https://arxiv.org/pdf/2102.08981?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained), and [COYO-700M](https://github.com/kakaobrain/coyo-dataset?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained).

At this stage, the LLM and visual encoder are typically frozen, and only the connector is trained. The goal is not to teach the model to be helpful; it’s to teach the connector to translate visual features into representations the LLM already understands.

#### Instruction Tuning

Instruction tuning teaches the model to be useful. A multimodal instruction sample is formally represented as a triplet (I,M,R), where ”I” is a natural language instruction (e.g., "describe the image in detail"), “M” is the multimodal input, and “R” is the ground truth response.

The MLLM learns to predict the answer “A” using the below equation, where “θ” are the model parameters:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c0012431-f52c-4d51-9941-3863484cb6f4/image.png)

Training follows the standard autoregressive objective function (next-token prediction loss) from LLM pre-training. The model minimizes this objective “L”, shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fb20cf74-55fc-4fe1-9e80-1b6de4814905/image.png)

Here, “N” is the length of the ground-truth response, and “R<i” denotes all preceding response tokens. The model is penalized if it assigns a low probability “p” to the correct next token “Ri”.

One thing to note is that only the response tokens contribute to the loss, while the instruction and image tokens are used as context but are not supervised.

This is what makes instruction tuning different from standard supervised fine-tuning: the model learns to follow new instructions rather than memorize task-specific input-output mappings.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/84f5f16b-a1a6-45fe-a36c-16c40d3672c4/Instruction_Tuning_Loss.png)

The quality and diversity of instructions matter more than raw quantity. A smaller, well-curated instruction set consistently outperforms a larger, noisy one.

### Alignment Tuning

Alignment tuning addresses a more subtle problem: models that are fluent and capable are not necessarily honest or safe. Two main techniques are used here.

RLHF (Reinforcement Learning from Human Feedback) operates in three steps.

First, a reward model rθ is trained on human preference pairs. Given a multimodal prompt “x” and a pair of responses (yw,yl) where “yw” is preferred, the reward model is trained by minimizing the following objective:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f8911206-db67-4049-9f1e-62c19ccb2bf1/image.png)

The reward model then guides a policy “πRLɸ” trained via Proximal Policy Optimization, with a KL Divergence penalty to prevent the model from drifting too far from its original behavior, resulting in the large, penalized objective below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cc3d1c2a-98f8-4de6-97e5-ce725544a823/image.png)

Here, “β” controls how strongly the policy is penalized for diverging from the reference.

The KL term is what keeps the model from collapsing into [reward-hacking behavior](https://www.anthropic.com/research/emergent-misalignment-reward-hacking?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained). Without it, models tend to produce outputs that score well on the reward model while becoming incoherent.

DPO (Direct Preference Optimization) simplifies the entire RLHF pipeline into a single binary classification loss, eliminating the need for a separate reward model altogether.

The learning objective is as follows and uses the ratio of how likely the model is to generate the winning/losing responses compared to the base reference model “πREF ”:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/512e7b26-d0f9-494d-b8d3-d8c541ed1848/image.png)

The intuition is elegant: the more the model increases the likelihood of the preferred responses past the baseline, the lower the objective becomes.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b82fdff9-3cee-49e3-829b-247d5f7c42ec/RLHF_vs_DPO.png)

DPO has become widely adopted because it achieves comparable alignment quality with a much simpler training loop.

### The Hallucination Problem

One of the most persistent failure modes of MLLMs is hallucination: generating text that is fluent and confident but inconsistent with the image.

A model might describe an object that is not in the scene, attribute the wrong color to something that is, or invent a spatial relationship between two objects.

This is not random noise; it reflects a structural tension in how MLLMs work. The LLM component carries strong statistical assumptions about what objects typically appear together, and when the visual signal is weak or ambiguous, those priors can override what the image actually shows.

Hallucinations fall into three categories of increasing complexity.

- Existence hallucination is the most basic: the model claims an object is present when it is not.
- Attribute hallucination describes real objects incorrectly, such as with the wrong color, wrong size, or wrong material.
- Relationship hallucination is the most subtle, involving false claims about how correctly identified objects relate to one another spatially or causally.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0505cbf7-ff89-464d-b174-e8eb5343855c/Types_of_MLLM_hallucinations.png)

Three broad strategies have emerged for addressing this.

- Pre-correction approaches collect training data that includes negative examples, cases where the correct response is to refrain from describing things not visible in the image.
- In-process-correction approaches intervene during generation. Visual Contrastive Decoding (VCD) observes that injecting noise into an image causes models to lean on language priors rather than visual content and designs a decoding scheme that contrasts noisy and clean visual signals to suppress hallucinations.
- Post-correction approaches treat the generated response as a draft and run a separate correction pipeline.

### Extended Capabilities

The basic MLLM architecture has been extended in several directions that significantly expand what these systems can do.

- Finer granularity of interaction. Early MLLMs treated the image as a single, whole input. More recent work supports region-level and pixel-level interaction where users can click on a specific object, draw a bounding box around a region, or sketch a shape, and the model responds with reference to exactly that part of the image. [LISA](https://arxiv.org/pdf/2308.00692?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) extends this further to support mask-level reasoning, where the model's response is grounded at the pixel level.
- More modalities. Beyond image and text, MLLMs have been extended to video, audio, 3D point clouds, depth maps, thermal imagery, and IMU data. [NExT-GPT](https://arxiv.org/abs/2309.05519?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) goes further still, supporting not just multimodal input but multimodal output, generating images, audio, and video alongside text by attaching diffusion models to the LLM.
- Domain-specific adaptation. The general architecture transfers well to specialized domains. [LLaVA-Med](https://arxiv.org/abs/2306.00890?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) adapts the approach to biomedical image understanding. [mPLUG-DocOwl](https://arxiv.org/abs/2307.02499?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) targets document understanding and OCR-free parsing. [CogAgent](https://arxiv.org/abs/2312.08914?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) and [Mobile-Agent](https://github.com/x-plug/mobileagent?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained) deploy MLLMs as GUI assistants capable of navigating smartphone interfaces and planning multi-step interactions.

### In-Context Learning and Chain of Thought

Two capabilities that made LLMs remarkable in the text domain have been successfully extended to the multimodal setting.

Multimodal In-Context Learning (M-ICL) allows the model to learn from a few demonstration examples provided directly in the prompt at inference time, with no gradient updates required.

The model sees two or three (image, instruction, response) examples and generalizes to a new image.

The key insight is that the model is not learning a pattern from data in the usual sense, but it’s learning instead from analogy, inferring what output format and reasoning style are expected from the demonstrations alone.

Sensitivity to example ordering and the relevance of demonstrations remain active areas of research.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d75af02b-93e9-4d33-98ca-50ed079ebef3/In-Context_Learning_for_Multimodal_LLMs.png)

Multimodal Chain of Thought (M-CoT) prompts the model to generate intermediate reasoning steps before arriving at a final answer.

Rather than jumping directly from image and question to answer, the model produces a rationale, a chain of observations and inferences that guides the final response.

This substantially improves performance on complex reasoning tasks. Approaches vary along two axes: chain structure (a linear chain of steps versus a tree-shaped decomposition with sub-questions and aggregation) and chain length (adaptive stopping versus a pre-defined number of steps).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/82ef958c-a049-4a8d-9d3d-88d3a9b20621/Multimodal_CoT_Reasoning_Chain__1_.png)

### Why do MLLMs Matter?

The convergence of language reasoning and visual perception represents a shift in what AI systems can do.

Traditional computer vision models are powerful but narrow because they classify, detect, and segment, but they do not explain, question, or compose. Traditional LLMs are general reasoners but have no sensory access to the world.

MLLMs combine both, and the result is a class of system capable of things neither could achieve alone: reading a handwritten equation in a photograph, explaining why a diagram is misleading, generating code from a screenshot of a UI, or navigating an unfamiliar app by looking at the screen.

The field is moving at a very high pace. The timeline from Flamingo in 2022 to [GPT-4V](https://openai.com/index/gpt-4v-system-card/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained), [LLaVA-1.5](https://github.com/haotian-liu/llava?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained), [CogVLM](https://arxiv.org/abs/2311.03079?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained), and their successors spans less than two years.

The core challenges that remain are long-context multimodal reasoning, robustness to adversarial inputs, conditional coverage across demographic subgroups, and scaling to more modalities.

These challenges are well-identified, and the tooling to attack them is improving rapidly. It can be said that the era of multimodal AI has only just begun.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici
- Yassine Ennaji

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multimodal-large-language-models-mllms-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team