Issue #95 Multimodal Edwin Dong, Anurag Shinde
Multimodal Large Language Models (MLLMs), Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook that dives into visual encoders, connectors, instruction tuning, multimodal hallucinations, and more.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
The monthly interview question bank can be found here. These are industry standard questions that are tested across major tech companies, AI research firms, and more. The questions are sorted by difficulty and topic. Check back frequently for more questions!
Most machine learning systems are built around a single modality. A language model reads text, an image classifier looks at pictures, and a speech recognizer processes audio.
Each is impressive within its functionality, but none resembles how humans actually perceive and reason about the world. We simultaneously read, see, hear, and combine modes effortlessly to understand what is happening around us.
Multimodal Large Language Models (MLLMs) are an attempt to close that gap. By connecting powerful language models to visual encoders and other modality-specific components, these models can receive an image, a video clip, or an audio recording alongside text and reason across all of them in a unified way.
MLLMs now power multimodal functionalities across major LLM providers, such as Gemini, Claude, and GPT. GPT-4V (Vision) led to a prominent breakthrough in MLLMs, and a large ecosystem of open and closed models has emerged rapidly around the same idea.
This edition is for PRO members.
