Issue #93 Multimodal Edwin Dong, Lalit Julapalli
Vision Language Models (VL-JEPA), Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook that will intuitively show VL-JEPA predicts semantic meaning, and how an embedding space is more efficient than a token space.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
The monthly interview question bank is now released; it can be found here. These are industry standard questions that are tested across major tech companies, AI research firms, and more. The questions are sorted by difficulty and topic. Check back frequently for more questions!
As we move forward with generative AI architectures, the industry is shifting from text-only interfaces toward multimodal systems capable of “seeing” the physical world.
At the core of this development are Vision Language Models (VLMs), combining computer vision (CV) and natural language processing (NLP) to interpret both images/videos and text simultaneously.
These VLMs typically are transformer-based, and they are built on the same autoregressive generation mechanism that powers LLM chatbots like ChatGPT, Claude, and Gemini.
The main difference is that VLMs take in visual input and a text prompt and output a response token “one-at-a-time,” a direct consequence of modern transformer attention.
VLMs are quite powerful for handling these multimodal inputs, but they are quite expensive, and a surprising amount of that cost goes toward modeling and computing things that may not matter for the final input. Essentially, the current state of VLMs is far from being the most optimized that they could be.
This article will break down VL-JEPA (Chen et al.), a new vision-language model architecture from Meta FAIR, described in this paper, which takes a different approach than standard VLMs.
Instead of generating token-by-token, VL-JEPA predicts. However, this prediction does not occur in the token space that generation occurs in. Rather, it occurs in the continuous embedding space itself and selectively converts into tokens/words.
Put together, VL-JEPA paves the way for a model that trains faster and uses fewer parameters for some tasks that VLMs couldn’t accomplish.
This edition is for PRO members.
