# Vision Language Models (VL-JEPA), Simply Explained #93 Mar 16 Multimodal https://www.aibutsimple.com/p/vision-language-models-vl-jepa-simply-explained As we move forward with generative AI architectures, the industry is shifting from text-only interfaces toward multimodal systems capable of “seeing” the physical world. At the core of this development are Vision Language Models (VLMs), combining computer vision (CV) and natural language processing (NLP) to interpret both images/videos and text simultaneously. These VLMs typically are transformer-based, and they are built on the same autoregressive generation mechanism that powers LLM chatbots like ChatGPT, Claude, and Gemini. The main difference is that VLMs take in visual input and a text prompt and output a response token “one-at-a-time,” a direct consequence of modern transformer attention. VLMs are quite powerful for handling these multimodal inputs, but they are quite expensive, and a surprising amount of that cost goes toward modeling and computing things that may not matter for the final input. Essentially, the current state of VLMs is far from being the most optimized that they could be. This article will break down [VL-JEPA (Chen et al.)](https://arxiv.org/pdf/2512.10942?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained), a new vision-language model architecture from Meta FAIR, described in this [paper](https://arxiv.org/pdf/2512.10942?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained), which takes a different approach than standard VLMs. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9ca073f5-0f43-4c51-b382-f7efb401ac1a/Copy_of_AI__But_Simple_Visuals_March_26.png) Instead of generating token-by-token, VL-JEPA predicts. However, this prediction does not occur in the token space that generation occurs in. Rather, it occurs in the continuous embedding space itself and selectively converts into tokens/words. Put together, VL-JEPA paves the way for a model that trains faster and uses fewer parameters for some tasks that VLMs couldn’t accomplish. --- ### Your AI is resolving tickets. Is it keeping customers? [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4ebc980d-1457-4952-8ac5-b3ae99403f27/1200x630__1_.png)](https://www.gladly.ai/resources/customer-service-reports-guides/leveraging-ai-automation/customer-expectations-report-2026/?utm_source=beehiiv&utm_medium=content-syndication&utm_campaign=2026-01-Content-2026-customer-expectations-report&utm_term=XIAEVJHTGJ&_bhiiv=opp_306d3315-af56-4c5c-80d8-181fd8deb8f0_f9c1d024) Resolution rates look great. But [Gladly's 2026 Customer Expectations Report](https://www.gladly.ai/resources/customer-service-reports-guides/leveraging-ai-automation/customer-expectations-report-2026/?utm_source=beehiiv&utm_medium=content-syndication&utm_campaign=2026-01-Content-2026-customer-expectations-report&utm_term=XIAEVJHTGJ&_bhiiv=opp_306d3315-af56-4c5c-80d8-181fd8deb8f0_f9c1d024) reveals the metric most CIOs are missing — and what the data says about where AI investments actually translate into retention, not just throughput. [See the findings](https://www.gladly.ai/resources/customer-service-reports-guides/leveraging-ai-automation/customer-expectations-report-2026/?utm_source=beehiiv&utm_medium=content-syndication&utm_campaign=2026-01-Content-2026-customer-expectations-report&utm_term=XIAEVJHTGJ&_bhiiv=opp_306d3315-af56-4c5c-80d8-181fd8deb8f0_f9c1d024) --- ### What are Vision Language Models? A VLM is any model that takes both visual input and textual input as a prompt and then generates a text output. For instance, if you ask a model to parse through a video, it will begin by looking through the video frames, then it will read your specific question, and then produce an answer word-by-word. The key detail is that last part. VLMs generally create their responses word-by-word. In the same way as autoregressive generation from text-only LLMs, VLMs like [LLaVa](https://github.com/haotian-liu/LLaVA?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) and [Qwen-VL](https://qwen.ai/blog?id=qwen-vl&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) use this approach. The main issue here is that the process above does more work than necessary. Consider the following: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd6ca3fd-907a-46c4-aab7-43c7e6060ed8/Copy_of_AI__But_Simple_Visuals_March_26__1_.png) Both A1 and A2 are perfectly reasonable answers to come from a text-based LLM. However, LLMs and VLMs are inherently trained to go word-by-word, which locks them into certain patterns. This is derived from their training cycles, where they are penalized for not following specific phrasings. This leads to models attributing compute to which words are probable over what semantic meaning or reasoning needs to be conveyed. Another issue is raised as a more intimate tie to the autoregressive nature of VLMs. Since generation happens one by one, you need the full message to be generated before acting on it, taking many seconds or even minutes. But in real-world production environments, it needs to be faster. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b79c7728-2140-4825-b0fd-f341f9e75a00/Copy_of_AI__But_Simple_Visuals_March_26__3_.png) Take a live video feed, potentially security camera footage or a vlog stream: The model should have the capability to continuously gain information on what it’s seeing and generate output only when something changes. With autoregressive decoding in standard VLMs, the model would be continuously generating tokens to update its world state. It doesn’t have a way to do background processing. VL-JEPA steps in to address both of these problems, focusing the model’s efforts somewhere other than token generation. ### What Does it Mean to “Predict Meaning?” When multiple phrasings for the same answer can be correct, VL-JEPA disregards having to sift through all of them, rather focusing on predicting what all of them share: the embedding space. The embedding space in this context is the vector form of a sentence. This vector is the result of a text encoder transforming this sentence from holding words to holding semantic content through numbers. This means that the two phrases from earlier, which meant the same thing, will be nearly adjacent in embedding space, regardless of how many words they share in common. However, in token space, as we see it currently, the sentences are functionally two separate entities. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/090abcd0-1db3-4d92-adcd-42227d946e4e/Copy_of_AI__But_Simple_Visuals_March_26__4_.png) Practically, because VL-JEPA isn’t generating language during training, it actually doesn’t need a computational heavy model decoder during that phase, saving resources. The decoder is required only at the time of inference, and even more beneficially, when it is actually required. For tasks such as classification, which don’t require output text, the decoder is never needed; VL-JEPA compares embeddings directly. In the paper, the authors show that this is an almost 50% reduction in trainable parameters, shown by the relative sizes of the VLMs and VL-JEPA models. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/16029c00-2e89-4c45-ad14-9fdc82bf1580/Copy_of_AI__But_Simple_Visuals_March_26__5_.png) By crunching together our keys, values, and queries matrices, along with their weight matrices, we calculate an attention score for each word, which then allows models to generate future words by projecting onto a vocabulary dataset of vector embeddings. These vector embeddings represent words. --- ### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) --- ### The Architecture of VL-JEPA VL-JEPA is built upon four core modules: the X-encoder, Predictor, Y-Encoder, and Y-Decoder. The X-Encoder’s job is to decipher visual input. Usually, it will take in an image or a sequence of video frames and compress it into a sequence of visual embeddings (SV), analogous to text tokens that LLMs use. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/35853a06-fccf-4d01-98b8-574f1bab0591/Copy_of_AI__But_Simple_Visuals_March_26__6_.png) However, these visual embeddings represent spatial and temporal features, rather than words. VL-JEPA depends on [V-JEPA 2](https://ai.meta.com/research/vjepa/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) as its X-Encoder, a self-supervised vision transformer. A quick note, the X-Encoder is frozen during training, so its weights don’t change. The X-Encoder is a pre-trained encoder that knows how to extract features and which features to extract. VLMs also use a frozen ([CLIP](https://openai.com/index/clip/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained)) encoder, but V-JEPA 2 possesses upgraded video understanding, as it has been trained with self-supervised prediction. The Predictor engine is the core mechanism of VL-JEPA. It takes in two inputs: the visual embeddings from the X-Encoder (SV) and the text prompt/query (XQ) from the user. - The Predictor’s output is another embedding representing the prediction it makes using both input embeddings. Architecturally, the Predictor is initialized from the last eight Transformer layers of Llama 3.2-1B. The researchers also took a modified approach, removing the casual mask from the architecture. - The casual mask removes the ability for tokens to attend to future tokens, but in the case of reading between temporal frames and textual inputs, the absence of a causal mask is necessary. The output of these layers gets pooled and projected to a shared embedding space. The shared embedding space where the training loss is calculated is crucial to update the predictor’s model weight. The projection step is what takes the Predictor’s representation to the Y-Encoder. The Y-Encoder takes in the target, which represents the “ground truth” for the output of the Predictor, and embeds the target into the same embedding space as the predictor’s output. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e70d22d3-555f-4813-a2e2-997c66ebe839/image.png) The Y-Encoder is based on the [EmbeddingGemma-300M](https://ai.google.dev/gemma/docs/embeddinggemma?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) embedding model, which is pretrained but not frozen. Unlike the X-Encoder that is frozen, the Y-Encoder and the Predictor are trained side-by-side. However, the learning rate for the Y-Encoder is 5% of the Predictor’s rate. This is a good choice, as the Predictor’s initial guesses will be far from the ground truth since it isn’t well adjusted yet. Letting the Y-Encoder move too fast degrades the quality of the target embeddings. The specific tool that aids the loss process is [InfoNCE](https://arxiv.org/abs/1807.03748?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained), which has two components: an alignment term that pulls predicted embeddings to target embeddings and a uniformity term that pushes all embeddings apart, preventing the model from collapsing to the same solution. The final part is the Y-Decoder, which is also the simplest piece. In fact, it is deliberately limited, as mentioned previously. It takes the predicted embedding and converts it back into human-readable text autoregressively. The Y-Decoder is also not involved during training; it’s used only when you need text output. The decoder is invoked selectively through the experiments, highlighting the efficiency gains. The VL-JEPA architecture is especially versatile, though, supporting generation, classification, retrieval, and discriminative visual question answering (VQA). The following experiments display these capabilities quantitatively. ### Testing the Architecture The actual paper covers a more comprehensive set of evaluations, but the two experiments we will dive into are about the core claims of VL-JEPA. The first is about showing the efficacy of predicting embeddings rather than tokens. The second is about the smart utilization of the decoder and specifically was used to decode live video streams. As explored in Section 4.5 of the paper, the authors set up a direct comparison between VL-JEPA and a standard VLM where the only difference was the target ground truth. All the initialization settings were the same, but the VLM used a 1B parameter LLama 3.2 decoder, whereas the VL-JEPA uses a 500M parameter model trained with prediction embeddings. Both were evaluated at regular checkpoints from millions of samples, on video captioning and video classification. At 500K samples, the models are roughly even ground, but VL-JEPA accelerates after that point. The separation of classification accuracy between the models grows until VL-JEPA leads with 41.0% to 27.2% with half of the parameters. The surprising result comes from the fact that this is coming entirely from just differences in embedding space and token space. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f7ee8d9c-398d-4900-9481-55d359471f5c/image.png) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8d10d72d-694a-41a7-a403-b68a02f9d4f5/image.png) The Selective Decoding experiment is also significantly revealing. VL-JEPA produces a continuous stream of semantic embeddings, which stay in embedding space until the meaning of the visual inputs actually shifts. - Standard VLMs decode at regular intervals, no matter what’s happening, potentially leading to a waste of compute. The authors tested this on a dataset of 218 videos that are 6 minutes on average, with a fixed number of annotations per video. The goal of the experiment was to predict those annotations with the least number of decodings. VL-JEPA discovered that decoding every 2.85 seconds matched the quality of decoding every single second, which is an effective 2.85x decrease in decoding operations. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/68661312-7393-486d-8beb-9dc3e5987deb/image.png) That’s very impressive considering that it is a natural consequence of the architecture, without any extensions of memory or compute power. ### The Bottom Line VL-JEPA makes a strong case that the paradigm with autoregressive generation is expanding and finding new pathways. Shifting the focus from token space to embedding space, Meta FAIR has produced a model that trains efficiently with fewer parameters and performs with similar and greater numbers than current VLMs. With that being said, VLMs are still the go-to for reasoning-heavy and agentic tasks, as VL-JEPA is not well charted for those types of operations. Beyond that, there is no clear picture of how this architecture performs after scaling to beyond 1.5B parameters, where most of its counterparts live. It’s an exciting new development that poses the foundation for new direction, but it’s not a total replacement. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained): - Sushant Waidande - Sai Krishna Pashapu - Bob Boncella - Demet Yazici - Yassine Ennaji --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to help our team push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=vision-language-models-vl-jepa-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team