# Joint-Embedding Predictive Architecture (JEPA), Simply Explained

#107 Jun 22 Architecture

https://www.aibutsimple.com/p/joint-embedding-predictive-architecture-jepa-simply-explained

In recent years, [Large Language Models (LLMs)](https://newsletter.aibutsimple.com/p/language-models-are-few-shot-learners-the-introduction-of-gpt-3-4099) and generative AI have been the main focus of AI research.

Although these models feature high-quality text generation (and outstanding fluency), in the pursuit of [Artificial General Intelligence (AGI)](https://newsletter.aibutsimple.com/p/agi-simply-explained), Yann LeCun and others argue they lack a "world model."

- Generative architectures like LLMs learn correlations between observations, but they don’t explicitly learn the abstract relationships (or rules) of the world that create those observations.

Additionally, from biology, humans and animals heavily rely on these abstract world models to learn enormous amounts of background knowledge, which we know as “common sense.”

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/24098e6a-fa75-44ad-9016-121faf43b560/JEPA__World_Model__2_.png)

The [Joint-Embedding Predictive Architecture (JEPA)](https://openreview.net/pdf?id=BZ5a1r-kVsf&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained) (Lecun, 2022) is a new predictive model that pulls from this biological inspiration to serve as a backbone for world modeling and higher-level reasoning.

---

### What You’ll Learn

1. Discriminative, generative, and predictive architectures
2. Joint embedding architectures (JEAs), the structure behind JEPA
3. The component breakdown of how JEPA works
4. JEPA variants and applications
5. The current state of JEPA and world models

### What’s Helpful to Know

- [Self-Supervised Learning (SSL)](https://newsletter.aibutsimple.com/p/self-supervised-learning-simply-explained)
  - SSL is a learning paradigm in which a learning system is trained to “fill in the blanks.” A model derives its own supervision (labels) from the data itself, learning relationships between observed and unobserved inputs.
- [World Models](https://newsletter.aibutsimple.com/p/world-models-simply-explained)
  - Models that learn how the world works, or how an environment changes over time. World models have a form of imagination, where they can simulate (in latent space) future states for reasoning and planning.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2424884a-c69c-49de-9481-f5005f9e0b4e/World_Models_Cover.png)

- Representation Collapse
  - When a model learns a trivial solution where many inputs map to the same representation, essentially making the embedding useless.
- Contrastive Learning
  - A self-supervised technique where a model learns to pull representations of related (positive) pairs closer together in embedding space, while pushing unrelated (negative) pairs apart.

---

### Discriminative vs. Generative Architectures

Before understanding how JEPA works, we need to further understand why JEPA was created.

For years, there were 2 main ways models were structured: discriminative and generative.

Formally, discriminative architectures learn the conditional probability distribution P(Y | X), where X is the input and Y is the target variable.

- For instance, a convolutional network classifies an image X as a class Y.

The problem with discriminative models is their reliance on supervision (e.g. labels, behavior) and how they are brittle to out-of-distribution data. Lecun argues that these types of models might not serve best as world models.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fdc2719d-32cd-4a87-a22c-cd79f642155b/JEPA__World_Model__1_.png)

Generative architectures typically learn the input data distribution P(X). The difference here is that they learn how the data is produced rather than predicting outputs from inputs.

- This also eliminates the need for human labels by using a learning objective defined from the data distribution itself.

They can still model a conditional probability P(Y | X), but do it indirectly through the learned P(X).

- Note: Generative architectures may learn a joint distribution P(X, Y) or even a conditional P(X | Y). Common examples like transformer pre-training learn P(X) from raw text.

One of the key flaws of generative models in modeling the world is the allocation of representational capacity.

For example, vision models predicting exact pixels expend compute modeling stochastic, high-frequency noise, like the exact texture of leaves blowing in the wind.

So, given these structures and their respective limitations, where does JEPA fall?

JEPA defines a sort of third pillar in model structure, as it’s non-generative, but not exactly discriminative. Lecun describes it as a predictive architecture.

JEPA’s predictive nature attempts to solve the “world model” problem with standard generative architectures but also isn’t constrained to the narrowness of discriminative architectures.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/894f853c-1760-4c74-8f1f-80310194658d/JEPA__World_Model__6_.png)

Understanding how it accomplishes this requires diving deeper into the underlying architecture.

---

### Joint Embedding Architectures (JEA)

Fundamentally, JEPA is a joint embedding architecture (JEA).

A joint embedding architecture is a system that takes related observations (e.g., different types or views of data) and projects them into the same vector space.

Joint embedding models are typically used for representation learning, which is to learn latent representations that capture the underlying structure and semantics of raw data.

To learn representations, they take a self-supervised learning approach. This is because a joint embedding objective can often be defined without human labels.

The key thing to understand is that joint embedding models are not new. JEPA is simply a modern take on joint embedding models to adapt them for world modeling.

Early joint embedding models began with Siamese models, and then:

- Transitioned into contrastive joint embedding models (such as [SimCLR](https://arxiv.org/abs/2002.05709?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained))
- Extended to multimodal embeddings (with [OpenAI’s CLIP](https://openai.com/index/clip/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained))
- Progressing even to non-contrastive distillation models (such as [BYOL](https://arxiv.org/abs/2006.07733?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained) or [DINO](https://arxiv.org/abs/2104.14294?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained))

The goal of these models was to align pairs of inputs close together in an embedding space.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/853c23ac-8634-42a4-9565-dfb1a8f66ed5/JEPA__World_Model__5_.png)

JEPA keeps the joint embedding framework but changes the learning objective by adding a “predictive” aspect.

So, rather than aligning two views of the same observation (like the contrastive approach), JEPA learns to predict a latent embedding of a target observation from a contextual observation.

Formally, if xc​ denotes context and xt​ denotes a target region or future observation, JEPA learns a predictor P such that:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a0dd4f5e-afae-464a-9f64-7c559247d336/image.png)

Here, E is an encoder that maps observations into a latent representation space. JEPA uses 2 encoders (E) that may have different architectures and parameter values from training.

The crucial portion for world modeling then, is that the joint embedding (double-encoder) structure of JEPA predicts representations rather than reconstructing observations.

- Both the context and target are encoded into some latent space, stripping away the unnecessary fine details to model abstract relationships instead.

---

### The JEPA Architecture

The joint embedding architecture of JEPA consists of 3 main components:

1. Context Encoder
2. Target Encoder
3. Predictor

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2a24f988-3831-4fb5-acba-a4de2189b34a/JEPA__World_Model__7_.png)

The context encoder processes the observed portion of the data x (e.g., visible image patches or past video frames) to produce a context representation sx.

Similarly, the target encoder processes the unobserved (or future) portion of the data y to produce a target representation sy.

To prevent representation collapse, the parameters of the target encoder are usually updated with an Exponential Moving Average (EMA) of the context encoder's parameters.

The predictor, arguably the key piece of JEPA, takes the context representation sx and a conditioning latent variable z (e.g., region where to predict) and outputs a latent embedding prediction of the target representation ŝy.

The network is optimized to minimize the regression loss (L1/L2) between the predicted embedding (ŝy) and the actual target embedding (sy).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/63a3e856-d28e-46e2-96b7-967d6bcf1fea/image.png)

From here, to train JEPA non-contrastively and effectively, the system should follow some criteria:

- Maximize the information content of x in sx
- Maximize the information content of y in sy
- Minimize the prediction error between sy and ŝy
- Minimize the information capacity of the latent variable z

The fourth criterion is interesting. Why should we want to minimize information contained within z?

Well, if the latent variable z has too much information, the predictor could ignore the context sx entirely and map z directly to sy, like a shortcut solution.

To prevent this shortcut, we regularize z to be sparse, low-dimensional, or discrete, only parameterizing actual uncertainty.

---

If you enjoy our content and custom visuals, consider sharing this newsletter with others or [upgrading](http://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained) so we can keep doing what we do.

### [Code AI Projects With AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/963eba00-ab31-4912-8c1f-1470a52f9ecc/aibutsimple_academy__2_.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Real-World Interview Questions and Cases (From Real Companies)
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience
  [Stay Ahead in AI With PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained)

---

### JEPA Variants

The multimodal nature of the JEPA blueprint has seen quick adaptation into many domains, thanks to its predictive nature within latent space.

It translates pretty seamlessly to many downstream tasks, not needing additional modality-specific encoders.

We’ll take a look at a couple JEPA variants below, starting with the first major adaptation for images.

#### I-JEPA, Image-Based Joint-Embedding

Image-based JEPA, [I-JEPA (Assran et al., 2023)](https://arxiv.org/abs/2301.08243?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained), is a foundational empirical implementation of the JEPA architecture.

It uses non-overlapping masked target regions that force the model to predict missing image content from broader visual context, forcing the network to learn semantic representations of objects and scenes.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/40e09fac-f83d-4b70-89d9-620e3d2fc074/JEPA__World_Model__10_.png)

First, the model samples several (typically four) target blocks independently from the image. These blocks are relatively large (15% to 20% of the image), ensuring the target representations hold high-level semantic meaning (such as object parts).

I-JEPA samples a context block covering 85% to 100% of the image after sampling the target blocks. The pixels that overlap with the target blocks are removed, so its size will decrease quite a bit.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/48544399-9747-42b8-b52a-66441a603a6c/image.png)

The context block gets passed into the context encoder, the target block gets fed to the target encoder, and latent embeddings are generated.

The predictor receives the context embeddings concatenated with learnable mask tokens containing the positional embeddings of the target locations, and it will deduce what objects exist at specific locations given the surrounding visual context.

Because I-JEPA uses latent embeddings rather than reconstructing observation pixels, it has remarkable scalability.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d2936b3c-fc7b-4f95-b0fb-01280c005a3e/image.png)

It outperforms Masked Autoencoders (MAEs) on ImageNet with a linear probe while requiring ~10x less compute.

#### V-JEPA, Video JEPA for Temporal Understanding

While I-JEPA captures spatial semantics, world modeling and complex planning require causal understanding.

Simply put, JEPA must understand not just what things look like but also what causes what to happen over time.

For instance:

- If a ball hits a wall, it bounces.
- If the steering wheel turns, the direction changes.

V-JEPA (Video JEPA, [Bardes et al., 2024](https://arxiv.org/abs/2404.08471?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained)) extends JEPA into the temporal dimension by masking video clips instead of images.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6f941d69-4ec2-4957-894a-7bf1487c606a/JEPA__World_Model__11_.png)

To date, there are three versions of V-JEPA: V-JEPA, V-JEPA 2, and V-JEPA 2.1. For all versions, their underlying backbone is roughly the same.

First, a video clip is chopped into spatiotemporal patches called tubelets, which are small chunks of size 2 frames by 16×16 pixels.

Two kinds of masks are sampled per clip to cover certain tublet-token regions.

- The short-range mask is a union of 8 sampled regions, covering 15% of each frame.
- The long-range mask is the union of 2 sampled regions, covering 70% of each frame.

For both masks, the aspect ratio is randomly sampled from the range of 0.75 to 1.5, so they vary in shape and aren’t squares. Most importantly, these blocks extend through the entire temporal duration of the clip.

Then, similar to any other JEPA variant, an encoder processes the visible tokens and produces representations. A predictor takes the encoder's output plus learnable mask tokens and tries to predict the representation of the masked region.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b0fc045a-6ac6-45de-afbe-05ff7a096987/image.png)

By pre-training entirely on self-supervised video, V-JEPA is able to learn high-level rules of the world, being later adapted (V-JEPA 2-AC) for robotics planning.

An extension of V-JEPA for vision-language applications is [VL-JEPA](https://newsletter.aibutsimple.com/p/vision-language-models-vl-jepa-simply-explained), and if you wanted to learn more, we’ve already discussed it in a previous article, [found here](https://newsletter.aibutsimple.com/p/vision-language-models-vl-jepa-simply-explained).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6160e466-5453-47c4-a127-704dad210629/Copy_of_AI__But_Simple_Visuals_March_26__6_.png)

---

### What to Take Away

The trajectory of recent AI research has seen a large focus on generative modeling. Autoregressive and pixel-reconstruction models, while fluent, are expending representation capacity on exact data reconstruction.

JEPA asks an important question: “Is predicting observations the most efficient objective, or should we predict abstract representations instead?"

By isolating the objective strictly to latent space, JEPA discards noise to focus entirely on structural, semantic, and temporal dynamics of a world environment.

Adaptable to many downstream tasks, accumulating many research variants (as of 2026), the JEPA framework positions itself as a useful backbone for higher-level reasoning and planning, ultimately designed for world modeling.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu (AI Developer, BTech, [LinkedIn](https://www.linkedin.com/in/sai-krishna-pashapu-997358283/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained), [GitHub](https://github.com/saikrishna-pashapu?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained))
- Bob Boncella
- Demet Yazici
- Imran Khan

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 8000+ ML engineers? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=joint-embedding-predictive-architecture-jepa-simply-explained)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team