Issue #107 Architecture Edwin Dong
Joint-Embedding Predictive Architecture (JEPA), Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook, exploring JEPA and its variants with Python code.
We'll load official pre-trained weights, investigate learned representations, build a training loop from scratch, and do some benchmarking across models. Please do read the full article before attempting the code tutorial.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
The monthly interview question bank can be found here. As we’re approaching the end of June, look out for a new update!
These are industry standard questions that are tested across major tech companies, AI research firms, and more. The questions are sorted by difficulty and topic.
Check back frequently for more questions, and let us know what type of questions you would like to see next!
In recent years, Large Language Models (LLMs) and generative AI have been the main focus of AI research.
Although these models feature high-quality text generation (and outstanding fluency), in the pursuit of Artificial General Intelligence (AGI), Yann LeCun and others argue they lack a "world model."
Generative architectures like LLMs learn correlations between observations, but they don’t explicitly learn the abstract relationships (or rules) of the world that create those observations.
Additionally, from biology, humans and animals heavily rely on these abstract world models to learn enormous amounts of background knowledge, which we know as “common sense.”
The Joint-Embedding Predictive Architecture (JEPA) (Lecun, 2022) is a new predictive model that pulls from this biological inspiration to serve as a backbone for world modeling and higher-level reasoning.
This edition is for PRO members.
