Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416

Yann LeCun details the fundamental limits of current LLMs and the architectures needed for genuine intelligence.

LLM Shortcomings

• Autoregressive token prediction prevents abstract planning and world modeling (0:00-20:07)
• Language supplies low-bandwidth data compared with the 10^15 bytes of visual input a child receives (0:00-10:01)
• LLMs pass exams yet cannot learn to drive or clear a table (0:00-10:01)

JEPA and World Models

• Joint Embedding Predictive Architectures learn abstract, noise-free representations instead of reconstructing pixels (20:09-50:17)
• Learned world models enable hierarchical planning by predicting future states from actions (40:15-1:00:20)

Open Source and Bias

• No single system can be unbiased; open-source platforms allow diverse fine-tuning across cultures and values (1:30:29-2:10:44)

AGI Is Gradual

• Progress requires hardware efficiency gains and iterative guardrail design, not a sudden breakthrough (2:00:41-2:47:05)

00:00: Introduction and Context

Speaker: Lex Fridman

• Introduces Yann LeCun and frames the conversation around open source AI, concentration of power in proprietary systems, and the belief that AI should empower humanity rather than concentrate control. LeCun argues that keeping AI locked behind closed doors would risk a future where a few companies control most information and decision-making diets. He emphasizes open sourcing as a path to widen access and reduce centralization.
• LeCun states his belief that people are fundamentally good and that open source AI can amplify human goodness by making people smarter. He pushes back against doomers who think humans are inherently bad or self-destructive in the face of powerful AI.
• Background notes: Yann LeCun is chief AI scientist at Meta, NYU professor, Turing Award winner, and a pioneer in AI. Meta AI has been a proponent of open-source AI, releasing models such as LLaMA 2, and planning LLaMA 3. He is known for criticizing alarmist views about AGI while acknowledging that AGI will likely emerge, but argues it can be guided and controlled rather than escape humanity.
• The host frames the conversation as a continuation of LeCun’s long-standing views about autonomy, openness, and the practical paths toward robust AI.

Notable line: "I see the danger of this concentration of power through proprietary AI systems as a much bigger danger than everything else." (00:00:00)

10:05: Limits of LLMs and Grounding in Reality

Speaker: Yann LeCun

• LeCun lists essential characteristics of intelligent behavior that LLMs lack: understanding the world (physical reality), persistent memory, reasoning, and planning. He argues autoregressive LLMs (e.g., GPT-4, LLaMA 2/3) cannot achieve human-level intelligence because they lack grounding in physical reality and robust world models.
• He notes the astonishingly large but ultimately insufficient scale of data used for training LLMs: roughly 10^13 tokens (~two bytes per token) from the internet, equating to about two terabytes of raw text. He contrasts this with a child’s 16,000 waking hours of experience and suggests that most of human knowledge comes from sensory observation and real-world interaction rather than language. He uses Moravec’s paradox to illustrate that high-level cognitive tasks are easier for computers than low-level physical interactions (e.g., driving, grabbing objects) for which humans excel.
• LeCun argues that much of what humans learn comes from direct sensory input and interaction with the world, not language. Language is a compressed, abstract representation, but many cognitive tasks rely on mental models and perceptual-grounded reasoning that language alone cannot capture.
• The debate about grounding: some researchers (notably computer vision and embodied AI researchers) emphasize embodied learning and environmental grounding; others focus on natural language and correlation with textual data. LeCun emphasizes that the environment is far richer than language alone and that a grounded AI requires more than text to learn robust world models.

Notable line: "Most of what we learn and most of our knowledge is through our observation and interaction with the real world, not through language." (00:00:00)

Blockquote: "If you expect the system to become intelligent just without having the possibility of doing those things [grounding in the real world], you're making a mistake." (10:05:00)

20:09: Vision, Video, and Latent Representations; The JEPA Idea

Speaker: Yann LeCun

• LeCun discusses the challenge of building a true world model from video data. He explains that predicting future video frames frame-by-frame (or frame-group) is far harder than predicting next words in text because video is high-dimensional and continuous, while language tokens are discrete. This makes learning a full video distribution impractical with current approaches.
• He introduces the Joint Embedding Predictive Architecture (JEPA) as an architecture that learns abstract representations by jointly encoding a raw input and its corrupted/augmented version, then predicting the representation of the clean input from the corrupted one. The goal is to learn representations that capture the essential structure of the world without needing to reconstruct every pixel detail.
• He outlines non-contrastive learning approaches (e.g., BYOL, VICReg, I-JEPA, DINO) that avoid collapse (where representations become constant) by using mechanisms beyond simple contrastive signals. JEPA leverages a predictor between two encoders (one fed with the clean input and one with the corrupted input), sharing weights to learn robust representations.
• The discussion contrasts reconstructive models (which attempt to reproduce pixel-perfect inputs) with representation-learning models that aim to extract high-level, useful features that can support downstream tasks such as recognition or planning.
• Video-specific variants include V-JEPA, which applies JEPA principles to video data by masking or transforming temporal segments, then predicting the full video representation from the masked version. Early results show that these representations enable downstream tasks like action recognition and assessing physical possibility (e.g., whether a video is physically plausible).

Notable line: "The big question is how do you train something like this? And until five years ago or six years ago, we didn't have particularly good answers... non-contrastive methods allow training without negative samples." (20:09:00)

Blockquote: "In video, you would have to predict the distribution of all possible frames in a video... we don't know how to represent distributions over high-dimensional continuous spaces in useful ways." (20:09:00)

30:13: JEPA vs Autoregressive LLMs; Abstraction and Planning

Speaker: Yann LeCun

• The core distinction: LLMs are autoregressive text predictors that generate one token at a time, left-to-right. They rely on language as the primary input modality and primarily predict next words. JEPA-style models operate in a latent representation space, focusing on predicting representations rather than pixel-level or token-level generation.
• The JEPA approach aims to learn abstract representations by filtering out details that are not easily predictable or relevant for higher-level reasoning (e.g., moving leaves in a tree). This higher level of abstraction helps the model generalize better, especially for tasks requiring understanding and planning, rather than memorizing pixel-level detail.
• LeCun argues that language is highly compressive and lower bandwidth for certain tasks, whereas visual and physical world understanding require richer representations and sensory inputs. Language alone is insufficient to build a robust world model; joint embedding approaches help lift the model to higher abstraction levels necessary for planning and control.
• He emphasizes that the predictive capability in JEPA is not about perfect reconstruction but about extracting stable, useful representations from data that can be used for reasoning and planning in a hierarchical fashion.
• The role of abstractions: human reasoning uses multiple levels of abstraction, from low-level motor control to high-level planning. A truly capable AI must learn at multiple levels of abstraction and be able to switch between them as needed.

Notable line: "Joint embedding lifts the level of abstraction of the representation... which is essential for intelligent systems." (30:13:00)

40:15: From Representations to Planning; Results and Implications

Speaker: Yann LeCun

• The conversation moves from theory to empirical results with video representations. V-JEPA demonstrates that learned representations from video can support downstream classification and action recognition tasks at a surprisingly high level, indicating that the representations capture meaningful dynamics and physical constraints.
• LeCun highlights the potential for world models to support planning. If a model can predict how the world evolves under actions, it can be used for model predictive control (MPC): planning a sequence of actions to achieve a desired goal. This is contrasted with purely predictive language models, which do not explicitly plan or simulate future states in the same way.
• He discusses the concept of hierarchical planning as necessary for long-horizon tasks (e.g., traveling from New York to Paris). The planning process should operate at multiple levels of abstraction: high-level goals (fly to Paris) and low-level motor commands (navigate city streets, board a plane). The key question is how to learn such hierarchical representations end-to-end rather than building them by hand.
• There is potential in robotics: with JEPA-like architectures and world models, robots could learn to plan actions in the real world, not just follow scripted behaviors.

Notable line: "A model that can predict the sequence of states corresponding to a sequence of commands... model predictive control." (40:15:00)

50:18: Planning, Tools, and the Role of Language in AI Systems

Speaker: Yann LeCun

• The discussion moves toward the integration of planning with tools and external systems. LLMs can be useful for high-level planning and abstraction, but to execute complex tasks (like going from New York to Paris or solving architectural problems), low-level grounding in perception, action, and environment is essential. JEPA-based systems could provide the internal world model and reasoning, while LLMs can perform high-level reasoning and interactions with external tools (search engines, calculators, robotics interfaces).
• LeCun emphasizes the potential of systems that learn to plan with world models and then pass control to action-oriented modules or tools for execution. He discusses the notion of Toolformer (conceptually) and other planning approaches that combine predictive representations with tool use to achieve tasks more efficiently than purely autoregressive generation.
• The vision includes a layered approach: an internal world model for planning, an abstract representation space for reasoning, and a language interface for communication and instruction. This aligns with a broader belief that AI systems should be capable of long-horizon planning and decision making rather than simply generating fluent text.

Notable line: "Hierarchical planning is absolutely necessary if you want to plan complex actions... you cannot plan a New York to Paris trip at millisecond granularity." (50:18:00)

01:00:00 Grounding Language with Experience; Common Sense vs Text

Speaker: Yann LeCun

• The dialogue explores whether language alone can support robust world models. LeCun argues that grounding in reality is necessary for true intelligence. He suggests that common sense and intuitive physics emerge from sensorimotor experience and interactions with the physical world, which language alone cannot provide.
• He notes that knowledge of the world is often not encoded in language and that there is a lot of tacit knowledge visible in daily life that is hard to capture in text. The argument is that language data is insufficient for constructing a shared, deep understanding of how the world works; embodied experiences provide richer, high-bandwidth information that can be leveraged to learn robust representations and planning capabilities.
• The conversation touches on the limits of language-based AI for common sense tasks and how joint embedding and non-contrastive learning from visual data can contribute toward building real-world understanding that language cannot fully supply on its own.

Notable line: "Language is a very approximate representation... there's a lot of tasks we accomplish by imagining mental models that have nothing to do with language." (01:00:00)

01:10: Hallucinations, Energy-Based Models, and the Limits of Autoregressive Generation

Speaker: Yann LeCun

• The discussion shifts to the phenomenon of hallucinations in LLMs. LeCun notes that the probability of producing a nonsensical or incorrect token grows with the sequence length due to the fundamental structure of autoregressive generation. He explains that a token-by-token generation process with independent-token assumptions leads to an exponential drift away from truth as you generate more tokens.
• He distinguishes between random, data-driven, prompt-dependent failures and fundamental limitations of the autoregressive approach. He emphasizes the long-tail of prompts and tasks that are not covered by pretraining or fine-tuning, which can result in surprising and incorrect outputs.
• The concept of energy-based models (EBMs) and their training is introduced as an alternative to purely predictive, next-word generation. EBMs score the compatibility of a pair (X, Y) and can be optimized to produce better answers by adjusting a hidden representation rather than generating tokens directly. He explains the need for regularization to avoid degeneracy and collapse in energy-based systems.
• The discussion also covers the idea that an energy-based approach could enable more deliberate, gradient-based inference, exploring representations and improving reasoning, rather than relying solely on generating likely word sequences.

Notable line: "The probability that an answer would be nonsensical increases exponentially with the number of tokens." (01:10:00)

01:20: Modeling and Optimization for Reasoning; EBMs and the Future of AI Reasoning

Speaker: Yann LeCun

• LeCun elaborates on energy-based models and the concept of optimizing over abstract representations to produce better answers. He describes a process where an encoder maps input X to a latent representation, a predictor estimates a representation of the answer Y, and a cost function evaluates the compatibility of the proposed answer with the input. The model is optimized to minimize energy for correct pairs and maximize energy for incorrect ones, using either contrastive or non-contrastive methods.
• He differentiates discrete token optimization (as in traditional LLM decoding) from continuous latent-space optimization. He argues that continuous optimization is far more efficient, enabling gradient-based refinement of answers rather than generating and selecting from a massive set of discrete token sequences.
• The concept of an energy-based system with an abstract representation (Z) is highlighted as a potential blueprint for future AI that can plan, reason, and adapt to new tasks. It would allow planning in a space of concepts rather than raw text, providing a path toward more general intelligence.

Notable line: "You can do this in abstract representation space; you can optimize to minimize the energy of the answer.'" (01:20:00)

01:30: Reinforcement Learning, RLHF, and the Path Forward

Speaker: Yann LeCun

• The conversation turns to reinforcement learning (RL) and RL with human feedback (RLHF). LeCun clarifies that RL is not inherently necessary for all AI progress, but can be useful when the world model is inaccurate or the objective is not perfectly specified. He argues that a world model and planning capability should come first, with RL used only to fine-tune or adapt models in cases where the environment or objectives necessitate exploration.
• He argues RLHF’s effectiveness comes from human feedback, which helps shape the reward signal and guide fine-tuning. However, RL-based fine-tuning is sample-inefficient; leveraging world-model-based planning and representation learning should be preferred for most tasks, with RL providing value primarily for long-horizon adaptation or to correct specific gaps in the model.
• He notes that RL techniques should be used sparingly and strategically, to adjust world models or critics when necessary, rather than as the primary training mechanism for all capabilities.
• He references “Toolformer” and other planning-related work as examples of integrating tools, external databases, or simulators into the reasoning process—bridging the gap between internal representations and external computation.

Notable line: "RL should not be abandoned completely, but its use should be minimized because it is incredibly inefficient in terms of samples." (01:30:00)

01:40: Open Source as a Path to Diversity and Safety; Guardrails and Governance

Speaker: Yann LeCun

• The discussion shifts to governance, openness, and the importance of diverse AI ecosystems. LeCun argues for open-source AI platforms to prevent a single or a few companies from controlling access to AI systems that mediate human knowledge and interaction. He emphasizes the importance of multilingual, culturally diverse, and locally trained models tuned for local needs.
• He argues that diversity in AI systems is essential for democracy and cultural preservation. Open-source platforms enable governments, NGOs, and diverse communities to fine-tune models on their data and ensure representation of language and culture. He cites examples from India and Africa where local-language AI deployment could help deliver essential services like healthcare information.
• LeCun addresses the business model question: can Meta monetize via open models? He explains that Meta can leverage a base model with services, ads, and business customers, leveraging an open foundation to accelerate ecosystem development. The revenue can come from providing AI-assisted services to users and enterprises while maintaining an open base model that others can build upon.
• He acknowledges the political and social sensitivities around content and bias, arguing that no AI can be truly unbiased for all audiences. The solution is diversity and guardrails: open systems with community-specific guardrails, while global guardrails ensure basic safety. The aim is to avoid centralized control of the digital diet while enabling local adaptations.

Notable line: "Open source platforms enable diversity in language, culture, and value systems; that’s the only way to avoid a biased, centralized AI diet." (01:40:00)

01:50: The Future of AI Governance, Industry Dynamics, and Market Forces

Speaker: Yann LeCun

• LeCun discusses the broader industry dynamics, noting that a handful of players will not be able to satisfy every community’s needs. Startups and open-source models are essential to create a diverse ecosystem of AI assistants that reflect different languages, cultures, and values. He mentions collaboration with governments and NGOs to tailor AI to local contexts and improve access to critical services.
• He responds to concerns about the political leanings of model developers by arguing that the more important factor is the customer’s acceptance and guardrails rather than the developers’ personal leanings. If a product is too biased or unsafe for any group, it risks backlash; openness and diversity help mitigate this risk.
• He references a spectrum of viewpoints on governance and the need for guardrails, while acknowledging that complete safety is not possible. The approach is iterative: design guardrails, deploy, observe, adjust, and improve continuously.

Notable line: "Diversity is better; Open source enables diversity; this will create an ecosystem with many different models suited to different communities." (01:50:00)

02:00: Hardware, Compute, and Accessibility of AGI

Speaker: Yann LeCun

• The discussion turns to computing hardware, scalability, and energy efficiency. LeCun notes that scale is necessary but not sufficient; we still far exceed the human brain in power efficiency. He estimates the brain operates around 25 watts, while GPUs operate at hundreds of watts, making immense efficiency improvements essential for ubiquitous, affordable AI.
• He discusses ongoing architectural innovations that enable scale without proportional increases in hardware costs. He mentions that progress is not solely hardware-driven; architectural innovations and more efficient implementations are crucial to bring AI toward human-like capabilities.

Notable line: "Scale is necessary but not sufficient; we’re off by a huge factor in power efficiency." (02:00:00)

02:10: AI Doomers, Timelines, and the Reality of Progress

Speaker: Yann LeCun

• LeCun addresses AI doomers and the misconception that AGI will be an abrupt event. He emphasizes gradual progress rather than a single breakthrough event. He argues that progress will be incremental, with milestones on the way to AGI, and that guardrails and governance will evolve in tandem with capabilities.
• He argues that big steps in AI have historically been disseminated quickly in industry, leading to rapid adaptation across the ecosystem. He cites examples from the diffusion and adoption of foundational ideas like self-supervised learning, transformers, and decoder-only architectures, which were rapidly reproduced after initial breakthroughs.

Notable line: "AGI is not coming as an event; it’s gradual progress that will unfold over years, not overnight." (02:10:00)

02:20: The Psychology of AI Doom and the Ballpoint Pen Analogy

Speaker: Yann LeCun

• A popular analogy is used to describe how people react to new technologies: a ballpoint pen’s potential for misuse triggers calls for bans and licensing, just as AI triggers calls for regulations and restrictions. LeCun uses this to illustrate the tendency of societies to overreact to new technology, fearing harm and seeking to control dissemination.
• He argues that history shows both positive and negative consequences from technological revolutions. Openness and a diversity of approaches help mitigate risks, while centralization can amplify risks if misuse occurs.

Notable line: "Open source platforms enable a diversity of models and guardrails; banning or licensing pens (or AI) would have unintended consequences and stifle progress." (02:20:00)

02:30: The Open Source Vision for LLaMA and Language Diversity

Speaker: Yann LeCun

• LeCun returns to the open-source vision, emphasizing that open-source models like LLaMA empower diverse communities to adapt AI for local languages and needs (e.g., languages in India and Africa). He notes collaborations to adapt LLaMA to multiple languages and contexts, enabling access to medical information and other critical services in regions with limited access to specialists.
• He discusses future generations of LLaMA and plans to improve multilingual and multimodal capabilities, and stresses JEPA-like architectures as a likely path forward rather than pure generative models.

Notable line: "Open source is the only way to preserve diversity of language, culture, and value systems in AI." (02:30:00)

02:40: The Hopeful Future: Printing Press, Democracy, and Human Progress

Speaker: Yann LeCun

• LeCun draws a parallel between AI assistants and the printing press: AI could massively raise human intelligence by providing everyone with smart, capable AI assistants. He likens AI to a digital extension of human intellect, potentially enabling a modern Enlightenment through better access to knowledge and reasoning tools.
• He acknowledges that new technology can also cause societal tensions and conflicts (as with the printing press). The key is to harness AI for education, knowledge sharing, and democratic discourse while implementing guardrails and safeguards to manage misuse.

Notable line: "AI will amplify human intelligence; everyone could have a staff of smart AI assistants, which could be better than us in some aspects. It could be wonderful for humanity." (02:40:00)

02:50: Closing Thoughts, Mentorship, and Advice for Students

Speaker: Yann LeCun

• LeCun offers career guidance for students and researchers: focus on learning representations and world models from observation; explore planning with learned world models; investigate hierarchical planning and learn how to adapt AI to real-world tasks without requiring massive scaling upfront.
• He emphasizes the importance of PhD work that advances core ideas such as JEPA-like architectures, planning with latent representations, hierarchical planning, and the integration of perception, action, and language in a unified framework.

Notable line: "If you want to do a PhD in this field, focus on training a world model by observation and planning with it; there are many opportunities beyond scaling up data." (02:50:00)

02:57: Looking Ahead: What Gives Hope for Humanity?

Speaker: Yann LeCun

• LeCun ends on an optimistic note: AI has the potential to make humanity smarter and help solve complex problems. He uses the analogy of the printing press and argues that although there are risks and challenges, the overall trajectory can be positive if AI systems are diverse, open, and guided by robust guardrails.
• He reiterates the core belief that people are fundamentally good, and that open-source AI can empower people to do more good in the world. He emphasizes democratic access to AI systems and the importance of open platforms as a means to protect freedom and diversity in the digital age.

Notable line: "AI will make humanity smarter; freedom and diversity in AI systems will protect democracy and knowledge sharing." (02:57:00)

02: The Open Source Gambit: Final Takeaways and Call to Action

• The discussion closes with a synthesis: to avoid monopolistic control and to maximize societal benefit, the AI community should pursue open-source foundations that enable diverse communities to tailor AI to their needs, languages, and cultures. Guardrails should be implemented in open systems to ensure safety while preserving the openness of the ecosystem.
• LeCun stresses that AGI is a long-term, gradual objective, and that progress will be incremental. The path forward should combine joint embedding and planning-based architectures (JEPA-like), non-contrastive self-supervised learning, and energy-based inference to enable robust world models and field-deployable robots.

Notable line: "Open source platforms are essential to preserve diversity and democracy in AI; AGI will be gradual, not an event, and we’ll need guardrails and diverse communities to steer it." (02:57:00)

Resources Mentioned

• LLaMA 2 and LLaMA 3 (Meta)
• LLaMA 2 languages and multilingual efforts (e.g., 22 official languages in India)
• BYOL (Bootstrap Your Own Latent) – DeepMind
• VICReg (Variance-Invariant-Covariance Regularization) – DeepMind/Fair
• I-JEPA (Joint Embedding Predictive Architecture) – FAIR
• DINO (Self-distillation with No Labels) – FAIR
• MAE (Masked Autoencoder) – He et al., used in vision representations
• Wav2Vec – Speech recognition and multilingual speech processing (Facebook/Meta)
• BYOL, VICReg, I-JEPA, DINO, DINO-v2 – self-supervised visual learning methods
• V-JEPA – Video JEPA variant for video representations
• Toolformer – planning with tools (DeepMind)
• Model Predictive Control (MPC) – planning with a world model
• Energy-Based Models (EBMs) – energy functions and optimization for inference
• Contrastive Learning – learning via similar/dissimilar pairs
• Non-Contrastive Learning – learning without explicit negative pairs
• RLHF (Reinforcement Learning with Human Feedback) – training reward models; fine-tuning
• RL (Reinforcement Learning) – learning via interaction and reward
• Moravec’s Paradox – low-level tasks easier for computers than high-level tasks; human-like perception hard for AI
• System 1 and System 2 – fast, automatic vs deliberate, planned reasoning
• Wav2Vec – multilingual speech recognition
• PyTorch/Deep Learning Tools – general references to the ecosystem (implied)
• Einstein-like discussions on cognitive architecture and planning in AI

Notable Quotes (verbatim, with timestamp)

00:00:00 — "I see the danger of this concentration of power through proprietary AI systems as a much bigger danger than everything else." 00:00:00 — "I believe that people are fundamentally good and so if AI, especially open source AI can make them smarter, it just empowers the goodness in humans." 10:05:00 — "Autoregressive LLMs can be useful, but they cannot deliver persistent memory, grounding in the real world, or robust planning." 10:05:00 — "The following is a conversation with Yann LeCun... He and Meta AI have been big proponents of open sourcing AI development, including LLaMA 2 and LLaMA 3." 20:09:00 — "The web-scale training of LLMs is not data that equals actual world experience; humans learn far more from observation than from language alone." 20:09:00 — "Linguistic data is a compressed, low-bandwidth channel to knowledge; the environment is richer than language." 30:13:00 — "The big difference with JEPA is that you learn an abstract representation rather than trying to reconstruct all pixels." 40:15:00 — "Model predictive control relies on a world model to plan sequences of actions; you can optimize to satisfy objectives." 50:18:00 — "Hierarchical planning is absolutely necessary for planning complex tasks like traveling from New York to Paris." 01:10:00 — "Hallucinations arise because autoregressive generation compounds errors; no free lunch here." 01:20:00 — "Energy-based models allow optimization over abstract representations rather than generation of discrete tokens." 01:40:00 — "Open source is essential for diversity; guardrails can be implemented in open architectures depending on community values." 02:40:57 — "AI will amplify human intelligence; everyone could have a staff of smart AI assistants." 02:57:00 — "AGI will be gradual, not a single event."

Topics

• Artificial Intelligence
• Open Source
• AGI / Advanced Machine Intelligence
• Self-Supervised Learning
• Joint Embedding Predictive Architecture (JEPA)
• Model Predictive Control (MPC)
• Energy-Based Models (EBMs)
• Robotics and Embodied AI
• AI Safety and Governance
• Multilingual and Multimodal AI
• Philosophical Foundations of Intelligence
• Moravec’s Paradox
• Turing Test and AI Evaluation