Mark Zuckerberg: First Interview in the Metaverse | Lex Fridman Podcast #398

Lex Fridman interviews Mark Zuckerberg inside the metaverse about codec avatars and mixed reality hardware that make virtual presence feel real.

Photorealistic Codec Avatars

• Codec avatars transmit only expressions on pre-scanned models, using far less bandwidth than video while preserving freckles, wrinkles, and eye movements that drive immersion (0:00-10:01).
• Subtle asymmetries and natural flaws, not perfection, create the sense that the other person is physically present.

Quest 3 Mixed Reality

• Quest 3 is the first mixed-reality headset priced for mainstream adoption, letting users see their real room while overlaying holograms and avatars (10:02-30:06).
• Controller-free hand tracking enables natural interaction such as playing a virtual piano on a physical table.

AI, Identity and Ethics

• Avatars raise identity questions; AI replicas of people or creators require predictable behavior to avoid misrepresentation (30:07-50:11).
• Open-sourcing Llama 2 accelerates community AI tools that will power future metaverse experiences.

The Next Computing Platform

• The metaverse delivers social and psychological benefits of co-location by shortcutting the laws of physics (50:12-1:04:28).

Major Topic: Introduction to Kodak Avatars and the Metaverse

SubtopicContext of the Conversation

• The host describes Mark Zuckerberg's appearance inside the metaverse as a photorealistic codec avatar with spatial audio, highlighting the sense of presence despite physical distance. This sets the frame for a discussion about presence, realism, and the future of human connection online.
• The opening tone emphasizes that the experience aims to feel like being in the same room, not merely a video call.

"the following is a conversation with Mark Zuckerberg inside the metaverse... this technology is incredible and I think it's the future of how human beings connect to each other in a deeply meaningful way on the internet" (Lex Fridman)

SubtopicWhat Kodak Avatars Are (Technical Foundation)

• Zuckerberg explains the Kodak Avatars project at Meta: scanning both participants in many expressions and building a computer model of each face and body, then collapsing that into a codec that transmits the intended appearance over the wire when wearing a headset. The goal is photorealism plus bandwidth efficiency compared to transmitting full video.
• The system aims to capture facial expressiveness, including subtleties like freckles, color variations, wrinkles, asymmetry, and eye behavior, which contribute to immersion.
• The avatar system is contrasted with cartoony avatars; Kodak Avatars are designed to preserve nuanced human expression while staying efficient to transmit.

SubtopicThe Science of Presence and Expression

• The discussion emphasizes that a large portion of communication is nonverbal; eyes, micro-expressions, and the corners of the eyes convey happiness, boredom, etc. The avatars are built to convey these nuances.
• There is recognition that “perfection” is not required for immersion; subtle flaws often enhance realism.
• The conversation touches on the potential to tune facial expression on a per-person basis, including how wide a smile should be, and whether users should be able to override their avatar’s expressiveness.

SubtopicCurrent State vs. Future Vision

• The duo describes a baseline: a very realistic avatar experience that surpasses the uncanny valley in practice, with ongoing refinements to microexpressions and per-user tuning.
• The vision expands beyond calls to collaborative experiences: in-person meetings, games, and mixed reality experiences where avatars blend with the physical environment.

SubtopicAccessibility and Scanning Process

• The current process involves lengthy scans (Pittsburgh lab) and the desire to enable rapid smartphone scans in two to three minutes that capture a broad set of expressions and basic identity.
• The ultimate goal is to reduce the cost and time of generating a high-fidelity avatar so more people can participate widely.

SubtopicHardware and Platforms (Quest Focus)

• The release plan emphasizes progressive rollout, starting with existing hardware (Meta headsets) and expanding to more people as scanning and processing pipelines improve.
• The discussion foregrounds Quest 3 as a major milestone in mixed reality (MR): improved displays, cameras, and the ability to overlay digital avatars into the physical world.

SubtopicMixed Reality vs. VR and Use Cases

• Mixed reality (MR) is presented as a major capability shift because it allows avatars to be anchored in real space and combined with physical surroundings, enabling safer and more natural interactions.
• The potential for meetings, remote collaboration, and casual social hangouts is highlighted, with Quest 3 enabling more natural integration of digital avatars into real rooms.

SubtopicReal-World Scenarios and Evolution

• Zuckerberg discusses how MR can improve workrooms and remote meetings by enabling participants to look and feel present around a table, combining physical presence with digital representation.
• The timeline is described as multi-year: rollouts will be gradual, with increasing fidelity and more applications (productivity, gaming, social experiences, remote work).

SubtopicIdentity, Expression, and Personalization

• There is an ongoing exploration of how to calibrate avatars to reflect genuine emotion while allowing individuals to tune or exaggerate expressions.
• Questions arise about how to handle identity (e.g., whether scans capture temporary appearances or stable identities) and how people may want to modify their avatar (weight, hair, etc.).

SubtopicFrequency and Feel of the Experience

• The emotional impact of being able to converse intimately with someone remotely is highlighted; the experience is described as profound and novel, potentially transforming long-distance conversations.
• The team acknowledges that 2D video may not convey the immersive quality of the 3D photorealistic avatar experience, but they anticipate continued evolution of how audiences perceive such experiences in video form.

SubtopicVision for the Next 5–10 Years

• The future envisions widespread MR glasses that blend physical reality with holograms and avatars, where the real world becomes a composite of physical and digital experiences.
• There is anticipation of AI-driven avatars and AI-assisted collaboration (AIS) that can inhabit meetings, augment human capabilities, and contribute with specialized personas.

SubtopicPractical Roadmap and Scaling

• The rollout strategy prioritizes expanding the scanner base, refining the capture pipeline, and integrating avatar experiences across apps and devices.
• Quest 3 is framed as the first device with mass-market MR capabilities, catalyzing a developer ecosystem to build new experiences around Kodak Avatars and MR interactions.

SubtopicPhilosophical and Social Implications

• The conversation probes the nature of reality in a blended physical-digital world and whether avatars are a new form of presence that redefines identity and relationship dynamics.
• The ethics of AI personas (AI-driven avatars) are raised, including the potential for AI to imitate or embody real people and the need for safeguards and governance.

SubtopicPractical Technology Details (Abridged)

• The Kodak Avatar pipeline relies on scanning, building a facial/body model, and encoding it for transmission over the headset network.
• The focus is on reading expressive signals (eyes, mouth, micro-expressions) with high fidelity while keeping bandwidth usage reasonable compared to raw video.
• Mixed reality relies on pass-through cameras, high-resolution sensors, and precise alignment of virtual elements with the user’s real-world perspective, including hand tracking as a key interaction modality.

SubtopicResources Mentioned (for deeper study)

• Kodak Avatars (Meta research project)
• Meta Quest 3 (mainstream MR headset)
• Meta Quest Pro (earlier MR-capable headset)
• Qualcomm’s next-generation MR chipset (used in Quest 3)
• Llama 2 (Open Source foundation model) and future foundation models
• AI Studio (tooling platform for creating AI personas and experiences)
• Jane Austen and Marcus Aurelius as example AI personas
• Snoop Dogg as Dungeon Master (AI persona application)
• Mixed reality pass-through and hand-tracking demonstrations (e.g., piano demo)
• Discussions around AI personas across Meta apps (Meta AI, Just Meta AI, etc.)

SubtopicKodak Avatars: Key Takeaways for Study

• Presence, not just images: The central claim is that realistic avatars can convey nuanced human emotion and presence in remote conversations, potentially transforming how we relate online.
• Bandwidth efficiency: Avatars compress expressive data into a codec, reducing bandwidth needs vs. video, while preserving realism.
• Per-person expressivity: Tuning avatar expressivity (e.g., smile width) to match individual differences is an active design question.
• Multi-sensory immersion: Spatial audio and photorealistic visuals contribute to a sense of actually being in the same room.
• Gradual rollout: The path to broad adoption is incremental—improve scans, reduce capture time, and integrate into existing apps and devices.

SubtopicMixed Reality as a Platform Shift

• MR (mixed reality) is framed as a foundational platform shift akin to smartphones, enabling holograms and avatars to coexist with the physical world in everyday spaces.
• The technology enables new kinds of collaboration: remote meetings with participants physically present as holograms, virtual instruments (e.g., piano), and shared digital objects anchored in real space.

SubtopicEthical and Social Considerations

• The possibility of simulating loved ones or historical figures raises questions about consent, memory, and the ethics of AI-driven impersonation.
• The role of AI personas in daily life (business assistants, customers, creators) invites policy and governance discussions around safety, accuracy, and trust.

SubtopicCreative and Economic Implications

• AI personas could transform customer service, education, entertainment, and social experiences by providing scalable, personalized interactions.
• The involvement of celebrities and public figures as AI personas (e.g., Snoop Dogg as a Dungeon Master) showcases potential for entertainment applications but also raises questions about licensing, rights, and authenticity.

SubtopicFinal Reflections and Future Possibilities

• The interview underscores that the next frontier is a truly blended real world—physical presence amplified by digital avatars and holograms, enabling richer, more natural social interactions.
• The conversation stresses gradual experimentation, community involvement (AI Studio), and ongoing refinement as core to reaching a more ubiquitous metaverse.

SubtopicMajor Takeaways for Study

• Presence is the core currency of social computing; photorealistic avatars are a route to deeper connection.
• Accessibility will come from fast, smartphone-based scans and more affordable MR hardware.
• The ecosystem will grow through developer platforms and AI personas, enabling scalable, varied interactions across apps.

SubtopicNotable Visuals and Concepts to Recreate (Descriptive)

• Visuals: A photorealistic avatar mirroring facial expressions, with spatial audio, placed in a shared virtual room that appears physically the same for both participants.
• Conceptual layouts: A dark room backdrop for a podcast, with the potential to render a real-world setting (e.g., office or living room) through MR overlays.
• Interaction cues: Hand tracking and gesture-based controls replacing traditional controllers in many MR scenarios.

Resources Mentioned

• Kodak Avatars (Meta research project)
• Meta Quest 3 ( mainstream MR headset)
• Meta Quest Pro (earlier MR headset)
• Qualcomm’s MR chipset (for Quest 3)
• Llama 2 (open-source foundation model)
• AI Studio (platform for AI personas)
• Jane Austen and Marcus Aurelius AI personas
• Snoop Dogg as a Dungeon Master AI persona
• Piano MR demo (hand-tracking-based musical interaction)
• Quest Pro/Quest 2 VR experiences and MR integration
• AR/VR content policies and AI governance discussions

Resources Mentioned

• Kodak Avatars (Meta research project)
• Meta Quest 3 (mainstream mixed reality headset)
• Meta Quest Pro (earlier MR headset)
• Qualcomm’s next-generation MR chipset used in Quest 3
• Llama 2 (open-source foundation model by Meta)
• AI Studio (Meta platform to build AI personas)
• Jane Austen AI persona (example)
• Marcus Aurelius AI persona (example)
• Snoop Dogg as a Dungeon Master AI persona (example)
• Piano demonstration (MR hand-tracking experience)
• Mixed reality pass-through and camera array details

Topics

• Metaverse
• Avatar Technology
• Mixed Reality
• AI Personalities
• Remote Communication
• Open Source AI
• Future of Work
• Privacy and Ethics