Lex Fridman

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

Nathan Lambert and Sebastian Raschka map the 2026 AI landscape: RLVR post-training, China’s open models, inference scaling, and the jagged road to AGI.

Key takeaways

10 points
  1. 1Inference scaling and RLVR produced the largest 2025 capability jump.
  2. 2China leads open-weight releases; US must respond with its own open models.
  3. 3Pre-training scaling continues but data quality now dominates returns.
  4. 4RLVR scales further than RLHF because rewards remain verifiable.
  5. 5Superhuman coder milestone is the next critical AGI checkpoint (~2031).
  6. 6Open models are essential for education and talent pipeline.
  7. 7Tool use reduces hallucinations but does not eliminate them.
  8. 8Work culture at frontier labs remains extremely intense.
  9. 9World models will be required for reliable planning and robotics.
  10. 10Authentic human experiences gain premium value as AI slop increases.

Summary

Nathan Lambert and Sebastian Raschka examine the state of AI in 2026, focusing on how inference scaling and RLVR are changing model behavior and economics.

China vs US Open Models

  • DeepSeek R1’s January 2025 release triggered the “DeepSeek moment,” proving near-SOTA performance at lower compute (0:00-12:04).
  • Chinese labs release permissive open-weight models to gain US developer mindshare while US firms avoid Chinese APIs for security reasons.

Model Usage Patterns

  • Users switch between fast non-thinking models (Gemini) and slow high-intelligence ones (Claude Opus 4.5) depending on task (12:07-24:09).
  • Speed wins for quick scripts; extended thinking wins for code and philosophy.

Scaling Laws & Inference Compute

  • Pre-training scaling continues but inference-time scaling (o1-style hidden thoughts) delivered the biggest 2025 leap (48:23-1:00:25).
  • RLVR now scales with log compute yielding linear gains on verifiable tasks.

Pre-, Mid-, Post-training Pipeline

  • Pre-training soaks up knowledge; mid-training adds long context; post-training (RLVR) unlocks skills (1:00:28-1:12:31).
  • Data quality and synthetic data now dominate returns.

RLVR vs RLHF

  • RLVR uses verifiable rewards on math/code, enabling self-correction and continuous scaling; RLHF saturates quickly (1:36:46-2:00:57).

AGI Timelines & Work

  • Superhuman coder milestone expected ~2031; full automation shifts humans to goal-setting (3:01:31-3:13:33).
  • Work culture remains 72-hour weeks at frontier labs; open models are critical for talent pipelines.

Deep read

Executive Summary

Nathan Lambert and Sebastian Raschka review the 2026 AI landscape. Inference-time scaling and RLVR delivered the largest recent gains while pre-training costs continue to rise. China leads in open-weight releases, forcing the US to respond with initiatives like the ADAM Project. The path to AGI hinges on continual learning, world models, and whether superhuman coding arrives before 2031.

Major Topic: State of AI in 2026 – Lex Fridman Podcast #490

00:00:00

Speaker: Lex Fridman Introduction and framing for a deep-dive into the state-of-the-art in AI, with Sebastian Raschka and Nathan Lambert. The episode frames progress in LLMs, coding, scaling laws, open vs closed models, China vs US dynamics, AI agents, GPUs, and the potential path to AGI. The guests are positioned as researchers, engineers, and educators who also communicate these topics accessibly.

"The following is a conversation all about the state-of-the-art in artificial intelligence, including some of the exciting technical breakthroughs and developments in AI that happened over the past year, and some of the interesting things we think might happen this upcoming year."
00:01:57

Speaker: Sebastian Raschka China vs US – who wins the AI race? Raschka emphasizes multi-timescale notions of winning. He argues there will be no single winner in 2026; access to technology will be shared as researchers rotate between labs, so no one entity owns the entire tech stack. The differentiator will be resources (budget/hardware) rather than proprietary ideas alone; the ecosystem will be defined by scale and capacity to deploy ideas, not a pure winner-take-all tech edge.

  • DeepSeek moment in Jan 2025 with open-weight DeepSeek R1 announced as state-of-the-art with lower compute costs; spurred a wave of frontier open-weight models in China (Kimi Moonshot, Minimax, Z.ai, MOe architectures in DeepSeek lineage).
  • Open-weight ecosystems in China grow rapidly; business models around openness versus API/API-like means and who pays for software are evolving. Open weights lower friction for Western uptake where security concerns limit cloud/API usage; firms may host models locally.
  • DeepSeek remains a player but faces competition; newer Chinese models leverage DeepSeek ideas and newer architectures, sometimes offering better performance due to more recent training, or simply due to architectural tweaks.
"Winning is a very broad term... there will be, The differentiating factor will be budget and hardware constraints. I don't think the ideas will be proprietary, but rather the resources needed to implement them."
00:12:07

Speaker: Nathan Lambert Nathan adds nuance: Anthropic’s Claude Opus 4.5 has hype. Gemini 3 was Google’s moment but open-ended differentiation is fading; Claude Code is a differentiator for Anthropic. He argues that organizational culture (Anthropic’s lower chaos) helps momentum. He also notes open vs closed dynamics: US firms benefit from broad consumer use (e.g., ChatGPT, Gemini) while Chinese open models push open access and cost-effective deployment. He predicts continued progress in both spheres but cautions about business models and the fact that many firms won’t pay for Chinese API access in Western markets.

"Anthropic is known for betting very hard on code, which is Claude Code. Gemini 3 was Google’s moment, but differentiation is lower; the ideas space is fluid, but the scale and culture of organizations matter a lot."
00:21:38

Speaker: Lex Fridman Transition to: Best AI for coding and how developers interact with different LLMs. The conversation moves to code tooling and how programmers select tools across different models depending on tasks (e.g., Codeium, Claude Code, Grok, Gemini). They discuss personal practices—using different models for different tasks (fast queries vs deep reasoning vs code) and how long-context capabilities influence practical workflows.

"For code and any sort of philosophical discussion, I use Claude Opus 4.5 with extended thinking. Also always with extended thinking."
00:28:29

Speaker: Sebastian Raschka Open-source vs closed-source LLMs – they discuss the landscape of open models (DeepSeek R1, Kimi, MiniMax, Z.ai, Moonshot, Qwen, Mistral AI, Gemma, gpt-oss-120b, Nemotron 3) and the tradeoffs around tool use, licensing, and deployment. Raschka emphasizes the potential of tool use (execution via Python interpreters, web searches) as a major unlock, and the importance of open licenses for enterprise adoption. He highlights the balance between open-source licenses (easy to deploy locally) and the business incentives that push toward closed platforms. He also discusses MoE (Mixture of Experts) models and the trend toward larger Chinese open-weight models with MoEs that outperform smaller Western counterparts in peak performance.

"gpt-oss-120b is actually a very strong model and does some things that other models don't do very well. Selfishly, I'll promote Western models, but the open models from China have friendlier licenses and make deployments easier for enterprises."
00:36:17

Speaker: Nathan Lambert Nathan continues with practical observations: tool-use is a major unlocking mechanism for open models and a lever to drive real-world usage. He notes the cost and latency differences: Chinese models often run locally with fewer GPUs per replica, potentially slower but cheaper to operate; US models tend to be better in output quality and user experience today, which fuels paid usage. He recounts a personal anecdote about using Gemini for fast, in-context tasks and Claude Opus 4.5 for coding and philosophy. He also mentions that many open models are opening up to tool calls, but trust and safety concerns remain.

"The simplest thing is: the US models are currently better, and we pay for marginal intelligence gains. Chinese models often require different deployment strategies, but their licenses are friendlier for open use."
01:04:12

Speaker: Lex Fridman Advance to: How AI is trained – Pre-training, Mid-training, Post-training. Raschka and Lambert discuss the data pipelines and training regimes. Pre-training uses vast corpora and can include synthetic data; mid-training addresses longer context and reduced catastrophe forgetting by focusing on higher-quality content; post-training includes supervised fine-tuning and RL-based methods (RLHF, RLVR). They emphasize the economics: pre-training is a cost center; serving and inference scale are the larger ongoing costs.

  • Pre-training: massive data (trillions of tokens) with next-token prediction; synthetic data and data curation matter for learning efficiency.
  • Mid-training: dedicates to specific capabilities (e.g., long-context) and combats catastrophic forgetting via curated data mixtures.
  • Post-training: RL-based fine-tuning (RLHF, RLVR, DPO, Constitutional AI) to unlock tool use, reasoning steps, and efficiency; RLVR scales training by letting the model improve through longer reasoning chains and verifiable tasks (math, code).
  • Data quality and licensing are crucial; data provenance and licensing impact model training and deployment, with ongoing policy debates and court cases (e.g., author rights and data licensing).
"Pre-training is the core knowledge acquisition; post-training unlocks capabilities via RL with verifiable rewards; middle training is about shaping and maintaining long-context abilities and quality data."
01:37:18

Speaker: Nathan Lambert & Sebastian Raschka Post-training explained: exciting research directions in LLMs. They discuss RLVR as the wave that unlocked long-context capabilities and tool-use in production models. They highlight that RLVR can dramatically improve performance quickly (e.g., a model improving from 15% to 50% accuracy on MATH-500 in a short run). They discuss concerns about data contamination when problems mirror training data and the difficulty of fully isolating training/test sets in large models. They reflect on the balance between RLHF (fine-tuning for style, tone, and formatting) and RLVR (problem-solving skills and longer reasoning). They also cover the compute dynamics: RL post-training is memory-bound and distributed, with distinct hardware considerations from pre-training. They emphasize continued value in pre-training, mid-training, and post-training, but note that current cost-benefit calculations often favor inference scaling and RLVR improvements over expanding pre-training budgets.

"RLVR turned out to be a perfect storm that enables longer reasoning, tool use, and better inference-time scaling—essentially a bridge from 'think fast' to 'think longer.'"
01:58:11

Speaker: Lex Fridman Advice for beginners on how to get into AI development and research. Nathan and Sebastian encourage building from scratch to truly understand LLMs—start with a small GPT-2 scale model, implement attention, positional embeddings, and KV caches, then gradually add ideas (Group Query Attention, RoPE, YaRN) to replicate OLMo-like architectures. They emphasize learning by reverse-engineering: inspect weights/configs in model hubs, replicate outputs, and verify against reference implementations. They discourage jumping straight to large frameworks; instead, learn fundamentals first, then use frameworks to scale, and eventually contribute to frontier labs by building evaluation tools and datasets, particularly in niche domains like reasoning, code, or math. They discuss the value of “struggle” in learning and warn against burnout from chasing too many threads at once.

"The goal is to understand how the LLM works; build a small model. Then reverse-engineer to learn the components and how to reproduce the outputs."
02:21:03

Speaker: Nathan Lambert Work culture in AI (72+ hour weeks) and the Silicon Valley bubble. They discuss the demanding work culture in frontier labs, the intense pace, and the risk of burnout. They reflect on the trade-offs of academia versus industry, noting OpenAI, Anthropic, and other frontier labs as high-pressure environments; yet these environments can offer considerable impact and compensation. They discuss 9-9-6 culture, the allure and risks of the Bay Area, and the possibility that the excitement around AI creates bubbles—some of which can be useful to accelerate progress, but may also have negative consequences if they detach from broader societal contexts.

"9-9-6 is a culture in some AI labs: 9am-9pm, six days a week. The intensity accelerates progress but comes at personal costs."
02:28:46

Speaker: Sebastian Raschka Silicon Valley bubble and policy implications. They discuss the role of policy (America’s Open Models/Adam Project) and the need for open models to sustain research and education. Raschka highlights the American Truly Open Models (ADAM) Project as a response to the rapid rise of Chinese open-weight ecosystems and a strategic effort to maintain US leadership in open AI models and infrastructure. He argues for broad open licensing to enhance education, research, and enterprise adoption in the US, citing NSF funding and policy initiatives that push for open-source/open-weight AI. The discussion touches on the balance between open data licensing, data rights, and the economics of model training and inference.

"Open models are an engine for AI research; the US should invest in open models to keep AI development accessible and competitive. The ADAM Project is one way to push this forward."
02:50:21

Speaker: Nathan Lambert Tool use, continual learning, and memory. They discuss how agents and tool use evolve, including the recursive language model concept and the potential of long-context mastery via memory mechanisms, memory layers, and context stratification. They discuss in-depth the role of tool-use in current closed models versus open models: closed models embed tools in the system with tight integration (proprietary apps, APIs), while open models rely on plugin-like approaches and user-chosen tools. They address memory, personalization (personal context/cache), and how to balance memory and compute costs (KV caches, memory footprint). They discuss the future of long-context as a tech constraint and potential architectural experiments (linear attention, sparse attention, MoE).

"In post-training, the model learns through RLVR; in memory/continual learning, you either evolve weights (continual learning) or rely on context windows (in-context learning)."
03:06:47

Speaker: Lex Fridman Timeline to AGI and the future of AI in industry. They discuss multiple viewpoints on AGI/ASI timelines; the conversation emphasizes that there is no agreed-upon definition and that many people think AGI is a moving target rather than a fixed milestone. They discuss “superhuman coder” and “AI researcher” trajectories as milestones on a path to broader AGI. They discuss the likelihood of fully automating software development versus maintaining human-in-the-loop collaboration, noting potential for a world with multiple agents across clouds and a distributed ecosystem rather than a single all-powerful model. They discuss economic implications and the possibility of mass automation for coding and other digital tasks, balanced against safety, governance, and human agency.

"The dream of a single AGI model is fading; the more likely scenario is a future with many capable agents collaborating with humans to solve complex tasks."
03:25:18

Speaker: Nathan Lambert Will AI replace programmers? They discuss the extent to which AI can automate coding and software development. Claude Code, Cursor, and other tools are highlighted as early agents that help with code but require human oversight and sensemaking. The conversation points to a future where AI handles routine coding tasks, but human designers and product folks still craft system architecture, integration, and safety. They discuss the balance between AI-assisted coding and preserving the human joy and learning that comes from problem-solving and debugging. They emphasize the importance of maintaining personal growth and avoiding over-reliance on AI to avoid burnout or atrophy of craft skills.

"The best coders will be those who orchestrate AI tools effectively, guiding and verifying results rather than handing control completely to the AI."
03:32:07

Speaker: Lex Fridman How AI will make money and future of big acquisitions in 2026. They discuss monetization strategies, including API services, enterprise licensing, and ad-supported models. They examine consolidation trends (Groq, Scale AI, Perplexity) and potential acquisitions (Anthropic, OpenAI, xAI) and the broader ecosystem. They discuss the role of licensing and open models, noting that open models may enable more widespread innovation and cheaper deployment, while closed models may yield premium experiences with deeper integration and higher margins. They discuss the possibility of platform-level monetization (ads) and the tension between user experience and monetization. They discuss the role of large-scale labs in defending research throughput and the risk of talent bottlenecks due to high compensation in industry labs.

"If everyone uses the same model, you need to differentiate via data, integration, or enterprise capabilities. The future of AI business will likely be a mix of open-model ecosystems and closed-platform advantages."
03:41:01

Speaker: Sebastian Raschka Future of AI labs, OpenAI, Anthropic, Google DeepMind, xAI, Meta. They discuss strategic positions of major players and the likelihood of continued progress. They emphasize: (a) open models serving as a research engine; (b) consolidation pressure; (c) continued innovation in tool use, reasoning, and agentic architectures; (d) the importance of education and policy to grow the AI workforce with strong open infrastructures. They discuss the “Manhattan Project for AI” as a hypothetical future but stress that the real value is in distributed, global collaboration across many labs rather than a single national project.

"We should invest in open models to enable broad innovation; big consolidation will occur but there will still be room for multiple players across a diverse ecosystem."
03:53:35

Speaker: Nathan Lambert Manhattan Project for AI and NVIDIA’s future GPU compute. They discuss the hardware backbone of AI progress: NVIDIA GPUs, Tanz the cost of running large clusters, and new chip architectures (Groq, Vera Rubin) optimized for inference vs training. They discuss the strategic advantage of vertically integrated stacks (NVIDIA, TPUs, Trainium, etc.) and the potential for specialized hardware to shift the costs of training and serving. They highlight the role of Blackwell/Google’s hardware, memory bottlenecks, and the need for scalable, reliable compute for next-gen models.

"The compute and hardware landscape will continue to shift; other players may push with novel architectures, but the incumbent advantage of ecosystems like NVIDIA’s CUDA remains strong for a while."
04:01:59

Speaker: Lex Fridman Future of human civilization. They discuss the societal and existential implications of AI progress, including the risk of misinformation, privacy concerns, and the management of AI-generated content. They stress the need for safety, governance, and responsible deployment as AI scales beyond lab settings into everyday life. They discuss the role of education, workforce transitions, and the importance of maintaining human agency and a sense of purpose alongside automation. They emphasize the necessity of focusing on positive societal outcomes and building tools to empower people rather than replacing them wholesale.

"The real impact is massive: enabling access to knowledge and tools that empower people globally. It’s a quiet force with enormous long-term effects, not a single leap."
04:08:15

Speaker: Lex Fridman Closing thoughts and reflections on the future. They end with gratitude toward the guests and a reminder of the ongoing, dynamic journey of AI development. They close with a quote from Albert Einstein about staying with the questions longer, inviting listeners to continue exploring, learning, and contributing to the field.

"It is not that I'm so smart, but I stay with the questions much longer."
04:13:59

Speaker: Lex Fridman End of the episode. Thanks and sponsors.

Resources Mentioned

  • DeepSeek R1, DeepSeek V3, DeepSeek V3.2 – open-weight frontier models (China)
  • Kimi Moonshot, MiniMax, Z.ai, Moonshot, Qwen, Mistral AI, Gemma – open-weight LLMs
  • OpenRouter, Perplexity – model serving/tools platforms
  • gpt-oss-120b – OpenAI’s open-weight 120B model with tool-use capabilities
  • DeepSeek R1 and MoE (Mixture of Experts) architectures, Multi-head Latent Attention, Group Query Attention, Sliding Window Attention
  • Qwen 3 – standout model for tool-use and math problems; Qwen base + RLVR discussions
  • Nemotron 3 – NVIDIA model with data release and MoE discussions
  • Llama/LLama 4 – Meta’s open-weight lineage; licensing and ecosystem dynamics
  • Claude Opus 4.5 – Anthropic model; Claude Code, extended thinking
  • Gemini 3 – Google’s model; marketing vs real user impact
  • o1, RLVR – OpenAI/DeepSeek-style large-scale reasoning and RL methods
  • o1, o2, o3 etc. – progression of reasoning models and scaling ideas
  • OLMo 3 – open-model data/architecture (Raschka reference to from-scratch materials)
  • Institute for Foundation Models / LM360 / Apertus / Manning books
  • Manning: Build a Large Language Model from Scratch; Build a Reasoning Model from Scratch
  • Box, Quo, UPLIFT Desk, Fin, Shopify, CodeRabbit, LMNT – sponsors (context only)
  • Box, Quo, Fin – platforms and tools for tooling discussions
  • AI2: Allen Institute for AI – data, models, and RLHF/ RLHF data anchors
  • RLHF – Reinforcement Learning from Human Feedback
  • RLVR – Reinforcement Learning with Verifiable Rewards
  • GRPO / PPO – policy optimization variations in RL
  • DPO – Direct Preference Optimization
  • Constitutional AI – alignment with automated preferences and AI feedback
  • AR vs non-AR (transformer) background: Attention Is All You Need, transformers, GPT lineage
  • ARXIV / arXiv – data sources for training / scientific papers
  • RHF, RLHF, and policy frameworks
  • World Models – Meta/others exploring world-model concepts in LLMs

Notable Quotes (verbatim)

"Winning is a very broad term... there will be, The differentiating factor will be budget and hardware constraints. I don't think the ideas will be proprietary, but rather the resources needed to implement them." — 00:12:07, Speaker: Sebastian Raschka
"Anthropic is known for betting very hard on code, which is the Claude Code thing... Gemini 3 was before this. And differentiation is lower." — 00:12:07, Speaker: Nathan Lambert
"The US models are currently better, and we pay for the marginal intelligence gains. Chinese open-weight models offer friendlier licenses and different deployment strategies." — 00:36:17, Speaker: Nathan Lambert
"RLVR turned out to be a perfect storm that enables longer reasoning, tool use, and better inference-time scaling." — 01:58:11, Speaker: Nathan Lambert
"Open models are an engine for AI research; the US should invest in open models to keep AI development accessible and competitive. The ADAM Project is one way to push this forward." — 02:50:21, Speaker: Nathan Lambert
"The real impact is massive: enabling access to knowledge and tools that empower people globally. It’s a quiet force with enormous long-term effects, not a single leap." — 04:08:15, Speaker: Lex Fridman

Key Takeaways

  • 2026 AI progress is multi-framed: no single winner; success is defined by resources and scale, not just breakthroughs. ⭐ HIGH PRIORITY
  • Open-weight models from China catalyzed a global move toward open AI tooling; the ecosystem is becoming more diverse and competitive. ⭐ HIGH PRIORITY
  • Tool use and external tools (calculators, Python interpreters, web search) are a foundational capability for modern LLMs and a key differentiator for open vs closed models. ⭐ HIGH PRIORITY
  • Post-training (RLVR, RLHF, DPO) is where practical, scalable improvements in reasoning and capability occur; the training regime matters as much as architecture. ⭐ HIGH PRIORITY
  • Long-context and memory are real bottlenecks; progress will come from smarter memory management and selective attention rather than brute-force scaling of context. ⭐ HIGH PRIORITY
  • Data quality and licensing are critical; synthetic data, data provenance, and licensing debates (copyright, licensing models) will shape how models are trained and deployed. ⭐ HIGH PRIORITY
  • The economics of training vs inference matter: pre-training is expensive but producing base models; inference scaling and post-training (RLVR) provide ongoing value and are often more cost-effective post-deployment. ⭐ HIGH PRIORITY
  • The AI ecosystem benefits from a mix of open and closed models; policy, open-source ecosystems (ADAM), and strategic investments will determine who leads in different segments (education, enterprise, research). ⭐ HIGH PRIORITY
  • The future of AI business will feature a mix of open weights and closed platforms, with potential acquisitions and consolidations among major players (Anthropic, OpenAI, Google DeepMind, xAI). ⭐ HIGH PRIORITY
  • Education and personal agency: learners should build from scratch to internalize fundamentals, then use tools to augment capabilities, while guarding against burnout and loss of craft. ⭐ HIGH PRIORITY

Action Items

  • For learners: build a small LLM from scratch (GPT-2 size), implement attention, KV cache, RoPE, and gradually add advanced components to mirror OLMo-like architectures. Validate outputs against reference implementations.
  • For researchers: pursue RLVR for hard-domain problems (math, code) and invest in data-curation pipelines; monitor data contamination risks and MRD (model-reported data) issues.
  • For practitioners: experiment with tool-use integration (web calls, calculators, interpreters) and consider hybrid deployments (local open models with trusted tool calls) to balance speed, privacy, and cost.
  • For policy and ecosystem builders: support open-weight AI initiatives (ADAM) and ensure licensing frameworks support both innovation and fair use; encourage data provenance and responsible data licensing.
  • For developers and product builders: explore multi-model workflows (Gemini, Claude, GPT-5 family) and tailor models to specific tasks (coding, math, reasoning) with mode-switching to optimize user experience and correctness.

Mental Models

  • Scaling Laws across Pre-training, Mid-training, Post-training, and Inference: Bigger models plus more data generally improve performance, but the ROI decreases as compute and data costs rise; optimal strategy often lies in a balance (inference-time scaling and RLVR can unlock big gains at lower cost).
  • Tool Use as a Core Skill: The ability to call external tools (calculators, interpreters, searches) is a fundamental construct that drastically improves reliability and utility; open models benefit greatly from plugin/tool ecosystems.
  • Agentic AI vs Single-Model Monolith: The future is likely a network of specialized agents across clouds and devices, each handling different tasks and collaborating, rather than one monolithic AGI entity.
  • Data Quality over Quantity: The quality and provenance of data (including synthetic data crafted with intent) often determines performance more than sheer data volume; data curation and verification are critical.
  • Memory and Long Context: Efficient long-context usage is not just about bigger contexts; it's about memory management, selective attention, and dynamic context condensation (e.g., compaction strategies, agent memory management).

Glossary

  • LLM: Large Language Model—a neural network trained to predict next tokens in text.
  • RLHF: Reinforcement Learning from Human Feedback—training where human preferences shape model outputs.
  • RLVR: Reinforcement Learning with Verifiable Rewards—scales RL with tasks whose outcomes can be objectively verified.
  • MoE: Mixture of Experts—an architectural approach where multiple sub-networks (experts) are selectively activated to handle different inputs.
  • KV cache: Key-Value cache—stores prior keys and values to accelerate attention and enable longer context processing.
  • Tool use: The model’s ability to call external programs, web searches, interpreters, or APIs to augment reasoning and factual accuracy.
  • OLMo: An open dataset/model lineage in the discussion; used as a touchstone for open models and data pipelines.
  • Pre-training / Mid-training / Post-training: Stages of model development; pre-training learns broad knowledge, mid-training tunes capabilities and long-context handling, post-training involves fine-tuning and alignment (e.g., RLHF/RLVR).
  • AGI / ASI: Artificial General Intelligence / Artificial Superintelligence—hypothetical future AI with broad or superhuman capabilities across tasks.

Quotes (verbatim)

  • "Winning is a very broad term... The differentiating factor will be budget and hardware constraints. I don't think the ideas will be proprietary, but rather the resources needed to implement them." — 00:12:07, Speaker: Sebastian Raschka
  • "Anthropic is known for betting very hard on code, which is the Claude Code thing... Gemini 3 was before this. And differentiation is lower." — 00:12:07, Speaker: Nathan Lambert
  • "The US models are currently better, and we pay for the marginal intelligence gains. Chinese open-weight models offer friendlier licenses and different deployment strategies." — 00:36:17, Speaker: Nathan Lambert
  • "RLVR turned out to be a perfect storm that enables longer reasoning, tool use, and better inference-time scaling." — 01:58:11, Speaker: Nathan Lambert
  • "Open models are an engine for AI research; the US should invest in open models to keep AI development accessible and competitive. The ADAM Project is one way to push this forward." — 02:50:21, Speaker: Nathan Lambert
  • "The real impact is massive: enabling access to knowledge and tools that empower people globally. It’s a quiet force with enormous long-term effects, not a single leap." — 04:08:15, Speaker: Lex Fridman

Topics

  • State Of AI 2026
  • Open vs Closed LLMs
  • AI Scaling Laws
  • Post-Training & RLVR
  • Tool Use & Agents
  • China AI Open Models
  • Pre-Training Economics
  • GPUs & AI Compute
  • Education & Learning In AI
  • AI Policy & Open-Source Adoption
  • Robotics & World Models
  • AGI Timelines
  • AI in Industry & Culture

Action Items

  • Build a minimal LLM from scratch on one GPU to internalize pre-training, SFT, and attention.
  • Prioritize verifiable-reward tasks (math, code) when experimenting with RLVR.
  • Use open-weight Chinese models for local or cost-sensitive deployments.
  • Track data quality metrics (almost-OCR yield, synthetic data ratio) rather than raw token count.
  • Allocate separate budgets for pre-training and long-horizon RLVR runs.
  • Evaluate models on both speed (non-thinking) and depth (extended thinking) modes.
  • Contribute evaluation failures to frontier labs for rapid adoption credit.
  • Learn tool-use patterns by forcing models to call external interpreters instead of memorizing.

Mental Models

Inference-Time Scaling
Allocate extra compute at inference for hidden chain-of-thought before emitting the first token; yields step-function gains on reasoning tasks.
RLVR vs RLHF
RLVR optimizes against objective correctness on verifiable problems and continues scaling; RLHF optimizes learned preferences and saturates.
Jagged Capability Curve
Models excel in domains with abundant verifiable data (frontend, math) and lag where data is scarce (distributed systems).

Glossary

RLVR
Reinforcement Learning with Verifiable Rewards: RL loop that grades model outputs against objective answers in math or code.
MoE
Mixture of Experts: architecture activating only a subset of expert sub-networks per token to scale parameters without proportional compute.
DeepSeek moment
January 2025 release of DeepSeek R1 that demonstrated near-SOTA open performance at low compute cost.
ADAM Project
American Truly Open Models initiative to build and host high-quality US open-weight models.

Notable Quotes

“Winning is a very broad term. I would say you mentioned the DeepSeek moment...”

Nathan Lambert · 0:00

“If there is code, and the code works, you know it's correct.”

Sebastian Raschka · 24:12

“The low-hanging fruit has mostly been taken... on reinforcement learning with verifiable rewards.”

Nathan Lambert · 48:23

Topics

#AI Models#Scaling Laws#Post-Training#AGI Timelines#Open Source AI

More from Lex Fridman Podcast

See all
Paul Rosolie: Uncontacted Tribes in the Amazon Jungle | Lex Fridman Podcast #489
3h 6m
Paul Rosolie: Uncontacted Tribes in the Amazon Jungle | Lex Fridman Podcast #489
Lex Fridman
Paul Rosolie details violent encounters with uncontacted Mashco Piro tribes and the lethal fight to shield the Amazon from narcos and loggers.
Infinity, Paradoxes, Gödel Incompleteness & the Mathematical Multiverse | Lex Fridman Podcast #488
3h 52m
Infinity, Paradoxes, Gödel Incompleteness & the Mathematical Multiverse | Lex Fridman Podcast #488
Lex Fridman
Cantor's infinities, Gödel incompleteness and the set-theoretic multiverse expose fundamental limits and plural realities in mathematics.
Deciphering Secrets of Ancient Civilizations, Noah's Ark, and Flood Myths | Lex Fridman Podcast #487
2h 5m
Deciphering Secrets of Ancient Civilizations, Noah's Ark, and Flood Myths | Lex Fridman Podcast #487
Lex Fridman
Mesopotamian cuneiform tablets predate and shape the Noah flood story while revealing how writing, games, and gods worked in the ancient world.
Michael Levin: Hidden Reality of Alien Intelligence & Biological Life | Lex Fridman Podcast #486
3h 18m
Michael Levin: Hidden Reality of Alien Intelligence & Biological Life | Lex Fridman Podcast #486
Lex Fridman
Biological systems exhibit goal-directed intelligence across scales via cognitive light cones, TAME protocols, and access to structured Platonic patterns rather than bottom-up emergence alone.
David Kirtley: Nuclear Fusion, Plasma Physics, and the Future of Energy | Lex Fridman Podcast #485
2h 36m
David Kirtley: Nuclear Fusion, Plasma Physics, and the Future of Energy | Lex Fridman Podcast #485
Lex Fridman
David Kirtley details how Helion's pulsed FRC fusion achieves direct electricity conversion at 80%+ efficiency, offering safe abundant power by 2028.
Dan Houser: GTA, Red Dead Redemption, Rockstar, Absurd & Future of Gaming | Lex Fridman Podcast #484
2h 45m
Dan Houser: GTA, Red Dead Redemption, Rockstar, Absurd & Future of Gaming | Lex Fridman Podcast #484
Lex Fridman
Dan Houser details the storytelling craft, open-world systems, and personal philosophy behind GTA and Red Dead while launching Absurd Adventures.