State of GPT | BRK216HFS

Andrej Karpathy explains how GPT assistants are built and used effectively.

Four Training Stages

• Pretraining consumes 99 % of compute on internet text to create base document completers (0:00-10:03).
• Supervised Finetuning (SFT) teaches initial helpful responses from human prompt-answer pairs (10:05-20:08).
• Reward Modeling trains a scorer on human rankings of multiple outputs (10:05-20:08).
• RLHF uses that scorer with PPO to produce ChatGPT-style assistants (10:05-20:08).

Base Models vs Assistants

• Base models complete documents; they do not answer questions unless prompted with examples.
• SFT + RLHF converts them into assistants humans prefer (10:05-20:08).

LLM vs Human Reasoning

• Humans use internal monologue, tools, and reflection; LLMs are sequential token simulators that need “tokens to think” (20:09-30:13).
• Techniques such as “let’s think step-by-step” and self-consistency spread reasoning across more tokens.

Practical Optimization

• Ask explicitly for expert performance; attach calculators, search, and RAG; prefer low-stakes co-pilot use cases (30:16-40:19).

00:00:00 Major Topic

00:00:00 Four-Stage Recipe for Training GPT Assistants

• Speaker: Andrej Karpathy outlines the emerging recipe for training GPT-like assistants, broken into four serial stages: pretraining, supervised finetuning (SFT), reward modeling, and reinforcement learning from human feedback (RLHF).
• The high-level pattern is: gather data for each stage, define an objective (loss), train a model, and end with an assistant model that benefits from the prior stages. The message is that this is still new and evolving, but the pipeline is becoming clearer.
• Key claim: the pretraining stage is where the bulk of compute happens (almost all FP operations and data processing), while the subsequent three finetuning stages require far less hardware and time.

> "this is where all of the computational work basically happens. This is 99 percent of the training compute time and also flops. This is where we are dealing with Internet scale datasets with thousands of GPUs in the supercomputer and also months of training potentially. The other three stages are finetuning stages that are much more along the lines of small few number of GPUs and hours or days."

00:05:00 Subtopic: Pretraining — data mixtures and tokenization

• The pretraining stage uses a large, mixed dataset often described as a data mixture. Examples include CommonCrawl (web scraping) and C4 (another CommonCrawl-derived dataset), plus high-quality sources like GitHub, Wikipedia, Books, Archives, and Stock Exchange content.
• Data are sampled according to specified proportions to form the training set for the GPT.
• Tokenization is the step that converts raw text into a sequence of integers (tokens) the model can operate on. Tokenization is a lossless translation between text and tokens.
• Common tokenization approach mentioned: Byte Pair Encoding (BPE), which merges text chunks into tokens progressively.
• The model views language as a sequence of tokens; it predicts the next token given the previous tokens, training via likelihood maximization.
• Key terminology: vocabulary size (order of tens of thousands of tokens), context length (typically 2k–100k tokens for modern models), and the turning point that larger context lengths enable longer-range dependencies.

00:08:00 Subtopic: Model scale, context, and sample computation

• The slide compares models like GPT-3 (175B parameters) with alternatives such as Meta’s LLaMA-65B. LLaMA can be more powerful per parameter due to longer training.
• Important observation: power cannot be judged solely by parameter count; total training tokens and training duration matter a lot. For example, LLaMA may train on about 1.4 trillion tokens vs. GPT-3’s ~300 billion.
• The key takeaway: model power emerges from training duration and data exposure as much as, if not more than, raw parameter count.
• Practical hyperparameters listed (in rough terms): vocabulary around a few 10k; context length up to 100k; model sizes around tens of billions of parameters; training on thousands of GPUs; training time on the order of weeks; cost in the millions of dollars for large runs.

00:12:00 Subtopic: How pretraining works — batching, context, and loss

• Data are laid out into B by T matrices (B = batch size, T = maximum context length). Documents are packed into rows and delimited by end-of-text tokens to signal new documents.
• For a given token position (cell), the model uses all prior tokens in the context to predict the next token; the model assigns a distribution over the vocabulary (size ~50k tokens in examples) to predict the next token.
• Training objective is essentially cross-entropy over the next-token prediction; weights are updated to increase the probability of correct next tokens.
• A simplified Shakespeare training example illustrates progression from random outputs to coherent language as training proceeds, with loss decreasing over time and the model producing more sensible text.

00:15:00 Subtopic: Pretraining outcomes — representations and why pretraining matters

• A key empirical insight: language models trained with massive data learn strong, general representations that transfer to many downstream tasks with little to no task-specific data—a phenomenon that enables efficient fine-tuning later on.
• The shift from “finetune on tasks” to “prompt and fine-tune minimally” is explained: early models (GPT-1, GPT-2) revealed that prompting could yield strong results without heavy fine-tuning, and this began the era of prompt-based capabilities.
• The audience is reminded to keep in mind that a base model is not yet a GPT assistant; it tends to complete documents rather than directly answer questions. Making a base model into an assistant requires additional steps (SFT and/or RLHF).

00:20:00 Subtopic: Base models vs assistant models; availability and licensing

• Base models (like the LLaMA family) are not assistants by themselves; they are document completers. To obtain an assistant behavior, you need SFT and/or RLHF.
• GPT-4 base model was not released; GPT-4 you interact with via API is an assistant model, not a base model.
• Example of available paths: GPT-3-based models via API (e.g., names like Devanshi); GPT-2-based models with weights available on GitHub; LLaMA as a non-commercially licensed base model.
• Some base models can be prompted to behave more like assistants (via carefully crafted few-shot prompts) but this is less reliable for long-term or complex tasks.

00:25:00 Subtopic: Supervised Finetuning (SFT) — data and objective

• SFT uses a dataset of high-quality prompt–response pairs created by human contractors. The data item structure is: a task prompt and an ideal response produced by humans following labeling instructions (helpful, truthful, harmless, etc.).
• The SFT phase continues language modeling training but uses this high-quality, low-quantity data rather than raw internet text. The goal is to produce an actual assistant: a model that can respond helpfully to prompts.
• A concrete example: a contractor writes a prompt like “write a short introduction about the relevance of the term monopsony,” and the contractor also provides an ideal response.

00:28:00 Subtopic: Reward Modeling (RM) — ranking completions

• RM introduces a data collection stage where the same prompt is used to generate multiple completions from the SFT model.
• Human raters rank these completions by quality. The model learns to predict a reward (a scalar score) for a given completion given the prompt and the model’s output.
• The RM dataset pairs the prompt with completions and associated human rankings; the training objective is to teach the model to assign higher rewards to better completions.
• The RM model is not an assistant by itself; it provides the scoring signal used for RLHF.

00:32:00 Subtopic: Reinforcement Learning from Human Feedback (RLHF) — policy optimization

• With a fixed reward model, RLHF trains the assistant by generating completions, appending the reward, and shaping the loss to maximize rewards (i.e., to produce higher-reward tokens in the future).
• The RLHF pipeline alternates between prompting the model to generate candidate completions and using the reward model to score those candidates; the language modeling objective is weighted by the achieved rewards.
• Result: an assistant model that aligns more closely with human preferences than SFT alone. Not all models use RLHF; some are SFT-only, some are base models, and some are RLHF models (e.g., ChatGPT-type systems).

00:36:00 Subtopic: Why RLHF? Intuition and limitations

• RLHF tends to yield higher human preference, as shown in early experiments (InstructGPT, PPO-based RLHF). Humans tend to prefer outputs from RLHF models over SFT models or base model prompts.
• A potential explanation: asymmetry between evaluating (which is easier) versus generating (which is computationally heavier per token) makes learning from human preferences via comparisons more effective than direct labeling of correct outputs.
• RLHF models can lose some entropy (become more peaky), reducing diversity in outputs. In some tasks, base models (which retain higher entropy) are better when you want a large set of diverse outputs.
• The speaker provides a practical heuristic: use base models for tasks requiring many diverse outputs (e.g., generating many Pokemon-like names), but prefer RLHF models when accuracy and alignment with human preferences matter more.

00:40:00 Subtopic: Benchmarks and availability of assistant models

• Berkeley researchers ranked available assistant models, providing Elo-style ratings (a relative performance metric).
• Notable top performers: GPT-4 is typically viewed as the strongest, followed by Claude, GPT-3.5, and then various open models like Vicuna, Koala, etc. The speaker notes most top models at the time are RLHF-based; many other models are SFT-based.
• The landscape is dynamic; model availability (weights, licensing) varies by model and by vendor.

00:44:00 Subtopic: How to apply GPT assistants to real problems — a concrete example

• The speaker uses a blog-writing task as a microcosm of problem solving: compare populations of California and Alaska and integrate the results into a sentence.
• He contrasts human internal monologue (inner reasoning, tool use, checks) with how a GPT model processes the same task: token-by-token generation with no explicit inner monologue. The model’s internal chain-of-thought is not accessible; what it does is predict the next token based on prior tokens.
• Main cognitive advantages of humans vs LLMs: humans can perform multi-step reasoning across several steps with internal deliberation; models are constrained to per-token computation and contextual memory.
• Modern transformers have large context windows (working memory) and can access a lot of factual knowledge encoded in weights, but they do not possess a true internal planning monologue unless explicitly prompted to reveal or simulate it.

00:48:00 Subtopic: Prompting as cognitive compensation; “tokens to think” and step-by-step reasoning

• Prompting is framed as a way to compensate for the cognitive differences between human reasoning and token-based generation. In practice, prompting techniques like few-shot prompts and asking for step-by-step reasoning help, especially on reasoning tasks.
• The speaker emphasizes that to elicit useful reasoning, you should prompt for a chain-of-thought or step-by-step reasoning; this tends to slow the model’s per-token work and can lead to better outcomes.
• Methods include self-consistency (sampling multiple completions and selecting the best) and asking the model to show its work, which helps with reliability.
• A notable idea: the model can sometimes recognize that it has not met an assignment and will admit it if prompted appropriately (e.g., asking, “Did you meet the assignment?”). This is a capability that improves with model size and prompting strategy.

00:52:00 Subtopic: Tools, memory, and retrieval for augmentation

• To overcome fixed context limits, practitioners can augment LLMs with external tools: calculators, code interpreters, and web search.
• Retrieval-augmented models bridge the gap between the model’s working memory and external sources. The context window is the model’s working memory, but you can preload it with relevant documents.
• Practical tooling: LlamaIndex (one data connector to multiple data types) can index documents, create embeddings, store them in a vector store, and retrieve relevant chunks to insert into prompts at test time.
• The general workflow: split relevant docs into chunks, embed them, store in a vector store, query the store for relevant chunks, and append them to the prompt for generation.

01:00:00 Subtopic: Constraint prompting and output formatting

• Constraint prompting is about enforcing a specific template or output format (e.g., enforcing JSON output). Microsoft’s Guidance is cited as an example in which formatting constraints are used to clamp the model’s output and ensure structured results.
• This approach helps enforce a predictable interface and can improve downstream usability.

01:04:00 Subtopic: Fine-tuning and PEFT (LoRA) — practical considerations

• Fine-tuning changes the model’s weights (as opposed to prompting alone). It is enabled by newer, more accessible methods such as Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA (Low-Rank Adaptation), which update only small, trainable components of the model while keeping the rest fixed.
• PEFT makes fine-tuning cheaper, allows very low-precision inference for updated components, and reduces the data and compute burden.
• Caveat: while SFT (continued language modeling on high-quality data) is relatively straightforward, RLHF remains more of a research undertaking, often unstable and less beginner-friendly.
• The speaker advises a pragmatic two-stage approach: first aim for maximum performance (likely via a strong base like GPT-4 with prompt engineering and possibly SFT), then explore fine-tuning if needed. RLHF is optional but can yield improvements if one can manage the complexity.

01:10:00 Subtopic: Practical guidance — where to start and how to iterate

• Prioritize performance: GPT-4 currently offers the strongest capabilities. Use detailed prompts fully describing the task, include relevant context, and incorporate prompt-engineering techniques.
• Use few-shot examples to demonstrate the desired behavior; combine prompts with tools and chains (reflections, multiple samples, and a plan for selecting the best results).
• If prompt engineering saturates, consider finetuning a model (preferably with LoRA-style parameter-efficient methods) as a next step, but be prepared for slower iterations and the need for human data pipelines.
• If you pursue RLHF, be aware that it’s an expert, research-heavy path with notable challenges and rapid changes; most teams should treat it as a higher-risk, longer-term investment.
• Cost optimization: explore lower-capacity models, shorter prompts, and more efficient inference to reduce costs while still achieving acceptable performance.

01:16:00 Subtopic: Use-case cautions and safety considerations

• LLMs have known limitations that matter in real applications: bias, hallucinations (fabrications), reasoning errors, knowledge cutoffs (e.g., post-2021), and vulnerability to prompt injection and other attacks.
• The recommendation is to use LLMs primarily in low-stakes settings, with human oversight as a co-pilot rather than fully autonomous agents.
• The speaker emphasizes balance: harness the power and the ecosystem while avoiding over-reliance in critical tasks.

01:22:00 Subtopic: Closing thoughts and GPT-4 ecosystem momentum

• GPT-4 is celebrated as a remarkable artifact with broad knowledge and capabilities (math, code, etc.), and there is a thriving ecosystem building around it.
• A brief demonstration: the speaker shows a small Python snippet that prompts GPT-4 to generate an inspirational message for Microsoft Build 2023, highlighting practical programmatic usage.
• The closing note: “GPT-4 is an amazing artifact... the power is accessible at your fingertips.”

01:25:00 Notable quotes and closing lines

• Key inspirational quote used as demonstration: GPT-4’s generated message to inspire the audience; the speaker notes that the demonstration confirms the practical ease of calling GPT-4 from code.

01:28:00 Recap: The practical recipe and ecosystem

• The overall message: the state of GPT involves a scalable, staged pipeline (pretraining → SFT → RM → RLHF) with increasing alignment complexity; there is a robust ecosystem of tools (retrieval, constraints, fine-tuning, and deployment patterns) that practitioners can leverage to build capable applications while staying mindful of limitations and safety concerns.

01:30:00 Resources and closing

• The talk references a collection of papers, datasets, frameworks, and tools throughout (e.g., LLaMA, GPT-3/4 family, ERM/RM, Tree of Thought, React, AlphaGo, AutoGPT, LlamaIndex, Guidance).
• The takeaway is to experiment with prompting, chain-of-thought strategies, tool use, and retrieval-augmented setups to maximize GPT-based systems’ utility in real-world applications.

01:32:00 End of deep dive

• The session concludes with a recap and a focus on practical steps for developers to leverage GPT-4 and its ecosystem in Build 2023 context.

01:34:00 Resources Mentioned (summarized)

• Datasets and models: CommonCrawl, C4, GitHub, Wikipedia, Books, Archives, LLaMA, GPT-3, GPT-4, GPT-2, Claude, Vicuna, Koala, etc.
• Papers and concepts: InstructGPT, PPO (policy optimization), Tree of Thought, React paper, AlphaGo, AutoGPT, LoRA (LoRA), Guidance (Microsoft), LlamaIndex, retrieval-augmented generation, vector stores, embeddings.
• Tools and platforms: OpenAI API, Python glue code, vectors, and prompts; Microsoft Build ecosystem references.

01:40:00 Closing emphasis

• The talk acknowledges the rapid evolution of the field and encourages practitioners to adopt a practical, cautious, and iterative approach to using GPT assistants in applications, while staying mindful of limitations and safety concerns.

Blockquotes (notable lines) "these transformers need tokens to think," "Did you meet the assignment? Actually GPT-4 knows very well that it did not meet the assignment." "Let things step-by-step." "Tree of Thought." "These are the assistant models that are probably available to you at this point." "GPT-4 is an amazing artifact."