Let's build GPT: from scratch, in code, spelled out.

A hands-on walkthrough that implements a GPT-style language model from scratch.

Core Architecture

• GPT stands for Generatively Pre-trained Transformer; the Transformer neural net performs the heavy lifting (0:00-12:02).
• Decoder-only blocks with causal masking let each token attend only to previous tokens (1:12:25-1:24:28).

Data and Tokenization

• Tiny Shakespeare (1 MB) is tokenized at character level into 65 unique integers (12:05-24:06).
• 90/10 train/val split; data is served in B=64 batches of T=256-token blocks (12:05-24:06).

Self-Attention Mechanism

• Each token emits query, key and value vectors; affinities are dot products scaled by sqrt(head_size) (1:00:22-1:12:24).
• Lower-triangular masking plus softmax yields data-dependent weighted aggregation (48:17-1:00:20).

Transformer Block

• Multi-head attention is followed by an MLP, both wrapped with residual connections and pre-layer-norm (1:24:28-1:36:31).
• Dropout (0.2) regularizes the 6-layer, 384-dim model (1:36:32-1:48:34).

Training and Scaling

• Adam (3e-4) on cross-entropy reduces loss from ~4.7 to 1.48 after 15 min on A100 (36:12-48:16).
• Pre-training on internet text followed by RLHF produces ChatGPT-style assistants (1:48:35-1:56:19).

Speaker: Andrej Karpathy

00:00:00 Introduction and Context: ChatGPT, Transformers, Shakespeare, and the Goal

• Karpathy frames the video around building a Generatively Pretrained Transformer (GPT) and relates it to ChatGPT, GPT-2/3, and the broader AI landscape.
• Emphasis on following the Attention is All You Need paper and the practical lineage via GPT-2/3, with a focus on understanding autoregressive language modeling, tensors, PyTorch nn basics, and the educational aim of a character-level transformer trained on a small dataset.
• The plan is to build intuition and implement a transformer-based language model (character level) on a toy dataset (tiny Shakespeare) to demonstrate how the system operates under the hood.

"Transformer is the neuron nut that actually does all the heavy lifting under the hood" (paraphrase of Karpathy’s framing)

• Notable resources mentioned early: Colab for the video, GitHub nanoGPT repo, OpenAI GPT-3 paper, ChatGPT blog, and general Colab/GPUs guidance.
• Key tradeoff highlighted: we won’t replicate production-grade ChatGPT (which is trained on vast internet data with extensive pre-training, fine-tuning, and RLHF) but we will explore the fundamentals via a smaller dataset and a simple GPT-style model.

00:07:52 Data, Tokenization, and Dataset Choice

• Data source: Tiny Shakespeare (concatenation of Shakespeare works, ~1 MB). Used to model character sequences and learn character-level dependencies.
• Concept: modeling how characters follow each other; given a chunk, predict the next character. This reveals patterns in the language style of Shakespeare without requiring word-level modeling.
• Tokenization approach chosen for this tutorial: character-level encoding. Each character maps to an integer ID; a reverse map exists to decode.
• Why not subword/tokenizers here? While subword models (e.g., BPE, SentencePiece, Byte-Pair Encoding, TikToken) are common in practice and reduce sequence length, the lecture emphasizes a simple, transparent character-level tokenizer to illustrate core ideas clearly.

Key term: tokens = units the model uses to represent text; for this setup, tokens are characters.

00:09:28 Tokenization, Train/Val Split

• Build two dictionaries: char2i and i2char for encoding/decoding between characters and integers.
• Vocabulary size is 65 characters (including space and punctuation).
• Data preprocessing steps: read text, extract the set of unique characters, sort them for deterministic ordering, and define vocab_size = 65.
• Train/validation split: use the first 90% for training and the last 10% for validation to measure overfitting and generalization.

00:14:27 Data Loader: Batches of Chunks of Data

• Training on chunks instead of the entire dataset: sample random chunks from the training data with a fixed block size (context length).
• Example of chunking: if block_size = 8, you feed in 8 characters as input (X) and 8 characters as targets (Y) that are the next characters for each position in X (offset by +1).
• The design exposes multiple training examples within a single chunk: an 8-character chunk yields 8 training examples, for contexts lengths 1..8.
• The reason to include all context lengths in training is to teach the model to predict next tokens starting from various-length contexts, which helps during inference when starting with little context.

01:00:18 Positional Encoding and the Crux: Version 4 Self-Attention

• The crucial transition is to implement self-attention, which enables tokens to attend to past tokens in the sequence.
• Early versions illustrate naive context aggregation; soon, matrix-multiplication-based attention is introduced for efficient computation and differentiable weighting.
• The notes introduce important concepts:

- Attention as a communication mechanism between positions in a sequence. - Attention operates over sets; it has no inherent notion of