Building makemore Part 5: Building a WaveNet

Andrej Karpathy extends the Part 3 MLP into a deeper, hierarchical network modeled after WaveNet by grouping consecutive embeddings instead of flattening everything at once.

Core Architecture Change

• Context length grows from 3 to 8 characters, improving validation loss from 2.10 to 2.02 (19:55-20:04).
• flatten_consecutive groups every two 10-dim embeddings, turning 4x8x10 into 4x4x20 and enabling progressive fusion (30:08-40:11).

PyTorch Refactoring

• Custom Embedding, Flatten, and Sequential modules encapsulate view and indexing so the forward pass collapses to a single model(x) call (10:03-20:04).
• Matrix multiply in PyTorch automatically broadcasts over leading batch dimensions, allowing 4x5x80 inputs to produce 4x5x200 outputs without loops (20:06-30:07).

BatchNorm Bug Fix

• Running stats were incorrectly kept over 4x68 channels; changing torch.mean to dim=(0,1) tracks only 68 channels and lifts performance to 2.022 (40:12-50:14).

Efficiency & Development

• The same hierarchical structure prefigures causal dilated convolutions that reuse intermediate activations inside CUDA kernels (50:15-56:19).
• Typical workflow alternates Jupyter shape checks with repository-scale experiments and an experimental harness for hyperparameter sweeps.

Major Topic: Introduction & Goals

Speaker: Andrej Karpathy

• Karpathy continues Makemore by deepening the character-level language model from Part 4 and morphing it toward a WaveNet-like architecture. The core idea is to handle longer prior context and fuse information progressively rather than squashing everything into a single hidden layer.
• He frames WaveNet as a hierarchical, auto-regressive model that predicts the next character using a depth-wise, tree-like fusion of previous characters. Although WaveNet in the original DeepMind paper uses dilated causal convolutions for efficiency, this video focuses first on the architectural intuition and then on the iterative PyTorch implementation and debugging process.
• The video also emphasizes the development workflow: lots of reading docs, tracking tensor shapes, moving between notebooks and repository code, and the difference between prototyping in notebooks vs. training in a repo.

> Blockquote (verbatim): "hi everyone today we are continuing our implementation of makemore our favorite character level language model ... we would like to make a deeper model that progressively fuses this information to make its guess about the next character in a sequence and so we'll see that as we make this architecture more complex we're actually going to arrive at something that looks very much like a WaveNet."

SubtopicStarter Code & Data Overview

• The starter code from Part 5 closely mirrors Part 3, with minimal changes. The dataset contains many examples where three characters are the input and the fourth is the target to predict; specifically, 182,000 examples of (three chars → next char).
• The architecture so far uses building-block modules (e.g., Linear, BatchNorm-like layer) to emulate PyTorch-style layer composition. The goal is to treat these modules as Lego bricks that can be stacked into a graph.
• The embed table C, a list of layers (linear, BatchNorm-like, etc.), and an output layer are initialized. Parameters require_grad is set to True for optimization, and the loss is defined (but acknowledged to be rough and in need of improvement).

> Blockquote (verbatim): "we have embedding table C and then we have a list of players and uh it's a linear feeds to Bachelor feeds to 10h and then a linear output layer and its weights are scaled down so we are not confidently wrong at the initialization."

SubtopicFixing the Learning Rate Plot & Training Diagnostics

• The code tweaks involve improving the learning-rate plot to get a cleaner view of optimization progress. The Decay schedule is shown to dramatically improve convergence by damping oscillations and helping settle near a local minimum.
• The plot becomes more interpretable after averaging or reshaping loss data (the video narrates creating a 2D view of long loss vectors to compute mean per-row, improving the visualization).
• The takeaway is that a cleaner learning-rate schedule and better loss visualization help diagnose training dynamics early, especially when the loss surface is highly non-convex or when batch sizes are small.

> Blockquote (verbatim): "the learning rate Decay subtracted a ton of energy out of the system and allowed us to settle into sort of the local minimum in this optimization."

SubtopicPyTorchifying Our Code: Layers, Containers, torch.nn, & Bugs

• The codebase is reorganized to be more PyTorch-like: move embedding and flattening logic into modules; replace ad-hoc operations with torch.nn-compatible layers.
• Discussion of BatchNorm1d (bath normalization) reveals: running_mean and running_var are updated outside backprop; the layer behaves differently in train vs eval modes; ensure the model is in the correct mode when evaluating or sampling to prevent garbage stats or NaNs in small-batch scenarios.
• The embedding table and the first layer operate as part of a manifest of layers; spatial operations (e.g., view/reshape) are used to temporarily reframe data shapes for efficient computation without copying memory.
• The forward pass becomes cleaner once the embedding, flattening, and initial blocks are encapsulated as modules and placed inside a torch.nn.Sequential container.

> Blockquote (verbatim): "we can create modules for both of these operations—the embedding operation and the flattening operation ... let’s construct those modules and now I can take out this C; C is the embeddings weight and it's inside layers so this should just work."

SubtopicImplementing WaveNet: Conceptual Overview & Dataset Context

• WaveNet is introduced as a hierarchical modeling approach (like a tree), where information from previous characters is fused progressively through multiple layers, rather than collapsing everything into a single large hidden representation at the input stage.
• The dataset context size is increased to 8, so the model will now condition on eight previous characters to predict the ninth. This expands the receptive field and allows deeper, more nuanced context modeling.
• The speaker demonstrates that naive scaling is not sufficient; a proper hierarchical fusion requires reorganizing the data flow so that the network sees short n-grams at early layers, then longer and longer contexts as you go deeper.

> Blockquote (verbatim): "this is the architecture and we're going to implement it in the course of this video so let's get started. so the starter code for part five is very similar to where we ended up in part three."

SubtopicDataset Context Expansion & Baseline Running

• The context window is bumped to 8; the baseline code is re-run on block_size 8 to observe how the larger context affects loss and sample quality.
• Initial results show a respectable validation loss around 2.0, with samples that look like names but not perfectly realistic; there is still room to push performance via architecture scaling and hyper-parameter tuning.

SubtopicImplementing WaveNet: Architecture Details & Debugging Compass

• The teacher describes the progressive fusion idea visually: two characters → two-character representations, then two bigrams → four-grams, and so on; the network depth grows and information is fused gradually.
• A rough worry is that a straightforward increase in block size without architectural changes yields diminishing returns or renders the network brittle; thus, the WaveNet-inspired hierarchical structure is pursued.
• The experiment includes a live debugging walkthrough where shapes are checked at every layer to ensure dimensional correctness, with a focus on how embedding, flattening, and linear transforms interact with the new block design.

> Blockquote (verbatim): "we now have to understand the shapes along the way so we can think about how to fuse the eight characters into groups of two and then two bigrams into four-grams and so on."

SubtopicTraining First Pass on WaveNet-like Model

• After reorganizing the architecture, the first training pass is run. Initial results show that with the larger context, learning begins to move, but there are still bugs to fix (notably BatchNorm handling and shape consistency).
• A critical lesson is that simply changing code paths can reveal hidden state management bugs (e.g., BatchNorm statistics and training/eval mode state) that can derail training if not properly synchronized.

SubtopicBug Fixes: BatchNorm1d & Model State

• A bug fix is applied to BatchNorm1d: the running statistics and per-channel statistics must be aggregated correctly when the input has more than a single sample (e.g., multiple batch dimensions or a 3D input). The initial version used an inappropriate reduction (over the wrong dimensions) causing unstable statistics due to the multi-dimensional batch.
• After the bug fix, a training run shows improved stability and a slight improvement in validation loss (2.022 vs 2.029), indicating more stable estimates of mean/variance when the data dimensions increase.

> Blockquote (verbatim): "the running mean shape is now one by one by 68. so we're only maintaining 68 means for every one of our channels and we're treating both the zeroth and the First Dimension as a batch Dimension which is exactly what we want."

SubtopicRe-training WaveNet with Bug Fix & Scaling Up

• The project uses iterative scaling: increase embedding dimension (from 10 to 24) and widen hidden layers, which inflates parameter count (e.g., to ~76k params). Training time increases, but the model begins to show meaningful improvements in validation loss (approaching ~1.99).
• The demonstration emphasizes the trade-off between model capacity and practical training time. With larger models, one should also consider robust experimental harnesses to systematically explore hyperparameters rather than ad-hoc tuning.

SubtopicLayer Reorganization, 3D Tensors & Shape Intuition

• The discussion deepens around the practicalities of multi-dimensional tensors in PyTorch and how to manipulate them without copying data. The embedding output is 4x8x10 (batch x context x embedding_dim).
• The Flatten operation is reimagined to support grouping consecutive items (n) instead of flattening everything into a single dimension. This enables the construction of a hierarchical architecture where two adjacent tokens are fused first, then four-grams, etc.
• A crucial design decision is to implement a FlattenConsecutive layer that reshapes the 4x8x10 tensor into 4x4x20 (i.e., 4 examples, 4 groups per example, each group containing the concatenated 2x10 embeddings).

> Blockquote (verbatim): "we want to take two consecutive elements and fuse them so we'd like to reshape into 4x4x20 rather than flattening into 4x80. view can do this; we can simply view the last two dimensions as 4x4x20."

SubtopicConcrete Implementation: FlattenConsecutive, Embedding, & Sequential Container

• The implementation introduces custom modules: Embedding (lookup-based on the vocab index) and Flatten (which fuses consecutive tokens into new feature groups). These modules live inside a pytorch-like container (PiTorch) to enable a neat sequential pass.
• A custom Sequential-like container is implemented (similar to torch.nn.Sequential) to glide data through the layers with a simple forward call. This significantly simplifies the code and makes the architecture more modular and readable.
• The forward pass becomes: input integers -> Embedding -> FlattenConsecutive (block-size-n fused) -> Linear -> (more layers) -> final logits, as described in the hierarchical wave-like structure.

SubtopicRewedging the WaveNet Architecture: Three-Layer Prototype & Shapes

• The architecture is prototyped with three layers (embedding, a hierarchical fusion step, and a final projection). The block-size for the first fusion is two, leading to an input size of 20 (two 10-d embeddings) for the first linear layer.
• As the architecture deepens, subsequent layers correspond to 4-grams and beyond, with corresponding input/output shapes that align with the hierarchical design.
• The result is a three-layer network that preserves the intent of progressive fusion, but the performance is still exploratory because the hyperparameters and architecture require tuning to exceed the baseline.

SubtopicDiagnostics: Comparing Naive vs Hierarchical, & Reproducing Results

• The author notes that the hierarchical version, with the same param count as the naive baseline, yields roughly the same validation loss but reveals potential for more efficient parameter use and better inductive biases.
• The experiment demonstrates that a meaningful performance improvement is possible with a well-chosen block-size and embedding dimension, even if the first iteration doesn’t dramatically beat the baseline.

SubtopicExperimental Harness, Future Directions, & Closing Thoughts

• The video closes with reflections on the development process and the need for an experimental harness to run many hyperparameter trials, track results, and manage experiments at scale.
• The WaveNet-inspired approach is presented as a stepping stone toward more advanced ideas like dilated causal convolutions (for computational efficiency) and residual/skip connections, plus a deeper dive into torch.nn itself.
• The author hints at a future plan to explore recurrent networks (RNNs, LSTMs) and Transformers, as well as better tooling to replicate experiments and systematically beat ad-hoc baselines.

> Blockquote (verbatim): "we implemented this architecture from the WaveNet paper but we did not implement this specific forward pass of it where you have a more complicated a linear layer sort of that is this gated linear layer kind of and there's residual connections and Skip connections and so on."

SubtopicFinal Reflections: Development Process, Documentation, & Next Steps

• Karpathy emphasizes the importance of reading PyTorch docs, checking shapes, and using Jupyter for prototyping while moving to a repository for actual experiments.
• He notes the limitations of the current documentation, the heavy emphasis on low-level engineering in official docs, and the need to keep experimenting with shapes and layers to understand how PyTorch components map to neural networks.
• The video ends with a roadmap: implement actual dilated causal convolutions, explore residual/skip connections, create a robust experimental harness, and possibly move toward fully PyTorch's torch.nn use. He invites viewers to experiment with different channel counts, embedding sizes, and hyperparameters to see what helps beat the current best of ~1.99 validation loss.

SubtopicGoing Forward: What’s Next in makemore Part 6 & Beyond

• The next steps include turning the WaveNet-inspired hierarchy into a practical module with dilated causal convolutions, exploring residual/skip connections, and building an experimental harness to systematically optimize hyperparameters.
• There’s also a plan to bridge toward traditional convolutional neural network interpretations (the convolution as an efficient implementation of the tree-like fusion) and to use this to explore more scalable architectures.
• He reiterates that there are many future directions: experiment with deeper networks, try different channel counts, look at more robust loss curves, compare to simpler baselines, and ultimately beat the current best results with better hyperparameters and architecture choices.

SubtopicFinal Encouragement: Practicality & Experimentation

• Karpathy emphasizes that beating a single metric (e.g., 1.993 validation loss) requires a careful experimental harness; he suggests trying more tuning—embedding size, hidden units, and block sizes—along with better training regimes and data handling.
• The takeaway is that building such models is as much about disciplined experimentation and tooling as it is about clever architectures. The process demonstrated—read docs, prototype, test, iterate, scale—remains the core workflow.

Resources Mentioned

• makemore on GitHub: https://github.com/karpathy/makemore
• Jupyter notebook for Part 5: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part5_cnn1.ipynb
• Colab notebook: https://colab.research.google.com/drive/1CXVEmCO_7r7WYZGb5qnjfyxTvQa13g5X?usp=sharing
• Karpathy’s website: https://karpathy.ai
• Twitter: https://twitter.com/karpathy
• Discord: https://discord.gg/3zy8kqD9Cp
• WaveNet (DeepMind 2016): https://arxiv.org/abs/1609.03499
• Bengio et al. 2003 MLP Language Model: https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

Notes on Diagrams & Visuals

• The WaveNet diagram is described as a stack of dilated, causal convolution layers; the crucial idea is that time steps influence future steps in a controlled, causal manner while dilation expands the receptive field. In practice, this is implemented via blocks of 1D convolutions with increasing dilation factors, allowing very deep networks to capture long-range dependencies efficiently.
• A schematic in the talk shows a tree-like fusion where successive pairs of characters are combined into bigger units (two characters → four-grams → eight-grams, etc.). This clarifies why context is integrated gradually rather than in a single giant input pass.

Notable Terms & Concepts (First Mentioned Here)

• WaveNet: A deep autoregressive model originally designed for raw audio, employing dilated causal convolutions to model long-range dependencies efficiently.
• Causal Convolution: A convolution where the output at time t only depends on inputs at times <= t, ensuring no information leaks from the future.
• Dilated Convolution: A convolution with gaps (dilation) between filter elements, increasing receptive field without increasing filter size.
• Embedding: A learnable dense vector representation for discrete tokens (characters, words) mapping from integers to real-valued vectors.
• BatchNorm1d: Batch normalization for 1D inputs; uses running statistics during evaluation and batch statistics during training, which can cause subtle bugs when batch dimensions are nonstandard.
• Flatten / FlattenConsecutive: Custom operation to concatenate or restructure groups of tokens (e.g., two consecutive characters) into a single feature vector to enable hierarchical processing.
• PyTorch / PiTorch: The Python library for deep learning; Karpathy’s teaching uses a PyTorch-like custom library called PiTorch to illustrate module composition while teaching.
• Logger vs. Loss: The goal is to monitor a loss function (cross-entropy on next-character prediction) across training and validation sets.
• Receptive Field: The amount of input context that can influence a given output; increasing this via depth and dilation allows modeling longer contexts.
• Auto-regressive: A model that generates the next element conditioned on previous elements.

Key Takeaways

• ⭐ HIGH PRIORITY: A WaveNet-like architecture can be built by progressively fusing context rather than collapsing it all at the input stage.
• Increasing context length (block size from 3 to 8) yields measurable gains in validation performance, showing the value of longer-range dependencies.
• Cleaning up the codebase to PyTorch-like patterns (Embedding, FlattenConsecutive, Sequential) substantially improves readability, debuggability, and maintainability.
• BatchNorm requires careful handling of batch dimensions and training/eval modes; mismanaging statistics can derail training, especially with multi-dimensional inputs.
• FlattenConsecutive enables a natural hierarchical representation (two-gram, four-gram, etc.) by reshaping inputs rather than flattening them arbitrarily; this aligns with the progressive fusion intuition.
• A simple three-layer, hierarchically fused model can achieve similar performance to a naive larger single-layer model, illustrating the potential efficiency gains from smarter architecture design.
• A robust experimental harness is essential for scalable, repeatable hyperparameter tuning; without it, it’s easy to miss substantial improvements or misinterpret results.
• Dilated causal convolutions are introduced as a future efficiency improvement to replace or complement the explicit hierarchical fusion, enabling long-range dependencies with fewer layers.
• The development process combines notebook prototyping and repository-based experimentation; documenting shapes and data flow is crucial to maintain correctness across iterations.
• The path forward includes exploring residual/skip connections, more scalable experimental tooling, and transitions toward using PyTorch’s torch.nn in a production-like workflow.

Action Items

• [ ] Study WaveNet’s core idea: hierarchical fusion of context (two chars → four-grams → eight-grams, etc.).
• [ ] Implement FlattenConsecutive with a configurable n (e.g., 2, then 4) to fuse tokens progressively.
• [ ] Build a three-layer WaveNet-like model with Embedding, FlattenConsecutive, and a sequence of Linear layers, ensuring shapes align at each stage.
• [ ] Debug BatchNorm1d behavior on multi-dimensional inputs; fix statistics reduction to avoid NaNs and unstable updates.
• [ ] Increase context length (block size) to 8 and record validation loss trend; compare with baseline to quantify gains.
• [ ] Experiment with embedding size and hidden unit counts (e.g., 24 embeddings, 68 hidden units) while keeping parameter counts reasonable.
• [ ] Develop an experimental harness to run multiple hyperparameter trials and compare results systematically.
• [ ] Explore dilated causal convolutions as a next step to replace or augment the hierarchical fusion for efficiency.
• [ ] Plan a path toward integrating PyTorch’s torch.nn more directly (switch from PiTorch to native torch.nn) to benefit from ecosystem tooling.
• [ ] Document all tensor shapes and module interfaces in the repository to prevent shape mismatches during future iterations.

Topics

• WaveNet
• Character-Level Language Modeling
• Dilated Convolutions
• Auto-regressive Models
• PyTorch / Deep Learning Tooling
• Hierarchical Feature Fusion
• Experimental Harness & Hyperparameter Tuning