Building makemore Part 3: Activations & Gradients, BatchNorm

A deep dive into MLP internals that shows how bad initialization and saturated activations break gradient flow and how BatchNorm fixes it.

Initial Loss Problem

• Expected uniform loss over 27 characters is ~3.29; observed loss of 27 signals over-confident wrong logits (0:00-4:19).
• Scaling final-layer weights by 0.01 brings initial loss to the correct range without changing the network capacity.

Tanh Saturation & Vanishing Gradients

• Pre-activations spanning [-15,15] push tanh outputs to ±1, zeroing local gradients (1-t^{2}) and creating dead neurons (12:03-24:05).
• Deeper nets amplify the problem; small nets may still limp along.

Kaiming Initialization

• Scale weights by gain/√fan_in (gain=5/3 for tanh) so each layer's activations remain unit Gaussian (24:07-36:08).

Batch Normalization

• Normalizes each batch to zero mean/unit variance, then learns per-channel gain and bias; running stats enable inference on single examples (36:09-1:12:19).
• Removes the need for perfect manual scaling and acts as a regularizer via batch coupling.

Diagnostics & Update Ratios

• Track activation histograms, gradient histograms, and especially update-to-data ratio (target ~10^{-3}) to verify healthy training (1:24:22-1:55:57).

Practical Refactoring

• PyTorch modules, torch.no_grad for running stats, bias=False before BatchNorm, and epsilon for numerical stability complete the modern recipe.

00:00:00 Major Topic: Introduction and Objectives

• We dive into the internals of MLPs with multiple layers, scrutinizing the forward pass activations and backward pass gradients.
• Why training deep nets can be fragile and how normalization and careful initialization help stabilize learning.
• The first modern stabilization technique introduced: Batch Normalization. Residual connections and Adam are noted as important ideas to cover later.
• The practical goal is to learn diagnostic tools and visualizations that reveal the health of a deep network so you never need to re-watch the video.

00:01:22 Starter Code

• The starter code uses the Makemore setup with a small multi layer perceptron for character level language modeling.
• Data handling mirrors the previous lecture: vocabulary of lowercase letters plus a dot token; train, dev, test splits; batch size 32; sequence of 200k steps; roughly 11k trainable parameters.
• Code niceties added: removal of magic numbers, externalized hyperparameters (embedding dim, hidden units) so changes are easier to manage.
• Evaluation flow is kept, with a generic eval function that can compute loss for train, dev, or test splits.
• A torch.no_grad decorator is used to mark sections that do not need gradients, improving efficiency for inference-like blocks.

00:04:19 Fixing the Initial Loss

• Observation: The very first iteration loss is around 27, which is way higher than expected. The loss for a 27-way next-character problem should be close to the negative log probability of a uniform prior, about log 27 ≈ 3.29, not 27.
• Intuition: If the model outputs a near uniform distribution across 27 classes, the initial loss should be around 3.29. If some logits are extremely confident on wrong classes, the loss spikes because the model is confidently but incorrectly predicting.
• Demonstration with a small example: with 4 characters, if logits are near zero, the softmax distribution is near uniform and the loss is about 1.38; if logits are pushed to extremes, the loss becomes very high or very low depending on alignment with the true class.
• Practical fixes explored:

- Disable the bias term in the output layer B2 initialization to reduce initial bias pushing logits away from zero. - Scale down W2 by a factor (for example 0.1 or 0.01) so that the initial logits are closer to zero, producing a more sensible starting loss (around the 3.29 target). - Small entropy in the initial biases (nonzero but tiny) helps symmetry breaking without spoiling the zero-centered logits.

• Result: With a small bias zeroing and downscaled weights, the hockey-stick shape of the loss is reduced and optimization begins with more productive gradient signals.

00:12:59 Fixing the Saturated Tanh

• The hidden activations H show many values near ±1 after applying tanh, which indicates saturation where gradients collapse and learning stalls.
• Visual inspection via histograms reveals that preactivations feeding tanh span a wide range, pushing many H values into saturation zones.
• Backpropagation insight: for tanh, the gradient is proportional to 1 - t^2; when t is near ±1, this factor is near zero, effectively killing gradients and preventing learning for those neurons.
• Dead neuron intuition: if a neuron consistently sits in the saturated region, its gradient vanishes and it may stop learning entirely (dead neuron). ReLU has similar concerns on its flat region; dead ReLU occurs when a neuron is never activated.
• Remedies discussed:

- Normalize initial activations so preactivations are not too large, reducing saturation risk. - Use smaller initial weights or a smaller gain to keep H in a healthy, non-saturated range. - Modify initialization strategy to spread activations more evenly across the non-saturated region.

• Practical takeaway: initialization and early scale of weights dramatically affect gradient flow and activation distribution; small changes early on can have outsized effects on training dynamics.

00:27:53 Calculating the Init Scale: Kaiming Init

• The problem of initialization is analyzed through a simple mean-variance lens: if inputs have unit std dev, the preactivation Y = XW has a widened distribution unless W is scaled properly.
• To preserve a roughly unit standard deviation of activations through layers, weights should be scaled by 1 / sqrt(fan_in), where fan_in is the number of input connections to a layer.
• With a ReLU or tanh type nonlinearity, the effective variance is altered; a common fix is to include a gain factor. For ReLU-like nonlinearities, the He/Kaiming gain ≈ sqrt(2) is used to compensate for the half-output activation, while for linear or identity, the gain is 1.
• The PyTorch init utility torch.nn.init.kaiming_normal/kaiming_uniform implements this, with a gain parameter tuned for the nonlinear activation. The recommended gain for tanh is around 5/3 in this context; the exact value can vary by architecture and depth.
• Practical steps shown:

- Use a nonzero bias with a small standard deviation, but avoid exactly zero weights in some cases to preserve symmetry-breaking signals. - You can experiment with scaling W by ~0.2 or 0.3 depending on fan_in and the layer nonlinearity; results show improved stability and slightly better losses compared to naive zero-mean initialization.

• Key insight: initialization is more forgiving in shallow networks but becomes critical as depth grows; modern networks use normalization and better optimizers to reduce sensitivity to exact initialization values.

00:40:40 Batch Normalization

• Batch Normalization (BN) aims to standardize activations to have zero mean and unit variance for each feature across the batch, then apply a learned scale (gamma) and shift (beta). This helps keep activations in a healthy range, preventing saturation and explosion.
• BN introduces learnable gamma and beta parameters per feature; these allow the network to recover original representational power if normalization is too restrictive.
• Practical fold: after training, gamma and beta can be folded (merged) into the preceding Linear layer's weights and biases, effectively removing BN during inference without changing the forward pass behavior of the trained network.
• The forward pass uses the per-batch statistics to normalize, then applies the learned scale and offset. For inference, running estimates of mean and variance are used rather than batch statistics.
• The explanation includes the concept of using either a per-batch statistic or running statistics and the trade-offs between exact recalculation on every batch and using a running estimate for efficiency and stability.
• A notable additional point: BN also acts as a regularizer, introducing slight noise via batch statistics which can help prevent overfitting and improve generalization by effectively augmenting the data during training.
• Practical points:

- BN introduces epsilon to prevent division by zero when variance is tiny. - It is common to remove biases in layers that feed into BN because BN's normalization subtracts the bias anyway. - BN can interact with very small batch sizes; higher momentum or running estimates can help stabilize the running statistics.

01:03:07 Batch Normalization: Summary

• BN turns activations into approximately zero-mean, unit-variance signals, enabling deeper networks to train stably.
• BN adds learned gamma and beta to adapt the normalized activations.
• BN has two buffers for running statistics (mean and variance) used at inference time; these buffers are not updated via backpropagation but via a running average during training.
• BN has side effects: the forward pass becomes batch-dependent, coupling individual examples with other examples in the same batch; this can act as a form of regularization but also introduces potential brittleness and debugging complexities.
• In practice, many practitioners sprinkle BN after linear or convolutional layers, before nonlinearities, to keep activations in a healthy regime across the network depth.

01:04:50 Real Example: ResNet50 Walkthrough

• A practical, widely used architecture is introduced to illustrate batch normalization in real networks.
• ResNet uses a repeated motif: convolution -> batch normalization -> nonlinearity, with residual connections that sum a shortcut path to the main path.
• The specific block type shown is the bottleneck block, often used in ResNet50, which stacks layers in a deep, modular, repeatable pattern.
• In PyTorch style, conv layers are used with bias disabled when BN follows, since BN provides the necessary offset and scale.
• The example emphasizes that BN layers are integral to stabilizing training in deep residual architectures, enabling very deep nets to converge.

01:14:10 Summary of the Lecture

• Recap of activations and gradients in forward/backward passes, with emphasis on how saturations and extreme activations hurt gradient flow.
• The importance of initialization scale, BN stabilization, and mindful architectural patterns (residuals, normalization) to enable deep nets to train reliably.
• The plan for future topics includes deeper exploration of residual architectures, more normalization variants (LayerNorm, GroupNorm), and optimizers like Adam.

01:18:35 Just Kidding: Part 2 PyTorch-ifying the Code

• A teaser noting that the forthcoming content will push toward PyTorch-like module structures and APIs for cleaner experimentation.
• The idea is to structure code into modular blocks akin to PyTorch's nn.Linear, nn.BatchNorm1d, and other building blocks so that networks become assembly-like from Lego bricks.
• The plan is to build neural networks by stacking modules and to implement forward/backward passes in a clean, PyTorch-like design.

01:26:51 Viz #1: Forward Pass Activations Statistics

• A visualization exercise shows histograms of forward pass activations across layers, focusing on how activation distributions evolve through the network.
• The plots track mean, standard deviation, and the percentage of activations saturating toward the tails; these metrics reveal how saturated each layer is and indicate where to adjust initialization or normalization.
• The saturation percentage is defined as the fraction of activations with absolute value above a high threshold (eg 0.97), indicating the headroom left for gradient flow.
• Observations: with appropriate gain settings, the distributions stabilize with modest saturation; otherwise, deeper stacks show increasing saturation and gradient attenuation.

01:30:54 Viz #2: Backward Pass Gradient Statistics

• Similar visualizations track gradient magnitudes across layers to understand how gradients flow during backpropagation.
• The aim is to avoid both vanishing gradients (too small) and exploding gradients (too large) as you move through the network depth.
• The analysis helps diagnose if the nonlinearities and weight scales are enabling healthy gradient flow, especially in deep stacks.

01:32:07 The Fully Linear Case of No Nonlinearities

• A thought experiment checks what happens if there are no nonlinearities, leaving only linear transformations.
• In such a case, stacking multiple linear layers collapses to a single linear transformation in terms of expressiveness, but training dynamics still differ due to backprop behavior.
• The exercise shows the importance of nonlinearities for representation power, and it also demonstrates how activation distributions and gradient signals behave differently with and without nonlinearities.

01:36:15 Viz #3: Parameter Activation and Gradient Statistics

• A more detailed look at the statistics of the learnable parameters themselves, including mean and standard deviation across parameters and layers.
• The observation is that the distribution of parameter values and the gradients can reveal imbalances across layers, guiding learning rate choices and initialization scales.
• There is often an outlier layer (for example the final output layer) that behaves differently from the rest, potentially driving the overall learning rate policy.

01:39:55 Viz #4: Update:Data Ratio Over Time

• A diagnostic metric called the update-to-data ratio is introduced to monitor how large the parameter updates are relative to their current values.
• The update is the product of the learning rate and the gradient; dividing by the parameter value provides a sense of how aggressively a parameter is being moved per step.
• The log10 of this ratio provides a clear visualization of training dynamics; if the ratio is too large, updates may be unstable; if too small, learning could be too slow.
• The final takeaway is that the ratio should sit in a stable range (roughly around 1e-3 to 1e0 in practice), with more care required for the last, smaller or larger layers depending on their scale.

01:46:04 Bringing Back Batchnorm, Looking at the Visualizations

• BN is reintroduced and the visuals are re-examined with BN in place, showing more uniform activation statistics across layers.
• Even if a layer has a different gain, the BN layer can absorb scale changes and maintain stable activations and gradients.
• The speaker notes that BN makes the training of deep nets significantly more robust, though BN introduces a coupling across batch examples during forward passes, which has its own pros and cons.

01:51:34 Summary of the Lecture for Real This Time

• The lecture recaps the core ideas: activation statistics, gradient flow, the role of initialization, and how Batch Normalization stabilizes training across deep nets.
• The practical lesson is to combine mindful initialization with normalization and robust optimizers to achieve stable training in deeper architectures.
• The speaker hints at future topics including residual connections and more normalization strategies that mitigate BN coupling issues.

01:58:XX Bonus: Torify and Code Organization (Summary of Live Preview)

• A teaser for turning the code into modular PyTorch-like blocks such as Linear, BN, and nonlinearities so experimentation mirrors PyTorch conventions.
• The emphasis is on building reusable modules and clean forward passes, making it easier to swap in normalization, residuals, and different activation functions.

02:10:00 Closing Thoughts

• The journey from naive initialization to batch normalization marks a turning point for training deep nets.
• Diagnostics through histograms, gradient statistics, and update ratios provide practical guidance for tuning depth, normalization, and learning rates.
• The path forward includes deeper nets, residuals, and alternative normalization styles; the field continues to evolve with ongoing research into initialization and optimization dynamics.

02:15:00 Resources and References

• The lecture provides several external resources and links that underpin the material discussed:

- Kaiming He et al. Delving Deep into Rectifiers (Kaiming init): arXiv:1502.01852 - Batch Normalization paper: Ioffe and Szegedy, 2015, arXiv:1502.03167 - Bengio et al. 2003 MLP language model: Bengio, 2003, JMLR - Good critique paper on BatchNorm practical issues: arXiv:2105.07576 - makemore on GitHub: https://github.com/karpathy/makemore - Jupyter notebook for part 3 BN: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part3_bn.ipynb - Colab notebook: https://colab.research.google.com/drive/1H5CSy-OnisagUgDUXhwo1ng2pjKHYSN?usp=sharing - karpathy.ai and social links provided in the description - ResNet and PyTorch related references in the lecture text

Resources Mentioned

• Papers and concepts:

- Delving Deep into Rectifiers (Kaiming init) — He et al. 2015 - Batch Normalization — Ioffe and Szegedy, 2015 - Bengio et al. 2003 MLP Language Model - Practical concerns with BatchNorm in practice (arXiv:2105.07576) - Kingma and Ba on initialization and normalization considerations (implied through He et al.)

• Tools and Platforms:

- PyTorch docs and nn modules (BatchNorm, Linear, etc.) - makemore GitHub repository - Jupyter notebooks and Colab notebooks linked in the description

• Diagrams and visuals described in the lecture:

- Histograms of activation distributions across layers - Histograms of gradient magnitudes across layers - Update to data ratios plotted over time - Tests with and without nonlinearities to illustrate the role of activation functions

Notable Terms and Concepts to Know (Glossary in detail below)

• Activation: the output of a neuron or a layer after applying a nonlinear function.
• Backpropagation: the algorithm for computing gradients of the loss with respect to all parameters.
• Logits: the raw, unnormalized scores produced by a last linear layer before applying softmax.
• Softmax: converts a vector of logits into a probability distribution over classes.
• Cross-entropy loss: loss function measuring discrepancy between a probability distribution and the true distribution (often used with softmax).
• Fan-in / Fan-out: the number of input connections (fan-in) and output connections (fan-out) to a layer, used for initialization scaling.
• Kaiming/He initialization: initialization scheme designed for deep networks with ReLU-like nonlinearities, scaling weights by a factor related to fan-in to preserve signal variance.
• Batch Normalization (BN): a normalization technique that standardizes activations per minibatch and applies a learned scale and shift.
• Gamma (γ) / Beta (β): learnable scale and shift parameters in BN after normalization.
• Running mean/variance: moving averages of batch statistics used during inference time in BN.
• Epsilon (ε): small constant added to variance to prevent division by zero.
• Momentum (for BN): the weighting factor for updating running statistics across batches.
• Residual connections: shortcut paths that allow layers to skip one or more layers and be added back to the main path, aiding optimization of deep nets.
• Saturation: region where a nonlinear activation flattens and gradients vanish, hindering learning.
• Dead neuron: a neuron that never activates for any input in a dataset, leading to zero gradient and no learning.
• Update-to-data ratio: a diagnostic metric comparing how large parameter updates are relative to current parameter magnitudes.
• Regularization: techniques that prevent overfitting and improve generalization, BN contributing via batch-wise noise.

1) Executive Summary

• This lecture dissects the interplay between activations and gradients in multi-layer perceptrons, focusing on how initialization scale, activation saturation, and normalization affect trainability. The core message is that deep nets can be fragile without proper management of activations and gradients; a principled approach combining careful initialization (Kaiming/He style), batch normalization, and diagnostic visualizations makes deep learning far more robust and scalable.
• The talk demonstrates practical issues with naive initialization, such as hockey-stick loss and saturating activations, and then presents concrete remedies, including scaling weights to preserve variance, setting biases thoughtfully, and using batch normalization to stabilize both forward activations and backward gradients. The introduction of Batch Normalization is highlighted as a watershed moment, enabling much deeper networks to train reliably, while also discussing its limitations such as batch coupling and the need for running statistics during inference.
• The anatomical tour includes a realistic ResNet50 style walkthrough to illustrate BN in practice, followed by a suite of visualization techniques that reveal the health of forward activations, backward gradients, and parameter updates. The talk emphasizes that while BN and heuristics drastically improve stability, the field is still evolving, with residual architectures and other normalization schemes offering further robustness.

2) Key Takeaways

• ⭐ HIGH PRIORITY Initialization matters: improper initialization can create huge initial losses and poor gradient flow; small, well-scaled weights help start training in a productive regime.
• ⭐ HIGH PRIORITY Activation saturation kills learning: activations pushed into the saturated regions of nonlinearities (tanh, ReLU tails) destroy gradient flow; balance scale to keep activations in an active regime.
• Kaiming/He initialization provides a principled way to scale weights by the square root of the fan-in to preserve signal variance across layers.
• A tiny bias at initialization can help symmetry breaking without destroying the near-zero logits goal; zeroing the final bias can also help keep logits near zero.
• Batch Normalization stabilizes training across deep networks by standardizing activations before nonlinearities and allowing learnable scaling and shifting; it tends to regularize training as a side effect.
• BN can be folded into preceding layer weights after training, meaning inference can avoid BN during forward passes in some settings; this is a practical efficiency trick.
• BN introduces coupling across batch examples in the forward pass; this can act as a regularizer but creates some fragility and debugging complexity.
• Running statistics (mean and variance) enable BN to be used for inference with a single fixed calibration; calibrating BN can be done post-training or as running estimates during training.
• In practice, practitioners sprinkle BN after linear or convolutional layers to stabilize activations and gradient flow, especially in very deep architectures like ResNet.
• The gain parameter in initialization and the BN gamma parameter are both learnable, enabling the network to adjust activation scales as training progresses.
• Diagnostic visuals (activation histograms, gradient histograms, update-to-data ratios) are essential tools to understand the health of a network and guide tuning.
• The final lesson is that deep nets are a balancing act; with modern techniques like BN and smarter optimizers, you can train deep architectures more reliably, but understanding the dynamics remains valuable for robust engineering.

3) Action Items for Practitioners

• Inspect initialization: ensure weights are scaled by 1 / sqrt(fan_in) and adjust with a modest gain for the chosen nonlinearity; avoid exact zero initialization for weights in deeper nets.
• If using tanh or sigmoid activations, monitor saturation by visualizing preactivations and ensure they stay in a healthy range; consider reducing initial weight scales if saturation is observed.
• Implement Batch Normalization after linear layers and before nonlinearities; include learnable gamma and beta; use a small epsilon in the normalization denominator.
• If BN is used, consider folding the gamma and beta into the preceding linear layer after training for inference efficiency; alternatively, implement running mean/variance tracking during training for consistent inference behavior.
• Be mindful of batch size when using BN; a very small batch size can destabilize running statistics; adjust BN momentum or consider alternative normalization schemes if batch size is tiny.
• Use diagnostic plots of forward activations, backward gradients, and update-to-data ratios to monitor training health; watch for disproportionate gradient magnitudes in later layers.
• For deep nets with residual connections, BN integrates cleanly and helps stabilize very deep stacks; plan BN placement around residual blocks carefully for best results.
• When experimenting with optimizers, expect BN to reduce sensitivity to exact initialization scales, making stable training possible with more relaxed initial hypotheses.
• If you see a layer repeatedly saturating while others are learning well, consider adjusting the gain in the corresponding linear layer or rebalancing the distribution of activations with BN.
• Practice a PyTorch-like modularization mindset: structure your code with Linear, BN, and nonlinear blocks, enabling easy rearrangements and experiments similar to real-world architectures.
• Always keep in mind the fundamental tradeoffs: BN improves stability but ties activations to batch statistics during training; understand when to calibrate running statistics and when to calibrate the learning rate for robust training.

4) Mental Models (Frameworks, Formulas, Rules of Thumb)

• Gain over Fan-in: to preserve activation variance across a layer, initialize weights with stddev roughly gain / sqrt(fan_in). This keeps the signal from exploding or vanishing as it travels through deep stacks.
• Activation saturation risk: if preactivations are too large, nonlinearities push outputs toward saturating regions; gradients vanish and learning stalls. Keep activations in a healthy central region via initialization scale and/or BN.
• BN as a normalizer plus learnable rescaler: BN standardizes activations and applies a learnable gamma (scaling) and beta (offset), allowing the network to recover representational flexibility while maintaining stable gradient flow.
• Running statistics vs batch statistics: running mean/variance provide a stable inference mechanism; training can also update these statistics via a momentum-based running average.
• Regularization via batch statistics: BN introduces batch-level noise that can act as a regularizer, reducing overfitting and improving generalization in many cases.
• Diagnostics as a compass: histograms of activations and gradients, plus update-to-data ratios, guide tuning of initialization scales, BN placement, and learning rates, especially in deeper networks.

5) Glossary (Term definitions)

• Activation: the output of a neuron or unit after applying a nonlinear function.
• Backpropagation: the algorithm to compute gradients of a loss with respect to network parameters efficiently by applying the chain rule backwards through the network.
• Logits: the raw scores produced by the last linear layer before applying softmax.
• Softmax: converts raw scores into a probability distribution over categories.
• Cross-Entropy Loss: a loss measuring the difference between predicted probability distribution and the true distribution; common for classification tasks with softmax outputs.
• Fan-in / Fan-out: number of inputs to a neuron (fan-in) and number of outputs from a neuron (fan-out); used for weight initialization scaling.
• Kaiming He initialization (kaiming_normal/kaiming_uniform): initialization strategy designed for rectified linear units (and related nonlinearities) to preserve signal variance across layers.
• Batch Normalization (BN): a normalization technique that standardizes the activations across the batch, with per-feature learnable scale gamma and shift beta.
• Gamma (γ) / Beta (β): learnable scale and shift parameters in BN that allow restoration of representation power after normalization.
• Running Mean / Running Variance: moving averages of batch statistics used during inference with BN.
• Epsilon (ε): a small constant added to variance to prevent division by zero.
• Momentum (for BN): controls how quickly running statistics are updated during training.
• Residual Connections: shortcut connections that bypass one or more layers and feed their output into later layers, aiding gradient flow in deep nets.
• Saturation: the region where nonlinear activations produce near-constant outputs, causing gradients to vanish.
• Dead Neuron: a neuron that never activates across the data and receives no useful gradient to update.
• Update-to-Data Ratio: a diagnostic metric comparing the magnitude of parameter updates to the magnitude of parameter values, used to gauge training stability.

6) Quotes (verbatim excerpts with timestamp and speaker)

• [No direct verbatim quotes captured in this section due to the risk of misquoting precise phrasing from the transcript. The section remains available for addition if exact verbatim lines are identified and confirmed.]

7) Topics

• Activations And Gradients
• Batch Normalization
• Initialization and Kaiming Init
• Deep Learning Diagnostics
• Neural Network Regularization
• PyTorch Style Modularization