The spelled-out intro to neural networks and backpropagation: building micrograd

Andrej Karpathy builds micrograd, a scalar autograd engine, to teach backpropagation and neural net training from first principles.

Core Engine

• Value objects wrap scalars and record children plus the operation that created them (0:00-12:02).
• __add__, __mul__, exp, pow and tanh each store a local _backward that applies the chain rule (1:12:21-1:24:24).
• backward() runs topological sort so every node’s gradient is computed exactly once after its parents (1:24:25-1:36:27).

Gradient Mechanics

• Gradients must be accumulated with +=; overwriting produces wrong results when a node is used twice (1:24:25-1:36:27).
• Local derivative of tanh is 1-tanh²; derivative of x^n is n*x^(n-1) (1:00:18-1:12:21).

Neuron to MLP

• A neuron computes w·x+b then applies tanh; Layer and MLP stack these units (1:48:30-2:00:32).
• Mean-squared-error loss turns the network output into a single scalar to minimize (1:48:30-2:00:32).

Training Loop

• Gradient descent subtracts learning-rate times grad from every parameter; zero_grad must be called each step (2:00:35-2:12:38).
• The identical forward/backward math appears in PyTorch tensors with requires_grad=True (1:36:28-1:48:29).

00:00:00 Major Topic — Introduction to the lecture and the goal of building micrograd

00:00:00 Speaker: Andrej Karpathy

• Overview: The aim is to show what neural network training looks like under the hood by starting from a blank Jupyter notebook and building a small autograd engine called micrograd. The emphasis is on intuition and the actual mechanics of gradient calculation, not on heavy production-grade tooling.
• Key claim: Backpropagation (automatic differentiation) is the mathematical core of training neural networks; micrograd demonstrates how to compute gradients for arbitrary scalar expressions, and then that machinery generalizes to neural nets.
• Important metaphor: Neural networks are just a class of mathematical expressions; backprop does not care about neural nets specifically but about propagating derivatives through an expression graph.

Quote: "micrograd is basically an autograd engine autograd is short for automatic gradient and really what it does is it implements backpropagation" (00:00:00)

00:00:25 Major Topic — micrograd overview

00:00:25 Subtopic — What micrograd does for you

• Micrograd wraps numbers in a lightweight Value object and builds up a computation graph from basic operations like addition, multiplication, and other simple ops. Each Value knows its data, its gradient (grad), and pointers to its children (the operands that created it).
• Forward pass: Evaluate the expression by computing g = f(a, b, ...). The example uses a, b as inputs to create c, d, e, f, g, and shows that g has a numerical value (e.g., 24.7).
• Backward pass: Start from the final output g and propagate gradients backward through the graph using the chain rule, yielding partial derivatives (dg/da, dg/db, etc.). These gradients tell you how sensitive the final output is to each internal node and to the inputs.
• Key takeaway: The engine is scalar-based (operates on single numbers) to keep the math clear and pedagogical; for production, you would move to tensor-based, vectorized computations for efficiency.

Quote: "backpropagation would be at the mathematical core of any modern deep neural network library like say pytorch or jaxx" (00:00:25)

00:08:08 Major Topic — Derivative intuition with a simple function

00:08:08 Subtopic — Numerical derivative intuition

• Definition: The derivative at x is the limit of [f(x+h) − f(x)] / h as h → 0. Since we cannot take h to exactly zero in floating point arithmetic, we approximate with a small h (e.g., 0.001).
• The derivative tells you the slope: how much f changes per unit change in x. For the simple quadratic f(x) = 3x^2 − 4x + 5, the derivative at x=3 is 14, which matches the analytic derivative.
• The numerical approach shows the same intuition as the analytic derivative: derivative is the sensitivity of the output to small changes in input.
• The section emphasizes differentiability and how the slope at a point governs how small nudges to x affect f(x).

Quote: "the definition of derivative is the limit as h goes to zero of [f(x+h)−f(x)]/h" (00:08:08)

00:14:12 Major Topic — Derivative with multiple inputs

00:14:12 Subtopic — Extending intuition to functions of several inputs

• Example: A function d = f(a, b, c) with d printed as 4. The talk shows how to measure partial derivatives ∂d/∂a, ∂d/∂b, ∂d/∂c by nudging each input by a small h while holding others fixed and observing the change in d.
• He explains that the slopes for a, b, c at a given point have intuitive signs (positive or negative) based on how a, b, c contribute to d (e.g., d = a·b + c or d = a·b + c, etc.).
• These exercises connect to the chain rule: derivatives of composite expressions are products of local derivatives along the computational path from inputs to the final output.

Quote: "the derivative of d with respect to a would be b, and the value of b is negative 3 which is the derivative that we have" (00:14:12)

00:19:09 Major Topic — Starting micrograd’s Value object and visualization

00:19:09 Subtopic — Building blocks and visualization

• He introduces the skeleton of the Value object, which wraps a scalar value and will be extended to support operations like add, mul, and more complex expressions.
• Operator overloading: Python's __add__, __mul__, etc., are implemented so that Value objects can participate in arithmetic expressions naturally (e.g., a + b, a * b).
• Visualization is shown via a graph drawing utility (Graphviz-like API) to visualize the expression graph, with leaves as input values and internal nodes representing intermediate expressions; this helps understand the forward pass as a graph, and later the backward pass as gradient propagation on that graph.
• The Value object stores: data (the numeric value), grad (the derivative of the final output w.r.t. this Value), and pointers to children/parents to represent the computation graph.

Quote: "micrograd basically allows you to build out mathematical expressions [...] where a and b are transformed into c, d and eventually e f and g" (00:19:09)

00:32:10 Major Topic — Manual backpropagation example #1: simple expression

00:32:10 Subtopic — Step-by-step backward through a simple expression

• The example shows a small expression built from a and b, then demonstrates calling backward on the final Value to compute gradients for each internal node and inputs.
• Highlights how local derivatives (e.g., for addition or multiplication) are used and how gradients propagate backward through the graph via the chain rule.
• The demonstration confirms that the computed gradients agree with analytic expectations (e.g., ∂g/∂a = 138, ∂g/∂b = 645 for the constructed expression), illustrating the correctness of backpropagation in a tiny graph.

Quote: "the derivative of g with respect to a which in this case happens to be 138 and the derivative of g with respect to b which also happens to be here 645" (00:32:10)

00:51:10 Major Topic — Preview of a single optimization step

00:51:10 Subtopic — From gradients to a parameter update

• He previews how gradient information is used to perform a single optimization step (gradient-based update) in a toy setup. This foreshadows how backpropagated gradients guide the parameter updates in training.
• The concept of using the gradient to adjust inputs/weights in the direction that reduces the loss is introduced, laying groundwork for the later sections on training with gradient descent.

Quote: "preview of a single optimization step" (00:51:10)

00:52:52 Major Topic — Manual backpropagation example #2: a neuron

00:52:52 Subtopic — A neuron model and forward/backward pass

• A simple neuron is described: inputs x, weights w, bias b; the neuron computes a weighted sum plus bias, then passes through a nonlinearity (tanh in the example).
• The talk introduces how to construct the neuron’s forward pass with Value objects for each intermediate quantity (x1w1, x2w2, sum, and the final activation) and how to backpropagate through the tanh nonlinearity.
• The tanh function is introduced as a nonlinearity used in biological/ML neurons; the backward pass requires its derivative, which is 1 - tanh(x)^2.
• This section demonstrates how to implement not only simple add/mul but also a nonlinearity and its gradient through the computation graph.

Quote: "we're taking the input data as an input and the weights of a neural network as an input and it's a mathematical expression and the output are your predictions of your neural net or the loss function" (00:00:25) [This paraphrase reflects the idea but a precise verbatim quote is captured earlier; see 00:52:52 area for tanh specifics]

01:09:02 Major Topic — Implementing the backward function for each operation

01:09:02 Subtopic — Local derivatives and backward routines

• The engine is extended to define a backward function for each operation (e.g., add, mul, tanh). Each backward computes how the final gradient (dL/dout) should contribute to each input’s grad via the local derivative (e.g., d(out)/d(a) for addition is 1, for multiplication it is the other input value, etc.).
• The importance of locally defined derivatives is stressed: if you know how the output changes with respect to the inputs for a single operation, you can propagate gradients through the entire graph by chaining these local derivatives.
• The example emphasizes that many different non-linearities can be supported as long as their local derivatives are known, and that the entire backpropagation harnesses the chain rule in a compositional way.

Quote: "the derivative l with respect to d is f and the derivative with respect to f is d" (01:09:02) [paraphrased concept; exact lines appear in context earlier]

01:17:32 Major Topic — Implementing the backward function for a whole expression graph

01:17:32 Subtopic — Backpropagation across multiple intermediate nodes

• The backward pass is extended to propagate gradients from the final output through a whole expression graph by applying the chain rule across each operation's local derivatives, using the stored graph structure (children, operation type).
• The process involves initializing the final output’s grad to 1, walking backward through the graph in topological order so that all dependencies are processed, and updating each node’s grad accordingly.
• This section also emphasizes the need to accumulate gradients when a value contributes to multiple paths (to fix the bug that arises when a node is used more than once in the graph).

Quote: "this backpropagating signal which is carrying the information of like what is the derivative of L with respect to all the intermediate nodes we can imagine it almost like flowing backwards through the graph" (01:17:32)

01:22:28 Major Topic — Fixing a backprop bug when one node is used multiple times

01:22:28 Subtopic — Accumulating gradients for shared nodes

• When a node is used more than once (e.g., a appears multiple times in a computation), its gradient contributions can come from multiple paths. The simple assignment (grad = something) overwrites previous contributions, leading to incorrect gradient values.
• The fix is to accumulate gradients with += rather than assign, initializing grads to zero. This mirrors the mathematical reality of the chain rule in multi-path graphs: the total derivative is the sum of contributions across all paths.

Quote: "the solution here ... accumulate these gradients plus equals; this will be okay remember because we are initializing them at zero" (01:22:28)

01:27:05 Major Topic — Breaking up a tanh, exercising with more operations

01:27:05 Subtopic — Decomposing tanh into primitive operations for more completeness

• The lecture re-implements tanh as a composition of primitive operations (via exponentials) to show you can break a nonlinearity into smaller, differentiable parts. This helps exercise with more operations (exponentiation, division, etc.) and demonstrates that you can achieve the same forward/backward behavior by composing a bunch of atomic operations.
• It also demonstrates code organization flexibility: you can choose to implement a nonlinearity as a single composite op or break it into atomic pieces, as long as you provide correct local derivatives for each piece.

Quote: "the level at which you implement your operations is totally up to you... if you can write the local gradients you can chain the gradient" (01:27:05)

01:39:31 Major Topic — PyTorch: comparison and integration

01:39:31 Subtopic — Comparing micrograd with PyTorch

• Karpathy walks through performing the same forward/backward flow in PyTorch to illustrate that the same core ideas apply in modern libraries, even though the implementation details differ (tensors, autograd, etc.).
• PyTorch uses multi-dimensional tensors (not just scalars), requires gradient tracking flags on leaf nodes, and provides a more extensive API, but the underlying gradient flow and backpropagation logic follow the same chain-rule principle.
• The demonstration shows backward propagation through tanh in PyTorch and the equivalence of gradients with micrograd, reinforcing the teaching: the core mathematics is the same, only the abstraction level is different.

Quote: "doing backward pass for tanh in PyTorch ... the backward pass kernel backpropagates through tanh" (01:39:31)

01:43:55 Major Topic — Building out a neural net library (multi-layer perceptron) in micrograd

01:43:55 Subtopic — Extending micrograd to MLPs

• The lecture expands micrograd to support building a small neural net: a multi-layer perceptron (MLP) with multiple layers of neurons, each layer fully connected to the next.
• It introduces a PyTorch-like organization: a neuron class with weights and biases, a layer class (a collection of neurons), and an MLP class that sequences layers according to a list of layer sizes.
• Each neuron computes a forward pass (weighted sum plus bias passed through a nonlinearity), and the entire network is built as a composition of these neurons, layers, and activations.
• This shows that neural nets are just specific, structured computation graphs built from a small set of primitive operations.

Quote: "neurons are fully connected to each other and this is a two-layer neural net" (01:43:55)

01:51:04 Major Topic — Creating a tiny dataset, writing the loss function

01:51:04 Subtopic — Loss definitions and training signals

• A very small binary classification dataset is created to illustrate training a small MLP. The goal is to push the network to output near the target labels (1.0 or -1.0).
• The loss function used is Mean Squared Error (MSE): the average of squared differences between predictions and targets. The loss is large when predictions are far from targets and zero when predictions perfectly match targets.
• The gradient of the loss with respect to network outputs is what ultimately propagates back through the network to adjust weights and biases.

Quote: "the loss is mean squared error; this is doing is we are going to basically iterate for y ground truth and y output in zip of y and yhat" (01:51:04)

01:57:56 Major Topic — Collecting all of the parameters of the neural net

01:57:56 Subtopic — Parameter collection and inspection

• The network’s trainable parameters (weights and biases) are gathered into a single list so that the optimization step can update all parameters at once. This mirrors PyTorch’s parameters() method in nn.Module.
• A nested structure is used: neuron parameters, then layer parameters (a layer holds several neurons), then the entire MLP’s parameters.
• This setup enables a simple gradient descent step: iterate over all parameters and update each by a small step proportional to its gradient.

Quote: "we gather up all of the parameters of the neural net so that we can operate on all of them simultaneously" (01:57:56)

02:01:12 Major Topic — Doing gradient descent optimization manually, training the network

02:01:12 Subtopic — A full, tiny training loop with micrograd

• The tutorial walks through a complete training loop: forward pass (compute predictions) → loss calculation → backward pass (compute gradients) → update parameters with a gradient descent step → repeat for many iterations.
• The learner discusses the effects of learning rate: too small is slow; too large can destabilize training or cause divergence. A learning-rate schedule can help (e.g., learning rate decay over iterations).
• It demonstrates a successful training run in a toy setting, showing how the network learns to make predictions closer to targets after several iterations of gradient descent.

Quote: "gradient descent optimization manually, training the network" (02:01:12)

02:14:03 Major Topic — Summary and how to go toward modern neural nets

02:14:03 Subtopic — Takeaways and the path forward

• The speaker summarizes that neural networks are a class of mathematical expressions; backpropagation provides gradients for all parameters; gradient descent uses those gradients to minimize the loss; and with enough scale, neural nets gain remarkable capabilities (e.g., GPT-like models) though the underlying math remains the same.
• The micrograd project demonstrates that the engine itself is surprisingly small (roughly 100 lines for the autograd engine, plus some for a tiny NN library). The rest is the same ideas reapplied at larger scale with tensors and optimized backends.
• The lecture emphasizes pedagogy: starting small, building intuition with a scalar autograd engine, and then scaling to tensors and large nets, is the right way to deeply understand neural network training.

Quote: "micrograd is what you need to train your networks and everything else is just efficiency" (02:14:03)

02:16:46 Major Topic — Walkthrough of the full code of micrograd on GitHub

02:16:46 Subtopic — Code tour highlights

• The engine (two files): the autograd engine and a minimal NN library atop it. Core pieces include the Value class, the operations (+, *, etc.), and the backward pass logic.
• The NN library (nn.pi) implements a tiny neural network API, roughly mirroring PyTorch’s nn.Module structure: Neuron, Layer, MLP, etc.
• A test suite (micrograd vs PyTorch) ensures that the forward/backward passes align with PyTorch expectations for the same expressions.
• The demonstration shows how micrograd’s operations can be composed to construct multi-layer neural nets and train them with simple gradient descent.

Quote: "Walkthrough of the full code of micrograd on GitHub" (02:16:46)

02:21:10 Major Topic — Real stuff: diving into PyTorch, finding their backward pass for tanh

02:21:10 Subtopic — How a modern framework implements backward for tanh

• The lecture contrasts micrograd’s simple, explicit local derivatives with PyTorch’s highly optimized, device-specific kernels. It shows where to look inside PyTorch for the tanh backward implementation (cpu vs gpu kernels), and notes the complexity that arises in real-world libraries.
• This section emphasizes that while PyTorch is much larger and optimized, the same chain-rule idea underpins its functionality. The difference is architectural and implementation-level complexity, not a different math foundation.

Quote: "the backward pass kernel for tanh ... the CPU kernel and the CUDA kernel for tanh backward" (02:21:10)

02:24:39 Major Topic — Conclusion

02:24:39 Subtopic — Final thoughts and resources

• Karpathy reiterates the availability of related resources: micrograd on GitHub, NN Zero to Hero notebooks, the author’s website and social media, and a Colab exercise. He notes the possibility of a follow-up video answering common questions.
• He emphasizes that the core concepts are universal across the size of models; understanding micrograd builds a strong foundation for understanding modern neural networks.
• He ends with encouragement to explore, subscribe, and engage in the community for further questions and iterations.

Quote: "I hope you enjoyed building out micrograd with me" (02:24:39)

02:25:20 Major Topic — Outtakes and wrap-up

02:25:20 Subtopic — Lighthearted closing

• The video closes with outtakes and a sense of humor about the learning process and the pitfalls of live demonstrations.

Quote: "outtakes :)" (02:25:20)

Resources Mentioned

• micrograd on GitHub: https://github.com/karpathy/micrograd
• jupyter notebooks for micrograd: https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd
• karpathy.ai (personal website): https://karpathy.ai
• Andrej Karpathy on Twitter: https://twitter.com/karpathy
• Google Colab exercise: https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing
• PyTorch: https://pytorch.org
• Graphviz/GraphViz-like graph visualization (GraphIs API mentioned for drawing graphs)
• Discord channel for Neural Networks: Zero to Hero series Discord: https://discord.gg/3zy8kqD9Cp

Notable Visuals and Explanations to Recreate

• A forward expression graph with leaves a and b, showing how c = a + b, d = a * b, e = c + d, and g = some final combination; Wy the backward pass can compute dg/da and dg/db by traversing the graph.
• A small neural unit (neuron) with x1, x2 inputs, weights w1, w2, bias, summing to n, then applying tanh to yield the output; demonstration of backward through tanh via its derivative 1 - tanh(x)^2.
• A tiny MLP example with multiple layers, a loss function (mean squared error), and gradient descent loop to show training dynamics (loss decreasing over iterations).
• A comparison between micrograd’s scalar engine and PyTorch’s tensor-based autograd to emphasize that the core ideas are the same; micrograd emphasizes understanding, PyTorch emphasizes scale and performance.

Summary of Key Concepts

• autograd/backpropagation allows efficient gradient computation for a loss with respect to neural network weights by traversing a computation graph backward using the chain rule.
• micrograd demonstrates a scalar-based autograd engine: a tiny Value type, forward graph construction via operator overloading, and backward propagation via per-operation local derivatives and topological ordering.
• The learning loop combines forward pass, loss computation, backward pass, and parameter updates (gradient descent), illustrating how neural nets learn from data.
• Real-world frameworks (like PyTorch) implement the same ideas at tensor scale with far more complexity and optimization, including kernel implementations for non-linearities, device-specific backends, and performance optimizations.
• Topological ordering is used to ensure gradients are propagated in a valid order from output back to inputs.

Vocabulary and Concepts Defined in Simple Terms

• autograd: automatic differentiation; automatically computing derivatives for a program’s output with respect to its inputs.
• backpropagation: the algorithm that computes gradients by applying the chain rule from the output back to inputs through the computation graph.
• chain rule: the rule that says the derivative of a composite function is the product of the derivatives along the path of composition.
• computation graph: a graph where nodes are operations/values and edges connect inputs to outputs; helps visualize forward computation and backward gradient flow.
• gradient: the vector of partial derivatives of a scalar function with respect to its inputs; tells you how to change inputs to increase or decrease the function.
• mean squared error (MSE): a common loss function that measures the average squared difference between predictions and targets.
• gradient descent: an optimization algorithm that updates parameters in the opposite direction of the gradient to minimize a loss function.
• topological sort: an ordering of a graph’s nodes such that every directed edge goes from earlier to later in the order; used to ensure correct backward propagation order.
• tanh: a smooth, S-shaped nonlinearity used in neural networks; derivative is 1 − tanh(x)^2.
• ReLU: another common nonlinearity (not the main focus here) with derivative 1 for positive input and 0 otherwise.
• tensor: a multi-dimensional array; PyTorch uses tensors to represent data and weights for efficient, parallel computation.
• autograd engine: the system responsible for recording computations and performing backward passes to compute gradients.

Notable Quotes (Verbatim with timestamps)

• "hello my name is andre and i've been training deep neural networks for a bit more than a decade and in this lecture i'd like to show you what neural network training looks like under the hood" — 00:00:00
• "backpropagation is this algorithm that allows you to efficiently evaluate the gradient of some kind of a loss function with respect to the weights of a neural network" — 00:00:00
• "> the forward pass is straightforward we will access that using the dot data attribute and the output of the forward pass the value of g is 24.7" — 00:00:00
• "micrograd is basically an autograd engine autograd is short for automatic gradient and really what it does is it implements backpropagation" — 00:00:25
• "the derivative of a simple function with one input" — 00:08:08
• "the derivative with multiple inputs" — 00:14:12
• "the value object will wrap the numbers themselves and then we are going to build out a mathematical expression" — 00:19:09
• "backpropagation is going to start at g and it's going to go backwards through that expression graph and it's going to recursively apply the chain rule" — 00:00:25
• "micrograd is what you need to train your networks and everything else is just efficiency" — 02:14:03
• "there's a lot of power that comes from only 150 lines of code" — 00:00:00
• "the level at which you implement your operations is totally up to you but you have to know how to differentiate through it" — 01:27:05

Topics

• Neural Networks
• Backpropagation
• Automatic Differentiation
• Micrograd
• Gradient Descent
• PyTorch Comparison
• Computation Graphs
• Activation Functions (tanh, soft nonlinearity)
• Topological Sort
• Model Training Loops