Backpropagation, One Neuron at a Time

When I first used PyTorch, these three lines did most of the work:

loss = loss_function(prediction, target)
loss.backward()
optimizer.step()

I could recite what they did. The network makes a prediction, measures how wrong it is, and nudges its weights. It was the middle line that bothered me. One function call, and every weight in the network somehow knew which direction to move. I could not point at a single weight and say this one went up because of that. My own code felt like a magic trick, and I would rather understand a trick than be impressed by it.

So for my from-scratch algorithms project, I deleted those three lines and rebuilt them in NumPy, where nothing is hidden. This post is the smallest version of what I found underneath: one input, one neuron, one training example — arithmetic you could do on the back of a receipt. Every formula below is attached to something you can drag, so you can poke at it instead of taking my word for it. At the end I add a hidden neuron, and you will see the story does not really change. There is just more of it.

If you have never trained a model, here is the whole thing in four moves. Everything that follows is just one of these, slowed all the way down:

1. Guess. Run the input through the network and get a prediction.
2. Grade the guess. Measure how far it was from the right answer. That single number is the loss.
3. Assign blame. Figure out how much each weight was responsible for that error. Those blame scores are the gradients — this step is backpropagation.
4. Nudge. Shift every weight a little in the direction that lowers the loss, then guess again.

Run that loop enough times and the guesses get good. A real network with millions of weights does the exact same four moves; it just has far more blame to spread around. The rest of this post is one neuron walking through the loop once — and the four sections below are those four moves in order.

First, make a prediction

A neuron is smaller than the word makes it sound. It takes a number in, turns one knob, and passes a number out. The knob is the weight $w$: it decides how much the input should count. There is also a bias $b$, a fixed offset that shifts the answer up or down before anything else happens. Multiply by the weight, add the bias, and you have the neuron’s raw score:

$$z=wx+b.$$

That score $z$ can be anything, from a large negative number to a large positive one. But if I am trying to answer a yes-or-no question — is this email spam, is this digit a seven — I want a probability, and a probability has to sit between 0 and 1. The sigmoid function is the squasher that gets me there:

$$p=\sigma(z)=\frac{1}{1+e^{-z}}.$$

It bends any number onto the range 0 to 1: a big positive $z$ lands near 1, a big negative $z$ lands near 0, and $z=0$ maps to exactly one-half. So I can read the output $p$ as confidence. If $p=0.8$, the neuron is saying “class 1, 80% sure.”

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

x = 2.0
w = -0.5
b = 0.2

z = w * x + b       # -0.8
p = sigmoid(z)       # about 0.31

Drag the sliders and watch the dot ride along the S-curve. Notice that a bigger weight does not always mean a bigger prediction — flip the sign of the input and cranking the weight up sends the prediction down instead. The weight and the input only matter together, as their product.

Change the neuron

Drag the knobs. The dot rides the sigmoid.

weighted input-0.600 × 1.500 + 0.200

z = wx + b-0.700

p = sigmoid(z)0.332

input x

1.50

weight w

-0.60

bias b

0.20

Turn “wrong” into a number

Suppose the correct label is $y=1$. A prediction of $p\approx0.31$ is clearly too low, but “too low” is not something you can do calculus on. Training needs a single number that grows as the prediction gets worse — a score for wrongness.

For yes-or-no problems, that score is binary cross-entropy:

$$\mathcal{L}(y,p)=-[y\log(p)+(1-y)\log(1-p)].$$

When $y=1$, the second term disappears:

$$\mathcal{L}(1,p)=-\log(p).$$

This gives confident mistakes a large loss. Predicting 0.49 when the answer is 1 is wrong, but predicting 0.01 is much worse.

How wrong is the prediction?

answer is 1answer is 0

prediction

0.30

cross-entropy

1.204

model prediction

0.30

Slide toward the wrong end. The loss does not creep up — it runs away toward infinity, which is the point: a confident mistake should hurt.

In NumPy, I clip the prediction before taking the logarithm. This prevents log(0), which is undefined.

def binary_cross_entropy(p, y):
    p = np.clip(p, 1e-12, 1.0 - 1e-12)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))

What a gradient tells us

The loss tells me how bad the prediction is. The gradient tells me what to do about it: how the loss would change if I nudged one weight a little and held everything else still.

$$\frac{\partial\mathcal{L}}{\partial w}$$

is that number. The curly $\partial$ just means “partial” — the rate of change of the loss with respect to $w$ alone, pretending every other value is frozen. Read it as: if I push $w$ up by a hair, which way does the loss move, and how fast?

• A positive gradient means increasing $w$ would increase the loss.
• A negative gradient means increasing $w$ would decrease the loss.
• A gradient close to zero means a small change in $w$ would barely affect the loss.

Backpropagation finds this number by walking backward through the same operations it used going forward. For our neuron, the forward path was a little assembly line:

$$w \longrightarrow z \longrightarrow p \longrightarrow \mathcal{L}.$$

Each stop on that line knows exactly one small thing: how much its output moves when its input wiggles. The chain rule is the rule for stitching those local answers into one. If nudging $w$ nudges $z$, and that nudge in $z$ nudges $p$, and that nudge in $p$ nudges the loss, then the total effect of $w$ on the loss is simply those three nudges multiplied together:

$$\frac{\partial\mathcal{L}}{\partial w} = \frac{\partial\mathcal{L}}{\partial p} \frac{\partial p}{\partial z} \frac{\partial z}{\partial w}.$$

The individual derivatives are

$$\frac{\partial\mathcal{L}}{\partial p} =-\frac{y}{p}+\frac{1-y}{1-p},$$ $$\frac{\partial p}{\partial z}=p(1-p), \qquad \frac{\partial z}{\partial w}=x.$$

Multiply those three together and something nice happens: the $p(1-p)$ from the sigmoid cancels the messy denominator from the loss, and almost everything collapses:

$$\boxed{ \frac{\partial\mathcal{L}}{\partial w}=(p-y)x }$$

The bias gradient is even simpler:

$$\frac{\partial\mathcal{L}}{\partial b}=p-y.$$

Before substituting the numbers, check the sign yourself.

Guess the direction first

x = 2, prediction = 0.31, target = 1

The gradient is (prediction − target) × x. Before you compute it, is it positive or negative?

NegativePositive

For $x=2$, $p\approx0.31$, and $y=1$:

$$\frac{\partial\mathcal{L}}{\partial w} =(0.31-1)(2)\approx-1.38.$$

The negative sign makes sense. The correct label is 1, so we need a larger prediction. With a positive input, increasing the weight increases $z$, which increases the prediction.

Update the weight

Gradient descent updates a parameter by moving against its gradient:

$$w_{\text{new}}=w_{\text{old}}-\eta\frac{\partial\mathcal{L}}{\partial w},$$

where $\eta$ is the learning rate.

Using $w=-0.5$, a learning rate of $0.2$, and our gradient of about $-1.38$:

$$w_{\text{new}}=-0.5-0.2(-1.38)\approx-0.224.$$

The weight becomes less negative, the prediction moves toward 1, and the loss falls. The figure below is that same update, drawn. The curve is the loss for every possible weight; the marble is where we are now. Take a step and it rolls downhill, leaving a breadcrumb behind — and the dashed tangent line is the gradient itself, steep where there is work to do and nearly flat once the loss bottoms out.

Roll downhill

x = 2, target = 1, learning rate = 0.2

weight

-0.500

prediction

0.310

gradient

-1.380

loss

1.171

Take one stepTake fiveReset

The dashed line is the gradient — the slope of the curve right under the marble. Each step slides the marble downhill by that slope, and the slope flattens out as the loss bottoms out.

The complete scalar training step is only a few lines:

z = w * x + b
p = sigmoid(z)
loss = binary_cross_entropy(p, y)

dw = (p - y) * x
db = p - y

w -= learning_rate * dw
b -= learning_rate * db

That is backpropagation for one sigmoid neuron. The forward pass stores values such as $x$ and $p$. The backward pass reuses them to calculate dw and db.

Add one hidden neuron

One neuron can only draw a single straight boundary between the two classes. Most real problems are not that tidy, so networks stack neurons into hidden layers that can bend the boundary. To see what that does to backprop, I will add exactly one hidden neuron with a tanh activation:

$$\begin{aligned} a &= w_1x+b_1, \\ h &= \tanh(a), \\ z &= w_2h+b_2, \\ p &= \sigma(z). \end{aligned}$$

The gradient for $w_1$ has a longer route to travel. Applying the same chain rule gives

$$\frac{\partial\mathcal{L}}{\partial w_1} =(p-y)w_2(1-h^2)x.$$

Every factor is one operation from the forward pass, read in reverse. Click them in order and watch the running product build up:

Build the product, factor by factor

An example forward pass, worked backward.

1. output error(p − y)2. second weight× w₂3. tanh slope× (1 − h²)4. input× x

-0.380.800.841.00= -0.380

How far the prediction landed from the target. Every other factor just scales this number.

This is the part that finally made it click for me. A deeper network does not need a new kind of math. It needs more local derivatives, kept in order and multiplied — the same move I just made for one neuron, run more times.

Here is the same calculation in code:

# Forward
a = w1 * x + b1
h = np.tanh(a)
z = w2 * h + b2
p = sigmoid(z)

# Backward
dz = p - y
dw2 = dz * h
db2 = dz

dh = dz * w2
da = dh * (1.0 - h**2)
dw1 = da * x
db1 = da

Names such as dz, dh, and da mean “the derivative of the loss with respect to this variable.” Writing the backward pass in the reverse order of the forward pass makes it much easier to check.

From one example to a batch

Training one example at a time is slow. With a batch, x, p, and y become NumPy arrays. The same formulas still work element by element, and then I average the gradients:

# x, p, and y each contain one value per training example
dz = p - y
dw = np.mean(dz * x)
db = np.mean(dz)

For a layer with several inputs and neurons, the weighted sum becomes matrix multiplication:

Z = X @ W + b

and the weight gradient is

dW = X.T @ dZ / X.shape[0]
db = np.mean(dZ, axis=0, keepdims=True)

The transpose is important because dW must have the same shape as W. When I debug this code, I print every array shape before changing the model or learning rate.

Check the derivative numerically

I do not have to trust my handwritten derivative. I can slightly increase and decrease the weight and measure how much the loss changes:

$$\frac{\partial\mathcal{L}}{\partial w} \approx \frac{\mathcal{L}(w+\varepsilon)-\mathcal{L}(w-\varepsilon)}{2\varepsilon}.$$

This is a finite-difference gradient check, and it is the best debugging trick I picked up on this project. It is far too slow to train with, but as a second opinion it is priceless: if my hand-derived formula and this measured slope agree, the derivation is almost certainly right. One catch — smaller $\varepsilon$ is not always better. Shrink it too far and floating-point rounding wrecks the measurement. Drag the slider and you can watch the gap find its sweet spot and then get worse again.

Two ways to find the same slope

Hand-derived vs. measured. They should nearly match.

epsilon = 10^-4

-4.00

backprop

-1.3799490

finite difference

-1.3799490

gap

1.08e-9

Shrink epsilon and the gap shrinks too — until it is so small that floating-point rounding takes over and the gap grows again. There is a sweet spot in the middle.

epsilon = 1e-5

loss_plus = loss_with_weight(w + epsilon)
loss_minus = loss_with_weight(w - epsilon)
numeric_dw = (loss_plus - loss_minus) / (2 * epsilon)

assert np.isclose(dw, numeric_dw, rtol=1e-5, atol=1e-7)

If the two values disagree, I check the backward pass before trying different hyperparameters.

The mistakes I check first

These caused most of the problems in my NumPy implementations:

1. A missing transpose. The matrix dimensions may still broadcast into something legal but incorrect.
2. A bias summed over the wrong axis. For a batch, bias gradients should combine examples, not output neurons.
3. A derivative evaluated at the wrong value. For example, the tanh derivative uses the hidden activation from the same forward pass.
4. Updating weights too early. Calculate all gradients first, then update every parameter.
5. A learning rate that is too large. Correct gradients can still overshoot the useful region.
6. Numerical overflow. Clip probabilities before taking a logarithm.

A small test helps more than a long training run. I usually try to make the network overfit a few examples and run gradient checks on several weights. If it cannot learn four examples, giving it four thousand will not fix the implementation.

What PyTorch is doing

PyTorch records the operations in the forward pass. When loss.backward() runs, it visits those operations in reverse, applies each local derivative, and stores the resulting gradients on the parameters.

The framework handles the bookkeeping, but the calculation is the same one used here:

$$\text{upstream gradient}\times\text{local derivative}.$$

That is the whole secret. loss.backward() is not doing anything I cannot do by hand for one neuron — it is just doing it for millions of them without complaining. I still reach for PyTorch when I actually want to train something; writing your own autograd for a real model would be a waste of a good library. But rebuilding these few equations in NumPy is the reason the magic trick stopped looking like magic. Now when a gradient comes out wrong, I know where to look instead of just trying different learning rates and hoping.