AI tutorials Mighty Professional
Build a Language Model · Learning from data

Neural Networks
from Scratch

Some decisions have rules too messy to write by hand, like whether a photo shows a cat. A neural network learns the rule from labeled examples instead. This page builds one from plain arrays and loops, with no libraries, up to a classifier that learns to separate two interleaved spirals of points, and you train it live in your browser.

Time~50 min LevelBeginner-friendly PrereqsComfortable with any programming language MathHigh-school algebra; calculus and basic linear-algebra notation help, but the prose carries you if not
◂ Build a Language Model Phase 1 · Learning from data Next · Automatic Differentiation ▸
What you'll be able to do by the end

Compute a neuron's and a layer's output by hand. Score a prediction with softmax and cross-entropy, and say why that pair is the usual choice for classifiers. Work out every weight's gradient with backpropagation and check one against a nudge-and-measure estimate. Choose a learning rate, activation and weight initialization that train, and read training and validation loss curves to tell overfitting from underfitting. You'll also have a classifier of roughly 150 to 200 lines, written in parallel in JavaScript, Python, C++ and Rust, that you can train on this page.

01Why neural networks?

Say you need code that decides whether a photo contains a cat. You can't write if (pixels look cat-shaped): no hand-written rule survives kittens, hairless breeds, plush toys and odd camera angles. What you can do is collect a few thousand photos labeled "cat" or "not cat", write a flexible function with millions of adjustable numbers, and let an algorithm adjust those numbers until the function labels the photos correctly. The rules are fitted to examples instead of programmed.

A neural network is that flexible function, and its adjustable numbers are called parameters. Training is the algorithm that adjusts them. A network trained this way can also label photos it has never seen, though it doesn't have to, and §14 shows how to check. The page builds each piece in order: a neuron and layers of neurons (§2 to §5), a loss that scores how wrong the network is (§6), gradient descent and backpropagation to shrink that score (§7 to §11), then a complete classifier you train yourself (§12 to §15).

Where the idea came from

Frank Rosenblatt's perceptron (1958) was the first artificial neuron with a rule for learning its weights[1]. In 1969 Minsky and Papert proved that a single-layer perceptron can't compute XOR[3], and neural-network research lost most of its funding and attention through the 1970s. Rumelhart, Hinton and Williams's 1986 paper Learning representations by back-propagating errors[2] is the one most often credited with making backpropagation the standard way to train multi-layer networks.

They didn't invent it. Seppo Linnainmaa described the general algorithm, reverse-mode automatic differentiation, in 1970[16], and Paul Werbos proposed applying it to neural networks in his 1974 PhD thesis. The 1986 paper is where it reached a wide audience, with experiments showing hidden layers learning useful internal representations.

02The neuron: a tiny decision maker

Start with a smaller decision. A spam filter looks at a few measurements of an email, say how many links it contains and whether the sender is in your contacts, and has to answer yes or no. An artificial makes exactly this kind of decision. It multiplies each measurement by a number that says how much that measurement matters, adds up the results, and answers yes if the total clears a bar.

Those per-input numbers are the neuron's weights. A positive weight pushes toward yes, a negative one pushes toward no, and a weight near zero means the input barely counts. The neuron also has one extra number, its bias, which is added to the total and sets how high the bar is.

Worked example

Inputs: x1 = 3 links, x2 = 0 (sender not in your contacts). Weights: w1 = 0.5, w2 = −2. Bias: b = −1.

Total: 0.5×3 + (−2)×0 + (−1) = 0.5. That's above zero, so the neuron says spam.

The same email from a contact (x2 = 1): 1.5 − 2 − 1 = −1.5, so it says not spam. The large negative weight on "known sender" outvotes the links.

That total is the neuron's pre-activation, written z. With inputs x1, x2, …, xn, weights w1, w2, …, wn and bias b, it's one multiply per input, all added up, plus the bias:

eq. 1 · pre-activation z = w1x1 + w2x2 + … + wnxn + b

In words: multiply each input by its weight, add the products, then add the bias. The result z is a single number. Hover any letter in the equation to see what it stands for.

Multiplying matching entries of two arrays and adding them up is a dot product, so in vector form z = w · x + b: a few lines of code in any language.

Need a refresher on weighted sums & dot products?

A weighted sum is exactly what it sounds like: a sum where each thing being added gets multiplied by its own number first. If a student's final grade is 0.6 × exam + 0.3 × homework + 0.1 × attendance, those 0.6, 0.3, 0.1 are weights.

The dot product is the weighted sum written compactly. Given two arrays w = [0.6, 0.3, 0.1] and x = [85, 92, 100], the dot product is 0.6×85 + 0.3×92 + 0.1×100 = 88.6. In JavaScript: w.reduce((s, wi, i) => s + wi * x[i], 0).

For a neuron, w is the array of weights, x is the array of inputs, and the dot product plus the bias gives the pre-activation z.

The neuron then turns z into its output with an activation function. Rosenblatt's perceptron, the first artificial neuron with a learning rule[1], used the simplest one there is: a hard threshold that turns the score into a yes or a no.

eq. 2 · perceptron output output = 1  if  z ≥ 00  if  z < 0

In words: if z is zero or positive, the neuron fires and outputs 1. If z is negative, it stays silent and outputs 0. The whole rule is one if-statement.

Two inputs make the rule easy to draw. Plot each input pair as a point on a plane. The points where w1x1 + w2x2 + b = 0 form a straight line. Points on one side give z > 0 and the neuron fires; points on the other side give z < 0 and it stays silent. Changing a weight tilts the line, and changing the bias slides it without changing its angle.

The widget starts with w1 = w2 = 1 and b = −0.3, and its dot sits at (0.4, 0.6), so z = 0.4 + 0.6 − 0.3 = 0.7 and the neuron fires. Drag the dot across the line and the output flips.

Live · Perceptron
z = w·x + b
0.000
output
fires (1)
Drag the dot, or move a slider. The shaded background shows where the neuron would fire (purple) or stay silent (blue), and the straight purple line between them is the decision boundary, where z = 0. The readout gives z for the dot and whether the neuron fires.

So a single perceptron draws one straight line and can only answer by which side of it a point is on. It can separate two groups of points only if they're linearly separable, meaning one straight line divides them. No choice of weights makes a single perceptron compute XOR, which outputs 1 when exactly one of its two inputs is 1[3]. That limit is why networks stack neurons in layers (§4). First, though, the hard threshold has a more immediate problem.

03Smoothing it out: the sigmoid neuron

Training adjusts weights in small steps and needs to know whether each small step helped. With a hard threshold it can't tell. Push z from −0.001 to +0.001 and the output jumps from 0 to 1; anywhere else, a small weight change moves the output by exactly nothing. There's no "almost firing." In calculus terms, the step's slope is zero everywhere except at the jump, so a learning rule that follows the slope gets no signal.

The fix is to replace the jump with a smooth S-shaped curve, so a small change in z always makes a small change in the output. Rumelhart, Hinton and Williams used the logistic sigmoid[2]. It squashes any number into the range 0 to 1: very negative inputs land near 0, very positive ones near 1, and zero lands exactly in the middle.

Worked example

z = 0: 1 / (1 + e0) = 1 / 2 = 0.5.

z = 2: e−2 ≈ 0.135, so 1 / 1.135 ≈ 0.881.

z = −2: e2 ≈ 7.39, so 1 / 8.39 ≈ 0.119. The two answers add to 1, because the curve is symmetric around z = 0.

The sigmoid computes that squashed value, which a network reads as how strongly the neuron fires:

eq. 3 · sigmoid σ(z) = 11 + e−z

In words: raise e (about 2.718) to the power −z, add 1, and take one over the result. Large positive z gives nearly 1, very negative z gives nearly 0, and z = 0 gives exactly 0.5. Around z = 0 the curve is gently sloped, so nudging z a little nudges the output a little.

Training also needs the curve's slope at each point, because backpropagation (§9) multiplies by it to work out how much each weight is to blame. At z = 0 the slope is 0.5 × (1 − 0.5) = 0.25. At z = 2 it's 0.881 × 0.119 ≈ 0.105. Both come from one short formula for the slope, written σ′ ("sigma prime"):

eq. 4 · sigmoid derivative σ′(z) = σ(z) · (1 − σ(z))

In words: the slope of the sigmoid at z is the sigmoid's value there times one minus that value. Code that has already computed σ(z) gets the slope with one multiply.

The slope is largest, 0.25, at z = 0, the curve's steepest point (its inflection point), and it decays toward zero in both directions. Turn on "show derivative" in the widget below and slide the input to see it.

Wait, what's a derivative again?

A derivative answers one question: if I nudge the input a tiny bit, how much does the output change? It's the slope of the function's graph at that point. Steep = big change, flat = small change.

For f(x) = x², the derivative is f′(x) = 2x. At x = 3, the slope is 6, meaning bumping x from 3.000 to 3.001 bumps the output by about 0.006.

For learning, derivatives say which direction to move each weight to reduce the loss. Without them, training would be guessing.

The prime symbol (′) is shorthand for "the derivative of." So σ′(z) reads as "sigma-prime of z," meaning "the slope of the sigmoid at z."

Live · Activation Functions
f(z)
0.5000
f′(z)
0.2500

f′(z) is the slope of the curve: how much the output changes per unit of input. Training needs it to know which way to nudge the weights.

The solid purple curve is the selected activation function and the dashed blue curve is its slope; slide z to move the marker along both, and use the buttons to switch functions. At the default, sigmoid at z = 0, the readout shows f(z) = 0.5 and f′(z) = 0.25. Sigmoid and tanh are S-shaped, ReLU and leaky ReLU are piecewise-linear, and GELU is a smoothed ReLU. All five have a slope defined everywhere (ReLU and leaky ReLU have a kink at 0, where code picks one side's slope), but ReLU's is zero for every negative input. §10 explains why those differences matter for deep networks.

04Stacking neurons into networks

One straight line is enough when the two classes sit on opposite sides of a line. Plenty of data doesn't: points of one class can form a ring around the other, or the two classes can wind around each other as two interleaved spirals, the dataset the trainer in §13 learns. No single line separates those.

Put two neurons side by side and you get two lines. Add a second layer of neurons that takes the first layer's outputs as its inputs, and it can combine those cuts into a curved region. With enough neurons, that region can follow a circle or a spiral. A feedforward network is built this way: layers of neurons, where every neuron in one layer takes inputs from every neuron in the layer before. The first layer is the input layer (it just holds the input numbers), the last is the output layer, and the ones in between are hidden layers, named because their outputs aren't part of the network's answer.

Three-layer feedforward network: 2 inputs, 3 hidden units, 2 outputs. A diagram of a small feedforward neural network. Two input neurons on the left are each connected to all three hidden neurons in the middle, which are each connected to two output neurons on the right. Every connection is one weight. input hidden output

The diagram above is a 2-3-2 network: two inputs, three hidden neurons, two outputs. Every line is one weight, and every circle except the inputs holds one bias. The whole thing is a function that turns two input numbers into two output numbers.

Writing every neuron's sum out by hand gets long, so code and math group a layer's numbers. Put the layer's weights in a grid, a matrix W, with one row per neuron and one column per input. Put its biases in a list b. Each neuron's pre-activation is then the dot product of its row with the input, plus its bias.

Worked example

The hidden layer of the forward-propagation widget in §5. Weight rows: (0.9, −0.6), (−0.4, 1.1), (0.6, 0.5). Biases: (0.1, −0.2, 0). Input: x = (0.6, 0.4).

Neuron 1: 0.9×0.6 + (−0.6)×0.4 + 0.1 = 0.54 − 0.24 + 0.1 = 0.40.
Neuron 2: −0.24 + 0.44 − 0.2 = 0.00.
Neuron 3: 0.36 + 0.20 + 0 = 0.56.

Sigmoid of each: σ(0.40) ≈ 0.599, σ(0) = 0.5, σ(0.56) ≈ 0.636. Those three numbers are the layer's output, and they become the next layer's input.

For layer number l, write its weight matrix W(l), its biases b(l), and its outputs (its activations) a(l). These two lines compute one layer's outputs from the previous layer's, which is all a network does to make a prediction:

eq. 5 · layer forward pass z(l) = W(l) a(l−1) + b(l) a(l) = σ(z(l))

In words: multiply the previous layer's outputs by this layer's weight matrix (each row gives one neuron's weighted sum), add the biases to get the pre-activations z, then apply the activation σ to each entry separately. The superscript (l) only labels which layer. A whole network is a few of these steps in a row.

Refresher: matrix-vector multiplication

A matrix is a rectangular grid of numbers. A vector is a column (or row) of numbers. Multiplying a matrix by a vector produces a new vector.

If W has 3 rows and 2 columns, and x is a 2-element vector, then W × x is a 3-element vector. Each output element is the dot product of one row of W with x.

For neural networks, that's exactly what's happening: each row of the weight matrix corresponds to one neuron, and dotting it with the previous layer's outputs gives that neuron's pre-activation. Three rows = three neurons in this layer.

In JavaScript: z[i] = W[i].reduce((s, w, j) => s + w * x[j], 0) + b[i].

Can one hidden layer learn anything?

In principle, close to it. A network with just one hidden layer can approximate any continuous function on a bounded input range as closely as you like, given enough neurons and a nonlinear activation such as the sigmoid. This is the universal approximation theorem (Cybenko 1989[17], Hornik 1991[18]). It says nothing about how many neurons that takes or whether training will find the weights. In practice, deeper networks often fit the same function with far fewer neurons than one very wide layer, which is a large part of why depth took over.

05Forward propagation

Making a prediction means running the layers in order. The input feeds the first hidden layer, that layer's outputs feed the next, and the last layer's outputs are the answer. This is forward propagation: apply the two layer equations from §4 one layer at a time and pass the activations forward.

For a classifier, the answer should be one probability per class. The output layer below therefore applies softmax instead of sigmoid: softmax turns its list of output numbers into probabilities that are all positive and add up to 1 (§6 shows how).

Worked example

Continue §4's example. The hidden outputs (0.599, 0.5, 0.636) go into the output layer, whose weight rows are (0.8, −0.7, 0.5) and (−0.5, 0.6, 0.9), with biases (0, 0.1).

Output 1: 0.8×0.599 − 0.7×0.5 + 0.5×0.636 ≈ 0.447.
Output 2: −0.5×0.599 + 0.6×0.5 + 0.9×0.636 + 0.1 ≈ 0.673.

Softmax turns (0.447, 0.673) into probabilities (0.44, 0.56), so the network picks class 2, but without much confidence.

The widget runs that same pass. Set the inputs, then click STEP to advance one phase at a time, or RUN to animate the whole pass:

Live · Forward Propagation 0 · inputs fed in.

Forward-propagation visualization

An animated two-to-three-to-two network. Each step lights up the next phase: feeding inputs in, computing hidden pre-activations, applying the activation function, computing output logits, and finally applying softmax. The current stage is also reported in the text status line.

click STEP to advance, or RUN to animate.
Each click of Step lights up the next phase, left to right; the status line names the phase. Edge thickness shows a weight's size, purple for positive and blue for negative, and the number on each edge is the weight. Node fill shows the activation value. At the default inputs (0.6, 0.4) the pass ends at probabilities [0.44, 0.56], the worked example's answer; move the sliders to see the prediction change.

In code, forward propagation is a short loop over the layers. This sketch applies sigmoid at every layer for brevity; the full listing in §12 leaves the output layer as raw numbers for softmax.

forward.js · pseudo-implementation
function forward(network, x) {
  let a = x;                              // activations of the previous layer
  const activations = [a];                // keep every layer's activations for backprop

  for (let l = 0; l < network.W.length; l++) {
    const W = network.W[l];               // weight matrix for this layer
    const b = network.B[l];               // bias vector for this layer
    const z = matVecMul(W, a).map((zi, i) => zi + b[i]);
    a = z.map(sigmoid);                  // element-wise activation
    activations.push(a);
  }
  return activations;                      // last entry is the network's output
}

activations[activations.length - 1] is the prediction. The function keeps every layer's activations because backpropagation (§9) reuses them to compute the gradients.

06Measuring wrongness: loss functions

A freshly built network has random weights and makes random predictions. To improve it, training needs one number that says how wrong it is across the training examples, so it can work out which way to nudge each weight to make that number smaller. That number is the loss (also called the cost). Lower is better, and a perfect prediction scores zero.

Mean squared error

The simplest loss, used in Nielsen's book[4] and many introductory examples, is mean squared error (MSE): measure how far each prediction is from its target, square that distance, and average over the examples. Squaring makes every error positive and punishes big misses more than small ones.

Worked example

Target y = (1, 0): the example belongs to class 1 of 2. Prediction ŷ = (0.8, 0.3).

Squared distance: (1 − 0.8)² + (0 − 0.3)² = 0.04 + 0.09 = 0.13. With the ½ factor from the formula below, this example's loss is 0.065.

Written out for n examples, with true output y and prediction ŷ for each:

eq. 6 · MSE C = 12n ∑x ‖y(x) − ŷ(x)‖2

In words: for each training example, take the distance between the prediction and the truth, square it, add those up, and divide by twice the number of examples. The ½ is there for convenience: it cancels when you take the slope, which comes out as (ŷ − y) instead of 2(ŷ − y).

What does Σ (the big capital sigma) actually mean?

Σ is shorthand for "sum up the following expression for every value of the index." The little symbol underneath says what to vary; the expression to its right is what gets added.

Plain English: ∑_{i=1}^{3} i² means "add up i² for i from 1 to 3," which is 1 + 4 + 9 = 14. In JS that's [1,2,3].reduce((s, i) => s + i*i, 0).

So ∑_x ‖y(x) − ŷ(x)‖² reads as: "loop over every training example x; for each one, compute the squared error; add them all up."

Cross-entropy: the better choice for classification

MSE works, but when the network is choosing between classes (spam or not, cat or dog, which word comes next) the usual loss is cross-entropy. Instead of measuring numeric distance, it asks how much probability the model gave the correct answer, and charges a lot when that probability is small.

That needs probabilities first. The output layer's raw numbers, called logits, can be any size and sign. Softmax makes each one positive by raising e to its power, then divides each by the total so they add up to 1.

Worked example

Logits (2.0, 0.5). Exponentiate: e2.0 ≈ 7.389 and e0.5 ≈ 1.649. Total ≈ 9.038.

Probabilities: 7.389 / 9.038 ≈ 0.818 and 1.649 / 9.038 ≈ 0.182. The bigger logit gets the bigger share, and the shares add to 1.

If class 1 is correct, the cross-entropy loss is −log(0.818) ≈ 0.201.

Softmax computes those probabilities for logit i out of all the logits j:

eq. 7 · softmax softmax(z)i = ezi∑j ezj

In words: raise e to the power of each logit, which makes every one positive, then divide each by the total of all of them. The result is a list of probabilities that add up to 1, in the same order of size as the logits.

Cross-entropy then scores those probabilities against the true label, written as a list y with a 1 for the correct class and 0 for the rest (a one-hot label):

eq. 8 · cross-entropy C = −∑i yi log(ŷi)

In words: only the correct class's term survives, because every other yi is 0. The loss is minus the log of the probability the model gave the correct class. Probability near 1 gives a loss near 0; probability near 0 sends the loss toward infinity.

What's "log"? Why use it for loss?

log here is the natural logarithm (sometimes written ln): the inverse of ex. So log(1) = 0, log(0.5) ≈ −0.69, log(0.1) ≈ −2.30, and log approaches −∞ as the input approaches 0.

Why use it for loss? It turns multiplications of probabilities into additions, which keeps the math and the numbers manageable. It punishes confidently wrong answers heavily: predicting 0.001 for the correct class gives a loss around 6.9, while predicting 0.1 gives 2.3. And paired with softmax, the gradient comes out to prediction − truth, a simple signal to compute.

If the true class is class 0 (so y = [1, 0]) and the model predicts ŷ = [0.9, 0.1], the loss is −log(0.9) ≈ 0.105. If the model says ŷ = [0.1, 0.9], the loss is −log(0.1) ≈ 2.303. Confident wrong answers cost a lot and confident right ones cost almost nothing, which pushes training hardest on the examples it gets most wrong.

Softmax + cross-entropy

When you pair softmax with cross-entropy and work through the calculus, almost everything cancels. You're left with one line: the gradient of the loss with respect to the logits is (ŷ − y), predicted minus actual. It stays large exactly when the model is confidently wrong, whereas sigmoid + MSE multiplies the error by a σ′ that is nearly zero there. That, plus the probabilistic reading of the output, is why softmax + cross-entropy is the standard pairing for classifiers.

07Learning by descent

The loss turns the network's weights into one number, and training means finding weights that make that number small. A network with millions of weights can't try combinations one by one. The standard strategy is gradient descent: repeatedly move every weight a little in whichever direction lowers the loss.

Picture standing on a hillside in dense fog. You can't see the bottom of the valley, but you can feel the slope under your feet. The gradient is that slope written as a vector: it points uphill, so you step the opposite way, feel the slope again, and step again. Repeat until the ground is flat, which is a low point, though not necessarily the lowest one.

Worked example

One weight, with loss C(w) = (w − 3)². The best weight is 3. Start at w = 1, where the loss is 4.

The slope there is 2(w − 3) = −4: negative, so increasing w lowers the loss. Step size η = 0.1: w becomes 1 − 0.1 × (−4) = 1.4, and the loss drops to (1.4 − 3)² = 2.56. The next step reaches 1.72 and a loss of about 1.64.

With η = 1.1 instead, the first step goes to 1 + 1.1 × 4 = 5.4. That's past 3, and the loss rises to 5.76: the step overshot the bottom.

With many weights, do the same thing for each one. For a weight w, compute ∂C / ∂w, how much the loss changes per unit change in that weight with every other weight held still. (If the curly-d notation is new to you, expand the box below.) The update rule then moves each weight a small step against its slope:

What's a partial derivative (∂)?

A regular derivative df/dx answers: "if I change x by a tiny bit, how does f change?" It works when f is a function of just one variable.

When f depends on many variables (like a loss that depends on every weight in the network), the question becomes: "if I change just w by a tiny bit and hold everything else fixed, how does f change?" That's a partial derivative, written with a curly ∂ instead of a straight d.

So ∂C / ∂w reads as: "the partial derivative of the loss C with respect to the weight w." It's a single number that says how sensitive the loss is to this particular weight, treating all other weights as constants.

The full vector of all these partials is the gradient. It points in the direction of steepest increase, so training moves the weights the opposite way to go downhill.

eq. 9 · gradient descent update w ← w − η · ∂C∂w

In words: the arrow means "assign." Replace each weight with itself minus η times its slope. Do this for every weight, every step, and the loss goes down, provided η is small enough.

The constant η ("eta") is the learning rate, how big a step to take. Too small and training crawls; too big and each step overshoots the valley floor, as the η = 1.1 step in the example did, so the weights oscillate or diverge. It's usually the first setting to tune.

Click anywhere on the loss surface below to drop a starting point, then hit DESCEND; the yellow path traces each step. On the elongated valley the loss is ½(w² + 12b²), so its slope across the valley grows 12 times faster than along it. The path bounces between the walls once η passes 1/12 ≈ 0.08 and flies off once it passes 2/12 ≈ 0.17 (the default 0.18 diverges there; lower it and descend again). The other surfaces show a saddle point and two separate minima:

Live · Gradient Descent
steps
0
loss
···

Click on the surface to place a starting point. Bright regions = low loss, dark = high. Green dots mark the minima. On the saddle surface, start close to the b = 0 line: the path slows to a crawl near the saddle point before tipping into one of the two wells. On the two-minima surface, which well you end in depends only on where you start.

You're looking down on a loss surface over two weights, w and b: bright is low loss, dark is high. Click to place a start, press Descend, and the yellow path traces each step; the readout counts steps and shows the current loss. A real network's surface has one dimension per weight, millions of them instead of two, and gradient descent never sees the whole surface, only the slope at its current point.
Stochastic gradient descent

Computing the gradient over the entire training set is expensive. The variant that makes neural nets practical is stochastic gradient descent (SGD): estimate the gradient from a small random batch of examples (a "mini-batch"), update the weights, then grab another batch. The estimate is noisy, which is acceptable, and the noise can help the optimizer move off saddle points and out of sharp local minima.

The same surface, in three dimensions

The contour map above is the usual way to draw it on paper. With the loss as height, the same kind of surface becomes a 3D terrain. Drag the surface below to rotate it, then hit Descend to see the path slide down the side of the hill:

Live · 3D Loss Surface drag to rotate

Drag the surface to rotate it. The brighter the color, the lower the loss. The yellow trace shows the path of gradient descent.

The height of the surface is the loss over two weights; drag to rotate it, pick a surface, and press Descend to send the yellow path downhill. On the double well, descent starts at w = 0.1, just past the ridge between the two minima, so it rolls into the nearer well and never visits the other one: which minimum gradient descent finds depends on where it starts.

08Beyond SGD: smarter optimizers

Plain SGD has problems. In a long narrow valley it either zigzags between the walls (large η) or crawls along the nearly flat floor (small η). Near a saddle point it slows almost to a stop, and on a flat region it crawls. Real loss surfaces have all three. Most modern training uses SGD with momentum or an adaptive method such as Adam. Each keeps a small running memory of past gradients and uses it to shape the next step.

Momentum: a ball with mass

In a long valley, the gradient keeps pointing the same way along the floor but flips sign from step to step across it. Momentum uses that difference. Plain SGD is a marble that stops the instant you stop pushing it; momentum SGD is a ball with mass. It keeps a running blend of recent gradients, picks up speed where they agree, and cancels out where they keep flipping.

Worked example

Blend factor β = 0.9, learning rate η = 0.1, velocity v starting at 0. Each step, v becomes 0.9 v − 0.1 g.

Gradient g = 1 every step (the valley floor): v goes −0.1, −0.19, −0.271, and keeps growing toward −1, ten times the plain SGD step of −0.1.

Gradient flipping +1, −1, +1, −1 (across the valley): v goes −0.1, 0.01, −0.091, 0.018. It stays small, so the zigzag is damped.

The update keeps a velocity v and moves the weights by it instead of by the raw gradient:

Momentum SGD v ← β v − η · ∇C w ← w + v

In words: each step, keep a fraction β (typically 0.9) of the previous velocity and add a fresh step of −η times the current gradient. Then move the weights by that velocity. Directions where the gradient agrees build up speed; directions where it alternates cancel.

RMSProp: adapt the step size per parameter

Different parameters can have wildly different gradient scales. A weight in the output layer might routinely see gradients of size 1.0; a weight five layers deep might see 0.001. Using the same learning rate for both is a compromise neither one wants. RMSProp[12] tracks a running average of the squared gradient for each parameter and divides each update by the square root of that average. Big-gradient parameters get smaller steps; small-gradient parameters get bigger ones.

Worked example

Two weights. One keeps seeing gradients of about 1.0, the other about 0.001. After many steps, the running averages of their squared gradients settle near 1.0 and 0.000001, whose square roots are 1.0 and 0.001.

Steps: η × 1.0 / 1.0 = η, and η × 0.001 / 0.001 = η. Both weights now move about the same distance per step, where plain SGD would move the second one a thousand times less.

The update keeps a running average s of each parameter's squared gradient and divides the step by its square root:

RMSProp s ← ρ s + (1 − ρ) · (∇C)2 w ← w − η · ∇Cs + ε

In words: keep a running average of each parameter's squared gradient, an exponential moving average in which each step keeps a fraction ρ of the old average and mixes in 1 − ρ of the new value, so recent steps count most. Then divide each update by the square root of that average. The tiny ε (typically 10−8) prevents division by zero when a parameter's gradients have all been zero so far.

Adam: momentum + RMSProp + bias correction

Adam[13] combines the previous two ideas. It keeps a momentum-style running average of the gradient, m, and an RMSProp-style running average of the squared gradient, v. Both start at zero, so for the first steps they're too small. Adam divides each by a correction factor that undoes this, a step called bias correction. Adam and its variant AdamW are the most common default optimizers in deep learning today.

Worked example

First step, default β1 = 0.9 and β2 = 0.999, gradient g = 2.

m = 0.1 × 2 = 0.2 and v = 0.001 × 4 = 0.004. Uncorrected, the step would be η × 0.2 / √0.004 ≈ 3.16 η.

Corrected: m̂ = 0.2 / (1 − 0.9) = 2 and v̂ = 0.004 / (1 − 0.999) = 4, so the step is η × 2 / √4 = η.

The full update, where t counts the steps taken so far:

Adam m ← β1 m + (1 − β1) · ∇C
v ← β2 v + (1 − β2) · (∇C)2
m̂ = m1 − β1t, v̂ = v1 − β2t
w ← w − η · m̂v̂ + ε

In words: track how the gradient has been pointing on average (m) and how big it has been on average (v). Divide each by 1 − βt to undo their start at zero; that factor approaches 1 as training goes on. Then take a momentum-style step, divided per parameter by the typical gradient size.

Racing the four optimizers down the same surface shows the differences. On the Rosenbrock function (a banana-shaped valley that is a standard optimizer stress test; the widget uses a rescaled version) at the default η = 0.05, plain SGD has covered only the first stretch of the curve after 320 steps, Momentum and Adam zigzag across the valley before settling onto its floor, and RMSProp follows the floor from the start and gets closest to the minimum. Toy surfaces like these show how each update rule behaves; they don't rank optimizers on real networks.

Live · Optimizer Race
SGD Momentum RMSProp Adam step 0
Press Race to start four optimizers from the same point: red is SGD, yellow Momentum, green RMSProp, purple Adam. Each one turns the gradient at its current position into a step in its own way, so the paths split apart. Try the other surfaces and learning rates. The Rosenbrock function is a common stress test: a narrow curved valley with a single minimum at (1, 1).

09Backpropagation: every gradient in one pass

Gradient descent needs ∂C/∂w for every weight in the network, potentially millions of them. The naive approach nudges one weight, reruns the whole forward pass, and measures the change in loss, once per weight: millions of forward passes per update. gets all the gradients from one backward sweep instead. It starts at the loss, works back through the layers, and reuses the numbers the forward pass already computed.

The quantity it passes backward is each neuron's error, written δ ("delta"): how much the loss changes per unit change in that neuron's pre-activation z. Once a neuron's error is known, the gradients of the weights and bias feeding into it follow with one multiply each.

Worked example

The smallest network that shows the idea: input x = 2, one hidden ReLU neuron with weight w1 = 0.5, one output with weight w2 = 3 and no activation, no biases, and squared-error loss C = ½(ŷ − y)² with target y = 2.

Forward: h = ReLU(0.5 × 2) = 1, ŷ = 3 × 1 = 3, C = ½(3 − 2)² = 0.5.

Backward: the output's error is ŷ − y = 1. The gradient for w2 is (what flowed into w2) × (error) = h × 1 = 1. Send the error back through w2 and through the ReLU, whose slope is 1 because its input was positive: the hidden error is 3 × 1 × 1 = 3. The gradient for w1 is x × 3 = 6.

Check by nudging: w1 = 0.501 gives h = 1.002, ŷ = 3.006 and C ≈ 0.50602, a change of about 0.006 for a nudge of 0.001, a slope of about 6.

Every number the backward pass used (h, x, w2 and the ReLU's slope) came from the forward pass or the stored weights, so nothing had to be recomputed. Nielsen states the general version as four equations[4]. Two pieces of notation first: ⊙ means multiply two equal-length lists slot by slot (the Hadamard product), and ∇aC is the list of slopes of the loss with respect to each output activation.

BP1 starts the backward pass at the output layer. It gives each output neuron's error from the loss and the output activation's slope:

BP1 · output layer error (general form) δ(L) = ∇aC ⊙ σ′(z(L))

In words: output error = (how the loss responds to each output activation) times (the activation's slope at that output), slot by slot. This is the general form, valid for any output activation and loss; it's what you'd use for sigmoid outputs with mean squared error. In the worked example the output had no activation (slope 1), so the error was just ŷ − y.

The code on this page leaves the output layer as raw logits and uses softmax with cross-entropy. Push those two through the chain rule and the σ′ factor, the softmax Jacobian (the table of how every probability depends on every logit) and most of the algebra cancel, leaving the one-liner from §6:

BP1′ · softmax + cross-entropy shortcut δ(L) = ŷ − y

In words: the output error is the predicted probabilities minus the one-hot label. For probabilities (0.6, 0.4) with class 2 correct, the label is (0, 1) and the error is (0.6, −0.6). There's no σ′ to evaluate and no Jacobian to build; the code in §12 uses exactly this line.

BP2 moves the error back one layer. It's the step that turned an output error of 1 into a hidden error of 3 in the worked example:

BP2 · propagate error backward δ(l) = ((W(l+1))⊤ δ(l+1)) ⊙ σ′(z(l))

In words: error at layer l = (next layer's weights, transposed, times next layer's error), then times this layer's activation slope, slot by slot. The transpose (rows and columns swapped) sends each error back along the same weights it came through. Repeat it layer by layer down to the first hidden layer.

The last two equations turn errors into gradients. For a bias:

BP3 · gradient wrt. bias ∂C∂bj(l) = δj(l)

In words: the bias gradient is the neuron's error itself. A bias adds straight into z, so one unit of bias moves z by exactly one unit.

And for a weight:

BP4 · gradient wrt. weight ∂C∂wjk(l) = ak(l−1) δj(l)

In words: weight gradient = (activation flowing in from the upstream neuron) times (error of the downstream neuron). A connection gets credit or blame in proportion to the signal that went through it. In the worked example, w1's gradient was x × 3 = 6.

The chain rule, in one minute

The chain rule is the calculus rule for differentiating composed functions. If y = f(g(x)), then dy/dx = f′(g(x)) × g′(x). In words: multiply the slopes through the chain.

A neural network is a long chain of compositions: input → weights → activation → weights → activation → … → loss. To get the slope of the loss with respect to an early weight, the chain rule says: multiply together the slopes of every step in between.

That multiplication chain is what BP2 does: pulling the error δ through the network step by step, multiplying by each layer's activation slope on the way. Backpropagation is just the chain rule, applied in an order that reuses computation instead of redoing it for every weight.

The four equations all follow from the chain rule of multivariable calculus[4], and the algorithm follows directly from them:

  1. Run a forward pass; cache every z(l) and a(l).
  2. Compute δ(L) at the output using BP1.
  3. For l = L − 1 down to 1: compute δ(l) using BP2.
  4. For each layer, accumulate gradients using BP3 and BP4.
  5. Update weights: W(l) ← W(l) − η · ∂C/∂W(l) and similarly for biases.

Per layer, the forward pass does one matrix-vector product and the backward pass does two (one to send δ back through the weights, one outer product for the weight gradient). A full gradient therefore costs about three forward passes, however many weights the network has, instead of one forward pass per weight. That ratio is why backprop scales to billions of parameters.

10Why hidden layers moved off sigmoid

Train a ten-layer network with sigmoid in every hidden layer and the last layers learn while the first ones barely change. BP2 explains why: every layer the error passes back through multiplies it by that layer's activation slope, σ′(z). The sigmoid's slope is at most 0.25, at z = 0, and falls toward zero as z moves away from zero in either direction (0.105 at z = 2, 0.018 at z = 4).

Worked example

Ten sigmoid layers, every neuron in the best case z = 0, so every slope is 0.25.

Slope factors alone: 0.2510 ≈ 9.5 × 10−7, about a millionth.

With every z at 2 instead, each factor is 0.105, and ten of them give about 1.6 × 10−10.

Unless the weights happen to be large enough to compensate, the gradient reaching the first layer is around a millionth of the gradient at the output, and those early weights barely move. This is the vanishing gradient problem[5], and through the 1990s and 2000s it made deep sigmoid and tanh networks very hard to train.

Several fixes arrived around 2006 to 2015 (layer-wise pretraining, better initialization, normalization layers, residual connections). The simplest, popularized around 2010 to 2011, is to replace sigmoid in the hidden layers with the rectified linear unit (ReLU), which passes positive inputs through unchanged and turns negative ones into zero. ReLU(3) = 3 and ReLU(−2) = 0:

eq. 10 · ReLU ReLU(z) = max(0, z)

In words: if z is positive, output z; if it's zero or negative, output 0. In code it's one line: return z > 0 ? z : 0.

The derivative is 1 for positive inputs and 0 for negative ones, so the activation contributes no shrinking factor: the gradient passes through unchanged for any neuron that's "on." Combined with better initialization, ReLU made much deeper networks trainable with plain gradient descent.

ReLU has a known failure mode: a neuron whose pre-activation is negative for every training example produces zero output and zero gradient, so it stops learning for good ("dying ReLU"). Leaky ReLU patches this with a small slope below zero (the widget uses 0.1z; 0.01z is also common). GELU multiplies z by a smooth weight that runs from 0 for very negative z to 1 for large positive z (the probability that a bell-curve random number falls below z), giving a curve with no kink that behaves like ReLU away from zero. GPT-2 uses it in every feed-forward block[6]; many recent large language models use gated variants such as SwiGLU instead. Try them in the activation widget above to see the shapes.

Initialization matters too

If all your weights start at zero, every neuron in a layer computes the same thing and learns the same thing. Gradient descent can never break that symmetry, so networks start from random weights. How big should the random numbers be? Too big and the signal grows layer after layer; too small and it fades away.

The size of random weights is measured by their variance, the average squared distance from their mean; for weights centered on zero, that's just their average square. A neuron with more inputs adds up more random terms, so its weights need to be smaller. Glorot and Bengio showed in 2010[7] that for tanh-like activations, weights with variance 1/nin keep the signal's size roughly constant from layer to layer, where nin is the number of inputs to the layer. ReLU zeroes the negative half of its inputs, which halves the signal's average square at every layer, so for ReLU networks He et al. showed in 2015[8] that you need twice that: variance 2/nin. Get the scale wrong on a deep network and the activations grow or shrink geometrically with depth.

The published Xavier rule

Glorot and Bengio's paper also derives a backward-pass condition, variance 1/nout, and its published "Xavier" rule averages the two to get 2/(nin + nout)[7]. Many libraries use the simpler 1/nin form, as the code on this page does.

The code below uses He initialization (variance 2/nin) when the activation is ReLU and variance 1/nin otherwise, drawn uniformly from [−limit, limit]. A uniform draw on that range has variance limit²/3, so the code sets limit = √(3 × target variance).

Worked example

A ReLU layer with 8 inputs. Target variance: 2 / 8 = 0.25. Limit: √(3 × 0.25) = √0.75 ≈ 0.87, so each weight starts somewhere between −0.87 and 0.87.

With tanh or sigmoid, the target is 1 / 8 = 0.125 and the limit is √0.375 ≈ 0.61.

11The training loop, end to end

Every piece is now in place: the forward pass makes a prediction, the loss scores it, backprop finds each weight's gradient, and the optimizer moves the weights. Training runs those steps over and over on batches of examples. In pseudocode:

training_loop · pseudocode
net = makeNetwork([2, 8, 8, 2])              // shape: 2 in, 2 hidden layers of 8, 2 out
data = loadTrainingData()

for epoch in 1..N:
  shuffle(data)
  for batch in batches(data, size=32):
    batchGradients = zeros_like(net)
    for (x, y) in batch:
      // forward pass (caching every layer's activations), loss, backward pass
      exampleGradients = backprop(net, x, y)
      accumulate(batchGradients, exampleGradients)
    applyUpdate(net, batchGradients / len(batch), lr=η)
  evaluate(net, validation_set)            // loss on held-out data, see §14

Worked example

1,000 training examples, batch size 32: one pass over the data takes 32 batches (31 full ones and a last one of 8), so 32 weight updates.

50 passes: 50 × 32 = 1,600 updates, and the network sees each example 50 times.

Each pass of the outer loop over the whole dataset is one epoch. The same steps (forward pass, loss, gradient via backprop, parameter update, repeat) train everything from LeCun's 1989 USPS digit classifier[14] to today's large language models; what changes are the layer types, the optimizer, and the scale. Karpathy's Zero-to-Hero series[15] follows this loop from a single neuron up to a small GPT.

12A working network, in your language

Below is the complete network (initialization, forward pass, backprop, SGD update) in four languages: JavaScript, Python, C++, and Rust. Pick whichever you're most comfortable with. The algorithm is identical in all four; only the syntax changes. Every variable has a descriptive name, and every non-obvious line gets a comment.

Following along in an editor?

Copy any version into a file with the matching extension. The JavaScript version works in Node.js or any browser. The Python version is pure stdlib (no NumPy). The C++ version compiles with -std=c++11 or later. The Rust version uses only std with a tiny hand-rolled pseudo-random number generator, or PRNG (in real code, use the rand crate). Each is roughly 150 to 200 lines. They are libraries: add a main or a training loop like the one in §11 to run them.

network · pick a language ↓
// ──────────────────────────────────────────────────────────────
// network.js · feed-forward neural net trained with backprop + SGD
// ──────────────────────────────────────────────────────────────

// Activation functions, paired with their derivatives.
// Note: each derivative is expressed in terms of the *post-activation*
// value `a`, not the pre-activation `z`. For our three choices this
// makes the derivative cheap to compute from values we already have.
const ACTIVATIONS = {
  relu: {
    fn:                       z => z > 0 ? z : 0,
    derivativeFromActivation: a => a > 0 ? 1 : 0,
  },
  tanh: {
    fn:                       z => Math.tanh(z),
    derivativeFromActivation: a => 1 - a * a,
  },
  sigmoid: {
    fn:                       z => 1 / (1 + Math.exp(-z)),
    derivativeFromActivation: a => a * (1 - a),
  },
};

// Build a fresh network. `layerSizes` is an array of integers,
// e.g. [2, 8, 8, 2] = 2 inputs, two hidden layers of 8, 2 outputs.
function makeNetwork(layerSizes, activationName = 'relu') {
  const weights = [];   // weights[layer][neuron][input]
  const biases  = [];   // biases[layer][neuron]

  for (let layer = 1; layer < layerSizes.length; layer++) {
    const fanIn  = layerSizes[layer - 1];   // inputs coming in
    const fanOut = layerSizes[layer];       // neurons in this layer

    // Target weight variance: 2/fanIn for ReLU (He init), 1/fanIn otherwise
    // (Xavier-style). Both keep activations from blowing up or vanishing
    // with depth. A uniform draw on [-limit, limit] has variance limit²/3,
    // so limit = sqrt(3 × targetVariance).
    const targetVariance = activationName === 'relu' ? 2 / fanIn : 1 / fanIn;
    const limit = Math.sqrt(3 * targetVariance);

    const layerWeights = [];
    for (let neuron = 0; neuron < fanOut; neuron++) {
      const weightRow = [];
      for (let input = 0; input < fanIn; input++) {
        weightRow.push((Math.random() * 2 - 1) * limit);
      }
      layerWeights.push(weightRow);
    }
    weights.push(layerWeights);
    biases.push(new Array(fanOut).fill(0));   // start biases at zero
  }

  return { layerSizes, weights, biases, activationName };
}

// Forward pass: turn an input vector into an output prediction.
// Returns both the activations (per layer) and the pre-activations,
// because we'll need both for backprop.
function forward(network, inputVector) {
  const activationsPerLayer    = [inputVector.slice()];
  const preActivationsPerLayer = [];
  const activation = ACTIVATIONS[network.activationName];

  for (let layer = 0; layer < network.weights.length; layer++) {
    const layerWeights        = network.weights[layer];
    const layerBiases         = network.biases[layer];
    const previousActivations = activationsPerLayer[activationsPerLayer.length - 1];

    const preActivations    = [];
    const currentActivations = [];
    const isOutputLayer = (layer === network.weights.length - 1);

    for (let neuron = 0; neuron < layerWeights.length; neuron++) {
      // Weighted sum + bias = pre-activation `z`.
      let z = layerBiases[neuron];
      const weightRow = layerWeights[neuron];
      for (let input = 0; input < weightRow.length; input++) {
        z += weightRow[input] * previousActivations[input];
      }
      preActivations.push(z);

      // Output layer stays as raw logits, we apply softmax separately
      // when we need probabilities. Hidden layers use the activation fn.
      currentActivations.push(isOutputLayer ? z : activation.fn(z));
    }

    preActivationsPerLayer.push(preActivations);
    activationsPerLayer.push(currentActivations);
  }

  return { activationsPerLayer, preActivationsPerLayer };
}

// Softmax: turn the raw output logits into a probability distribution.
// Subtracting the max first is a numerical-stability trick, same answer,
// no overflow.
function softmax(logits) {
  const maxLogit = Math.max(...logits);
  const exponentials = logits.map(z => Math.exp(z - maxLogit));
  const total = exponentials.reduce((sum, value) => sum + value, 0);
  return exponentials.map(value => value / total);
}

// Backpropagation for softmax + cross-entropy loss.
// Returns the gradient of the loss with respect to every weight and
// bias in the network, plus the loss itself.
function backprop(network, inputVector, trueClassIndex) {
  const { activationsPerLayer } = forward(network, inputVector);
  const numLayers = network.weights.length;
  const activation = ACTIVATIONS[network.activationName];

  // Allocate zero-initialized gradient arrays with the same shape
  // as the weights and biases.
  const weightGradients = network.weights.map(layer =>
    layer.map(row => row.map(() => 0))
  );
  const biasGradients = network.biases.map(layer => layer.map(() => 0));

  // BP1: at the output, the error simplifies (for softmax+CE) to
  //      (predicted probability) − (one-hot true label).
  const outputLogits     = activationsPerLayer[numLayers];
  const outputProbabilities = softmax(outputLogits);
  let errorAtCurrentLayer = outputProbabilities.slice();
  errorAtCurrentLayer[trueClassIndex] -= 1;

  // Walk backward through the layers, computing gradients as we go.
  for (let layer = numLayers - 1; layer >= 0; layer--) {
    const previousLayerActivations = activationsPerLayer[layer];

    // BP3 + BP4: gradient of the loss wrt this layer's biases and weights.
    for (let neuron = 0; neuron < network.weights[layer].length; neuron++) {
      biasGradients[layer][neuron] = errorAtCurrentLayer[neuron];
      for (let input = 0; input < previousLayerActivations.length; input++) {
        weightGradients[layer][neuron][input] =
          errorAtCurrentLayer[neuron] * previousLayerActivations[input];
      }
    }

    // BP2: propagate the error backward to the previous layer,
    // scaled by the activation function's derivative.
    if (layer > 0) {
      const errorAtPreviousLayer = new Array(network.weights[layer - 1].length).fill(0);
      for (let source = 0; source < errorAtPreviousLayer.length; source++) {
        let weightedError = 0;
        for (let target = 0; target < network.weights[layer].length; target++) {
          weightedError += network.weights[layer][target][source] * errorAtCurrentLayer[target];
        }
        errorAtPreviousLayer[source] =
          weightedError * activation.derivativeFromActivation(activationsPerLayer[layer][source]);
      }
      errorAtCurrentLayer = errorAtPreviousLayer;
    }
  }

  // Cross-entropy loss for this example.
  const loss = -Math.log(Math.max(1e-9, outputProbabilities[trueClassIndex]));
  return { weightGradients, biasGradients, loss, outputProbabilities };
}

// One SGD step: subtract a small multiple of each gradient.
function applyGradients(network, gradients, learningRate) {
  for (let layer = 0; layer < network.weights.length; layer++) {
    for (let neuron = 0; neuron < network.weights[layer].length; neuron++) {
      network.biases[layer][neuron] -= learningRate * gradients.biasGradients[layer][neuron];
      for (let input = 0; input < network.weights[layer][neuron].length; input++) {
        network.weights[layer][neuron][input] -=
          learningRate * gradients.weightGradients[layer][neuron][input];
      }
    }
  }
}
# ───────────────────────────────────────────────────────────────
# network.py · feed-forward neural net trained with backprop + SGD
#
# Pure stdlib, no NumPy. For real work you would use NumPy or PyTorch,
# but the loops here make every operation explicit and easy to follow.
# ───────────────────────────────────────────────────────────────
import math
import random

# Activation functions paired with their derivatives. Each derivative
# is expressed in terms of the post-activation `a`, not the pre-activation `z`,
# so we can compute it cheaply from values we already have.
def relu(z):                                   return z if z > 0 else 0.0
def relu_derivative(a):                        return 1.0 if a > 0 else 0.0
def tanh(z):                                   return math.tanh(z)
def tanh_derivative(a):                        return 1.0 - a * a
def sigmoid(z):                                return 1.0 / (1.0 + math.exp(-z))
def sigmoid_derivative(a):                     return a * (1.0 - a)

ACTIVATIONS = {
    'relu':    (relu,    relu_derivative),
    'tanh':    (tanh,    tanh_derivative),
    'sigmoid': (sigmoid, sigmoid_derivative),
}

def make_network(layer_sizes, activation_name='relu'):
    """Build a network. layer_sizes = [n_in, n_hidden, ..., n_out]."""
    weights = []   # weights[layer][neuron][input]
    biases  = []   # biases[layer][neuron]

    for layer in range(1, len(layer_sizes)):
        fan_in  = layer_sizes[layer - 1]
        fan_out = layer_sizes[layer]

        # Target weight variance: 2/fan_in for ReLU (He init), 1/fan_in otherwise
        # (Xavier-style). A uniform draw on [-limit, limit] has variance limit²/3,
        # so limit = sqrt(3 × target_variance).
        target_variance = 2 / fan_in if activation_name == 'relu' else 1 / fan_in
        limit = math.sqrt(3 * target_variance)

        layer_weights = [
            [(random.random() * 2 - 1) * limit for _ in range(fan_in)]
            for _ in range(fan_out)
        ]
        weights.append(layer_weights)
        biases.append([0.0] * fan_out)   # biases start at zero

    return {
        'layer_sizes':     layer_sizes,
        'weights':         weights,
        'biases':          biases,
        'activation_name': activation_name,
    }

def forward(network, input_vector):
    """Run one example through the network. Returns activations and pre-activations
    per layer, we need both for backprop."""
    activations_per_layer     = [list(input_vector)]
    pre_activations_per_layer = []
    activation_fn, _ = ACTIVATIONS[network['activation_name']]

    for layer in range(len(network['weights'])):
        layer_weights        = network['weights'][layer]
        layer_biases         = network['biases'][layer]
        previous_activations = activations_per_layer[-1]
        is_output_layer      = (layer == len(network['weights']) - 1)

        pre_activations = []
        new_activations = []
        for neuron_index in range(len(layer_weights)):
            # Weighted sum + bias = pre-activation z.
            z = layer_biases[neuron_index]
            for input_index in range(len(layer_weights[neuron_index])):
                z += layer_weights[neuron_index][input_index] * previous_activations[input_index]
            pre_activations.append(z)

            # Output layer keeps raw logits; hidden layers apply the activation fn.
            new_activations.append(z if is_output_layer else activation_fn(z))

        pre_activations_per_layer.append(pre_activations)
        activations_per_layer.append(new_activations)

    return activations_per_layer, pre_activations_per_layer

def softmax(logits):
    """Turn raw scores into probabilities. Subtracting the max is a stability trick."""
    max_logit    = max(logits)
    exponentials = [math.exp(z - max_logit) for z in logits]
    total        = sum(exponentials)
    return [value / total for value in exponentials]

def backprop(network, input_vector, true_class_index):
    """Compute gradients of the cross-entropy loss with respect to all weights and biases."""
    activations_per_layer, _ = forward(network, input_vector)
    num_layers = len(network['weights'])
    _, activation_derivative = ACTIVATIONS[network['activation_name']]

    # Allocate zero gradients with the same shape as the parameters.
    weight_gradients = [
        [[0.0] * len(row) for row in layer]
        for layer in network['weights']
    ]
    bias_gradients = [[0.0] * len(layer) for layer in network['biases']]

    # BP1: output error for softmax + cross-entropy is just (probs − one_hot).
    output_logits        = activations_per_layer[num_layers]
    output_probabilities = softmax(output_logits)
    error_at_current_layer = list(output_probabilities)
    error_at_current_layer[true_class_index] -= 1.0

    for layer in range(num_layers - 1, -1, -1):
        previous_layer_activations = activations_per_layer[layer]

        # BP3 + BP4: bias and weight gradients for this layer.
        for neuron in range(len(network['weights'][layer])):
            bias_gradients[layer][neuron] = error_at_current_layer[neuron]
            for input_idx in range(len(previous_layer_activations)):
                weight_gradients[layer][neuron][input_idx] = (
                    error_at_current_layer[neuron] * previous_layer_activations[input_idx]
                )

        # BP2: send the error one layer further back, scaled by the activation slope.
        if layer > 0:
            error_at_previous_layer = [0.0] * len(network['weights'][layer - 1])
            for source in range(len(error_at_previous_layer)):
                weighted_error = 0.0
                for target in range(len(network['weights'][layer])):
                    weighted_error += network['weights'][layer][target][source] * error_at_current_layer[target]
                error_at_previous_layer[source] = (
                    weighted_error
                    * activation_derivative(activations_per_layer[layer][source])
                )
            error_at_current_layer = error_at_previous_layer

    loss = -math.log(max(1e-9, output_probabilities[true_class_index]))
    return weight_gradients, bias_gradients, loss, output_probabilities

def apply_gradients(network, weight_gradients, bias_gradients, learning_rate):
    """One SGD step: subtract a small multiple of each gradient."""
    for layer in range(len(network['weights'])):
        for neuron in range(len(network['weights'][layer])):
            network['biases'][layer][neuron] -= learning_rate * bias_gradients[layer][neuron]
            for input_idx in range(len(network['weights'][layer][neuron])):
                network['weights'][layer][neuron][input_idx] -= (
                    learning_rate * weight_gradients[layer][neuron][input_idx]
                )
// ─────────────────────────────────────────────────────────────────
// network.cpp · feed-forward neural net trained with backprop + SGD
// Requires C++11 or later (uses <random>, range-for, auto, etc.).
// Compile: g++ -std=c++11 -O2 network.cpp -o network
// ─────────────────────────────────────────────────────────────────
#include <vector>
#include <cmath>
#include <random>
#include <algorithm>
#include <string>
#include <cstddef>

// Pair an activation function with its derivative-from-activation.
struct Activation {
    double (*apply)(double z);
    double (*derivative_from_activation)(double a);
};

static double relu_fn(double z) { return z > 0.0 ? z : 0.0; }
static double relu_d (double a) { return a > 0.0 ? 1.0 : 0.0; }
static double tanh_fn(double z) { return std::tanh(z); }
static double tanh_d (double a) { return 1.0 - a * a; }
static double sigmoid_fn(double z) { return 1.0 / (1.0 + std::exp(-z)); }
static double sigmoid_d (double a) { return a * (1.0 - a); }

static Activation choose_activation(const std::string& name) {
    if (name == "relu")    return { relu_fn,    relu_d };
    if (name == "tanh")    return { tanh_fn,    tanh_d };
    return { sigmoid_fn, sigmoid_d };
}

struct Network {
    std::vector<int> layer_sizes;
    std::vector<std::vector<std::vector<double>>> weights;  // weights[layer][neuron][input]
    std::vector<std::vector<double>> biases;                 // biases[layer][neuron]
    std::string activation_name;
};

struct ForwardResult {
    std::vector<std::vector<double>> activations;       // includes input as layer 0
    std::vector<std::vector<double>> pre_activations;
};

struct Gradients {
    std::vector<std::vector<std::vector<double>>> weight_gradients;
    std::vector<std::vector<double>> bias_gradients;
    double loss;
    std::vector<double> output_probabilities;
};

Network make_network(const std::vector<int>& layer_sizes,
                     const std::string& activation_name = "relu") {
    Network net;
    net.layer_sizes     = layer_sizes;
    net.activation_name = activation_name;

    std::mt19937 rng(std::random_device{}());
    std::uniform_real_distribution<double> uniform(-1.0, 1.0);

    for (std::size_t layer = 1; layer < layer_sizes.size(); ++layer) {
        int fan_in  = layer_sizes[layer - 1];
        int fan_out = layer_sizes[layer];

        // Target weight variance: 2/fan_in for ReLU (He init), 1/fan_in otherwise
        // (Xavier-style). A uniform draw on [-limit, limit] has variance limit²/3,
        // so limit = sqrt(3 × target_variance).
        double target_variance = (activation_name == "relu") ? 2.0 / fan_in : 1.0 / fan_in;
        double limit = std::sqrt(3.0 * target_variance);

        std::vector<std::vector<double>> layer_weights(fan_out, std::vector<double>(fan_in));
        for (int neuron = 0; neuron < fan_out; ++neuron)
            for (int input = 0; input < fan_in; ++input)
                layer_weights[neuron][input] = uniform(rng) * limit;

        net.weights.push_back(layer_weights);
        net.biases.push_back(std::vector<double>(fan_out, 0.0));
    }
    return net;
}

ForwardResult forward(const Network& net, const std::vector<double>& input_vector) {
    ForwardResult result;
    result.activations.push_back(input_vector);
    Activation activation = choose_activation(net.activation_name);

    for (std::size_t layer = 0; layer < net.weights.size(); ++layer) {
        const auto& layer_weights = net.weights[layer];
        const auto& layer_biases  = net.biases[layer];
        const auto& previous       = result.activations.back();
        const bool is_output_layer = (layer == net.weights.size() - 1);

        std::vector<double> pre_activations(layer_weights.size());
        std::vector<double> activations    (layer_weights.size());

        for (std::size_t neuron = 0; neuron < layer_weights.size(); ++neuron) {
            double z = layer_biases[neuron];
            for (std::size_t input = 0; input < layer_weights[neuron].size(); ++input)
                z += layer_weights[neuron][input] * previous[input];
            pre_activations[neuron] = z;
            activations[neuron]     = is_output_layer ? z : activation.apply(z);
        }

        result.pre_activations.push_back(pre_activations);
        result.activations.push_back(activations);
    }
    return result;
}

std::vector<double> softmax(const std::vector<double>& logits) {
    double max_logit = *std::max_element(logits.begin(), logits.end());
    std::vector<double> exponentials(logits.size());
    double total = 0.0;
    for (std::size_t i = 0; i < logits.size(); ++i) {
        exponentials[i]  = std::exp(logits[i] - max_logit);
        total           += exponentials[i];
    }
    for (double& value : exponentials) value /= total;
    return exponentials;
}

Gradients backprop(const Network& net,
                    const std::vector<double>& input_vector,
                    int true_class_index) {
    ForwardResult fwd = forward(net, input_vector);
    Activation activation = choose_activation(net.activation_name);
    std::size_t num_layers = net.weights.size();

    // Zero-shaped gradient containers.
    Gradients grads;
    grads.weight_gradients.resize(num_layers);
    grads.bias_gradients.resize(num_layers);
    for (std::size_t layer = 0; layer < num_layers; ++layer) {
        grads.weight_gradients[layer].assign(net.weights[layer].size(), {});
        for (std::size_t neuron = 0; neuron < net.weights[layer].size(); ++neuron)
            grads.weight_gradients[layer][neuron].assign(net.weights[layer][neuron].size(), 0.0);
        grads.bias_gradients[layer].assign(net.biases[layer].size(), 0.0);
    }

    // BP1: output error for softmax + cross-entropy is just (probs - one_hot).
    std::vector<double> output_probabilities = softmax(fwd.activations.back());
    std::vector<double> error_at_current_layer = output_probabilities;
    error_at_current_layer[true_class_index] -= 1.0;

    for (int layer = static_cast<int>(num_layers) - 1; layer >= 0; --layer) {
        const auto& previous_layer_activations = fwd.activations[layer];

        // BP3 + BP4: bias and weight gradients for this layer.
        for (std::size_t neuron = 0; neuron < net.weights[layer].size(); ++neuron) {
            grads.bias_gradients[layer][neuron] = error_at_current_layer[neuron];
            for (std::size_t input = 0; input < previous_layer_activations.size(); ++input)
                grads.weight_gradients[layer][neuron][input] =
                    error_at_current_layer[neuron] * previous_layer_activations[input];
        }

        // BP2: propagate error one layer further back.
        if (layer > 0) {
            std::vector<double> error_at_previous_layer(net.weights[layer - 1].size(), 0.0);
            for (std::size_t source = 0; source < error_at_previous_layer.size(); ++source) {
                double weighted_error = 0.0;
                for (std::size_t target = 0; target < net.weights[layer].size(); ++target)
                    weighted_error += net.weights[layer][target][source] * error_at_current_layer[target];
                error_at_previous_layer[source] =
                    weighted_error * activation.derivative_from_activation(fwd.activations[layer][source]);
            }
            error_at_current_layer = error_at_previous_layer;
        }
    }

    grads.loss = -std::log(std::max(1e-9, output_probabilities[true_class_index]));
    grads.output_probabilities = output_probabilities;
    return grads;
}

void apply_gradients(Network& net, const Gradients& grads, double learning_rate) {
    for (std::size_t layer = 0; layer < net.weights.size(); ++layer) {
        for (std::size_t neuron = 0; neuron < net.weights[layer].size(); ++neuron) {
            net.biases[layer][neuron] -= learning_rate * grads.bias_gradients[layer][neuron];
            for (std::size_t input = 0; input < net.weights[layer][neuron].size(); ++input)
                net.weights[layer][neuron][input] -= learning_rate * grads.weight_gradients[layer][neuron][input];
        }
    }
}
// ───────────────────────────────────────────────────────────────────
// network.rs · feed-forward neural net trained with backprop + SGD.
// Uses only `std`. For real work, pull in the `rand` crate and replace
// the tiny PRNG at the bottom of this file.
// ───────────────────────────────────────────────────────────────────

use std::f64;

#[derive(Clone, Copy)]
pub enum Activation { ReLU, Tanh, Sigmoid }

impl Activation {
    /// Apply the activation function to a pre-activation `z`.
    pub fn apply(self, z: f64) -> f64 {
        match self {
            Activation::ReLU    => if z > 0.0 { z } else { 0.0 },
            Activation::Tanh    => z.tanh(),
            Activation::Sigmoid => 1.0 / (1.0 + (-z).exp()),
        }
    }

    /// Derivative expressed in terms of the *post-activation* value `a`.
    /// Cheap to compute from values we already have.
    pub fn derivative_from_activation(self, a: f64) -> f64 {
        match self {
            Activation::ReLU    => if a > 0.0 { 1.0 } else { 0.0 },
            Activation::Tanh    => 1.0 - a * a,
            Activation::Sigmoid => a * (1.0 - a),
        }
    }
}

pub struct Network {
    pub layer_sizes: Vec<usize>,
    pub weights:     Vec<Vec<Vec<f64>>>,  // [layer][neuron][input]
    pub biases:      Vec<Vec<f64>>,
    pub activation:  Activation,
}

pub struct Gradients {
    pub weight_gradients:     Vec<Vec<Vec<f64>>>,
    pub bias_gradients:       Vec<Vec<f64>>,
    pub loss:                 f64,
    pub output_probabilities: Vec<f64>,
}

impl Network {
    /// Build a fresh network with the given shape.
    pub fn new(layer_sizes: &[usize], activation: Activation) -> Self {
        let mut weights: Vec<Vec<Vec<f64>>> = Vec::new();
        let mut biases:  Vec<Vec<f64>>      = Vec::new();

        for layer in 1..layer_sizes.len() {
            let fan_in  = layer_sizes[layer - 1];
            let fan_out = layer_sizes[layer];

            // Target weight variance: 2/fan_in for ReLU (He init), 1/fan_in otherwise
            // (Xavier-style). A uniform draw on [-limit, limit] has variance limit²/3,
            // so limit = sqrt(3 × target_variance).
            let target_variance = match activation {
                Activation::ReLU => 2.0 / fan_in as f64,
                _                  => 1.0 / fan_in as f64,
            };
            let limit = (3.0 * target_variance).sqrt();

            let layer_weights: Vec<Vec<f64>> = (0..fan_out)
                .map(|_| (0..fan_in)
                    .map(|_| (rand_uniform() * 2.0 - 1.0) * limit)
                    .collect())
                .collect();
            weights.push(layer_weights);
            biases.push(vec![0.0; fan_out]);
        }

        Self { layer_sizes: layer_sizes.to_vec(), weights, biases, activation }
    }

    /// Run one input vector through the network. Returns (activations, pre-activations)
    /// per layer; the input is included as `activations[0]`.
    pub fn forward(&self, input_vector: &[f64]) -> (Vec<Vec<f64>>, Vec<Vec<f64>>) {
        let mut activations_per_layer: Vec<Vec<f64>> = vec![input_vector.to_vec()];
        let mut pre_activations_per_layer: Vec<Vec<f64>> = Vec::new();

        for (layer, layer_weights) in self.weights.iter().enumerate() {
            let layer_biases = &self.biases[layer];
            let previous     = activations_per_layer.last().unwrap().clone();
            let is_output_layer = layer == self.weights.len() - 1;

            let mut pre_activations = Vec::with_capacity(layer_weights.len());
            let mut activations     = Vec::with_capacity(layer_weights.len());

            for (neuron, weight_row) in layer_weights.iter().enumerate() {
                let mut z = layer_biases[neuron];
                for input in 0..weight_row.len() {
                    z += weight_row[input] * previous[input];
                }
                pre_activations.push(z);
                activations.push(if is_output_layer { z } else { self.activation.apply(z) });
            }

            pre_activations_per_layer.push(pre_activations);
            activations_per_layer.push(activations);
        }

        (activations_per_layer, pre_activations_per_layer)
    }

    /// Backpropagation for cross-entropy + softmax.
    pub fn backprop(&self, input_vector: &[f64], true_class: usize) -> Gradients {
        let (activations_per_layer, _) = self.forward(input_vector);

        let mut weight_gradients: Vec<Vec<Vec<f64>>> = self.weights.iter()
            .map(|layer| layer.iter().map(|row| vec![0.0; row.len()]).collect())
            .collect();
        let mut bias_gradients: Vec<Vec<f64>> = self.biases.iter()
            .map(|b| vec![0.0; b.len()])
            .collect();

        // BP1: softmax(logits) − one_hot(true_class).
        let output_probabilities = softmax(activations_per_layer.last().unwrap());
        let mut error_at_current_layer: Vec<f64> = output_probabilities.clone();
        error_at_current_layer[true_class] -= 1.0;

        for layer in (0..self.weights.len()).rev() {
            let previous_layer_activations = &activations_per_layer[layer];

            // BP3 + BP4: bias and weight gradients for this layer.
            for neuron in 0..self.weights[layer].len() {
                bias_gradients[layer][neuron] = error_at_current_layer[neuron];
                for input in 0..previous_layer_activations.len() {
                    weight_gradients[layer][neuron][input] =
                        error_at_current_layer[neuron] * previous_layer_activations[input];
                }
            }

            // BP2: send error one layer further back.
            if layer > 0 {
                let mut error_at_previous_layer = vec![0.0; self.weights[layer - 1].len()];
                for source in 0..error_at_previous_layer.len() {
                    let mut weighted_error = 0.0;
                    for target in 0..self.weights[layer].len() {
                        weighted_error += self.weights[layer][target][source] * error_at_current_layer[target];
                    }
                    error_at_previous_layer[source] = weighted_error
                        * self.activation.derivative_from_activation(activations_per_layer[layer][source]);
                }
                error_at_current_layer = error_at_previous_layer;
            }
        }

        Gradients {
            weight_gradients,
            bias_gradients,
            loss: -output_probabilities[true_class].max(1e-9).ln(),
            output_probabilities,
        }
    }

    /// One SGD step: subtract a small multiple of each gradient.
    pub fn apply_gradients(&mut self, grads: &Gradients, learning_rate: f64) {
        for layer in 0..self.weights.len() {
            for neuron in 0..self.weights[layer].len() {
                self.biases[layer][neuron] -= learning_rate * grads.bias_gradients[layer][neuron];
                for input in 0..self.weights[layer][neuron].len() {
                    self.weights[layer][neuron][input] -=
                        learning_rate * grads.weight_gradients[layer][neuron][input];
                }
            }
        }
    }
}

fn softmax(logits: &[f64]) -> Vec<f64> {
    let max_logit = logits.iter().copied().fold(f64::NEG_INFINITY, f64::max);
    let exponentials: Vec<f64> = logits.iter().map(|&z| (z - max_logit).exp()).collect();
    let total: f64 = exponentials.iter().sum();
    exponentials.iter().map(|e| e / total).collect()
}

// Tiny xorshift64 PRNG. Single-threaded only: `static mut` is unsynchronized,
// so concurrent calls would be undefined behavior. For real (multi-threaded
// or production) code, drop this and use the `rand` crate.
fn rand_uniform() -> f64 {
    use std::time::{SystemTime, UNIX_EPOCH};
    static mut STATE: u64 = 0;
    unsafe {
        if STATE == 0 {
            STATE = SystemTime::now()
                .duration_since(UNIX_EPOCH).unwrap()
                .subsec_nanos() as u64 | 1;
        }
        STATE ^= STATE << 13;
        STATE ^= STATE >> 7;
        STATE ^= STATE << 17;
        (STATE as f64) / (u64::MAX as f64)
    }
}
What's intentionally missing

The listings trade speed and robustness for readability. A production implementation would add:

Every line traces back to an equation from earlier sections. Large language models are trained with the same loop of forward pass, loss, backprop and update, scaled up with matrix libraries, GPUs, attention layers, better optimizers and billions of parameters.

13Train it yourself

The widget below runs a compact copy of the JavaScript above (same algorithm, same initialization, shorter variable names) in your browser. Pick a dataset, tweak the architecture, hit TRAIN. The left panel shows the predicted class probability over the input space, with the white line marking the decision boundary where both classes are equally likely; the chart under the controls plots the average training loss per epoch.

Live · Neural Network Trainer click on the left canvas to add points · shift-click for class 0
INPUT SPACE · decision boundary forms as the network learns
epoch
0
loss
···
accuracy
···
throughput
···
Pick a dataset and press TRAIN, then watch the white boundary bend around the points while the loss chart falls and the accuracy climbs. Click the left canvas to drop a class-1 point, or shift-click for class 0, and the network adapts. Class 0 (blue) draws as squares and class 1 (purple) as circles, so the two are distinguishable without color. Architecture: 2 → hidden → hidden → 2, trained with SGD at batch size 1 and cross-entropy loss.

Things to try

14Overfitting and how to spot it

A photo classifier can score 100% on the photos it trained on and still mislabel the next photo a user uploads. A network with enough capacity can memorize its training set, noise included. Give 100 noisy points to a network with 1,000 weights and, trained long enough, it can bend its decision boundary around every one of them. On the training data it looks perfect; on new data it does worse than a simpler fit would.

This is : learning the noise in the training set as if it were a real pattern. The standard way to catch it is to hold out some of your data, never let the network train on it, and measure the loss on that held-out set during training too. When the training loss keeps falling but the held-out loss doesn't, the model has stopped learning things that carry over to new data.

Train / validation / test

The standard split is three sets. Training set: what the network sees and learns from. Validation set: held out, used to pick hyperparameters and decide when to stop training. Test set: held out even from you the engineer. Touch it once at the end to report final performance. Use it more than that and you're effectively training on it through the feedback loop.

The widget below trains a deliberately oversized network on 40 noisy points, jittered enough that a few land on the wrong side of the gap between the two classes. The solid green curve is loss on the training set. The dashed red curve is loss on a held-out validation set the network never sees. Validation loss usually bottoms out within the first 50 epochs; after that, training loss keeps falling while validation loss climbs.

Worked example

The widget's network is 2 → 16 → 16 → 2. Parameters: (2×16 + 16) + (16×16 + 16) + (16×2 + 2) = 48 + 272 + 34 = 354 weights and biases.

Against 40 training points, that's almost 9 adjustable numbers per point, plenty to fit each point's noise individually.

Live · Train vs. Validation
filled = training · outlined = validation · ◼ class 0 · ● class 1
training loss (solid) validation loss (dashed)
epoch
0
train loss
···
val loss
···
Press TRAIN. The left panel shows the decision boundary with training points filled and validation points outlined; the right panel plots both losses per epoch. One network is scored on two sets of points. The decision boundary chases noise in the training set as the green solid curve drops further. The dashed red validation curve eventually starts climbing: from there on, the network is memorizing instead of generalizing. Drop the noise toward 0.02 and the classes stop overlapping, and the climb goes away.

How to recognize it

The diagnostic pattern is:

How to fix it

Four standard cures, often combined:

  1. More data. This is the cleanest fix: with a thousand points, the noise averages out and a network this size no longer has the capacity to memorize individual examples. Often not an option in practice.
  2. Smaller network. Fewer parameters means less capacity to memorize.
  3. Weight decay (L2 regularization). Penalize the network for having large weights, which pushes it toward smoother functions. Train with the weight-decay slider at 0.01 or above and the validation curve climbs far less than it does at 0.
  4. Early stopping. Track validation loss and keep a copy of the weights from its best epoch; once it has stopped improving for a while, stop and use that copy.

There are more techniques: dropout, which randomly zeros out neurons during training; batch normalization; and data augmentation. The four above are the baseline most training setups start from.

15Your turn: the code playground

The playground below runs the same compact JavaScript library as the trainer, exposed as the MPGNet object, so you can build your own network, train it, and predict from the editor. The Python, C++, and Rust ports from §12 do the same thing; copy any of them into your own editor to run there. Hit RUN (or Ctrl+Enter / Cmd+Enter) to execute; output prints on the right.

▸ playground.js · live JavaScript, in your browser

Edit anything: the data, the architecture, the learning rate, a held-out test set. The library follows the §12 JavaScript with shorter names: makeNetwork(layerSizes, { activation }) returns { W, B } (weights and biases), backprop(net, input, classIndex) returns { dW, dB, loss, probs }, and forward, predict, applyGrads and softmaxVec round it out. It also lives at window.MPGNet for use from the browser console.

16Where to go from here

The network built on this page is called a multilayer perceptron (MLP) or feedforward neural network. It is the base that later architectures build on, and fully connected MLP blocks still sit inside most of them. On its own it handles small tabular datasets and 2D toys, and struggles with the images, audio and text that made deep learning famous.

Continue with Automatic Differentiation from Scratch, which turns the hand-written backward pass built here into a tape that differentiates any program. The Build a Language Model hub maps the rest of the series: training at depth, evaluation, embeddings, transformers, a trained GPT, alignment and serving, with convolutional networks and diffusion as adjacent pages.

Images: convolutions

A 256×256 grayscale image has 65,536 pixels. Connecting every pixel to each of, say, 500 neurons in an MLP's first hidden layer takes about 33 million weights for that layer alone, and the network has to learn the same edge detector separately at every position. Convolutional neural networks (CNNs) exploit the image's structure: a small filter of shared weights slides over the image, producing a feature map that records where its pattern appears. The algorithm from this page (backprop on a chain of differentiable operations) still drives training; only the layer types change.

Sequences: attention

For text, audio, or any data that comes in a sequence, an MLP needs a fixed input size and has no built-in notion of order. The standard sequence models before 2017, recurrent networks, read one token at a time, so relating token i to token i+50 means carrying information through every position in between. Vaswani et al.'s 2017 Attention Is All You Need[9] replaces recurrence with scaled dot-product attention. Each token scores how relevant every other token is to it, turns the scores into weights with softmax, and takes a weighted mix of the other tokens' content, so every pair of positions connects directly:

scaled dot-product attention Attention(Q, K, V) = softmax( QK⊤√dk )V

In words: each token gets three vectors, a query Q (what it's looking for), a key K (what it offers) and a value V (its content). Take the dot product of every query with every key, divide by √dk to keep the numbers moderate, softmax each row into weights, and use them to mix the values. Each token ends up with its own weighted summary of the sequence.

Stack blocks of attention followed by a small MLP (with residual connections and normalization around each) and you have a , the architecture behind nearly every major language model since 2018. Karpathy's nanoGPT[10] is a compact reference implementation of a GPT-style transformer: about 300 lines for the model, 300 for the training loop.

You don't need GPT-4 to do interesting things

Eldan and Li's TinyStories paper[11] showed that transformers with under 10 million parameters, some with a single transformer block, can write fluent, consistent short stories when trained on a synthetic dataset restricted to words a typical 3- to 4-year-old understands. The authors report that every model in the paper trains on a single V100 GPU in at most 30 hours, so the whole pipeline is within reach of a hobby budget.

Building a tiny transformer takes the same forward/backward/update loop with different layers in the middle. Karpathy's series[15] and Perivolaropoulos's Train Your Own LLM From Scratch workshop[19] both walk through one step by step.

You finished

You now have a code-level picture of how neural networks are trained: forward pass, loss, backpropagation, update. Larger models change the layer types, the optimizer and the scale, but they are trained by the same loop.

The final exam

Five questions covering the whole tutorial. If you can answer all five without scrolling back up, you've got the fundamentals.

17Sources & further reading

Numbered citations refer to the superscripts above. Links are given where a stable free copy exists.

A note on originality

The prose in this tutorial is original writing. The mathematical equations themselves are canonical statements that appear identically across every neural-network textbook (there's only one way to write the sigmoid function or the four backpropagation equations), and they are attributed at the points where they are introduced. The labeling convention BP1–BP4 follows Nielsen [4], who derives all four in detail. The scaled dot-product attention formula appears verbatim in Vaswani et al. [9]. The interactive demos, code, and CSS on this page are written from scratch for this tutorial.

  1. Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408. The original artificial neuron.
  2. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. Brought backpropagation into the mainstream.
  3. Minsky, M. & Papert, S. (1969). Perceptrons. MIT Press. Proves the limits of single-layer perceptrons, XOR among them; widely cited as a cause of the 1970s decline in neural-network research.
  4. Nielsen, M. A. (2015). Neural Networks and Deep Learning. Determination Press. Free online: neuralnetworksanddeeplearning.com. Chapter 2 derives the four backprop equations (BP1–BP4) used in §9; chapter 1 uses the quadratic cost of eq. 6.
  5. Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich. The first formal analysis of the vanishing-gradient problem.
  6. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI. PDF. The GPT-2 paper; the released model code (src/model.py) applies GELU in every feed-forward block.
  7. Glorot, X. & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS. The Xavier initialization paper: derives the forward (1/nin) and backward (1/nout) variance conditions and proposes 2/(nin + nout).
  8. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV. The He / Kaiming initialization paper: variance 2/nin for ReLU layers, and the 30-layer comparison against Xavier initialization.
  9. Vaswani, A., Shazeer, N., Parmar, N. et al. (2017). Attention Is All You Need. NeurIPS. arXiv:1706.03762. The Transformer paper.
  10. Karpathy, A. nanoGPT. GitHub: github.com/karpathy/nanoGPT. A compact GPT implementation; its README describes a ~300-line model and a ~300-line training loop.
  11. Eldan, R. & Li, Y. (2023). TinyStories: How Small Can Language Models Be and Still Speak Coherent English? arXiv:2305.07759. Coherent story generation below 10 million parameters; all models trained on a single V100 GPU in at most 30 hours.
  12. Hinton, G. (2012). Lecture 6e: rmsprop, Divide the gradient by a running average of its recent magnitude. Slide deck from the Coursera Neural Networks course. RMSProp's first public appearance.
  13. Kingma, D. P. & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv:1412.6980. Source of the Adam update and its bias correction; Section 3 of the paper notes that without the correction the initial steps are much larger.
  14. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4), 541–551. An early convolutional network trained end to end with backpropagation, on U.S. Postal Service ZIP-code digits; the MNIST work in Gradient-based learning applied to document recognition (LeCun et al., 1998) followed nine years later.
  15. Karpathy, A. Neural Networks: Zero to Hero. YouTube series, 2022–2023. karpathy.ai/zero-to-hero.html. Builds from a scalar autograd engine up to a small GPT with the same forward, loss, backward, update loop.
  16. Linnainmaa, S. (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. M.Sc. thesis, University of Helsinki. Reverse-mode automatic differentiation, which is the algorithm we now call backpropagation, sixteen years before Rumelhart-Hinton-Williams.
  17. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4), 303–314. The universal approximation theorem for one hidden layer of sigmoid units.
  18. Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2), 251–257. Extends the result to any bounded, nonconstant activation function.
  19. Perivolaropoulos, A. (2026). Train Your Own LLM From Scratch. GitHub: github.com/angelos-p/llm-from-scratch. A hands-on workshop that builds a ~10M-parameter GPT which trains on a laptop in under an hour.

See also