Convolutional Networks from Scratch
A photo is a grid of numbers, and a dense layer that gave every pixel its own weights would need tens of millions of them for even a tiny image. A convolutional network slides one small filter across the whole image instead, so the same few weights can find an edge wherever it appears. This page works through that filter by hand: what size comes out, how color channels combine, what happens when the image shifts, and how training updates a weight that's used in many places.
Work out a convolution layer's output size and weight count from its kernel size, stride, padding and dilation. Explain why an RGB convolution adds its three color channels into one map, and what a 1×1 convolution is for. Predict how much of the input one output can depend on after a stack of layers, and why a one-pixel shift can change a pooled output. Write the backward pass for a shared kernel and check it against a finite difference.
01One small filter, reused everywhere
Say you want a network to tell whether a 32×32 color photo contains a cat. That photo is 32 × 32 × 3 = 3,072 numbers. A dense layer, the kind covered on the Neural Networks page, gives every input-output pair its own weight. A dense layer that outputs 16 numbers per pixel would need 3,072 × 16,384 ≈ 50 million weights, and it would have to learn what an ear looks like separately at every position. A learns one small grid of weights instead and slides it across the image, computing the same weighted sum at every position.[1]
That small grid is the kernel. At each position it sits on a patch of the image the same size as itself. Multiply each pixel in the patch by the kernel weight on top of it, add up the products, and that's one output number. Move one pixel to the right and do it again. Because the same weights are used everywhere, a kernel that responds to a vertical edge responds to it anywhere in the image. That's an assumption built into the layer, not knowledge: the weights still have to be learned from data before the kernel detects anything useful.[1]
Worked example
The image in the first widget is dark (0) on its left half and bright (1) on its right half. The vertical-edge kernel has the row (−1, 0, 1), repeated three times.
On a flat bright patch every row is (1, 1, 1): −1 + 0 + 1 = 0 per row, so the output is 0.
On the patch where the edge begins every row is (0, 0, 1): 0 + 0 + 1 = 1 per row, three rows, so the output is 3.
The kernel answers 0 on flat areas and 3 where dark meets bright. A bright-to-dark edge would give −3.
Written out, the output at one position is a sum over the kernel's offsets. Every convolution layer on this page computes this; later sections add channels and spacing options to it.
In words: put the kernel's top-left corner on (row, column), multiply each kernel weight by the pixel under it, add everything up, then add the bias. The widget below uses one input grid, moves one pixel at a time, and has no bias.
Strictly, this is cross-correlation. The textbook convolution from signal processing flips the kernel left-to-right and top-to-bottom before sliding it. Deep learning libraries such as PyTorch skip the flip and still call the layer a convolution.[2] For learned kernels the difference doesn't matter, because training simply learns flipped weights. It matters when you copy a hand-written filter from a signal-processing source: the flip mirrors it, and for an edge kernel like the one above, mirroring changes the sign of every response.
The grid of outputs is a feature map: one number per position saying how strongly the patch there matched the kernel. It's not a list of detected objects. A large value means "this patch looks like the kernel", and a pattern running the other way can give a large negative value. An activation such as ReLU, which keeps positive values and zeroes out negative ones, then decides which responses pass to the next layer.
The widget has three hand-picked kernels so you can check the arithmetic yourself. The horizontal-edge kernel outputs zero everywhere on this image, because the brightness changes from column to column and never from row to row. The box average blurs. A trained network doesn't use hand-picked kernels: it learns its own through the loss and the backward pass, which section 06 covers.
The weight count is the other payoff. Images usually have several numbers per pixel, one per channel: a color photo has three, red, green and blue. A layer with Cin input channels, Cout output channels and a K×K kernel has Cout × Cin × K² weights, plus one bias per output channel. The image's width and height don't appear anywhere in that count.[2]
Worked example
3 input channels (RGB), 16 output channels, 3×3 kernels: 16 × 3 × 9 = 432 weights plus 16 biases.
That's the same whether the image is 32×32 or 1024×1024. The dense layer at the top of this section needed about 50 million weights for the 32×32 case.
02Stride, padding and dilation set the output size
Before you stack layers you need to know what size comes out of each one. A 3×3 kernel on a 7×7 image only fits in 5×5 positions, so the image shrinks a little at every layer. Many networks also shrink it on purpose, so later layers have fewer positions to compute. Three settings control the output size.[3]
- Stride is how far the kernel moves between placements. Stride 2 skips every other position, which roughly halves each side of the output.
- Padding adds extra values, usually zeros, around the border so the kernel can hang partly off the edge.
- Dilation spreads the kernel's samples apart. With dilation 2, a 3-wide kernel covers 5 pixels but still has only 3 weights.
Worked example
One row of 7 pixels, a 3-wide kernel, padding 1, stride 2. Padding makes the row 9 wide, positions −1 to 7.
The kernel's left edge can sit at −1, 1, 3 and 5. At 7 it would run off the end. That's 4 outputs.
Now set dilation to 2. The kernel covers 5 positions, so its left edge can only sit at −1, 1 and 3: 3 outputs, still from 3 weights each.
The general rule counts the same thing: how many stride-sized steps the kernel can take across the padded row.
O = floor((I + 2 × padding − K_eff) / stride) + 1
In words: first work out how wide the kernel really is once the gaps are included. Then take the padded input width, subtract the kernel width, divide by the stride and round down: that's how many steps fit. Add 1 for the starting position. For the example, (7 + 2 − 3) / 2 + 1 = 4, and with dilation 2, (7 + 2 − 5) / 2 + 1 = 3. The formula assumes the same padding on both sides.
Zero padding is a guess about what lies outside the image, and the kernel treats it like real data. At the border it sees real pixels next to zeros, which can look like an edge that isn't in the scene. Reflection or circular padding make different guesses and give different numbers. A common recipe keeps the size unchanged: with an odd kernel size K, stride 1 and dilation 1, pad by (K − 1)/2 on each side, which is 1 for a 3×3 kernel. With stride above 1, definitions of "same size" differ, and some of them pad one side more than the other.[3]
Image generators and segmentation networks often need to go the other way, from a small map to a bigger one. A common tool is transposed convolution: each input value stamps a scaled copy of the kernel onto a larger output, and overlapping stamps add up. It's the same arithmetic the backward pass uses to send gradients back to input pixels. Despite the name it's not an inverse. A stride-2 layer with one input and one output channel produces roughly a quarter as many numbers as it reads, so information is gone and no layer can bring it back. The output_padding option some libraries offer only picks between output sizes that would otherwise be ambiguous; it doesn't restore lost values.[3]
03Color channels add up to one response
A color photo isn't one grid but three: red, green and blue. After the first layer the channels are whatever the previous layer's kernels computed, and there are usually more of them; ResNet's first layer outputs 64.[7] A kernel looking for a red-to-green edge has to see red and green together, so an ordinary convolution doesn't filter each channel on its own. Each output channel has one small kernel per input channel. It runs each kernel over its channel, adds the results into a single map, adds one bias, and only then applies the activation.[2]
Worked example
The next widget has two input channels and one output channel, so it has 1 × 2 × 9 = 18 weights and one bias (set to 0).
At the top-left output, channel 0's kernel contributes 1.50 and channel 1's kernel contributes −1.50.
The output is 1.50 + (−1.50) + 0 = 0.
With channels, the formula from section 01 gains one more sum, over the input channels.
input[c, row + u, column + v] × weight[o, c, u, v]
In words: for output channel o, run the kernel weight[o, c] over input channel c, do that for every input channel, then add all the results and one bias. Each output channel has its own complete set of kernels, one per input channel.
In the widget, channel 0 brightens from left to right and channel 1 brightens from top to bottom. Channel 0's kernel responds to left-to-right change; channel 1's kernel subtracts its bottom row from its top row, so a top-to-bottom brightening gives a negative number. Both ramps are perfectly even, so every position gets the same two numbers. The gain slider scales channel 1's kernel. At gain 1 the contributions cancel and the summed map is zero everywhere. That's a legitimate answer, not a bug: the layer combines evidence from both channels before any activation sees it.
Two variations are common. A 1×1 convolution has a kernel one pixel wide, so it never looks at neighbors. It only mixes channels, separately at each pixel, like a small dense layer applied to every pixel's list of channel values. That makes it a cheap way to raise or lower the channel count. Two 1×1 layers with nothing between them are just two matrix products and could be merged into one; put an activation between them and they can't.
A depthwise convolution goes the other way: each input channel gets its own spatial kernel (or several), and channels are never mixed. Following it with a 1×1 layer splits one layer's job in two, one step for space and one for channels. Depthwise is the extreme case of grouped convolution, where channels are split into groups that never connect, and the weight count becomes Cout × (Cin / groups) × K².[2]
Worked example
64 input channels to 64 output channels with 3×3 kernels. An ordinary layer: 64 × 64 × 9 = 36,864 weights.
Depthwise 3×3 (groups = 64): 64 × 1 × 9 = 576 weights. Then a 1×1 layer: 64 × 64 × 1 = 4,096. Together: 4,672, about an eighth of the ordinary layer.
04What happens when the picture moves
A cat classifier should say "cat" whether the cat sits in the middle of the photo or a few pixels to the left. Convolution gets part of the way there. Slide the input one pixel and a stride-1 convolution's output slides one pixel too, with the same values, away from the borders. That property is : the output moves with the input. A classifier needs something stronger, an answer that doesn't change at all when the input moves. That's called invariance. A CNN, a network built from convolution layers, moves toward invariance by discarding position in stages, and those stages are where shifts cause trouble.
The usual way to discard position is downsampling: keep fewer positions. Max pooling with a 2-wide window and stride 2 splits a row into pairs and keeps the larger value of each pair.[4] A stride-2 convolution also produces outputs at only every second position. In both cases a one-pixel shift can move a value from one window into the next, which changes the output instead of just sliding it.[5]
Worked example
Take the row (0, 0, 1, 0.3, 0, 0, 0, 0). Max pooling over pairs gives (0, 1, 0, 0): the 1 and the 0.3 share the second pair.
Shift right by one: (0, 0, 0, 1, 0.3, 0, 0, 0). The 1 is still in the second pair, but the 0.3 has moved into the third, so the output is (0, 1, 0.3, 0). The shift added a response instead of moving the old one.
Shift by two instead and the output is (0, 0, 1, 0): the original output moved one place, a clean shift.
Average pooling keeps the total, 0.65 before and after the one-pixel shift, but splits it across two outputs: (0, 0.65, 0, 0) becomes (0, 0.5, 0.15, 0).
The widget runs this example. It uses circular shifts, where values pushed off the right end come back on the left, so the border never interferes and every change comes from the pooling windows.
Signal processing calls this aliasing: sampling too sparsely makes the result depend on exactly where the samples fall. The standard fix is to blur slightly before downsampling, so no detail is finer than the samples can capture. Zhang (2019) added that blur to the downsampling layers of standard CNNs and reported outputs that changed less under small shifts, along with higher ImageNet accuracy across several common architectures.[5] The blur costs some fine detail, and it reduces shift sensitivity rather than removing it.
At the very end, a classifier needs one answer for the whole image. Global average pooling gets there by averaging each channel's whole map into one number, which removes position completely. A detector that has to say where the cat is can't do that, because position is exactly what it must report.
05How much of the image can one output see?
A 3×3 kernel sees a 3×3 patch, far too small to recognize a face. CNNs get around that by stacking layers. A second-layer output combines a 3×3 block of first-layer outputs, each of which saw its own 3×3 patch, so it depends on a 5×5 patch of the input. The set of input pixels one output can depend on is its , and its size tells you whether a network can see enough context for the task.[6]
Downsampling makes it grow faster. After a stride-2 layer, neighboring outputs sit 2 input pixels apart, so the next 3-wide kernel reaches 4 more input pixels instead of 2. Bookkeeping needs two numbers per layer: the span, how many input positions one output covers, and the jump, how many input pixels apart neighboring outputs are.
Worked example
Three layers with 3-wide kernels and stride 1, in one dimension. The span starts at 1 and each layer adds 2: 1 + 2 + 2 + 2 = 7 input positions.
Give the first layer stride 2. It still adds 2, because its inputs are 1 pixel apart, but it sets the jump to 2. The next two layers add 2 × 2 = 4 each: 1 + 2 + 4 + 4 = 11. Neighboring final outputs are 2 input positions apart.
The general recurrence walks from the input up through the layers, updating both numbers at each one.
jump_next = jump × stride
In words: each layer widens the span by its kernel's reach, which is (K − 1) × dilation steps, each step jump input pixels wide. A stride multiplies the jump for every layer after it. Start with span 1 and jump 1 at the input. This works for a plain chain of layers; a network with branches needs the count for each path.
The receptive field says what can affect an output, not how much. A zero weight cuts a path completely; a ReLU that outputs zero for this input cuts it for this input only. Dilation can leave holes, so a large span may contain positions no kernel ever reads. When judging coverage, count the positions actually sampled.[6]
Networks with branches need one more check. A residual connection adds a stack of layers' output back onto that stack's own input, written F(x) + x by He et al.[7] The addition is entry by entry, so both tensors need the same shape, and entry (row, column) in each must refer to the same place in the image. When the layers change the channel count or downsample, He et al. pass the shortcut through a learned projection, F(x) + Wsx, so the shapes match.
06Training a shared filter: add up every use
Training needs the derivative of the loss with respect to each kernel weight, so the optimizer knows which way to nudge it. The chain rule works as it does for a dense layer, which the Neural Networks page derives: a weight's derivative is how much the loss changes with each output the weight affects, times how much that output changes with the weight. What's new is sharing. A kernel weight is used at every output position, so its derivative is a sum over all of them. Code that overwrites the derivative inside the position loop, instead of adding to it, keeps only the last position and trains on the wrong number.
Worked example
The widget's input is the 3×3 grid (1, 2, 0 / 0, 1, 3 / 2, 0, 1) and the 2×2 kernel is (0.2, −0.1 / 0.3, 0.1). The four outputs are 0.1, 1.0, 0.5 and 0.
Every target is 1 and the loss is half the sum of squared errors, so the loss's derivative with respect to each output is output − 1: −0.9, 0, −0.5 and −1.
The top-left kernel weight multiplies the top-left pixel of each patch: 1, 2, 0 and 1. Its derivative is (−0.9)(1) + (0)(2) + (−0.5)(0) + (−1)(1) = −1.9, the number the code in section 07 asserts.
In general, for one input channel, stride 1 and no padding, the derivative for each kernel weight is the same sum.
∂L/∂output[row, column] × input[row + u, column + v]
In words: for each output position, take how much the loss wants that output to change and multiply it by the pixel this weight touched there. Add those up over every position. With more input channels, each kernel gets its own sum; with a minibatch, a group of examples processed together, you also add over the examples.
To test backward code, compute the same derivative a second, independent way. A central finite difference nudges one weight up by a small step ε, runs the forward pass, nudges it down by ε, runs it again, and divides the change in loss by 2ε. No backward code is involved, so agreement between the two is strong evidence the backward pass is right.[8]
Real networks make the check harder. If the nudge carries some value across a ReLU's corner at zero, the finite difference averages two different slopes and won't match the backward pass, even when the code is correct. A step that's too small gets lost in rounding. So check in double precision on a small case, keep values away from activation corners, and try several step sizes instead of trusting one tolerance.[9]
07A complete forward and backward pass
The programs below compute the same 3×3 input and 2×2 kernel as the last widget. They assert the hand-computed output (0.1) and gradient (−1.9), then check every weight against a central finite difference. The arrays are small enough that every index is visible, and both programs use double precision, so rounding error stays far below the check's tolerance.
#include <array>
#include <cassert>
#include <cmath>
#include <vector>
struct Result { std::vector<double> output; std::array<double,4> gradient{}; double loss = 0; };
Result evaluate(const std::array<double,9>& input,
const std::array<double,4>& kernel) {
Result result;
// Valid 2x2 cross-correlation on a 3x3 input, one channel, bias zero.
for (int row = 0; row < 2; ++row) for (int column = 0; column < 2; ++column) {
double output = 0;
for (int kernelRow = 0; kernelRow < 2; ++kernelRow)
for (int kernelColumn = 0; kernelColumn < 2; ++kernelColumn)
output += input[(row+kernelRow)*3+column+kernelColumn]
* kernel[kernelRow*2+kernelColumn];
result.output.push_back(output);
const double difference = output - 1.0; // Every target is one in this harness.
result.loss += 0.5 * difference * difference;
for (int kernelRow = 0; kernelRow < 2; ++kernelRow)
for (int kernelColumn = 0; kernelColumn < 2; ++kernelColumn)
// += collects uses of one shared weight at different output positions.
result.gradient[kernelRow*2+kernelColumn] += difference
* input[(row+kernelRow)*3+column+kernelColumn];
}
return result;
}
int main() {
const std::array<double,9> input{1,2,0, 0,1,3, 2,0,1};
const std::array<double,4> kernel{0.2,-0.1,0.3,0.1};
const Result result = evaluate(input,kernel);
assert(std::abs(result.output[0]-0.1) < 1e-12);
assert(std::abs(result.gradient[0]+1.9) < 1e-12);
for (int index = 0; index < 4; ++index) {
auto plus = kernel, minus = kernel;
const double epsilon = 1e-5;
plus[index] += epsilon; minus[index] -= epsilon;
const double numerical = (evaluate(input,plus).loss
- evaluate(input,minus).loss)/(2*epsilon);
// This independently checks the backward pass against the forward loss.
assert(std::abs(numerical-result.gradient[index]) < 1e-8);
}
}
struct Result { output: Vec<f64>, gradient: [f64;4], loss: f64 }
fn evaluate(input: &[f64;9], kernel: &[f64;4]) -> Result {
let mut result = Result { output: Vec::new(), gradient: [0.0;4], loss: 0.0 };
// Valid 2x2 cross-correlation on a 3x3 input, one channel, bias zero.
for row in 0..2 { for column in 0..2 {
let mut output = 0.0;
for kernel_row in 0..2 { for kernel_column in 0..2 {
output += input[(row+kernel_row)*3+column+kernel_column]
* kernel[kernel_row*2+kernel_column];
}}
result.output.push(output);
let difference = output - 1.0; // Every target is one in this harness.
result.loss += 0.5 * difference * difference;
for kernel_row in 0..2 { for kernel_column in 0..2 {
// += collects uses of one shared weight at different output positions.
result.gradient[kernel_row*2+kernel_column] += difference
* input[(row+kernel_row)*3+column+kernel_column];
}}
}}
result
}
fn main() {
let input = [1.0,2.0,0.0, 0.0,1.0,3.0, 2.0,0.0,1.0];
let kernel = [0.2,-0.1,0.3,0.1];
let result = evaluate(&input,&kernel);
assert!((result.output[0]-0.1).abs() < 1e-12);
assert!((result.gradient[0]+1.9).abs() < 1e-12);
for index in 0..4 {
let mut plus = kernel; let mut minus = kernel;
let epsilon = 1e-5;
plus[index] += epsilon; minus[index] -= epsilon;
let numerical = (evaluate(&input,&plus).loss
- evaluate(&input,&minus).loss)/(2.0*epsilon);
// This independently checks the backward pass against the forward loss.
assert!((numerical-result.gradient[index]).abs() < 1e-8);
}
}
The fixed sizes leave out batching, multiple channels, padding, dilation, the bias gradient, and the gradient with respect to the input, which a real layer needs to pass the error back to the layer before it. Adding channels adds one more sum to the forward pass and one more gradient array per kernel. Adding a batch adds a sum over examples. Decide whether the loss is a sum or an average over the batch before you scale the gradient.
To turn this into a small image classifier, run a convolution, apply ReLU, optionally downsample, and repeat. Then average each channel's map to one number and feed those numbers to a dense layer with a softmax. The Neural Networks page derives the gradient for that last layer; the convolution backward pass, plus the input gradient, carries it back into the shared kernels. Production libraries use memory-layout-aware, vectorized or GPU kernels, and a plain scalar loop like this one is a useful reference to test them against.
08Checks before training a larger network
Two implementations can produce the same output shape while padding different sides, flipping the kernel, or ordering channels differently. Compare actual values on a small input that isn't symmetric. A symmetric image or kernel can hide all of these mismatches.
Build up the tests in order. Start with one patch whose products you can check by hand. Then use a single bright pixel away from the center, which shows whether outputs land in the right place. Then put nonzero values in more than one input channel to check the channel sum. For the backward pass, include weights used at several positions and input pixels covered by overlapping patches. Tests that only compare shapes catch none of these.
Check the input pipeline too. A correct model gives bad answers if inference feeds it BGR instead of RGB channel order, normalizes pixels differently, or resizes images differently from training. None of these is a convolution bug, which is why they're easy to miss. Keep preprocessing in one explicit place and compare one image end to end against a reference.
Split the data before you fit any preprocessing or choose an architecture. Neighboring frames from one video are nearly identical, so a random split by frame puts near-copies in both the training and test sets, and the test score can reward memorizing near-copies instead of generalizing. Split by the unit you want the model to generalize across, such as whole videos, scenes, sessions or people.
09What's next
Diffusion Models is the series' other adjacent page: the same training machinery used to generate images instead of classifying them. On the main path, Transformers and Attention replaces the fixed 3×3 neighborhood with mixing that's computed from the input. Each position decides how much to draw from every other position, using weights that are recomputed for every input, while the matrices that produce those weights are trained like any other parameters.
10Sources
The cited equations define the algorithms. Widget readouts report the displayed toy arrays and settings; they are not hardware benchmarks.
- Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, 1998. Gradient-Based Learning Applied to Document Recognition. Local receptive fields and shared trainable kernels.
- PyTorch contributors, Conv2d documentation. Conv2d. Cross-correlation, channel connectivity, dilation, and output shapes.
- Vincent Dumoulin and Francesco Visin, 2016, revised 2018. A guide to convolution arithmetic for deep learning. Stride, padding, dilation, and transposed convolution.
- PyTorch contributors, MaxPool2d documentation. MaxPool2d. The per-channel pooling operator and its padding convention.
- Richard Zhang, 2019. Making Convolutional Networks Shift-Invariant Again. Downsampling, shift sensitivity, and filtering before subsampling.
- André Araujo, Wade Norris, Jack Sim, 2019. Computing Receptive Fields of Convolutional Neural Networks. Dependency spans, unused positions, and multiple paths.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2016. Deep Residual Learning for Image Recognition. Residual addition and dimension-matching projections.
- PyTorch contributors, Gradcheck mechanics. Gradcheck mechanics. Numerical checks of analytic derivatives.
- PyTorch contributors, gradcheck API documentation. gradcheck. Double-precision checks and limitations at nondifferentiable points.