Convolutional Networks from Scratch

01A local operation with shared weights

A dense layer gives each input-to-output connection its own weight. A convolutional layer gives a small neighborhood a weight array and reuses that array at different positions. Moving an edge across an image therefore reuses the same detector. The layer still needs training data that makes the detector useful. Locality and weight sharing constrain the function; they do not supply knowledge of objects.^[1]

The operator used here is cross-correlation, matching PyTorch Conv2d. The kernel is the array of filter weights. Mathematical convolution reverses it along its spatial axes. Learned filters can use either convention if training and inference agree, but copying a hand-written edge filter between conventions can reverse its response sign.^[2]

The sum visits the kernel offsets u and v. The widget uses one input channel, unit stride, no padding, and bias zero.

A feature map is the resulting response array. It is not a list of detected objects. A positive response can mean an edge of one orientation, a color combination, or a pattern useful to a later layer. Reversing an edge can make the response negative. An activation such as ReLU then changes which responses pass forward.

The first widget uses a vertical step image and three hand-selected kernels. The horizontal detector returns zero because this image changes across columns, not across rows. The blur averages each neighborhood. These kernels illustrate the arithmetic; a trained network chooses its weights through the loss and backward pass.

For an ungrouped layer with input-channel count Cᵢ, output-channel count Cₒ, and a K×K kernel, the kernel contains CₒCᵢK² weights, plus Cₒ biases if enabled. The number does not grow with image width or height. A hypothetical 3-to-16-channel, 3×3 layer has 432 kernel weights and 16 biases. Those counts follow directly from the tensor shape; they are not a model-size benchmark.^[2]

02Stride, padding, and dilation

The kernel's placement determines the output's geometry. Stride moves the whole kernel. Padding extends the boundary. Dilation separates the samples inside the kernel. These are distinct operations.^[3]

I is one input dimension, K the number of kernel coefficients on that axis, and O the output dimension. This form assumes equal padding on both sides and a kernel that fits the padded input.

For a 7-position axis, three coefficients, padding one and stride two, there are four legal placements: their left edges are −1, 1, 3, and 5. Increasing dilation to two widens the footprint from three to five positions, while still sampling only three coefficients. With the same padding and stride, the output has three positions.

Zero padding inserts an assumption about the world outside the crop. At a boundary, the detector sees real pixels next to zeros. Reflection or circular padding changes that assumption and the numerical result. Padding preserves a chosen shape; it does not recover missing observations. For odd kernels, unit stride and dilation one, symmetric padding of (K−1)/2 preserves the input size. Stride greater than one needs more care, including asymmetric padding in some definitions of "same."^[3]

Transposed convolution applies the transpose of the forward linear operator. It is also part of how gradients are distributed back to input positions. It is not generally an inverse: subsampling can discard information, and overlapping contributions add together. Output padding resolves a shape ambiguity for some strided configurations; it does not restore the lost values.^[3]

03Channels are summed before activation

An ordinary convolution does more than run one spatial filter independently on red, green, and blue. Each output channel has a separate spatial kernel for each input channel. The channel contributions are summed, one output bias is added, and the activation is applied afterward.^[2]

o selects an output channel; c selects an input channel. Each output channel has its own full set of kernels.

The widget has two input channels and one output channel. One kernel measures a horizontal ramp; the other measures changes between rows. The gain scales only the second channel's kernel. A negative gain reverses its contribution. Cancellation in the summed map is a legitimate result: the layer combines evidence before any nonlinearity is applied.

With grouped convolution, an output channel sees only its group's inputs. With depthwise convolution, each input channel has its own spatial filtering path, possibly with multiple output filters per input. A following 1×1 convolution can mix channels without mixing neighboring positions. The group count changes the parameter formula to Cₒ(Cᵢ/groups)K² for compatible channel counts.^[2]

A 1×1 kernel is therefore useful even though its spatial footprint is one pixel: it learns a channel combination at each location. It can expand or reduce the channel dimension. Applying a nonlinear activation between two such channel-mixing layers changes the representable function; collapsing the two matrices is valid only when the intervening operations allow it.

04Shifts and downsampling

Away from boundaries, a unit-stride convolution with shared weights is translation equivariant: shifting the input shifts the response map. An equivariant feature map preserves location. An invariant prediction stays the same under a transformation. Confusing those properties leads to incorrect claims about what a CNN guarantees.

A stride-two operation samples one phase of the spatial grid. A one-pixel input shift selects a different phase. The output need not be a simple shifted copy. Max pooling also discards positions within its windows; that can make a response insensitive to some movements, but not arbitrary shifts. Its window boundaries still matter.^[4]

The small signal below contains a peak and a weaker neighbor. Circular shifts avoid zero-padding effects, so changes come from the pooling windows. At the initial one-position shift, the two nonzero values fall into separate windows instead of the same window. Max pooling produces a second response. Average pooling preserves their total under this circular, non-overlapping setup, but redistributes it between outputs.

Aliasing is a sampling problem. Filtering before downsampling suppresses high spatial frequencies, reducing phase sensitivity, with a corresponding loss of detail. Zhang's anti-aliased CNN work applies this idea to network downsampling. It does not establish exact invariance to arbitrary translation or prove that blurring improves every task.^[5]

For a classifier, global average pooling can remove the remaining position axis by averaging each channel's spatial map. For localization, throwing away that axis would discard information the output needs. The task determines whether preserving position is useful.

05How much input can one output use?

A later layer combines earlier responses, so its receptive field grows through the stack. Tracking only the last layer's kernel size misses this accumulated dependency. Track both the receptive-field span and the spacing between adjacent outputs in original input coordinates.^[6]

Start with span=1 and jump=1 at the input. jump is the distance between neighboring current-layer positions in input coordinates. This recurrence is for a single sequential path.

Three 3-coefficient layers at unit stride and dilation one span seven input positions. If only the first layer has stride two, the span becomes eleven, and neighboring final outputs are two input positions apart. These numbers come from adding 2, then 4, then 4 to the initial span of one.

The theoretical dependency set describes what can influence the output. It does not say that every position has equally strong influence. A zero weight removes a path's contribution; a gated activation can remove it for a particular input. Repeated dilation can also leave holes even while the bounding span becomes large. Distinguish the span from the sampled set when evaluating coverage.^[6]

With branching paths, each branch can have a different receptive field or alignment. A residual addition needs compatible tensor shapes. To combine features at the same image location, the paths also need matching spatial alignment. He et al. write the block as F(x)+x, or F(x)+W_s x when a learned projection matches dimensions. It does not make arbitrary mismatched arrays addable.^[7]

06One kernel, many gradient contributions

The backward pass uses the same chain rule as the foundation article. The new bookkeeping is parameter reuse. If one kernel weight affects four outputs, the loss derivative includes its contribution through all four outputs. Overwriting that derivative inside the spatial loop keeps only the last use.

This is one input channel with unit stride and no padding. Additional connected channels have separate kernel coefficients; a minibatch adds another sum.

The widget uses four targets, each equal to one, and a sum of half squared errors. For that loss, the derivative with respect to each output is output−1. The analytic calculation collects the patch contributions. A central finite difference perturbs one weight and compares the forward losses. It supplies an independent check of the backward code.^[8]

A nonlinear network adds qualifications. A finite-difference interval crossing a ReLU corner may disagree with a one-sided branch derivative even when the backward pass is implemented correctly. Very small perturbations can disappear in floating-point rounding. Use double precision for a small reference case, check away from activation corners, and sweep the step size instead of treating one tolerance as proof for every configuration.^[9]

07A complete forward and backward kernel

The following programs compute the same 3×3-input, 2×2-kernel case as the last widget. They assert a hand-computed output and gradient, then check every weight numerically. The arrays are small enough that the indexing remains visible. Both programs use double precision and contain their own assertion checks.

#include <array>
#include <cassert>
#include <cmath>
#include <vector>
struct Result { std::vector<double> output; std::array<double,4> gradient{}; double loss = 0; };
Result evaluate(const std::array<double,9>& input,
                const std::array<double,4>& kernel) {
    Result result;
    // Valid 2x2 cross-correlation on a 3x3 input, one channel, bias zero.
    for (int row = 0; row < 2; ++row) for (int column = 0; column < 2; ++column) {
        double output = 0;
        for (int kernelRow = 0; kernelRow < 2; ++kernelRow)
        for (int kernelColumn = 0; kernelColumn < 2; ++kernelColumn)
            output += input[(row+kernelRow)*3+column+kernelColumn]
                    * kernel[kernelRow*2+kernelColumn];
        result.output.push_back(output);
        const double difference = output - 1.0; // Every target is one in this harness.
        result.loss += 0.5 * difference * difference;
        for (int kernelRow = 0; kernelRow < 2; ++kernelRow)
        for (int kernelColumn = 0; kernelColumn < 2; ++kernelColumn)
            // += collects uses of one shared weight at different output positions.
            result.gradient[kernelRow*2+kernelColumn] += difference
                * input[(row+kernelRow)*3+column+kernelColumn];
    }
    return result;
}
int main() {
    const std::array<double,9> input{1,2,0, 0,1,3, 2,0,1};
    const std::array<double,4> kernel{0.2,-0.1,0.3,0.1};
    const Result result = evaluate(input,kernel);
    assert(std::abs(result.output[0]-0.1) < 1e-12);
    assert(std::abs(result.gradient[0]+1.9) < 1e-12);
    for (int index = 0; index < 4; ++index) {
        auto plus = kernel, minus = kernel;
        const double epsilon = 1e-5;
        plus[index] += epsilon; minus[index] -= epsilon;
        const double numerical = (evaluate(input,plus).loss
                                - evaluate(input,minus).loss)/(2*epsilon);
        // This independently checks the backward pass against the forward loss.
        assert(std::abs(numerical-result.gradient[index]) < 1e-8);
    }
}

struct Result { output: Vec<f64>, gradient: [f64;4], loss: f64 }
fn evaluate(input: &[f64;9], kernel: &[f64;4]) -> Result {
    let mut result = Result { output: Vec::new(), gradient: [0.0;4], loss: 0.0 };
    // Valid 2x2 cross-correlation on a 3x3 input, one channel, bias zero.
    for row in 0..2 { for column in 0..2 {
        let mut output = 0.0;
        for kernel_row in 0..2 { for kernel_column in 0..2 {
            output += input[(row+kernel_row)*3+column+kernel_column]
                    * kernel[kernel_row*2+kernel_column];
        }}
        result.output.push(output);
        let difference = output - 1.0; // Every target is one in this harness.
        result.loss += 0.5 * difference * difference;
        for kernel_row in 0..2 { for kernel_column in 0..2 {
            // += collects uses of one shared weight at different output positions.
            result.gradient[kernel_row*2+kernel_column] += difference
                * input[(row+kernel_row)*3+column+kernel_column];
        }}
    }}
    result
}
fn main() {
    let input = [1.0,2.0,0.0, 0.0,1.0,3.0, 2.0,0.0,1.0];
    let kernel = [0.2,-0.1,0.3,0.1];
    let result = evaluate(&input,&kernel);
    assert!((result.output[0]-0.1).abs() < 1e-12);
    assert!((result.gradient[0]+1.9).abs() < 1e-12);
    for index in 0..4 {
        let mut plus = kernel; let mut minus = kernel;
        let epsilon = 1e-5;
        plus[index] += epsilon; minus[index] -= epsilon;
        let numerical = (evaluate(&input,&plus).loss
                        - evaluate(&input,&minus).loss)/(2.0*epsilon);
        // This independently checks the backward pass against the forward loss.
        assert!((numerical-result.gradient[index]).abs() < 1e-8);
    }
}

The fixed dimensions deliberately omit batching, multiple channels, padding, dilation, bias gradients, and the gradient with respect to the input. Adding channels introduces a sum in the forward pass and a matching contribution per connected kernel in the backward pass. Adding a batch introduces another accumulated sum. Decide whether the loss is summed or averaged before scaling its derivative.

To make a small classifier, replace an early dense layer with a convolution, apply an activation, optionally downsample, then reduce the spatial maps and feed a classification head. The cross-entropy backward pass already derived in the foundation article supplies the head's gradient. The convolution backward pass distributes it to the shared weights. A production implementation needs layout-aware kernels, vectorization or GPU execution, and verified agreement on padding and shape rules; the scalar loop defines the reference behavior.

08Checks before training a larger network

Start with a single patch whose products can be checked by hand. Then test an off-center impulse to verify spatial alignment. Use nonzero values in more than one input channel to verify channel summation. For a backward pass, check both reused weights and inputs reached by overlapping patches. Shape-only assertions miss these errors.

Validate the full input path too. A correct model can fail when inference swaps RGB and BGR, changes normalization, or resizes images differently from training. These are changes to the function's input, not arithmetic defects in convolution. Keep preprocessing explicit and compare an end-to-end reference example.

Separate related samples before fitting preprocessing or choosing an architecture. Adjacent frames from one capture can be highly similar; a frame-level random split can evaluate memorization of the capture rather than generalization to another one. The useful split unit follows the claim being evaluated, such as a scene, session, or subject.

09What's next

Transformers and Attention replaces a fixed spatial neighborhood with input-dependent mixing between sequence positions. The weights used to form queries, keys, and values are still trained parameters; the resulting attention weights are recomputed for each input.

10Sources

The cited equations define the algorithms. Widget readouts report the displayed toy arrays and settings; they are not hardware benchmarks.

1. Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, 1998. Gradient-Based Learning Applied to Document Recognition. Local receptive fields and shared trainable kernels.
2. PyTorch contributors, Conv2d documentation. Conv2d. Cross-correlation, channel connectivity, dilation, and output shapes.
3. Vincent Dumoulin and Francesco Visin, 2016, revised 2018. A guide to convolution arithmetic for deep learning. Stride, padding, dilation, and transposed convolution.
4. PyTorch contributors, MaxPool2d documentation. MaxPool2d. The per-channel pooling operator and its padding convention.
5. Richard Zhang, 2019. Making Convolutional Networks Shift-Invariant Again. Downsampling, shift sensitivity, and filtering before subsampling.
6. André Araujo, Wade Norris, Jack Sim, 2019. Computing Receptive Fields of Convolutional Neural Networks. Dependency spans, unused positions, and multiple paths.
7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2016. Deep Residual Learning for Image Recognition. Residual addition and dimension-matching projections.
8. PyTorch contributors, Gradcheck mechanics. Gradcheck mechanics. Numerical checks of analytic derivatives.
9. PyTorch contributors, gradcheck API documentation. gradcheck. Double-precision checks and limitations at nondifferentiable points.