AI tutorialsMighty Professional
Build a Language Model · Other model families

Diffusion Models from Scratch

A diffusion model generates an image by starting from pure random noise and removing a little of it at a time until a picture is left. It learns that from a simple training game: take a real example, add a known amount of noise, and ask the network to guess the noise. This page builds the whole loop on a toy dataset small enough to compute exactly: adding noise, what the best possible guess looks like, training a predictor, generating samples step by step, and steering them toward a label with guidance.

Time~45 minLevelIntermediatePrereqsNeural Networks for regression and gradients; Gaussian distributions are explained here.StackC++20 & Rust · browser demos
◂ Build a Language ModelAdjacent · Other model familiesBack to the hub ▸
What you'll be able to do by the end

Write the formula that jumps a clean example to any noise level in one line. Explain why the best noise prediction is an average, and why that average can be a value no real example has. Run one deterministic DDIM sampling step by hand. Say what a classifier-free guidance scale of 0, 1 or 3 does to a model's prediction. Convert between noise, clean-value and velocity predictions so a sampler reads a model's output correctly.

01Add noise with a recipe you control

Say you want a program that generates new images that look like the ones in a training set. There's no formula for "a realistic photo" to aim at. A gets around that. Training takes real examples and adds random noise to them in a fixed, known way. The training code drew that noise itself, so it knows exactly what the noise was and can ask a network to predict it, with no human labeling. To generate, the model starts from pure noise and repeatedly removes some of the noise it predicts.[1]

The recipe blends the clean value with noise drawn from a Gaussian, the bell-curve distribution, using two weights: one shrinks the signal and one scales the noise. Their squares add up to 1, which is why the recipe is called variance-preserving: if the clean data has variance 1, the noisy data does too. One number, ᾱ ("alpha-bar"), sets the noise level. It's the fraction of the signal's power that's left: ᾱ = 1 is clean, and ᾱ near 0 is almost pure noise. Training can jump straight to any noise level in one line, without adding noise step by step.[1]

Worked example

Clean value x₀ = 2, noise draw ε = −0.8, noise level ᾱ = 0.4.

Signal weight √0.4 ≈ 0.632. Noise weight √0.6 ≈ 0.775.

Noisy value x_t = 0.632 × 2 + 0.775 × (−0.8) ≈ 1.265 − 0.620 = 0.645.

These are the defaults of the last widget on this page, which reads x_t = 0.64523.

The full recipe builds ᾱ from a per-step schedule, then gives the one-line jump used in the example.

α_t = 1 − β_t; ᾱ_t = product of α₁ through α_t
x_t = √ᾱ_t × x₀ + √(1 − ᾱ_t) × ε

In words: step t keeps a fraction α_t of the signal's power and replaces the rest, β_t, with noise. Multiply those kept fractions over every step so far and you get ᾱ_t, the signal power left after t steps. Any noisy version is then √ᾱ_t times the clean value plus √(1 − ᾱ_t) times a fresh draw ε from a standard bell curve (mean 0, variance 1).

For one fixed clean value x₀, the noisy x_t is spread around √ᾱ_t x₀ with variance 1 − ᾱ_t, where variance is the average squared distance from the mean. That's only the noise's spread. Across a whole dataset the clean values vary too, so the total variance of x_t is ᾱ_t times the data's variance, plus 1 − ᾱ_t.

Even a small 256×256 color image is 196,608 numbers, far too many for exact answers. So this page uses a dataset with one number per example. Half the values come from a bell curve centered at −2 and half from one centered at +2, each with variance 0.16 (a standard deviation of 0.4). That's a mixture of two Gaussians: two clearly separate clusters, standing in for two kinds of picture. At noise level ᾱ, each cluster's center shrinks to ±2√ᾱ and its variance becomes 0.16ᾱ + 1 − ᾱ.

Follow a mixture as noise increases

You're looking at 96 examples from the two-cluster dataset after adding noise (left) and the exact density of the noisy data (right). Drag the noise level: at 0 the clusters sit at ±2, and as noise rises they slide toward 0 and blur into one bell curve. At the default of 55, ᾱ = 0.42, so the cluster centers have moved to about ±1.30 and each has variance 0.65. Height only spreads the dots out. Each example keeps its own noise draw as you drag so you can follow it; that's for comparison, and it isn't how the forward process draws noise step by step.

How fast ᾱ falls from 1 to 0 across the steps is the noise schedule, and it decides how much training the model gets at each noise level. Nichol and Dhariwal proposed a cosine-shaped schedule with a small offset.[2] The widget's slider uses a plain curve, ᾱ = cos²(slider/100 × 90°), which covers the full range but isn't their exact schedule.

02The best guess can be a value no example has

Hand a model a noisy value and ask for the clean one. Often several clean values could have produced it: a noisy 0 in the toy dataset could have come from the −2 cluster or the +2 cluster, equally likely. A model trained with squared error learns to output the average of the possibilities, each weighted by how likely it is. That's the . Squared error rewards it because it gives the smallest expected squared miss, even when the average itself is a value that almost never occurs in the data.

Worked example

Noisy value x_t = 0 at ᾱ = 0.4. The two clusters are equally likely sources.

If it came from the −2 cluster, the best guess for the clean value is about −1.81. From the +2 cluster, about +1.81. The posterior mean averages them: 0, in the gap where almost no clean data lies.

Why squared error picks 0: if the truth is −1.81 or +1.81 with equal odds, guessing 0 costs 1.81² ≈ 3.3 on average. Guessing +1.81 costs nothing half the time and 3.61² ≈ 13.0 the other half, so about 6.5 on average.

For one cluster, the best guess has a closed form. With s = √ᾱ for the signal weight and σ² = 1 − ᾱ for the noise variance, it starts at the cluster's center and moves toward what the observation suggests.

E[x₀ | x_t, component] = μ + (s τ² / (s²τ² + σ²)) × (x_t − s μ)

In words: start from the cluster's center μ, then correct it by how far the observation x_t lands from where that center would sit after noise, s μ. The correction is bigger when the cluster's own spread τ² is large compared with the noise σ²: the less noise, the more the observation is trusted. For the two-cluster mixture, compute this for each cluster and average the two, weighting each by its Bayes responsibility: how likely that cluster is to have produced x_t.

Where the formula comes from

Multiply the cluster's bell curve for x₀ by the bell curve for x_t given x₀, which is the noise recipe, and rearrange the exponent into a single square in x₀ (completing the square). The result is another bell curve. Its center is the formula above and its variance is τ²σ² / (s²τ² + σ²), which is 0.145 in the worked example. The responsibility of each cluster is its noisy density at x_t times its share of the data (one half each here), divided by the total over both clusters.

Compare posterior modes with their mean

You're looking at the density of noisy observations with your chosen observation marked (left), and the exact density of the clean value given that observation, with its mean marked (right). Drag the observation and the noise level. At the defaults (observation 0, ᾱ = 0.4) the right panel has two humps near ±1.8 and the mean sits at 0 between them. Move the observation toward ±2 and one hump takes over, pulling the mean onto it. Every curve here is exact arithmetic for the known mixture.

The best noise guess follows from the best clean guess by rearranging the recipe: E[ε | x_t] = (x_t − s E[x₀ | x_t]) / σ, for σ > 0. It's also tied to the score of the noisy data, the slope of its log-density, which points toward where noisy data is more likely: E[ε | x_t] = −σ ∇log p_t(x_t). So predicting the noise and pointing uphill toward likely data are the same job. Song and Ermon built a generator directly on scores of noisy data.[3] This page computes the score exactly from the known mixture.

That's also why the averaging isn't fatal. A diffusion sampler doesn't take one posterior mean of a heavily noised sample and stop. It takes a step in the predicted direction, predicts again at a lower noise level, and repeats. At low noise the observation pins down which cluster it came from, and the best guess stops sitting between them.

03Train a predictor on noise you added

Training is the easy part, because the training code makes its own targets. Pick a clean example, pick a noise level, draw noise, build x_t with the recipe, and ask the network ε̂_θ(x_t, t) to predict the noise you drew. The loss is the squared error between that prediction and the real noise; Ho et al.'s simplified objective is exactly this.[1]

Worked example

Take the example from section 01: x₀ = 2, ᾱ = 0.4, ε = −0.8, so x_t ≈ 0.645.

If the network predicts ε̂ = 0, this example's loss is (−0.8 − 0)² = 0.64. A prediction of −0.5 would cost (−0.8 + 0.5)² = 0.09.

The best possible predictor for the toy mixture guesses ε̂ ≈ −0.49 here, because other clean-and-noise pairs also land on 0.645. Its loss on this example is (−0.8 + 0.49)² ≈ 0.096, not 0.

Averaged over many examples, noise draws and noise levels, that gives the training objective.

loss = E[ || ε − ε̂_θ(x_t,t) ||² ]

In words: the average squared gap between the noise you added and the noise the network predicts. The network sees the noisy sample and the noise level t. It never sees the clean example.

No network can recover the exact noise from x_t alone: the clean value is hidden, and different clean-plus-noise pairs produce the same x_t. Under squared error the best possible predictor outputs the expected noise given x_t, the average from section 02. A real network approximates that function from a finite training set.

The next widget trains a tiny model, a straight line ε̂ = w x_t + b, by gradient descent on 64 fixed noisy examples at one noise level. A line can't follow the mixture's curved best-guess function, so the loss falls and then levels off well above zero.

Fit a noise predictor with gradient descent

You're looking at the 64 training pairs, each a noisy value against the noise that was added (dots), with the fitted line, and the training loss after each update. Press Step or Play to take gradient steps; changing ᾱ or the learning rate restarts training. At the defaults the loss starts at 1.016 with w = b = 0 and levels off near 0.693 within about 20 steps, at w ≈ 0.336. That's as good as a straight line at one noise level can do.

A real network also takes the noise level as an input, and it needs to: the same noisy value calls for a different noise guess at different levels. Check the loss at each noise level as well as the average, because an average over all levels can hide a range where the model is bad.

Two more cautions. Changing what the network predicts (the noise, the clean value, or the velocity from section 06), or how the loss weights each noise level, changes what training optimizes, even though the predictions convert into each other exactly. And a low denoising loss alone doesn't show that samples look right, cover the data's variety, or hold up under a different sampler.

04Generate by stepping toward less noise

To generate, start from pure noise at a very high noise level. Ask the model what noise it sees, remove part of it, and land at a slightly lower noise level. Repeat until the noise is gone. Some samplers add a little fresh randomness at each step. The used here, with its randomness setting η at 0, adds none.[4]

Each step does two things. First it estimates the clean value by running the noise recipe backward with the predicted noise. Then it rebuilds a noisy value at the next, lower noise level from that estimate and the same predicted noise.

Worked example

Current value x_t = 0.645 at ᾱ = 0.4, stepping to ᾱ_next = 0.7. The exact predictor for the toy mixture says ε̂ ≈ −0.490.

Clean estimate: x̂₀ = (0.645 − 0.775 × (−0.490)) / 0.632 = 1.025 / 0.632 ≈ 1.621.

Next value: √0.7 × 1.621 + √0.3 × (−0.490) ≈ 1.356 − 0.269 = 1.087.

With the true noise, −0.8, the clean estimate would be exactly 2. The code in section 07 checks that case.

Written out, the two parts of the step are:

x̂₀ = (x_t − √(1 − ᾱ_t) × ε̂) / √ᾱ_t
x_next = √ᾱ_next × x̂₀ + √(1 − ᾱ_next) × ε̂

In words: estimate the clean value by subtracting the predicted noise at its current weight and undoing the signal's shrink. Then blend that estimate with the same predicted noise, using the next level's weights. ᾱ_next is larger than ᾱ_t, so each step carries more signal and less noise. Both parts come from one model prediction.

With η = 0 the only randomness is the starting noise. The same starting draw always gives the same output, and different draws give different outputs. A stochastic sampler, such as the original DDPM one, draws fresh noise at every intermediate step; DDPM's final step adds none.[1]

Run deterministic reverse trajectories

You're looking at 64 samples as they move (left) and the paths of six of them (right). Press Step to run one reverse update for every sample, or Play to run them all; changing the number of steps restarts. The samples start as random draws from a bell curve centered at 0 and, after the default 24 steps, end in two clusters near −2 and +2. The noise predictor is the exact best guess for the known mixture, not the line trained above. Fewer steps make each update coarser: at 8 steps the clusters end slightly closer to 0.

The demo starts at ᾱ = 10⁻⁴, not exactly 0, so a trace of the data remains, and a standard bell curve is a close stand-in for the noisy data there. Even with the exact predictor, the finite number of steps and that approximate start keep the output from being an exact draw from the mixture. When samples look wrong, both the predictor and the sampler are suspects.

In an image model, x_t is a whole array of pixels, or of latent values from a compressed version of the image, and the network predicts a noise array of the same shape. The update above runs on each number separately, but the network looks at all of them together to decide what noise it sees. The one-number toy isolates the sampling arithmetic; it doesn't generate anything like an image.

05Steer generation with guidance

A model that generates from a prompt has to make what the prompt asks for. So the network also takes a condition, such as a class label or an encoded text prompt. makes it follow the condition more strongly. At each step it computes two noise predictions, one with the condition and one without, and pushes further in the direction where they differ. In Ho and Salimans' method, training drops the condition on a random fraction of examples, so one network learns to make both predictions.[5]

Worked example

In the guidance widget, the condition means "from the +2 cluster". At x_t = −1 and ᾱ = 0.4, the unconditional prediction is ε̂ ≈ 0.25 and the conditional one is ε̂ ≈ −2.64.

With guidance scale g = 2: 0.25 + 2 × (−2.64 − 0.25) = 0.25 − 5.78 = −5.53.

The guided prediction overshoots the conditional one, so the step pushes the sample harder toward +2.

The combination is one line.

ε̂_guided = ε̂_unconditional + g × (ε̂_conditional − ε̂_unconditional)

In words: start from the unconditional prediction and move toward the conditional one by g times the gap between them. g = 0 ignores the condition, g = 1 gives the plain conditional prediction, and g above 1 goes past it. Ho and Salimans write the scale as w, where g = 1 + w.[5]

"Classifier-free" means no separately trained classifier network is needed to steer the sampler, unlike the earlier classifier-guidance method. The condition itself is still there. In the widget, the unconditional data is the two-cluster mixture and the condition picks the +2 cluster. Both predictions are exact, so you can see exactly what combining them does.

Combine conditional and unconditional scores

You're looking at the score, the direction toward more likely data, for the unconditional and conditional models (left), and the guided score (right). Drag the guidance scale and ᾱ. At the default scale of 2, the readout gives the matching noise predictions at x_t = −1: 0.2451 without the condition, −2.6422 with it, and −5.5294 guided. The best noise guess is −√(1 − ᾱ) times the score, so the same blend applies to either. A high scale is an extrapolation, not a probability, and doesn't guarantee better output.

What distribution does guidance aim at?

At each noise level, the guided score is the slope of g log p_cond + (1 − g) log p_uncond. Where it can be normalized, that matches a density proportional to p_condg p_uncond1−g at that level. This holds level by level; it doesn't by itself prove that a finite guided sampler ends at that density.[5]

Raising the scale makes samples follow the condition more closely and reduces their variety. Ho and Salimans measure this as a trade-off between sample quality and diversity, similar to the one classifier guidance gives.[5] The best setting depends on the model and the task, so measure it.

06Noise, clean value or velocity: what the model predicts

A diffusion model's output doesn't have to be the noise. Some models predict the clean value x₀ instead, and some predict a blend of the two called the velocity, v.[6] All three carry the same information when they're exact, but the sampler has to know which one it's getting. Feed a velocity output to a sampler that expects noise and it produces wrong samples without any error.

With s = √ᾱ as the signal weight and σ = √(1 − ᾱ) as the noise weight, the velocity is v = sε − σx₀. The name is a convention; it doesn't describe anything moving.

Worked example

Same values as before: ᾱ = 0.4, x₀ = 2, ε = −0.8, so s ≈ 0.632, σ ≈ 0.775 and x_t ≈ 0.645.

Velocity: v = 0.632 × (−0.8) − 0.775 × 2 ≈ −0.506 − 1.549 = −2.055.

Back to the clean value: s × x_t − σ × v ≈ 0.408 + 1.592 = 2.000. Back to the noise: σ × x_t + s × v ≈ 0.500 − 1.300 = −0.800.

The three conversions, all using the same two weights:

x_t = s × x₀ + σ × ε; v = s × ε − σ × x₀
x₀ = s × x_t − σ × v
ε = σ × x_t + s × v

In words: the noisy value and the velocity are two different blends of the same clean value and noise. Because s² + σ² = 1, the pair (x_t, v) is the pair (x₀, ε) rotated, and rotating back recovers either one exactly. Getting x₀ from a noise prediction instead means dividing by s.

With exact values all three routes agree. With a model's imperfect predictions they don't. Recovering x₀ from a noise prediction multiplies the noise error by −σ/s, which grows without limit as ᾱ approaches 0: at ᾱ = 0.01, a noise error of 0.1 becomes a clean-value error of about 1. Salimans and Ho discuss the prediction target and the loss weighting together for this reason; the targets converting into each other doesn't make the training objectives equivalent.[6]

Reconstruct from exact target representations

You're looking at the clean value, the noise, the noisy value and the velocity they produce (left), and the clean value recovered from the noise and from the velocity (right). Drag ᾱ, the clean value or the noise. At the defaults (ᾱ = 0.4, x₀ = 2, ε = −0.8) the readout shows x_t = 0.64523 and v = −2.05516, and both routes recover exactly 2. This checks the algebra with exact values. It says nothing about how well a trained model predicts any of them.

Store the prediction type with the model weights and the schedule. Converting needs the current s and σ, and matching array shapes won't catch a mix-up.

07A deterministic DDIM step in two languages

The programs below implement one deterministic DDIM step, from ᾱ = 0.4 to 0.7, on the section 01 example. Their checks feed in the exact noise, −0.8, so the results are known in advance: the clean estimate must be 2, a step all the way to ᾱ = 1 must land on 2, and the velocity conversion must recover 2. There's no trained model in the code.

An asserted deterministic reverse step, complete programs
#include <cassert>
#include <cmath>
struct Step { double clean; double next; };
Step ddimStep(double noisy, double predictedNoise,
              double alphaBar, double nextAlphaBar) {
    assert(alphaBar > 0 && alphaBar < 1);
    assert(nextAlphaBar >= alphaBar && nextAlphaBar <= 1);
    // eta=0: no fresh random draw enters this reverse update.
    // Invert the known corruption using this level's prediction.
    const double clean = (noisy-std::sqrt(1-alphaBar)*predictedNoise)
                       / std::sqrt(alphaBar);
    // Reuse that prediction as the deterministic direction at the next level.
    const double next = std::sqrt(nextAlphaBar)*clean
                      + std::sqrt(1-nextAlphaBar)*predictedNoise;
    return {clean,next};
}
int main() {
    const double clean = 2.0, noise = -0.8;
    const double alphaBar = 0.4, nextAlphaBar = 0.7;
    const double noisy = std::sqrt(alphaBar)*clean+std::sqrt(1-alphaBar)*noise;
    const Step result = ddimStep(noisy,noise,alphaBar,nextAlphaBar);
    assert(std::abs(result.clean-clean) < 1e-12);
    const double expected = std::sqrt(nextAlphaBar)*clean
                          + std::sqrt(1-nextAlphaBar)*noise;
    assert(std::abs(result.next-expected) < 1e-12);
    const Step final = ddimStep(noisy,noise,alphaBar,1.0);
    assert(std::abs(final.next-clean) < 1e-12); // Final noise coefficient is zero.
    const double velocity = std::sqrt(alphaBar)*noise-std::sqrt(1-alphaBar)*clean;
    const double fromVelocity = std::sqrt(alphaBar)*noisy-std::sqrt(1-alphaBar)*velocity;
    assert(std::abs(fromVelocity-clean) < 1e-12);
}
struct Step { clean: f64, next: f64 }
fn ddim_step(noisy: f64, predicted_noise: f64,
             alpha_bar: f64, next_alpha_bar: f64) -> Step {
    assert!(alpha_bar > 0.0 && alpha_bar < 1.0);
    assert!(next_alpha_bar >= alpha_bar && next_alpha_bar <= 1.0);
    // eta=0: no fresh random draw enters this reverse update.
    // Invert the known corruption using this level's prediction.
    let clean = (noisy-(1.0-alpha_bar).sqrt()*predicted_noise)/alpha_bar.sqrt();
    // Reuse that prediction as the deterministic direction at the next level.
    let next = next_alpha_bar.sqrt()*clean
             + (1.0-next_alpha_bar).sqrt()*predicted_noise;
    Step { clean,next }
}
fn main() {
    let clean = 2.0; let noise = -0.8;
    let alpha_bar: f64 = 0.4; let next_alpha_bar: f64 = 0.7;
    let noisy = alpha_bar.sqrt()*clean+(1.0-alpha_bar).sqrt()*noise;
    let result = ddim_step(noisy,noise,alpha_bar,next_alpha_bar);
    assert!((result.clean-clean).abs() < 1e-12);
    let expected = next_alpha_bar.sqrt()*clean+(1.0-next_alpha_bar).sqrt()*noise;
    assert!((result.next-expected).abs() < 1e-12);
    let final_step = ddim_step(noisy,noise,alpha_bar,1.0);
    assert!((final_step.next-clean).abs() < 1e-12); // Final noise coefficient is zero.
    let velocity = alpha_bar.sqrt()*noise-(1.0-alpha_bar).sqrt()*clean;
    let from_velocity = alpha_bar.sqrt()*noisy-(1.0-alpha_bar).sqrt()*velocity;
    assert!((from_velocity-clean).abs() < 1e-12);
}

A full sampler also needs a model call at every level, the same time input the model was trained with, the model's prediction type, and the same data or latent scaling used in training. A stochastic sampler needs its own noise-variance formula and independent random draws; borrowing the variance formula from a different sampler changes the method.

The program only accepts current noise levels with 0 < ᾱ < 1. At ᾱ = 0 the clean estimate would divide by zero, and ᾱ = 1 means the sample is already clean. The widget starts at a small positive ᾱ and treats the final target level as ᾱ = 1. Code that evaluates one of these endpoints needs to define what happens there instead of dividing by zero.

08Separate predictor errors from sampler errors

Test in stages. First run the sampler with exact targets on a known distribution, like this page's mixture. Next test the trained predictor at individual noise levels. Only then judge generated samples. A correct sampler can't make up for a wrong time index, wrong data normalization, or a prediction in the wrong convention.

This page's widgets are split the same way. The posterior widget is exact Bayes arithmetic, the line trainer fits one fixed dataset, and the sampler runs with a known exact predictor. Testing the parts separately catches errors that a good-looking final animation can hide.

A good-looking sample proves little

A sample can look plausible and still fail to match a specific original, ignore the requested condition, or cover only part of the data. Measure the property you need; the denoising formula guarantees none of these.

09What's next

This is the last adjacent page in the series; the Build a Language Model hub shows where it fits. On the main path, AI Inference and Quantization covers running a trained model: numeric stability, storing numbers as small integers, memory traffic, and sampling from output probabilities.

10Sources

The cited equations define the algorithms. Widget readouts report the displayed toy arrays and settings; they are not hardware benchmarks.

  1. Jonathan Ho, Ajay Jain, Pieter Abbeel, 2020. Denoising Diffusion Probabilistic Models. Gaussian forward marginals, noise-prediction training, and reverse sampling.
  2. Alexander Quinn Nichol and Prafulla Dhariwal, 2021. Improved Denoising Diffusion Probabilistic Models. Noise scheduling, including the offset cosine schedule.
  3. Yang Song and Stefano Ermon, 2019. Generative Modeling by Estimating Gradients of the Data Distribution. Scores of noise-perturbed distributions and iterative generation.
  4. Jiaming Song, Chenlin Meng, Stefano Ermon, 2021. Denoising Diffusion Implicit Models. Deterministic reverse steps and sampling on selected noise levels.
  5. Jonathan Ho and Tim Salimans, 2022. Classifier-Free Diffusion Guidance. Combining conditional and unconditional denoising predictions.
  6. Tim Salimans and Jonathan Ho, 2022. Progressive Distillation for Fast Sampling of Diffusion Models. The velocity parameterization and reconstruction identities.