Diffusion Models from Scratch
A diffusion model generates an image by starting from pure random noise and removing a little of it at a time until a picture is left. It learns that from a simple training game: take a real example, add a known amount of noise, and ask the network to guess the noise. This page builds the whole loop on a toy dataset small enough to compute exactly: adding noise, what the best possible guess looks like, training a predictor, generating samples step by step, and steering them toward a label with guidance.
Write the formula that jumps a clean example to any noise level in one line. Explain why the best noise prediction is an average, and why that average can be a value no real example has. Run one deterministic DDIM sampling step by hand. Say what a classifier-free guidance scale of 0, 1 or 3 does to a model's prediction. Convert between noise, clean-value and velocity predictions so a sampler reads a model's output correctly.
01Add noise with a recipe you control
Say you want a program that generates new images that look like the ones in a training set. There's no formula for "a realistic photo" to aim at. A gets around that. Training takes real examples and adds random noise to them in a fixed, known way. The training code drew that noise itself, so it knows exactly what the noise was and can ask a network to predict it, with no human labeling. To generate, the model starts from pure noise and repeatedly removes some of the noise it predicts.[1]
The recipe blends the clean value with noise drawn from a Gaussian, the bell-curve distribution, using two weights: one shrinks the signal and one scales the noise. Their squares add up to 1, which is why the recipe is called variance-preserving: if the clean data has variance 1, the noisy data does too. One number, ᾱ ("alpha-bar"), sets the noise level. It's the fraction of the signal's power that's left: ᾱ = 1 is clean, and ᾱ near 0 is almost pure noise. Training can jump straight to any noise level in one line, without adding noise step by step.[1]
Worked example
Clean value x₀ = 2, noise draw ε = −0.8, noise level ᾱ = 0.4.
Signal weight √0.4 ≈ 0.632. Noise weight √0.6 ≈ 0.775.
Noisy value x_t = 0.632 × 2 + 0.775 × (−0.8) ≈ 1.265 − 0.620 = 0.645.
These are the defaults of the last widget on this page, which reads x_t = 0.64523.
The full recipe builds ᾱ from a per-step schedule, then gives the one-line jump used in the example.
x_t = √ᾱ_t × x₀ + √(1 − ᾱ_t) × ε
In words: step t keeps a fraction α_t of the signal's power and replaces the rest, β_t, with noise. Multiply those kept fractions over every step so far and you get ᾱ_t, the signal power left after t steps. Any noisy version is then √ᾱ_t times the clean value plus √(1 − ᾱ_t) times a fresh draw ε from a standard bell curve (mean 0, variance 1).
For one fixed clean value x₀, the noisy x_t is spread around √ᾱ_t x₀ with variance 1 − ᾱ_t, where variance is the average squared distance from the mean. That's only the noise's spread. Across a whole dataset the clean values vary too, so the total variance of x_t is ᾱ_t times the data's variance, plus 1 − ᾱ_t.
Even a small 256×256 color image is 196,608 numbers, far too many for exact answers. So this page uses a dataset with one number per example. Half the values come from a bell curve centered at −2 and half from one centered at +2, each with variance 0.16 (a standard deviation of 0.4). That's a mixture of two Gaussians: two clearly separate clusters, standing in for two kinds of picture. At noise level ᾱ, each cluster's center shrinks to ±2√ᾱ and its variance becomes 0.16ᾱ + 1 − ᾱ.
How fast ᾱ falls from 1 to 0 across the steps is the noise schedule, and it decides how much training the model gets at each noise level. Nichol and Dhariwal proposed a cosine-shaped schedule with a small offset.[2] The widget's slider uses a plain curve, ᾱ = cos²(slider/100 × 90°), which covers the full range but isn't their exact schedule.
02The best guess can be a value no example has
Hand a model a noisy value and ask for the clean one. Often several clean values could have produced it: a noisy 0 in the toy dataset could have come from the −2 cluster or the +2 cluster, equally likely. A model trained with squared error learns to output the average of the possibilities, each weighted by how likely it is. That's the . Squared error rewards it because it gives the smallest expected squared miss, even when the average itself is a value that almost never occurs in the data.
Worked example
Noisy value x_t = 0 at ᾱ = 0.4. The two clusters are equally likely sources.
If it came from the −2 cluster, the best guess for the clean value is about −1.81. From the +2 cluster, about +1.81. The posterior mean averages them: 0, in the gap where almost no clean data lies.
Why squared error picks 0: if the truth is −1.81 or +1.81 with equal odds, guessing 0 costs 1.81² ≈ 3.3 on average. Guessing +1.81 costs nothing half the time and 3.61² ≈ 13.0 the other half, so about 6.5 on average.
For one cluster, the best guess has a closed form. With s = √ᾱ for the signal weight and σ² = 1 − ᾱ for the noise variance, it starts at the cluster's center and moves toward what the observation suggests.
In words: start from the cluster's center μ, then correct it by how far the observation x_t lands from where that center would sit after noise, s μ. The correction is bigger when the cluster's own spread τ² is large compared with the noise σ²: the less noise, the more the observation is trusted. For the two-cluster mixture, compute this for each cluster and average the two, weighting each by its Bayes responsibility: how likely that cluster is to have produced x_t.
Where the formula comes from
Multiply the cluster's bell curve for x₀ by the bell curve for x_t given x₀, which is the noise recipe, and rearrange the exponent into a single square in x₀ (completing the square). The result is another bell curve. Its center is the formula above and its variance is τ²σ² / (s²τ² + σ²), which is 0.145 in the worked example. The responsibility of each cluster is its noisy density at x_t times its share of the data (one half each here), divided by the total over both clusters.
The best noise guess follows from the best clean guess by rearranging the recipe: E[ε | x_t] = (x_t − s E[x₀ | x_t]) / σ, for σ > 0. It's also tied to the score of the noisy data, the slope of its log-density, which points toward where noisy data is more likely: E[ε | x_t] = −σ ∇log p_t(x_t). So predicting the noise and pointing uphill toward likely data are the same job. Song and Ermon built a generator directly on scores of noisy data.[3] This page computes the score exactly from the known mixture.
That's also why the averaging isn't fatal. A diffusion sampler doesn't take one posterior mean of a heavily noised sample and stop. It takes a step in the predicted direction, predicts again at a lower noise level, and repeats. At low noise the observation pins down which cluster it came from, and the best guess stops sitting between them.
03Train a predictor on noise you added
Training is the easy part, because the training code makes its own targets. Pick a clean example, pick a noise level, draw noise, build x_t with the recipe, and ask the network ε̂_θ(x_t, t) to predict the noise you drew. The loss is the squared error between that prediction and the real noise; Ho et al.'s simplified objective is exactly this.[1]
Worked example
Take the example from section 01: x₀ = 2, ᾱ = 0.4, ε = −0.8, so x_t ≈ 0.645.
If the network predicts ε̂ = 0, this example's loss is (−0.8 − 0)² = 0.64. A prediction of −0.5 would cost (−0.8 + 0.5)² = 0.09.
The best possible predictor for the toy mixture guesses ε̂ ≈ −0.49 here, because other clean-and-noise pairs also land on 0.645. Its loss on this example is (−0.8 + 0.49)² ≈ 0.096, not 0.
Averaged over many examples, noise draws and noise levels, that gives the training objective.
In words: the average squared gap between the noise you added and the noise the network predicts. The network sees the noisy sample and the noise level t. It never sees the clean example.
No network can recover the exact noise from x_t alone: the clean value is hidden, and different clean-plus-noise pairs produce the same x_t. Under squared error the best possible predictor outputs the expected noise given x_t, the average from section 02. A real network approximates that function from a finite training set.
The next widget trains a tiny model, a straight line ε̂ = w x_t + b, by gradient descent on 64 fixed noisy examples at one noise level. A line can't follow the mixture's curved best-guess function, so the loss falls and then levels off well above zero.
A real network also takes the noise level as an input, and it needs to: the same noisy value calls for a different noise guess at different levels. Check the loss at each noise level as well as the average, because an average over all levels can hide a range where the model is bad.
Two more cautions. Changing what the network predicts (the noise, the clean value, or the velocity from section 06), or how the loss weights each noise level, changes what training optimizes, even though the predictions convert into each other exactly. And a low denoising loss alone doesn't show that samples look right, cover the data's variety, or hold up under a different sampler.
04Generate by stepping toward less noise
To generate, start from pure noise at a very high noise level. Ask the model what noise it sees, remove part of it, and land at a slightly lower noise level. Repeat until the noise is gone. Some samplers add a little fresh randomness at each step. The used here, with its randomness setting η at 0, adds none.[4]
Each step does two things. First it estimates the clean value by running the noise recipe backward with the predicted noise. Then it rebuilds a noisy value at the next, lower noise level from that estimate and the same predicted noise.
Worked example
Current value x_t = 0.645 at ᾱ = 0.4, stepping to ᾱ_next = 0.7. The exact predictor for the toy mixture says ε̂ ≈ −0.490.
Clean estimate: x̂₀ = (0.645 − 0.775 × (−0.490)) / 0.632 = 1.025 / 0.632 ≈ 1.621.
Next value: √0.7 × 1.621 + √0.3 × (−0.490) ≈ 1.356 − 0.269 = 1.087.
With the true noise, −0.8, the clean estimate would be exactly 2. The code in section 07 checks that case.
Written out, the two parts of the step are:
x_next = √ᾱ_next × x̂₀ + √(1 − ᾱ_next) × ε̂
In words: estimate the clean value by subtracting the predicted noise at its current weight and undoing the signal's shrink. Then blend that estimate with the same predicted noise, using the next level's weights. ᾱ_next is larger than ᾱ_t, so each step carries more signal and less noise. Both parts come from one model prediction.
With η = 0 the only randomness is the starting noise. The same starting draw always gives the same output, and different draws give different outputs. A stochastic sampler, such as the original DDPM one, draws fresh noise at every intermediate step; DDPM's final step adds none.[1]
The demo starts at ᾱ = 10⁻⁴, not exactly 0, so a trace of the data remains, and a standard bell curve is a close stand-in for the noisy data there. Even with the exact predictor, the finite number of steps and that approximate start keep the output from being an exact draw from the mixture. When samples look wrong, both the predictor and the sampler are suspects.
In an image model, x_t is a whole array of pixels, or of latent values from a compressed version of the image, and the network predicts a noise array of the same shape. The update above runs on each number separately, but the network looks at all of them together to decide what noise it sees. The one-number toy isolates the sampling arithmetic; it doesn't generate anything like an image.
05Steer generation with guidance
A model that generates from a prompt has to make what the prompt asks for. So the network also takes a condition, such as a class label or an encoded text prompt. makes it follow the condition more strongly. At each step it computes two noise predictions, one with the condition and one without, and pushes further in the direction where they differ. In Ho and Salimans' method, training drops the condition on a random fraction of examples, so one network learns to make both predictions.[5]
Worked example
In the guidance widget, the condition means "from the +2 cluster". At x_t = −1 and ᾱ = 0.4, the unconditional prediction is ε̂ ≈ 0.25 and the conditional one is ε̂ ≈ −2.64.
With guidance scale g = 2: 0.25 + 2 × (−2.64 − 0.25) = 0.25 − 5.78 = −5.53.
The guided prediction overshoots the conditional one, so the step pushes the sample harder toward +2.
The combination is one line.
In words: start from the unconditional prediction and move toward the conditional one by g times the gap between them. g = 0 ignores the condition, g = 1 gives the plain conditional prediction, and g above 1 goes past it. Ho and Salimans write the scale as w, where g = 1 + w.[5]
"Classifier-free" means no separately trained classifier network is needed to steer the sampler, unlike the earlier classifier-guidance method. The condition itself is still there. In the widget, the unconditional data is the two-cluster mixture and the condition picks the +2 cluster. Both predictions are exact, so you can see exactly what combining them does.
What distribution does guidance aim at?
At each noise level, the guided score is the slope of g log p_cond + (1 − g) log p_uncond. Where it can be normalized, that matches a density proportional to p_condg p_uncond1−g at that level. This holds level by level; it doesn't by itself prove that a finite guided sampler ends at that density.[5]
Raising the scale makes samples follow the condition more closely and reduces their variety. Ho and Salimans measure this as a trade-off between sample quality and diversity, similar to the one classifier guidance gives.[5] The best setting depends on the model and the task, so measure it.
06Noise, clean value or velocity: what the model predicts
A diffusion model's output doesn't have to be the noise. Some models predict the clean value x₀ instead, and some predict a blend of the two called the velocity, v.[6] All three carry the same information when they're exact, but the sampler has to know which one it's getting. Feed a velocity output to a sampler that expects noise and it produces wrong samples without any error.
With s = √ᾱ as the signal weight and σ = √(1 − ᾱ) as the noise weight, the velocity is v = sε − σx₀. The name is a convention; it doesn't describe anything moving.
Worked example
Same values as before: ᾱ = 0.4, x₀ = 2, ε = −0.8, so s ≈ 0.632, σ ≈ 0.775 and x_t ≈ 0.645.
Velocity: v = 0.632 × (−0.8) − 0.775 × 2 ≈ −0.506 − 1.549 = −2.055.
Back to the clean value: s × x_t − σ × v ≈ 0.408 + 1.592 = 2.000. Back to the noise: σ × x_t + s × v ≈ 0.500 − 1.300 = −0.800.
The three conversions, all using the same two weights:
x₀ = s × x_t − σ × v
ε = σ × x_t + s × v
In words: the noisy value and the velocity are two different blends of the same clean value and noise. Because s² + σ² = 1, the pair (x_t, v) is the pair (x₀, ε) rotated, and rotating back recovers either one exactly. Getting x₀ from a noise prediction instead means dividing by s.
With exact values all three routes agree. With a model's imperfect predictions they don't. Recovering x₀ from a noise prediction multiplies the noise error by −σ/s, which grows without limit as ᾱ approaches 0: at ᾱ = 0.01, a noise error of 0.1 becomes a clean-value error of about 1. Salimans and Ho discuss the prediction target and the loss weighting together for this reason; the targets converting into each other doesn't make the training objectives equivalent.[6]
Store the prediction type with the model weights and the schedule. Converting needs the current s and σ, and matching array shapes won't catch a mix-up.
07A deterministic DDIM step in two languages
The programs below implement one deterministic DDIM step, from ᾱ = 0.4 to 0.7, on the section 01 example. Their checks feed in the exact noise, −0.8, so the results are known in advance: the clean estimate must be 2, a step all the way to ᾱ = 1 must land on 2, and the velocity conversion must recover 2. There's no trained model in the code.
#include <cassert>
#include <cmath>
struct Step { double clean; double next; };
Step ddimStep(double noisy, double predictedNoise,
double alphaBar, double nextAlphaBar) {
assert(alphaBar > 0 && alphaBar < 1);
assert(nextAlphaBar >= alphaBar && nextAlphaBar <= 1);
// eta=0: no fresh random draw enters this reverse update.
// Invert the known corruption using this level's prediction.
const double clean = (noisy-std::sqrt(1-alphaBar)*predictedNoise)
/ std::sqrt(alphaBar);
// Reuse that prediction as the deterministic direction at the next level.
const double next = std::sqrt(nextAlphaBar)*clean
+ std::sqrt(1-nextAlphaBar)*predictedNoise;
return {clean,next};
}
int main() {
const double clean = 2.0, noise = -0.8;
const double alphaBar = 0.4, nextAlphaBar = 0.7;
const double noisy = std::sqrt(alphaBar)*clean+std::sqrt(1-alphaBar)*noise;
const Step result = ddimStep(noisy,noise,alphaBar,nextAlphaBar);
assert(std::abs(result.clean-clean) < 1e-12);
const double expected = std::sqrt(nextAlphaBar)*clean
+ std::sqrt(1-nextAlphaBar)*noise;
assert(std::abs(result.next-expected) < 1e-12);
const Step final = ddimStep(noisy,noise,alphaBar,1.0);
assert(std::abs(final.next-clean) < 1e-12); // Final noise coefficient is zero.
const double velocity = std::sqrt(alphaBar)*noise-std::sqrt(1-alphaBar)*clean;
const double fromVelocity = std::sqrt(alphaBar)*noisy-std::sqrt(1-alphaBar)*velocity;
assert(std::abs(fromVelocity-clean) < 1e-12);
}
struct Step { clean: f64, next: f64 }
fn ddim_step(noisy: f64, predicted_noise: f64,
alpha_bar: f64, next_alpha_bar: f64) -> Step {
assert!(alpha_bar > 0.0 && alpha_bar < 1.0);
assert!(next_alpha_bar >= alpha_bar && next_alpha_bar <= 1.0);
// eta=0: no fresh random draw enters this reverse update.
// Invert the known corruption using this level's prediction.
let clean = (noisy-(1.0-alpha_bar).sqrt()*predicted_noise)/alpha_bar.sqrt();
// Reuse that prediction as the deterministic direction at the next level.
let next = next_alpha_bar.sqrt()*clean
+ (1.0-next_alpha_bar).sqrt()*predicted_noise;
Step { clean,next }
}
fn main() {
let clean = 2.0; let noise = -0.8;
let alpha_bar: f64 = 0.4; let next_alpha_bar: f64 = 0.7;
let noisy = alpha_bar.sqrt()*clean+(1.0-alpha_bar).sqrt()*noise;
let result = ddim_step(noisy,noise,alpha_bar,next_alpha_bar);
assert!((result.clean-clean).abs() < 1e-12);
let expected = next_alpha_bar.sqrt()*clean+(1.0-next_alpha_bar).sqrt()*noise;
assert!((result.next-expected).abs() < 1e-12);
let final_step = ddim_step(noisy,noise,alpha_bar,1.0);
assert!((final_step.next-clean).abs() < 1e-12); // Final noise coefficient is zero.
let velocity = alpha_bar.sqrt()*noise-(1.0-alpha_bar).sqrt()*clean;
let from_velocity = alpha_bar.sqrt()*noisy-(1.0-alpha_bar).sqrt()*velocity;
assert!((from_velocity-clean).abs() < 1e-12);
}
A full sampler also needs a model call at every level, the same time input the model was trained with, the model's prediction type, and the same data or latent scaling used in training. A stochastic sampler needs its own noise-variance formula and independent random draws; borrowing the variance formula from a different sampler changes the method.
The program only accepts current noise levels with 0 < ᾱ < 1. At ᾱ = 0 the clean estimate would divide by zero, and ᾱ = 1 means the sample is already clean. The widget starts at a small positive ᾱ and treats the final target level as ᾱ = 1. Code that evaluates one of these endpoints needs to define what happens there instead of dividing by zero.
08Separate predictor errors from sampler errors
Test in stages. First run the sampler with exact targets on a known distribution, like this page's mixture. Next test the trained predictor at individual noise levels. Only then judge generated samples. A correct sampler can't make up for a wrong time index, wrong data normalization, or a prediction in the wrong convention.
This page's widgets are split the same way. The posterior widget is exact Bayes arithmetic, the line trainer fits one fixed dataset, and the sampler runs with a known exact predictor. Testing the parts separately catches errors that a good-looking final animation can hide.
A sample can look plausible and still fail to match a specific original, ignore the requested condition, or cover only part of the data. Measure the property you need; the denoising formula guarantees none of these.
09What's next
This is the last adjacent page in the series; the Build a Language Model hub shows where it fits. On the main path, AI Inference and Quantization covers running a trained model: numeric stability, storing numbers as small integers, memory traffic, and sampling from output probabilities.
10Sources
The cited equations define the algorithms. Widget readouts report the displayed toy arrays and settings; they are not hardware benchmarks.
- Jonathan Ho, Ajay Jain, Pieter Abbeel, 2020. Denoising Diffusion Probabilistic Models. Gaussian forward marginals, noise-prediction training, and reverse sampling.
- Alexander Quinn Nichol and Prafulla Dhariwal, 2021. Improved Denoising Diffusion Probabilistic Models. Noise scheduling, including the offset cosine schedule.
- Yang Song and Stefano Ermon, 2019. Generative Modeling by Estimating Gradients of the Data Distribution. Scores of noise-perturbed distributions and iterative generation.
- Jiaming Song, Chenlin Meng, Stefano Ermon, 2021. Denoising Diffusion Implicit Models. Deterministic reverse steps and sampling on selected noise levels.
- Jonathan Ho and Tim Salimans, 2022. Classifier-Free Diffusion Guidance. Combining conditional and unconditional denoising predictions.
- Tim Salimans and Jonathan Ho, 2022. Progressive Distillation for Fast Sampling of Diffusion Models. The velocity parameterization and reconstruction identities.