AI Inference and Quantization from Scratch
A trained model only pays off once it runs fast and cheaply enough on real hardware. This page covers the arithmetic that decides that: computing softmax without overflow, storing weights as 8-bit or 4-bit integers and measuring what that costs, multiplying those integers correctly, working out whether a layer is held back by memory or by arithmetic, and what temperature and top-p do when the model picks its next token.
Compute softmax for any scores without overflow. Turn an array of floats into 8-bit codes with a scale and a zero point, and say how much error rounding and clipping add. Multiply quantized vectors with integer arithmetic and get the right answer. Estimate from a layer's size and a chip's bandwidth whether the layer is waiting on memory or on arithmetic, and why a batch of one usually waits on memory. Say exactly what temperature and top-p do to the next-token probabilities.
01Keep a reference, and keep softmax finite
Inference means running a trained model on new inputs: a chatbot reading your prompt and picking its next word, say. Making that cheaper (smaller number formats, faster kernels, a different sampling rule) changes the answers a little, and each change touches a different part of the system. So before you optimize anything, save the unmodified model's outputs on some real inputs. Every change on this page gets measured against that reference.
The first thing that can go wrong isn't an optimization at all. A language model's last layer produces one raw score per token in its vocabulary, called a logit. Softmax turns logits into probabilities: raise e to each score, then divide each result by the total so they add up to 1. Raise e to a large score, though, and the result is too big for a floating-point number. It becomes infinity, and infinity divided by infinity is NaN.
Worked example
Logits (2, 1, 0). e² = 7.39, e¹ = 2.72, e⁰ = 1, total 11.11. Probabilities: 0.665, 0.245, 0.090.
Add 1000 to every logit: (1002, 1001, 1000). The largest 64-bit float is about 1.8 × 10³⁰⁸, which is e709.8, so e1002 overflows and every probability becomes NaN.
Subtract the largest logit first: (0, −1, −2). e⁰ = 1, e⁻¹ = 0.368, e⁻² = 0.135, total 1.503. Probabilities: 0.665, 0.245, 0.090. Same answer, no overflow.
That shift is the standard fix. It works because adding the same number to every logit multiplies the top and bottom of each fraction by the same factor, which cancels.[1]
In words: subtract the biggest logit from every logit, raise e to each result, and divide each by their total. The biggest shifted logit is 0, so no exponential is larger than 1 and nothing overflows.
The formula needs at least one finite logit. If every token is masked out with a score of −∞, there's nothing to normalize, so check for that case explicitly. Very negative shifted logits can underflow to exactly 0; that's a different effect from NaN, and it just means those tokens get probability 0.
What the shift can't fix
The shifted formula stops the exponentials overflowing, but it can't recover information that was already gone. If two logits were so large that rounding made them equal before softmax ever ran, subtracting the maximum won't separate them again. Blanchard, Higham and Higham analyze the rounding error of the shifted softmax and the closely related log-sum-exp in detail.[1]
When you port a model or change its number format, compare the intermediate arrays as well as the final answer: each layer's output and the final probabilities. Two versions can pick the same top token while their probabilities differ a lot. And when two tokens are nearly tied, a tiny rounding difference can flip which one has the highest score, the argmax, without anything being badly wrong. Check both the numbers and the task result.
02Store each number as a small integer
A 7-billion-parameter model stored in 16-bit floats needs 14 GB for its weights alone, more than many consumer GPUs have. Store each weight as an 8-bit integer and it's 7 GB; at 4 bits, about 3.5 GB. Replacing real numbers with a small set of integer codes like this is quantization. This section shows how the codes work and what they cost in accuracy.
The scheme here, , picks a range of real values to cover, splits it into equal steps, and stores the number of the step each value is nearest to. Two extra numbers turn a code back into a real value: the scale, which is the size of one step, and the zero point, which is the code that stands for real 0. A code on its own means nothing: the same codes with a different scale are different numbers.[2]
Worked example
Cover −3 to 3 with 8-bit codes 0 to 255. Scale = 6 / 255 = 0.0235. Zero point = round(3 / 0.0235) = round(127.5) = 128.
Store 1.1: 1.1 / 0.0235 = 46.75, which rounds to 47; add the zero point to get code 175.
Read it back: 0.0235 × (175 − 128) = 1.106. The error is 0.006, under half a step (0.012).
Now narrow the range to −1 to 1 and store 2.7. It lands past the top, so it's clamped to code 255 and reads back as 0.996. The error is 1.7, and all of it comes from clipping.
Written out, storing a value and reading it back are:
reconstructed = scale × (integer − zero_point)
In words: to store, divide by the step size, round to a whole number, shift by the zero point, and clamp into the code range. To read back, undo the shift and multiply by the step size. Only the rounding and the clamp lose information.
With b bits there are 2b codes, so a common choice sets scale = (high − low) / (2b − 1). The range is widened if needed so it includes 0, and the zero point is rounded to an integer, so real 0 gets an exact code; Jacob et al. require this because efficient kernels often pad arrays with zeros.[2] Rounding the zero point can shift the ends of the range slightly.
That leaves two kinds of error. Rounding error, for values inside the range, is at most about half a step. Clipping error, for values outside it, can be as big as the value itself. Picking the range trades one for the other: a wide range never clips but spreads the codes farther apart, while a narrow one gives fine steps and chops off the big values. The widget lets you move that trade-off on eight fixed values. Its mean squared error describes only this array.
Exact halves: which way to round
A value that lands exactly halfway between two codes has to go one way or the other, and implementations disagree. This page's code and widgets round halves away from zero, before adding the zero point. Other formats and runtimes round halves to the nearest even integer. The word "round" doesn't tell you which, so check the format's specification when you need two implementations to match bit for bit.
A simpler variant, symmetric quantization, makes the range run from −max to +max so the zero point is just 0. There's one less number to track and the multiply gets simpler. But if the values are lopsided, say all between −0.1 and 3, almost half the codes cover values that never occur. Which scheme works best depends on the tensors and on what the kernels support as much as on the bit count.
03One big value can spoil everyone's range
A layer's weight matrix has one row per output, and each row is called an output channel. Suppose one row holds a weight of 12 while another row's weights are all around 0.1. A single scale for the whole matrix (per-tensor) has to stretch to 12, and the small row's weights all round to within a step or two of zero. Giving each row its own scale (per-channel) fixes that, at the cost of storing one scale per row and needing a kernel that applies them.[3]
Worked example
Symmetric 8-bit codes run from −127 to 127. The matrix's largest weight is 12, so a shared scale is 12 / 127 = 0.0945.
The small row's 0.07 becomes round(0.07 / 0.0945) = round(0.74) = 1, which reads back as 0.0945. Its −0.03 becomes code 0 and reads back as 0.
With its own scale, 0.1 / 127 = 0.000787, the same 0.07 becomes round(88.9) = 89 and reads back as 0.07008. The error drops from 0.025 to 0.00008.
This compares how well one small matrix survives the round trip. It shows per-channel scales losing less information here, not that a particular model keeps its accuracy.
Large language models have a harder version of the problem in their activations, the values passed from one layer to the next. Dettmers et al. found that in large transformers a few feature dimensions of the activations carry values far bigger than the rest. These are called outliers. Their method, LLM.int8(), quantizes most of each matrix multiply to 8 bits with a separate scale for every row of the activations and every column of the weights, and sends the outlier dimensions through a separate 16-bit multiply.[4]
SmoothQuant takes another route. It divides each activation input channel by a factor and multiplies the matching input channel of the weights by the same factor. The product doesn't change, but the activations' range gets smaller and easier to quantize, and some of the difficulty moves into the weights.[5]
Worked example
One activation channel reaches 60 while the others stay under 2. Its weights are about 0.02.
Pick a factor of 8 for that channel. The activation becomes 60 / 8 = 7.5 and the weight 0.02 × 8 = 0.16. Their product is unchanged: 60 × 0.02 = 7.5 × 0.16 = 1.2. The activation's largest value fell from 60 to 7.5, so a shared activation scale can be 8 times finer.
As a matrix identity, with s holding one positive factor per input channel:
In words: divide each input channel of the activations X by its factor and multiply the same input channel of the weights W by that factor. The exact product stays the same; only how hard each side is to quantize changes. The factors must line up with the dimension the multiply sums over.
Keep the two problems apart. The widget's per-channel scales were one per output row of the weights; activation outliers sit in input features of the activations. Those are different axes of different tensors. When someone says per-channel quantization fixed an error, ask which tensor and which axis.
Weights are fixed, so their ranges are known before deployment. Activations depend on the input. Static quantization fixes the activation ranges ahead of time by running representative inputs through the model and recording what values appear, a step called . Dynamic quantization measures each activation's range while the model runs instead, and ONNX Runtime supports both.[3] Dynamic ranges cost extra work on every call; fixed ranges can clip inputs that look unlike the calibration set. Test whichever you pick on held-out inputs that resemble real traffic.
04Subtract the zero points before you multiply
The point of integer codes is to run a layer's dot products (multiply matching entries, add them up) with cheap integer arithmetic instead of floating point. That works, but only if you subtract each vector's zero point from its codes before multiplying. Multiply the raw codes and the zero-point offsets get counted as if they were signal.[2]
Worked example
Both vectors use scale 0.1 and zero point 10. Input (0.5, −0.2) stores as codes (15, 8); weights (0.3, 0.4) store as (13, 14).
The real dot product: 0.5 × 0.3 + (−0.2) × 0.4 = 0.15 − 0.08 = 0.07.
Centered codes: (15 − 10)(13 − 10) + (8 − 10)(14 − 10) = 15 − 8 = 7. Times 0.1 × 0.1 gives 0.07.
Raw codes: 15 × 13 + 8 × 14 = 307. Times 0.01 gives 3.07, off by a factor of 44.
real_output ≈ input_scale × weight_scale × accumulator
In words: subtract the zero points, multiply matching codes, add the products in a wide integer, and convert to a real number once at the end by multiplying by both scales. The integer arithmetic is exact; the ≈ is the error from storing the inputs as codes in the first place.
This reference stops at a real number. A fully integer pipeline instead converts the accumulator straight into the next layer's integer codes, adds the bias in matching units, and clamps.
The running total, the accumulator, can get large, so its width matters. With 8-bit codes and any zero point, a centered code is at most 255 in size, so each product is at most 255² = 65,025 and a length-K dot product is at most K × 65,025 before bias. A signed 32-bit integer, int32, holds about 2.1 billion, so this worst case fits for K up to 33,025. A 4096-long dot product (at most 266 million) fits easily, but int32 isn't safe for any length without that check.
A deployed integer-only network also has to handle the bias, stored in the same units as the product of the two scales, and the output, which needs its own scale and zero point. The framework and the hardware kernel must agree on where each conversion happens. Jacob et al. lay out the whole integer pipeline.[2]
A weight-only quantized model works differently. Its kernel can unpack the 4- or 8-bit weights back to floats on the fly and do floating-point math, so the activations never leave floating point. That's a different arithmetic path from an all-int8 matrix multiply. A model file's bit count doesn't tell you which path runs, how much unpacking it needs, or whether the kernel got any faster.
05Is the layer waiting on memory or on math?
When a model generates text for one user, each new token runs every layer on a single input vector. Each layer reads its whole weight matrix from memory but does only one multiply and one add per weight. Whether that's slow because of arithmetic or because of memory decides what will speed it up. The answers the question with two numbers about the hardware: its peak arithmetic rate and its memory bandwidth.[6]
What decides between them is arithmetic intensity, the number of operations done per byte moved. At low intensity, memory can't feed the arithmetic units fast enough. At high intensity, arithmetic is the limit. Say which memory you're counting bytes from: an estimate for on-chip cache and one for device memory are different models.
Worked example
A made-up chip: 10 TFLOP/s peak, 200 GB/s memory bandwidth. A 4096 × 4096 layer with 2-byte weights, one input row.
Work: 2 × 4096² ≈ 33.6 million operations. Traffic: 33.6 MB of weights, plus 16 KB of input and output. Intensity ≈ 1 operation per byte.
Memory time: 33.6 MB / 200 GB/s = 0.168 ms. Arithmetic time: 33.6 million / 10 trillion = 0.0034 ms. The layer is memory-bound by about 50×.
To keep the arithmetic busy, this chip needs 10 T / 200 G = 50 operations per byte. A batch of 64 rows reaches 62, and the layer becomes compute-bound.
time ≥ max(operation_count / peak, byte_count / bandwidth)
In words: a layer can't go faster than the arithmetic units allow or faster than memory can deliver its bytes, so its time is at least the larger of the two. Real kernels land above this bound: launch overhead, dependencies between steps, conversion work and imperfect use of the hardware all add time.
Take a weight matrix with M rows and K columns applied to B input rows. The work is 2BMK operations (each multiply and each add counts as one). The weights are read once for the whole batch, so a bigger batch does more work per weight byte. The widget's model also counts reading the inputs and writing the outputs once each, and ignores every other tensor.
So batching helps: serve 32 users at once and each weight read does 32 users' work. The catch is waiting, because a request may sit in a queue while a batch forms. Throughput across all users, time to the first token, and each request's latency are separate measurements, and this bound covers only the execution of one layer. Queueing time comes on top.
A transformer generating text also keeps a cache. For every earlier token, each layer stores that token's attention keys and values, vectors it computed once and reuses for every later token instead of recomputing. This key-value (KV) cache takes 2 × layers × batch × KV heads × tokens × head width × bytes per entry, as the transformer page derives.[7]
Worked example
32 layers, 8 KV heads of width 128, 2-byte entries, one sequence: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token. A 4096-token conversation holds 512 MiB of cache.
Quantizing the weights doesn't shrink that cache. It grows with every token, and with full attention each new token reads all of it.
06Temperature and top-p reshape the next-token odds
A chatbot's model gives a probability for every possible next token, and something still has to pick one. Always taking the most likely token, called greedy decoding, tends to produce repetitive text; sampling from the full distribution sometimes picks nonsense from the long tail of unlikely tokens.[8] Two settings sit between those extremes.
Temperature T divides every logit before softmax. T below 1 widens the gaps between logits, so the top token gets more of the probability. T above 1 shrinks them and flattens the distribution. T = 0 would divide by zero, so it's handled as a separate rule: take the top token.
Top-p, or , sorts tokens by probability and keeps the smallest set from the top whose probabilities add up to at least p. It rescales those to sum to 1 and samples from them. The token that pushes the total past p stays in. Unlike top-k, which always keeps k tokens, the number kept follows the distribution: a confident model keeps few, an unsure one keeps many.[8]
Worked example
Logits (3, 2, 1, 0, −1, −2). At T = 1, softmax gives 0.634, 0.233, 0.086, 0.032, 0.012, 0.004.
At T = 0.5 the logits double to (6, 4, 2, 0, −2, −4), and the top token's probability rises to 0.865.
Top-p with p = 0.9 at T = 1: running totals 0.634, then 0.867 (still under 0.9), then 0.953, which crosses it. Keep three tokens. Divide each by 0.953: 0.665, 0.245, 0.090.
Neither setting fixes a wrong model. If the logits favor a wrong answer, a lower temperature makes the model pick it more often. Cutting the tail changes how varied the output is and how often a long-shot token appears; it doesn't make the text more factual. Holtzman et al. studied how decoding rules change generated text, and their results don't name one best setting for every application.
Write down the order of the steps. Cutting the tail before applying temperature can keep a different set of tokens than applying temperature first. Break ties in a fixed way if you need reproducible output, and check that at least one token survives. Keep the model's probabilities and the sampler's final probabilities as separate arrays.
07A checked integer dot product
The two programs below, in C++ and Rust with the same logic, store four values as unsigned 8-bit codes, compute the dot product in integers, and convert to a real number at the end. They use the widget's vectors, so the float answer is 1.56. The assertions check that the integer result equals the dot product of the read-back vectors, that an exact 0 stays exact, and that the result lands within 0.04 of 1.56. They also check that huge inputs clip to codes 0 and 255 and that exact halves round away from zero. Four terms keep the accumulator at or below 4 × 65,025 = 260,100. The codes sit in 32-bit integers for readability rather than packed bytes.
#include <algorithm>
#include <array>
#include <cassert>
#include <cmath>
#include <cstdint>
struct Quantized { std::array<std::int32_t,4> codes; double scale; std::int32_t zero; };
Quantized quantize(const std::array<double,4>& values, double low, double high) {
assert(std::isfinite(low) && std::isfinite(high));
assert(low <= 0 && high >= 0 && high > low);
const double scale = (high-low)/255.0;
assert(std::isfinite(scale) && scale > 0);
// Include zero exactly; its code is rounded with the same tie convention.
const std::int32_t zero = static_cast<std::int32_t>(std::round(-low/scale));
Quantized result{{},scale,zero};
for (int index = 0; index < 4; ++index) {
assert(std::isfinite(values[index]));
// Clip while still floating point so huge finite inputs cannot overflow an integer cast.
const double code = std::clamp(std::round(values[index]/scale)+zero,0.0,255.0);
result.codes[index] = static_cast<std::int32_t>(code);
}
return result;
}
double dot(const Quantized& left, const Quantized& right) {
std::int32_t accumulator = 0;
for (int index = 0; index < 4; ++index)
// Codes are already widened; centering removes offsets before multiplication.
accumulator += (left.codes[index]-left.zero)*(right.codes[index]-right.zero);
return left.scale*right.scale*accumulator; // Return a real value for this reference.
}
int main() {
const auto left = quantize({-1,0.3,2,0.1},-2,3);
const auto right = quantize({0.5,-0.2,1,1.2},-1,2);
double reconstructed = 0;
for (int index = 0; index < 4; ++index)
reconstructed += left.scale*(left.codes[index]-left.zero)
* right.scale*(right.codes[index]-right.zero);
assert(std::abs(dot(left,right)-reconstructed) < 1e-12);
const auto zeros = quantize({0,0,0,0},-2,3);
assert(dot(zeros,right) == 0); // Exact real zero survives the affine representation.
assert(std::abs(dot(left,right)-1.56) < 0.04); // Float dot is 1.56 for this input.
const auto extremes = quantize({1e100,-1e100,0,1},-2,3);
assert(extremes.codes[0] == 255 && extremes.codes[1] == 0); // Clipping precedes conversion.
const auto ties = quantize({0.5,-0.5,0,0},-127.5,127.5);
assert(ties.codes[0]-ties.zero == 1 && ties.codes[1]-ties.zero == -1);
}
struct Quantized { codes: [i32;4], scale: f64, zero: i32 }
fn quantize(values: [f64;4], low: f64, high: f64) -> Quantized {
assert!(low.is_finite() && high.is_finite());
assert!(low <= 0.0 && high >= 0.0 && high > low);
let scale = (high-low)/255.0;
assert!(scale.is_finite() && scale > 0.0);
// Include zero exactly; its code is rounded with the same tie convention.
let zero = (-low/scale).round() as i32;
let mut result = Quantized { codes: [0;4],scale,zero };
for (index,value) in values.iter().enumerate() {
assert!(value.is_finite());
// Clip while still floating point so huge finite inputs cannot overflow integer addition.
let code = ((value/scale).round()+f64::from(zero)).clamp(0.0,255.0);
result.codes[index] = code as i32;
}
result
}
fn dot(left: &Quantized, right: &Quantized) -> f64 {
let mut accumulator: i32 = 0;
for index in 0..4 {
// Codes are already widened; centering removes offsets before multiplication.
accumulator += (left.codes[index]-left.zero)*(right.codes[index]-right.zero);
}
left.scale*right.scale*f64::from(accumulator) // Real-valued reference result.
}
fn main() {
let left = quantize([-1.0,0.3,2.0,0.1],-2.0,3.0);
let right = quantize([0.5,-0.2,1.0,1.2],-1.0,2.0);
let mut reconstructed = 0.0;
for index in 0..4 {
reconstructed += left.scale*f64::from(left.codes[index]-left.zero)
* right.scale*f64::from(right.codes[index]-right.zero);
}
assert!((dot(&left,&right)-reconstructed).abs() < 1e-12);
let zeros = quantize([0.0;4],-2.0,3.0);
assert_eq!(dot(&zeros,&right),0.0); // Exact real zero survives affine coding.
assert!((dot(&left,&right)-1.56).abs() < 0.04); // Float dot is 1.56 for this input.
let extremes = quantize([1e100,-1e100,0.0,1.0],-2.0,3.0);
assert!(extremes.codes[0] == 255 && extremes.codes[1] == 0); // Clipping precedes conversion.
let ties = quantize([0.5,-0.5,0.0,0.0],-127.5,127.5);
assert!(ties.codes[0]-ties.zero == 1 && ties.codes[1]-ties.zero == -1);
}
This is a reference calculation, written to be easy to check. It doesn't pack 4-bit codes two to a byte, use vector instructions, fuse the bias and activation into the multiply, or convert the result back into codes for the next layer. A deployment needs each of those to match its model graph and its hardware backend. The browser widgets likewise show representations and results; none of them is an int8 speed test.
Keep representative inputs and the unquantized model's outputs. Then compare in order: how well each tensor survives the round trip, how far each layer's outputs move, how the real task scores on held-out data, and finally how fast the actual deployment path runs at realistic batch sizes and sequence lengths. Each step answers a different question, and passing one doesn't imply passing the next.
08Measure what actually ran
A smaller model file doesn't guarantee a faster run. A runtime can fall back to a higher-precision kernel for any operator it has no low-bit kernel for. Check which backend and kernels were selected before you benchmark. If a range or an axis is wrong, compare intermediate values layer by layer before you start tuning the sampler.
When you report a benchmark, list the hardware, runtime version, batch size, prompt and output lengths, warmup, and whether data transfers and tokenization are included. Keep memory occupied (bytes sitting allocated) separate from memory traffic (bytes read per token): a weight matrix occupies its memory once but is read again for every token.
The widgets compute exact errors for the arrays they show. A model needs its own accuracy and robustness evaluation, including inputs that push past the calibration ranges. How much error is acceptable depends on the task.
09What's next
Serving Language Models takes the quantized model to many users at once: prefill against decode on the roofline, the memory the key-value cache takes, continuous batching, paged caches with prefix sharing, chunked prefill and speculative decoding.
10Sources
The cited equations define the algorithms. Widget readouts report the displayed toy arrays and settings; they are not hardware benchmarks.
- Pierre Blanchard, Desmond J. Higham, Nicholas J. Higham, 2019 arXiv preprint. Accurate Computation of the Log-Sum-Exp and Softmax Functions. Shifted exponential formulas and floating-point behavior.
- Benoit Jacob et al., 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. Affine quantization, zero points, integer accumulators, and rescaling.
- ONNX Runtime contributors, quantization documentation. Quantize ONNX models. Calibration, static and dynamic quantization, and error diagnosis.
- Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. Outlier-sensitive quantization and mixed-precision decomposition.
- Guangxuan Xiao et al., 2023. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. Equivalent channel rescaling between activations and weights.
- Samuel Williams, Andrew Waterman, David Patterson, 2008 technical report, published 2009. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures. Arithmetic intensity and compute versus bandwidth bounds.
- Hugging Face Transformers contributors, cache documentation. How caching works. Cache payload dimensions and reuse during generation.
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi, 2020. The Curious Case of Neural Text Degeneration. Temperature, probability-tail truncation, and nucleus sampling.