IEEE-754 Floating Point

01Why floats

A game world needs a bullet's position to the millimeter and a planet's orbit in the millions of kilometers, often in the same frame. A 32-bit float covers that dynamic range in a fixed number of bits, with arithmetic the CPU and GPU run in hardware. That range, not raw speed, is why games use them: integer add and multiply are usually just as fast, but integers can't cheaply hold fractions or span that range, and fixed-point trades the range away for uniform precision.

The catch is that a float is an approximation with exactly defined rounding, and a handful of conventions decide how it rounds. Get those conventions wrong and you get the classic float bugs. Get them right and the bugs become predictable.

02The bit layout

A debugger shows a position as 0x40D00000, or a network packet carries four bytes that should be a speed. To know what number those bits hold, and why some patterns mean infinity or NaN, you need the layout. A binary32 float (IEEE-754's 32-bit format, C++'s float and Rust's f32) is three fields in 32 bits: 1 sign bit, 8 exponent bits, and 23 fraction bits^[1][2].

The sign says positive or negative. The fraction holds the digits. The exponent says where the binary point goes, the same job the power of ten does in 6.5 × 10³. This formula turns the three fields back into the number, and it's what the hardware does every time it reads a float:

In words: write a 1, a binary point, then the 23 fraction bits, which gives a number from 1 up to (not including) 2. Multiply it by 2 raised to the stored exponent minus 127, and flip the sign if the sign bit is set. The bias of 127 is what lets one unsigned 8-bit field encode both tiny and huge magnitudes. Hover any symbol to see what it stands for.

The exponent is stored with a bias of 127, a fixed offset you subtract on the way out, so a stored 1..254 means an actual −126..+127. The leading 1. is implicit for normal numbers and not stored, which is why 23 stored fraction bits give 24 bits of precision. The 1 plus the fraction is the significand, the standard's word for what older texts call the "mantissa". Stored exponent 0 is reserved for zero and subnormals; stored exponent 255 for infinity and NaN (Not a Number, what 0/0 produces). binary64 (a double) uses the same scheme with 11 exponent bits (bias 1023) and 52 fraction bits.

The editor below shows the exact value of any binary32 and color-codes the three fields:

In code, C++20's std::bit_cast reads a float's bits without the undefined behavior of casting through a pointer^[5], and Rust's to_bits / from_bits do the same^[4]. Masks and shifts then pull the three fields apart:

// C++20: std::bit_cast is the UB-free way to reinterpret. (Reading a float
// through a uint32_t* breaks strict aliasing: undefined behavior.)
#include <bit>
uint32_t bits = std::bit_cast<uint32_t>(value);
float back = std::bit_cast<float>(bits);

uint32_t sign     = bits >> 31;
uint32_t exponent = (bits >> 23) & 0xFF;       // 8 bits
uint32_t fraction = bits & 0x7FFFFF;            // 23 bits

// to_bits / from_bits are the safe, stable way (const since 1.83).
let bits: u32 = value.to_bits();
let back: f32 = f32::from_bits(bits);

let sign     = bits >> 31;
let exponent = (bits >> 23) & 0xFF;            // 8 bits
let fraction = bits & 0x7F_FFFF;               // 23 bits

03Normals & subnormals

A reverb tail fades, a damped spring settles, a velocity is multiplied by 0.98 every frame. Each one shrinks a value toward zero, and sooner or later it gets smaller than the smallest number the normal encoding can hold. What happens next matters for both correctness and speed.

Normal numbers have that implicit leading 1 and a stored exponent of 1..254. When a result gets too small to normalize (below 2⁻¹²⁶ for binary32), the format doesn't jump straight to zero. It switches to subnormals (also called denormals): stored exponent 0 and an implicit leading bit of 0, which fill the gap down to zero with evenly spaced values. Precision drains away a bit at a time instead of all at once, which is called gradual underflow^[1].

Subnormals keep precision degrading gracefully toward zero instead of falling off a cliff. They have a performance story too, on some hardware a sharp one, which gets its own section at the end.

04Signed zero, infinity, NaN

A character's position turns into NaN, and on the next frame its skinned mesh disappears from the screen: every vertex multiplied by a NaN matrix is NaN too. The reserved exponents encode these special values, and each has a sharp edge.

• Signed zero. +0.0 and −0.0 compare equal under == but have different bits and behave differently: 1.0 / +0.0 is +∞, 1.0 / −0.0 is −∞. Functions like copysign and atan2 tell them apart: atan2(+0.0, −1.0) is +π but atan2(−0.0, −1.0) is −π, so a heading computed from a velocity whose y is −0 reads −180° instead of +180°.
• Infinity. From overflow or division by zero. Arithmetic on it mostly stays infinite; ∞ − ∞ is NaN.
• NaN. From 0.0/0.0, ∞ − ∞, sqrt(−1). NaN is not equal to anything, including itself, which is exactly why x != x is the portable NaN test (use std::isnan / f32::is_nan).

05Rounding, and why 0.1 isn't exact

A spawn timer adds 0.1f every tick and fires when timer == 1.0f, but that check is never true. The reason is rounding: most results of float arithmetic land between two representable floats and have to snap to one of them.

IEEE-754's default rule is round-to-nearest, ties-to-even: snap to the nearest representable value, and on an exact halfway case pick the one whose last significand bit is 0^[2].

Rounding is also why 0.1 isn't exact. A fraction is representable in binary only if it's n / 2k. One tenth is 1/(2×5), and that factor of 5 makes it a non-terminating, repeating binary fraction, the same way 1/3 repeats in decimal. The nearest binary32 to 0.1 is slightly larger than one tenth^[6].

In double precision the famous 0.1 + 0.2 != 0.3 follows: three separate roundings, then a fourth on the addition, land on 0.30000000000000004. In binary32 the same errors happen to cancel and 0.1f + 0.2f == 0.3f holds, which is luck, not exactness. Integers, by contrast, are exact up to 2²⁴ in binary32 (2⁵³ in a double), so not everything is approximate, just any value that isn't a sum of powers of two within the precision budget.

06Machine epsilon and ULP

An open-world game streams in a town 20 km from the world origin, and the player's gun starts to shimmer by a couple of millimeters every frame. Nothing changed in the code. What changed is the size of the numbers, because floats are not evenly spaced.

A ULP (unit in the last place) is the gap between one float and the next, and it doubles at every power of two, because the 23 fraction bits always split the range from one power of two to the next into the same 2²³ steps. Near 1.0 the gap is 2⁻²³; just below 16,777,216 (2²⁴) the gap is 1.0, and above it, 2.0^[1].

The stepper walks one ULP at a time:

Wrong answers, and why: epsilon() is the gap at 1.0, not the smallest float and not the rounding bound; and far-from-origin jitter is a magnitude problem that doubles fix or rounding don't actually solve.

07Catastrophic cancellation

Two crates sit 10 cm apart in that town 20 km out, and the physics code subtracts their positions to get the gap between them. The answer comes back 0.4 mm short, a 0.4% error.

That's catastrophic cancellation: subtract two nearly-equal numbers that already carry rounding error and the matching leading digits cancel, leaving the old error as most of what's left. The subtraction itself is exact; it exposes error that was already there^[1].

08Summing a million numbers

A profiler adds up a million frame timings, or a stats screen totals the distance a player walked over a long session. Add a long list of floats naively and the running total grows large while each new term stays small, so each addition rounds away part of the term, and the error accumulates.

Kahan summation fixes this by keeping a second float, the compensation, that holds exactly what the last addition rounded away, and feeding it back into the next term^[8].

float sum = 0.0f, c = 0.0f;        // c carries the lost low-order bits
for (float x : values) {
    float y = x - c;               // add back last round's lost part
    float t = sum + y;             // big + small: low bits of y fall off here
    c = (t - sum) - y;             // recover exactly what fell off
    sum = t;
}

let (mut sum, mut c) = (0.0f32, 0.0f32);   // c carries the lost low-order bits
for &x in &values {
    let y = x - c;                // add back last round's lost part
    let t = sum + y;              // big + small: low bits of y fall off here
    c = (t - sum) - y;            // recover exactly what fell off
    sum = t;
}

Naive summation's error bound grows with the number of terms; Kahan's stays roughly constant, effectively independent of the count until it approaches 1/epsilon (about 8 million terms for a float)^[8]. The race below runs the worked example 60,000 times:

09FMA, order, and -ffast-math

A replay recorded on one build plays back differently on another build of the same source, on the same PC. Three facts about how float arithmetic gets evaluated explain most cases like this: fused multiply-add, evaluation order, and -ffast-math.

Fused multiply-add (FMA) computes a*b + c with a single rounding of the full-precision product-plus-sum, instead of rounding the product and then the sum^[9]. It's more accurate for that expression and, on hardware with an FMA unit, often as fast as a bare multiply. C++ spells it std::fma; Rust spells it f32::mul_add.

float r = std::fma(a, b, c);   // a*b + c, rounded once. Fast only if FP_FAST_FMAF is defined
                                  // (hardware FMA); otherwise a slow software routine.

let r = a.mul_add(b, c);      // (a*b) + c, rounded once.

The catch: a compiler may contract a plain a*b + c into a single FMA on its own, which changes the result versus the two-rounding form. "More accurate" is not "identical," and whether contraction happens depends on the compiler and flags, so the same source gives different numbers on different toolchains^[9][10]. With the numbers above, the line return a*b + c; compiled by GCC 13 at -O2 -mfma becomes one vfmadd instruction and returns 0.000488340854644775390625; add -ffp-contract=off and it returns 0.00048828125.

Float addition is not associative. (a+b)+c can differ from a+(b+c) because each step rounds^[1]. So summation order, SIMD lane-reduction order (SIMD runs one instruction on several values at once, then combines the lanes at the end), and multi-threaded partial-sum order all change the result. None of this is random: a fixed order is deterministic, and the answer changes only when the order does.

10Comparing floats

A unit test checks that a transformed point lands where it should, and fails with "expected 1048576, got 1048576.25". They're only two floats apart. a == b on floats is exact equality, and after any rounding or reordering that's usually not what you want. There are three tolerance strategies, and each fails somewhere^[7].

• Absolute (|a−b| ≤ ε, with ε a fixed small number): works near zero, fails at large magnitudes, where a fixed ε is smaller than one ULP and the test degrades into ==.
• Relative (|a−b| ≤ ε·max(|a|,|b|), so the tolerance scales with the values): works across magnitudes, breaks down near zero, and is meaningless across zero (opposite signs).
• ULP-based: reinterpret the bits as integers and subtract; the result is the count of representable floats between them. Intuitive, but undefined across signs and gigantic when comparing a tiny number to 0.0.

Dawson's practical answer: combine an absolute "safety net" for the near-zero case with a relative or ULP comparison otherwise, and pick the absolute tolerance from the scale of your problem. His summary is worth memorizing: "zero is a huge nuisance, avoid it if possible"^[7].

The widget runs all three tests on two floats a few ULP apart, at every magnitude from 2⁻²⁴ to 2²⁴:

11Determinism across machines

In a lockstep game, every machine runs the same simulation from the same player inputs and sends only the inputs over the network. A real-time strategy (RTS) game uses it because thousands of units would be too much state to send. It only works if every machine computes bit-identical results, and identical source can produce different float results on different machines. That is why a lockstep RTS or a fighting-game replay can desync on a teammate's PC, and why the Game Loop tutorial, next in the series, needs more than a fixed timestep to make a simulation deterministic.

The causes are specific; floats are not random^[11]:

• Different compilers and optimization levels reassociate and contract differently.
• FMA contraction on one platform but not another.
• x87 80-bit intermediates vs SSE 32/64-bit on x86. x87 is the original x86 floating-point unit, which keeps intermediates at 80 bits by default; SSE rounds every result to exactly float or double.
• Transcendental functions. IEEE-754 requires + − × ÷ sqrt fma and remainder to be correctly rounded, but it does not require it of sin, cos, exp, so vendor math libraries legitimately differ^[11].
• FTZ/DAZ mode differences (next section).

12The subnormal cliff

The music stops, the reverb tail fades out, and the audio thread, which was fine during the loud part, suddenly misses its deadline and the sound crackles. On some CPUs a quiet signal costs more than a loud one, because as it decays it fills with subnormals.

On some hardware, arithmetic that produces or consumes subnormals drops into a microcode-assisted slow path that is dramatically slower than the same operation on normal numbers. Bruce Dawson measured a worst case of roughly a 175× slowdown for SSE addition and multiplication with subnormals on a Core 2, and about a 140-cycle penalty on a Sandy Bridge mulps, where he could produce no slowdown at all on addps^[12]. The size depends on the CPU and the operation, so quote it with its CPU rather than as a universal number.

The fix is FTZ (flush-to-zero, for outputs) and DAZ (denormals-are-zero, for inputs), two bits in the x86 MXCSR control register (the per-thread register that holds SSE's rounding mode and exception settings) that make the CPU treat subnormals as zero, trading correctness near zero for speed. It is a per-thread CPU mode, so set it on every audio or physics worker thread, not just the main one^[13]. Rust has no sound equivalent: its compiler assumes the default MXCSR, flipping these bits is undefined behavior, and the _mm_setcsr intrinsic has been deprecated since Rust 1.75 for that reason^[15]. Rust code keeps subnormals out of its feedback paths instead.

#include <xmmintrin.h>   // FTZ (flush-to-zero, outputs)
#include <pmmintrin.h>   // DAZ (denormals-are-zero, inputs; SSE3)
_MM_SET_FLUSH_ZERO_MODE(_MM_FLUSH_ZERO_ON);        // MXCSR bit 15
_MM_SET_DENORMALS_ZERO_MODE(_MM_DENORMALS_ZERO_ON); // MXCSR bit 6; call per thread

// Rust assumes the default MXCSR, so setting FTZ/DAZ is UB (and
// _mm_setcsr is deprecated). Flush tiny values out of feedback state
// yourself instead, e.g. in a recursive filter or reverb tail.
const DENORMAL_FLOOR: f32 = 1.0e-15;       // about -300 dB: inaudible

fn flush_tiny(sample: f32) -> f32 {
    if sample.abs() < DENORMAL_FLOOR { 0.0 } else { sample }
}

// state = flush_tiny(input + feedback * state);

13Pitfalls

14What's next

Next is The Game Loop & Time. Its determinism section picks up from §11: a fixed timestep is necessary for a replayable simulation, and the float behavior on this page is why it isn't sufficient. After it come Platform & Window and Input. This page pairs with 3D Math as the numerics foundation, and the bit tricks here are the same muscle as Bit Shifting: the famous fast inverse square root, Quake III's trick that reads a float's bits as an integer to approximate 1/√x, is exactly this bit-level reinterpretation^[14]. The full path is on the series hub.

1. David Goldberg. "What Every Computer Scientist Should Know About Floating-Point Arithmetic." ACM Computing Surveys, 1991. docs.oracle.com. Bit layout, ULP and machine epsilon, gradual underflow, catastrophic vs benign cancellation (the 70-ULP quadratic example), non-associativity.
2. IEEE. "IEEE Standard for Floating-Point Arithmetic," IEEE 754-2019. ieeexplore.ieee.org. binary32/binary64 definitions, round-to-nearest-ties-to-even as the default, the correctly-rounded operation set.
3. cppreference. std::numeric_limits<T>::epsilon / min / denorm_min. en.cppreference.com. epsilon = gap from 1.0 to the next float = FLT_EPSILON; min() = smallest normal; denorm_min() = smallest subnormal.
4. Rust. Primitive type f32. doc.rust-lang.org. to_bits/from_bits, mul_add (single rounding), next_up/next_down (stable 1.86), EPSILON = 1.19209290e-07, MIN_POSITIVE, NaN sign non-portability.
5. cppreference. std::bit_cast. en.cppreference.com. C++20, the UB-free reinterpretation of a float's bits (vs type-punning through a pointer).
6. Exploring Binary. "Why 0.1 Does Not Exist In Floating-Point." exploringbinary.com. The exact nearest value to 0.1 and the repeating-binary reason it isn't representable.
7. Bruce Dawson. "Comparing Floating Point Numbers, 2012 Edition." 2012. randomascii.wordpress.com. Absolute vs relative vs ULP comparison, where each fails, the combined approach, and "zero is a huge nuisance."
8. "Kahan summation algorithm." Wikipedia. en.wikipedia.org. The compensated-summation algorithm and its error bound versus naive summation (and the Neumaier and pairwise variants).
9. cppreference. std::fma. en.cppreference.com. Single-rounding semantics of a*b+c, FP_FAST_FMA, and the relationship to compiler contraction (FP_CONTRACT).
10. Simon Byrne. "Beware of fast-math." simonbyrne.github.io. The sub-flags -ffast-math enables, including reassociation breaking Kahan, finite-math killing isnan, and GCC enabling FTZ/DAZ process-wide, even from shared libraries until GCC 13.
11. Glenn Fiedler. "Floating Point Determinism." Gaffer On Games, 2010. gafferongames.com. The causes of cross-platform float divergence, Intel's note that the IEEE operations match across platforms under the same control settings while x87 transcendentals may not, and the Battlezone II lockstep case (transcendentals wrapped to stay at single precision).
12. Bruce Dawson. "That's Not Normal: the Performance of Odd Floats." 2012. randomascii.wordpress.com. Measured subnormal slowdowns with the specific CPUs, and the FTZ/DAZ control-register fix.
13. Intel. "Set the FTZ and DAZ Flags," oneAPI C++ Compiler Developer Guide. intel.com. The _MM_SET_FLUSH_ZERO_MODE / _MM_SET_DENORMALS_ZERO_MODE intrinsics and the MXCSR FTZ (bit 15) / DAZ (bit 6) bits.
14. "Fast inverse square root." Wikipedia. en.wikipedia.org. The Quake III 0x5f3759df bit-level hack, its ~0.17% error after one Newton step, and the modern hardware rsqrt instructions that supersede it.
15. Rust. core::arch::x86_64::_mm_setcsr. doc.rust-lang.org. Deprecated since 1.75: Rust assumes MXCSR's masking, rounding and denormals-are-zero bits stay at their defaults, so changing them is immediate undefined behavior.