IEEE-754 Floating Point

01Why floats

A 32-bit float buys you a huge dynamic range, from sub-millimeter to kilometers, in a fixed number of bits, with arithmetic the CPU and GPU run in hardware. That range, not raw speed, is the reason games use them: integer add and multiply are usually just as fast, but integers can't cheaply hold fractions or span that range, and fixed-point trades the range away for uniform precision.

The catch is that a float is an approximation with exactly defined rounding, and a handful of conventions decide how it rounds. Get those conventions wrong and you get the classic float bugs. Get them right and the bugs become predictable.

02The bit layout

A binary32 float is three fields in 32 bits: 1 sign bit, 8 exponent bits, and 23 fraction bits^[1][2].

Three fields rebuild the number. The sign bit flips it positive or negative. The fraction bits sit after an implicit leading 1. to form a significand in the range [1, 2). The stored exponent shifts that significand by a power of two, after subtracting the bias of 127 so one 8-bit field can encode both tiny and huge magnitudes. Hover any symbol to see what it stands for.

The exponent is stored with a bias of 127, so a stored 1..254 means an actual −126..+127. The leading 1. is implicit for normal numbers and not stored, which is why 23 stored fraction bits give 24 bits of significand precision. (The standard's word is significand; "mantissa" is the older name for the same thing.) Stored exponent 0 is reserved for zero and subnormals; stored exponent 255 for infinity and NaN. binary64 (a double) uses the same scheme with 11 exponent bits (bias 1023) and 52 fraction bits.

Click the bits. The editor below shows the exact value of any binary32, color-codes the three fields, and flags the subnormal and infinity/NaN regions. Try the presets, then zero out the exponent field; the number goes subnormal and the implicit leading bit flips off:

// C++20: std::bit_cast is the UB-free way to reinterpret. (Casting through
// a uint32_t* is type-punning and technically undefined behavior.)
#include <bit>
uint32_t bits = std::bit_cast<uint32_t>(value);
float back = std::bit_cast<float>(bits);

uint32_t sign     = bits >> 31;
uint32_t exponent = (bits >> 23) & 0xFF;       // 8 bits
uint32_t fraction = bits & 0x7FFFFF;            // 23 bits

// to_bits / from_bits are the safe, stable way (const since 1.83).
let bits: u32 = value.to_bits();
let back: f32 = f32::from_bits(bits);

let sign     = bits >> 31;
let exponent = (bits >> 23) & 0xFF;            // 8 bits
let fraction = bits & 0x7F_FFFF;               // 23 bits

03Normals & subnormals

Normal numbers have that implicit leading 1 and a stored exponent of 1..254. When a result gets too small to normalize (below 2⁻¹²⁶ for binary32), the format doesn't jump straight to zero. It switches to subnormals (also called denormals): stored exponent 0, implicit leading bit 0, filling the gap to zero with constant absolute spacing. This is gradual underflow^[1].

Subnormals keep precision degrading gracefully toward zero instead of falling off a cliff. They have a performance story too, on some hardware a sharp one, which gets its own section at the end.

04Signed zero, infinity, NaN

The reserved exponents encode the special values, and each has a sharp edge.

• Signed zero. +0.0 and −0.0 compare equal under == but have different bits and behave differently: 1.0 / +0.0 is +∞, 1.0 / −0.0 is −∞. Functions like copysign and atan2 tell them apart.
• Infinity. From overflow or division by zero. Arithmetic on it mostly stays infinite; ∞ − ∞ is NaN.
• NaN. From 0.0/0.0, ∞ − ∞, sqrt(−1). NaN is not equal to anything, including itself, which is exactly why x != x is the portable NaN test (use std::isnan / f32::is_nan).

05Rounding, and why 0.1 isn't exact

When a result can't be represented exactly, it rounds. IEEE-754's default is round-to-nearest, ties-to-even: on an exact halfway case, pick the result whose last significand bit is 0^[2].

This is why 0.1 isn't exact. A fraction is representable in binary only if it's n / 2k. One tenth is 1/(2×5), and that factor of 5 makes it a non-terminating, repeating binary fraction, the same way 1/3 repeats in decimal. The nearest binary32 to 0.1 is slightly larger than one tenth^[6], so 0.1 + 0.2 != 0.3: three separate roundings, then a fourth on the addition. Integers, by contrast, are exact up to 2²⁴ in binary32 (2⁵³ in a double), so not everything is approximate, just any value that isn't a sum of powers of two within the precision budget.

06Machine epsilon and ULP

Floats are not evenly spaced. A ULP (unit in the last place) is the gap between one float and the next, and it doubles at every power of two. Near 1.0 the gap is 2⁻²³; just below 16,777,216 (2²⁴) the gap is 1.0, and above it, 2.0^[1].

The stepper walks one ULP at a time. The gap grows with magnitude; past 2²⁴ it reaches 2.0 and stepping skips the odd integers:

Wrong answers, and why: epsilon() is the gap at 1.0, not the smallest float and not the rounding bound; and far-from-origin jitter is a magnitude problem that doubles fix or rounding don't actually solve.

07Catastrophic cancellation

Catastrophic cancellation happens when you subtract two nearly-equal numbers that already carry rounding error: the equal leading digits cancel, promoting the low-order error to the front. The subtraction itself is exact; it exposes error that was already there^[1].

Goldberg's example: computing b² − 4ac with b=3.34, a=1.22, c=2.28 in 3-digit arithmetic. The true answer is 0.0292, but b² rounds to 11.2 and 4ac to 11.1, so the result is 0.1, an error of about 70 ULP^[1]. The multiplications introduced the error; the subtraction merely revealed it. The fix is a numerically stable reformulation (for the quadratic formula, compute the larger-magnitude root first, then use the product-of-roots identity), not "throw more precision at it."

08Summing a million numbers

Add up a long list of floats naively and the running total grows large while each new term is small, so each addition rounds away part of the term. The error accumulates. Kahan summation keeps a compensation term that recovers what was lost^[8].

float sum = 0.0f, c = 0.0f;        // c carries the lost low-order bits
for (float x : values) {
    float y = x - c;               // add back last round's lost part
    float t = sum + y;             // big + small: low bits of y fall off here
    c = (t - sum) - y;             // recover exactly what fell off
    sum = t;
}

let (mut sum, mut c) = (0.0f32, 0.0f32);   // c carries the lost low-order bits
for &x in &values {
    let y = x - c;                // add back last round's lost part
    let t = sum + y;              // big + small: low bits of y fall off here
    c = (t - sum) - y;            // recover exactly what fell off
    sum = t;
}

Naive summation error grows with the count of terms; Kahan's stays roughly constant, effectively independent of n until n approaches 1/epsilon^[8]. The race below adds a large value and then a long stream of tiny ones. Watch the naive total stall while Kahan keeps climbing:

09FMA, order, and -ffast-math

Three related facts about how arithmetic is evaluated, each a source of "different answer on a different build."

Fused multiply-add computes a*b + c with a single rounding of the full-precision product-plus-sum, instead of rounding the product and then the sum^[9]. It's more accurate for that expression and, on hardware with an FMA unit, often as fast as a bare multiply. C++ spells it std::fma; Rust spells it f32::mul_add.

float r = std::fma(a, b, c);   // a*b + c, rounded once. Fast only if FP_FAST_FMA.

let r = a.mul_add(b, c);      // (a*b) + c, rounded once.

The catch: a compiler may contract a*b + c into a single FMA on its own, which changes the result versus the two-rounding form. "More accurate" is not "identical," and whether contraction happens depends on the compiler and flags, so the same source gives different numbers on different toolchains^[9][10].

Float addition is not associative. (a+b)+c can differ from a+(b+c) because each step rounds^[1]. So summation order, SIMD lane-reduction order, and multi-threaded partial-sum order all change the result. It isn't randomness; a fixed order is perfectly deterministic, the answer just changes when the order does.

10Comparing floats

a == b on floats is exact bit equality, and after any rounding or reordering that's usually not what you want. There are three tolerance strategies, and each fails somewhere^[7].

• Absolute (|a−b| ≤ ε): works near zero, fails at large magnitudes, where a fixed ε is smaller than one ULP and the test degrades into ==.
• Relative (|a−b| ≤ ε·max(|a|,|b|)): works across magnitudes, breaks down near zero, and is meaningless across zero (opposite signs).
• ULP-based: reinterpret the bits as integers and subtract; the magnitude is the count of representable floats between them. Intuitive, but undefined across signs and gigantic when comparing a tiny number to 0.0.

Dawson's practical answer: combine an absolute "safety net" for the near-zero case with a relative or ULP comparison otherwise, and pick the absolute tolerance from the scale of your problem. His summary is worth memorizing: "zero is a huge nuisance, avoid it if possible"^[7].

Drag the magnitude. A fixed absolute tolerance works near zero but falls apart at large magnitudes, where ε drops below one ULP and the test silently becomes ==; relative and ULP stay scale-invariant:

11Determinism across machines

Identical source can produce different float results on different machines. This is the thread the Game Loop tutorial kept pointing at, and it's why a lockstep RTS or a fighting-game replay can desync on a teammate's PC.

The causes are specific, and listing them is the point, because "floats are random" is wrong^[11]:

• Different compilers and optimization levels reassociate and contract differently.
• FMA contraction on one platform but not another.
• x87 80-bit intermediates vs SSE 32/64-bit on x86.
• Transcendental functions. IEEE-754 requires + − × ÷ sqrt fma and remainder to be correctly rounded, but it does not require it of sin, cos, exp, so vendor math libraries legitimately differ^[11].
• FTZ/DAZ mode differences (next section).

12The subnormal cliff

On some hardware, arithmetic that produces or consumes subnormals drops into a microcode-assisted slow path that is dramatically slower than the same operation on normal numbers. Bruce Dawson measured an SSE multiply with a subnormal hitting roughly a 175× slowdown on a Core 2, and about a 140-cycle penalty on a Sandy Bridge mulps^[12]. The size is hardware-dependent (often negligible on recent AMD, and on addition), so quote it with its CPU rather than as a universal number.

The fix is FTZ (flush-to-zero, for outputs) and DAZ (denormals-are-zero, for inputs), two bits in the x86 MXCSR control register that make the CPU treat subnormals as zero, trading correctness near zero for speed. It is a per-thread CPU mode, so set it on every audio or physics worker thread, not just the main one^[13].

#include <xmmintrin.h>   // FTZ (flush-to-zero, outputs)
#include <pmmintrin.h>   // DAZ (denormals-are-zero, inputs; SSE3)
_MM_SET_FLUSH_ZERO_MODE(_MM_FLUSH_ZERO_ON);        // MXCSR bit 15
_MM_SET_DENORMALS_ZERO_MODE(_MM_DENORMALS_ZERO_ON); // MXCSR bit 6; call per thread

// No std API; use the x86 intrinsics (unsafe) on each worker thread.
use std::arch::x86_64::{_mm_getcsr, _mm_setcsr};
unsafe { _mm_setcsr(_mm_getcsr() | 0x8040); }   // set FTZ (bit 15) and DAZ (bit 6)

13Pitfalls

14What's next

This closes the determinism thread from the Game Loop and pairs with 3D Math as the numerics foundation. The bit tricks here are the same muscle as the Bit Shifting tutorial (the famous fast inverse square root is exactly this bit-level reinterpretation^[14]). Next on the build path is Platform & Window, then the Vulkan renderer. The full path is on the series hub.

1. David Goldberg. "What Every Computer Scientist Should Know About Floating-Point Arithmetic." ACM Computing Surveys, 1991. docs.oracle.com. Bit layout, ULP and machine epsilon, gradual underflow, catastrophic vs benign cancellation (the 70-ULP quadratic example), non-associativity.
2. IEEE. "IEEE Standard for Floating-Point Arithmetic," IEEE 754-2019. ieeexplore.ieee.org. binary32/binary64 definitions, round-to-nearest-ties-to-even as the default, the correctly-rounded operation set.
3. cppreference. std::numeric_limits<T>::epsilon / min / denorm_min. en.cppreference.com. epsilon = gap from 1.0 to the next float = FLT_EPSILON; min() = smallest normal; denorm_min() = smallest subnormal.
4. Rust. Primitive type f32. doc.rust-lang.org. to_bits/from_bits, mul_add (single rounding), next_up/next_down (stable 1.86), EPSILON = 1.19209290e-07, MIN_POSITIVE, NaN sign non-portability.
5. cppreference. std::bit_cast. en.cppreference.com. C++20, the UB-free reinterpretation of a float's bits (vs type-punning through a pointer).
6. Exploring Binary. "Why 0.1 Does Not Exist In Floating-Point." exploringbinary.com. The exact nearest value to 0.1 and the repeating-binary reason it isn't representable.
7. Bruce Dawson. "Comparing Floating Point Numbers, 2012 Edition." randomascii.wordpress.com. Absolute vs relative vs ULP comparison, where each fails, the combined approach, and "zero is a huge nuisance."
8. "Kahan summation algorithm." Wikipedia. en.wikipedia.org. The compensated-summation algorithm and its error bound versus naive summation (and the Neumaier and pairwise variants).
9. cppreference. std::fma. en.cppreference.com. Single-rounding semantics of a*b+c, FP_FAST_FMA, and the relationship to compiler contraction (FP_CONTRACT).
10. Simon Byrne. "Beware of fast-math." simonbyrne.github.io. The sub-flags -ffast-math enables, including reassociation breaking Kahan, finite-math killing isnan, and crtfastmath.o setting FTZ/DAZ process-wide at link time.
11. Glenn Fiedler. "Floating Point Determinism." gafferongames.com. The causes of cross-platform float divergence and the Battlezone II lockstep case (forced single precision, wrapped transcendentals).
12. Bruce Dawson. "That's Not Normal: the Performance of Odd Floats." randomascii.wordpress.com. Measured subnormal slowdowns with the specific CPUs, and the FTZ/DAZ control-register fix.
13. Intel. "Set the FTZ and DAZ Flags," oneAPI C++ Compiler Developer Guide. intel.com. The _MM_SET_FLUSH_ZERO_MODE / _MM_SET_DENORMALS_ZERO_MODE intrinsics and the MXCSR FTZ (bit 15) / DAZ (bit 6) bits.
14. "Fast inverse square root." Wikipedia. en.wikipedia.org. The Quake III 0x5f3759df bit-level hack, its ~0.17% error after one Newton step, and the modern hardware rsqrt instructions that supersede it.