3D Math for Games

01The conventions are the hard part

The vector and matrix algebra in this tutorial is the same algebra you saw in school. What makes 3D math for games its own skill is that a single rotation can be written several equally correct ways, and a formula copied from a source with different conventions renders upside down, mirrored, or inside out. The algebra doesn't change between engines and APIs; the conventions do, and most of the debugging time goes into them^[1][2].

Four independent choices trip people up, and they are genuinely independent, so you can mix them in any combination:

• Storage order: row-major or column-major. How the numbers sit in memory.
• Vector convention: column vectors (v′ = M·v, matrix on the left) or row vectors (v′ = v·M, matrix on the right).
• Handedness: left- or right-handed world space. Which way the cross product points.
• Clip space: where +Y ends up on screen and whether the depth range is [−1, 1] or [0, 1].

This tutorial picks one of each and holds it throughout: column-major storage, column vectors (v′ = M·v), right-handed world space, and Vulkan clip space. The first three are the defaults of the two libraries the code uses, glm for C++^[3] and glam for Rust^[4]. Vulkan clip space takes a little configuration in each (§5): glm defaults to OpenGL's depth range, and neither library flips Y for you. Stated once; every sample below obeys it.

02Vectors

A 3D vector is three numbers, but the two products you do with it carry most of the geometry in a game: the dot product measures alignment, the cross product builds a perpendicular.

The dot product collapses two vectors to a single number that encodes the angle between them:

Two routes to the same number. Left: multiply the matching components and add, pure arithmetic. Right: multiply the two lengths by the cosine of the angle between them. The right-hand form is why the dot product measures alignment: cos θ is 1 when the arrows point the same way, 0 when they are perpendicular, and negative once they point more than 90° apart. Hover any symbol to see what it stands for.

Its sign tells you which half-space b is in relative to a: positive in front, negative behind, zero perpendicular. That sign test powers CPU-side back-face checks, the N·L term in diffuse lighting, and "is the enemy in front of me" checks (a field-of-view test compares the dot product against the cosine of half the view angle).

The cross product (3D only) returns a vector perpendicular to both inputs, with length equal to the area of the parallelogram they span:

The result is a new vector pointing perpendicular to both a and b, straight out of the plane they share. Each output component criss-crosses the other two axes; that pattern is the whole formula. Its length equals the area of the parallelogram a and b span, and because that area carries sin θ, the cross product is largest when the arrows are perpendicular and shrinks to zero when they are parallel.

Projection of a onto b drops a perpendicular from the tip of a onto the line through b:

How much of a points along b? Drop a straight line from the tip of a onto b's direction; the shadow it casts is the projection. The fraction (a·b)/(b·b) works out to how many copies of b fit into that shadow, and multiplying by b aims the answer back along b. When b is already unit length the bottom is 1 and the whole thing collapses to (a·b) b.

Normalization scales a vector to unit length while keeping its direction:

Divide a vector by its own length and the direction survives but the length becomes exactly 1, a unit vector, written with a hat: â. Directions you compare or feed into lighting should almost always be normalized first. The trap: a zero-length vector has |a| = 0, so this divides by zero and produces a NaN that then spreads through every transform it touches, so guard against it.

And when you actually need the angle between two vectors, acos(a·b) (for unit vectors) is numerically poor near 0 and π; atan2(|a×b|, a·b) is the robust form.

Drag the blue arrow, or use the sliders. The dot product flips sign as b swings past 90°, and the parallelogram (the cross product's magnitude) swells to its largest when the arrows are perpendicular:

struct Vec3 { float x, y, z; };

inline float dot(Vec3 a, Vec3 b) { return a.x*b.x + a.y*b.y + a.z*b.z; }

inline Vec3 cross(Vec3 a, Vec3 b) {
    return { a.y*b.z - a.z*b.y,      // x
             a.z*b.x - a.x*b.z,      // y
             a.x*b.y - a.y*b.x };    // z
}

inline float length(Vec3 v) { return std::sqrt(dot(v, v)); }

inline Vec3 normalize(Vec3 v) {
    float len = length(v);
    return len > 1e-8f ? Vec3{ v.x/len, v.y/len, v.z/len } : v;  // guard the zero vector
}
// In practice: glm::dot, glm::cross, glm::length, glm::normalize.

#[derive(Clone, Copy)]
struct Vec3 { x: f32, y: f32, z: f32 }

fn dot(a: Vec3, b: Vec3) -> f32 { a.x*b.x + a.y*b.y + a.z*b.z }

fn cross(a: Vec3, b: Vec3) -> Vec3 {
    Vec3 { x: a.y*b.z - a.z*b.y,      // x
           y: a.z*b.x - a.x*b.z,      // y
           z: a.x*b.y - a.y*b.x }     // z
}

fn length(v: Vec3) -> f32 { dot(v, v).sqrt() }

fn normalize(v: Vec3) -> Vec3 {
    let len = length(v);
    if len > 1e-8 { Vec3 { x: v.x/len, y: v.y/len, z: v.z/len } } else { v }  // guard zero
}
// In practice: glam's Vec3::dot, ::cross, ::length, ::normalize_or_zero.

03Matrices

A 4×4 matrix packs a 3×3 linear part (rotation, scale, shear) and a translation into one object that the GPU can apply to every vertex with a single multiply. Composing transforms is matrix multiplication, and the order matters: T·R ≠ R·T.

The thing that actually causes bugs is that two independent conventions get conflated. Storage order (row- vs column-major) is how the floats sit in memory. Vector convention (column vs row vectors) is which side the matrix goes on. They are orthogonal: any of the four combinations is valid^[2].

Three facts follow, and each maps to a real bug:

• With column vectors, a model→world→view chain is written right-to-left, in the order the GPU applies it: M_view · M_world · M_model · v. Read it as "model first."
• Translation lives in the 4th column under column vectors, the 4th row under row vectors. Put it in the wrong place and objects fly off to infinity.
• GPUs care about memory layout, not your math convention. GLSL matrices in a uniform block (a std140 block, in Vulkan) default to column-major; upload row-major bytes without transposing, or without declaring the block row_major, and you silently transpose your transform.

One more scoping note: don't say "every engine stores column-major." The OpenGL and glTF lineage does, and so do glm and glam; DirectX's XMMATRIX is row-major with row vectors. Both are correct; they are different lineages^[2].

Poke at it. The widget composes a transform from translation, rotation, and a scale that starts non-uniform (1.6 by 1), and prints the resulting column-major matrix. Flip the application order and the square shears; set both scales equal and the two orders agree again:

// glm: column-major storage, column vectors. Read right-to-left:
// scale first, then rotate, then translate.
glm::mat4 model = glm::mat4(1.0f);              // identity
model = glm::translate(model, position);          // T
model = model * glm::mat4_cast(rotation);        // R (quaternion → matrix)
model = glm::scale(model, scale);                 // S
// Equivalent: model = T * R * S; applied to a point as model * vec4(p, 1).

// glam: column-major storage, column vectors. Same order.
let model = Mat4::from_scale_rotation_translation(
    scale,        // S
    rotation,     // R (a Quat)
    position,     // T
);
// Builds T * R * S directly; applied to a point as model * p.extend(1.0).

Wrong answers, and why: column-major storage does not force row vectors (the conventions are independent); the transpose-on-upload symptom is a memory-layout mismatch, not a quaternion or near/far issue.

04Coordinate spaces & handedness

A vertex passes through a chain of spaces: object space → world space → view (camera) space → clip space → NDC → screen. Each arrow is one matrix, except the last two, which are the perspective divide and the viewport transform the GPU does for you.

Handedness is fixed by the basis orientation. With +X right and +Y up: if +Z points toward you, the system is right-handed; if +Z points into the screen, it's left-handed. Formally, it is the sign of the triple product (X × Y) · Z with the three axes written in a reference right-handed frame.

05Model → view → projection

The three matrices that get a model onto the screen. Model takes object space to world space. View takes world to eye space, and is the inverse of the camera's world transform. Projection takes eye space to clip space, a homogeneous space where the divide hasn't happened yet.

After the vertex shader writes clip-space (x, y, z, w), the GPU does the perspective divide (xyz / w) to get NDC, then the viewport transform to get framebuffer pixels. The projection matrix is where the API's clip-space conventions live, and this is where OpenGL and Vulkan differ in ways that produce upside-down or depth-broken renders.

Sources: the Vulkan specification's vertex post-processing chapter and Wellings' coordinate-system writeup^[5][6].

Two consequences for the projection matrix:

• The [0, 1] depth range changes the matrix's third row, which is why libraries ship distinct variants: glm gates it with GLM_FORCE_DEPTH_ZERO_TO_ONE (and exposes perspectiveRH_ZO vs perspectiveRH_NO)^[3]; glam splits Mat4::perspective_rh (zero-to-one, for Vulkan, WebGPU, Direct3D) from perspective_rh_gl (negative-one-to-one, for OpenGL)^[4].
• The Y-down NDC is handled one of a few ways: negate gl_Position.y in the shader, bake a Y-flip into the projection, or set a negative VkViewport.height, which negates clip-space Y in the viewport transform. Negative viewport height came with VK_KHR_maintenance1 and is core since Vulkan 1.1^[16].

The widget renders the same scene for a chosen target API. Untick the box to feed it a projection written for the other API, or mismatch the import handedness; the readout names each thing that breaks:

// Define this BEFORE including glm so depth maps to [0, 1] for Vulkan:
//   #define GLM_FORCE_DEPTH_ZERO_TO_ONE
glm::mat4 proj = glm::perspective(glm::radians(60.0f), aspect, 0.1f, 1000.0f);
proj[1][1] *= -1.0f;  // flip Y: glm assumes GL's Y-up NDC, Vulkan's is Y-down
// Alternative to the flip: set a negative VkViewport.height.

// glam's perspective_rh already targets [0, 1] depth (Vulkan/wgpu/D3D).
let mut proj = Mat4::perspective_rh(60.0_f32.to_radians(), aspect, 0.1, 1000.0);
proj.y_axis.y *= -1.0;  // flip Y for Vulkan's Y-down NDC (or use a negative viewport height)
// Use perspective_rh_gl instead if you ever target OpenGL's -1..1 depth.

06Quaternions

A quaternion is four numbers that encode an orientation. A unit quaternion q = (w, x, y, z) represents a rotation of angle θ about a unit axis n as:

A quaternion's four numbers are not arbitrary: pack the cosine of half the angle into the first slot, and the axis scaled by the sine of half the angle into the other three. The orientation is stored directly, with no sequence of separate spins to go wrong, which is exactly what dodges gimbal lock (next subsection).

The half-angle is intrinsic, not a quirk: it falls out of the sandwich product used to rotate a vector, v′ = q v q⁻¹ (with v written as a pure quaternion, and q⁻¹ = the conjugate for unit q).

Two properties matter constantly. First, double cover: q and −q are the same rotation. That's harmless when you apply a quaternion and decisive when you interpolate two (next section). Second, the budget: four floats versus nine for a 3×3 matrix, and composition is one quaternion product versus a matrix multiply.

Versus Euler angles, and gimbal lock

Euler angles store an orientation as three sequential rotations (yaw, pitch, roll). They read nicely in an inspector and fail in a specific way:

Gimbal lock is not "caused by quaternions being missing" or "by 90° rotations in general"; it is the middle-axis degeneracy of a three-angle sequence. Quaternions trade that singularity for an ambiguity: q and −q are the same rotation, which matters when interpolating (§7). And a quaternion built from Euler angles inherits their lock at that pose; the benefit comes from composing rotations as quaternions instead of rebuilding them from three angles.

The rig below is three nested gimbals driven by Euler angles. Drive the middle ring to 90° and the outer and inner rings snap parallel; yaw and roll then produce the same motion. Flip to the quaternion track and each slider instead turns the model about its own axis, composed as a quaternion product, so the three sliders move it in three independent directions at any pose:

Versus rotation matrices, and drift

Both represent the same rotations, and the GPU ultimately wants a matrix for the vertex transform, so quaternions are usually converted to a matrix at the end of the chain. Pick by operation: quaternions are cheaper to store, compose, interpolate, and renormalize; matrices are cheaper to apply to many vectors, which is why linear-blend skinning shaders usually consume matrices.

Repeated quaternion multiplication accumulates floating-point error, so a quaternion that should stay unit-length slowly drifts off the unit sphere; left uncorrected, it introduces scale/shear distortion when converted to a matrix. The fix is to renormalize periodically, every few multiplications, not necessarily after every single one. Rotation matrices drift the same way (they lose orthonormality) and need re-orthonormalization rather than mere column-normalization.

07Interpolating rotations

Blending two orientations is the daily job of an animation system. There are three contenders, and the "obvious correct" one is not always the right pick.

SLERP (spherical linear interpolation, Shoemake 1985^[11]) walks the great-circle arc between two unit quaternions at constant angular velocity:

Ω (omega) is the angle between the two orientations, read straight off their dot product (cos Ω = q₀ · q₁). The two sine weights slide from all-q₀ at t = 0 to all-q₁ at t = 1, and because they ride the curve of the sphere rather than a straight chord, the blend turns at a steady rate the whole way across.

LERP interpolates the components in a straight line: a chord through the sphere, not along it. The result is not unit length and its rotation speed is non-uniform. NLERP is LERP followed by a renormalize; it rides the same arc as SLERP but not at constant angular velocity, running faster in the middle and slower at the ends^[12].

The widget animates t from 0 to 1 over real seconds and draws all three paths plus a live angular-velocity strip. The circle is a slice through quaternion space, where a rotation of θ puts q₁ at θ/2 from q₀ (the half angle of eq. 5). Crank the angle to make the methods diverge, and toggle off the shortest-path fix to see the classic bug:

Wrong answers, and why: normalizing the inputs doesn't address which arc you travel (the double cover does); dropping to LERP trades the long-way bug for a non-unit, variable-speed result. And NLERP is the one without constant angular velocity, so that can't be its advantage.

08Building & decomposing transforms

Many engines store a local transform as translation, rotation (a quaternion), and scale, and build the matrix on demand. glTF defines a node's transform as exactly T·R·S with the rotation as a quaternion^[7].

Composing is the easy direction:

With column vectors you read the chain right to left, the order it actually runs: scale the object, then rotate it, then translate it into the world. Swap two of these and the result changes, because rotation and non-uniform scale do not commute (the §3 widget shows the shear).

Decomposing back out is where the edges are: translation is the last column, scale is the length of each of the three basis columns, and the rotation is what's left after dividing the columns by their scales.

09Random numbers

Procedural placement, loot rolls, particle jitter, AI variation: games lean on random numbers, and the C standard library's rand() is the wrong tool for all of them.

rand() is implementation-defined: there's no required algorithm and no quality guarantee. Common implementations are LCGs whose low-order bits are weak, and RAND_MAX can be as small as 32767^[15]. On top of that, rand() % n introduces modulo bias unless n divides RAND_MAX + 1^[15].

Reach for a modern small-state generator. PCG applies a permutation to an LCG's output: small fast state, strong statistical quality, multiple independent streams^[14]. The xoshiro/xoroshiro family is comparably fast with tiny state. Both are plenty for gameplay.

// PCG32 (O'Neill). State is 64 bits; output is a permuted 32-bit word.
struct Pcg32 {
    uint64_t state = 0x853c49e6748fea9bULL;
    uint64_t inc   = 0xda3e39cb94b95bdbULL;   // stream selector, must be odd

    uint32_t next() {
        uint64_t oldState = state;
        state = oldState * 6364136223846793005ULL + inc;   // LCG step
        // Output uses the OLD state, so the permutation overlaps the next step.
        uint32_t xorshifted = (uint32_t)(((oldState >> 18) ^ oldState) >> 27);
        uint32_t rotation = (uint32_t)(oldState >> 59);   // top 5 bits pick the rotate
        // Rotate right by `rotation`; (32 - rotation) & 31 keeps the shift below 32.
        return (xorshifted >> rotation) | (xorshifted << ((32 - rotation) & 31));
    }
};

// PCG32 (O'Neill). Same multiplier and permutation; seed it with the same
// constants as the C++ tab: state 0x853c49e6748fea9b, inc 0xda3e39cb94b95bdb.
struct Pcg32 { state: u64, inc: u64 }   // inc (stream) must be odd

impl Pcg32 {
    fn next_u32(&mut self) -> u32 {
        let old_state = self.state;
        self.state = old_state.wrapping_mul(6364136223846793005).wrapping_add(self.inc);
        // Output uses the OLD state, so the permutation overlaps the next step.
        let xorshifted = (((old_state >> 18) ^ old_state) >> 27) as u32;
        let rotation = (old_state >> 59) as u32;   // top 5 bits pick the rotate
        xorshifted.rotate_right(rotation)   // the permutation, as one intrinsic
    }
}
// In practice: the `rand_pcg` / `rand` crates ship this and the bounded sampling.

10Pitfalls

Most bugs in this tutorial are convention mismatches. Symptom first, then the convention behind it:

11What's next

This is Phase 2 of Build a Game Engine. The next module is IEEE-754 Floating Point, the other half of the numerics phase and the reason a fixed timestep alone doesn't guarantee identical results across machines. After it come The Game Loop & Time, the Platform layer, and your first triangle in Vulkan, where the clip-space conventions from §5 stop being abstract.

1. Eric Lengyel. Foundations of Game Engine Development, Vol. 1: Mathematics. Terathon Software, 2016. The primary textbook reference for the vector, matrix, and quaternion fundamentals here.
2. Tomas Akenine-Möller, Eric Haines, Naty Hoffman, et al. Real-Time Rendering, 4th ed., ch. 4 "Transforms." realtimerendering.com. Supports the row- vs column-vector conventions, transform composition, and quaternions.
3. G-Truc. glm manual. github.com/g-truc/glm. Column-major storage, column vectors, right-handed and [−1,1] depth defaults; GLM_FORCE_DEPTH_ZERO_TO_ONE and the _ZO/_NO projection variants.
4. Cameron Hart. glam documentation. docs.rs/glam. Column-major storage (public x_axis…w_axis columns); perspective_rh ([0,1]) vs perspective_rh_gl ([−1,1]); to_scale_rotation_translation's requirement that the input be a 3D affine transform.
5. Matthew Wellings. "The New Vulkan Coordinate System." matthewwellings.com. Vulkan's Y-down NDC, [0,1] depth, right-handed NDC (OpenGL's is left-handed), and a projection-matrix correction for porting OpenGL code.
6. The Khronos Group. Vulkan Specification, vertex post-processing (clip volume and viewport transform). docs.vulkan.org. The clip-volume bound 0 ≤ z_c ≤ w_c.
7. The Khronos Group. glTF 2.0 Specification. registry.khronos.org. Right-handed, +Y up, meters and radians, node transform as T·R·S with a quaternion rotation.
8. Epic Games. "Coordinate System and Spaces in Unreal Engine." dev.epicgames.com. Left-handed (cross product and axes follow the left-hand rule), +Z up, +X forward, and the handedness warning for assets imported from other tools. With "Units of Measurement in Unreal Engine," dev.epicgames.com: centimeters as the default distance unit.
9. Unity Technologies. "Quaternion and euler rotations in Unity," Unity 6.3 Manual. docs.unity3d.com. Left-handed with +X right, +Y up, +Z forward; rotations stored internally as quaternions and shown as Euler angles, applied Z, then X, then Y.
10. "Gimbal lock." Wikipedia. en.wikipedia.org/wiki/Gimbal_lock. The loss of one DOF when the middle Euler axis reaches ±90°, as a representation artifact.
11. Ken Shoemake. "Animating Rotation with Quaternion Curves." SIGGRAPH 1985, Computer Graphics 19(3), 245–254. dl.acm.org. The origin of SLERP (and SQUAD); constant-angular-velocity interpolation.
12. Arseny Kapoulkine. "Approximating slerp." zeux.io. NLERP follows the SLERP arc but not at constant velocity; small-angle equivalence and the sin Ω → 0 fallback.
13. Jonathan Blow. "Understanding Slerp, Then Not Using It." number-none.com. The commutativity / constant-velocity / minimal-torque tradeoff and the case for NLERP in games.
14. Melissa O'Neill. "PCG: A Family of Simple Fast Space-Efficient Statistically Good Algorithms for Random Number Generation." 2014. pcg-random.org. The permutation-on-LCG construction and its statistical quality.
15. Paul Hsieh. "Misconceptions about rand()." azillionmonkeys.com. rand() is implementation-defined with weak low bits and a possibly small RAND_MAX; % n introduces modulo bias.
16. The Khronos Group. VK_KHR_maintenance1 reference page. docs.vulkan.org. Allows a negative VkViewport::height to invert Y in the clip-to-framebuffer transform; all its functionality is core in Vulkan 1.1.