Build a Game Engine · Audio

The Audio Engine

Your game's sound clicks every few seconds, drops out when a level loads, and distorts when ten things explode at once. Rendering is soft real-time: a late frame makes the game stutter. Audio is hard real-time: a buffer that isn't ready when the device needs it is an audible click or dropout. That constraint, a callback that must finish before the device runs dry, shapes the subsystem: no locks or allocation in the callback, a lock-free ring to hand it work, and a mixer sized to fit its deadline. This page builds that subsystem in C++ and Rust, and fixes the clicks, the dropouts and the distortion along the way.

Time~55 min LevelSenior PrereqsHelpful, not required: the Lock-free Queues tutorial (the ring is reused here) and The Game Loop (audio is a second clock). StackC++ (miniaudio) · Rust (cpal)

What you'll be able to do by the end

Write an audio callback that never allocates, locks or blocks, and spot the C++ calls that allocate in disguise.
Pick a buffer size by trading latency against the risk of underruns.
Send play commands from the game thread through a lock-free ring without freeing memory on the audio thread.
Mix voices with headroom and a limiter instead of letting them clip.
Resample, pan with equal power, and attenuate by distance the way game audio APIs do.

01The real-time callback

You don't push sound to the speakers when it suits you; the hardware asks for it, on its own clock. The audio layer (the OS API, or a library such as miniaudio or cpal on top of it) runs a dedicated high-priority thread that calls a function you registered, handing it a buffer to fill before the hardware needs it. It's a pull model: the system pulls samples from you, on its schedule, on a thread you don't control. cpal describes it as "a dedicated, high-priority thread responsible for delivering audio data to the system's audio device in a timely manner"^[10]. A 128-frame buffer at 48 kHz gives you about 2.67 ms to fill it.

Worked example

At 48 kHz with 256-frame buffers, the callback runs 48,000 ÷ 256 = 187.5 times a second, and each call has 256 ÷ 48,000 = 5.33 ms before the device needs the next buffer. Mixing 32 stereo voices into it is 256 × 32 × 2 = 16,384 multiply-adds, a tiny fraction of a millisecond on a modern core. The danger isn't the work you planned. It's one call that stalls: a lock the game thread holds for 8 ms makes this callback miss its 5.33 ms deadline, and the listener hears a click.

What the callback must never do

Inside the callback: no heap allocation (malloc/free/new/delete), no mutex a non-real-time thread can hold, no file/network/syscall I/O, and no unbounded work. They're banned for their unbounded worst case, not their average speed: the allocator may take a contended lock or fault a page in from disk; a mutex can cause priority inversion^[3]. Bencina's rule is to judge callback code by its worst-case execution time, never its average (amortized) cost^[1]. try_lock on a std::mutex is not a loophole: if another thread is waiting, the unlock that follows has to wake it through the OS scheduler, which is a system call^[2]. For a stream of commands the fix is a lock-free queue. (Where a lock is unavoidable, Doumler's option is a spinlock the audio thread only ever try-locks, so its side costs one atomic operation while the other thread spins with backoff^[2].)

In C++ the traps are easy to miss: destroying the last std::shared_ptr to an object runs its delete, std::function may heap-allocate, vector::push_back can reallocate. Any of those is an allocation in disguise. The mixing callback below is allocation- and lock-free:

The mixing callback (no alloc, no locks)

// miniaudio 0.11.x. Runs on the high-priority audio thread.
// RULE: no malloc/free, no mutex the game thread holds, no I/O, no unbounded loops.
void dataCallback(ma_device* device, void* pOutput, const void*, ma_uint32 frameCount) {
    Mixer* mixer = (Mixer*)device->pUserData;     // pre-allocated, owned elsewhere
    float* output = (float*)pOutput;                 // interleaved stereo: L,R,L,R,...

    drainPlayCommands(mixer);                       // wait-free pops from the SPSC ring (§4)
    memset(output, 0, sizeof(float) * frameCount * 2);  // start from silence (bounded)

    for (Voice& voice : mixer->voices) {           // fixed-capacity array, no allocation
        if (!voice.active) continue;
        for (ma_uint32 frame = 0; frame < frameCount; ++frame) {
            float sample = voice.data[voice.position];  // mono source
            output[frame*2 + 0] += sample * voice.gainLeft;   // mixing == summation
            output[frame*2 + 1] += sample * voice.gainRight;
            if (++voice.position >= voice.length) { voice.active = false; break; }
        }
    }
    // the sum may exceed [-1,1]; clamp/limit on the master bus, not here (§5)
}

// cpal 0.18.x. The closure runs on cpal's high-priority audio thread; the stream is stereo f32.
let stream = device.build_output_stream(
    config,                                             // StreamConfig, taken by value since cpal 0.18
    move |output: &mut [f32], _: &cpal::OutputCallbackInfo| {
        drain_play_commands(&mut mixer);                 // wait-free pops from the SPSC ring (§4)
        for s in output.iter_mut() { *s = 0.0; }          // silence

        for voice in mixer.voices.iter_mut() {            // fixed-capacity, no allocation
            if !voice.active { continue; }
            for frame in output.chunks_mut(2) {            // 2 samples == one stereo frame
                let sample = voice.data[voice.position];   // mono source
                frame[0] += sample * voice.gain_left;        // summation
                frame[1] += sample * voice.gain_right;
                voice.position += 1;
                if voice.position >= voice.length { voice.active = false; break; }
            }
        }
        // sum may exceed [-1,1]; limit on the master bus, not here (§5)
    },
    move |err| eprintln!("audio error: {err}"),
    None,                                              // timeout (cpal 0.15+ param)
)?;

What's intentionally missing

No device-format negotiation/conversion (the engine converts at the edges); no master-bus limiter (§5); no stealing when all slots are full; no retire ring to free finished buffers off the audio thread (freeing here would break the allocation rule); no sample-accurate command scheduling (commands apply at block boundaries, not exact sample offsets); no declick ramp on voice start/stop (instant gain changes click).

02PCM fundamentals

Before you can mix anything you need to know what the numbers in those buffers mean. (pulse-code modulation) represents a waveform as evenly spaced amplitude samples^[4]. Four terms you must keep straight:

Sample rate (Hz) sets the bandwidth: the highest frequency it can hold is half the rate, the Nyquist limit. 48 kHz is the common game/OS rate; 44.1 kHz is CD's.
Bit depth sets dynamic range, the span from the quietest to the loudest representable level, for integer formats (16-bit ≈ 96 ). float32 samples are zero-centered, nominally in [−1, 1].
Channels are concurrent streams (L/R, 5.1).
A frame is one sample per channel at one instant (a stereo frame is two samples).

Worked example

One second of stereo float32 at 48 kHz is 48,000 frames × 2 channels × 4 bytes = 384,000 bytes; the same second as 16-bit integers is 192,000 bytes. 48 kHz can represent frequencies up to 24 kHz, above the roughly 20 kHz limit of human hearing. 16 bits give 65,536 levels, and 20 × log₁₀(65,536) = 96.3 dB of range between the smallest step and full scale, which is where "16-bit ≈ 96 dB" comes from.

Two collisions to avoid

An audio frame is one sample per channel, not a video frame. The two meet in the game loop, so say which one you mean. And float32 in [−1, 1] is the internal processing format (what miniaudio and cpal's f32 hand you); device and file formats are frequently 16- or 24-bit integer PCM, and the engine converts at the edges. What float buys is headroom during mixing: the sum can exceed ±1 without losing data until the final clamp. (It adds no resolution at the speaker; the device format sets that.) Layout is interleaved (L,R,L,R) for most device APIs and WAV, planar (LLL…RRR) for Web Audio.

03Buffer & latency

A player presses jump and the sound comes a beat late; turn the buffer down to fix it and the audio starts crackling. The callback buffer size sets two things at once: the latency floor (you can't react faster than one buffer) and how tight the deadline is. 256 frames at 48 kHz is about 5.3 ms; 64 frames about 1.3 ms. Smaller buffers mean lower latency but a shorter window to finish in, so a smaller hiccup (a USB stall, a background flush, an over-budget mix) is enough to cause an underrun (a starved output, heard as a click or dropout).

Worked example

A jump sound triggered just after a callback starts has to wait for the next one, up to a full buffer: 5.33 ms at 256 frames, 21.3 ms at 1024. On top of that, the OS mixer and the device add buffering of their own. Halving the buffer halves that wait but also halves the time each callback has to finish, so a 3 ms stall that was harmless at 256 frames (5.33 ms deadline) causes an underrun at 128 (2.67 ms).

The widget races the device draining one buffer against the callback computing the next:

Drag Buffer and tick "callback waits on a lock", then watch the underrun count and the strip of recent callbacks. The left bar is the buffer the device is playing; it drains in one buffer period, which is also the callback's deadline and the latency floor. The right bar is the callback computing the next buffer. Each callback's duration is modeled as a mixing cost proportional to the buffer size, plus random scheduling jitter, plus a rare 1 to 3 ms OS hiccup; the checkbox adds a 2 to 10 ms lock wait. When a callback outlasts its deadline the playing buffer runs dry (CLICK) and its bar in the strip turns red. At the starting 512 frames (a 10.67 ms deadline) the lock wait makes about one callback in five miss; at 1024 frames it never causes an underrun; at 256 about three in four miss; and at 64 to 128 frames even an OS hiccup is enough. Time is slowed 130×; the costs are illustrative, not measured.

04The ring to the audio thread

The player fires a gun on the game thread, and the gunshot has to start on the audio thread within a buffer or two. The game thread can't call into the audio thread directly and can't share a mutex with it, so the hand-off is a single-producer, single-consumer lock-free ring: the game thread enqueues "play this sound" commands, the callback dequeues them at the top of each block. The C++ tab uses the SpscRing from the Lock-free Queues tutorial as-is; the Rust tab uses the rtrb crate, because that tutorial's Rust ring requires T: Copy and this command carries an Arc. PortAudio's ring buffer states it "only works when there is a single reader and a single writer" with a power-of-two capacity^[5].

Don't free on the audio thread

The consumer side must be wait-free: bounded steps, never blocks, never allocates. The subtle trap is lifetime: if you pass an owning pointer (or an Arc/shared_ptr) and the audio thread drops the last reference, that free runs on the audio thread and breaks the allocation rule. Either the game thread owns the buffer's lifetime (the audio thread only borrows) or you hand finished voices back on a second ring for the game thread to free. Either ring can fill, so pick the policy up front (drop and count, or size the ring for the worst burst). On that retire ring the audio thread is the producer, and it must never wait for space.

Worked example

The C++ PlayCommand below is an 8-byte pointer, a 4-byte length and two 4-byte gains: 20 bytes, padded to 24 for the pointer's alignment. A 256-slot ring is 6,144 bytes. At 187.5 callbacks a second and a 60 Hz game loop, the callback drains the ring about three times per game frame, so a 40-sound explosion in one frame uses 40 of the 256 slots and is gone within one buffer.

Push a play-command (game thread) and drain it (audio thread)

// Trivially copyable, as the Lock-free Queues SpscRing requires. The game thread
// owns the sample memory and keeps it alive while any voice can still read it.
struct PlayCommand { const float* data; uint32_t length; float gainLeft, gainRight; };
constexpr size_t kCommandCapacity = 256;                  // power of two, per SpscRing
using CommandRing = SpscRing<PlayCommand, kCommandCapacity>;

// Game thread (NOT real-time): push fails if the ring is full.
void requestPlay(CommandRing& ring, const PlayCommand& cmd) {
    if (!ring.push(cmd)) { /* drop or count an overflow; never block audio */ }
}

// Audio thread: wait-free pops, capped at the ring's capacity so a game thread
// pushing at the same time can't keep the loop running. Never frees.
void drainPlayCommands(Mixer* mixer) {
    PlayCommand cmd;
    for (size_t drained = 0; drained < kCommandCapacity && mixer->pendingCommands.pop(cmd); ++drained) {
        Voice* voice = findFreeVoice(mixer);       // scan the fixed voice array
        if (!voice) continue;                          // all slots busy: drop it (no stealing here)
        voice->data = cmd.data; voice->length = cmd.length;
        voice->gainLeft = cmd.gainLeft; voice->gainRight = cmd.gainRight;
        voice->position = 0; voice->active = true;
    }
}

// rtrb 0.4 SPSC ring (Producer/Consumer halves). The game thread clones the Arc
// into each command and keeps its own copy of every loaded sound, so the audio
// thread never drops the LAST reference (that drop would free on the audio thread).
use rtrb::{Consumer, Producer};
struct PlayCommand { data: Arc<[f32]>, gain_left: f32, gain_right: f32 }
const COMMAND_CAPACITY: usize = 256;                     // rtrb::RingBuffer::new(COMMAND_CAPACITY)

// Game thread: push fails when full. Never block the audio thread.
fn request_play(producer: &mut Producer<PlayCommand>, cmd: PlayCommand) {
    let _ = producer.push(cmd);                       // drop on full; do not spin
}

// Audio thread: wait-free pops, capped at the ring's capacity so a game thread
// pushing at the same time can't keep the loop running.
fn drain_play_commands(mixer: &mut Mixer) {
    for _ in 0..COMMAND_CAPACITY {
        let Ok(cmd) = mixer.consumer.pop() else { break };   // Err(Empty): drained
        let Some(voice) = mixer.find_free_voice() else { continue };  // all busy: drop it
        voice.length = cmd.data.len();
        voice.data = cmd.data;          // drops the slot's old Arc: a refcount decrement, never the last one
        voice.gain_left = cmd.gain_left; voice.gain_right = cmd.gain_right;
        voice.position = 0; voice.active = true;
    }
}

05Mixing

Ten grenades go off at once and the speakers crackle. Mixing N voices is summation: add the (gain-scaled) samples, per channel, per frame. Because independent signals add, the sum routinely leaves [−1, 1]. In float that's non-destructive until output, but the final clamp to the device range will hard-clip anything past ±1, squaring off the peaks into harmonic distortion.

Headroom and a limiter, not divide-by-N, not hard-clip

The fix is headroom (mix with peaks below the ceiling^[6]) plus a limiter on the master bus: a compressor with a very high ratio and a fast attack, which turns the gain down smoothly as peaks approach the ceiling, so the waveform keeps its shape instead of being squared off^[13]. Two wrong "fixes": dividing the sum by N attenuates everything (one quiet voice shouldn't get quieter because another exists); and relying on hard-clipping distorts. The limiter has costs too: pushed hard, its gain changes become audible as pumping and it flattens transients^[13]. Float not overflowing mid-sum does not mean you can ignore levels: the clamp at the device boundary still clips.

Worked example

Three voices that each peak at 0.5 can line up to 1.5, which is 20 × log₁₀ 1.5 = 3.5 dB over full scale, so the device clamp flattens those peaks. Divide-by-N doesn't fix it properly: with 4 voices playing, a footstep that peaks at 0.8 on its own comes out at 0.2, 12 dB quieter, just because three other sounds exist. Mixing each voice with about 6 dB of headroom (peaks around 0.5) and letting a limiter catch the rare pile-up keeps the footstep at its level.

The widget sums tone voices into a master output:

The left column shows each voice, a tone; the right trace is the master, their sum, between the ±1 rails. Drag Voices and Gain, then tick the soft clip. At 100% gain each voice peaks at 0.4, so two voices top out at 0.8 and three can reach 1.2. Past ±1 the master peaks flatten against the rails (red), the clipping distortion. Turn on the soft clip and the peaks round back under the ceiling. (The demo uses a tanh soft-clipper as a simple stand-in; a true limiter instead reduces gain to keep the waveform's shape below the ceiling rather than waveshaping the whole signal.) Either way, peak control costs some dynamic range.

06Resampling

Your footstep sounds were recorded at 44.1 kHz and the device runs at 48 kHz. The designer also wants each step pitched slightly differently. Playing a sound at a different pitch, or matching a 44.1 kHz asset to a 48 kHz device, means computing sample values at fractional positions: resampling. The quality ladder runs nearest-neighbor → linear → windowed-sinc, a filter that blends many neighboring samples with carefully chosen weights.

Worked example

Going from 44.1 kHz to 48 kHz, each output sample advances 44,100 ÷ 48,000 = 0.91875 source samples. Output sample 10 sits at source position 9.1875, so linear interpolation returns 0.8125 × source[9] + 0.1875 × source[10]. Pitching a sound up an octave advances 2 source samples per output sample, which also makes it play in half the time.

Linear interpolation aliases

Naive drop/duplicate and even linear interpolation introduce aliasing. Linear interpolation's frequency response is sinc²(fT), a weak low-pass whose first sidelobe suppresses spectral images by only about 26 dB and which isn't flat in the passband, so high-frequency images fold back as audible alias tones^[7]. Production resamplers use windowed-sinc / polyphase filters, trading taps for quality. Linear is acceptable only if the signal is already heavily oversampled. Pitch-by-resampling also changes duration and shifts the formants along with the pitch (the chipmunk effect); keeping duration and formants fixed takes a separate pitch-shifting algorithm.

07Spatial audio

An enemy walks from your left to your right, and its footsteps should follow it without getting quieter in the middle. Placing a sound in space starts with stereo panning. Equal-power panning maps the pan angle θ ∈ [0, π/2] to L = cos θ, R = sin θ. Because loudness tracks power (amplitude²) and cos²θ + sin²θ = 1, the total power is flat across the pan; at center each channel is 0.707 (−3 dB)^[6]. Linear panning instead dips to −3 dB total at center, the audible "hole in the middle."

eq. 1 \cdot equal-power pan L = cos θ, R = sin θ, L ² + R ² = 1

In words: turn the pan position into an angle from 0 (left) to a quarter turn (right) and use its cosine and sine as the two channel gains. The squares of the gains always add to 1, so the total power, and with it the loudness, stays the same wherever the sound is. Hover any term to see what it stands for.

Worked example

At center, θ = π/4: equal-power gives L = R = 0.707, and 0.707² + 0.707² = 1.0, so total power stays at 0 dB. Linear panning gives L = R = 0.5, and 0.5² + 0.5² = 0.5, which is 10 × log₁₀ 0.5 = −3.0 dB: the sound gets quieter as it passes through the middle.

The caveat a mix engineer will raise

"Constant power is flat" assumes the two channels combine incoherently (power-additive), which holds for two loudspeakers in a room. If they combine coherently (a true stereo-to-mono fold-down, or a listener exactly equidistant), equal-power's center is +3 dB hot while linear's is flat, which is why the −4.5 dB compromise pan law (the geometric mean of the two) exists and why pro tools expose selectable pan laws^[6]. Equal-power is a sound default for game stereo, as long as you know the assumption.

The widget plots both laws across the sweep:

Beyond panning: distance attenuation makes far sounds quieter. Physics says intensity falls as 1/r², but a raw 1/r² curve blows up at r→0, so game audio APIs clamp it; OpenAL's default is an inverse-distance model with a reference distance, a tunable rolloff, and the distance clamped to a max^[8]: gain = ref ÷ (ref + rolloff × (distance − ref)). With a 1 m reference and a rolloff of 1, a sound 10 m away plays at 1 ÷ (1 + 9) = 0.1, or −20 dB, and anything closer than 1 m plays at full gain instead of blowing up. Doppler shifts pitch from relative velocity (computed from velocities, then applied as a resample): a car approaching at 20 m/s plays 343 ÷ (343 − 20) = 1.062 times higher, about one semitone. And (head-related transfer function) filtering convolves a mono source with the head-related impulse responses, recordings of how a click from that direction reaches each ear, for a direction to place it in full 3D over headphones; it's what Steam Audio and similar spatializers do^[12]. HRTFs are binaural (for headphones) and differ from head to head; with a generic set measured on someone else, listeners make markedly more front/back and up/down confusions than with real sources^[14].

08APIs & libraries

The callback in §1 has to be driven by something different on every platform you ship. Each OS has a native low-level API that moves buffers to the device on a fixed schedule: WASAPI on Windows (your own thread waits on an event or polls, then writes into the rendering endpoint buffer; the audio engine mixes app streams in shared mode; exclusive mode bypasses it)^[9], CoreAudio / Audio Units on macOS/iOS (a render callback that pulls samples from you), ALSA on Linux (you write periods into the PCM device's ring buffer, blocking or polled). The deadline is the same; who owns the thread and how you are woken differ.

In practice you use a cross-platform layer. C++: miniaudio (a single-file C library, v0.11.25) hides all of the above behind one data callback, running its own audio thread on backends such as WASAPI that don't call you^[11]. Rust: cpal (v0.18.2, the low-level device/stream layer) and rodio (v0.22.2, higher-level playback on top)^[10]. Many shipped games use middleware, a full audio engine plus authoring tool (FMOD, Wwise), or their engine's built-in audio instead, with Steam Audio or similar for 3D audio. Under the authoring tools and effects, those run a voice mixer of the same shape as the one built here.

Wrong answers, and why: the callback rule is about unbounded worst case (not average speed or corruption); and clipping is fixed with headroom plus a limiter, not by dividing by N or accepting the distortion.

09Pitfalls

malloc/lock in the callbackUnbounded worst case → missed deadline → click. Use a lock-free ring.

Freeing on the audio threadAn Arc/shared_ptr drop is a free. Retire buffers back to the game thread.

Dividing the mix by NAttenuates everything. Use headroom + a limiter instead.

Audio frame vs video frameAn audio frame is one sample per channel. Say which you mean.

Linear resampling on raw audioIt aliases (26 dB images). Oversample first or use windowed-sinc.

Linear panning−3 dB hole in the middle. Use equal-power (with the fold-down caveat).

Pure 1/r² distanceBlows up at r→0. Use a clamped rolloff with a reference distance.

Tiny buffer for "low latency"Tighter deadline → underruns. Balance latency against robustness.

10What's next

The next module, Audio DSP & the FFT, covers the signal processing that runs inside this callback: filters, spectra and reverb. With audio in place the engine can run a loop, draw sprites, take input, load assets, and make sound, which is what the 2D-game capstone after it wires together into a small playable game in C++ and Rust. After that, the series turns to 3D. The full path is on the series hub.

Ross Bencina. "Real-time audio programming 101: time waits for nothing" (2011). rossbencina.com. The canonical no-allocate / no-lock / no-block callback rules; judge code by worst-case execution time, not average-case (amortized) complexity.
Timur Doumler. "Using locks in real-time audio processing, safely" (2020). timur.audio (archived). A std::mutex is unsafe on the audio thread even via try_lock, because the unlock may have to wake a waiting thread (a system call); a spinlock the audio thread only try-locks, with backoff on the other thread, is the safe locking option.
Android Open Source Project. "Avoiding Priority Inversion." source.android.com. Priority inversion in audio manifests as glitches/dropouts; the canonical mechanism.
"Pulse-code modulation." Wikipedia. en.wikipedia.org. Sampling and quantization: a continuous waveform becomes integer samples at a fixed sample rate and bit depth.
PortAudio. pa_ringbuffer.h reference. portaudio.com. The lock-free ring works only single-reader / single-writer, with a power-of-two capacity.
Anders Øland and Roger Dannenberg. "Loudness Concepts & Pan Laws" (CMU). cs.cmu.edu. Equal-power panning (cos/sin, flat power, −3 dB center), the linear hole-in-middle, the −4.5 dB compromise law, and the headroom convention of metering down from 0 dB.
Julius O. Smith III (CCRMA). "Linear Interpolation Frequency Response." ccrma.stanford.edu. Linear interpolation is sinc²(fT), about 26 dB image suppression, so it aliases; windowed-sinc is the fix.
The OpenAL 1.1 Specification. openal.org. The inverse-distance-clamped default attenuation model and the Doppler formula (speed of sound 343.3, factor 1.0).
Microsoft. "About WASAPI." learn.microsoft.com. The app writes a rendering endpoint buffer; the audio engine mixes app streams; shared vs exclusive mode.
cpal (Rust). docs.rs/cpal (v0.18.2) and rodio (v0.22.2). The high-priority audio thread; build_output_stream with the timeout parameter.
David Reid. miniaudio. miniaud.io (v0.11.25). A single-file C audio library; the data callback shape, and the rule that the device must not be started or stopped from inside the callback.
Valve. Steam Audio. valvesoftware.github.io/steam-audio. HRTF-based binaural spatialization (the headphone 3D-audio technique), with occlusion and reverb.
"Dynamic range compression." Wikipedia. en.wikipedia.org. A limiter is a compressor with a high ratio and fast attack; limiting keeps the signal under the threshold with far less distortion than hard clipping; gain pumping.
Elizabeth M. Wenzel, Marianne Arruda, Doris J. Kistler, Frederic L. Wightman. "Localization using nonindividualized head-related transfer functions." JASA 94(1), 1993. pubmed.ncbi.nlm.nih.gov. With another person's HRTFs, listeners' front/back and up/down confusions rose significantly compared with free-field sources.

01The real-time callback

02PCM fundamentals

03Buffer & latency

04The ring to the audio thread

05Mixing

06Resampling

07Spatial audio

08APIs & libraries

09Pitfalls

10What's next

See also