Building the Trainer

01Three parts and one interface

The trainer from Self-Play Reinforcement Learning keeps everything in one Python process: the rules, your bots, the network and the update. It's easy to follow and slow to run. At its 2,810 games an hour, the 79,360 games of this page's training run would take 28 hours of playing and updating; the trainer on this page spent 84 minutes on them.^[1]

A trainer meant to run for days splits into three parts, each in the language that suits it:

1. An engine in a compiled language. It applies the rules, builds what the network sees and plays your flood-fill, Voronoi and alpha-beta bots, for hundreds of games held in one object, stepped together in one call and spread over the CPU's cores. The project's engine is Rust; C++ works the same way.
2. A C interface. A handful of plain functions, such as "make 512 games", "write what each player to move sees into this array" and "play these 512 moves", that Python can call directly. Section 02 builds one.
3. A learner in Python. PyTorch runs the network on the GPU and computes the gradients. Around it is Python you write: the PPO loop, the opponents, the gate (the test a new network must pass to replace the best one so far) and the checkpoints.

The network goes on the GPU because it's fast there: on the project's machine it trained on 61,377 positions a second, against 842 with four CPU threads on a CPU that other jobs kept busy.^[1] The engine and bots stay on the CPU, which suits flood fills and tree searches.

A common question is whether the learning algorithm itself comes from a library such as PyTorch. PyTorch does two jobs: its autograd package "provides classes and functions implementing automatic differentiation of arbitrary scalar valued functions",^[2] so from the loss you write it works out how to nudge every weight, and it runs the network on the GPU. The rest is your code: the 332 lines of ppo.py from Self-Play Reinforcement Learning, and the 116 of trainer.py here.^[1]

Built by hand, the search route needs a simulator proven against the referee (Referees and Simulators), a harness that measures with intervals (Measuring Bot Strength) and a search bot; learning adds a network (Neural Networks for Game Bots), this trainer and an exporter (Shipping a Network in 100,000 Characters). Capstone: Build the Lab lists each with its check.

02Call compiled code from Python

Python can't call a Rust or C++ function directly: each language names functions, passes arguments and lays out data its own way. All three can use C's way, and almost any language can call a function that follows C's rules, so a C interface is where they meet.

Those rules are C's ABI, its application binary interface: which registers carry the arguments, how big each type is, how a pointer comes back. Calling across languages through it is a foreign function interface, or FFI.

In Rust, extern "C" "makes this function adhere to the C calling convention", in the Rustonomicon's words, and #[no_mangle] "turns off Rust's name mangling, so that it has a well defined symbol to link to".^[4] A cdylib crate builds "a dynamic library to be loaded from another language": a .so file on Linux, a .dll on Windows.^[5] In C++, extern "C" { } does both jobs.

Python's standard ctypes module, "a foreign function library for Python", loads the library and makes the calls.^[6] It can't read signatures from the library, so you declare each function's argument types (argtypes) and return type (restype), and nothing checks you: a call "with incorrect arguments can corrupt memory or crash Python". A function you didn't declare is assumed to return a C int, 32 bits on the usual platforms, which cuts a 64-bit pointer in half.

The interface is small. The toy engine of section 10 has five functions, and the project's has the same shape with more outputs:

The games live inside the library as one object. Python holds only its address, a handle, and passes it to every call; env_free destroys the object, once. Calls on one batch must not overlap: nothing inside locks it.

The arrays go the other way. Python allocates them once with numpy and passes their addresses, and the library writes straight into them: nothing is converted or copied between the languages, which is what zero-copy means here. numpy's ctypes.data_as gives the address, and "the returned pointer will keep a reference to the array".^[7] torch.from_numpy wraps the same memory: "The returned tensor and ndarray share the same memory."^[8] Any copy after that is your choice: the trip to the GPU, or keeping an observation that the next call would overwrite (section 12).

A crash inside the library is a crash of the trainer. In current Rust, a panic (Rust's stop on an unhandled error) can't unwind out of an extern "C" function: the "panic will cause the process to safely abort".^[4] In C++, noexcept functions end the same way, through std::terminate. Better still, keep game events from failing at all: the project's tron_step treats an unknown move number as a crash of the player who sent it, as CodinGame treats an invalid command.

03One iteration, end to end

A trainer counts progress in iterations: play a batch of games, learn from them, and repeat thousands of times. In the author's lab, the first Back Track King network to reach Legend (on the permanent ladder, after the contest) came from iteration 3,350, after 917,630 games.^[3] What one iteration does, and what each part costs, shows where a faster trainer has to save time.

Self-Play Reinforcement Learning builds the algorithm, PPO (Proximal Policy Optimization): play games with the current network, give each decision an advantage (how much better the game went after that move than the network's value prediction expected), then make moves with positive advantages more likely, in clipped steps: once a move's probability has moved a little, the loss stops rewarding further change.^[9] An iteration does that once, in five stages:

1. Rollout. Each turn, the engine steps every game on the CPU and the network picks the learner's moves in all of them in one GPU pass; opponents' moves come from past copies of the network or from your bots. Each learner decision is stored with what it saw, its move, that move's probability and the value prediction.
2. Advantages. A backward sweep over each player's decisions gives each one its advantage with generalized advantage estimation (GAE), which mixes the network's value predictions with the game's result through a discount γ and a blending factor λ.^[10] It's cheap arithmetic.
3. PPO update. The decisions are shuffled into minibatches of 2,048, one gradient step each; three passes over the batch, three epochs, make the update.
4. Weight average. The author's lab keeps a moving average of the weights, an EMA (average = 0.995 × average + 0.005 × new), and gates and exports the average.^[3] Pduhard, 3rd in Back Track King, averaged saved checkpoints and found the average "reliably stronger than any single one of them, by 10 to 15 points".^[11] The project skips this stage.
5. Screen. Every few iterations the learner plays the champion, the best network so far, on seeds training never uses (section 08). Every iteration ends with a checkpoint and a status file (section 06).

The lab's iteration advances 512 games by 32 turns: the fixed-length segments of the PPO paper's algorithm, in which "each of N (parallel) actors collect T timesteps of data".^[9] Games carry over from one iteration to the next. The project plays 512 complete games per iteration instead, so its iterations vary and run larger: 130,448 learner decisions on average.^[1]

The model's 9.75 s is below the run's real 12.6 s per iteration (11.71 hours for 3,350), mostly because medians hide slow iterations; benchmark games against the lab's own bots every 150 iterations and checkpoint saves add the rest.^[3] Either way, an iteration is hundreds of games.

04Why one game at a time is slow

A common question once a first trainer runs: a game is 100 turns between two players at 60 ms a move, so shouldn't it take a second or two? Multiply it out: 100 × 2 × 60 ms is 12 seconds of thinking per game, before the referee and two bot processes have even started.

That 60 ms is a search bot's time budget: it searches until its clock says answer. A trainer has no clock. Its network answers in one fixed pass, computed for 512 positions in parallel on the GPU, so the trainer keeps hundreds of games in one process: one engine call steps them all, one pass decides every learner move.

The project measured the real thing: throughput.py runs two iterations of the same trainer code on each engine, with the network on the GPU. The machine, an Intel Core i9-11900KF (8 cores) with an NVIDIA RTX 3080 Ti, was shared with other training jobs and heavily loaded (a load average of 49 to 66 during training), so the project's wall-clock numbers are pessimistic.^[1]

The real runs agree: 2,810 games an hour for the single-process trainer with 32 games per iteration, and 63,054 for the batched one, then 53,698 after a memory fix (section 06).^[1]

Can a faster or overclocked GPU speed this up without hurting the result? Measure where the time goes first. In the project's batched run the update, the one stage that runs almost all on the GPU, was under a third of each iteration (10.1 of 32.7 s); the rest was the rollout, where engine, bots and network take turns.^[1] The author's lab got its gains by removing waits, with the recipe and GPU clocks unchanged.

05Your bots inside the simulator

Self-play alone can drift: the learner gets better at beating its own recent copies and worse at everything else (Self-Play Reinforcement Learning). A share of games against your own fixed bots keeps it honest, and those bots have to run inside the trainer's loop, at its speed.

The project ports each bot into the engine: tron_bot_moves asks the Rust flood-fill, Voronoi or alpha-beta bot for its move in every game a mask selects, split over CPU threads. It's fast and there's no separate program to crash, but a port is a second copy of your bot and needs checking like the engine: the project's Rust bots matched this series' browser bots on all 27,506 moves of 100 games.^[1]

The other way runs your submitted program as a subprocess: the trainer sends it the text CodinGame would and turns its printed command into the seat's move, so, in the author's lab's notes, "it plays exactly as on CodinGame". The lab sends every game's input before collecting any answer, and its notes report its bots answering 128 games in about 9 ms a turn.^[3]

Opponents come from three places: the learner itself, a pool of its past copies called snapshots, and your fixed bots, called native bots here. The lab's default, which the project copies, is 25% native, 37.5% self-play and 37.5% snapshots. Self-play gives the learner both seats, so it's 0.75 ÷ 1.375 = 54.5% of its decisions.

In the pool, a snapshot the learner still loses to is drawn more often. Its weight is (1 − rate)² + 0.02, where rate = (points + 2) ÷ (games + 4) is the learner's score against it, pulled toward 0.5 while games are few.^[3] Pduhard describes the same idea: "the ones that beat us are drawn more often".^[11]

The weighting only bites when an old snapshot still wins. In the project's run, the records against its 32 snapshots ended between 0.49 and 0.63, and each got 2.0% to 3.7% of the pool's games.^[1]

Long runs also meet an opponent that keeps crashing. The lab's rule-based C++ bot for Spring Challenge 2022, kept byte for byte, held pointers into a list that grew and was never cleared, and crashed around turn 212 of long games when the list moved in memory. Training survived it; the gate didn't, and the run failed at one iteration until its launcher gave up after 20 attempts with a success exit code, so nothing restarted it.^[3]

The fix gives three rules worth copying. Leave a crashed or silent bot's game out of the score and log its map, seat and turn: counting it as a win rewards the learner for breaking the bot. Fail the gate's check against an opponent that finished no games. Make a launcher that gives up exit with an error code. The bot still crashed about 4,000 times a day, so the lab also patched a copy with one line that clears the list each turn; it gave the original's reply on every turn checked, 94,145 of them against three opponents.^[3]

06Runs that survive a crash

A run that lasts a day gets interrupted: an opponent crashes, the machine reboots, the GPU fills up, or you stop it to submit. Each interruption should cost at most the iteration in progress.

So save a checkpoint after every iteration, with more than the weights: PyTorch's tutorial on saving models says "it is important to also save the optimizer's state_dict, as this contains buffers and parameters that are updated as the model trains."^[12] The project's latest.pt holds the learner, the champion, the optimizer (Adam keeps two running averages per weight), the pool with its records and the counters, the next training seed among them: 1.9 MB.^[1]

Write it to a temporary file, then rename that over the old one. The rename is atomic: a reader, your next run included, sees the old checkpoint or the new one, never half of each. Python's documentation for os.replace: "If successful, the renaming will be an atomic operation (this is a POSIX requirement)."^[13] That covers a crash or a kill; a power cut can still lose bytes the operating system hadn't written yet, unless you fsync the file (make the operating system write it to the disk) before the rename, which the project doesn't.

--resume loads latest.pt and carries on with the same counters, pool, optimizer and champion. Its random generators restart from the seed and the iteration number, so it continues the run without replaying exactly what an uninterrupted one would have done.

Ctrl+C (the SIGINT signal) only sets a flag: the loop finishes the iteration, saves and exits with code 0, and a second Ctrl+C quits at once.

A common request is one file to run until stopped that saves on Ctrl+C and hands you the bot. In the project that's trainer.py --minutes 100000 --export: Ctrl+C finishes the iteration, saves latest.pt and champion.pt, and runs the exporter of Shipping a Network in 100,000 Characters. Saving takes a moment. The export takes minutes, because it compiles the bot and replays tens of thousands of turns against the network: 11 minutes on the project's loaded machine,^[1] and about 2 minutes 40 seconds for the lab's last Back Track King export, in two encodings (Capstone: Build the Lab).^[3]

To survive crashes and reboots, let the operating system run the trainer: on Linux, a systemd unit, enabled so it starts at boot, with Restart=on-failure, which restarts the service "when the process exits with a non-zero exit code, is terminated by a signal" other than the few it counts as clean, SIGINT among them, or times out.^[15] A trainer stopped on purpose exits 0 and stays stopped; a launcher that gives up must exit non-zero, or it sits idle like the lab's launcher in section 05. The restart command adds --resume when a checkpoint exists.

A shared GPU needs care too. PyTorch caches freed GPU memory, and torch.cuda.empty_cache() "releases all unused cached memory from PyTorch so that those can be used by other GPU applications";^[16] the project's update calls it every iteration. Its first batched version still held 3.6 GiB, the whole batch on the GPU, on a card with 11.6 of 12.3 GB in use. At iteration 51 the run was stopped with Ctrl+C and resumed after two changes: planes packed eight cells to a byte, and each 2,048-decision minibatch run as four passes of 512 whose gradients add up to the same step. Memory fell to 0.43 GiB. The update slowed from 7.8 to 11.2 s an iteration on average, and the run went from 63,054 games an hour before the change to 53,698 after it.^[1]

Test the stop and the resume before you trust them. The project's checks/durability.sh sends Ctrl+C to a small run after two iterations, resumes it and prints what happened: the run stopped after the third iteration with exit code 0, and --resume continued at iteration 4. The stop at iteration 51 was the same test at full size.^[1]

07Guardrails that keep the numbers honest

A trainer that runs all night can draw a convincing curve from a bug. These rules, set before the long run, catch the bugs that look like progress.

1. Give every seed a purpose, in writing. A seed fixes a game's start, so a range of seeds is a set of maps. Reserve ranges before training and never mix them. A list like the project's, below, is a seed ledger.
2. Never gate on training maps. JOiy (13th in Back Track King) found his promotions had been "decided on the same 20 000-map database training drew from", and that "a checkpoint can clear that gate by memorising boards"; he moved the gate, benchmarks and holdout games to 2,000 maps training never used.^[17]
3. Run the regression tests before every long run. The project's: 8 engine tests, the Rust bots against the browser bots move by move, the Python engine against the Rust one on 4,472 positions, and the Rust network against PyTorch. A broken engine still trains; it learns the wrong game.
4. Build observations only from what the seat was sent. The lab's rule: "An observation is a pure function of the protocol input the seat has received."^[3] Light cycles hide nothing, but with fog of war, an observation read from the engine's state can show the network what its live version will never see.
5. Keep fixed opponents in the mix and in the measurements: native-bot games in training (section 05) and a curve against bots that never change (section 08).
6. Prove the exported bot decides like the network. Shipping a Network in 100,000 Characters replays every turn of hundreds of games through both.
7. Save the settings with the run. The project writes every argument to config.json when a run starts.

08Read the dashboard without fooling yourself

Once a run goes overnight, you need a way to see what it's doing. The trainer rewrites a small status file after every iteration, atomically, so a script, a terminal or a desktop widget can read it any time. The project's, at the end of its run (trimmed):

{
 "iteration": 155,
 "games": 79360,
 "decisions": 20219507,
 "train_seconds": 6023.8,
 "games_per_hour": 47428,
 "champion_iteration": 60,
 "champion_games": 30720,
 "pool": [{"iteration": 0, "games": 2679, "points": 1686.0}, ... 31 more snapshots],
 "last_curve": {"iteration": 144, "games": 73728,
                "voronoi": {"wins": 286, "games": 400, "rate": 0.715, "low": 0.6689, "high": 0.757},
                "alphabeta": {"wins": 202, "games": 400, "rate": 0.505, "low": 0.4562, "high": 0.5537},
                ... the flood-fill bot and the clone},
 "session_seconds": 4232.2
}

metrics.jsonl keeps a line per iteration and curve.jsonl one per strength measurement, and watch -n 30 cat runs/ppo/status.json is already a dashboard; the author's lab reads the same kind of files with a status command and a desktop widget.^[3] The games_per_hour field divides by the whole training time, gates and curve points included, so it reads 47,428.

The numbers come in two kinds. The screen measures the learner against the champion, which moves whenever it's replaced; the curve measures it against bots that never change (your teacher, the clone it started from, the alpha-beta bot). A promotion can come with a dip against one fixed bot, and every number is a sample with an interval (Measuring Bot Strength).

Screens near 0.5 are normal. Once gains get small, the learner and the champion, its own best earlier version, are close to level, and the gate waits for a clear win, so it goes quiet well before learning stops.

In the project's run, the screens after iteration 60 read 0.48 to 0.56, all inside the range a level learner produces 95% of the time (53 to 75 wins of 128), and there was no promotion in the last 48,640 games.^[1]

What about a run with no visible progress for eight hours? Late in a run that's often normal: gains near a plateau are a few points, and seeing a few points takes hundreds of games per comparison (Measuring Bot Strength works out how many). Check the curve before calling a run stalled.

How long should you run it? Until the gate stops promoting and the curve against your fixed bots flattens. The author's lab's first Back Track King network to reach Legend came after 917,630 games, about 11.7 hours on one RTX 3080 Ti; the run went on to 21.8 million games, and its later local gains didn't show on the live ladder.^[3] Self-Play Reinforcement Learning reads the project's curve.

09Settings to start from

Every setting in a trainer is a choice someone made for one game. Start from a set that worked somewhere, change one thing at a time and measure each change. The table puts the author's lab's defaults beside the project's, which were chosen for light cycles. Most of the lab's were set for Back Track King and carried to its other games; the native-bot share came later, after a Spring Challenge 2022 run drifted, and the Back Track King run never used it.

The lab's Back Track King run itself trained at 1.5 × 10⁻⁴ with a 0.55 screen until about iteration 21,600, then at 5 × 10⁻⁵ with the 0.52 default.^[3] Training through the export's rounding is quantization-aware training, which Shipping a Network in 100,000 Characters explains.

The project raised its learning rate and entropy bonus from a first guess (1 × 10⁻⁴ and 0.003) because the clone's move probabilities were so peaked that a short run barely moved them.^[1] JOiy's Back Track King settings differ again: "512 envs, horizon 128, LR 7.5e-5, entropy 0.001, PPO epochs 3, 4 minibatches, clip 0.2".^[17] Treat each column as one game's answer.

10The trainer in miniature, in code

The project's files are long because they do everything. These listings are the same design in miniature: toy games behind a C interface in C++ and in Rust, the Python loader that drives either library, and a PyTorch loop with PPO's clipped update, a checkpoint, resume, a status file and Ctrl+C.

The toy game is a corridor of nine cells. A token starts in the middle and steps left or right each turn. One end is the goal, chosen by the game's seed: reaching it scores +1, the other end −1, and 16 steps without reaching either is a draw. The observation is 10 bytes, one per cell plus the goal's side, and the policy is one linear layer from those 10 numbers to two logits, one score per move that a softmax turns into the moves' probabilities.

// A batch of toy games behind a C ABI, with the same functions and numbers as toy_env.rs.
// Build: g++ -std=c++20 -O2 -shared -fPIC toy_env.cpp -o libtoy_env_cpp.so
//
// The toy game: a corridor of 9 cells. A token starts in the middle and steps left or right; the
// goal is one end, picked by the game's seed. Reaching the goal scores +1, the other end -1, and a
// game that reaches neither end in 16 steps is a draw.
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <vector>

namespace {
constexpr std::size_t CELLS = 9;
constexpr std::size_t OBSERVATION = CELLS + 1;  // one byte per cell (1 where the token is), then the goal side
constexpr std::size_t START = CELLS / 2;
constexpr std::uint32_t MAX_STEPS = 16;

// SplitMix64: turns a seed into well-mixed bits, the same numbers in C++ and Rust.
// Unsigned arithmetic wraps, like Rust's wrapping_add and wrapping_mul.
std::uint64_t mix(std::uint64_t seed) {
    std::uint64_t bits = seed + 0x9E3779B97F4A7C15ULL;
    bits = (bits ^ (bits >> 30)) * 0xBF58476D1CE4E5B9ULL;
    bits = (bits ^ (bits >> 27)) * 0x94D049BB133111EBULL;
    return bits ^ (bits >> 31);
}

struct Game {
    std::size_t position = START;
    std::size_t goal;
    std::uint32_t steps = 0;
    bool done = false;
    explicit Game(std::uint64_t seed) : goal((mix(seed) & 1) == 1 ? CELLS - 1 : 0) {}
};
}  // namespace

// Python only ever sees a pointer to this, so its layout can stay C++'s business.
struct Batch {
    std::vector<Game> games;
};

// extern "C" turns off C++ name mangling, so ctypes finds each function by its plain name.
// noexcept: an exception must not unwind into Python's C frames; it ends the process instead.
extern "C" {

// A batch of `count` games; game i starts from seed `first_seed + i`.
Batch* env_new(std::uint32_t count, std::uint64_t first_seed) noexcept {
    auto* batch = new Batch;
    batch->games.reserve(count);
    for (std::uint64_t index = 0; index < count; ++index) batch->games.emplace_back(first_seed + index);
    return batch;  // the caller owns it now; env_free gives it back
}

void env_free(Batch* batch) noexcept { delete batch; }

// Bytes per game in env_observe, so the caller can size its buffer without hard-coding it.
std::uint32_t env_observation_size() noexcept { return static_cast<std::uint32_t>(OBSERVATION); }

// Writes every game's observation into `out` (count × OBSERVATION bytes); a finished game writes zeros.
void env_observe(const Batch* batch, std::uint8_t* out) noexcept {
    for (const Game& game : batch->games) {
        std::memset(out, 0, OBSERVATION);
        if (!game.done) {
            out[game.position] = 1;
            out[CELLS] = game.goal == CELLS - 1 ? 1 : 0;
        }
        out += OBSERVATION;
    }
}

// One action per game (1 steps right, anything else steps left). Writes each game's reward for
// this step and whether it has ended. A finished game ignores its action and scores 0.
void env_step(Batch* batch, const std::int32_t* actions, float* rewards, std::uint8_t* done) noexcept {
    for (std::size_t index = 0; index < batch->games.size(); ++index) {
        Game& game = batch->games[index];
        rewards[index] = 0.0f;
        if (!game.done) {
            // A running game is never at an end, so the position can't leave the corridor.
            game.position = actions[index] == 1 ? game.position + 1 : game.position - 1;
            game.steps += 1;
            if (game.position == 0 || game.position == CELLS - 1) {
                rewards[index] = game.position == game.goal ? 1.0f : -1.0f;
                game.done = true;
            } else if (game.steps == MAX_STEPS) {
                game.done = true;  // a draw: no reward either way
            }
        }
        done[index] = game.done ? 1 : 0;
    }
}

}  // extern "C"

//! A batch of toy games behind a C ABI. Python makes one call per step for every game in the batch.
//! Build: rustc --edition 2021 --crate-type cdylib -O toy_env.rs   (writes libtoy_env.so)
//!
//! The toy game: a corridor of 9 cells. A token starts in the middle and steps left or right; the
//! goal is one end, picked by the game's seed. Reaching the goal scores +1, the other end -1, and a
//! game that reaches neither end in 16 steps is a draw.
use std::slice;

const CELLS: usize = 9;
const OBSERVATION: usize = CELLS + 1; // one byte per cell (1 where the token is), then the goal side
const START: usize = CELLS / 2;
const MAX_STEPS: u32 = 16;

/// SplitMix64: turns a seed into well-mixed bits, the same numbers in C++ and Rust.
fn mix(seed: u64) -> u64 {
    let mut bits = seed.wrapping_add(0x9E37_79B9_7F4A_7C15);
    bits = (bits ^ (bits >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
    bits = (bits ^ (bits >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
    bits ^ (bits >> 31)
}

struct Game {
    position: usize,
    goal: usize,
    steps: u32,
    done: bool,
}

impl Game {
    fn new(seed: u64) -> Game {
        let goal = if mix(seed) & 1 == 1 { CELLS - 1 } else { 0 };
        Game { position: START, goal, steps: 0, done: false }
    }
}

/// Python only ever sees a pointer to this, so its layout can stay Rust's business.
pub struct Batch {
    games: Vec<Game>,
}

/// A batch of `count` games; game i starts from seed `first_seed + i`.
#[no_mangle] // Rust 2024 spells this #[unsafe(no_mangle)]
pub extern "C" fn env_new(count: u32, first_seed: u64) -> *mut Batch {
    let games = (0..u64::from(count)).map(|index| Game::new(first_seed + index)).collect();
    // into_raw hands ownership to the caller; env_free takes it back.
    Box::into_raw(Box::new(Batch { games }))
}

/// # Safety
/// `batch` must come from `env_new` and must not be used afterwards.
#[no_mangle]
pub unsafe extern "C" fn env_free(batch: *mut Batch) {
    if !batch.is_null() {
        drop(Box::from_raw(batch));
    }
}

/// Bytes per game in `env_observe`, so the caller can size its buffer without hard-coding it.
#[no_mangle]
pub extern "C" fn env_observation_size() -> u32 {
    OBSERVATION as u32
}

/// Writes every game's observation into `out`; a finished game writes zeros.
/// # Safety
/// `out` must hold count × OBSERVATION bytes.
#[no_mangle]
pub unsafe extern "C" fn env_observe(batch: *const Batch, out: *mut u8) {
    let games = &(*batch).games;
    let out = slice::from_raw_parts_mut(out, games.len() * OBSERVATION);
    for (game, row) in games.iter().zip(out.chunks_exact_mut(OBSERVATION)) {
        row.fill(0);
        if !game.done {
            row[game.position] = 1;
            row[CELLS] = u8::from(game.goal == CELLS - 1);
        }
    }
}

/// One action per game (1 steps right, anything else steps left). Writes each game's reward for
/// this step and whether it has ended. A finished game ignores its action and scores 0.
/// # Safety
/// `actions`, `rewards` and `done` must each hold one entry per game.
#[no_mangle]
pub unsafe extern "C" fn env_step(batch: *mut Batch, actions: *const i32, rewards: *mut f32, done: *mut u8) {
    let games = &mut (*batch).games;
    let actions = slice::from_raw_parts(actions, games.len());
    let rewards = slice::from_raw_parts_mut(rewards, games.len());
    let done = slice::from_raw_parts_mut(done, games.len());
    for (index, game) in games.iter_mut().enumerate() {
        rewards[index] = 0.0;
        if !game.done {
            // A running game is never at an end, so the position can't leave the corridor.
            game.position = if actions[index] == 1 { game.position + 1 } else { game.position - 1 };
            game.steps += 1;
            if game.position == 0 || game.position == CELLS - 1 {
                rewards[index] = if game.position == game.goal { 1.0 } else { -1.0 };
                game.done = true;
            } else if game.steps == MAX_STEPS {
                game.done = true; // a draw: no reward either way
            }
        }
        done[index] = u8::from(game.done);
    }
}

"""Python side of the toy C ABI (toy_env.rs or toy_env.cpp): many games stepped by one call, with
numpy arrays that the library writes into.

    python toy_env.py ./libtoy_env.so         # the Rust build
    python toy_env.py ./libtoy_env_cpp.so     # the C++ build: same calls, same numbers
"""
import ctypes
import sys

import numpy as np

U8, I32, F32 = (ctypes.POINTER(kind) for kind in (ctypes.c_uint8, ctypes.c_int32, ctypes.c_float))


def load(path):
    """Opens the library and declares every function's signature. ctypes can't read signatures from
    a library, and its default return type, a C int, would cut a 64-bit pointer in half."""
    library = ctypes.CDLL(path)
    signatures = {
        'env_new': ([ctypes.c_uint32, ctypes.c_uint64], ctypes.c_void_p),
        'env_free': ([ctypes.c_void_p], None),
        'env_observation_size': ([], ctypes.c_uint32),
        'env_observe': ([ctypes.c_void_p, U8], None),
        'env_step': ([ctypes.c_void_p, I32, F32, U8], None),
    }
    for name, (arguments, result) in signatures.items():
        function = getattr(library, name)
        function.argtypes, function.restype = arguments, result
    return library


class ToyBatch:
    """`count` games behind one pointer; game i starts from seed first_seed + i."""

    def __init__(self, library, count, first_seed):
        self.library, self.count = library, count
        self.handle = library.env_new(count, first_seed)
        # Python owns these arrays and the library writes straight into them, so nothing is
        # serialized or copied on the way back.
        self.observations = np.zeros((count, library.env_observation_size()), np.uint8)
        self.rewards = np.zeros(count, np.float32)
        self.done = np.zeros(count, np.uint8)

    def observe(self):
        """The same array every call, overwritten in place: copy it to keep it."""
        self.library.env_observe(self.handle, self.observations.ctypes.data_as(U8))
        return self.observations

    def step(self, actions):
        # The library reads `count` int32 values from this address, so hand it exactly that.
        actions = np.ascontiguousarray(actions, dtype=np.int32)
        assert actions.shape == (self.count,)
        self.library.env_step(self.handle, actions.ctypes.data_as(I32), self.rewards.ctypes.data_as(F32),
                              self.done.ctypes.data_as(U8))
        return self.rewards, self.done.astype(bool)

    def close(self):
        if self.handle:
            self.library.env_free(self.handle)
            self.handle = None


def check(path, count=1000):
    """Walks every game toward its goal: each one must win in exactly 4 steps."""
    games = ToyBatch(load(path), count, first_seed=1)
    address = games.observations.ctypes.data
    goal_right = games.observe()[:, -1].copy()
    steps, total = 0, np.zeros(count, np.float32)
    while games.observe().any():            # a finished game observes all zeros
        rewards, _ = games.step(goal_right)  # 1 = right, 0 = left
        total += rewards
        steps += 1
    assert games.observations.ctypes.data == address, 'the library must fill our array in place'
    assert steps == 4 and (total == 1).all()
    print(f'{path}: {count} games won in {steps} steps; goal on the right in {int(goal_right.sum())}')
    games.close()


if __name__ == '__main__':
    check(sys.argv[1])

"""The trainer's loop in miniature: the toy games from toy_env.py, a tiny PyTorch policy and PPO's
clipped update, plus what a long run needs: a checkpoint written atomically after every iteration,
resume, a status file, and Ctrl+C that finishes the iteration before stopping.

    python toy_train.py ./libtoy_env.so --out toy-run --iterations 20
"""
import argparse
import json
import os
import signal
from pathlib import Path

import numpy as np
import torch

from toy_env import ToyBatch, load

GAMES, EPOCHS, MINIBATCH = 512, 3, 1024
CLIP, GAMMA, LEARNING_RATE = 0.2, 0.95, 0.05


def rollout(policy, library, first_seed, device):
    """Plays GAMES complete games with the current policy and returns every decision with its advantage,
    plus the share of games won."""
    games = ToyBatch(library, GAMES, first_seed)
    states, actions, log_probs, owners = [], [], [], []
    outcome = np.zeros(GAMES, np.float32)
    while (alive := np.flatnonzero(games.observe().any(1))).size:
        state = torch.from_numpy(games.observations[alive].astype(np.float32)).to(device)
        with torch.no_grad():
            choice = torch.distributions.Categorical(logits=policy(state))
            action = choice.sample()
        moves = np.zeros(GAMES, np.int32)
        moves[alive] = action.cpu().numpy()
        rewards, _ = games.step(moves)
        outcome += rewards
        states.append(state)
        actions.append(action)
        log_probs.append(choice.log_prob(action))
        owners.append(torch.from_numpy(alive))
    games.close()
    owners = torch.cat(owners)
    # A decision's return is the game's result, discounted once per decision still to come.
    remaining = torch.zeros(len(owners), dtype=torch.long)
    seen = torch.zeros(GAMES, dtype=torch.long)
    for index in reversed(range(len(owners))):
        remaining[index] = seen[owners[index]]
        seen[owners[index]] += 1
    returns = torch.from_numpy(outcome)[owners] * GAMMA ** remaining.float()
    # Advantage: better or worse than the batch's average decision, scaled to unit spread.
    advantages = (returns - returns.mean()) / (returns.std() + 1e-8)
    batch = (torch.cat(states), torch.cat(actions), torch.cat(log_probs), advantages.to(device))
    return batch, float((outcome == 1).mean())


def update(policy, optimizer, batch):
    """EPOCHS passes over the batch in shuffled minibatches of PPO's clipped loss."""
    states, actions, old_log_probs, advantages = batch
    for _ in range(EPOCHS):
        for index in torch.randperm(len(actions), device=states.device).split(MINIBATCH):
            choice = torch.distributions.Categorical(logits=policy(states[index]))
            ratio = (choice.log_prob(actions[index]) - old_log_probs[index]).exp()
            advantage = advantages[index]
            # No credit for moving the ratio past 1 ± CLIP in the advantage's direction.
            loss = -torch.min(ratio * advantage, ratio.clamp(1 - CLIP, 1 + CLIP) * advantage).mean()
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()


def write_atomically(path, write):
    """Write a temporary file, then rename it over the old one: a reader, or a crash, sees the
    old file or the new one, never half of one."""
    temporary = path.with_name(path.name + '.tmp')
    write(temporary)
    os.replace(temporary, path)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('library')
    parser.add_argument('--out', default='toy-run')
    parser.add_argument('--iterations', type=int, default=20)
    args = parser.parse_args()
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    library, out = load(args.library), Path(args.out)
    out.mkdir(exist_ok=True)
    torch.manual_seed(0)
    policy = torch.nn.Linear(library.env_observation_size(), 2).to(device)   # logits for left, right
    optimizer = torch.optim.Adam(policy.parameters(), lr=LEARNING_RATE)
    counters = dict(iteration=0, games=0)
    checkpoint = out / 'latest.pt'
    if checkpoint.exists():
        # Resume: weights, the optimizer's running averages and the counters, so seeds continue too.
        saved = torch.load(checkpoint, map_location=device)
        policy.load_state_dict(saved['policy'])
        optimizer.load_state_dict(saved['optimizer'])
        counters = saved['counters']
        torch.manual_seed(counters['iteration'])
    stop = []

    def request_stop(signum, frame):
        if stop:
            raise KeyboardInterrupt  # a second Ctrl+C quits at once
        stop.append(True)
        print('stopping after this iteration', flush=True)
    signal.signal(signal.SIGINT, request_stop)

    while counters['iteration'] < args.iterations and not stop:
        batch, won = rollout(policy, library, 1 + counters['iteration'] * GAMES, device)
        update(policy, optimizer, batch)
        counters['iteration'] += 1
        counters['games'] += GAMES
        payload = dict(policy=policy.state_dict(), optimizer=optimizer.state_dict(), counters=counters)
        write_atomically(checkpoint, lambda path: torch.save(payload, path))
        status = json.dumps(dict(counters, decisions=len(batch[1]), won=round(won, 3)))
        write_atomically(out / 'status.json', lambda path: path.write_text(status))
        print(status, flush=True)


if __name__ == '__main__':
    main()

Both libraries compile with warnings as errors (g++ and clang++ -std=c++20 -Wall -Wextra -Werror -O2 -shared -fPIC; rustc --edition 2021 --crate-type cdylib -D warnings -O). On all three builds the loader's check wins all 1,000 games in 4 steps, with the array filled in place, and toy_train.py prints the same 20 lines on a given device. Its sampled policy wins about a quarter of its games in the first iteration (26.6% on the CPU, 24.2% on the GPU, which draws different random numbers) and 99% or more from the sixth on; played greedily on 10,000 unseen seeds, it wins every game. A Ctrl+C mid-run finished the iteration, saved and exited with code 0; resuming continued at the next iteration.

11Project step 4: scale it

cargo build --release --manifest-path engine/Cargo.toml     # libtron.so: the engine and its C interface
python checks/python_engine.py 30                           # the Python engine still matches Rust
cd train
python throughput.py --start ../runs/bc/clone.pt --iterations 2           # games per hour, both engines
python trainer.py --start ../runs/bc/clone.pt --out ../runs/ppo --minutes 100 \
    --games 512 --lr 2e-4 --entropy 0.005 --curve-every 12 --curve-seeds 200
python trainer.py ... --resume       # the same arguments: continue after Ctrl+C or a crash
bash ../checks/durability.sh         # Ctrl+C and resume, tested on a 16-game run

//! The C ABI: many games behind one pointer, so Python makes one call per step for the whole batch
//! instead of one per game. Python owns every buffer (numpy arrays) and Rust writes into them, so no
//! observation is copied twice. Calls on one batch must not overlap.
use crate::alphabeta::{alphabeta_bot, Options};
use crate::board::{CELLS, WIDTH};
use crate::bots::{flood_bot, voronoi_bot, voronoi_scores};
use crate::features::{move_features, move_targets, planes, MOVE_FEATURES, PLANES};
use crate::game::Game;
use crate::net::{Net, Shape};
use std::slice;

pub struct Batch {
    games: Vec<Game>,
}

/// Bots the engine can play itself, by number.
pub const FLOOD: u32 = 0;
pub const VORONOI: u32 = 1;
pub const ALPHABETA: u32 = 2;

/// Runs `work(index, game, output)` for every game, split over `threads` scoped threads.
/// `output` is the game's own slice of a per-game output buffer, so threads never share one.
fn parallel<T: Send>(games: &[Game], out: &mut [T], per_game: usize, threads: usize, work: impl Fn(usize, &Game, &mut [T]) + Sync) {
    if games.is_empty() {
        return;
    }
    let threads = threads.clamp(1, games.len());
    let chunk = games.len().div_ceil(threads);
    std::thread::scope(|scope| {
        for (number, (game_chunk, out_chunk)) in games.chunks(chunk).zip(out.chunks_mut(chunk * per_game)).enumerate() {
            let work = &work;
            scope.spawn(move || {
                for (index_in_chunk, (game, slot)) in game_chunk.iter().zip(out_chunk.chunks_mut(per_game)).enumerate() {
                    work(number * chunk + index_in_chunk, game, slot);
                }
            });
        }
    });
}

/// A batch of `count` games; game i starts from seed `first_seed + i`.
#[no_mangle]
pub extern "C" fn tron_new(count: u32, first_seed: u64) -> *mut Batch {
    let games = (0..count as u64).map(|index| Game::new(first_seed + index)).collect();
    Box::into_raw(Box::new(Batch { games }))
}

/// # Safety
/// `batch` must come from `tron_new` and must not be used afterwards.
#[no_mangle]
pub unsafe extern "C" fn tron_free(batch: *mut Batch) {
    if !batch.is_null() {
        drop(Box::from_raw(batch));
    }
}
// ... tron_reset(batch, index, seed) starts a new game in one slot.

/// For every game: the planes of the player to move (count × PLANES × CELLS bytes), the room and
/// Voronoi margin after each of its moves (count × 4 × MOVE_FEATURES floats), the cell each direction
/// enters (count × 4, -1 off the board), which directions are legal (count × 4) and whose turn it is
/// (count). Finished games get zeros.
/// # Safety
/// The buffers must hold the sizes above.
#[no_mangle]
pub unsafe extern "C" fn tron_observe(batch: *const Batch, threads: u32, plane_out: *mut u8, feature_out: *mut f32, targets_out: *mut i32, legal_out: *mut u8, to_move_out: *mut u8) {
    let games = &(&*batch).games;
    let game_count = games.len();
    let plane_out = slice::from_raw_parts_mut(plane_out, game_count * PLANES * CELLS);
    parallel(games, plane_out, PLANES * CELLS, threads as usize, |_, game, out| {
        if game.done() {
            out.fill(0);
        } else {
            let heads = [Some(game.heads[0]), Some(game.heads[1])];
            planes(&game.cells, game.to_move, &heads, out);
        }
    });
    let feature_out = slice::from_raw_parts_mut(feature_out, game_count * 4 * MOVE_FEATURES);
    parallel(games, feature_out, 4 * MOVE_FEATURES, threads as usize, |_, game, out| {
        out.fill(0.);
        if !game.done() {
            let heads = [Some(game.heads[0]), Some(game.heads[1])];
            for (direction, values) in move_features(&game.cells, game.to_move, &heads).iter().enumerate() {
                out[direction * MOVE_FEATURES..(direction + 1) * MOVE_FEATURES].copy_from_slice(values);
            }
        }
    });
    let targets_out = slice::from_raw_parts_mut(targets_out, game_count * 4);
    let legal_out = slice::from_raw_parts_mut(legal_out, game_count * 4);
    let to_move_out = slice::from_raw_parts_mut(to_move_out, game_count);
    for (i, game) in games.iter().enumerate() {
        let (targets, legal) = move_targets(&game.cells, game.heads[game.to_move]);
        for direction in 0..4 {
            targets_out[i * 4 + direction] = targets[direction];
            legal_out[i * 4 + direction] = u8::from(legal[direction] && !game.done());
        }
        to_move_out[i] = game.to_move as u8;
    }
}

/// One move per game for the player to move (0 UP, 1 RIGHT, 2 DOWN, 3 LEFT). Anything else crashes
/// the mover, as an invalid command does on CodinGame. Finished games ignore it.
/// # Safety
/// `moves` must hold one entry per game.
#[no_mangle]
pub unsafe extern "C" fn tron_step(batch: *mut Batch, moves: *const i32) {
    let games = &mut (&mut *batch).games;
    let moves = slice::from_raw_parts(moves, games.len());
    for (game, &direction) in games.iter_mut().zip(moves) {
        game.step(usize::try_from(direction).unwrap_or(usize::MAX));
    }
}

/// 1 for every finished game.
/// # Safety
/// `out` must hold one byte per game.
#[no_mangle]
pub unsafe extern "C" fn tron_done(batch: *const Batch, out: *mut u8) {
    let games = &(&*batch).games;
    let out = slice::from_raw_parts_mut(out, games.len());
    for (slot, game) in out.iter_mut().zip(games) {
        *slot = u8::from(game.done());
    }
}

/// Two scores per game: 1 for the winner, 0 for the loser, 0 0 while the game runs.
/// # Safety
/// `out` must hold two integers per game.
#[no_mangle]
pub unsafe extern "C" fn tron_scores(batch: *const Batch, out: *mut i32) {
    let games = &(&*batch).games;
    let out = slice::from_raw_parts_mut(out, games.len() * 2);
    for (i, game) in games.iter().enumerate() {
        for seat in 0..2 {
            out[i * 2 + seat] = i32::from(game.winner == Some(seat));
        }
    }
}
// ... tron_plies (the moves made so far in each game) follows the same pattern.

/// The move a hand-written bot (FLOOD, VORONOI or ALPHABETA with `max_depth` and `node_budget`)
/// would play for the player to move, in every game where `wanted` is 1 (-1 elsewhere and in
/// finished games). The games are split over `threads` threads.
/// # Safety
/// `wanted` and `out` must hold one entry per game.
#[no_mangle]
pub unsafe extern "C" fn tron_bot_moves(batch: *const Batch, bot: u32, max_depth: u32, node_budget: u64, threads: u32, wanted: *const u8, out: *mut i32) {
    let games = &(&*batch).games;
    let wanted = slice::from_raw_parts(wanted, games.len());
    let out = slice::from_raw_parts_mut(out, games.len());
    let options = Options { max_depth: max_depth as usize, node_budget };
    parallel(games, out, 1, threads as usize, |index, game, slot| {
        slot[0] = if wanted[index] == 0 || game.done() {
            -1
        } else {
            (match bot {
                FLOOD => flood_bot(game),
                VORONOI => voronoi_bot(game),
                _ => alphabeta_bot(game, options),
            }) as i32
        };
    });
}
// ... tron_voronoi_scores, tron_frame, tron_heads and tron_width follow the same pattern.
// ... tron_net_new, tron_net_free and tron_net_evaluate run the Rust network for the parity check.

"""Python side of the engine's C ABI (engine/src/ffi.rs): many light-cycle games stepped by one call.

Build the library first: cargo build --release --manifest-path engine/Cargo.toml
"""
import ctypes
from pathlib import Path

import numpy as np

LIBRARY = Path(__file__).resolve().parents[1]/'engine/target/release/libtron.so'
PLANES, HEIGHT, WIDTH, MOVE_FEATURES = 8, 20, 30, 2
CELLS = HEIGHT*WIDTH
NAMES = ['UP', 'RIGHT', 'DOWN', 'LEFT']
FLOOD, VORONOI, ALPHABETA = 0, 1, 2
# The page-5 bot: iterative deepening to 8 plies, at most 10,000 nodes a move (a stand-in for the clock).
ALPHABETA_DEPTH, ALPHABETA_BUDGET = 8, 10_000

U8, I32, F32 = ctypes.POINTER(ctypes.c_uint8), ctypes.POINTER(ctypes.c_int32), ctypes.POINTER(ctypes.c_float)


def load():
    lib = ctypes.CDLL(str(LIBRARY))
    signatures = {
        'tron_new': ([ctypes.c_uint32, ctypes.c_uint64], ctypes.c_void_p),
        'tron_free': ([ctypes.c_void_p], None),
        'tron_reset': ([ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint64], None),
        'tron_observe': ([ctypes.c_void_p, ctypes.c_uint32, U8, F32, I32, U8, U8], None),
        'tron_step': ([ctypes.c_void_p, I32], None),
        'tron_done': ([ctypes.c_void_p, U8], None),
        'tron_scores': ([ctypes.c_void_p, I32], None),
        'tron_plies': ([ctypes.c_void_p, I32], None),
        'tron_bot_moves': ([ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint32, ctypes.c_uint64, ctypes.c_uint32, U8, I32],
                           None),
        'tron_voronoi_scores': ([ctypes.c_void_p, ctypes.c_uint32, I32], None),
        'tron_frame': ([ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint32, ctypes.c_char_p, ctypes.c_uint32], ctypes.c_int32),
        'tron_heads': ([ctypes.c_void_p, I32], None),
        'tron_width': ([], ctypes.c_uint32),
        'tron_net_new': ([ctypes.c_uint32, ctypes.POINTER(ctypes.c_uint32), ctypes.c_uint32, ctypes.c_uint32,
                          ctypes.c_uint32, F32, ctypes.c_uint64], ctypes.c_void_p),
        'tron_net_free': ([ctypes.c_void_p], None),
        'tron_net_evaluate': ([ctypes.c_void_p, ctypes.c_uint32, U8, F32, I32, U8, F32, F32], None),
    }
    for name, (arguments, result) in signatures.items():
        function = getattr(lib, name)
        function.argtypes, function.restype = arguments, result
    assert lib.tron_width() == WIDTH, 'stale engine library'
    return lib


LIB = load()


def ptr(array, kind):
    """A C pointer to a numpy array's memory (the array must stay alive and contiguous)."""
    assert array.flags['C_CONTIGUOUS']
    return array.ctypes.data_as(kind)


class Batch:
    """`count` games in one Rust object. Game i starts from seed first_seed + i; `reset` starts another."""

    def __init__(self, count, first_seed=0, threads=4):
        self.count, self.threads = count, threads
        self.handle = LIB.tron_new(count, first_seed)
        # Rust writes observations straight into these arrays.
        self.planes = np.zeros((count, PLANES, HEIGHT, WIDTH), np.uint8)
        self.features = np.zeros((count, 4, MOVE_FEATURES), np.float32)
        self.targets = np.zeros((count, 4), np.int32)
        self.legal = np.zeros((count, 4), np.uint8)
        self.to_move = np.zeros(count, np.uint8)

    @classmethod
    def from_seeds(cls, seeds, threads=4):
        """One game per seed."""
        batch = cls(len(seeds), threads=threads)
        for index, seed in enumerate(seeds):
            batch.reset(index, int(seed))
        return batch

    def reset(self, index, seed):
        LIB.tron_reset(self.handle, index, seed)

    def observe(self):
        """For every game, what the player to move sees: planes, room and margin after each move, the
        cell each move enters, legal moves, and whose turn it is. Returns copies the caller may keep."""
        LIB.tron_observe(self.handle, self.threads, ptr(self.planes, U8), ptr(self.features, F32), ptr(self.targets, I32),
                         ptr(self.legal, U8), ptr(self.to_move, U8))
        return (self.planes.copy(), self.features.copy(), self.targets.copy(), self.legal.astype(bool),
                self.to_move.astype(np.int64))

    def step(self, moves):
        moves = np.ascontiguousarray(moves, dtype=np.int32)
        LIB.tron_step(self.handle, ptr(moves, I32))

    def done(self):
        out = np.zeros(self.count, np.uint8)
        LIB.tron_done(self.handle, ptr(out, U8))
        return out.astype(bool)

    def scores(self):
        """[count, 2]: 1 for the winner's seat, 0 for the loser's, 0 0 while running."""
        out = np.zeros((self.count, 2), np.int32)
        LIB.tron_scores(self.handle, ptr(out, I32))
        return out

    def plies(self):
        out = np.zeros(self.count, np.int32)
        LIB.tron_plies(self.handle, ptr(out, I32))
        return out

    def bot_moves(self, bot, wanted=None, depth=ALPHABETA_DEPTH, budget=ALPHABETA_BUDGET):
        """The move a hand-written bot (FLOOD, VORONOI, ALPHABETA) would make in the games where `wanted`
        is True (all games by default); -1 elsewhere."""
        wanted = np.ones(self.count, np.uint8) if wanted is None else np.ascontiguousarray(wanted, dtype=np.uint8)
        out = np.zeros(self.count, np.int32)
        LIB.tron_bot_moves(self.handle, bot, depth, budget, self.threads, ptr(wanted, U8), ptr(out, I32))
        return out
    # ... voronoi_scores, frame and heads follow the same pattern.

    def close(self):
        if self.handle:
            LIB.tron_free(self.handle)
            self.handle = None

    __del__ = close

# ... RustNet loads a PyTorch model's weights into engine/src/net.rs for the parity check.

"""The batched trainer (page 15): the same PPO loop as ppo.py, with the games stepped by the Rust engine
behind its C ABI (hundreds of games per call), the network on the GPU, and what a long run needs: a
checkpoint after every iteration, resume, Ctrl+C that finishes the iteration and saves, a status file,
and a strength curve against fixed opponents on seeds training never sees.

    python trainer.py --start ../runs/bc/clone.pt --out ../runs/ppo --minutes 100
    python trainer.py --start ../runs/bc/clone.pt --out ../runs/ppo --minutes 160 --resume   # 60 more
    python trainer.py ... --minutes 100000 --export    # until Ctrl+C, then export the champion
"""
import json
import signal
import subprocess
import sys
import time
from dataclasses import asdict
from pathlib import Path

import torch

from evaluate import evaluate
from ppo import COUNTERS, Config, arguments, save, train
from tron_env import Batch

CURVE_OPPONENTS = ('flood', 'voronoi', 'alphabeta', 'clone')


def rust_games(seeds):
    return Batch.from_seeds(list(seeds), threads=8)


def write_atomically(path, write):
    """Write to a temporary file, then rename it over the old one: a crash never leaves half a file."""
    temporary = path.with_suffix(path.suffix+'.tmp')
    write(temporary)
    temporary.replace(path)


def curve_point(model, state, args):
    """Strength of `model` against the yardsticks, appended to curve.jsonl with the games played so far."""
    clock = time.time()
    was_training = model.training
    results = evaluate(model.eval(), CURVE_OPPONENTS, args.curve_seeds, args.start)
    model.train(was_training)
    row = dict(iteration=state['iteration'], games=state['games'], decisions=state['decisions'],
               train_seconds=state['seconds'], evaluation_seconds=round(time.time()-clock, 1),
               **{name: {key: round(value, 4) if isinstance(value, float) else value for key, value in result.items()}
                  for name, result in results.items()})
    with open(Path(args.out)/'curve.jsonl', 'a') as log:
        log.write(json.dumps(row)+'\n')
    print('  curve:', ', '.join(f"{name} {result['rate']:.3f} [{result['low']:.3f}, {result['high']:.3f}]"
                                for name, result in results.items()),
          flush=True)
    return row


def main():
    parser = arguments(__doc__)
    parser.add_argument('--resume', action='store_true', help='continue from latest.pt in --out')
    parser.add_argument('--curve-every', type=int, default=12, help='iterations between strength measurements')
    parser.add_argument('--curve-seeds', type=int, default=200, help='evaluation seeds, each played in both seats')
    parser.add_argument('--export', action='store_true', help='when the run stops, export the champion (page 16)')
    parser.set_defaults(games=512, screen_seeds=64, confirm_seeds=256)
    args = parser.parse_args()
    config = Config(**{field: getattr(args, field) for field in asdict(Config())})
    out = Path(args.out)
    out.mkdir(parents=True, exist_ok=True)
    checkpoint = torch.load(out/'latest.pt', weights_only=False) if args.resume else None
    if not args.resume:
        (out/'config.json').write_text(json.dumps(dict(args=vars(args), engine='rust'), indent=1))

    # Ctrl+C asks the loop to stop after the current iteration; a second Ctrl+C stops at once.
    stop = {'requested': False}

    def request_stop(signum, frame):
        if stop['requested']:
            raise KeyboardInterrupt
        stop['requested'] = True
        print('\nstopping after this iteration (Ctrl+C again to quit now)', flush=True)
    signal.signal(signal.SIGINT, request_stop)

    started = time.time()

    def on_iteration(state):
        state['stop'] = stop['requested']
        if state['iteration'] % args.curve_every == 0:
            state['last_curve'] = curve_point(state['learner'], state, args)
        payload = dict(learner=state['learner'].state_dict(), champion=state['champion'].state_dict(),
                       optimizer=state['optimizer'].state_dict(), pool=state['pool'].entries,
                       state={key: state[key] for key in COUNTERS})
        write_atomically(out/'latest.pt', lambda path: torch.save(payload, path))
        status = dict(iteration=state['iteration'], games=state['games'], decisions=state['decisions'],
                      train_seconds=state['seconds'],
                      games_per_hour=round(state['games']/max(state['seconds'], 1)*3600),
                      champion_iteration=state['champion_iteration'], champion_games=state['champion_games'],
                      pool=[dict(iteration=entry['iteration'], games=entry['games'], points=entry['points'])
                            for entry in state['pool'].entries],
                      last_curve=state.get('last_curve'), session_seconds=round(time.time()-started, 1))
        write_atomically(out/'status.json', lambda path: path.write_text(json.dumps(status, indent=1)))

    if not args.resume:
        # The starting point of the curve: the cloned network before any self-play.
        from play import load
        curve_point(load(args.start, 'cuda' if torch.cuda.is_available() else 'cpu'),
                    dict(iteration=0, games=0, decisions=0, seconds=0.), args)
    state = train(args, config, rust_games, on_iteration, checkpoint)
    print(f"stopped at iteration {state['iteration']} after {state['games']:,} games; "
          f"champion from iteration {state['champion_iteration']} -> {out/'champion.pt'}", flush=True)
    save(state['champion'], out/'champion.pt')
    if args.export:
        exporter = Path(__file__).resolve().parents[1]/'export/export.py'
        subprocess.run([sys.executable, str(exporter), '--model', str(out/'champion.pt'), '--out', str(out/'export')],
                       check=True)


if __name__ == '__main__':
    main()

"""Games per hour of the same PPO iteration on the two engines (page 15's before and after): plain Python
games stepped one by one (tron.Games) against the Rust batch behind the C ABI (tron_env.Batch).

    python throughput.py --start ../runs/bc/clone.pt
"""
import argparse
import copy
import json
import time

import numpy as np
import torch

import tron
from ppo import Config, Pool, ppo_update, rollout
from play import load
from trainer import rust_games


def measure(engine, new_games, games, iterations, start, device):
    """Seconds per iteration (rollout and update) for `games` games per iteration."""
    config = Config(games=games, lr=2e-4, entropy=.005)
    learner = load(start, device).train()
    optimizer = torch.optim.Adam(learner.parameters(), lr=config.lr)
    pool = Pool(config.pool_size)
    pool.add(learner, 0)
    random = np.random.default_rng(0)
    generator = torch.Generator(device=device).manual_seed(0)
    rows = []
    for iteration in range(iterations):
        opponents = [(0, copy.deepcopy(learner).eval())]*config.pool_slots
        clock = time.time()
        batch, stats = rollout(learner, pool, opponents, config, random, 20_000_000+iteration*games, new_games, device,
                               generator)
        rolled = time.time()
        ppo_update(learner, optimizer, batch, config, device, torch.Generator().manual_seed(0))
        done = time.time()
        rows.append(dict(rollout=rolled-clock, update=done-rolled, decisions=stats['new_decisions']))
    seconds = sum(row['rollout']+row['update'] for row in rows)
    return dict(engine=engine, games_per_iteration=games, iterations=iterations,
                rollout_seconds=round(sum(row['rollout'] for row in rows)/iterations, 2),
                update_seconds=round(sum(row['update'] for row in rows)/iterations, 2),
                games_per_hour=round(games*iterations/seconds*3600))


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--start', required=True)
    parser.add_argument('--iterations', type=int, default=2)
    args = parser.parse_args()
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    results = [measure('python', tron.Games, 64, args.iterations, args.start, device),
               measure('rust', rust_games, 64, args.iterations, args.start, device),
               measure('rust', rust_games, 512, args.iterations, args.start, device)]
    for row in results:
        print(json.dumps(row), flush=True)


if __name__ == '__main__':
    main()

12Where trainers go wrong

A signature you didn't declare. ctypes assumes a C int return, so a 64-bit handle comes back cut in half, and the crash happens later, somewhere else.

A stale library. Python loads whatever .so is on disk. Rebuild after every engine change, and have the loader check a constant the library reports: the project's compares the board width.

A short or temporary buffer. The library writes past the end of a short array without any error, so size arrays from the library's own constants, and pass addresses with data_as: numpy warns that a bare .ctypes.data keeps no reference, so a pointer from a temporary array points at freed memory.^[7]

Keeping the observation array. The library overwrites it on every call. Copy what you keep; the project's observe() returns copies.

A checkpoint without the optimizer or the counters. Adam restarts its averages from zero, and the run replays training seeds it has already used.

Games per hour without saying what was counted. The project's batched run reads 63,054 games an hour for rollouts and updates before the memory change, and 47,428 over the whole run with gates. Quote the hardware, the load and what the count includes.

13What's next

Shipping a Network in 100,000 Characters takes the champion this trainer saved, the network from iteration 60, and turns it into one Rust source file under CodinGame's 100,000-character limit. It rounds the weights to a few bits, with a short fine-tune through the rounding, packs them into text, generates the bot, proves it plays the same moves as the PyTorch network turn by turn, and times it against Tron Battle's 100 ms limit.

The project so far, with this step's files in teal:

project/
├── engine/                    Rust: rules, bots, features, network, C interface
│   ├── Cargo.toml             builds an rlib and a cdylib (libtron.so)
│   └── src/
│       ├── board.rs           grid, distances, room, Voronoi territory
│       ├── game.rs            rules, seeded starts, CodinGame frames
│       ├── bots.rs            flood-fill and Voronoi bots (the teacher)
│       ├── alphabeta.rs       the alpha-beta bot (a yardstick)
│       ├── features.rs        what the network sees
│       ├── net.rs             the network's forward pass in Rust
│       ├── ffi.rs             the C interface Python calls (explained on this page)
│       └── lib.rs
├── train/
│   ├── tron_env.py            drives the engine from Python (explained on this page)
│   ├── model.py               the network in PyTorch
│   ├── parity.py              Rust and PyTorch choose the same moves
│   ├── record.py              records the teacher's decisions
│   ├── bc.py                  behavior cloning
│   ├── play.py                matches, both seats, Wilson intervals
│   ├── evaluate.py            a network against the yardsticks
│   ├── tron.py                the engine in plain Python
│   ├── ppo.py                 the single-process PPO trainer
│   ├── trainer.py             the batched trainer: checkpoints, resume, Ctrl+C, status
│   └── throughput.py          games per hour, Python engine against the Rust batch
├── checks/
│   ├── python_engine.py       the Python engine against the Rust engine, ply by ply
│   └── durability.sh          Ctrl+C and resume on a small run
├── data/                      recorded demonstrations
└── runs/
    ├── bc/                    the clone
    ├── ppo-single/            the single-process run
    └── ppo/                   the batched run
        ├── latest.pt          everything needed to resume, rewritten every iteration
        ├── champion.pt        the best network so far
        ├── learner.pt         the learner when the run stopped
        ├── status.json        progress, rewritten every iteration
        ├── metrics.jsonl      one line per iteration
        ├── curve.jsonl        strength against the yardsticks every 12 iterations
        └── config.json        every setting the run started with

14Sources

Quotes are from each source's own text. Numbers marked as the project's were measured with the code on this page and the series' earlier pages; numbers marked as the author's lab are first-hand records from one lab and aren't general results. Widget readouts are computed in the page from the models and records their captions name.

1. Build a Legend Bot running project, step 4, run on October 30, 2026 on one machine (Intel Core i9-11900KF, 8 cores and 16 threads; NVIDIA GeForce RTX 3080 Ti, 12 GB, shared with other training jobs). Games per hour (runs/logs/throughput.log), the single-process and batched runs (runs/ppo-single/, runs/ppo/ and runs/logs/ppo.log, with the restart at iteration 51 and the allocator warnings), the stop-and-resume test (checks/durability.sh), the CPU and GPU benchmark (checks/bench_devices.py) and the engine checks. The code is on this page and the series' earlier pages.
2. PyTorch 2.14 documentation, Automatic differentiation package (torch.autograd). The scope of autograd: automatic differentiation of scalar-valued functions.
3. Author's lab records, 2026 (private repository). The framework decision, protocol bots inside the simulator, their crash handling and the observation rule, docs/multi-game.md; the Spring Challenge 2022 crash loop and memory fix, AGENTS.md (October 17 and 18 entries); the Back Track King run's settings, timings and learning rates, experiments/runs/nn-main/config.json, metrics.jsonl and sessions.jsonl; the trainer's defaults, framework/rl/trainer.py; the October 22 throughput changes, experiments/performance-20261022.json; the length of the October 24 export, from the folder times in experiments/runs/nn-main/submissions/.
4. The Rustonomicon, Foreign Function Interface, sections "Calling Rust code from C" and "FFI and unwinding". extern "C", no_mangle, and a panic at a boundary that may not unwind aborting the process.
5. The Rust Reference, Linkage. The cdylib crate type.
6. Python documentation, ctypes: A foreign function library for Python (read for Python 3.15). argtypes, restype, the default int return type, and the warning about incorrect arguments.
7. NumPy documentation, numpy.ndarray.ctypes. data_as keeps a reference to the array; the bare data address doesn't.
8. PyTorch 2.14 documentation, torch.from_numpy. The tensor shares the array's memory.
9. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017. Proximal Policy Optimization Algorithms, arXiv:1707.06347. The clipped objective, and Algorithm 1: each iteration, N parallel actors collect T timesteps each, followed by K epochs of minibatch updates.
10. John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, Pieter Abbeel, 2016. High-Dimensional Continuous Control Using Generalized Advantage Estimation, ICLR 2016. Advantages blended with γ and λ.
11. Pduhard, 2026. Summer Challenge 2026, Back Track King: 3rd, README of his post-mortem repository, section "Training". 512 games at once in C++, half against past versions, and averaged checkpoints.
12. Matthew Inkawhich, PyTorch Tutorials, Saving and Loading Models, section "Saving & Loading a General Checkpoint for Inference and/or Resuming Training". A checkpoint for resuming needs the optimizer's state.
13. Python documentation, os.replace. A successful rename is atomic (a POSIX requirement).
14. Python documentation, signal, "Execution of Python signal handlers". A handler runs later, in the interpreter; a long computation in C runs to the end first.
15. systemd.service(5), Restart=. Which exits count as clean, and when on-failure restarts a service.
16. PyTorch 2.14 documentation, CUDA semantics, "Memory management". The caching allocator and empty_cache().
17. JOiy (gist hosted by tcordel), 2026, last updated October 10, 2026. JOiy's bot: Summer challenge 2026, section 5, "Training". Held-out maps for the gate, and his PPO settings.
18. CodinGame. Tron Battle, game statement, read on 2026-10-30. Battles of 2 to 4 cycles, with one opponent in the first league.