Self-Play Reinforcement Learning

01Learn from results

The clone from Behavior Cloning plays its teacher, the Voronoi bot, level: 196 wins in 400 games, an interval of .441 to .539.^[1] Training it longer won't help, because its loss only scores how closely it copies. To pass the teacher, the network needs a signal that says which games it won.

That signal is a reward, a number the learner receives after it acts: in the project, +1 for a win and −1 for a loss, once, after the game's last move. Sutton and Barto state the learner's goal as maximizing "the total amount of reward it receives."^[2]

One complete game is an episode. A decision's return is the reward that follows it, discounted: every step of delay multiplies a reward by the discount factor γ (gamma), a number just below 1.

In words: the next reward, plus γ times the one after, plus γ² times the one after that, to the end of the game. With one reward at the end, the return is the result times γ raised to the number of decisions still to come.

The discount decides how far back a result's credit reaches. Set it too low and the early moves learn almost nothing: putibuzu, 20th in Back Track King (Summer Challenge 2026), wrote that his bot learned the early game badly because "gamma was too small, so the terminal-only reward didn't propagate back to the beginning."^[3]

The hard part is that one result is shared by about 335 moves, counting both players'. The next two sections divide it among them.

02Make winning moves more likely

The network turns a position into a probability for each legal move (Neural Networks for Game Bots). After a game you know which moves it played and how the game ended, which suggests the simplest rule: make the moves of won games more likely and the moves of lost games less likely.

The network's rule for choosing moves is its policy, and training it this way is a policy gradient method; Ronald Williams named the family REINFORCE in 1992.^[4] For each move played, backpropagation finds the direction in weight space that makes the move more likely, and the update steps along it, scaled by the outcome. Because every move shares one result, the updates are noisy: a won game's weak moves get rewarded too.

The fix is to compare the result with what was expected. The network's value head predicts the result from a position, from −1 for a sure loss to +1 for a sure win; used to judge moves, it is called the critic.^[2] A move's advantage is how much better it turned out than the critic expected. Schulman et al. define it as the move's value minus the position's, which "measures whether or not the action is better or worse than the policy's default behavior."^[5]

In words: find the direction that makes the played move more likely, and step along it by the learning rate times the advantage. A positive advantage pushes the move up, a negative one pushes it down, and an advantage near zero leaves it almost alone.

Subtracting the critic's estimate leaves the average update unchanged, because any baseline that doesn't depend on the move averages out to zero,^[2] while it can make each update far less noisy (Reinforcement Learning shows why). When trainer code talks about advantages, it means this number.

03Give each move its share of the result

An advantage needs to know what happened after the move. You can wait for the end of the game and compare the result with the critic's estimate, or look just one move ahead. Each choice has a cost.

Waiting for the result is right on average but noisy, because everything that happened later lands on this move. Looking one move ahead uses the TD error: the reward right after the move, plus γ times the critic's estimate of the next position, minus its estimate before the move, which is calm but only as right as the critic. Generalized advantage estimation (GAE), from Schulman et al., blends the two with a second number, λ (lambda), between 0 and 1.^[5]

In words: a move's credit is its own TD error plus γλ times the next move's credit, computed backwards from the end of the game. At λ = 0 only the move's own surprise counts; at λ = 1 every later surprise counts in full, and together they add up to the result minus the critic's estimate.

Schulman et al. describe the trade: λ = 1 is right on average whatever the critic's accuracy, "but it has high variance due to the sum of terms", while λ = 0 "induces bias" unless the critic is exact, "but it typically has much lower variance."^[5] Values in between worked best in their experiments. The project uses λ = 0.95 with γ = 0.995; reCurse, who won CodinGame's Fall Challenge 2023 with a network trained by PPO, used γ = 0.99 and λ = 0.95.^[6]

04Reuse each batch, but not too far

Playing games costs far more than learning from them: in the project's single-process run, 32 games took 39.2 seconds to play and 1.8 seconds to learn from.^[1] So you want to use each batch more than once, but after a few updates the network isn't the one that played those games any more.

PPO, Proximal Policy Optimization, from Schulman et al. in 2017, makes several passes over each batch, called epochs, while keeping each update close to (proximal to) the policy that played the games.^[7] For every recorded move it computes a ratio: the move's probability under the network now, divided by its probability when the game was played. The ratio starts at 1. Once it has moved more than ε (epsilon) from 1 in the direction the advantage asks for, PPO stops rewarding the update for moving it further.

In words: score each move twice, with its ratio as it is and with the ratio held between 1 − ε and 1 + ε, and keep the lower score; training raises the average. In the paper's words, "we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse."^[7]

The paper suggested ε = 0.2, the best of three values on its robot-control tasks; its Atari runs used 0.1, shrunk to zero during training.^[7] The project uses 0.2, three epochs and minibatches of 2,048 decisions, and adds the critic's squared error at half weight and an entropy bonus of 0.005, which Mnih et al. found "improved exploration by discouraging premature convergence to suboptimal deterministic policies."^[8] The author's lab also ends a batch's epochs once the KL divergence between old and new move probabilities passes 0.05;^[9] the project doesn't, and its updates moved the policy by a KL divergence of about 0.001 to 0.002.^[1]

Published CodinGame bots use similar settings. JOiy, 13th in Back Track King (Legend), lists "512 envs, horizon 128, LR 7.5e-5, entropy 0.001, PPO epochs 3, 4 minibatches, clip 0.2".^[10] Pduhard, 3rd: "512 games run at once in C++, both seats are played by the current network".^[11]

A common question when you first read the code: what does optimizer.step() do? One PyTorch update is three calls: optimizer.zero_grad() clears the last update's slopes, loss.backward() computes the slope of the loss for every weight, and optimizer.step() moves every weight a step against its slope. Section 11 runs all three. The arrays they work on are tensors, arrays with any number of axes.

05Choose who the learner plays

In self-play the learner's opponent is itself, so the opposition gets stronger exactly as fast as the learner does. That's how TD-Gammon learned backgammon: a network that, in Gerald Tesauro's words, "trains itself to be an evaluation function for the game of backgammon by playing against itself and learning from the outcome."^[12] But the latest version of yourself is one opponent, and a narrow one.

Strategies can beat each other in a circle, as rock, paper and scissors do, and a learner that only trains to beat its last version can follow the circle forever. DeepMind's AlphaStar paper names the trap: self-play "may chase cycles (for example, where A defeats B, and B defeats C, but A loses to C) indefinitely without making progress." It also gives the classic fix. Fictitious self-play trains against "a uniform mixture of all previous policies", and in a two-player game where one side's gain is the other's loss, that mixture converges to a Nash equilibrium, a mix nothing beats on average.^[13]

Real trainers mix the two. OpenAI Five played "the latest policy against itself for 80% of games" and older versions in the rest, to "avoid strategy collapse".^[14] AlphaStar drew past opponents more often the more they beat the learner,^[13] and so did Pduhard in Back Track King: "Half the games are against the network itself; the other half against old versions of it, kept in a pool, and the ones that beat us are drawn more often."^[11] How big a pool you need depends on the game: in the games Czarnecki et al. studied, small populations kept cycling and larger ones converged.^[15]

The project uses the author's lab's default mix.^[9] A quarter of each batch's games are against the teacher, a bot that never changes; of the rest, half are the learner against itself and half against a snapshot from its pool. A game against itself gives the learner both seats, so those 37.5% of games supply about 54.5% of its decisions. Every 5 iterations the learner joins the pool, which keeps up to 32 snapshots, and each batch draws 4, favoring the ones that still beat it.

06Better against itself, worse against everyone else

A pool keeps the learner from going round in circles, but every opponent in it is still the learner's own past, and the learner can find a habit that beats all of them and loses to everything else. Tesauro warned about this in 1995: in a deterministic game a self-play learner "might develop some horrible strategy that nevertheless gives self-consistent results when implemented against itself".^[12]

Drift has caught a contest winner. reCurse trained his Fall Challenge 2023 network only against its current self; in the contest's last days he "started seeing weird behaviors and many more losses than I was used to", and an earlier checkpoint "started winning more consistently again." Next time, he wrote, he would look into "nice league systems using AlphaStar as inspiration."^[6] Training curves don't show drift either. JOiy: "A run can look healthy on every curve (stable entropy, a well-fit critic, regular promotions) and still be learning nothing that transfers to the ladder."^[10]

The guard is opponents that don't learn, used everywhere a number is produced:

1. Keep fixed bots in the training mix, as the project's quarter of games against its teacher does.
2. Measure every checkpoint against fixed bots, on seeds no training game uses.
3. Gate on fixed bots, and roll back a learner that keeps losing screens to its champion.
4. Check how the games end. In the lab's run, game lengths and tie-break wins showed the drift.

A common question: it shouldn't take long to beat a Silver bot, so what's the win rate against the original? Ask that of every checkpoint. A score against the pool can rise while the bot gets worse; the score against a fixed bot is the one that falls when it does.

When a run stalls, as Spring Challenge 2022's did in Gold, check the lab's five causes there: this drift; an action the cloned teacher never used (Behavior Cloning); greedy play (section 10); heroes deciding independently (Neural Networks for Game Bots); and no opponent in the style of the league's best (Case Study: Five Legends by Search).

07Project step 3: a self-play trainer in one process

python ../checks/python_engine.py 30                     # Python engine = Rust engine, turn by turn
python ppo.py --start ../runs/bc/clone.pt --out ../runs/ppo-single --minutes 40 \
    --games 32 --gate-every 5 --screen-seeds 16 --confirm-seeds 64
python evaluate.py ../runs/ppo-single/champion.pt --seeds 200   # the four yardsticks

"""Self-play PPO in one Python process (page 14).

Each iteration plays a batch of complete games: the learner against itself, against past snapshots of
itself (the pool) and against the Voronoi bot (the teacher). Every learner decision gets an advantage
from GAE, then a few epochs of the clipped PPO update change the weights. Every few iterations a
champion gate plays the learner against the best weights so far on held-out seeds: a screen first,
then a confirmation on fresh seeds.

The engine is plain Python (tron.Games) unless a trainer passes another one with the same calls;
trainer.py passes the batched Rust engine.

    python ppo.py --start ../runs/bc/clone.pt --out ../runs/ppo-single --minutes 30
"""
import argparse
import copy
import json
import time
from dataclasses import asdict, dataclass
from pathlib import Path

import numpy as np
import torch

import tron
from model import choose, entropy, policy, tf32_convolutions
from play import load, match, tensors

TRAINING_SEEDS = 10_000_000   # each training game takes the next seed from here
SCREEN_SEEDS = 3_000_000      # the gate's screen: the same held-out seeds every time
CONFIRM_SEEDS = 3_100_000     # the gate's confirmation: fresh seeds every time
TEACHER = 1                   # the Voronoi bot's number in both engines


@dataclass
class Config:
    games: int = 64             # complete games per iteration
    teacher_share: float = .25  # games against the Voronoi bot
    self_share: float = .5      # of the other games: the learner in both seats (else a pool snapshot)
    pool_slots: int = 4         # snapshots drawn from the pool for one iteration
    pool_size: int = 32
    snapshot_every: int = 5     # iterations between snapshots
    gamma: float = .995         # discount per decision
    lam: float = .95            # GAE: 0 trusts the critic's next value, 1 waits for the result
    clip: float = .2
    epochs: int = 3
    minibatch: int = 2048       # decisions per gradient step
    microbatch: int = 512       # decisions per forward/backward pass inside a step (GPU memory)
    lr: float = 2e-4
    entropy: float = .005       # bonus for keeping some doubt between moves (exploration)
    value_coef: float = .5
    max_grad_norm: float = 1.
    gate_every: int = 10
    screen_seeds: int = 32      # played in both seats
    screen_pass: float = .55
    confirm_seeds: int = 128


class Pool:
    """Past snapshots of the learner. A snapshot the learner still loses to is drawn more often:
    weight (1 - rate)^2 + 0.02, where rate = (points + 2) / (games + 4) is the learner's score against it,
    pulled toward 0.5 while there are few games."""

    def __init__(self, size):
        self.size, self.entries = size, []

    def add(self, model, iteration):
        weights = {name: tensor.detach().cpu().clone() for name, tensor in model.state_dict().items()}
        self.entries.append(dict(weights=weights, iteration=iteration, points=0., games=0))
        if len(self.entries) > self.size:
            del self.entries[1]          # keep the first snapshot; drop the oldest after it

    def rate(self, entry):
        return (entry['points']+2)/(entry['games']+4)

    def sample(self, count, random):
        weights = np.array([(1-self.rate(entry))**2+.02 for entry in self.entries])
        return random.choice(len(self.entries), size=count, p=weights/weights.sum())


def gae(values, reward, gamma, lam):
    """Advantages and value targets for one player's decisions in one game. The only reward is the
    result (+1 win, -1 loss) after the last decision."""
    advantages = np.zeros(len(values), np.float32)
    running = 0.
    for step in reversed(range(len(values))):
        last = step == len(values)-1
        next_value = 0. if last else values[step+1]   # nothing follows the end of the game
        delta = (reward if last else 0.)+gamma*next_value-values[step]   # TD error: better or worse than expected
        running = delta+gamma*lam*running
        advantages[step] = running
    return advantages, advantages+values


@torch.no_grad()
def act(model, planes, features, targets, legal, device, generator):
    """Sampled moves (temperature 1), their log-probabilities, and the critic's values."""
    inputs = tensors(device, planes, features, targets, legal)
    logits, value = model(*inputs)
    move = choose(logits, inputs[3], generator=generator)
    log_probs = torch.log_softmax(logits, 1)
    return move.cpu().numpy(), log_probs.gather(1, move[:, None]).squeeze(1).cpu().numpy(), value.cpu().numpy()


def rollout(learner, pool, opponents, config, random, first_seed, new_games, device, generator):
    """Plays config.games complete games and returns the learner's decisions with advantages."""
    count = config.games
    games = new_games(range(first_seed, first_seed+count))
    # Who the learner plays in each game: -2 the teacher, -1 itself, k >= 0 the k-th drawn snapshot.
    kind = np.where(random.random(count) < config.teacher_share, -2,
                    np.where(random.random(count) < config.self_share, -1,
                             random.integers(len(opponents), size=count)))
    learner_seat = random.integers(2, size=count)
    log = []                    # per step: the learner decisions made in that step
    while not (done := games.done()).all():
        planes, features, targets, legal, to_move = games.observe()
        moves = np.zeros(count, np.int64)
        has_move = legal.any(1)
        learner_turn = ~done & ((kind == -1) | (to_move == learner_seat))
        # A player with no legal move crashes whatever it plays: no decision is recorded.
        record = learner_turn & has_move
        if record.any():
            move, logp, value = act(learner, planes[record], features[record], targets[record], legal[record], device,
                                    generator)
            moves[record] = move
            log.append(dict(game=np.nonzero(record)[0], seat=to_move[record], planes=planes[record],
                            features=features[record], targets=targets[record], legal=legal[record], move=move,
                            logp=logp, value=value))
        teacher_turn = ~done & ~learner_turn & (kind == -2)
        if teacher_turn.any():
            moves[teacher_turn] = games.bot_moves(TEACHER, wanted=teacher_turn)[teacher_turn]
        for slot, (_, model) in enumerate(opponents):
            turn = ~done & ~learner_turn & (kind == slot) & has_move
            if turn.any():
                moves[turn] = act(model, planes[turn], features[turn], targets[turn], legal[turn], device, generator)[0]
        games.step(moves)
    scores, plies = games.scores(), games.plies()
    games.close()
    for game in np.nonzero(kind >= 0)[0]:
        entry = pool.entries[opponents[kind[game]][0]]
        entry['points'] += float(scores[game, learner_seat[game]])
        entry['games'] += 1
    # One trajectory per (game, seat): that player's decisions in the order they were made.
    batch = {key: np.concatenate([step[key] for step in log]) for key in log[0]}
    trajectory = batch['game']*2+batch['seat']
    order = np.argsort(trajectory, kind='stable')
    batch = {key: value[order] for key, value in batch.items()}
    trajectory = trajectory[order]
    starts = np.flatnonzero(np.r_[True, trajectory[1:] != trajectory[:-1]])
    ends = np.r_[starts[1:], len(trajectory)]
    batch['advantage'] = np.zeros(len(trajectory), np.float32)
    batch['return'] = np.zeros(len(trajectory), np.float32)
    for begin, end in zip(starts, ends):
        won = scores[batch['game'][begin], batch['seat'][begin]] == 1
        batch['advantage'][begin:end], batch['return'][begin:end] = gae(
            batch['value'][begin:end], 1. if won else -1., config.gamma, config.lam)
    teacher_games = kind == -2
    stats = dict(new_games=count, new_decisions=len(trajectory), plies=round(float(plies.mean()), 1),
                 teacher_games=int(teacher_games.sum()),
                 teacher_score=float(scores[teacher_games, learner_seat[teacher_games]].mean())
                 if teacher_games.any() else None)
    del batch['game'], batch['seat']
    return batch, stats


def unpack_planes(packed):
    """Planes packed 8 per byte (np.packbits) back to 0/1 bytes [n, 8, 20, 30], on the GPU."""
    bits = (packed[:, :, None] >> torch.arange(7, -1, -1, device=packed.device)) & 1
    return bits.flatten(1).view(-1, tron.PLANES, tron.HEIGHT, tron.WIDTH)


def ppo_update(model, optimizer, batch, config, device, generator):
    """config.epochs passes over the batch in random minibatches of the clipped PPO loss. Each
    minibatch is one optimizer step; its gradient is summed over micro-batches to save GPU memory."""
    # The whole batch goes to the GPU once, planes packed 8 per byte; minibatches are cut out there.
    batch = dict(batch, planes=np.packbits(batch['planes'].reshape(len(batch['planes']), -1), axis=1))
    tensors = {key: torch.from_numpy(value).to(device) for key, value in batch.items()}
    size = len(tensors['move'])
    totals, updates = np.zeros(5), 0
    for _ in range(config.epochs):
        order = torch.randperm(size, generator=generator).to(device)
        for start in range(0, size, config.minibatch):
            index = order[start:start+config.minibatch]
            # Advantages are normalised over the whole minibatch, as if it were one pass.
            mean, std = tensors['advantage'][index].mean(), tensors['advantage'][index].std()
            optimizer.zero_grad()
            step = np.zeros(5)
            for chunk in index.split(config.microbatch):
                sub = {key: value[chunk] for key, value in tensors.items()}
                with tf32_convolutions():
                    log_probs, value = policy(model, unpack_planes(sub['planes']), sub['features'],
                                              sub['targets'].long(), sub['legal'])
                new_logp = log_probs.gather(1, sub['move'].long()[:, None]).squeeze(1)
                ratio = (new_logp-sub['logp']).exp()        # pi_new(move) / pi_old(move)
                advantage = (sub['advantage']-mean)/(std+1e-8)
                # Clipped surrogate: no reward for pushing the ratio past 1 +/- clip in the advantage's direction.
                policy_loss = -torch.min(ratio*advantage, ratio.clamp(1-config.clip, 1+config.clip)*advantage).mean()
                value_loss = (value-sub['return']).square().mean()
                bonus = entropy(log_probs, sub['legal']).mean()
                loss = policy_loss+config.value_coef*value_loss-config.entropy*bonus
                share = len(chunk)/len(index)               # this chunk's part of the minibatch mean
                with tf32_convolutions():
                    (share*loss).backward()
                with torch.no_grad():
                    approx_kl = (sub['logp']-new_logp).mean()
                    clipped = ((ratio-1).abs() > config.clip).float().mean()
                step += share*np.array([policy_loss.item(), value_loss.item(), bonus.item(), approx_kl.item(),
                                        clipped.item()])
            torch.nn.utils.clip_grad_norm_(model.parameters(), config.max_grad_norm)
            optimizer.step()
            totals += step
            updates += 1
    del tensors
    if device == 'cuda':
        torch.cuda.empty_cache()        # hand the cached memory back to the other jobs on the GPU
    names = ('policy_loss', 'value_loss', 'entropy', 'approx_kl', 'clip_fraction')
    return {name: round(float(total), 5) for name, total in zip(names, totals/updates)}


def gate(learner, champion, confirmations, config, new_games):
    """Screen the learner against the champion on the same held-out seeds every time; if it scores at
    least config.screen_pass, confirm on fresh seeds and promote only if the 95% interval's lower end
    is above 0.5. Both play greedily."""
    learner.eval()
    screen = match(learner, champion, SCREEN_SEEDS, config.screen_seeds, new_games=new_games)
    record = dict(screen=screen, promoted=False)
    if screen['rate'] >= config.screen_pass:
        first = CONFIRM_SEEDS+confirmations*config.confirm_seeds
        confirm = match(learner, champion, first, config.confirm_seeds, new_games=new_games)
        record.update(confirm=confirm, promoted=confirm['low'] > .5)
    learner.train()
    return record


COUNTERS = ('iteration', 'games', 'decisions', 'confirmations', 'champion_iteration', 'champion_games', 'next_seed',
            'seconds')


def train(args, config, new_games, on_iteration=None, checkpoint=None):
    """The training loop shared by both engines. on_iteration(state) runs after every iteration
    (trainer.py uses it for checkpoints, the status file and the strength curve); `checkpoint`
    resumes a run that trainer.py saved."""
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    resumed = checkpoint['state']['iteration'] if checkpoint else 0
    torch.manual_seed(args.seed+resumed)
    generator = torch.Generator(device=device).manual_seed(args.seed+resumed)
    cpu_generator = torch.Generator().manual_seed(args.seed+resumed)
    random = np.random.default_rng([args.seed, resumed])
    out = Path(args.out)
    out.mkdir(parents=True, exist_ok=True)
    learner = load(args.start, device).train()
    champion = copy.deepcopy(learner).eval()
    optimizer = torch.optim.Adam(learner.parameters(), lr=config.lr)
    pool = Pool(config.pool_size)
    pool.add(learner, 0)
    state = dict(iteration=0, games=0, decisions=0, confirmations=0, champion_iteration=0, champion_games=0,
                 seconds=0., next_seed=TRAINING_SEEDS, out=out, learner=learner, champion=champion,
                 optimizer=optimizer, pool=pool, stop=False)
    if checkpoint:
        learner.load_state_dict(checkpoint['learner'])
        champion.load_state_dict(checkpoint['champion'])
        optimizer.load_state_dict(checkpoint['optimizer'])
        pool.entries = checkpoint['pool']
        state.update({key: checkpoint['state'][key] for key in COUNTERS})
    clock = time.time()-state['seconds']
    while not state['stop'] and time.time()-clock < args.minutes*60:
        state['iteration'] += 1
        iteration = state['iteration']
        # Draw this iteration's pool opponents (each a frozen copy of an old learner).
        opponents = []
        for index in pool.sample(config.pool_slots, random):
            model = copy.deepcopy(learner).eval()
            model.load_state_dict(pool.entries[index]['weights'])
            opponents.append((index, model))
        started = time.time()
        batch, stats = rollout(learner, pool, opponents, config, random, state['next_seed'], new_games, device,
                               generator)
        state['next_seed'] += config.games
        rolled = time.time()
        losses = ppo_update(learner, optimizer, batch, config, device, cpu_generator)
        state['games'] += stats['new_games']
        state['decisions'] += stats['new_decisions']
        row = dict(iteration=iteration, games=state['games'], decisions=state['decisions'], **stats, **losses,
                   rollout_seconds=round(rolled-started, 2), update_seconds=round(time.time()-rolled, 2))
        if iteration % config.snapshot_every == 0:
            pool.add(learner, iteration)
        if iteration % config.gate_every == 0:
            record = gate(learner, champion, state['confirmations'], config, new_games)
            state['confirmations'] += 'confirm' in record
            if record['promoted']:
                champion.load_state_dict(learner.state_dict())
                state['champion_iteration'], state['champion_games'] = iteration, state['games']
            row['gate'] = record
        state['seconds'] = round(time.time()-clock, 1)
        row.update(seconds=state['seconds'], champion_iteration=state['champion_iteration'])
        with open(out/'metrics.jsonl', 'a') as log:
            log.write(json.dumps(row)+'\n')
        print(json.dumps({key: value for key, value in row.items() if key != 'gate'}), flush=True)
        if 'gate' in row:
            print('  gate:', json.dumps(row['gate']), flush=True)
        if on_iteration:
            on_iteration(state)
    save(learner, out/'learner.pt')
    save(champion, out/'champion.pt')
    return state


def save(model, path):
    torch.save({'shape': model.shape, 'weights': {name: tensor.cpu() for name, tensor in model.state_dict().items()}},
               path)


def arguments(description=__doc__):
    parser = argparse.ArgumentParser(description=description, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('--start', required=True, help='the behavior-cloned network')
    parser.add_argument('--out', required=True)
    parser.add_argument('--minutes', type=float, default=30)
    parser.add_argument('--seed', type=int, default=1)
    for field, value in asdict(Config()).items():
        parser.add_argument('--'+field.replace('_', '-'), type=type(value), default=value)
    return parser


def main():
    args = arguments().parse_args()
    config = Config(**{field: getattr(args, field) for field in asdict(Config())})
    (Path(args.out)).mkdir(parents=True, exist_ok=True)
    (Path(args.out)/'config.json').write_text(json.dumps(dict(args=vars(args), engine='python'), indent=1))
    train(args, config, tron.Games)


if __name__ == '__main__':
    main()

"""Light cycles in plain Python with numpy: the same rules, seeds, Voronoi bot and planes as the Rust
engine (engine/src), so the single-process trainer can play its games without the C interface.
checks/python_engine.py replays games in both and compares every plane and every move."""
from collections import deque

import numpy as np

WIDTH, HEIGHT = 30, 20
CELLS = WIDTH*HEIGHT
UP, RIGHT, DOWN, LEFT = range(4)
UNREACHED = 1 << 20
PLANES = 8
MASK64 = (1 << 64)-1


def neighbor(cell, direction):
    """The cell one step away, or None off the board."""
    x, y = cell % WIDTH, cell//WIDTH
    if direction == UP:
        return cell-WIDTH if y > 0 else None
    if direction == RIGHT:
        return cell+1 if x+1 < WIDTH else None
    if direction == DOWN:
        return cell+WIDTH if y+1 < HEIGHT else None
    return cell-1 if x > 0 else None


# The on-board neighbours of every cell, computed once.
NEIGHBORS = [[next_cell for next_cell in (neighbor(cell, direction) for direction in range(4)) if next_cell is not None]
             for cell in range(CELLS)]


class SplitMix64:
    def __init__(self, seed):
        self.state = seed & MASK64

    def next(self):
        self.state = (self.state+0x9E3779B97F4A7C15) & MASK64
        mixed = self.state
        mixed = ((mixed ^ (mixed >> 30))*0xBF58476D1CE4E5B9) & MASK64
        mixed = ((mixed ^ (mixed >> 27))*0x94D049BB133111EB) & MASK64
        return mixed ^ (mixed >> 31)

    def below(self, bound):
        return self.next() % bound


class Game:
    def __init__(self, seed):
        random = SplitMix64(seed)
        first = random.below(CELLS)
        second = random.below(CELLS)
        while second == first:
            second = random.below(CELLS)
        self.cells = [0]*CELLS            # 0 free, p + 1 for player p's trail
        self.cells[first], self.cells[second] = 1, 2
        self.heads = [first, second]
        self.headings = [RIGHT if head % WIDTH < WIDTH//2 else LEFT for head in self.heads]
        self.to_move, self.plies, self.winner = 0, 0, None

    def legal(self, player):
        return [direction for direction in (UP, RIGHT, DOWN, LEFT)
                if (target := neighbor(self.heads[player], direction)) is not None and self.cells[target] == 0]

    def step(self, direction):
        player = self.to_move
        target = neighbor(self.heads[player], direction)
        if target is None or self.cells[target]:
            self.winner = 1-player             # a crash: the other player wins
        else:
            self.cells[target] = player+1
            self.heads[player], self.headings[player] = target, direction
        self.plies += 1
        self.to_move = 1-player


def distances(cells, start):
    """Breadth-first distances from `start` over free cells."""
    dist = [UNREACHED]*CELLS
    dist[start] = 0
    queue = deque([start])
    while queue:
        cell = queue.popleft()
        for next_cell in NEIGHBORS[cell]:
            if cells[next_cell] == 0 and dist[next_cell] == UNREACHED:
                dist[next_cell] = dist[cell]+1
                queue.append(next_cell)
    return dist


def territory(cells, heads):
    """Free cells each head reaches strictly first, and whether no free cell is reachable by both."""
    first, second = np.array(distances(cells, heads[0])), np.array(distances(cells, heads[1]))
    free = np.array(cells) == 0
    counts = [int((free & (first < second)).sum()), int((free & (second < first)).sum())]
    shared = (free & (first < UNREACHED) & (second < UNREACHED)).any()
    return counts, not shared


def flood_count(cells, head):
    return sum(distance < UNREACHED for distance in distances(cells, head))-1


def voronoi_bot(game):
    """The page-4 bot: best (Voronoi margin * 1000 + room) after the move; straight, right, left, back."""
    me, heading = game.to_move, game.headings[game.to_move]
    best, best_score = heading, None
    for direction in (heading, (heading+1) % 4, (heading+3) % 4, (heading+2) % 4):
        target = neighbor(game.heads[me], direction)
        if target is None or game.cells[target]:
            continue
        cells = game.cells.copy()
        cells[target] = me+1
        heads = game.heads.copy()
        heads[me] = target
        counts, _ = territory(cells, heads)
        score = (counts[me]-counts[1-me])*1000+flood_count(cells, target)
        if best_score is None or score > best_score:
            best, best_score = direction, score
    return best


def move_features(game):
    """Room and Voronoi margin after each move, divided by 600 (zeros for illegal moves)."""
    me = game.to_move
    out = np.zeros((4, 2), np.float32)
    for direction in range(4):
        target = neighbor(game.heads[me], direction)
        if target is None or game.cells[target]:
            continue
        cells = game.cells.copy()
        cells[target] = me+1
        heads = game.heads.copy()
        heads[me] = target
        counts, _ = territory(cells, heads)
        out[direction] = np.float32(flood_count(cells, target))/np.float32(CELLS), \
            np.float32(counts[me]-counts[1-me])/np.float32(CELLS)
    return out


def observe(game):
    """What the player to move sees: planes [8, 20, 30] (0/1), room and margin after each move, the
    cell each move enters (-1 off the board) and which moves are legal. Same as engine/src/features.rs."""
    me = game.to_move
    cells = np.array(game.cells)
    mine = np.array(distances(game.cells, game.heads[me]))
    theirs = np.array(distances(game.cells, game.heads[1-me]))
    free = cells == 0
    planes = np.zeros((PLANES, CELLS), np.uint8)
    planes[0] = 1
    planes[1] = cells == me+1
    planes[2] = ~free & (cells != me+1)
    planes[3, game.heads[me]] = 1
    planes[4, game.heads[1-me]] = 1
    planes[5] = free & (mine < theirs)
    planes[6] = free & (theirs < mine)
    planes[7] = free & (mine < UNREACHED)
    targets = [neighbor(game.heads[me], direction) for direction in range(4)]
    legal = [target is not None and game.cells[target] == 0 for target in targets]
    return (planes.reshape(PLANES, HEIGHT, WIDTH), move_features(game),
            np.array([-1 if target is None else target for target in targets]), np.array(legal))


class Games:
    """Many Python games behind the same calls as the Rust batch (tron_env.Batch), so the trainer can
    run on either engine."""

    def __init__(self, seeds):
        self.games = [Game(seed) for seed in seeds]
        self.count = len(self.games)

    def observe(self):
        planes = np.zeros((self.count, PLANES, HEIGHT, WIDTH), np.uint8)
        features = np.zeros((self.count, 4, 2), np.float32)
        targets, legal = np.full((self.count, 4), -1), np.zeros((self.count, 4), bool)
        for i, game in enumerate(self.games):
            if game.winner is None:
                planes[i], features[i], targets[i], legal[i] = observe(game)
        return planes, features, targets, legal, np.array([game.to_move for game in self.games])

    def step(self, moves):
        for game, move in zip(self.games, moves):
            if game.winner is None:
                game.step(int(move))

    def done(self):
        return np.array([game.winner is not None for game in self.games])

    def scores(self):
        out = np.zeros((self.count, 2), np.int32)
        for i, game in enumerate(self.games):
            if game.winner is not None:
                out[i, game.winner] = 1
        return out

    def plies(self):
        return np.array([game.plies for game in self.games])

    def bot_moves(self, bot, wanted):
        """Only the Voronoi bot (the teacher) lives in the Python engine."""
        return np.array([voronoi_bot(game) if wants and game.winner is None else -1
                         for game, wants in zip(self.games, wanted)])

    def close(self):
        pass

"""The pure-Python engine (train/tron.py) against the Rust engine (through the C ABI): same start cells,
same planes and move features (bit for bit), same legal moves and the same Voronoi-bot move at every
ply of games that mix the Voronoi bot with random legal moves.

    python checks/python_engine.py [games]
"""
import sys
from pathlib import Path

import numpy as np

sys.path.insert(0, str(Path(__file__).resolve().parents[1]/'train'))
import tron                                    # noqa: E402
from tron_env import VORONOI, Batch            # noqa: E402

games = int(sys.argv[1]) if len(sys.argv) > 1 else 50
random = np.random.default_rng(0)
checked, mismatches = 0, 0
for k in range(games):
    seed = 8_000_000+k
    python_game, rust = tron.Game(seed), Batch(1, threads=1)
    rust.reset(0, seed)
    while python_game.winner is None:
        planes, features, targets, legal, _ = rust.observe()
        p_planes, p_features, p_targets, p_legal = tron.observe(python_game)
        teacher = tron.voronoi_bot(python_game)
        same = (np.array_equal(planes[0], p_planes) and np.array_equal(features[0], p_features)
                and np.array_equal(targets[0], p_targets) and np.array_equal(legal[0], p_legal)
                and teacher == rust.bot_moves(VORONOI)[0])
        checked += 1
        mismatches += not same
        options = python_game.legal(python_game.to_move)
        # Mostly the Voronoi bot, sometimes a random legal move, so positions vary.
        move = int(random.choice(options)) if options and random.random() < .2 else teacher
        python_game.step(move)
        rust.step(np.array([move]))
    assert rust.done()[0] and rust.scores()[0, python_game.winner] == 1
print(f'{games} games, {checked} positions, {mismatches} mismatches')
sys.exit(1 if mismatches else 0)

Building the Trainer runs the same train() on the Rust engine with 512 games per iteration: 79,360 games in 100 minutes. Every 12 iterations it measured the learner, playing its most likely move, against the teacher, the clone and the alpha-beta bot on the evaluation seeds. That run gives the project's strength curve:^[1]

Against the alpha-beta bot the learner climbed from the clone's .300 to about .5 and stopped. On 1,000 fresh games, both seats, on seeds no run had touched:^[1]

The network never passed the alpha-beta bot: it is level at best, and both intervals contain .5. Self-play took it from the clone's .295 to about .5 against the search and no further. When to Use Machine Learning gave the reason to expect it: light cycles offer three moves a turn and are decided by forcing tactics, where a search you wrote is hard to beat. In the author's lab, too, the Tron Battle bot that reached Legend was an alpha-beta search with no network. Networks Inside Search puts the network inside a search.

08How much self-training

A common worry: people say self-play needs millions of games, your trainer counts iterations, and a restart just threw away 100 of them. How much is that? It depends on what an iteration holds.

In the PPO paper an iteration is one batch: "each of N (parallel) actors collect T timesteps of data", then the network trains on it.^[7] The project's iterations are 32 or 512 complete games. The author's lab's Back Track King trainer advanced 512 games by 32 turns each; its games lasted 55 to 60 turns, so 100 iterations were about 27,000 finished games and 21 minutes.^[9]

The project's answer to how much self-training it needed:^[1]

1. The clone: 3,000 recorded games and under five minutes of training.
2. Passing the teacher: a few hundred games; .623 against it after 640.
3. Flattening: from 12,288 games to 79,360, .667 to .723 against the teacher and .672 to .700 against the clone.
4. The gate agreed: promotions at 5,120, 10,240 and 30,720 games, and none in the last 48,640.
5. The alpha-beta bot: not passed; .43 by 12,288 games, .46 by 30,720, then .45 to .51.

From 35,840 games on, the screens sat at .48 to .56: learner and champion were about equally strong, and 128 games can't tell a few points apart. The one confirmation, at 76,800 games, had a lower bound of .492. A gate that has stopped promoting while the fixed yardsticks stay flat is what a plateau looks like.

The whole batched project run was 79,360 games, 8.6% of that. Game counts don't carry over from one game to another, and neither does the look of a plateau. The lab's gate never stopped promoting (101 promotions, the last at iteration 73,800 of 74,335),^[9] because a champion only has to beat its own past versions in local games; its plateau showed only on the live ladder (Case Study: Back Track King follows the lab's run to its end).

A common question near the top of a ladder: the bot is a few places from first, so is there anything to tweak? By then the gains left can be smaller than the noise of one live result. After the lab's network reached the top few places of Back Track King's Legend league (permanent ladder, October 25, 2026), reward variants, a league of frozen opponents, a search, distillation and Q-learning were all tried, and none scored above the noise band of identical resubmissions.^[9]

09A more frequent signal, the same goal

A common question once self-play runs: how do you get a learning signal more often than once per game? Reward shaping pays some reward during the game. Paid carelessly, it changes what the learner optimizes.

Ng, Harada and Russell recount Randløv and Alstrøm's simulated bicycle, rewarded for progress toward a goal: "The agent learned to ride in tiny circles near the start state because no penalty was incurred for riding away from the goal." They also proved which bonuses are safe. Give every position a score Φ (phi), its potential, and pay γ × Φ(after) − Φ(before) for each move. Over a game the payments cancel down to an amount fixed by the starting position, so the best policy can't change; being potential-based is "a necessary and sufficient condition for it to guarantee consistency with the optimal policy".^[19]

In words: pay the learner for raising the potential and charge it for lowering it, with the potential set to 0 when the game ends. Everything paid along the way comes back out, so only the result decides which way of playing is best.

That collapse is reward hacking: the learner maximizes the reward you wrote, which isn't the game you meant, with solutions "valid in some literal sense" that "don't meet the designer's intent".^[20] Krakovna et al. describe a boat-racing agent paid for hitting green blocks that learned to circle and hit the same blocks again and again, because "shaping rewards can change the optimal policy if they are not potential-based".^[21]

The top players of Back Track King (Summer Challenge 2026) disagree about shaping. Indrill, 1st with deep Q-learning, added each turn's change in the projected final score gap between the players to the result, to "have a less delayed reward".^[22] Pduhard, 3rd, paid each turn's income gap and ±30 for the result, and lists "Two attempts at shaping the reward" among what didn't work.^[11] JOiy, 13th, found that per-turn shaping "teaches the network to optimise the proxy", and his best bots trained on a reward paid only at the end.^[10] delineate, who won Spring Challenge 2026 (Troll Farm), used heavy shaping to teach his network the basic tasks; once it chose its own targets, "it would find ways to game the reward shaping", so his final stages trained on the endgame score difference with no shaping.^[23] Treat shaping as a tool with a known failure mode.

The project uses no shaping. If you add a potential, build it from inputs the network already reads, such as the room and Voronoi margin it gets for every move.

10Greedy or sampled

During training the learner samples its moves, drawing each at random in proportion to its probability, so it keeps trying moves it rates a little lower. When the bot plays for real you choose: the most likely move every turn, greedy play, or keep sampling.

Between the two sits the temperature τ (tau) from Neural Networks for Game Bots: divide every logit (the network's raw score for a move) by τ before the softmax turns the scores into probabilities, so τ near 0 is greedy play and large τ is close to random. A greedy policy can loop: where two moves tie, or nearly tie, and each leads back to the other, it goes back and forth forever. Sampling breaks the tie by chance.

The project trains at τ = 1 and plays greedily, and here greedy play is fine: the champion scores .693 [.646, .736] against its teacher greedy and .655 [.607, .700] sampling at τ = 1 on the same seeds, a difference inside both intervals.^[1] Its policy is sharp: the average entropy of its move probabilities stayed near 0.1, where three equally likely moves would give 1.1. Two contest winners played greedily: reCurse's policy "is played with the action having the highest probability", and Indrill's "selection is a simple max. No randomness".^[6][22] Measure both before you submit, and if sampling wins, train at the temperature you will play at.

11GAE and the clipped loss in code

These complete programs compute GAE and PPO's clipped loss on the worked examples of sections 03 and 04 and check every number. The C++ and Rust programs work out by hand the slope of the loss on each move's logit; the PyTorch tab lets autograd do it and takes one optimizer.step(). All three agree on every number they print (the PyTorch tab skips the two single-move checks): the clipped decision gets a slope of exactly 0, and one gradient step with a learning rate of 1 moves the chosen move from 0.25 to 0.315 in the second decision and from 0.50 to 0.421 in the third, whose advantage is −1.

// GAE and PPO's clipped loss, checked against the worked examples on this page.
// It prints each result and exits with 1 if any number differs from the page's.
#include <algorithm>
#include <array>
#include <cmath>
#include <cstdio>
#include <vector>

// Credit for one player's decisions in one game. values[i] is the critic's estimate before
// decision i; the only reward is the result (+1 win, -1 loss), after the last decision.
struct Credit {
    std::vector<double> advantages;     // how much better each move did than the critic expected
    std::vector<double> value_targets;  // what the critic trains toward: advantage + value
};

Credit gae(const std::vector<double>& values, double reward, double discount, double gae_lambda) {
    const int count = static_cast<int>(values.size());
    Credit credit{std::vector<double>(count), std::vector<double>(count)};
    double running = 0.0;  // the advantage of the decision after this one
    for (int step = count - 1; step >= 0; --step) {
        const bool last = step == count - 1;
        const double next_value = last ? 0.0 : values[step + 1];  // nothing follows the end
        const double step_reward = last ? reward : 0.0;            // only the result pays
        // TD error: what happened (reward, then the critic's next estimate) minus what was expected.
        const double td_error = step_reward + discount * next_value - values[step];
        // Later surprises count as well, shrinking by discount * gae_lambda for each step back.
        running = td_error + discount * gae_lambda * running;
        credit.advantages[step] = running;
        credit.value_targets[step] = running + values[step];
    }
    return credit;
}

// One decision's share of PPO's clipped objective.
struct ClippedTerm {
    double ratio;      // the move's probability now / its probability when the game was played
    double objective;  // min(ratio * advantage, clipped ratio * advantage)
    double slope;      // d objective / d ratio: 0 once the clip has taken over
};

ClippedTerm clipped_term(double new_log_prob, double old_log_prob, double advantage, double clip) {
    const double ratio = std::exp(new_log_prob - old_log_prob);
    const double clipped_ratio = std::clamp(ratio, 1.0 - clip, 1.0 + clip);
    // The smaller of the two: no credit for moving the ratio past the clip.
    const double objective = std::min(ratio * advantage, clipped_ratio * advantage);
    // Flat once the ratio has moved past the clip in the direction the advantage asks for.
    const bool flat = (advantage > 0 && ratio > 1.0 + clip) || (advantage < 0 && ratio < 1.0 - clip);
    return {ratio, objective, flat ? 0.0 : advantage};
}

constexpr int kMoves = 3;
using Logits = std::array<double, kMoves>;

struct Decision {
    Logits logits;        // the network's scores for the three moves now
    int chosen;           // the move played when the game was recorded
    double old_log_prob;  // the log probability of that move back then
    double advantage;     // from GAE
};

Logits softmax(const Logits& logits) {
    const double top = *std::max_element(logits.begin(), logits.end());  // keeps exp() in range
    Logits probabilities{};
    double total = 0.0;
    for (int k = 0; k < kMoves; ++k) {
        probabilities[k] = std::exp(logits[k] - top);
        total += probabilities[k];
    }
    for (double& probability : probabilities) probability /= total;
    return probabilities;
}

// The minibatch's policy loss (minus the mean objective, because optimizers minimize) and its
// slope on every logit: the numbers PyTorch's backward() computes for the same batch.
double clipped_loss(const std::vector<Decision>& batch, double clip, std::vector<Logits>& gradients) {
    const double count = static_cast<double>(batch.size());
    double total = 0.0;
    gradients.assign(batch.size(), Logits{});
    for (std::size_t i = 0; i < batch.size(); ++i) {
        const Decision& decision = batch[i];
        const Logits probabilities = softmax(decision.logits);
        const ClippedTerm term = clipped_term(std::log(probabilities[decision.chosen]), decision.old_log_prob,
                                              decision.advantage, clip);
        total += term.objective;
        if (term.slope == 0.0) continue;  // clipped: this decision adds nothing to the gradient
        // d ratio / d logit k = ratio * ((k == chosen) - p_k): raising the chosen move's logit raises its
        // probability, and raising any other logit takes probability away from it.
        for (int k = 0; k < kMoves; ++k) {
            const double is_chosen = k == decision.chosen ? 1.0 : 0.0;
            gradients[i][k] = -term.slope * term.ratio * (is_chosen - probabilities[k]) / count;
        }
    }
    return -total / count;
}

int failures = 0;
// Prints one row of results and counts any that differ from the page's numbers by 1e-6 or more.
void expect(const char* what, const std::vector<double>& actual, const std::vector<double>& expected) {
    std::printf("%-32s", what);
    for (std::size_t i = 0; i < actual.size(); ++i) {
        const bool ok = std::abs(actual[i] - expected[i]) < 1e-6;
        std::printf(" %10.6f%s", actual[i], ok ? "" : " (MISMATCH)");
        if (!ok) ++failures;
    }
    std::printf("\n");
}

int main() {
    // Section 03: three moves, the critic's values before them, then a loss. No discount.
    const std::vector<double> values{0.2, 0.6, 0.1};
    expect("lost, lambda 0", gae(values, -1.0, 1.0, 0.0).advantages, {0.4, -0.5, -1.1});
    expect("lost, lambda 1", gae(values, -1.0, 1.0, 1.0).advantages, {-1.2, -1.6, -1.1});
    expect("lost, lambda 0.5", gae(values, -1.0, 1.0, 0.5).advantages, {-0.125, -1.05, -1.1});
    expect("won, lambda 0.5", gae(values, +1.0, 1.0, 0.5).advantages, {0.375, -0.05, 0.9});
    expect("value targets, lost, 0.5", gae(values, -1.0, 1.0, 0.5).value_targets, {0.075, -0.45, -1.0});

    // The widget's 12-turn game at the project's settings: credit for the blunder on turn 8.
    const std::vector<double> game{0.00, 0.05, 0.02, 0.10, 0.45, 0.50, 0.48, 0.55, 0.10, 0.15, 0.25, 0.30};
    expect("blunder, game won / lost",
           {gae(game, +1.0, 0.995, 0.95).advantages[7], gae(game, -1.0, 0.995, 0.95).advantages[7]},
           {0.284085, -1.312591});

    // Section 04: a move whose probability rose from 0.30 to 0.40 since the game was played.
    const double clip = 0.2;
    const ClippedTerm good = clipped_term(std::log(0.40), std::log(0.30), +1.0, clip);
    const ClippedTerm bad = clipped_term(std::log(0.40), std::log(0.30), -1.0, clip);
    expect("advantage +1: objective, slope", {good.objective, good.slope}, {1.2, 0.0});
    expect("advantage -1: objective, slope", {bad.objective, bad.slope}, {-4.0 / 3.0, -1.0});

    // One minibatch of three decisions, each of which played move 0.
    std::vector<Decision> batch{
        {{std::log(0.40), std::log(0.35), std::log(0.25)}, 0, std::log(0.30), +1.0},  // past the clip
        {{std::log(0.25), std::log(0.50), std::log(0.25)}, 0, std::log(0.30), +1.0},  // inside it
        {{std::log(0.50), std::log(0.30), std::log(0.20)}, 0, std::log(0.40), -1.0},  // rose, but was bad
    };
    std::vector<Logits> gradients;
    expect("minibatch loss", {clipped_loss(batch, clip, gradients)}, {-0.261111111});
    const std::vector<std::vector<double>> gradient_expected{
        {0.0, 0.0, 0.0}, {-0.208333333, 0.138888889, 0.069444444}, {0.208333333, -0.125, -0.083333333}};
    const std::vector<std::vector<double>> after_expected{
        {0.40, 0.35, 0.25}, {0.315381634, 0.445727777, 0.238890590}, {0.421437667, 0.352898187, 0.225664147}};
    for (std::size_t i = 0; i < batch.size(); ++i) {
        expect("gradient", {gradients[i].begin(), gradients[i].end()}, gradient_expected[i]);
        // One plain gradient step with a learning rate of 1: logits -= gradient.
        for (int k = 0; k < kMoves; ++k) batch[i].logits[k] -= gradients[i][k];
        const Logits after = softmax(batch[i].logits);
        expect("probabilities after one step", {after.begin(), after.end()}, after_expected[i]);
    }
    std::printf("%s\n", failures ? "FAILED" : "all checks passed");
    return failures ? 1 : 0;
}

// GAE and PPO's clipped loss, checked against the worked examples on this page.
// It prints each result and exits with 1 if any number differs from the page's.

// Credit for one player's decisions in one game. values[i] is the critic's estimate before
// decision i; the only reward is the result (+1 win, -1 loss), after the last decision.
struct Credit {
    advantages: Vec<f64>,    // how much better each move did than the critic expected
    value_targets: Vec<f64>, // what the critic trains toward: advantage + value
}

fn gae(values: &[f64], reward: f64, discount: f64, gae_lambda: f64) -> Credit {
    let count = values.len();
    let mut credit = Credit { advantages: vec![0.0; count], value_targets: vec![0.0; count] };
    let mut running = 0.0; // the advantage of the decision after this one
    for step in (0..count).rev() {
        let last = step == count - 1;
        let next_value = if last { 0.0 } else { values[step + 1] }; // nothing follows the end
        let step_reward = if last { reward } else { 0.0 }; // only the result pays
        // TD error: what happened (reward, then the critic's next estimate) minus what was expected.
        let td_error = step_reward + discount * next_value - values[step];
        // Later surprises count as well, shrinking by discount * gae_lambda for each step back.
        running = td_error + discount * gae_lambda * running;
        credit.advantages[step] = running;
        credit.value_targets[step] = running + values[step];
    }
    credit
}

// One decision's share of PPO's clipped objective.
struct ClippedTerm {
    ratio: f64,     // the move's probability now / its probability when the game was played
    objective: f64, // min(ratio * advantage, clipped ratio * advantage)
    slope: f64,     // d objective / d ratio: 0 once the clip has taken over
}

fn clipped_term(new_log_prob: f64, old_log_prob: f64, advantage: f64, clip: f64) -> ClippedTerm {
    let ratio = (new_log_prob - old_log_prob).exp();
    let clipped_ratio = ratio.clamp(1.0 - clip, 1.0 + clip);
    // The smaller of the two: no credit for moving the ratio past the clip.
    let objective = (ratio * advantage).min(clipped_ratio * advantage);
    // Flat once the ratio has moved past the clip in the direction the advantage asks for.
    let flat = (advantage > 0.0 && ratio > 1.0 + clip) || (advantage < 0.0 && ratio < 1.0 - clip);
    ClippedTerm { ratio, objective, slope: if flat { 0.0 } else { advantage } }
}

const MOVES: usize = 3;
type Logits = [f64; MOVES];

struct Decision {
    logits: Logits,    // the network's scores for the three moves now
    chosen: usize,     // the move played when the game was recorded
    old_log_prob: f64, // the log probability of that move back then
    advantage: f64,    // from GAE
}

fn softmax(logits: &Logits) -> Logits {
    let top = logits.iter().copied().fold(f64::NEG_INFINITY, f64::max); // keeps exp() in range
    let mut probabilities = [0.0; MOVES];
    let mut total = 0.0;
    for k in 0..MOVES {
        probabilities[k] = (logits[k] - top).exp();
        total += probabilities[k];
    }
    for probability in probabilities.iter_mut() {
        *probability /= total;
    }
    probabilities
}

// The minibatch's policy loss (minus the mean objective, because optimizers minimize) and its
// slope on every logit: the numbers PyTorch's backward() computes for the same batch.
fn clipped_loss(batch: &[Decision], clip: f64, gradients: &mut Vec<Logits>) -> f64 {
    let count = batch.len() as f64;
    let mut total = 0.0;
    gradients.clear();
    gradients.resize(batch.len(), [0.0; MOVES]);
    for (i, decision) in batch.iter().enumerate() {
        let probabilities = softmax(&decision.logits);
        let term = clipped_term(probabilities[decision.chosen].ln(), decision.old_log_prob, decision.advantage, clip);
        total += term.objective;
        if term.slope == 0.0 {
            continue; // clipped: this decision adds nothing to the gradient
        }
        // d ratio / d logit k = ratio * ((k == chosen) - p_k): raising the chosen move's logit raises its
        // probability, and raising any other logit takes probability away from it.
        for k in 0..MOVES {
            let is_chosen = if k == decision.chosen { 1.0 } else { 0.0 };
            gradients[i][k] = -term.slope * term.ratio * (is_chosen - probabilities[k]) / count;
        }
    }
    -total / count
}

// Prints one row of results and counts any that differ from the page's numbers by 1e-6 or more.
fn expect(what: &str, actual: &[f64], expected: &[f64], failures: &mut u32) {
    print!("{:<32}", what);
    for (value, wanted) in actual.iter().zip(expected) {
        let ok = (value - wanted).abs() < 1e-6;
        print!(" {:10.6}{}", value, if ok { "" } else { " (MISMATCH)" });
        if !ok {
            *failures += 1;
        }
    }
    println!();
}

fn main() {
    let mut failures = 0;
    // Section 03: three moves, the critic's values before them, then a loss. No discount.
    let values = [0.2, 0.6, 0.1];
    expect("lost, lambda 0", &gae(&values, -1.0, 1.0, 0.0).advantages, &[0.4, -0.5, -1.1], &mut failures);
    expect("lost, lambda 1", &gae(&values, -1.0, 1.0, 1.0).advantages, &[-1.2, -1.6, -1.1], &mut failures);
    expect("lost, lambda 0.5", &gae(&values, -1.0, 1.0, 0.5).advantages, &[-0.125, -1.05, -1.1], &mut failures);
    expect("won, lambda 0.5", &gae(&values, 1.0, 1.0, 0.5).advantages, &[0.375, -0.05, 0.9], &mut failures);
    expect("value targets, lost, 0.5", &gae(&values, -1.0, 1.0, 0.5).value_targets, &[0.075, -0.45, -1.0],
           &mut failures);

    // The widget's 12-turn game at the project's settings: credit for the blunder on turn 8.
    let game = [0.00, 0.05, 0.02, 0.10, 0.45, 0.50, 0.48, 0.55, 0.10, 0.15, 0.25, 0.30];
    let blunder = [gae(&game, 1.0, 0.995, 0.95).advantages[7], gae(&game, -1.0, 0.995, 0.95).advantages[7]];
    expect("blunder, game won / lost", &blunder, &[0.284085, -1.312591], &mut failures);

    // Section 04: a move whose probability rose from 0.30 to 0.40 since the game was played.
    let clip = 0.2;
    let good = clipped_term(0.40_f64.ln(), 0.30_f64.ln(), 1.0, clip);
    let bad = clipped_term(0.40_f64.ln(), 0.30_f64.ln(), -1.0, clip);
    expect("advantage +1: objective, slope", &[good.objective, good.slope], &[1.2, 0.0], &mut failures);
    expect("advantage -1: objective, slope", &[bad.objective, bad.slope], &[-4.0 / 3.0, -1.0], &mut failures);

    // One minibatch of three decisions, each of which played move 0.
    let decision = |probabilities: [f64; MOVES], old: f64, advantage: f64| Decision {
        logits: probabilities.map(f64::ln),
        chosen: 0,
        old_log_prob: old.ln(),
        advantage,
    };
    let mut batch = vec![
        decision([0.40, 0.35, 0.25], 0.30, 1.0),  // past the clip
        decision([0.25, 0.50, 0.25], 0.30, 1.0),  // inside it
        decision([0.50, 0.30, 0.20], 0.40, -1.0), // rose, but was bad
    ];
    let mut gradients = Vec::new();
    expect("minibatch loss", &[clipped_loss(&batch, clip, &mut gradients)], &[-0.261111111], &mut failures);
    let gradient_expected = [[0.0, 0.0, 0.0], [-0.208333333, 0.138888889, 0.069444444], [0.208333333, -0.125, -0.083333333]];
    let after_expected = [[0.40, 0.35, 0.25], [0.315381634, 0.445727777, 0.238890590], [0.421437667, 0.352898187, 0.225664147]];
    for i in 0..batch.len() {
        expect("gradient", &gradients[i], &gradient_expected[i], &mut failures);
        // One plain gradient step with a learning rate of 1: logits -= gradient.
        for k in 0..MOVES {
            batch[i].logits[k] -= gradients[i][k];
        }
        expect("probabilities after one step", &softmax(&batch[i].logits), &after_expected[i], &mut failures);
    }
    println!("{}", if failures > 0 { "FAILED" } else { "all checks passed" });
    std::process::exit(if failures > 0 { 1 } else { 0 });
}

# The same checks with PyTorch: GAE in plain Python, then one PPO update step in which autograd
# computes the slopes the C++ and Rust programs work out by hand. Exits with 1 on a mismatch.
import sys

import torch

failures = 0


def expect(what, actual, expected):
    """Prints one row of results and counts any that differ from the page's numbers by 1e-6 or more."""
    global failures
    row = []
    for value, wanted in zip(actual, expected):
        ok = abs(value-wanted) < 1e-6
        row.append(f'{value:10.6f}' + ('' if ok else ' (MISMATCH)'))
        failures += 0 if ok else 1
    print(f'{what:<32}', ' '.join(row))


def gae(values, reward, discount, gae_lambda):
    """Advantages and value targets for one player's decisions in one game; the only reward is the
    result (+1 win, -1 loss), after the last decision. Same arithmetic as gae() in ppo.py."""
    advantages = [0.0]*len(values)
    running = 0.0                                    # the advantage of the decision after this one
    for step in reversed(range(len(values))):
        last = step == len(values)-1
        next_value = 0.0 if last else values[step+1]   # nothing follows the end
        step_reward = reward if last else 0.0          # only the result pays
        td_error = step_reward+discount*next_value-values[step]
        running = td_error+discount*gae_lambda*running
        advantages[step] = running
    return advantages, [advantage+value for advantage, value in zip(advantages, values)]


# Section 03: three moves, the critic's values before them, then a loss. No discount.
values = [0.2, 0.6, 0.1]
expect('lost, lambda 0', gae(values, -1.0, 1.0, 0.0)[0], [0.4, -0.5, -1.1])
expect('lost, lambda 1', gae(values, -1.0, 1.0, 1.0)[0], [-1.2, -1.6, -1.1])
expect('lost, lambda 0.5', gae(values, -1.0, 1.0, 0.5)[0], [-0.125, -1.05, -1.1])
expect('won, lambda 0.5', gae(values, 1.0, 1.0, 0.5)[0], [0.375, -0.05, 0.9])
expect('value targets, lost, 0.5', gae(values, -1.0, 1.0, 0.5)[1], [0.075, -0.45, -1.0])

# The widget's 12-turn game at the project's settings: credit for the blunder on turn 8.
game = [0.00, 0.05, 0.02, 0.10, 0.45, 0.50, 0.48, 0.55, 0.10, 0.15, 0.25, 0.30]
expect('blunder, game won / lost', [gae(game, 1.0, 0.995, 0.95)[0][7], gae(game, -1.0, 0.995, 0.95)[0][7]],
       [0.284085, -1.312591])

# Section 04: one minibatch of three decisions, each of which played move 0. float64 so the
# numbers match the C++ and Rust programs to the last digit printed.
clip = 0.2
probabilities = torch.tensor([[0.40, 0.35, 0.25],     # past the clip
                              [0.25, 0.50, 0.25],     # inside it
                              [0.50, 0.30, 0.20]],    # rose, but was bad
                             dtype=torch.float64)
logits = probabilities.log().requires_grad_()       # the weights this step will change
chosen = torch.tensor([0, 0, 0])
old_log_probs = torch.tensor([0.30, 0.30, 0.40], dtype=torch.float64).log()
advantages = torch.tensor([1.0, 1.0, -1.0], dtype=torch.float64)
optimizer = torch.optim.SGD([logits], lr=1.0)        # the plain step of the C++ and Rust programs

new_log_probs = torch.log_softmax(logits, 1).gather(1, chosen[:, None]).squeeze(1)
ratio = (new_log_probs-old_log_probs).exp()
# The same min as ppo.py: the clipped term takes over only where it is the more pessimistic one.
loss = -torch.min(ratio*advantages, ratio.clamp(1-clip, 1+clip)*advantages).mean()
optimizer.zero_grad()                               # gradients add up across backward() calls
loss.backward()                                     # autograd fills logits.grad
expect('minibatch loss', [loss.item()], [-0.261111111])
gradient_expected = [[0.0, 0.0, 0.0], [-0.208333333, 0.138888889, 0.069444444], [0.208333333, -0.125, -0.083333333]]
for row, wanted in zip(logits.grad.tolist(), gradient_expected):
    expect('gradient', row, wanted)
optimizer.step()                                    # one update: logits -= lr * logits.grad
after_expected = [[0.40, 0.35, 0.25], [0.315381634, 0.445727777, 0.238890590], [0.421437667, 0.352898187, 0.225664147]]
for row, wanted in zip(torch.softmax(logits, 1).tolist(), after_expected):
    expect('probabilities after one step', row, wanted)
print('FAILED' if failures else 'all checks passed')
sys.exit(1 if failures else 0)

12Where self-play goes wrong

A NaN from the mask in the entropy bonus. An illegal move's log-probability is −∞, and 0 × −∞ is NaN: in the project the first PPO step turned every weight into NaN. Zero the masked log-probabilities first, as entropy() in model.py does (Neural Networks for Game Bots).

Reading the clone's .500 against itself as a measurement. Light cycles have no draws, and two identical deterministic players play the same game from both seats of a seed, so each wins one: the curve's first point against the clone is that mirror.

A learning rate set for a random network. A clone's policy is already sharp: in a short test at 1 × 10⁻⁴, the project's updates moved it by only 0.0001 to 0.0004 of KL divergence, so the runs used 2 × 10⁻⁴ and an entropy bonus of 0.005.^[1]

A shaping term the network can't see, which stays in the advantages and acts as a real reward (section 09), and training at one temperature while playing at another (section 10).

Training one against one for a game of four. A network that has only met one opponent has never practiced a battle of three or four cycles.

13What's next

Building the Trainer runs this loop faster and longer. The games move back to the Rust engine, 512 at a time behind the C interface, the learner trains on the GPU, and the trainer gains checkpoints, a clean stop on Ctrl+C, resuming and a status file. It measures games per hour before and after on the same machine.

The project so far, with this step's files in teal:

project/
├── engine/                    Rust: rules, bots, features, network, C interface
│   ├── Cargo.toml
│   └── src/
│       ├── board.rs           grid, distances, room, Voronoi territory
│       ├── game.rs            rules, seeded starts, CodinGame frames
│       ├── bots.rs            flood-fill and Voronoi bots (the teacher)
│       ├── alphabeta.rs       the alpha-beta bot (a yardstick)
│       ├── features.rs        what the network sees
│       ├── net.rs             the network's forward pass in Rust
│       ├── ffi.rs             the C interface Python calls
│       └── lib.rs
├── train/
│   ├── tron_env.py            drives the engine from Python
│   ├── model.py               the network in PyTorch
│   ├── parity.py              Rust and PyTorch choose the same moves
│   ├── record.py              records the teacher's decisions
│   ├── bc.py                  behavior cloning
│   ├── play.py                matches, both seats, Wilson intervals
│   ├── evaluate.py            a network against the yardsticks
│   ├── tron.py                light cycles in plain Python
│   └── ppo.py                 self-play PPO in one process
├── checks/
│   └── python_engine.py       the Python engine against the Rust one
├── data/
│   ├── bc-train.npz           496,402 decisions from 3,000 games
│   └── bc-heldout.npz         49,864 decisions from 300 games
└── runs/
    ├── bc/
    │   ├── clone.pt           the clone's weights
    │   └── bc.json            its training log
    └── ppo-single/
        ├── config.json        the run's settings
        ├── metrics.jsonl      one line per iteration, gates included
        ├── learner.pt         the last weights
        └── champion.pt        the gate's champion

14Sources

Quotes are from each author's own text. Numbers marked as the project's were measured with the code on this page and the series' earlier pages; numbers marked as the author's lab are first-hand records from one lab and aren't general results. Widget readouts are computed in the page, exactly or from seeded episodes.

1. Build a Legend Bot running project, steps 2 and 3, run on October 30, 2026 on one machine (Intel Core i9-11900KF; NVIDIA RTX 3080 Ti shared with other training jobs). The clone's matches (runs/logs/eval-clone.log); the single-process run (runs/ppo-single/, runs/logs/eval-single-champion.log); the batched run's strength curve and gates (runs/ppo/curve.jsonl, runs/ppo/metrics.jsonl); the final 1,000-game test (runs/logs/final-alphabeta-test.log); the Python-against-Rust engine check; and the champion sampled at temperature 1 against its teacher. The code is on this page and the series' earlier pages.
2. Richard S. Sutton and Andrew G. Barto, 2018, second edition. Reinforcement Learning: An Introduction, MIT Press. Section 3.2 (reward), 3.3 (return and discount rate), 13.4 (a baseline that doesn't depend on the action) and 13.5 (the critic).
3. putibuzu, 2026. Rails and soup, CodinGame forum, Summer Challenge 2026 feedback and strategies, post 4. A discount factor too small for a reward that comes only at the end.
4. Ronald J. Williams, 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Machine Learning 8, 229–256 (author's preprint). REINFORCE: weight changes proportional to the reward minus a baseline.
5. John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, Pieter Abbeel, 2016. High-Dimensional Continuous Control Using Generalized Advantage Estimation, ICLR 2016. The advantage function, the TD residual, GAE and its trade between bias and variance, and section 4 on GAE as reward shaping with the critic as the potential.
6. reCurse, 2024. reCurse's Codingame Fall Challenge 2023 Postmortem (gist created 2024-01-08, last updated 2026-09-20). PPO with γ = 0.99 and λ = 0.95, greedy play, and the late losses from training only against the current network.
7. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017. Proximal Policy Optimization Algorithms, arXiv:1707.06347 (preprint). The probability ratio, the clipped objective, several epochs of minibatch updates per batch, an iteration in Algorithm 1, and the clip settings for the robot-control and Atari runs.
8. Volodymyr Mnih et al., 2016. Asynchronous Methods for Deep Reinforcement Learning, ICML 2016. Adding the policy's entropy to the objective to discourage early convergence on one action.
9. Author's lab records, 2026 (private repository). The trainer's KL stop, opponent mix and pool weights, framework/ppo_selfplay.py and framework/rl/trainer.py; Spring Challenge 2022's drift, rollback and guards, and its greedy and sampled results, docs/multi-game.md and experiments/runs/spring2022/evaluations.jsonl; the rewards, framework/rl/rewards.py, games/backtrackking/rl.py and games/spring2022/rl.py; the wild-mana diagnosis, experiments/spring2022-gold-20261029.json; Back Track King's main run and its promotions, experiments/runs/nn-main/metrics.jsonl and evaluations.jsonl; its best live rank and later variants, AGENTS.md (October 25 and 26 entries) and experiments/live-submissions/summer-challenge-2026-back-track-king/submitted.jsonl; Troll Farm's run and diagnosis, experiments/runs/trollfarm/metrics.jsonl and docs/postmortem-lessons.md.
10. JOiy, 2026. JOiy's bot: Summer challenge 2026 (gist, last updated 2026-10-10). His PPO settings, curves that look healthy without the bot improving on the ladder, and a terminal-only reward against per-turn shaping.
11. Pduhard, 2026. Summer Challenge 2026, Back Track King: 3rd, README. 512 games at once with both seats trained, half the games against a pool that draws the snapshots that beat the network more often, the income-gap reward, and two shaping attempts that didn't work.
12. Gerald Tesauro, 1995. Temporal Difference Learning and TD-Gammon, Communications of the ACM 38(3), 58–68 (transcription published with the ACM's permission). A network trained by playing itself, and the risk of self-play in deterministic games.
13. Oriol Vinyals et al., 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature 575, 350–354 (authors' manuscript). Self-play that chases cycles, fictitious self-play, and opponents drawn by how often they beat the learner.
14. Christopher Berner et al. (OpenAI), 2019. Dota 2 with Large Scale Deep Reinforcement Learning, arXiv:1912.06680, section 3.2 and appendix N. 80% of games against the latest policy and 20% against past versions.
15. Wojciech Marian Czarnecki et al., 2020. Real World Games Look Like Spinning Tops, NeurIPS 2020. How population size decides whether self-play training converges or cycles, in their two-player games.
16. David Silver et al., 2017. Mastering the game of Go without human knowledge, Nature 550, 354–359 (authors' manuscript). The evaluator that promoted a network only above 55% of 400 games.
17. David Silver et al., 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play, Science 362, 1140–1144 (preprint). Self-play games always generated by the latest network, with no gate.
18. CodinGame. Tron Battle, game statement. Battles of 2 to 4 cycles, with one opponent in the first league.
19. Andrew Y. Ng, Daishi Harada, Stuart Russell, 1999. Policy invariance under reward transformations: Theory and application to reward shaping, ICML 1999. Potential-based shaping and its guarantee, and Randløv and Alstrøm's bicycle learner.
20. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, 2016. Concrete Problems in AI Safety, arXiv:1606.06565, section 4 on reward hacking.
21. Victoria Krakovna et al., 2020. Specification gaming: the flip side of AI ingenuity, DeepMind blog, April 21. Shaping that isn't potential-based, and the boat that circled to hit the same green blocks.
22. Indrill, 2026. Ranked 1st, CodinGame forum, Summer Challenge 2026 feedback and strategies, post 5. Deep Q-learning, a reward reshaped with a projected final score, and greedy selection.
23. delineate, 2026. Codingame Spring Challenge 2026 (delineate) (gist). Shaping to teach the first steps, a network that gamed it, and training without it.