Behavior Cloning

01Copy the bot you already have

The running project has a network now, built in Neural Networks for Game Bots. It reads a light-cycle board as a stack of grids, takes two extra numbers for each move, and outputs a probability for each legal move and a predicted result. The two numbers are the free cells still reachable after the move and the Voronoi margin after it: the cells you reach before your opponent, minus the cells it reaches first. With random weights it plays random legal moves. Self-play, where the network improves by playing against itself, could start there, but its first games would go to learning not to drive into dead ends, which the Voronoi bot you wrote in Evaluation Functions already avoids.

Behavior cloning skips that stage. You run your bot, the teacher, through your simulator, record each position it faced and the move it chose, and train the network to predict that move from that position. Each recorded pair is a demonstration. The teacher's move is the label, the right answer the network is graded against, so this is supervised learning, the kind of training When to Use Machine Learning used to fit three weights to recorded positions.

The idea comes from robotics and control. In 1999 Michael Bain and Claude Sammut described cloning as feeding "the logged data from skilled, human operators" to "an induction program which outputs a control strategy", and their abstract adds that early clones "were brittle and difficult to understand."^[1] A decade earlier, Dean Pomerleau's ALVINN learned road following from 1,200 simulated road images labeled with the correct turn curvature, and its author noted that the network "must not solely be shown examples of accurate driving, but also how to recover (i.e. return to the road center) once a mistake has been made."^[2] Recovery comes back in section 06.

In a contest the teacher costs nothing extra: it's the bot you already wrote and measured. RamzaBeoulve, 4th in Back Track King (Summer Challenge 2026), did exactly this: "My network didn't start from scratch. I first trained it with behavioral cloning on my best heuristic bot to give it a strong starting point and avoid spending days before getting decent PPO results."^[3] Starting self-play from a clone is a warm start: training begins from weights that already play the teacher's game.

02Record the teacher through your simulator

A clone needs many demonstrations, from positions like the ones it will meet: the project records about half a million. Your simulator from Referees and Simulators records that many in minutes, because it runs the rules inside your own process, without the referee or its time limits.

Recording is a loop over seeded games. Each seed fixes one game's start, so a recording can be replayed exactly. The loop stores what the network will need on every one of the teacher's turns. The details below decide whether the clone learns the game or a narrow slice of it.

1. Play the teacher in both seats, on many seeds. In turn-based light cycles player 0 moves first, so a teacher recorded from one seat shows the clone half of the openings.
2. Vary the opponent. The project plays every seed three times: the teacher against itself (both seats recorded), against the flood-fill bot (it moves wherever the most cells stay reachable), and against the alpha-beta bot from Minimax and Alpha-Beta, which searches several moves ahead. Against one opponent the recordings only cover the positions that opponent leads to.
3. Store what the network reads. For the player to move: the grids and per-move numbers the network reads, which moves are legal, the move the teacher chose, and, once the game ends, whether that player won. The value head, the network's second output, learns to predict that result. For a game with a big board you can store only the moves and rebuild the inputs later by replaying each game through your simulator.
4. Turn each printed command into an action. The network outputs one probability per move, so every answer the teacher gives must become an index into that output. A light-cycle bot prints UP, DOWN, LEFT or RIGHT: one of four outputs. When the teacher runs inside your engine, as the project's does, it returns the index directly. A Back Track King bot prints several PLACE_TRACKS commands and a DISRUPT per turn; the author's lab parsed each one into its network's slots and skipped the turns its network couldn't express.^[4]
5. Keep only decisions with a real choice. With one legal move, the legality mask from Neural Networks for Game Bots already decides; the loss is zero, so the example only costs time.
6. Mark the contested decisions. A decision is contested when the teacher's own scores for its legal moves differ. When they're all equal, the teacher's tie-break picked the move: the project's bots go straight first, then right, then left. Contested decisions are where the teacher's judgment shows, so the clone is measured on them separately.
7. Hold out games. Record a second set on seeds you never train on. Every agreement number on this page is measured there unless it is labeled as training agreement.

The project's recording, 1,000 seeds played three ways on the 30 × 20 board, took 125 seconds: 3,000 games, 266 moves a game on average, and 496,402 decisions with a choice, of which 39.7% were contested.^[5] The other 60% were ties the teacher broke with its preference order. A held-out set of 300 games on other seeds gave 49,864 decisions, 39.8% of them contested.

03Train the network to predict the teacher

Training adjusts the network's weights until it gives the teacher's move a high probability in each recorded position. For that it needs a number that says how wrong the network is on one decision.

The network outputs one score per move, a logit. The softmax turns the scores into probabilities that add up to 1, and the mask sets every illegal move's probability to exactly 0. The loss is the cross-entropy: minus the natural log of the probability the network gave the teacher's move. A confident right answer costs almost nothing, and a confident wrong one costs a lot.^[6]

Written out, the probability of a move and the loss on one decision are:

In words: raise e to each legal move's score and divide by the total, which gives probabilities that add up to 1. Then charge minus the log of the probability the network gave the move the teacher actually played. Training lowers the average charge over all the recorded decisions.

Training works through the decisions in a random order, a minibatch of a few hundred at a time; one pass over all of them is an epoch. The project updates the weights with Adam, which scales each weight's step by the recent size of its slopes, so weights fed large and small inputs learn at a similar pace.^[7] To check a clone, measure its agreement: the share of decisions where its most likely legal move is the teacher's move. Measure it on the held-out games; agreement on the training games also rewards memorizing them.

The project trains the value head at the same time. On every decision it predicts the game's result for the player to move, +1 for a win and −1 for a loss, and the squared error of that prediction is added to the loss at half weight. Self-play's updates lean on that prediction (the next page shows how), and the recordings already hold the results to train it on.

The widget runs the whole loop in your browser, on a smaller game. Its teacher is the flood-fill bot from Evaluation Functions, which moves wherever the most cells stay reachable, playing itself on a 15 × 10 board. Its network sees only the 9 × 9 cells around its head, turned so the cycle always faces up, and has one hidden layer. At the end the clone plays its teacher, and the score comes with a 95% interval: the range of true win rates that fit the result (Measuring Bot Strength shows how to compute it).

So the 98.8% says little on its own. A rule with no judgment at all, which takes the first legal move in the order straight, right, left, agrees with the teacher on 4,783 of the 4,842 held-out decisions (98.8%) and on 57 of the 116 contested ones (49.1%). Those are the network's numbers after its first epoch: so far it has learned the tie-break and nothing else. Played against the flood-fill bot, that rule scores 67 of 200. The contested line is the one to watch.

04Agreement is not strength

Agreement counts every decision the same, and games don't. Many decisions in a light-cycle game are ties, and a few decide who runs out of room first.

A first version of the project's network shows the gap. It read the board's grids only and took each move's score from the cell next to the head, without the two per-move numbers. Cloned from the Voronoi bot on the same 496,402 decisions, it agreed with its teacher on 94.8% of held-out decisions and 88.9% of the contested ones. Then it played the evaluation seeds, 200 seeds in both seats:^[5]

It lost three games in four to the bot it agreed with 94.8% of the time, because its mistakes fell in the wrong places. On a sample of 6,000 held-out decisions, the legal moves left different amounts of room in 2.1% of them: one move led into a smaller region than another. The clone agreed with its teacher on 81.9% of those, against 94.9% elsewhere, and on 16.5% of them it chose a move with less room than the best one. Nothing in its convolutions counts the cells of a region or compares two pockets, and doubling its width to 32 channels only moved contested agreement from 88.9% to 90.1%.

The fix was to hand the network the two numbers the teacher ranks moves by: each move's score now also reads the room left after the move and the Voronoi margin after it, the inputs Neural Networks for Game Bots builds. Trained the same way, that network agreed on 97.85% of held-out decisions and 96.0% of the contested ones, and played level with its teacher: 196 wins in 400 games, an interval of .441 to .539. Section 05 has the full numbers.

So a clone gets three checks, and only the last measures strength:

1. Agreement on held-out decisions. It catches a broken pipeline, such as labels shifted to the wrong move, and little else.
2. Agreement on contested decisions. It shows whether the clone copies the teacher's judgment or only its tie-break.
3. Games against the teacher and your other yardsticks, on seeds kept out of training, both seats, with a Wilson interval from Measuring Bot Strength.

Expect a perfect clone to score exactly half against its teacher. Turn-based light cycles have no draws, and two identical bots that always make the same move in the same position play the same game from both seats of a seed, so each wins one of the two. A clone at .490, with an interval from .441 to .539, plays level with its teacher. It hasn't beaten it, and nothing in cloning aims to: the loss only rewards copying.

05Project step 2: clone the Voronoi bot

cargo build --release --manifest-path ../engine/Cargo.toml                          # the engine and its C interface
python record.py --first-seed 2000000 --seeds 100 --out ../data/bc-heldout.npz    # 300 games, 18 s
python record.py --first-seed 1000000 --seeds 1000 --out ../data/bc-train.npz     # 3,000 games, 125 s
python bc.py --train ../data/bc-train.npz --heldout ../data/bc-heldout.npz --out ../runs/bc --channels 16 --epochs 8
python evaluate.py ../runs/bc/clone.pt --seeds 200 --against flood,voronoi,alphabeta

"""Record the Voronoi bot's games through the engine: every position where it had to choose between two
or more legal moves, the move it chose, and whether that player went on to win. These are the
demonstrations behavior cloning learns from.

    python record.py --first-seed 1000000 --seeds 1000 --out ../data/bc-train.npz
"""
import argparse
import time

import numpy as np

from tron_env import ALPHABETA, FLOOD, VORONOI, Batch

TEACHER = VORONOI
# Each seed is played three times: the teacher against itself (both seats recorded), against the
# flood-fill bot and against the alpha-beta bot (one seat recorded, alternating with the seed), so the
# teacher is seen against different styles.
OPPONENTS = (VORONOI, FLOOD, ALPHABETA)


def record(first_seed, seeds, threads=6):
    count = len(OPPONENTS)*seeds
    batch = Batch(count, threads=threads)
    bots = np.zeros((count, 2), np.int64)             # the bot in each seat of each game
    for seed in range(seeds):
        for slot, opponent in enumerate(OPPONENTS):
            game = seed*len(OPPONENTS)+slot
            batch.reset(game, first_seed+seed)
            teacher_seat = seed % 2
            bots[game, teacher_seat], bots[game, 1-teacher_seat] = TEACHER, opponent
    rows = {name: [] for name in ('planes', 'features', 'targets', 'legal', 'move', 'contested', 'game', 'seat')}
    while not (done := batch.done()).all():
        planes, features, targets, legal, to_move = batch.observe()
        bot = bots[np.arange(count), to_move]
        moves = np.zeros(count, np.int32)
        for kind in (TEACHER, FLOOD, ALPHABETA):
            turn = ~done & (bot == kind)
            if turn.any():
                moves[turn] = batch.bot_moves(kind, wanted=turn)[turn]
        # Keep the teacher's decisions with a real choice. A decision is contested when the legal moves
        # score differently for the teacher; otherwise its straight-first tie-break decided.
        keep = ~done & (bot == TEACHER) & (legal.sum(1) >= 2)
        if keep.any():
            scores = batch.voronoi_scores()[keep]
            legal_kept = legal[keep]
            best = np.where(legal_kept, scores, np.iinfo(np.int32).min).max(1)
            worst = np.where(legal_kept, scores, np.iinfo(np.int32).max).min(1)
            rows['planes'].append(np.packbits(planes[keep].reshape(keep.sum(), -1), axis=1))
            rows['features'].append(features[keep])
            rows['targets'].append(targets[keep])
            rows['legal'].append(legal_kept)
            rows['move'].append(moves[keep])
            rows['contested'].append(best != worst)
            rows['game'].append(np.nonzero(keep)[0])
            rows['seat'].append(to_move[keep])
        batch.step(moves)
    data = {name: np.concatenate(value) for name, value in rows.items()}
    # The value target: +1 if the player who made the decision won the game, -1 if it lost.
    winners = batch.scores()
    data['won'] = np.where(winners[data['game'], data['seat']] == 1, 1, -1).astype(np.int8)
    data['game'] += first_seed*len(OPPONENTS)       # globally unique game ids
    data['plies'] = batch.plies()
    batch.close()
    return data


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--first-seed', type=int, required=True)
    parser.add_argument('--seeds', type=int, required=True)
    parser.add_argument('--threads', type=int, default=6)
    parser.add_argument('--out', required=True)
    args = parser.parse_args()
    clock = time.time()
    data = record(args.first_seed, args.seeds, args.threads)
    np.savez_compressed(args.out, **data)
    print(f"{len(OPPONENTS)*args.seeds} games ({data['plies'].mean():.0f} plies on average), "
          f"{len(data['move']):,} decisions with a choice, {data['contested'].mean():.1%} contested, "
          f"{time.time()-clock:.0f} s -> {args.out}")


if __name__ == '__main__':
    main()

"""Behavior cloning: train the network to predict the Voronoi bot's move (cross-entropy) and its value
head to predict who wins (mean squared error), then measure agreement on held-out games, overall and on
contested decisions.

    python bc.py --train ../data/bc-train.npz --heldout ../data/bc-heldout.npz --out ../runs/bc
"""
import argparse
import json
import time
from pathlib import Path

import numpy as np
import torch
import torch.nn.functional as F

from model import PLANES, HEIGHT, WIDTH, TronNet


class Demonstrations:
    """Recorded decisions (record.py). Planes stay bit-packed until a minibatch needs them."""

    def __init__(self, path):
        data = np.load(path)
        self.packed = data['planes']
        self.features = torch.from_numpy(data['features'])
        self.targets = torch.from_numpy(data['targets'].astype(np.int64))
        self.legal = torch.from_numpy(data['legal'])
        self.move = torch.from_numpy(data['move'].astype(np.int64))
        self.won = torch.from_numpy(data['won'].astype(np.float32))
        self.contested = data['contested']
        self.games = len(np.unique(data['game']))

    def __len__(self):
        return len(self.move)

    def batch(self, index, device):
        planes = np.unpackbits(self.packed[index], axis=1)[:, :PLANES*HEIGHT*WIDTH]
        planes = torch.from_numpy(planes.reshape(-1, PLANES, HEIGHT, WIDTH))
        return (planes.to(device), self.features[index].to(device), self.targets[index].to(device),
                self.legal[index].to(device), self.move[index].to(device), self.won[index].to(device))


def loss_terms(model, planes, features, targets, legal, move, won):
    logits, value = model(planes, features, targets, legal)
    imitation = F.cross_entropy(logits, move)      # -log p(teacher's move)
    value_loss = F.mse_loss(value, won)
    agree = (logits.argmax(1) == move).float()
    return imitation, value_loss, agree


@torch.no_grad()
def agreement(model, data, device, size=4096):
    """How often the network's most likely move is the teacher's move: all decisions, contested ones."""
    model.eval()
    agrees = []
    for start in range(0, len(data), size):
        index = np.arange(start, min(start+size, len(data)))
        agrees.append(loss_terms(model, *data.batch(index, device))[2].cpu().numpy())
    model.train()
    agrees = np.concatenate(agrees)
    return float(agrees.mean()), float(agrees[data.contested].mean())


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--train', required=True)
    parser.add_argument('--heldout', required=True)
    parser.add_argument('--out', required=True)
    parser.add_argument('--channels', type=int, default=16)
    parser.add_argument('--epochs', type=int, default=8)
    parser.add_argument('--batch', type=int, default=512)
    parser.add_argument('--lr', type=float, default=2e-3)
    parser.add_argument('--value-weight', type=float, default=.5)
    parser.add_argument('--limit', type=int, default=0, help='use only the first N training decisions')
    parser.add_argument('--seed', type=int, default=0)
    args = parser.parse_args()
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    torch.manual_seed(args.seed)
    out = Path(args.out)
    out.mkdir(parents=True, exist_ok=True)
    train, heldout = Demonstrations(args.train), Demonstrations(args.heldout)
    count = min(len(train), args.limit) if args.limit else len(train)
    model = TronNet(args.channels).to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=args.lr)
    steps = args.epochs*(count//args.batch)
    schedule = torch.optim.lr_scheduler.OneCycleLR(optimizer, args.lr, total_steps=steps, pct_start=.1)
    random = np.random.default_rng(args.seed)
    history, clock = [], time.time()
    for epoch in range(1, args.epochs+1):
        order = random.permutation(count)
        totals = np.zeros(3)
        for start in range(0, count-args.batch+1, args.batch):
            imitation, value_loss, agree = loss_terms(model, *train.batch(order[start:start+args.batch], device))
            loss = imitation+args.value_weight*value_loss
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            schedule.step()
            totals += [imitation.item(), value_loss.item(), agree.mean().item()]
        totals /= count//args.batch
        overall, contested = agreement(model, heldout, device)
        row = dict(epoch=epoch, imitation_loss=round(totals[0], 4), value_loss=round(totals[1], 4),
                   train_agreement=round(totals[2], 4), heldout_agreement=round(overall, 4),
                   heldout_contested_agreement=round(contested, 4), seconds=round(time.time()-clock, 1))
        history.append(row)
        print(json.dumps(row), flush=True)
    torch.save({'shape': model.shape, 'weights': {name: tensor.cpu() for name, tensor in model.state_dict().items()}},
               out/'clone.pt')
    summary = dict(args=vars(args), train_decisions=count, train_games=train.games, heldout_decisions=len(heldout),
                   heldout_games=heldout.games, heldout_contested=int(heldout.contested.sum()),
                   parameters=sum(p.numel() for p in model.parameters()), history=history)
    (out/'bc.json').write_text(json.dumps(summary, indent=1))


if __name__ == '__main__':
    main()

"""Matches between networks and the hand-written bots. Every seed is played twice, seats swapped, and
the result comes with a 95% Wilson interval."""
import math

import numpy as np
import torch

from model import TronNet, choose
from tron_env import ALPHABETA, FLOOD, VORONOI, Batch

BOTS = {'flood': FLOOD, 'voronoi': VORONOI, 'alphabeta': ALPHABETA}


def wilson(points, games, z=1.96):
    """95% Wilson score interval for a win rate of points / games."""
    rate = points/games
    centre = (rate+z*z/(2*games))/(1+z*z/games)
    half = z*math.sqrt(rate*(1-rate)/games+z*z/(4*games*games))/(1+z*z/games)
    return centre-half, centre+half


def tensors(device, *arrays):
    """numpy observation arrays -> torch tensors on `device` (targets become int64 for gather)."""
    out = []
    for array in arrays:
        tensor = torch.from_numpy(np.ascontiguousarray(array))
        out.append((tensor.long() if tensor.dtype in (torch.int32, torch.int64) else tensor).to(device))
    return out


@torch.no_grad()
def greedy_moves(model, planes, features, targets, legal):
    """The network's most likely legal move in each position (the first one on a tie)."""
    device = next(model.parameters()).device
    logits, _ = model(*tensors(device, planes, features, targets, legal))
    return choose(logits, torch.from_numpy(legal).to(device), greedy=True).cpu().numpy().astype(np.int32)


def load(path, device='cpu'):
    """A model saved as {'shape': ..., 'weights': ...}, in evaluation mode."""
    saved = torch.load(path, weights_only=True, map_location=device)
    model = TronNet(**saved['shape']).to(device)
    model.load_state_dict(saved['weights'])
    return model.eval()


def decide(player, games, turn, planes, features, targets, legal):
    """Moves by `player` (a bot name or a network, greedy) in the games where `turn` is True."""
    if isinstance(player, str):
        return games.bot_moves(BOTS[player], wanted=turn)[turn]
    return greedy_moves(player, planes[turn], features[turn], targets[turn], legal[turn])


def rust_games(seeds):
    return Batch.from_seeds(seeds, threads=6)


def match(player, opponent, first_seed, seeds, new_games=rust_games):
    """`player` against `opponent` on seeds first_seed .. first_seed + seeds - 1, each seed twice (`player`
    as player 0, then as player 1). There are no draws in turn-based light cycles: someone always crashes first.
    `new_games(seeds)` builds the engine: the Rust batch by default, tron.Games for pure Python."""
    count = 2*seeds
    games = new_games([first_seed+game//2 for game in range(count)])
    player_seat = np.tile([0, 1], seeds)
    while not (done := games.done()).all():
        planes, features, targets, legal, to_move = games.observe()
        moves = np.zeros(count, np.int32)
        for mover, turn in ((player, to_move == player_seat), (opponent, to_move != player_seat)):
            turn = turn & ~done
            if turn.any():
                moves[turn] = decide(mover, games, turn, planes, features, targets, legal)
        games.step(moves)
    wins = int(games.scores()[np.arange(count), player_seat].sum())
    plies = games.plies()
    games.close()
    low, high = wilson(wins, count)
    return dict(wins=wins, games=count, rate=wins/count, low=low, high=high, plies=float(plies.mean()))


def describe(result):
    return (f"{result['wins']}-{result['games']-result['wins']} ({result['rate']:.3f} "
            f"[{result['low']:.3f}, {result['high']:.3f}])")

"""Strength of a saved network against the yardsticks: the flood-fill bot, the Voronoi bot (the teacher),
the alpha-beta bot and the behavior-cloned network, on the evaluation seeds (never used in training).

    python evaluate.py ../runs/bc-c16/clone.pt [--seeds 200] [--against flood,voronoi,alphabeta,clone]
"""
import argparse
import json
import time

import torch

from play import describe, load, match

EVALUATION_SEEDS = 5_000_000        # seeds 5,000,000 .. 5,000,000 + n - 1, each played in both seats
CLONE = '../runs/bc/clone.pt'


def evaluate(model, against, seeds, clone_path=CLONE, device='cuda'):
    results = {}
    for name in against:
        opponent = load(clone_path, device) if name == 'clone' else name
        results[name] = match(model, opponent, EVALUATION_SEEDS, seeds)
    return results


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('model')
    parser.add_argument('--seeds', type=int, default=200)
    parser.add_argument('--against', default='flood,voronoi,alphabeta,clone')
    parser.add_argument('--clone', default=CLONE)
    args = parser.parse_args()
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    clock = time.time()
    results = evaluate(load(args.model, device), args.against.split(','), args.seeds, args.clone, device)
    for name, result in results.items():
        print(f'{args.model} vs {name}: {describe(result)}, {result["plies"]:.0f} plies a game')
    print(json.dumps({'model': args.model, 'seconds': round(time.time()-clock, 1), 'results': results}))


if __name__ == '__main__':
    main()

06Small mistakes compound

A clone trained on the teacher's games has never seen the teacher make a mistake, so it has never seen the positions that follow one. When the clone slips, it lands somewhere new, guesses, and may slip again.

Supervised learning assumes the examples it is tested on come from the same distribution as the ones it trained on. A clone breaks that assumption, because its own moves decide which positions it sees next. Stéphane Ross and Andrew Bagnell proved in 2010 what that costs. Suppose the clone picks a different action from its teacher's with probability ε on the positions the teacher visits. Then over a task of T steps its cost can exceed the teacher's by as much as T² × ε, against T × ε if its mistakes didn't change what it saw, and some tasks reach that bound.^[9]

As a formula, with each step's cost between 0 and 1:

In words: a clone that is wrong on a small share ε of the teacher's positions can end up, over T steps, worse than its teacher by up to T² times that share. One factor of T counts the steps; the other counts how long a single mistake can keep the clone off the teacher's path.

The fix is to show the clone its own mistakes, with the teacher's answers. DAgger, from Ross, Geoffrey Gordon and Bagnell in 2011, runs in rounds: train a clone on the teacher's games, run the clone, ask the teacher what it would have done in every position the clone reached, add those labeled positions to the data, and train again on all of it. In its simplest form, the one the authors report "often performs best in practice", every round after the first runs only the latest clone. Counting mistakes as the cost, its guarantee grows like T × ε instead of T² × ε, with ε now measured on the positions the clones reach.^[10] It needs a teacher you can ask about any position, and a bot you wrote is one: run it on the clone's positions inside your simulator.

In a bot contest the same thing happens on a larger scale. A cloned light-cycle bot that turns once where its teacher would have gone straight soon faces wall shapes its teacher never made. The self-play of the next page fixes part of this on its own, because the network trains on positions from its own games. DAgger is the cheaper fix when the clone is all you want: each round is one more recording run through your simulator, with the teacher labeling the clone's positions.

07The action nobody taught

A clone copies what its teacher did, and it also copies what the teacher never did. If your bot never uses an action, cloning teaches the network that the action is always wrong, and self-play may never find out otherwise.

Exploration doesn't rescue an action like that. Self-play explores by sampling moves from the network's own probabilities, and an action at one in a million is almost never sampled, so it almost never earns a reward, good or bad, that training could learn from. The usual remedy is an entropy bonus, which adds the policy's entropy to what training maximizes so the network keeps some spread in its choices.^[11] Entropy measures that spread: −Σ p × ln p over the moves, zero when one move has probability 1. The bonus's push on an action's score is proportional to that action's probability, so it barely touches an action near zero.

As a formula, at temperature 1:

In words: −ln p is how surprising an action is, and the entropy H is the average surprise over all the actions. The push on an action's score is its probability times how much more surprising it is than average. A rare action is very surprising, so the bracket is large, but the leading p shrinks the push to almost nothing.

Two fixes target it more directly. A logit offset adds a fixed amount to the forgotten action's score, enough for sampling to try it again. An entropy term over kinds of action (all moves as one kind, WIND as a second, SHIELD as a third) keeps a whole kind from fading out once it is alive. Neither makes the action good; they let self-play find out whether it is.

Check for forgotten actions before self-play starts: count how often the clone picks each kind of action in its own games, and compare with the opponents you want to beat. An action your teacher never used and strong players use often is the one to watch.

08Pick the teacher

A clone can only be as good as what it copies, and only on positions like the ones it was shown.

Your own bot is usually the best teacher. You can run it on any position, it plays every turn of every game, you know its strength from your measurements, and when the clone disagrees with it you can read its code to see why.

A common question: does handing the trainer a decent hand-written bot help, or should the network start from scratch? It helps, as a starting point. The network begins self-play at the bot's level instead of from random moves, and the bot stays on as the first yardstick the network has to beat. The project's clone above matched its teacher after under five minutes of training, and the next page measures what self-play does from there.

Other players' bots make weaker teachers than they look. You only see their games: one move per turn, in the positions those games happened to reach, and you can't ask them about any other position.

09Keep early self-play near the clone

Self-play starts from the clone, and its first updates can undo the copy before they improve on it.

A common guard is an anchor: add a penalty to the self-play loss that grows as the network's move probabilities move away from the clone's. The usual measure of that distance is the KL divergence, which is zero when two sets of probabilities agree and grows as they differ. A small weight on the penalty lets self-play change moves where winning pays for it and holds the rest of the clone in place.

JOiy, 13th in Back Track King (Legend), trained by PPO self-play with "anchor KL 0.02 fixed on the starting policy" and reports: "Without it, PPO walks away from its starting play in a few hundred updates and never comes back. With it, the run keeps the starting policy's opening shape while improving inside it."^[12] AlphaStar, which started from a network trained on human StarCraft II games, kept a KL term toward that supervised policy in its reinforcement learning loss.^[13] The author's lab used the same 0.02 toward its clone in the Back Track King run, and its trainer fades the penalty to zero over the first 1,500 iterations.^[4] The anchor has a cost too: it holds the network near the clone's mistakes as well as its skill.

10A softmax policy in code

These complete programs run the method end to end on a small scale. They record the flood-fill bot's decisions in 100 games on a 15 × 10 board, train a softmax policy (a score per move computed as a weighted sum of inputs, turned into probabilities by the softmax) with Adam on all the decisions at once, and measure agreement on 100 held-out games. They train two policies on the same recordings. The first sees only the 9 × 9 view around its head, like the widget's network without its hidden layer. The second also gets, for each move, how many cells it gives up against the best legal move: the teacher's own measure, as in the project's network.

All three versions record the same 7,690 training decisions and 7,847 held-out ones, 192 of them contested, and print the same counts. The view-only policy agrees with its teacher on 7,787 held-out decisions (99.2%) but on only 136 of the 192 contested ones (70.8%). With the room input, agreement rises to 7,844 (99.96%) overall and 189 (98.4%) on contested decisions. Contested decisions are 2.4% of this held-out set, so they can move the overall number by 2.4 points at most. Each program asserts these figures: the agreement numbers as lower bounds, and the view-only policy's contested agreement as staying under 80%. The PyTorch version writes no slope formula: autograd computes the slopes, and torch.optim.Adam applies the same update as the C++ and Rust loops.

// Behavior cloning in miniature: record the flood-fill bot's decisions on a 15 x 10
// light-cycle board, train a softmax policy to predict its move, and measure how often
// the clone agrees with its teacher on held-out games, overall and on contested decisions.
// Two policies learn from the same recordings: one sees only the 9 x 9 view around its
// head, the other also sees how many cells each move gives up against the best move.
#include <algorithm>
#include <array>
#include <cassert>
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <limits>
#include <vector>

constexpr int kWidth = 15, kHeight = 10, kCells = kWidth * kHeight;
constexpr int kRadius = 4, kViewSize = (2 * kRadius + 1) * (2 * kRadius + 1);  // 81 cells
constexpr int kMoves = 3;                                  // straight, right turn, left turn
constexpr int kStepX[4] = {0, 1, 0, -1};                   // directions: up, right, down, left
constexpr int kStepY[4] = {-1, 0, 1, 0};
constexpr int kTurn[kMoves] = {0, 1, 3};                   // added to the heading, modulo 4

// SplitMix64: a small seeded generator, so every run plays the same games (the Rust and
// Python versions use the same one and record the same decisions).
struct SplitMix64 {
    std::uint64_t state;
    std::uint64_t next() {
        state += 0x9E3779B97F4A7C15ull;
        std::uint64_t mixed = state;
        mixed = (mixed ^ (mixed >> 30)) * 0xBF58476D1CE4E5B9ull;
        mixed = (mixed ^ (mixed >> 27)) * 0x94D049BB133111EBull;
        return mixed ^ (mixed >> 31);
    }
    int below(int bound) { return static_cast<int>(next() % static_cast<std::uint64_t>(bound)); }
};

struct Game {
    std::array<int, kCells> cells{};                       // 0 free; 1 or 2 a cycle's trail
    std::array<int, 2> head{}, heading{};
    int toMove = 0;                                        // players move one at a time
};

// Two start cells at least 6 apart; each cycle faces the far side of the board.
Game newGame(std::uint64_t seed) {
    SplitMix64 random{seed};
    int first = 0, second = 0;
    do {
        first = random.below(kCells);
        second = random.below(kCells);
    } while (std::abs(first % kWidth - second % kWidth) + std::abs(first / kWidth - second / kWidth) < 6);
    Game game;
    game.head = {first, second};
    game.cells[first] = 1;
    game.cells[second] = 2;
    for (int player = 0; player < 2; ++player)
        game.heading[player] = game.head[player] % kWidth < kWidth / 2 ? 1 : 3;
    return game;
}

bool isFree(const Game& game, int x, int y) {
    return x >= 0 && y >= 0 && x < kWidth && y < kHeight && game.cells[y * kWidth + x] == 0;
}

// The cell a move enters, or -1 if the move crashes.
int targetCell(const Game& game, int chosen) {
    const int direction = (game.heading[game.toMove] + kTurn[chosen]) % 4;
    const int x = game.head[game.toMove] % kWidth + kStepX[direction];
    const int y = game.head[game.toMove] / kWidth + kStepY[direction];
    return isFree(game, x, y) ? y * kWidth + x : -1;
}

// Free cells still reachable after entering `cell`: the room a move leaves. The game is a
// copy, so marking cells as seen leaves the real one alone. The count can overstate: past
// a choke point it adds up both sides, though the cycle can only fill one of them.
int roomAfter(Game game, int cell) {
    game.cells[cell] = 3;
    std::vector<int> queue{cell};
    int count = 0;
    for (std::size_t next = 0; next < queue.size(); ++next) {
        const int x = queue[next] % kWidth, y = queue[next] / kWidth;
        for (int direction = 0; direction < 4; ++direction) {
            const int nextX = x + kStepX[direction], nextY = y + kStepY[direction];
            if (!isFree(game, nextX, nextY)) continue;
            game.cells[nextY * kWidth + nextX] = 3;        // seen
            queue.push_back(nextY * kWidth + nextX);
            ++count;
        }
    }
    return count;
}

// The 9 x 9 view around the head, turned so the cycle always faces up: 1 for a wall, a
// trail or the edge, 0 for a free cell. Turning it lets one set of weights serve every heading.
std::array<double, kViewSize> viewAround(const Game& game) {
    std::array<double, kViewSize> view{};
    const int x = game.head[game.toMove] % kWidth, y = game.head[game.toMove] / kWidth;
    const int heading = game.heading[game.toMove];
    int index = 0;
    for (int along = -kRadius; along <= kRadius; ++along)          // row 0 is farthest ahead
        for (int across = -kRadius; across <= kRadius; ++across) {   // column 0 is farthest left
            int stepX = across, stepY = along;                       // heading up
            if (heading == 1) { stepX = -along; stepY = across; }
            if (heading == 2) { stepX = -across; stepY = -along; }
            if (heading == 3) { stepX = along; stepY = -across; }
            view[index++] = isFree(game, x + stepX, y + stepY) ? 0.0 : 1.0;
        }
    return view;
}

struct Decision {
    std::array<double, kViewSize> view{};
    std::array<double, kMoves> cellsLost{};                // room given up against the best move
    std::array<bool, kMoves> legal{};
    int teacherMove = 0;
    bool contested = false;                                // the legal moves leave different room
};

// Plays the flood-fill bot against itself and records every decision with a choice.
void recordGame(std::uint64_t seed, std::vector<Decision>& decisions) {
    Game game = newGame(seed);
    while (true) {
        Decision decision;
        std::array<int, kMoves> room{};
        int legalCount = 0, best = -1;
        for (int chosen = 0; chosen < kMoves; ++chosen) {
            const int cell = targetCell(game, chosen);
            decision.legal[chosen] = cell >= 0;
            if (cell < 0) continue;
            ++legalCount;
            room[chosen] = roomAfter(game, cell);
            if (best < 0 || room[chosen] > room[best]) best = chosen;   // ties: straight, right, left
        }
        if (legalCount == 0) return;                       // the player to move crashes
        if (legalCount >= 2) {                             // one legal move: nothing to learn
            decision.view = viewAround(game);
            decision.teacherMove = best;
            for (int chosen = 0; chosen < kMoves; ++chosen) {
                if (!decision.legal[chosen]) continue;
                decision.cellsLost[chosen] = room[best] - room[chosen];
                if (room[chosen] != room[best]) decision.contested = true;
            }
            decisions.push_back(decision);
        }
        const int cell = targetCell(game, best);
        game.cells[cell] = game.toMove + 1;
        game.heading[game.toMove] = (game.heading[game.toMove] + kTurn[best]) % 4;
        game.head[game.toMove] = cell;
        game.toMove = 1 - game.toMove;
    }
}

// The policy's numbers in one array: a row of view weights per move, then a bias per move,
// then the weight on cells lost.
constexpr int kBiasIndex = kMoves * kViewSize, kRoomIndex = kBiasIndex + kMoves, kParameters = kRoomIndex + 1;
using Parameters = std::array<double, kParameters>;

// One score (logit) per move: the view weighted by that move's row, plus its bias, minus
// the room weight times the cells the move gives up.
std::array<double, kMoves> logits(const Parameters& parameters, const Decision& decision, bool seesRoom) {
    std::array<double, kMoves> scores{};
    for (int chosen = 0; chosen < kMoves; ++chosen) {
        double score = parameters[kBiasIndex + chosen];
        for (int cell = 0; cell < kViewSize; ++cell) score += parameters[chosen * kViewSize + cell] * decision.view[cell];
        if (seesRoom) score -= parameters[kRoomIndex] * decision.cellsLost[chosen];
        scores[chosen] = score;
    }
    return scores;
}

// Softmax over the legal moves only: an illegal move gets probability exactly 0.
std::array<double, kMoves> probabilities(const std::array<double, kMoves>& scores, const std::array<bool, kMoves>& legal) {
    double largest = -std::numeric_limits<double>::infinity();
    for (int chosen = 0; chosen < kMoves; ++chosen) if (legal[chosen]) largest = std::max(largest, scores[chosen]);
    std::array<double, kMoves> odds{};
    double total = 0.0;
    for (int chosen = 0; chosen < kMoves; ++chosen) {
        if (!legal[chosen]) continue;
        odds[chosen] = std::exp(scores[chosen] - largest);     // subtracting the largest avoids overflow
        total += odds[chosen];
    }
    for (double& value : odds) value /= total;
    return odds;
}

// Slope of the mean cross-entropy, -ln p(teacher's move), for every parameter. For a
// softmax, the slope on a move's score is its probability, minus 1 for the teacher's move.
Parameters slopes(const Parameters& parameters, const std::vector<Decision>& decisions, bool seesRoom) {
    Parameters slope{};
    for (const Decision& decision : decisions) {
        const auto odds = probabilities(logits(parameters, decision, seesRoom), decision.legal);
        for (int chosen = 0; chosen < kMoves; ++chosen) {
            if (!decision.legal[chosen]) continue;           // probability 0, not the teacher's: no slope
            const double error = odds[chosen] - (chosen == decision.teacherMove ? 1.0 : 0.0);
            for (int cell = 0; cell < kViewSize; ++cell) slope[chosen * kViewSize + cell] += error * decision.view[cell];
            slope[kBiasIndex + chosen] += error;
            if (seesRoom) slope[kRoomIndex] -= error * decision.cellsLost[chosen];
        }
    }
    for (double& value : slope) value /= static_cast<double>(decisions.size());
    return slope;
}

// Adam over the whole training set at once. Each parameter steps by the learning rate times
// its running average slope, divided by its running slope size, so the room weight (fed
// counts up to about 100) and the view weights (fed 0 or 1) learn at a similar pace.
Parameters train(const std::vector<Decision>& decisions, bool seesRoom, int steps, double learningRate) {
    Parameters parameters{}, average{}, averageSquare{};   // start from zero: every legal move equally likely
    const double beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8;
    for (int step = 1; step <= steps; ++step) {
        const Parameters slope = slopes(parameters, decisions, seesRoom);
        const double correction1 = 1.0 - std::pow(beta1, step), correction2 = 1.0 - std::pow(beta2, step);
        for (int index = 0; index < kParameters; ++index) {
            average[index] = beta1 * average[index] + (1.0 - beta1) * slope[index];
            averageSquare[index] = beta2 * averageSquare[index] + (1.0 - beta2) * slope[index] * slope[index];
            const double denominator = std::sqrt(averageSquare[index]) / std::sqrt(correction2) + epsilon;
            parameters[index] -= learningRate / correction1 * average[index] / denominator;
        }
    }
    return parameters;
}

struct Agreement { int agreed = 0, decisions = 0, contestedAgreed = 0, contested = 0; };

// How often the policy's most likely legal move is the teacher's move.
Agreement agreement(const Parameters& parameters, const std::vector<Decision>& decisions, bool seesRoom) {
    Agreement result;
    for (const Decision& decision : decisions) {
        const auto scores = logits(parameters, decision, seesRoom);
        int choice = -1;
        for (int chosen = 0; chosen < kMoves; ++chosen)
            if (decision.legal[chosen] && (choice < 0 || scores[chosen] > scores[choice])) choice = chosen;
        const bool agreed = choice == decision.teacherMove;
        result.agreed += agreed;
        ++result.decisions;
        if (decision.contested) { result.contestedAgreed += agreed; ++result.contested; }
    }
    return result;
}

int main() {
    std::vector<Decision> training, heldOut;
    for (std::uint64_t game = 0; game < 100; ++game) recordGame(1000 + game, training);
    for (std::uint64_t game = 0; game < 100; ++game) recordGame(2000 + game, heldOut);   // never trained on
    std::printf("training: %zu decisions with a choice; held out: %zu\n", training.size(), heldOut.size());
    std::array<Agreement, 2> results;
    for (const bool seesRoom : {false, true}) {
        const Parameters policy = train(training, seesRoom, 300, 0.05);
        const Agreement result = agreement(policy, heldOut, seesRoom);
        results[seesRoom] = result;
        std::printf("%s: agrees on %d of %d held-out decisions, %d of %d contested ones\n",
                    seesRoom ? "view + cells lost" : "view only        ",
                    result.agreed, result.decisions, result.contestedAgreed, result.contested);
    }
    // The page quotes these numbers; the program checks them.
    assert(results[0].agreed * 1000 >= 992 * results[0].decisions);           // view only: 99.2% overall,
    assert(results[0].contestedAgreed * 100 < 80 * results[0].contested);      // under 80% on contested ones
    assert(results[1].agreed * 1000 >= 999 * results[1].decisions);           // with cells lost: 99.9% overall
    assert(results[1].contestedAgreed * 1000 >= 984 * results[1].contested);   // and 98.4% on contested ones
}

// Behavior cloning in miniature: record the flood-fill bot's decisions on a 15 x 10
// light-cycle board, train a softmax policy to predict its move, and measure how often
// the clone agrees with its teacher on held-out games, overall and on contested decisions.
// Two policies learn from the same recordings: one sees only the 9 x 9 view around its
// head, the other also sees how many cells each move gives up against the best move.
const WIDTH: i32 = 15;
const HEIGHT: i32 = 10;
const CELLS: usize = (WIDTH * HEIGHT) as usize;
const RADIUS: i32 = 4;
const VIEW_SIZE: usize = ((2 * RADIUS + 1) * (2 * RADIUS + 1)) as usize; // 81 cells
const MOVES: usize = 3; // straight, right turn, left turn
const STEP_X: [i32; 4] = [0, 1, 0, -1]; // directions: up, right, down, left
const STEP_Y: [i32; 4] = [-1, 0, 1, 0];
const TURN: [usize; MOVES] = [0, 1, 3]; // added to the heading, modulo 4

// SplitMix64: a small seeded generator, so every run plays the same games (the C++ and
// Python versions use the same one and record the same decisions).
struct SplitMix64 { state: u64 }

impl SplitMix64 {
    fn next(&mut self) -> u64 {
        self.state = self.state.wrapping_add(0x9E37_79B9_7F4A_7C15);
        let mut mixed = self.state;
        mixed = (mixed ^ (mixed >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
        mixed = (mixed ^ (mixed >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
        mixed ^ (mixed >> 31)
    }
    fn below(&mut self, bound: usize) -> usize { (self.next() % bound as u64) as usize }
}

#[derive(Clone)]
struct Game {
    cells: [u8; CELLS], // 0 free; 1 or 2 a cycle's trail
    head: [usize; 2],
    heading: [usize; 2],
    to_move: usize, // players move one at a time
}

// Two start cells at least 6 apart; each cycle faces the far side of the board.
fn new_game(seed: u64) -> Game {
    let mut random = SplitMix64 { state: seed };
    let (mut first, mut second);
    loop {
        first = random.below(CELLS);
        second = random.below(CELLS);
        let distance = (first as i32 % WIDTH - second as i32 % WIDTH).abs() + (first as i32 / WIDTH - second as i32 / WIDTH).abs();
        if distance >= 6 { break; }
    }
    let mut game = Game { cells: [0; CELLS], head: [first, second], heading: [0; 2], to_move: 0 };
    game.cells[first] = 1;
    game.cells[second] = 2;
    for player in 0..2 { game.heading[player] = if (game.head[player] as i32 % WIDTH) < WIDTH / 2 { 1 } else { 3 }; }
    game
}

fn is_free(game: &Game, x: i32, y: i32) -> bool {
    x >= 0 && y >= 0 && x < WIDTH && y < HEIGHT && game.cells[(y * WIDTH + x) as usize] == 0
}

// The cell a move enters, or None if the move crashes.
fn target_cell(game: &Game, chosen: usize) -> Option<usize> {
    let direction = (game.heading[game.to_move] + TURN[chosen]) % 4;
    let x = game.head[game.to_move] as i32 % WIDTH + STEP_X[direction];
    let y = game.head[game.to_move] as i32 / WIDTH + STEP_Y[direction];
    if is_free(game, x, y) { Some((y * WIDTH + x) as usize) } else { None }
}

// Free cells still reachable after entering `cell`: the room a move leaves. The game is a
// copy, so marking cells as seen leaves the real one alone. The count can overstate: past
// a choke point it adds up both sides, though the cycle can only fill one of them.
fn room_after(mut game: Game, cell: usize) -> i32 {
    game.cells[cell] = 3;
    let mut queue = vec![cell];
    let mut count = 0;
    let mut next = 0;
    while next < queue.len() {
        let (x, y) = (queue[next] as i32 % WIDTH, queue[next] as i32 / WIDTH);
        next += 1;
        for direction in 0..4 {
            let (next_x, next_y) = (x + STEP_X[direction], y + STEP_Y[direction]);
            if !is_free(&game, next_x, next_y) { continue; }
            game.cells[(next_y * WIDTH + next_x) as usize] = 3; // seen
            queue.push((next_y * WIDTH + next_x) as usize);
            count += 1;
        }
    }
    count
}

// The 9 x 9 view around the head, turned so the cycle always faces up: 1 for a wall, a
// trail or the edge, 0 for a free cell. Turning it lets one set of weights serve every heading.
fn view_around(game: &Game) -> [f64; VIEW_SIZE] {
    let mut view = [0.0; VIEW_SIZE];
    let (x, y) = (game.head[game.to_move] as i32 % WIDTH, game.head[game.to_move] as i32 / WIDTH);
    let heading = game.heading[game.to_move];
    let mut index = 0;
    for along in -RADIUS..=RADIUS {        // row 0 is farthest ahead
        for across in -RADIUS..=RADIUS {   // column 0 is farthest left
            let (step_x, step_y) = match heading {
                0 => (across, along),
                1 => (-along, across),
                2 => (-across, -along),
                _ => (along, -across),
            };
            view[index] = if is_free(game, x + step_x, y + step_y) { 0.0 } else { 1.0 };
            index += 1;
        }
    }
    view
}

struct Decision {
    view: [f64; VIEW_SIZE],
    cells_lost: [f64; MOVES], // room given up against the best move
    legal: [bool; MOVES],
    teacher_move: usize,
    contested: bool, // the legal moves leave different room
}

// Plays the flood-fill bot against itself and records every decision with a choice.
fn record_game(seed: u64, decisions: &mut Vec<Decision>) {
    let mut game = new_game(seed);
    loop {
        let mut room = [0; MOVES];
        let mut legal = [false; MOVES];
        let mut best: Option<usize> = None;
        for chosen in 0..MOVES {
            if let Some(cell) = target_cell(&game, chosen) {
                legal[chosen] = true;
                room[chosen] = room_after(game.clone(), cell);
                if best.map_or(true, |current| room[chosen] > room[current]) { best = Some(chosen); } // ties: straight, right, left
            }
        }
        let Some(best) = best else { return }; // the player to move crashes
        if legal.iter().filter(|&&open| open).count() >= 2 { // one legal move: nothing to learn
            let mut decision = Decision { view: view_around(&game), cells_lost: [0.0; MOVES], legal, teacher_move: best, contested: false };
            for chosen in (0..MOVES).filter(|&chosen| legal[chosen]) {
                decision.cells_lost[chosen] = (room[best] - room[chosen]) as f64;
                if room[chosen] != room[best] { decision.contested = true; }
            }
            decisions.push(decision);
        }
        let cell = target_cell(&game, best).expect("the teacher's move is legal");
        game.cells[cell] = game.to_move as u8 + 1;
        game.heading[game.to_move] = (game.heading[game.to_move] + TURN[best]) % 4;
        game.head[game.to_move] = cell;
        game.to_move = 1 - game.to_move;
    }
}

// The policy's numbers in one array: a row of view weights per move, then a bias per move,
// then the weight on cells lost.
const BIAS_INDEX: usize = MOVES * VIEW_SIZE;
const ROOM_INDEX: usize = BIAS_INDEX + MOVES;
const PARAMETERS: usize = ROOM_INDEX + 1;
type Parameters = [f64; PARAMETERS];

// One score (logit) per move: the view weighted by that move's row, plus its bias, minus
// the room weight times the cells the move gives up.
fn logits(parameters: &Parameters, decision: &Decision, sees_room: bool) -> [f64; MOVES] {
    let mut scores = [0.0; MOVES];
    for chosen in 0..MOVES {
        let mut score = parameters[BIAS_INDEX + chosen];
        for cell in 0..VIEW_SIZE { score += parameters[chosen * VIEW_SIZE + cell] * decision.view[cell]; }
        if sees_room { score -= parameters[ROOM_INDEX] * decision.cells_lost[chosen]; }
        scores[chosen] = score;
    }
    scores
}

// Softmax over the legal moves only: an illegal move gets probability exactly 0.
fn probabilities(scores: &[f64; MOVES], legal: &[bool; MOVES]) -> [f64; MOVES] {
    let mut largest = f64::NEG_INFINITY;
    for chosen in (0..MOVES).filter(|&chosen| legal[chosen]) { largest = largest.max(scores[chosen]); }
    let mut odds = [0.0; MOVES];
    let mut total = 0.0;
    for chosen in (0..MOVES).filter(|&chosen| legal[chosen]) {
        odds[chosen] = (scores[chosen] - largest).exp(); // subtracting the largest avoids overflow
        total += odds[chosen];
    }
    for value in odds.iter_mut() { *value /= total; }
    odds
}

// Slope of the mean cross-entropy, -ln p(teacher's move), for every parameter. For a
// softmax, the slope on a move's score is its probability, minus 1 for the teacher's move.
fn slopes(parameters: &Parameters, decisions: &[Decision], sees_room: bool) -> Parameters {
    let mut slope = [0.0; PARAMETERS];
    for decision in decisions {
        let odds = probabilities(&logits(parameters, decision, sees_room), &decision.legal);
        for chosen in 0..MOVES {
            if !decision.legal[chosen] { continue; } // probability 0, not the teacher's: no slope
            let error = odds[chosen] - if chosen == decision.teacher_move { 1.0 } else { 0.0 };
            for cell in 0..VIEW_SIZE { slope[chosen * VIEW_SIZE + cell] += error * decision.view[cell]; }
            slope[BIAS_INDEX + chosen] += error;
            if sees_room { slope[ROOM_INDEX] -= error * decision.cells_lost[chosen]; }
        }
    }
    for value in slope.iter_mut() { *value /= decisions.len() as f64; }
    slope
}

// Adam over the whole training set at once. Each parameter steps by the learning rate times
// its running average slope, divided by its running slope size, so the room weight (fed
// counts up to about 100) and the view weights (fed 0 or 1) learn at a similar pace.
fn train(decisions: &[Decision], sees_room: bool, steps: i32, learning_rate: f64) -> Parameters {
    // Start from zero: every legal move equally likely.
    let (mut parameters, mut average, mut average_square) = ([0.0; PARAMETERS], [0.0; PARAMETERS], [0.0; PARAMETERS]);
    let (beta1, beta2, epsilon) = (0.9_f64, 0.999_f64, 1e-8);
    for step in 1..=steps {
        let slope = slopes(&parameters, decisions, sees_room);
        let (correction1, correction2) = (1.0 - beta1.powi(step), 1.0 - beta2.powi(step));
        for index in 0..PARAMETERS {
            average[index] = beta1 * average[index] + (1.0 - beta1) * slope[index];
            average_square[index] = beta2 * average_square[index] + (1.0 - beta2) * slope[index] * slope[index];
            let denominator = average_square[index].sqrt() / correction2.sqrt() + epsilon;
            parameters[index] -= learning_rate / correction1 * average[index] / denominator;
        }
    }
    parameters
}

#[derive(Default, Clone, Copy)]
struct Agreement { agreed: usize, decisions: usize, contested_agreed: usize, contested: usize }

// How often the policy's most likely legal move is the teacher's move.
fn agreement(parameters: &Parameters, decisions: &[Decision], sees_room: bool) -> Agreement {
    let mut result = Agreement::default();
    for decision in decisions {
        let scores = logits(parameters, decision, sees_room);
        let mut choice: Option<usize> = None;
        for chosen in (0..MOVES).filter(|&chosen| decision.legal[chosen]) {
            if choice.map_or(true, |current| scores[chosen] > scores[current]) { choice = Some(chosen); }
        }
        let agreed = choice == Some(decision.teacher_move);
        result.agreed += agreed as usize;
        result.decisions += 1;
        if decision.contested {
            result.contested_agreed += agreed as usize;
            result.contested += 1;
        }
    }
    result
}

fn main() {
    let (mut training, mut held_out) = (Vec::new(), Vec::new());
    for game in 0..100 { record_game(1000 + game, &mut training); }
    for game in 0..100 { record_game(2000 + game, &mut held_out); } // never trained on
    println!("training: {} decisions with a choice; held out: {}", training.len(), held_out.len());
    let mut results = [Agreement::default(); 2];
    for sees_room in [false, true] {
        let policy = train(&training, sees_room, 300, 0.05);
        let result = agreement(&policy, &held_out, sees_room);
        results[sees_room as usize] = result;
        println!(
            "{}: agrees on {} of {} held-out decisions, {} of {} contested ones",
            if sees_room { "view + cells lost" } else { "view only        " },
            result.agreed, result.decisions, result.contested_agreed, result.contested
        );
    }
    // The page quotes these numbers; the program checks them.
    assert!(results[0].agreed * 1000 >= 992 * results[0].decisions); // view only: 99.2% overall,
    assert!(results[0].contested_agreed * 100 < 80 * results[0].contested); // under 80% on contested ones
    assert!(results[1].agreed * 1000 >= 999 * results[1].decisions); // with cells lost: 99.9% overall
    assert!(results[1].contested_agreed * 1000 >= 984 * results[1].contested); // and 98.4% on contested ones
}

# Behavior cloning in miniature, with PyTorch doing the training: the same games and
# decisions as the C++ and Rust versions, the same two policies, the same checks.
import torch
import torch.nn.functional as F

WIDTH, HEIGHT = 15, 10
CELLS = WIDTH * HEIGHT
RADIUS = 4
VIEW_SIZE = (2 * RADIUS + 1) ** 2  # 81 cells
MOVES = 3  # straight, right turn, left turn
STEP_X = (0, 1, 0, -1)  # directions: up, right, down, left
STEP_Y = (-1, 0, 1, 0)
TURN = (0, 1, 3)  # added to the heading, modulo 4
MASK = (1 << 64) - 1


class SplitMix64:
    """The same seeded generator as the C++ and Rust versions, so the same games."""

    def __init__(self, seed):
        self.state = seed

    def next(self):
        self.state = (self.state + 0x9E3779B97F4A7C15) & MASK
        mixed = self.state
        mixed = ((mixed ^ (mixed >> 30)) * 0xBF58476D1CE4E5B9) & MASK
        mixed = ((mixed ^ (mixed >> 27)) * 0x94D049BB133111EB) & MASK
        return mixed ^ (mixed >> 31)

    def below(self, bound):
        return self.next() % bound


class Game:
    def __init__(self, seed):
        """Two start cells at least 6 apart; each cycle faces the far side of the board."""
        random = SplitMix64(seed)
        while True:
            first, second = random.below(CELLS), random.below(CELLS)
            if abs(first % WIDTH - second % WIDTH) + abs(first // WIDTH - second // WIDTH) >= 6:
                break
        self.cells = [0] * CELLS  # 0 free; 1 or 2 a cycle's trail
        self.head = [first, second]
        self.cells[first], self.cells[second] = 1, 2
        self.heading = [1 if cell % WIDTH < WIDTH // 2 else 3 for cell in self.head]
        self.to_move = 0  # players move one at a time


def is_free(cells, x, y):
    return 0 <= x < WIDTH and 0 <= y < HEIGHT and cells[y * WIDTH + x] == 0


def target_cell(game, chosen):
    """The cell a move enters, or None if the move crashes."""
    direction = (game.heading[game.to_move] + TURN[chosen]) % 4
    x = game.head[game.to_move] % WIDTH + STEP_X[direction]
    y = game.head[game.to_move] // WIDTH + STEP_Y[direction]
    return y * WIDTH + x if is_free(game.cells, x, y) else None


def room_after(game, cell):
    """Free cells still reachable after entering `cell`: the room a move leaves. The count can
    overstate: past a choke point it adds up both sides, though the cycle can only fill one of them."""
    cells = game.cells.copy()  # mark cells as seen on a copy, not on the real game
    cells[cell] = 3
    queue = [cell]
    for current in queue:  # the loop also visits cells appended while it runs
        x, y = current % WIDTH, current // WIDTH
        for direction in range(4):
            next_x, next_y = x + STEP_X[direction], y + STEP_Y[direction]
            if is_free(cells, next_x, next_y):
                cells[next_y * WIDTH + next_x] = 3  # seen
                queue.append(next_y * WIDTH + next_x)
    return len(queue) - 1


def view_around(game):
    """The 9 x 9 view around the head, turned so the cycle always faces up: 1 for a wall, a
    trail or the edge, 0 for a free cell. Turning it lets one set of weights serve every heading."""
    x, y = game.head[game.to_move] % WIDTH, game.head[game.to_move] // WIDTH
    heading = game.heading[game.to_move]
    view = []
    for along in range(-RADIUS, RADIUS + 1):  # row 0 is farthest ahead
        for across in range(-RADIUS, RADIUS + 1):  # column 0 is farthest left
            step_x, step_y = ((across, along), (-along, across), (-across, -along), (along, -across))[heading]
            view.append(0.0 if is_free(game.cells, x + step_x, y + step_y) else 1.0)
    return view


def record_game(seed, decisions):
    """Plays the flood-fill bot against itself and records every decision with a choice."""
    game = Game(seed)
    while True:
        room, legal, best = [0] * MOVES, [False] * MOVES, None
        for chosen in range(MOVES):
            cell = target_cell(game, chosen)
            if cell is not None:
                legal[chosen] = True
                room[chosen] = room_after(game, cell)
                if best is None or room[chosen] > room[best]:  # ties: straight, right, left
                    best = chosen
        if best is None:
            return  # the player to move crashes
        if sum(legal) >= 2:  # one legal move: nothing to learn
            cells_lost = [float(room[best] - room[chosen]) if legal[chosen] else 0.0 for chosen in range(MOVES)]
            contested = any(legal[chosen] and room[chosen] != room[best] for chosen in range(MOVES))
            decisions.append((view_around(game), cells_lost, legal, best, contested))
        cell = target_cell(game, best)
        game.cells[cell] = game.to_move + 1
        game.heading[game.to_move] = (game.heading[game.to_move] + TURN[best]) % 4
        game.head[game.to_move] = cell
        game.to_move = 1 - game.to_move


def tensors(decisions):
    """One tensor per field, one row per decision."""
    views, cells_lost, legal, teacher_move, contested = zip(*decisions)
    return (torch.tensor(views, dtype=torch.float64), torch.tensor(cells_lost, dtype=torch.float64),
            torch.tensor(legal), torch.tensor(teacher_move), torch.tensor(contested))


def logits(policy, data, sees_room):
    """One score per move: the view weighted by that move's row, plus its bias, minus the room
    weight times the cells the move gives up. Illegal moves get -inf: probability exactly 0."""
    weights, bias, room_weight = policy
    views, cells_lost, legal = data[0], data[1], data[2]
    scores = views @ weights.T + bias
    if sees_room:
        scores = scores - room_weight * cells_lost
    return scores.masked_fill(~legal, float('-inf'))


def train(data, sees_room, steps, learning_rate):
    """Adam over the whole training set at once, from zero: every legal move equally likely."""
    policy = [torch.zeros(MOVES, VIEW_SIZE, dtype=torch.float64, requires_grad=True),
              torch.zeros(MOVES, dtype=torch.float64, requires_grad=True),
              torch.zeros((), dtype=torch.float64, requires_grad=True)]
    optimizer = torch.optim.Adam(policy, lr=learning_rate, betas=(0.9, 0.999), eps=1e-8)
    for _ in range(steps):
        loss = F.cross_entropy(logits(policy, data, sees_room), data[3])  # mean of -ln p(teacher's move)
        optimizer.zero_grad()
        loss.backward()  # autograd works out every slope
        optimizer.step()
    return policy


@torch.no_grad()
def agreement(policy, data, sees_room):
    """How often the policy's most likely legal move is the teacher's move."""
    agreed = logits(policy, data, sees_room).argmax(1) == data[3]  # ties go to the first move
    contested = data[4]
    return int(agreed.sum()), len(agreed), int(agreed[contested].sum()), int(contested.sum())


training, held_out = [], []
for game in range(100):
    record_game(1000 + game, training)
for game in range(100):
    record_game(2000 + game, held_out)  # never trained on
print(f"training: {len(training)} decisions with a choice; held out: {len(held_out)}")
training, held_out = tensors(training), tensors(held_out)
results = []
for sees_room in (False, True):
    policy = train(training, sees_room, 300, 0.05)
    agreed, decisions, contested_agreed, contested = agreement(policy, held_out, sees_room)
    results.append((agreed, decisions, contested_agreed, contested))
    name = "view + cells lost" if sees_room else "view only        "
    print(f"{name}: agrees on {agreed} of {decisions} held-out decisions, {contested_agreed} of {contested} contested ones")
# The page quotes these numbers; the program checks them.
assert results[0][0] * 1000 >= 992 * results[0][1]  # view only: 99.2% overall,
assert results[0][2] * 100 < 80 * results[0][3]  # under 80% on contested ones
assert results[1][0] * 1000 >= 999 * results[1][1]  # with cells lost: 99.9% overall
assert results[1][2] * 1000 >= 984 * results[1][3]  # and 98.4% on contested ones

11Where cloning goes wrong

Recording one seat, one opponent or a handful of maps. The clone learns the positions it was shown. A teacher that only played itself from one seat shows half the openings and one style of opponent.

Measuring on the training games. Agreement on the games you trained on also rewards memorizing them. Hold out seeds and keep them out of every training run.

Labels that don't line up with the outputs. If a printed command maps to the wrong move index, training still runs, and agreement still rises on the moves that happen to line up. Check the mapping by hand on a few positions: replay the recorded moves through your engine and compare with the teacher's own game.

Forgetting the mask at play time. Training with the mask never pushes illegal moves down, because the mask removes them from every loss. Without the mask, the project's clone would have chosen an illegal move in 15,590 of the 33,266 decisions (47%) of its games against itself.^[5]

A teacher with habits you don't want. The clone copies the habits along with the skill: an action the teacher never uses, a tie-break that drifts toward walls. Self-play has to unlearn them, and an action near zero probability may never come back without help (section 07).

Expecting the clone to beat its teacher. Nothing in cloning rewards winning. A clone that plays level has done its job; passing the teacher is self-play's job.

12What's next

Self-Play Reinforcement Learning starts from this clone. It plays the clone against itself and against a pool of its earlier versions, scores each game's result, and makes the moves that led to wins more likely with PPO, the method RamzaBeoulve trained his cloned network with. It measures how the clone's strength grows with the number of self-play games, against its teacher, against the clone itself and against the alpha-beta bot.

The project so far, with this step's files in teal:

project/
├── engine/                    Rust: rules, bots, features, network, C interface
│   ├── Cargo.toml
│   └── src/
│       ├── board.rs           grid, distances, room, Voronoi territory
│       ├── game.rs            rules, seeded starts, CodinGame frames
│       ├── bots.rs            flood-fill and Voronoi bots (the teacher)
│       ├── alphabeta.rs       the alpha-beta bot (a yardstick)
│       ├── features.rs        what the network sees
│       ├── net.rs             the network's forward pass in Rust
│       ├── ffi.rs             the C interface Python calls (Building the Trainer)
│       └── lib.rs
├── train/
│   ├── tron_env.py            drives the engine from Python (Building the Trainer)
│   ├── model.py               the network in PyTorch
│   ├── parity.py              Rust and PyTorch choose the same moves
│   ├── record.py              records the teacher's decisions
│   ├── bc.py                  behavior cloning
│   ├── play.py                matches, both seats, Wilson intervals
│   └── evaluate.py            a network against the yardsticks
├── data/
│   ├── bc-train.npz           496,402 decisions from 3,000 games
│   └── bc-heldout.npz         49,864 decisions from 300 games
└── runs/bc/
    ├── clone.pt               the clone's weights
    └── bc.json                its training log

13Sources

Quotes are from each author's own text. Numbers marked as the project's were measured with the code on this page; numbers marked as the author's lab are first-hand records from one lab and aren't general results. Widget readouts are computed in the page from seeded games.

1. Michael Bain, Claude Sammut, 1999. A Framework for Behavioural Cloning, in Machine Intelligence 15, Oxford University Press, pp. 103–129. Cloning as induction over logs of skilled operators, and the brittleness of early clones.
2. Dean A. Pomerleau, 1989. ALVINN: An Autonomous Land Vehicle in a Neural Network, Advances in Neural Information Processing Systems 1, pp. 305–313. Training on 1,200 simulated road images, and the need to show the network how to recover from mistakes.
3. RamzaBeoulve, 2026. 4th place solution, CodinGame forum, Summer Challenge 2026 feedback and strategies, post 10. Behavior cloning of his heuristic bot as the start of PPO training.
4. Author's lab records, 2026 (private repository). Back Track King: parsing the bots' commands into the network's slots and the cloning warm start from the contest bot and two variants, games/backtrackking/nn_bc.py and docs/provenance.md; the KL anchor of 0.02, docs/nn-training.md, and its fade over 1,500 iterations, framework/rl/trainer.py; the replay-cloned stand-ins of the top six, experiments/runs/standins-20261024/yardstick/verdict.json and docs/research-training.md. Spring Challenge 2022: the SHIELD diagnosis, offset and self-play counts, experiments/spring2022-gold-20261029.json and games/spring2022/README.md; the Legend bot's spells over 125 Gold battles, docs/postmortem-lessons.md.
5. Build a Legend Bot running project, step 2, run on October 30, 2026 on one machine (Intel Core i9-11900KF; NVIDIA RTX 3080 Ti shared with other training jobs). Recording, training and evaluation logs (runs/logs/record.log, bc.log, eval-clone.log), the first network design's runs and matches, the room diagnosis on 6,000 held-out decisions, and the mask check on the clone's own games. The code is on this page, Neural Networks for Game Bots and Building the Trainer.
6. Christopher M. Bishop, 2006. Pattern Recognition and Machine Learning, Springer, sections 4.3.4 and 5.2. The cross-entropy error for choosing one of several classes.
7. Diederik P. Kingma, Jimmy Lei Ba, 2015. Adam: A Method for Stochastic Optimization, ICLR 2015. Per-weight step sizes from running averages of each slope and its square.
8. CodinGame. Tron Battle, game statement. The 30 × 20 grid and 2 to 4 cycles per battle.
9. Stéphane Ross, J. Andrew Bagnell, 2010. Efficient Reductions for Imitation Learning, AISTATS 2010, PMLR 9:661–668. Theorem 2.1: the T²ε bound for a clone trained on the teacher's states, and that it is tight.
10. Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, 2011. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011, PMLR 15:627–635. DAgger, its simplest form, and its bound linear in T.
11. Volodymyr Mnih et al., 2016. Asynchronous Methods for Deep Reinforcement Learning, ICML 2016. Adding the policy's entropy to the objective to discourage early convergence to one action.
12. JOiy, 2026. JOiy's bot: Summer challenge 2026 (gist, last updated 2026-10-10). The KL anchor to the starting policy and what it prevented.
13. Oriol Vinyals et al., 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature 575, 350–354 (authors' manuscript). Supervised training on human games first, then a KL term toward that supervised policy during reinforcement learning.