Measuring Bot Strength

01A win rate is a measurement

The new version won 6 of 10 games against the old one. If the two versions were exactly as strong, the new one would still win 6 or more of 10 games 37.7% of the time. If it were truly better, winning 55% of its games, it would do so 50.4% of the time. Ten games can't tell those two situations apart.

Think of each game as a weighted coin flip. The new version has a true win rate p against the old one: the fraction of games it would win over thousands of games. Ten games give you an estimate of p, the observed rate, and the estimate wobbles. How much it wobbles follows the binomial distribution: the chances of every possible number of wins in n independent games with the same win chance.

Draws count as half a win. That's how chess scores them: the US Chess rating system counts 1 for a win, 0.5 for a draw and 0 for a loss.^[1] With draws you're measuring a points rate, and the intervals below treat it like a win rate. Draws make results less variable than coin flips, so the intervals come out slightly wider than they need to be, which errs on the safe side.

An error bar turns the measurement into something you can act on. A confidence interval is a range of true win rates that fit what you observed; a 95% interval comes from a procedure that covers the true rate in 95% of experiments. For win rates the Wilson score interval is a good default. It stays inside 0 to 1, and it holds up for small samples and for rates near 0 or 1, where the textbook "rate ± 1.96 standard errors" breaks down.^[2]

The Wilson interval for w wins (or points) in n games computes a center and a half-width from the observed rate. With z = 1.96 it gives 95% confidence:

In words: start from the observed rate p̂ = w / n and pull it slightly toward 0.5, by an amount that only counts when n is small. The half-width shrinks like 1 / √n, so four times the games halves the error bar. The interval runs from center minus half-width to center plus half-width.

02How many games a decision needs

Before you run a test, decide what gain you're looking for. A change worth 5 points, a 55% win rate against the old version, needs a very different test from one worth 20 points.

Two settings shape the test. The confidence level controls false alarms: at 95%, a version that is no better still looks different 5% of the time, better in half of those cases and worse in the rest. The power is the chance of detecting a real gain of the size you named; 80% is a common choice. Then the games needed follow, and they grow with the square of 1 over the gain: halve the gain you want to see, and you need four times the games.

The formula is the normal approximation NIST gives for testing a proportion, with both win rates near 50%. It tells you how many games to play before you play them:^[3]

In words: add the two normal-table values and square the sum, multiply by the variance of one game, and divide by the squared gain. The variance term is why games near 50% are the most expensive to call: p(1 − p) is largest there.

Often you can't run a duel. You compare two versions by how each does against the same opponents, the boss or a panel of older bots. Then the gain is the difference between two noisy rates, and the variance of a difference is the sum of both variances: twice the games, 1,570 per version for 5 points, if each version plays its own maps. Playing both versions on the same maps helps, because a hard map then hurts both alike and drops out of the difference. If the two versions' results on a map have correlation ρ, the variance shrinks by a factor (1 − ρ): at ρ = 0.5 you're back to 785 per version.

A common question, once the numbers come in: "Should we swap to 800 games instead of 96?" For a duel at 95% confidence and 80% power, 96 games only detect gains of about 14 points, and 800 detect about 5. Comparing two versions on separate maps needs more: 96 games per version detect about 20 points, 800 about 7. So for a change worth a few points, 800 games is the right direction and still not enough.

The same arithmetic explains why progress near the top seems to stall. Late gains are small (a few points at a time in the author's lab's late training), and every halving of the gain you need to see costs four times the games. A gain of 2 points needs nearly 5,000 games in a duel, which is why trainers screen briefly and confirm at length (section 07).^[4]

03Same maps, both seats

Maps differ. On one map your bot starts boxed into a corner; on another it has room to grow. Seats can differ too: in some games moving first, or starting on one side, is worth points. If the new version happens to draw the easy maps and the old one the hard ones, map luck can hide a real gain or invent one.

The fix is a paired comparison: play both versions on the same maps, in both seats, and compare them map by map. A map both versions lose, or both win, says nothing about which one is better, so map difficulty drops out of the comparison.

Swapping seats on every map cancels seat bias the same way. Local tools can do it for you: cg-brutaltester's -s option repeats each game on the same map "with positions changed".^[5] CodinGame's own arena does something similar. In 2016 a staff member wrote that for games where seat order makes a difference, it plays the seat permutations and counts them as a single result.^[6]

Pairing changes how you count, too. Two games on the same map aren't independent, so an interval computed as if they were comes out too narrow. Make the map the unit: compare per-map results, and when you resample (section 05), resample whole maps with both seats together. The author's lab records the per-game Wilson interval for reference only, because "two seats on one map are correlated", and decides on map-level intervals.^[4]

04The mirror trap

You change one tie-break in your bot, play the new version against the old one from mirrored starting positions, and it wins 80 of 100 games, which looks decisive. Here it measures almost nothing.

A bot is deterministic when it always makes the same move in the same position. In a game where both players move at the same time, two copies of one deterministic bot, started from mirror-image positions, make mirror-image moves. They play the same game, rotated, and it ends level every time. Change anything, even a rule that only decides exact ties, and the symmetry breaks: the games stop being level, and one side starts collecting wins. Those wins come from breaking the mirror. They say little about strength against anyone else.

The widget plays light cycles, the game behind CodinGame's Tron Battle: each bot steers a cycle that leaves a wall behind it, and a bot that runs into a wall loses. Here both cycles move at the same time, which is what makes a perfect mirror possible. The original bot moves wherever it ends up with the biggest lead in cells it can reach before its opponent (its Voronoi territory, from Evaluation Functions).

A common question when a learning project stalls: "We read how first place did it, so why is ours not improving?" In the author's lab it had two causes. The learner could only re-rank the original bot's own candidate plans, so it could never play a move the original hadn't proposed (Neural Networks for Game Bots covers that design). And its benchmark was this mirror, so the numbers that should have exposed the cap looked like progress. The tests against the two variants did expose it.

Check your own game before you trust a head-to-head test: play the original against an exact copy of itself, from identical or mirrored starts. If every game ends level, your game mirrors, and results against the original measure the mirror.

05Versions that beat each other in a circle

Version 2 beats version 1. Version 3 beats version 2. Then version 1 beats version 3. Which one do you ship?

Strength is non-transitive when results go round in a circle, like rock, paper, scissors. Czarnecki and colleagues hypothesize that real games are shaped like a spinning top: an upright axis of transitive strength, and around it cycles whose number depends on the level of strength. Their simplest example is "a cycle of length 3 that one finds e.g. in the Rock Paper Scissors game."^[7]

Bot versions pick up cycles easily. A change tuned against one opponent learns to counter that opponent's habits, and the counter can be worth nothing, or less, against anyone else.

The fix is a panel: several bots that play differently, a round robin in which every bot plays every other on every map in both seats, and one rating per bot fitted to all the games at once. The Bradley–Terry model gives each bot a strength s and says bot i beats bot j with a chance that depends only on their two strengths.^[8]

Written out, the model turns two strengths into a winning chance, and a logarithm turns strengths into the familiar Elo points:

In words: a bot's chance of beating another is its share of their combined strength. Measured in Elo points, that's the logistic curve US Chess uses for expected scores: equal ratings give 0.5, a 191-point edge gives 0.75.^[1]

With many bots, no single pairing decides the ratings: fitting finds the strengths that best explain every result at once. A simple iteration finds them. Set each bot's strength to its total wins divided by the sum, over its opponents, of games played divided by the two bots' current combined strength; repeat until nothing moves. That's Zermelo's algorithm from 1929, which Lange, Hunter and Yang showed in 2000 is an MM (minorization–maximization) algorithm. It converges to the best fit as long as no group of bots won every game, or lost every game, against the rest.^[8] Section 09 implements it.

To put error bars on the ratings, use the bootstrap: redraw the data with replacement many times, refit each time, and read the spread of the refitted ratings.^[9] Redraw whole maps or whole rounds, never single games, so that games which share a map stay together.

One more panel member pays for itself: a twin, an exact copy of your current bot under another name, perturbed only through its random seed. Its rating gap from the original is pure noise, so it measures the noise floor, the smallest difference your panel can resolve.

06Tune on some maps, test on others

Suppose you try ten versions of a change, test each on 200 games, and keep the best. If all ten were exactly as strong as the original, the one you keep still measures about 0.554 on average, and on fresh maps it drops back to 0.500. You kept the luckiest.

Picking the best of several noisy measurements picks the one with the luckiest noise. The winner's score overstates its strength, and the overstatement grows with the number of versions you compare and shrinks with the games each one gets.

The fix is to give maps fixed jobs. A seed ledger splits the seed space into ranges: training games, validation games for tuning and for promotion gates, and final tests. A final test uses fresh seeds once, for a decision you committed to before looking, and those seeds are then spent. Evaluating Models describes the same three-way split for machine-learning datasets.

07Screen, then confirm

A trainer that improves a bot produces a new candidate every few minutes, and a gate decides which candidates become the new champion. A gate is a test you run hundreds of times, so its false alarms add up.

A single short test makes a leaky gate. A candidate exactly as strong as the champion passes a 512-game test at 0.52 or better 17.7% of the time; across 100 such candidates that's about 18 false promotions, and each one leaves the champion weaker than the record says. A second, longer test on maps the candidate wasn't selected on stops most of them.

A common question once a gate is running: "Why are bots promoted that don't beat the old baseline?" A gate compares each candidate with the current champion on validation maps, and the old baseline is a different opponent. Three things let a promoted bot lose to it. Some promotions are false alarms, like gate A's 8 in the widget. Strength isn't transitive (section 05), so beating the champion doesn't imply beating the baseline. And a chain of small real gains against each new champion can drift away from what the baseline punishes, because a lineage that only plays itself measures progress against itself.

A related one: "Why does the vs-baseline percentage go down sometimes?" Every benchmark is a sample. The lab's baseline check used 800 games, about ±3.5 points at 95%, so a 2-point drop between two checks is noise. The learner can also improve against its champion while slipping against the baseline, for the reasons above. A drop that persists across several checks is the one to act on.

And: "How often should I generate a new baseline?" When a live result confirms that a candidate is stronger. Then keep the older baselines in the panel, so a regression against any of them still shows up.

08What only the live ladder can answer

Your local panel says the new version is clearly better. You submit, and the score barely moves, or drops. Both can be true at once: local tests measure the opponents you have, and the ladder is made of opponents you don't have.

Local opponents are usually your own lineage: earlier versions, variants and twins. The ladder holds other people's bots, built on other ideas, and a change can help against your lineage while doing nothing, or harm, against theirs. On top of that, a live score carries the rating noise from How Bot Contests Work: in the author's lab, identical code moved by about a point between submissions.

A common question after a big local win: "The new bot beats the old Legend bot 100% of the time?" That's a local result against one opponent. The lab's first Legend network beat the original contest bot 500–0–12 on 256 fresh maps in both seats. That says it's far stronger than that bot. It says little about where it lands on a ladder of bots built differently.

Another: "The last two submissions kept our rank but dropped our score." In the author's lab, identical code moved by about a point from one submission to the next, so a drop smaller than that is no evidence the bot got worse.

And the hardest one: "Realistically, any chance of moving up? What are the odds it's actually better?" Compare the gap with the noise band. On October 24, 2026, the three bots just above the lab's sat 0.39 to 0.58 points higher, well inside the spread of identical code: a resubmission could move it a place or two either way by luck. The top two stood about 5 and 8 points higher on October 26, several times the noise band, which only a real improvement closes. None of the dozen or so variants the lab tried after that scored detectably better.^[4]

When you do test live, decide the rule before the result exists: how many games, which opponents count, and what score means better, worse or no difference. Writing that down first is pre-registration, and it stops you from reading noise as a result after the fact.

09The statistics in code

These complete programs implement the three tools of this page: the Wilson interval (section 01), the games-needed formulas (section 02), and a Bradley–Terry fit by Zermelo's iteration (section 05), with an optional pseudo-draw prior that keeps an unbeaten bot's rating finite. The checks in main use numbers worked out above: 55 of 100 gives 0.452 to 0.644; a 5-point gain needs 785 games in a duel and 1,570 per version on separate maps; 75 to 25 is 190.8 Elo; a rock-paper-scissors cycle rates every bot 0. The last check is the property that defines the best fit: at the fitted ratings, every bot's predicted points equal the points it took in the games.

// measure.cpp: error bars, games needed and Bradley-Terry ratings for bot matches.
// Build: g++ -std=c++20 -O2 measure.cpp -o measure && ./measure
#include <algorithm>
#include <cassert>
#include <cmath>
#include <cstdio>
#include <vector>

struct Interval { double low, high; };

// Wilson score interval for `points` won in `games` games (a draw is half a point).
// z = 1.959964 gives 95% confidence.
Interval wilson(double points, double games, double z = 1.959964) {
    double rate = points / games, zSquared = z * z;
    double center = (rate + zSquared / (2 * games)) / (1 + zSquared / games);
    double half = z * std::sqrt(rate * (1 - rate) / games + zSquared / (4 * games * games)) /
                  (1 + zSquared / games);
    return {std::max(0.0, center - half), std::min(1.0, center + half)};
}

// Normal-table values: 95% two-sided confidence and 80% power.
constexpr double kZConfidence = 1.959964, kZPower = 0.841621;

// Games for a duel (new version against old) to detect a win rate of 0.5 + gain.
int gamesForDuel(double gain) {
    double zSum = kZConfidence + kZPower;
    return static_cast<int>(std::ceil(zSum * zSum * 0.25 / (gain * gain) - 1e-9));
}

// Games per version when both versions play the same opponents and you compare their win rates.
// correlation is 0 when each version gets its own maps, higher when they share maps and seats.
int gamesPerVersion(double gain, double correlation) {
    double zSum = kZConfidence + kZPower;
    double variance = 2 * 0.25 * (1 - correlation);  // a difference of two noisy rates
    return static_cast<int>(std::ceil(zSum * zSum * variance / (gain * gain) - 1e-9));
}

// The logistic link between an Elo gap and the expected score (US Chess's formula).
double expectedScore(double eloGap) { return 1 / (1 + std::pow(10.0, -eloGap / 400)); }

using Table = std::vector<std::vector<double>>;  // table[i][j] = points player i took from player j

// Bradley-Terry: player i beats j with chance strength[i] / (strength[i] + strength[j]).
// Fitted by Zermelo's iteration (an MM algorithm). prior adds that many points each way to every
// pair that met, so a player who never lost or never won keeps a finite rating.
// Returns Elo ratings with the panel's average at 0.
std::vector<double> bradleyTerry(const Table& points, double prior = 0) {
    std::size_t players = points.size();
    std::vector<double> strength(players, 1.0);
    for (int iteration = 0; iteration < 100000; ++iteration) {
        std::vector<double> next(players);
        for (std::size_t i = 0; i < players; ++i) {
            double won = 0, weight = 0;
            for (std::size_t j = 0; j < players; ++j) {
                double games = points[i][j] + points[j][i];
                if (i == j || games == 0) continue;
                won += points[i][j] + prior;
                weight += (games + 2 * prior) / (strength[i] + strength[j]);
            }
            next[i] = won / weight;  // the strength at which i's predicted wins equal its real wins, others held fixed
        }
        double logMean = 0;  // ratings only mean something relative to each other: pin the average
        for (double value : next) logMean += std::log(value);
        logMean /= static_cast<double>(players);
        double change = 0;
        for (std::size_t i = 0; i < players; ++i) {
            next[i] /= std::exp(logMean);
            change = std::max(change, std::fabs(std::log(next[i] / strength[i])));
        }
        strength = next;
        if (change < 1e-13) break;
    }
    std::vector<double> elo(players);
    for (std::size_t i = 0; i < players; ++i) elo[i] = 400 * std::log10(strength[i]);
    return elo;
}

bool near(double value, double target, double tolerance) { return std::fabs(value - target) < tolerance; }

int main() {
    // 55 wins in 100 games: the interval still includes 0.5.
    Interval fiftyFive = wilson(55, 100);
    assert(near(fiftyFive.low, 0.45245, 1e-4) && near(fiftyFive.high, 0.64385, 1e-4));
    assert(fiftyFive.low < 0.5);
    // 6 wins in 10 games: anything from about 0.31 to 0.83.
    Interval sixOfTen = wilson(6, 10);
    assert(near(sixOfTen.low, 0.31267, 1e-4) && near(sixOfTen.high, 0.83182, 1e-4));

    // A five-point gain at 95% confidence and 80% power.
    assert(gamesForDuel(0.05) == 785);
    assert(gamesPerVersion(0.05, 0.0) == 1570);  // separate maps
    assert(gamesPerVersion(0.05, 0.5) == 785);   // shared maps, correlation 0.5

    // Two bots, 75 to 25: three times as strong, 400 * log10(3) = 190.85 Elo apart.
    std::vector<double> pair = bradleyTerry({{0, 75}, {25, 0}});
    assert(near(pair[0] - pair[1], 190.849, 1e-3));
    assert(near(expectedScore(pair[0] - pair[1]), 0.75, 1e-9));

    // Rock, paper, scissors: each bot beats the next 70 to 30. No bot is stronger overall.
    std::vector<double> cycle = bradleyTerry({{0, 70, 30}, {30, 0, 70}, {70, 30, 0}});
    for (double rating : cycle) assert(near(rating, 0, 1e-6));

    // At the fitted ratings, every bot's predicted points equal the points it really took.
    Table panel = {{0, 30, 12, 9}, {10, 0, 22, 14}, {8, 18, 0, 25}, {11, 6, 15, 0}};
    std::vector<double> elo = bradleyTerry(panel);
    for (std::size_t i = 0; i < panel.size(); ++i) {
        double actual = 0, predicted = 0;
        for (std::size_t j = 0; j < panel.size(); ++j) {
            if (i == j) continue;
            actual += panel[i][j];
            predicted += (panel[i][j] + panel[j][i]) * expectedScore(elo[i] - elo[j]);
        }
        assert(near(predicted, actual, 1e-6));
    }

    // A bot that never lost: without a prior its rating runs off to infinity; one pseudo-draw keeps it finite.
    std::vector<double> unbeaten = bradleyTerry({{0, 20}, {0, 0}}, 0.5);
    assert(std::isfinite(unbeaten[0]) && near(unbeaten[0] - unbeaten[1], 400 * std::log10(41.0), 1e-3));

    std::printf("55/100: [%.4f, %.4f]  6/10: [%.4f, %.4f]\n", fiftyFive.low, fiftyFive.high, sixOfTen.low,
                sixOfTen.high);
    std::printf("games: duel %d, per version %d (separate maps) or %d (shared, 0.5)\n", gamesForDuel(0.05),
                gamesPerVersion(0.05, 0.0), gamesPerVersion(0.05, 0.5));
    std::printf("75-25: %+.1f Elo  panel: %+.1f %+.1f %+.1f %+.1f\n", pair[0] - pair[1], elo[0], elo[1], elo[2],
                elo[3]);
    std::puts("all checks passed");
}

// measure.rs: error bars, games needed and Bradley-Terry ratings for bot matches.
// Build: rustc --edition 2021 -O measure.rs && ./measure

struct Interval {
    low: f64,
    high: f64,
}

// Wilson score interval for `points` won in `games` games (a draw is half a point).
// z = 1.959964 gives 95% confidence.
fn wilson(points: f64, games: f64, z: f64) -> Interval {
    let (rate, z_squared) = (points / games, z * z);
    let center = (rate + z_squared / (2.0 * games)) / (1.0 + z_squared / games);
    let half = z * (rate * (1.0 - rate) / games + z_squared / (4.0 * games * games)).sqrt()
        / (1.0 + z_squared / games);
    Interval { low: (center - half).max(0.0), high: (center + half).min(1.0) }
}

// Normal-table values: 95% two-sided confidence and 80% power.
const Z_CONFIDENCE: f64 = 1.959964;
const Z_POWER: f64 = 0.841621;

// Games for a duel (new version against old) to detect a win rate of 0.5 + gain.
fn games_for_duel(gain: f64) -> u32 {
    let z_sum = Z_CONFIDENCE + Z_POWER;
    (z_sum * z_sum * 0.25 / (gain * gain) - 1e-9).ceil() as u32
}

// Games per version when both versions play the same opponents and you compare their win rates.
// correlation is 0 when each version gets its own maps, higher when they share maps and seats.
fn games_per_version(gain: f64, correlation: f64) -> u32 {
    let z_sum = Z_CONFIDENCE + Z_POWER;
    let variance = 2.0 * 0.25 * (1.0 - correlation); // a difference of two noisy rates
    (z_sum * z_sum * variance / (gain * gain) - 1e-9).ceil() as u32
}

// The logistic link between an Elo gap and the expected score (US Chess's formula).
fn expected_score(elo_gap: f64) -> f64 {
    1.0 / (1.0 + 10f64.powf(-elo_gap / 400.0))
}

type Table = Vec<Vec<f64>>; // table[i][j] = points player i took from player j

// Bradley-Terry: player i beats j with chance strength[i] / (strength[i] + strength[j]).
// Fitted by Zermelo's iteration (an MM algorithm). prior adds that many points each way to every
// pair that met, so a player who never lost or never won keeps a finite rating.
// Returns Elo ratings with the panel's average at 0.
fn bradley_terry(points: &Table, prior: f64) -> Vec<f64> {
    let players = points.len();
    let mut strength = vec![1.0; players];
    for _ in 0..100_000 {
        let mut next = vec![0.0; players];
        for i in 0..players {
            let (mut won, mut weight) = (0.0, 0.0);
            for j in 0..players {
                let games = points[i][j] + points[j][i];
                if i == j || games == 0.0 {
                    continue;
                }
                won += points[i][j] + prior;
                weight += (games + 2.0 * prior) / (strength[i] + strength[j]);
            }
            next[i] = won / weight; // the strength at which i's predicted wins equal its real wins, others held fixed
        }
        // Ratings only mean something relative to each other: pin the average.
        let log_mean = next.iter().map(|value: &f64| value.ln()).sum::<f64>() / players as f64;
        let mut change: f64 = 0.0;
        for i in 0..players {
            next[i] /= log_mean.exp();
            change = change.max((next[i] / strength[i]).ln().abs());
        }
        strength = next;
        if change < 1e-13 {
            break;
        }
    }
    strength.iter().map(|value| 400.0 * value.log10()).collect()
}

fn near(value: f64, target: f64, tolerance: f64) -> bool {
    (value - target).abs() < tolerance
}

fn main() {
    // 55 wins in 100 games: the interval still includes 0.5.
    let fifty_five = wilson(55.0, 100.0, 1.959964);
    assert!(near(fifty_five.low, 0.45245, 1e-4) && near(fifty_five.high, 0.64385, 1e-4));
    assert!(fifty_five.low < 0.5);
    // 6 wins in 10 games: anything from about 0.31 to 0.83.
    let six_of_ten = wilson(6.0, 10.0, 1.959964);
    assert!(near(six_of_ten.low, 0.31267, 1e-4) && near(six_of_ten.high, 0.83182, 1e-4));

    // A five-point gain at 95% confidence and 80% power.
    assert_eq!(games_for_duel(0.05), 785);
    assert_eq!(games_per_version(0.05, 0.0), 1570); // separate maps
    assert_eq!(games_per_version(0.05, 0.5), 785); // shared maps, correlation 0.5

    // Two bots, 75 to 25: three times as strong, 400 * log10(3) = 190.85 Elo apart.
    let pair = bradley_terry(&vec![vec![0.0, 75.0], vec![25.0, 0.0]], 0.0);
    assert!(near(pair[0] - pair[1], 190.849, 1e-3));
    assert!(near(expected_score(pair[0] - pair[1]), 0.75, 1e-9));

    // Rock, paper, scissors: each bot beats the next 70 to 30. No bot is stronger overall.
    let cycle = bradley_terry(&vec![vec![0.0, 70.0, 30.0], vec![30.0, 0.0, 70.0], vec![70.0, 30.0, 0.0]], 0.0);
    assert!(cycle.iter().all(|&rating| near(rating, 0.0, 1e-6)));

    // At the fitted ratings, every bot's predicted points equal the points it really took.
    let panel: Table = vec![
        vec![0.0, 30.0, 12.0, 9.0],
        vec![10.0, 0.0, 22.0, 14.0],
        vec![8.0, 18.0, 0.0, 25.0],
        vec![11.0, 6.0, 15.0, 0.0],
    ];
    let elo = bradley_terry(&panel, 0.0);
    for i in 0..panel.len() {
        let (mut actual, mut predicted) = (0.0, 0.0);
        for j in 0..panel.len() {
            if i == j {
                continue;
            }
            actual += panel[i][j];
            predicted += (panel[i][j] + panel[j][i]) * expected_score(elo[i] - elo[j]);
        }
        assert!(near(predicted, actual, 1e-6));
    }

    // A bot that never lost: without a prior its rating runs off to infinity; one pseudo-draw keeps it finite.
    let unbeaten = bradley_terry(&vec![vec![0.0, 20.0], vec![0.0, 0.0]], 0.5);
    assert!(unbeaten[0].is_finite() && near(unbeaten[0] - unbeaten[1], 400.0 * 41f64.log10(), 1e-3));

    println!(
        "55/100: [{:.4}, {:.4}]  6/10: [{:.4}, {:.4}]",
        fifty_five.low, fifty_five.high, six_of_ten.low, six_of_ten.high
    );
    println!(
        "games: duel {}, per version {} (separate maps) or {} (shared, 0.5)",
        games_for_duel(0.05),
        games_per_version(0.05, 0.0),
        games_per_version(0.05, 0.5)
    );
    println!(
        "75-25: {:+.1} Elo  panel: {:+.1} {:+.1} {:+.1} {:+.1}",
        pair[0] - pair[1],
        elo[0],
        elo[1],
        elo[2],
        elo[3]
    );
    println!("all checks passed");
}

Both print the same lines and exit with status 0:

55/100: [0.4524, 0.6439]  6/10: [0.3127, 0.8318]
games: duel 785, per version 1570 (separate maps) or 785 (shared, 0.5)
75-25: +190.8 Elo  panel: +76.3 -12.0 -6.0 -58.3
all checks passed

10Other ways a benchmark lies

A common question after the mirror trap: "Is there any other glaring issue like this?" These are the measurement mistakes this page covers, plus a few more the author's lab ran into, each with its fix.

The psyleague warning, in full: "Most of the referees for CodinGame detect if your bot goes above allowed time for each turn. If you spawn too many workers your bots will start timing out randomly."^[12] The lab's guardrails make the same point about training signals: "Training reward, optimizer updates and imitation agreement do not establish stronger play."^[4]

11What's next

Learning from the Leaderboard turns the bots above you into a source of ideas: where to find post-mortems, how to turn a write-up into changes you can test, replaying logged games with your simulator, and running a pre-registered live panel against the players you need to beat.

12Sources

Formulas come from the statistics sources below. Numbers from the author's lab are first-hand and marked as such. Widget readouts come from the models described in their captions, and the mirror widget plays real light-cycle games with the kernel's bots.

1. Mark E. Glickman and Thomas Doan, version of April 6, 2026. The US Chess Rating system. Scoring 1, 0.5 and 0 for a win, draw and loss, and the logistic expected score 1 / (1 + 10^{−(R − R_i)/400}) (section 4.2).
2. NIST/SEMATECH. e-Handbook of Statistical Methods, section 7.2.4.1: Confidence intervals. The Wilson interval, its attribution to Wilson (1927) and its recommendation by Brown, Cai and DasGupta (2001) and Agresti and Coull (1998).
3. NIST/SEMATECH. e-Handbook of Statistical Methods, section 7.2.4.2: Sample sizes required. The normal-approximation sample size for detecting a change in a proportion at a given confidence and power.
4. Author's lab records, 2026 (private repository). Seat-grouped intervals and the seed ledger: docs/self-play.md. The mirror and the 0.610 result: AGENTS.md (October 13 to 14 rebuild), docs/provenance.md step 4 and experiments/decision-repair-20261013.json. The 95–5 and 348–334–2 results and the panel ladder: the contest repository's IMPROVEMENT_REVIEW_20260916.md and PANEL_LADDER.md. The promotion gate and the 800-game baseline check: docs/nn-training.md. Final-test rules and the quoted guardrail: docs/training-guardrails.md. Live results: experiments/live-submissions/summer-challenge-2026-back-track-king/comparison-20261024.json, evaluations-20261024.json, submitted.jsonl, experiments/nn-final-20261014.json and AGENTS.md (October 24 to 26 entries).
5. dreignier, 2026. cg-brutaltester README. The -s option that replays each map with positions changed.
6. _CG_Maxime (CodinGame staff), 2016-05-16. Smash The Code: question about the final ranking, post 41. Seat permutations summed and counted as a single result.
7. Wojciech M. Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi and Max Jaderberg, 2020. Real World Games Look Like Spinning Tops. NeurIPS 2020. Transitive strength plus cycles, and rock, paper, scissors as the simplest cycle.
8. David R. Hunter, 2004. MM algorithms for generalized Bradley–Terry models (archived PDF). Annals of Statistics 32(1):384–406. The Bradley–Terry model, Zermelo's iteration as an MM algorithm, and the condition under which it converges.
9. Bradley Efron, 1979. Bootstrap Methods: Another Look at the Jackknife. Annals of Statistics 7(1):1–26. Resampling with replacement to estimate the spread of a statistic.
10. Fishtest documentation. The SPRT for a Brownian motion. The SPRT's bounds, log(β/(1 − α)) and log((1 − β)/α), and ±log(19) at α = β = 0.05.
11. Stockfish developers. Fishtest Mathematics. Fishtest's generalized SPRT on game pairs (the pentanomial model) with bounds in normalized Elo.
12. Psyho (FakePsyho), 2024. psyleague README. The warning that too many parallel workers make bots time out.