How the AI is trained

PPO in plain words

PPO (Proximal Policy Optimization) is a reinforcement-learning method: the network learns by playing, not by copying labelled examples. Nobody tells it which card was the right pick. It finds out only at the end of a run how far it got, and has to work out for itself which of its choices were responsible. This section explains the idea without the maths; the rest of the page assumes it.

The policy

The network being trained is a policy: shown a situation (the deck, relics, HP, map, and the options on offer) it gives each option a probability. At a card reward it might say 55% Inflame, 30% Pommel Strike, 15% skip. Learning means shifting those probabilities towards choices that lead to better runs.

Play, then learn

Training alternates two phases, the loop in the figure above:

1. Play. The current net plays a batch of full runs. During training it samples from its probabilities instead of always taking the favourite, so it keeps trying options it is unsure about. Every decision is recorded: what it saw, what it chose, and how likely that choice was.
2. Learn. Once the batch is in, the net is updated so that choices which turned out well become more likely and choices which turned out badly become less likely. Then the updated net plays the next batch.

One batch teaches very little on its own. A training leg is typically 150 iterations of this loop, tens of thousands of runs, and a champion is the product of several legs.

Which choice deserves the credit?

A run has dozens of decisions and one outcome. A win does not mean every pick in it was good, and a death does not mean every pick was bad: the run may have been lost to one unlucky elite. PPO handles this with a second output, the critic (the value head): from any situation it predicts how well the run will go from here.

The credit for a choice, its advantage, is how much better things went than the critic expected at the moment of choosing. Suppose the critic expects a return of 0.40 at a card reward, the net takes Inflame, and the critic's estimate one decision later is 0.46: the pick gets an advantage of about +0.06. A choice that leaves the run worse off than expected gets a negative advantage. Any single advantage is noisy, because the fights and draws in between are random, but over thousands of runs the luck averages out and the real effect of the choice remains. The critic is trained at the same time, to predict returns ever more accurately.

"GAE" in the settings below is how the advantage blends the critic's next estimate with what actually happened over the rest of the run; λ = 0.97 leans mostly on what actually happened.

Small steps: the "proximal" part

The obvious rule, raise every probability in proportion to its advantage, is unstable: one batch of lucky runs can push a probability from 30% to 95%, and a policy that has lurched that far is playing a different game from the one the batch measured. PPO limits each update. For every recorded decision it compares the new probability of the choice with the old one, and once that ratio leaves the band 0.8–1.2 the choice earns no further credit. The net can still change a lot over many iterations, but only a little per batch, which also makes it safe to learn from each batch more than once (here, two passes).

Keeping it curious

Left alone, a policy tends to become certain too early, always taking the same option and never learning what the others are worth. A small entropy bonus rewards keeping some probability spread across the options. It is held flat early in a leg and decays towards the end, as the net settles.

Training versus playing

Sampling and the entropy bonus exist only to learn. When the trained net plays for real, in the gates, the published win rates and the advisor, it takes its highest-probability option every time, and the critic is not used to choose anything; its estimate is only shown, as V.

The environment

• One episode is one whole run, from the opening choice to a win, a death or a 3,000-step cap.
• One step is one non-combat decision with two or more options: path, card reward, boss relic, event, shop, rest site, treasure, full-belt potion swap, and the deck changes the game asks for (remove, upgrade, transform, enchant).
• Fights are not steps. The combat search resolves them inside the environment, so from the policy's point of view a fight is a stochastic transition that costs HP, potions and sometimes the run.
• Single-option prompts are taken automatically and logged. Nothing else is resolved on the policy's behalf: there is no silent fallback.
• Ascension is sampled uniformly from A0 to A10 for every game, so each net covers the whole range.

Getting the environment to actually present every decision took work. Until a fix in late July, event rooms reached the policy before their options existed, so events were resolved without it: the event embedding of five champions had moved by exactly 0.000000 during training. After the fix the policy saw 6.7 event decisions per run instead of 1.2, and two earlier "no effect" results had to be withdrawn.

Reward

The return is graded progress plus potential-based shaping:

return  = 0.5 · floors_reached / total_floors  +  0.5 · won
shaping = Φ(s′) − Φ(s),   Φ = 0.1 · hp_fraction + 0.035 · upgraded_cards − 0.307

total_floors is 45 (46 at A10, which adds a second boss). The discount is γ = 1. Reaching Act 3 and dying is worth more than dying in Act 1, but only a win reaches 1.

A useful negative result: with γ = 1, any critic of the form V = m − Φ makes every temporal-difference residual independent of the shaping coefficients, which was confirmed numerically to 3e-16. Shaping can only matter through critic error, so the coefficients are standardised rather than tuned.

PPO settings

The exact recipe, for readers who know PPO. Each line maps onto a step of the plain-words version above.

Collection & compute

Collection dominates. A pool of CPU worker processes each plays whole runs through the Rust engine, with the policy evaluated in PyTorch on one thread per worker, and most of that time is combat search. In the leg behind the current Silent champion (250/1,000 simulations, 960 games per iteration) collection took 97% of the wall-clock, and a PPO update about half a minute on one GPU. On one worker on one core, the October 2026 engine and search play the same six Ironclad runs 2.46× faster with identical decisions, and 3.16× faster with the profile-guided build training now uses. On the whole machine (46 workers, the update on one RTX 3090) a typical 960-game iteration takes 208 s instead of 609 s (2.93×), of which the update is 34 s. Part of the gain is not the engine: the new code skips the search when a decision has only one legal move. The hours below were measured before these changes. Details: Benchmarks.

Fights during training use a cheaper search than at evaluation: 250 simulations per decision and 1,000 at bosses, against 1,000 and 4,000 in the gate. Moving to cheap searches with more games per iteration, at equal compute, gained 2.5 points (p = 0.028), short of its pre-registered 3-point bar but adopted for the recent legs.

One leg is a single training run continuing from the previous champion. Necrobinder, Regent and Defect used the earlier recipe (1,000/4,000 simulations, 256 games per iteration, the PPO update on the CPU), and Defect ran its last ~90 iterations on half the machine; Silent's 41 hours include three iterations re-collected after a restart. A champion is the product of several legs; most took between 31 and 82 hours on the engine of the time.

Starting points

No lineage starts from random weights. Each began with behaviour cloning on runs played by an earlier agent from the project. The cloned Ironclad net matched the teacher's choice 81% of the time and won 33.2% of runs against the teacher's 38.7%; PPO took it from there.

Every later leg warm-starts from the champion before it. When a leg adds an input, such as the act boss's identity or a new feature column, the new weights start at zero, so the warm-started net plays bit-identically to its parent until training moves them.

Champion lineages

Each step was promoted on its own gate against the one before. Gains are paired, pooled over A0–A10.

Gains are in percentage points, each from its own gate. * Promoted by decision after its gate missed significance (p = 0.07). "Deck changes exposed" means removal, upgrade, transform and enchant prompts reach the net instead of being auto-resolved. These prompts come up less than once per run, and the gains of those legs are not attributed to them: the baselines never trained in the new environment.

The promotion gate

Training win rate is not evidence. A candidate is promoted only through a gate whose rule is committed before any of its runs exist:

• Paired. Candidate and deployed champion play the same 200 seeds at each of A0–A10: 2,200 paired runs, with one engine binary for both arms.
• Fresh seeds. Each gate uses a new seed panel that no training or earlier gate has touched. Rates are never compared across panels: one identical net scored 43.1% on one panel and 47.2% on another.
• Deployed settings. Greedy policy, 1,000/4,000-sim honest search: exactly what ships.
• Decision rule. Promote if the pooled gain is positive with McNemar exact p < 0.01, and no single ascension level drops by more than 5 points. The most recent gates also report a seed-clustered p-value, because the 11 runs on one seed are not independent (design effect about 2.2×).
• Export checks. The exported ONNX model is played against the PyTorch checkpoint (Regent: 190 of 200 runs identical), and peak token and action-slot use is checked against the fixed budget.
• Attribution is separate. A gate answers "is it better?", not "why?". Each promotion record states what the gain cannot be credited to.

just promote-champion rewrites the manifest and the code generated from it; just check-champions, part of just ci, fails if anything drifts from the manifest's pinned hashes.

What we learned

• "High ascension is a combat ceiling" was wrong. It rested on three null results. Once the environment presented every decision, Defect's largest gains came at A7 and A8 (+18.5 and +21.0 in one leg).
• Defaults bite. Learning-rate annealing was silently off by default and cost about 6 points before anyone noticed.
• Prioritised level replay did nothing measurable on Silent: −1.1 ± 1.3 points against a control arm with the same compute. The Ironclad leg that used it gained, but PLR, the longer horizon and a re-opened learning-rate anneal are confounded there, so it is not credited.
• Up-weighting hard ascensions during training cost 2 points.
• Small effects are hard to see. The quick paired screen used for triage has 1.7 points of noise between identical arms and cannot resolve effects below about 8 points; only full gates are trusted.
• A cheap combat surrogate is not good enough to train on. Agents trained against it ended runs with 34–50% of their HP to spare: they learned a game easier than the real one.

Results

Win rates for every character and ascension level are in the main docs: Defect 76.1%, Ironclad 74.9%, Silent 59.9%, Regent 53.8% and Necrobinder 51.7% averaged over A0–A10, in the simulator at the advisor's 8,000 simulations per combat decision. The promotion gates run at the cheaper 1,000/4,000 budget, so their numbers are lower and only meaningful as paired comparisons.

In the real game, at Ascension 8, the champions deployed in August 2026 (Ironclad deckmut_it150, Silent asc_it300) won 13 of 40 and 11 of 40 runs, driven through the Spire Mentor advisor at 8,000 simulations per combat decision and graded by the game's own run history. The current champions have not yet been measured in the real game.