The Benchmark

What Makes Runs Comparable

• Same start. Every episode launches the game fresh and loads the same save by name.
• Same game time. The budget is counted in game ticks (1,800 per game minute; 25 game minutes by default), so every model gets the same amount of in-game time.
• Same speed. The game runs at a fixed speed of 40. The host sets it with the game's speed keys before the budget starts and the model cannot change it, so speed is not a lever: it cannot trade game time for how often the model looks.
• Thinking is free. The game is paused while the model thinks and while it only reads, so a slow model is not penalised in game time. A real-time limit (6 hours by default) only stops runs that have gone badly wrong; the record shows when it ended a run.
• Same version. Every run records its benchmark version (such as 1.0.0), which fixes the prompts, tools, run rules, scoring and the code behind them. Compare runs that share a major and minor version. Each run also records the harness commit, hashes of its prompt and tools, the preparation images and the model's settings.

Scenarios

A run's benchmark name selects its rules: the host lowercases it, turns other characters into -, and loads prompt/benchmarks/<name>.md if it exists. "Oasis by the Sea construction" loads oasis-by-the-sea-construction.md. Without a rules file, the operator's instruction alone defines the task.

All are Free Build games saved at the very start of play, with soldiers and defences not allowed. The starting goods are still arriving when a save loads: Oasis by the Sea-1 shows 120 gold and 12 wood, and the rest of the package arrives within about 25 game seconds: wood first, then stone, bread and the last of the gold (see Scoring). The Oasis rules tell the model this, and the game reference tells it that starting goods rise at first. Large Barren Desert benchmark start shows 4,080 gold, 43 wood and 12 meat. Its package is 5,000 gold, 55 meat, about 245 wood, 50 stone, 15 iron, 20 pitch and 20 ale: the gold, the meat and 192 wood arrive within about 27 game seconds and fill the stockpile with wood, and the rest waits until it has room. The Sinkhole start was saved a little after the start (game tick 2,114), with its whole package of 2,000 gold, 55 wood, 20 stone and 55 meat already in, and a granary and four joined stockpiles standing.

The saves are not included. To make one, start a Free Build game on that map and save it straight away under the exact name in the table. A save made later starts from a different position, so its runs are not comparable. npm run episodes loads a save by typing its name into the game's Load Game search box, so names may contain only letters, digits, spaces and hyphens.

The no-military rule is an instruction in the prompt. The harness does not offer military buildings in its placement tools, but it does not block a model that clicks them by hand. The scorecard reports troop counts.

Versions

A benchmark version names one fixed benchmark: the same prompts, tools, run rules, scoring and the code that carries them out. Versions follow semantic versioning, MAJOR.MINOR.PATCH, starting at 1.0.0:

A change that leaves behaviour as it was (comments, refactors, how a failed request is retried) keeps the version and is recorded as a relock.

• Fingerprint. The harness hashes every file that decides how a run plays or is scored: harness/server and harness/shared (except tests, the report and the video renderer), prompt/, the game-machine helpers in tools/ubuntu/, the reader in src/ with its build files, and package-lock.json. It uses git's blob IDs, so a clean checkout gives its commit's fingerprint on any platform.

• Versions file. benchmark-versions.json lists each version with the fingerprints it accepts, the date and what changed. npm test fails while the current files are not listed, so a change cannot slip in unrecorded. Record it in the same commit:

  npm run benchmark-version -- check                           # which version is this code?
  npm run benchmark-version -- bump patch "what changed"       # or minor, or major (table above)
  npm run benchmark-version -- relock "why"                    # same behaviour, same version

  Every entry says what changed or why behaviour did not.

• Changelog. CHANGELOG.md has a section for each version, newest first, headed ## [MAJOR.MINOR.PATCH] - YYYY-MM-DD with the date the version was first recorded. Its Benchmark list says what changed in play or scoring; its Harness list gathers the changes since the previous version that left runs as they were, including relocks; until the next version they wait under ## [Unreleased]. A test fails while a version in benchmark-versions.json has no section or a different date, so add the section in the commit that bumps the version. The format is fixed so the website can read it.

• Package version. package.json and package-lock.json carry the benchmark version; bump sets them before taking the fingerprint, and a test checks they match.

• Tags. The git tag vMAJOR.MINOR.PATCH marks the commit each version was released at, and each tag has a GitHub release whose notes are its changelog section.

• Each run records benchmark in run.json: the version (null when the files are not a listed version), the fingerprint, and a hash of the private preparation images (null when the guide is text only), which are not in the repository and so not in the fingerprint. A version also needs committed code: uncommitted behavioural edits change the fingerprint.

• The game machine runs its own copy of tools/ubuntu/, src/ and the CMake files. Before every run npm run episodes compares them, file by file, with the checkout that plays the run, copies over the ones that differ and, when the reader's sources are among them, rebuilds the Windows programs the game runs (build/ubuntu-windows). A series still refuses to start if they differ afterwards.

• npm run episodes refuses code without a version unless --unversioned, for trials. npm run report shows each run's version.

Rounds

Every agent run on a benchmark plays on the version of that benchmark's last run, so the models on one map stay comparable after later versions change the prompt or the tools. benchmark-rounds.json records each benchmark's round: its version, fingerprint and the commit to play it from. It is not in the fingerprint, so moving a round changes no version.

• Same version. A run whose round is the checkout's version plays here, as before.
• An earlier version. npm run episodes plays the run from the round's commit: it makes a git worktree under .versions/<version>-<commit> (ignored by git) on first use, links this checkout's .env, harness/runtime/ and .internal/ into it and installs its dependencies; copies that version's game-machine files over (above); starts that version's host on the first free port after --port (its log in the worktree's host.log); and runs that version's own runner with the same arguments. Runs, series and saves land in this checkout's harness/runtime/ as usual. This checkout's host stays up but does not list the run until it restarts; do not restart a host while the other one has a run going, since a starting host marks unfinished runs interrupted.
• What an earlier version gets from this checkout. The round's code plays as its version recorded it, with two additions that change how a run is carried, not what it plays or scores:
  • Its host starts with round-preload.ts, which adds the prompt cache key to its requests for models that need one. The key is part of the request, not of what the model reads.
  • A version made before the run overlay (1.5.6 and earlier) gets this checkout's overlay.py on the game machine, which that version's files do not include, and this checkout's runner feeds it from the round's host.
• A new round. A benchmark's first valid run sets its round to the checkout's version. --new-round plays on the checkout's version whatever the round, and moves the round there once the run is valid: that is how a new version starts a round, as when every model was run again on Large Barren Desert at 1.5.6. The runner then says to commit benchmark-rounds.json.
• --unversioned trials, --dry-run, --idle and --human ignore rounds, and so does the Custom benchmark.

Scoring: Net Worth

The scored benchmark ranks runs by net worth when the run ends: gold, plus every stored good valued at the game's marketplace sell price per unit.

The prices are a fixed table in the game (no difficulty or demand effects), read from the game code and spot-checked in play. They are in harness/server/market-prices.ts.

Weapons and armour are stored only in an armory, so they count only once one stands. The score has always valued them, but until 1.5.6 the rules told the agent that they add nothing and build_structure did not offer the Armory. Since 1.6.0 it does, set_workshop chooses what each workshop makes, and the rules list their prices.

Baseline. npm run episodes -- --idle measures what doing nothing scores: it loads the save, sets the game speed as an agent run does, lets the game run untouched for the same game time an agent gets, and writes the scorecard and the results to harness/runtime/episodes/idle-<time>/. On Oasis by the Sea-1 with 25 game minutes at speed 40 doing nothing scores 1,304: the starting package of 1,000 gold, 50 wood, 25 stone and 50 bread (worth 1,425) arrives, the game places the granary by itself (5 wood), and the peasants who move in eat 29 of the bread. Runs too short for the game to place the granary end lower: 1,225, with all the bread eaten. With 33⅓ game minutes (2,000 game seconds) doing nothing scores 1,220 on Oasis by the Sea benchmark start, where the granary is already placed (all the bread is eaten), and 5,192 on Large Barren Desert benchmark start, which starts with 5,000 gold. Left alone, the desert's stockpile stays full of wood, so the part of its package that waits for room (stone, iron, pitch, ale and more wood, about 1,150 at the sell price) never arrives; a run that frees storage receives it, and its growth includes it. With 45 game minutes doing nothing scores 2,195 on The Sinkhole start: its 2,000 gold, 55 wood and 20 stone stay, and the 55 meat is eaten.

Growth is net worth minus the baseline for the same save and game minutes, so it shows what a run achieved beyond leaving the game alone. The measured baselines are in harness/server/baselines.ts; a run whose save and budget have none has no growth.

Reference games. npm run episodes -- --save "…" --map "…" --human is a game played by a person at the game machine, scored like an agent's:

1. The runner launches the game, loads the save, sets the benchmark speed and leaves the game paused.
2. The player clicks the game and presses P to start. The budget counts game time, so time spent paused is free, as an agent's thinking is. The runner logs each game minute with the net worth so far.
3. When the budget is used up, the runner pauses the game. It then reads the scorecard, keeps a screenshot of the screen, and saves the game in the Save dialog as human, the save and the date and time (for example human oasis by the sea 1010-1432).
4. Everything goes to harness/runtime/episodes/human-<time>/: scorecard.json (with human.playedSeconds and human.ended), the results, screenshot.jpg and the .sav.

The player is not bound by an agent's limits, such as the minimum turn. If the reader has no valid sample for 2 minutes (the memory guard may have closed the game), the scorecard uses the last reading and human.ended is readings_lost.

The reference games agents are compared with are in referenceGames in baselines.ts, by map, and how the person played each is in its own file in prompt/reference-games/. The first two were played for 25 real minutes at speed 40 (about 2,000 game seconds), The Sinkhole for 45 game minutes (2,698 game seconds; 33 min 45 s of real time at speed 40), so its agent runs take --game-minutes 45:

The agent sees them only in the interview after its episode.

Since 1.5.6 the agent is told that net worth counts most and population and popularity less, without the weights, and that every model plays the same save for the same game time, so growth up to the last minute counts (each map's rules file and objective). From 1.2.0 to 1.5.5 it was told that all three count, without how they are weighed. The website combines them into one number, the Crusader Bench score: 100 × (0.8 × net worth growth + 0.1 × population growth + 0.1 × popularity ÷ 100), each growth measured on the run's map from doing nothing (0) to the reference game (1). The harness's own reports rank runs by net worth, and the scorecard records population and popularity beside it. Buying at the marketplace lowers net worth (the buy price is higher than the sell price); selling leaves it unchanged.

Running Episodes

npm run episodes runs a series of episodes (3 by default) without an operator. The episodes are independent games of the same save; --playbook makes them a learning series. For each episode it:

1. Starts the game through Steam on the game machine and connects to it.
2. Waits for the main menu, opens Load Game, searches for the save by name and loads it (up to 3 attempts), then confirms the map with the reader. While it waits it clicks the screen centre once a second, which skips the intro videos and the promo splash (plain cover art on the main menu, away from its buttons). The main menu and the Load Game dialog are recognised as soon as they show, from a few pixel colours of a capture (tools/runner/screen_probe.py, which needs Pillow on the host, and harness/server/menu-screen.ts); the runner log gives the seconds from launch to each. Without Pillow it waits the old way: 45 s, then until the screen stops changing, which the animated menu never does, so about 105 s of timeouts.
3. Pauses the game and starts the agent run with the given budget.
4. When the run ends, takes the final reading and writes the scorecard, episode.json, and the results into the run's folder.
5. If the game is still running and paused, saves it in the game (Game Options → Save) as <model> <version> <series id end> e<episode>[a<attempt>], for example glm 5-3 flash 1-1-1 ae6fc4 e3a2 (the game takes 32 characters, so a long model name is shortened), and copies the file into the run's folder as <name>.sav, to load and watch later. The Save dialog starts with the loaded save's name, so saving without replacing it would overwrite the benchmark save. The runner moves the cursor to the end of that name before clearing it: a click on the field can land inside a long name, which left "rt" of Large Barren Desert benchmark start behind the new name and the save under a name the runner did not look for. When no save under the new name appears, the runner opens the dialog once more and tries again (attempts: 2): once only the spaces of the typed name reached the field. Before the first save the runner keeps a copy of the benchmark save on the game machine (~/.local/state/crusader-arena/save-backups/). It compares the benchmark save's hash after every save, puts the copy back if it changed, and then pauses the series. A failed save does not affect the score; game_save in episode.json records the name and file, or the error. The .sav files are not published.
6. Disconnects and closes the game. With --record, renders the run video.
7. If the episode counts (see below), interviews the agent about it.

When an episode counts and there is a next one, step 7's interview and the next episode's steps 1–2 (starting the game, connecting and loading its save) run at the same time: neither touches the other's state, so the next episode's game is ready to go as soon as the interview ends. The next episode's agent run (step 3) still waits for the interview, since the dashboard refuses a second run while one is in progress. A stop, a stale dashboard or a changed benchmark save (step 5) between episodes still waits for the interview and closes a game the series ends up not playing after all.

Each episode ends in one of four ways (series.ts):

So no model gets a second attempt after a failure of its own, and a failure that is not the model's never costs it an episode. Every attempt stays in series.json; an episode's result is its last attempt.

Stopping and resuming. Press Ctrl-C in the runner at any moment: it stops the dashboard run, closes the game and pauses the series (a second Ctrl-C, 2 seconds or more later, quits at once and leaves cleaning up to you). Continue it with:

npm run episodes -- --resume <series id>

The series continues at its first episode without a result, with the settings it was started with (in a learning series, from the playbook the episode before it left). --resume refuses setting flags (pass only --restart, --keep-game, --unversioned, --no-interview or --port) and refuses to continue when the benchmark version, code commit, preparation images or the model profile's ID, endpoint, reasoning, output limit or providers have changed. If harness or prompt files change while a series runs, the series pauses before the next episode: the dashboard would otherwise run it on the older code.

npm run episodes -- --save "Oasis by the Sea-1" --map "Oasis by the Sea" \
  --benchmark "Oasis by the Sea construction" --model "Kimi K3" \
  --prompt-file prompt/objectives/oasis-by-the-sea.txt --game-minutes 25

Before starting, the dashboard must be running (npm run dev, from a terminal; see Setup), not connected and not running a run. It must also have been started after the last change to the harness or prompt files, its code must be a benchmark version with no uncommitted changes, and the game machine's helper and reader files must match this checkout; the runner checks all of this.

Learning Series

Only with --playbook. The episodes of a series play the same save, each from the start. One thing carries over: the agent's playbook, Markdown notes of up to 8 KB, which measures how much the agent learns from playing.

• Start. The preparation message says which episode of how many this is and shows the playbook as the previous episode left it (empty in the first).
• During play the agent can read and change the playbook with playbook_read, playbook_write and playbook_edit (Tools). They cost no game time.
• After the episode the host shows the agent the final reading, net worth included, and its playbook, and asks for the complete next version. The request repeats the last gameplay request plus that message, so it is cached like the compaction request. If the model calls a tool instead, it is asked again as text only. A reply that is empty or over 8 KB leaves the playbook as the agent last wrote it during play.
• Score. The last episode's net worth is the series' score, if that episode is valid (see the scorecard). The report also shows each episode's net worth and the change from episode 1 to the last, and marks invalid ones.

The runner keeps each series in harness/runtime/series/<id>/:

• series.json: the save, benchmark and model; status (running, paused or completed) and, when paused, why (stopped); settings, everything the episodes run with (model settings, instruction, run settings, benchmark version and commit); and results, every attempt of every episode with its outcome, run, net worth and validity;
• playbook-after-episode-<n>.md, the playbook episode n left, which starts the next;
• runner.log, the runner's own log.

Each run folder also holds its final playbook.md, and an episode's runs record their attempt number. Runs started from the dashboard are never part of a series.

The Scorecard

episode.json holds:

• net_worth, goods_value, and net_worth_baseline / net_worth_growth when a baseline exists for the save and game minutes;
• gold, population, housing, popularity, total_food, the nonzero goods, the map-wide structure count and own troops;
• the map, date and game time;
• a run summary: status, stop reason, budget used, turns, tokens, inference timing and memory;
• source: final for a reading taken after the final pause, or last_observed when the game was gone at the end (for example after a memory-guard stop) and the last valid reading from the run was used instead;
• game_save: the in-game save of the finished episode (name, file, bytes, sha256), or the error that stopped it;
• valid: whether net_worth is a full-budget score. It is false, with the reasons in invalid, when the run did not complete, a limit other than the game-time budget ended it, the final pause was not confirmed, or source is last_observed. Only valid episodes should be compared or used as a series' score. The runner's summary line says which episodes are not.

Runs started from the dashboard do not write episode.json.

The Results

Every run the runner makes records the whole state of the game at its end in one format, whatever played it: an agent's episode, a reference game, the do-nothing baseline or a dry run (results.ts). results.json holds it beside the scorecard and results.md shows it as tables:

A value the game was not read for is null, never 0: a reader older than the building counts leaves buildings null. Results are written by runs made since they were added.

The Interview

After each episode that counts (valid or model_failure), once it is scored and the game saved and closed, the runner has the host interview the agent about its game. The interview is not part of the benchmark: the score is already recorded, and nothing from it reaches a later episode (in a learning series it comes after the reflection that writes the next playbook).

• The same conversation. The host keeps the finished run's conversation until the next run starts and asks in it, so the agent remembers its game as far as its context reaches after any compactions. The request repeats the run's system prompt and tools, so the provider's cache covers the game. The tools no longer work: a tool call gets a refusal and the combined question below is asked on, twice at most.

• Four questions in one message (prompt/interview.md), asked together under their own titled headings, with instructions to answer each one under the same heading so the reply can be told apart into four answers:

  1. Its result beside the reference game on its map: the game time each played, net worth, gold, stored goods, population, housing and popularity, and what doing nothing scores. How does it see the difference?
  2. How the person played and what their settlement held at the end. What would it do differently from the start?
  3. What does it think of the benchmark: the tools, what it could see, the time limit and the instructions?
  4. Anything else?

  Asking once instead of once per question cut a four-question interview from four sequential requests to one. When the reply does not come back under the four headings, it is kept whole as a single answer instead of guessed at.

• Maps with a reference game only. A map needs both its entry in referenceGames and its file in prompt/reference-games/; without them the interview is skipped. When the run's budget differs from the person's game time by more than 2%, the first question says the two are not like for like.

• The record. interview.md, the transcript, and interview.json, the same with each answer's reasoning where the provider returns it, refused tool calls, tokens and time, and the interview's cost, go into the run's folder, rewritten after each answer, and are published with the run. The combined request's tokens, reasoning and refused calls are kept on the first of the four answers, so the totals are not counted four times over. The run's own files stay as the run ended: the interview's cost is not in the run's cost and its requests are not in events.jsonl. npm run report (below) reads interview.json into its own columns, apart from the run's.

• Failures. A transient provider error is retried as during play. Any other failure, a stop (Ctrl-C) or 30 minutes without an end ends the interview with no answers recorded (it is now one request, not four, so there are no earlier answers to keep); the episode's result never changes. --no-interview leaves interviews out.

Comparing Runs

npm run report prints one Markdown table with a row per run and, when the runs include learning series, a second table with a row per series: net worth by episode, the final episode's net worth (the series' score), the change from episode 1, and the series' cost. It only reads files, so it is safe to run during a benchmark.

npm run report                                              # every run under harness/runtime/runs
npm run report -- --benchmark "Oasis by the Sea" --model kimi --since 2026-10-01
npm run report -- --runs /path/to/runs --json > report.json

Filters ignore case and punctuation. --model matches the profile name or the provider's model ID; --since takes an ISO date or time.

Columns, in order: run, start time, model, model settings (reasoning, output limit, providers), harness version (commit, uncommitted-change hash, and prompt and tool hashes), benchmark, map, how it ended, game seconds, wall seconds, turns, tokens (total, uncached input, cache reads, cache writes, the share of input read from the cache, output), cost in US dollars (what OpenRouter billed, or ~ before it when worked out from the run's tokens and its model's prices; see Context), the post-episode interview's own tokens and cost (from interview.json, apart from the run's own; blank where a run has no interview), tokens per game minute, population, housing, popularity, net worth, growth, gold, food, structures, troops, where the score came from, building placements attempted / placed / failed, anchor-tool calls / placed / failed, retries, retry methods, retries skipped, peak game memory, tool errors and tool usage.

"How it ended" is one of: game_time (budget used), wall_limit, turn_limit, memory_guard, error: <last error>, stopped, interrupted, running or incomplete. For runs without episode.json, the score is rebuilt from the last reading in the run's tool results and labelled last_tool_observation, without growth. Missing data stays blank rather than guessed; ~ marks game time derived from reader ticks, * marks an unfinished run and ! a net worth that is not a full-budget score (valid: false in episode.json).

Costs of Earlier Runs

Runs from before the host priced requests on endpoints other than OpenRouter, such as the first Claude runs through Anthropic's API, recorded tokens but no cost. Enter the model's prices in its settings, then:

npm run cost            # lists the runs it can price and what each cost; changes nothing
npm run cost -- --write # records those costs

It prices each request's recorded tokens at the profile's prices, as a live run would, and writes a request_cost event after each request (marked backfilled) and the total and prices into run.json. It leaves out running runs, runs that already have a cost, and OpenRouter runs from before billed amounts were recorded. Restart the host afterwards to see the costs in the dashboard, and package a published series again (Publishing) to carry them into the dataset.

Limits

• One environment. The benchmark has only run on one Ubuntu machine with one game build. See Status.
• Single reading. The score is one reader sample after the final pause. Reads are checked twice but are not atomic snapshots.
• No control of randomness. Neither the game's nor the model's randomness is fixed, so the change across a series mixes learning with chance. Run several series, or compare with a series run without --playbook.
• Harness versions. Prompt and tool changes affect results. Compare runs with the same Harness value in npm run report.
• The memory guard can end a run early (see Status); the scorecard then uses the last valid reading.
• Not built yet: scores for the diagnostic benchmarks, military or defence scoring, rankings across runs, and a published comparison of models.

Adding a Benchmark

Create prompt/benchmarks/<name>.md. Start it with front matter; military: false removes the troop, siege and fortification sections from the game reference:

---
military: false
---
# Benchmark: <title>

**Task.** ...
**Score.** ...
**Not allowed.** ...

## The map
## Start

Then start runs with the matching benchmark name. Automatic scoring exists only for net worth; a new scoring rule needs code in harness/server/episodes.ts.