How the Benchmark Works

The Game

Stronghold Crusader is a real-time castle and economy game by Firefly Studios, first released in 2002 and remastered in 2025 as Stronghold Crusader: Definitive Edition, the version the benchmark plays. It is set in the Middle East during the Crusades and seen from an isometric, top-down view.

A town starts with a keep, a campfire and a handful of peasants. Peasants need houses and food; they fell trees, quarry stone, mine iron, farm wheat and apples, and carry everything to a stockpile or a granary. Wheat goes to a mill, flour to a bakery. Popularity, driven by food, taxes, religion and fear, decides whether people stay, and taxes and the marketplace turn the town into gold.

The benchmark uses Free Build: no enemies and no objective beyond growing the town. Military buildings are off limits. That leaves an economic task, scored on what the town holds when the time runs out: its net worth, its population and its popularity, combined into one number.

Why a Game

A game is a visual environment that responds to every action, and it produces a score that can be checked. Net worth, population and popularity are read from the game’s memory after the final pause, so no judge is involved. Every episode on a map starts from the same save, at the same speed, with the same amount of game time, and a person has played each map for the same time, so there is a human result to measure against.

Episodes are also long. One episode is about 25 to 150 turns, and each turn changes what the next screenshot shows. That is closer to extended computer work than answering a single question.

What It Measures

Crusader Bench measures whether a model can turn what it sees on screen into actions that work in a live game. An episode tests several abilities together in one long task, and ends in a single number.

The Task

Three maps are scored, each from its own Free Build save made at the very first moment of play. A model plays each map separately, and a person played each one for the same game time, the reference games every model is measured against.

• Oasis by the Sea. A keep, a stockpile, a granary, a single peasant and a starting package of 1,000 gold, 50 wood, 25 stone and 50 bread still being paid in. Iron ore lies north of the keep, stone to the west, farmland on an oasis to the south-west.
• Large Barren Desert. A keep, a stockpile, a granary and a single peasant, with a much larger starting package: 5,000 gold, 55 meat, about 245 wood, 50 stone, 15 iron, 20 pitch and 20 ale. There is no oasis grass anywhere, so no farm can be built and food has to come from elsewhere. Stone lies about 100 tiles west of the keep and iron ore about 60 tiles east.
• The Sinkhole. A keep, four joined stockpiles, a granary and five peasants, with 2,000 gold, 55 wood, 20 stone and 55 meat already in. The keep stands on the inner edge of a wide ring of oasis grass around a sunken basin, whose uneven rim cannot be built on. Iron ore lies in the basin 50 to 90 tiles east of the keep and a large stone deposit about 85 tiles away. It is played for longer, 45 game minutes.

The model is told:

The rules sent with the instruction describe the map and say that the settlement is judged at the end on its net worth, its population and its popularity, without giving the weights. Soldiers and defences are not allowed. The placement tools do not offer military buildings, and the scorecard reports troop counts.

What Makes Games Comparable

• Same start. The game is launched fresh for every episode and the same save is loaded by name.
• Same game time. Every episode on a map gets the same game time, counted in the game’s own ticks, 30 to a game second: the length of the reference game a person played there. 33 min 20 s (2,000 game seconds) on Oasis by the Sea and Large Barren Desert; 45 min (2,700 game seconds) on The Sinkhole.
• Same speed. The harness fixes the game speed at 40 before the clock starts. The model cannot change it, so it cannot trade game time against how often it looks.
• Thinking costs no game time. The game is paused while the model thinks and while it only reads, so a slow model loses no game time. A six-hour real-time limit only stops games that have stalled.
• Same version. Every game records its benchmark version and a fingerprint of the prompts, tools and code behind it. Each map is played on one version, its round: the games on a map are compared only within it, and a model new to a map plays that map’s version. The maps can be on different versions (now v1.5 and v1.6).

A Turn

A turn is one model reply and its tool calls. Reading tools run with the game still paused; the first tool that acts, waits or looks resumes it. A turn that runs the game lasts at least 8 game seconds, which caps an episode at 250 acting turns and keeps cost in check. A reply can take at most eight actions, and every request shows the game time left. After a turn that acted without looking, the host sends a text reading of the town rather than a screenshot unless the turn needs one, which keeps the conversation small.

Scoring

A game is scored on the three things the agent is told its settlement is judged on: its net worth, its population and its popularity when the game time runs out. Each is measured on the game’s own map, from what doing nothing leaves to the game a person played there, and the three are combined into the Crusader Bench score, which runs from 0 for doing nothing to 100 for the person’s game.

All three are read once, after the harness pauses the game at the end of the budget. A model’s score on a map is the mean of its full-budget episodes there, and its score over every map is the mean of its map scores, so each map counts once however many episodes were played on it. The home page compares models over every map only when each has a full-budget score on all of them; until then a model appears on the maps it has played.

Net Worth

Prices are the game’s fixed marketplace sell prices, read from the game’s code and checked in play. Selling at the marketplace leaves net worth unchanged; buying lowers it, because the buy price is higher than the sell price. Buildings count only through what they produce.

The Two Ends of Each Map’s Scale

The maps start with very different wealth and leave very different room to grow. Measuring a game as a share of the person’s gain on the same map makes each map count the same, however much it has to give.

Doing nothing is measured by letting the save run untouched for the same game time as its episodes: on Oasis by the Sea the starting package arrives and the peasants who move in eat the bread; on Large Barren Desert the 5,000 gold stays in the treasury; on The Sinkhole the peasants eat the 55 meat and the 2,000 gold stays. All three end with 4 peasants and a popularity of 0. Popularity is measured on its own scale of 0 to 100 rather than from doing nothing, because what doing nothing leaves depends only on when the food runs out, and the person ended at 100 or 99 on every map.

The Weights

The agent is told that the three count, and from patch 1.5.6 that net worth counts most, but never the weights, so the weights are a choice, kept fixed for the version. Net worth carries 80%, population and popularity 10% each.

• Net worth is the open-ended measure. It is what the economy produced and kept, and it has no ceiling. Population is capped by housing and popularity by 100.
• Popularity mostly works through population. In the game, popularity above 50 brings peasants and below 50 sends them away, so weighing it heavily would count the same success twice. On the games so far it rises and falls with both population and net worth.
• Small weights still decide close calls. Today’s models differ more in how many people they keep than in how much wealth they build, so on the first five games on Oasis by the Sea population and popularity decided about a quarter of the order. With equal weights they would have decided most of it.

In a sum like this, weights act as exchange rates, and they differ by map with the person’s gain there. One more peasant is worth as much net worth as 11.5 gold on Oasis by the Sea, 2.4 gold on Large Barren Desert and 9.8 gold on The Sinkhole, and one more point of popularity as much as 22.9 on Oasis by the Sea, 6.0 on Large Barren Desert and 49.3 on The Sinkhole.

Limits on Each Part

Population growth and popularity stop at the person’s level, which for popularity is the top of the scale. Net worth growth can run from −1 to 2: a game that ends below doing nothing, for example by spending its gold on buildings that bring nothing back, scores below 0, and a game far ahead of the person on one map cannot carry its average alone. No game so far comes near either end.

Why This Form

The score uses the same scale as the human-normalised score of the Atari benchmarks, where 0 is random play and 100 a professional games tester, and the same fixed ends that the UN’s Human Development Index sets for each of its measures. Three other ways of combining the parts were tried on the games and set aside:

• A product of the parts (a geometric mean) rewards balance, but it swings widely for a game just above doing nothing and has no value for one below it.
• Net worth multiplied by popularity gives popularity more say over the order than net worth: a 10% fall in popularity costs 10% of the score.
• Net worth reduced for low population and popularity keeps net worth in charge but makes any population worth nothing when net worth does not grow, although the agent is told population counts.

What the Score Cannot Tell

• One reference game per map. The top of each scale is a single game played by a person. If that game was stronger on one map than on another, scores on that map read lower. Changing a reference game changes every score on its map, so a new one would be a change to the scoring.
• Starting goods on Large Barren Desert. Left alone, the desert’s stockpile stays full of wood and about 1,150 of its starting goods never arrive. A game that frees room receives them, which on its own is about a quarter of the person’s gain there, or about 19 points.
• Few games. Most models have one episode per map, and games of the same model vary by roughly 10 to 15%. Scores a few points apart should be read as level.

Before the Score

Until the score was introduced, games were ranked by net worth alone, with population and popularity reported beside it. The games on versions 1.0 and 1.1 played 25 game minutes, for which there is no game played by a person, so they have no Crusader Bench score and are still ranked by net worth.

A Person’s Game

To set the top of each map’s scale, and to show how far the models are from a practised player, a person played each save at the benchmark’s speed, pausing allowed, and was scored the same way: 25 real minutes on Oasis by the Sea and Large Barren Desert, about 33 min 20 s of game time, and 33 min 45 s on The Sinkhole, about 45 game minutes. The harness loaded the save and left it paused; when the time was up it paused the game, read the scorecard and saved the game.

On every map the person ran a bread economy and sold the bread for gold; on the first two, wheat bought at the marketplace kept the mills and bakeries from ever stopping. On Oasis by the Sea that meant wheat farms, mills and 57 bakeries; 13 iron mines and 10 quarries with 33 ox tethers; a church and a cathedral; and 54 gallows, gibbets and other bad things, whose fear made the workers 140% efficient, with no taxes and full rations to keep popularity at 100. On Large Barren Desert, where no farm can be built, it meant 95 bakeries and 12 mills fed only by bought wheat, 11 inns, 13 quarries and 7 iron mines with 37 ox tethers, a church and a cathedral, and 21 bad things for 110% efficiency, with low taxes, no rations and 250 people in 30 hovels. On The Sinkhole it meant 31 wheat farms, 19 mills and 161 bakeries, 16 inns kept in bought ale, 38 iron mines and 10 quarries with 29 ox tethers, 9 blacksmiths, 24 water pots as jobs for the growing town, a church and a cathedral, and 105 bad things for 130% efficiency, with no taxes and half rations of bread: 506 people in 62 hovels.

The agent sees none of this during play; it is shown the person’s result and how they played only in the interview afterwards. The game’s own files are not published, only the scorecard, the final reading and a screenshot.

Episodes and Learning Series

A model plays one or more episodes, each a fresh game of the same save. Episodes are independent: nothing carries from one to the next. The games on the current version are one episode each.

A learning series is the exception. Its episodes are played in a row, and one thing carries over: the model’s playbook, Markdown notes of up to 8 KB. The model can read and edit it during play at no cost in game time, and after each episode it is shown its final reading, net worth included, and asked to rewrite the playbook for the next attempt. The games on versions 1.0 and 1.1 were learning series of three.

• Score: on the current version, the Crusader Bench score, the mean over a model’s full-budget episodes on a map. On versions 1.0 and 1.1, the net worth of the last episode.
• Learning: in a learning series, the change from episode 1 to the last, a measure of how much the model improved from its own notes.

Neither the game’s nor the model’s randomness is fixed, so one episode mixes skill with chance, and a learning series mixes learning with it too. Results from several episodes per model are more reliable than one.

The Interview

After each episode that counts, once it is scored and the game is saved and closed, the harness interviews the model about its game, in the conversation it played in, with the tools switched off. The interview is not part of the benchmark: the score is already recorded, and nothing from it reaches a later episode. It is asked in one message with four questions under their own headings:

1. Its result beside the person’s game on its map: the game time each played, net worth, gold, stored goods, population, housing and popularity, and what doing nothing scores. How does it see the difference?
2. How the person played and what their settlement held at the end. What would it do differently from the start?
3. What it thinks of the benchmark: the tools, what it could see, the time limit and the instructions.
4. Anything else.

The transcript is published with the game, with the interview’s own tokens and cost kept apart from the game’s. Every game’s page shows it in full. The model remembers its game only as far as its context reaches after any compactions, so what it says about the game can differ from the record.

The Record

Every game publishes, besides its score, the files that let anyone check it and read how it went:

• The results. The whole state of the game at the end, read after the final pause: net worth with every stored good, population, housing and idle peasants, popularity and the factors that move it, taxes, rations, worker efficiency, storage, and the player’s buildings by name.
• The trace. Every message, tool call, result and screenshot the model saw, a readable log, a per-minute timeline of the economy, the model’s own notes, a video of the episode with the thinking pauses cut, and a final overview of the settlement.
• The analysis. A reading of the game written afterwards from these files, by a model (so far Claude Opus 5.5, including for its own game) in a separate session from the one that played: how the game went minute by minute, what held it back, what worked, what it cost, how it compares with the other games on the same save, and what its interview got right or wrong. It is published with the game and shown on its page. It is a reading, not part of the score, and its claims can be checked against the files it names.

The Agent

The agent is a vision-capable language model with tool calling. Each request carries a reading of the town: the clock and budget, date, gold, population and housing, popularity and its factors, tax, food and rations, stored goods and the room left for them, placement mode, camera position and recent game messages, with a 1280-pixel-wide screenshot of the game window when the turn needs one. Long conversations are compacted into a handoff the model writes itself.

It has 26 tools, plus three playbook tools in a learning series:

Look

observe, wait_and_observe, flat_viewScreenshots, waiting until a condition is met or a time has passed (with or without a screenshot), and a flattened view that shows free ground.

Build

build_structure, place_near, expand_storage, find_sites, demolish_buildingPlace buildings by map tile or screenshot pixel and verify them with the reader, let the tile map find spots that fit, grouped by the ground under them, or demolish one of the agent's own.

Camera

center_on, map_overview, go_to_tile, save_view, go_to_view, navigate_minimapMove around the map, read the whole map's resources, bookmark views.

Economy

status, get_inventory, inspect_building, set_tax, set_rations, market_tradeRead the state of the town, inspect buildings, set taxes and rations, buy and sell at the marketplace in the game's own steps.

Reference

building_infoOne building's cost, footprint and menu position from the game reference.

Raw input

game_actionOne click, drag, key or scroll at screenshot pixels, for anything the other tools do not cover.

Memory

update_plan, notebook_read, notebook_write, notification_historyA plan, a notebook that survives context compaction, and every message the game showed.

Playbook

playbook_read, playbook_write, playbook_editLearning series only: the notes that carry over to the next episode.

What It Cannot Do

• See anything but the game window.
• Pause the game, open its menu or change its speed.
• Use a shell, files, a browser or the network.
• Play against other people: the reader refuses multiplayer games and the harness will not run one.

How It Is Built

A harness runs the agent: it pauses the game while the model thinks, carries out its tool calls, keeps the clock and records everything. A small reader inside the game supplies the numbers a player would read off the game’s panels, such as gold, stored goods, population and popularity, so the model does not have to read them from pixels. The reader only reads and changes no game files. A recorder captures the game window so every episode can be replayed.

The harness and the reader each have a page in the documentation.

Full-Budget Episodes

A score is a full-budget score when the episode completed, the game-time budget ended it, the final pause was confirmed and the score came from the final reading. Each episode ends in one of these ways:

The harness marks an episode invalid when anything but the game-time budget ended it, however close to the end. This site counts an episode that played at least 99% of its game time (1,980 of 2,000 game seconds, or 2,673 of 2,700) and was read after the final pause as a full-budget score: the settlement it leaves is the one a full game would leave. GLM 5.3 Flash’s version 1.5 game on Oasis by the Sea, which it ended 5 game seconds early, is one.

Every attempt is kept in the record and shown on each game’s page. An episode the model ended earlier than that is still read from the game, so its figures are shown, marked as not a full-budget score.

Versions

A benchmark version names one fixed benchmark: the prompts, tools, rules of play, scoring and the code that carries them out. Versions follow MAJOR.MINOR.PATCH.

The harness fingerprints every file that decides how a game plays or is scored, and refuses to run unversioned code in a benchmark series. Leaderboards on this site are grouped by major and minor version. They rank only the games listed in the dataset’s evaluation.json, and of a model’s games on one map only those on the newest patch; every other game keeps its report, marked as not ranked. On this version, Oasis by the Sea was played on 1.5.0 and 1.5.1, and Large Barren Desert was played on 1.5.6, and The Sinkhole was played on 1.6.2.The games on versions 1.0 and 1.1 played 25 game minutes from a different save as learning series of three, so they are not comparable with the current version. What each version changed is in the changelog.

Limitations

• One setup. All games so far come from one machine running one build of the game through Steam and Proton.
• Few games. Randomness is not fixed, so one episode per model gives a preliminary result. Small differences between models may be noise.
• One reading. The score is a single sample of the game’s memory after the final pause, read twice and checked, not an atomic snapshot.
• Three scored maps. Each has a single reference game played by a person, which sets the top of its scale. Other scenarios in the repository are diagnostic; military play is not scored.
• Costs are not all bills. OpenRouter reports what it billed. Anthropic’s and Moonshot’s APIs report tokens but no bill, so those games’ costs are their tokens at the model’s list prices, marked ~ on this site.
• Submitted games are checkable, not verified. A community score comes from the submitter’s machine; its video, screenshots and readings make it checkable.

Data, Licence and Citation

Every published game is in the Hugging Face dataset: series, episode and per-minute timeline tables, the full trace of every message, tool call and screenshot, the results, the interview, the analysis, any playbooks and a video of each episode, with the person’s reference games beside them. This site is built from those files.

• Logs, tables, playbooks and model outputs are released under CC BY 4.0.
• Game footage (screenshots, final overviews and videos) shows Stronghold Crusader: Definitive Edition, © Firefly Studios. It is not covered by that licence and is shared for non-commercial research.
• The code is on GitHub under the MIT License, with full documentation.

Load the Data

from datasets import load_dataset

repo = "hesamation/crusader-arena-runs"
series   = load_dataset(repo, "series",   split="train")
episodes = load_dataset(repo, "episodes", split="train")
timeline = load_dataset(repo, "timeline", split="train")

Cite

To cite Crusader Bench in a paper, name the benchmark version, since scores compare only within one:

@misc{crusaderbench2026,
  title        = {Crusader Bench: Independent Analysis of AI Agents Playing Stronghold Crusader},
  author       = {{Crusader Bench}},
  year         = {2026},
  howpublished = {\url{https://crusbench.com}},
  note         = {Benchmark version 1.5 and 1.6. Data: \url{https://huggingface.co/datasets/hesamation/crusader-arena-runs}}
}

Responsible Use

Crusader Bench is for research on AI agents in single-player games. It is an independent project, not affiliated with or endorsed by Firefly Studios. The reader loads its own code into the running game, which the game’s licence terms may not permit; anyone reproducing games does so with their own copy of the game and at their own risk.

Back to the leaderboard