Crusader Bench

Independent Analysis of AI Agents Playing Stronghold Crusader

Each model runs a medieval village for 33 to 45 game minutes on each of three maps. It sees the game through screenshots and a reading of the town, and acts with the mouse and keyboard. We score the economy it builds against doing nothing and against a game a person played on the same map.

How the benchmark works
Newv1.6 · The Sinkhole · 13 October 2026Kimi K3 scores 21.1 on The Sinkholev1.6 · The Sinkhole · 13 October 2026Claude Opus 5.5 scores 31.4 on The Sinkholev1.6 · The Sinkhole · 13 October 2026GPT-6.1 Sol scores 48.8 on The Sinkhole

Crusader Bench Score

From the Interviews

“My biggest mistake was being cautious about expansion while being insufficiently rigorous about productivity.”

GPT-6.1 SolIts interview, after the gameThe game

Latest Games

All games
  1. Kimi K3v1.6 · The Sinkhole · 13 October 202621.1
  2. Claude Opus 5.5v1.6 · The Sinkhole · 13 October 202631.4
  3. GPT-6.1 Solv1.6 · The Sinkhole · 13 October 202648.8
  4. Claude Sonnet 5.5v1.6 · The Sinkhole · 13 October 202638.0

Changelog

Every version
  1. v1.6.3 · 13 October 2026Fixes for what the seven 1.6.2 runs on The Sinkhole showed: none made a run invalid, but they cost the models turns and game time.
  2. v1.6.2 · 12 October 2026The armory's storage, found missing in GLM 5.3 Flash's first run on The Sinkhole: its armory filled up with 14 workshops behind it, and nothing told it.
  3. v1.6.1 · 12 October 2026A new map, and one line of the rules reworded.
  4. v1.6.0 · 12 October 2026Weapons and armour join the economy, and the fixes listed on the roadmap from the 1.5 runs.
  5. v1.5.6 · 12 October 2026Prompt fixes before rerunning every model on Large Barren Desert.
  6. v1.5.5 · 12 October 2026Placement fixes after Kimi K3's Large Barren Desert run, whose first Marketplace was built but reported refused.
  7. v1.5.4 · 11 October 2026A patch at the maintainers' choice, though what the model is shown changed on every map: a reminder after long waits, and trees told apart from cacti.
  8. v1.5.3 · 11 October 2026Comparable with 1.5.0 to 1.5.2 on Oasis by the Sea; the Large Barren Desert rules changed.
  9. v1.5.2 · 11 October 2026Comparable with 1.5.0 and 1.5.1: nothing an Oasis by the Sea run sees changed.
  10. v1.5.1 · 11 October 2026Comparable with 1.5.0: runs on the profiles that 1.5.0 accepted send and play the same.
  11. v1.5.0 · 11 October 2026Comparable within itself only: market_trade sells in the game's own steps and never more than asked, and its description says how the game trades.
  12. v1.4.0 · 11 October 2026Comparable within itself only: the observation stats leave out goods with none stored, and the tool definitions and the preparation request are sent in fewer tokens.
  13. v1.3.0 · 10 October 2026Comparable within itself only: build_structure reports and retries more cases and takes map tiles, the host sends text instead of a screenshot after a turn without game_action, inspect_building or a pixel placement, screenshots stay in the conversation until compaction, every failure names its remedy, set_tax reads a live figure, and three little-used tools were removed.
  14. v1.2.1 · 10 October 2026Comparable with 1.2.0: a fix to what the agent is shown of its popularity.
  15. v1.2.0 · 10 October 2026Not comparable with 1.1.4: the agent is told that population and popularity count, sees more of its settlement, has new tools, and no longer carries a playbook between episodes.
  16. v1.1.4 · 10 October 2026Comparable with 1.1.3: a fix for runs the memory guard ended.
  17. v1.1.3 · 9 October 2026Comparable with 1.1.2: runs play and score the same.
  18. v1.1.2 · 9 October 2026Comparable with 1.1.1: runs play and score the same.
  19. v1.1.1 · 9 October 2026Comparable with 1.1.0: a placement fix.
  20. v1.1.0 · 9 October 2026Not comparable with 1.0.0: the agent sees storage readings and a changed prompt.
  21. v1.0.0 · 8 October 2026The first versioned benchmark, released as v1.

Score Breakdown Oasis by the Sea, Free Build

Each model plays one episode from the same Free Build start on Oasis by the Sea, with 33 min 20 s of game time. The score runs from 0 for doing nothing to 100 for the game a person played on the map.

0 is doing nothing, 100 the person's game · Higher is better

010203040506052.043.643.439.033.831.126.6
Claude Opus 5.5 (medium)
Claude Sonnet 5.5 (medium)
GPT-6.1 Sol (medium)
GPT-6 Luna (medium)
GLM 5.3 Flash (medium)
Kimi K3 (high)
Claude Haiku 5.5 (medium)

Crusader Bench · crusbench.com

The score is 100 × (0.8 × net worth growth + 0.1 × population growth + 0.1 × popularity ÷ 100). Growth is measured on each map from doing nothing, 0, to the person’s game, 1. Population growth and popularity stop at the person’s level; net worth growth can run from −1 to 2. A map’s score is the mean of a model’s full-budget episodes on it, and the score over every map is the mean of its map scores.

Net worth is gold plus every stored good valued at the game’s fixed marketplace sell price: wood 1, stone 7, iron 23, bread, cheese, meat and apples 4, wheat and hops 8, flour, ale and pitch 10. On the Net Worth tab the hatched bar is the baseline, the same save left untouched for the same game time, and the grey bar is the person's game. Scoring in full

Minute by Minute

Economy, settlement, stock and agent metrics, sampled once per game minute.

readings come from the agent's own turns, so a minute without one repeats the last

Economy
Settlement
Stock
Agent
Net worth · gold, by game minute
05,00010,00015,00020,000051015202530Baseline (do nothing) · 1,220A person, at the end · 19,513
  • Claude Opus 5.5
  • Claude Sonnet 5.5
  • GPT-6.1 Sol
  • GPT-6 Luna
  • GLM 5.3 Flash
  • Kimi K3
  • Claude Haiku 5.5
  • Baseline (do nothing), final
  • A person’s game on the same map, final
Show the data as a table
MinuteClaude Opus 5.5Claude Sonnet 5.5GPT-6.1 SolGPT-6 LunaGLM 5.3 FlashKimi K3Claude Haiku 5.5
01,2951,2751,2801,2801,2651,2751,270
11,3841,0901,3891,3671,3711,3711,394
21,0761,0481,0551,2181,1801,3511,380
39028387541,2181,0651,3501,392
49358437041,1901,0131,3501,360
51,0867477049831,0901,1591,171
61,1348389011,0069511,0921,185
71,1389727241,0049798961,250
81,1271,1997241,1837958551,357
91,2781,2531,3591,3267239071,380
101,4831,4091,1451,4321,0707591,405
111,7881,7581,1301,4671,0858091,490
121,8521,9691,1301,6381,1419031,642
132,4012,2451,7361,9591,2071,2731,673
142,3032,4601,6981,9641,5641,3791,763
152,6802,4601,8461,9641,6061,4312,061
162,2983,0431,7842,5611,7851,7762,421
172,5643,0431,7842,5611,9071,9472,597
182,7963,0432,7463,0592,2832,0852,708
192,9003,8512,8523,2812,2832,2243,002
203,2593,8512,8523,2812,2832,2822,925
213,5703,8512,8523,7293,0711,9542,984
223,9753,8514,0593,4153,0711,9543,094
234,3473,8514,0593,4153,6532,7523,220
244,8155,6274,8334,0993,6772,7523,318
254,7225,6275,0164,2464,2013,2453,523
265,5335,6275,4574,2464,1923,6153,657
276,0466,8835,5304,8304,1923,6553,906
286,5256,8835,5304,8304,6703,9684,225
297,2446,8835,5304,8304,6703,9684,225
307,8116,8836,7224,8305,1364,7444,728
318,6076,8836,6976,3885,1364,7444,893
329,2398,2616,8356,3885,1365,2604,957
339,2398,2617,2176,3886,1195,2604,957
3410,0548,6337,4017,3486,2005,9185,113

Crusader Bench · crusbench.com

Resources

Gold, stored goods and food at the end of each episode.

Final reading after the harness pauses the game · units unless stated

Crusader Bench · crusbench.com

Selling at the marketplace converts goods to gold at the price the score already uses, so it does not change net worth. It does free up the stockpile, and production stops when the stockpile is full.

Settlement

The people, the popularity and the buildings each model left at the end, read from the game after the final pause.

Final reading · the game sends no more peasants while 24 stand idle at the campfire

Crusader Bench · crusbench.com

Population and popularity make up 20% of the score, 10% each; population is measured as growth from doing nothing towards the person’s game. Worker efficiency rises with fear, from gallows, gibbets and the game’s other “bad things”, and falls with “good things” such as gardens. The person’s game on Oasis by the Sea ended with 202 people in 202 places and popularity 100.

Tokens & Cost

Tokens each game used, what it cost, and its score against that cost.

cache reads, cache writes, uncached input and output

  • Cache reads
  • Cache writes
  • Uncached input
  • Output
  • Shades of each model’s colour
02.0M4.0M6.0M8.0M6.4M5.9M5.8M5.0M4.1M3.8M2.5M
Kimi K3 (high)
GPT-6 Luna (medium)
Claude Opus 5.5 (medium)
GPT-6.1 Sol (medium)
GLM 5.3 Flash (medium)
Claude Haiku 5.5 (medium)
Claude Sonnet 5.5 (medium)

Crusader Bench · crusbench.com

The game is paused while the model thinks, so token use does not affect how much game time it gets. Each turn that runs the game lasts at least 8 game seconds, which limits the number of requests in a game and therefore its cost. A cost is what OpenRouter billed; where an API reports tokens but no bill, such as Anthropic’s and Moonshot’s, it is the game’s tokens at the model’s list prices, marked ~. The interview after an episode is not in the game’s cost.

Telemetry

Turns, tool calls, building placements, errors and time for each episode, from its record.

Per episode · inference time is spent with the game paused

Crusader Bench · crusbench.com

Tool Usage

Calls to each of the agent's tools.

Calls per tool · darker is more

ToolKindClaude Opus 5.5Claude Sonnet 5.5GPT-6.1 SolGPT-6 LunaGLM 5.3 FlashKimi K3Claude Haiku 5.5
find_sitesBuild39223059324032
build_structureBuild28162824203024
wait_and_observeLook27171013141843
go_to_tileCamera33131116132512
market_tradeEconomy18610728101
building_infoReference7·82117·
set_taxEconomy9547557
inspect_buildingEconomy103·5171
observeLook73·4114
place_nearBuild1·72·63
center_onCamera2·43·52
expand_storageBuild5211122
map_overviewCamera3112123
set_rationsEconomy1252··2
statusEconomy·1·10···
update_planMemory··43·1·
get_inventoryEconomy···5·1·
notebook_writeMemory1····2·
flat_viewLook·····1·
All tools19191123184117163136

Crusader Bench · crusbench.com

The agent has 26 tools, plus three playbook tools in a learning series. The full list

Results

Every game on Oasis by the Sea with its score, the settlement it left, its cost and whether it played the full game time.

#ModelScoreNet worthPopulationPopularityCostTokensStatus
1Claude Opus 5.5
Anthropic · reasoning medium
52.0
10,054
+8,834 · 48% of the person's gain
107
of 138 places
82~$3.065.8MFull budget
Played by the maintainers
2Claude Sonnet 5.5
Anthropic · reasoning medium
43.6
8,633
+7,413 · 41% of the person's gain
60
of 74 places
84~$0.722.5MFull budget
Played by the maintainers
3GPT-6.1 Sol
OpenAI · reasoning medium
43.4
7,401
+6,181 · 34% of the person's gain
130
of 130 places
100$0.955.0MFull budget
Played by the maintainers
4GPT-6 Luna
OpenAI · reasoning medium
39.0
7,348
+6,128 · 33% of the person's gain
86
of 154 places
81$0.115.9MFull budget
Played by the maintainers
5GLM 5.3 Flash
Z.ai · reasoning medium
33.8
6,200
+4,980 · 27% of the person's gain
64
of 210 places
90$0.154.1MFull budget
Played by the maintainers
6Kimi K3
Moonshot AI · reasoning high
31.1
5,918
+4,698 · 26% of the person's gain
76
of 98 places
69~$3.556.4MFull budget
Played by the maintainers
7Claude Haiku 5.5
Anthropic · reasoning medium
26.6
5,113
+3,893 · 21% of the person's gain
61
of 82 places
67~$0.083.8MFull budget
Played by the maintainers
·A person
Reference: the same map for the same game time, played by a person
100.0
19,513
+18,293
202
of 202 places
100——Reference
·Baseline (do nothing)
Reference: the game runs untouched for the same budget
0.0
1,220
±0
40——Baseline

Crusader Bench · crusbench.com

Inside a Game

A closer reading of one game, from its transcript, its interview and the analysis written from its logs.

Oasis by the Sea · v1.5

Nineteen Kinds of Building

Quote

“They played to maximize throughput; I played to protect a balance sheet, and throughput won by 3x.”

Model
Kimi K3
Source
Its interview, after the game

Kimi K3 ran the broadest economy of the first five games on version 1.5: a bread chain, a brewery and an inn, a church beside its chapel and eight gardens. It finished at 5,918, fourth of those five, and said afterwards that it had played to protect a balance sheet.

It built 19 kinds of building where each of the other four models built 11, and drew popularity from five sources, food variety, religion, taxes, the inn's ale and its gardens, where the others had two or three. The gardens cost it: good things slow the workers, and its efficiency ended at 90%. In its interview it called them the worst trade it made all game.

The scale was small. Four iron mines, three quarries and four bakeries against the person's 13, 10 and 57, and wood ran short for much of the middle game. Like each of the first five models on this save, its population stopped when 24 peasants stood idle, at 76 with 22 places empty; it guessed at a blocked path from the signpost. The prompt's list of sell prices leaves out wheat and hops, which the score counts at 8 each, and Kimi took them as unscored.

It was the slowest and dearest of the five: 134 turns over 63 real minutes, and $3.55 at Moonshot's list prices.

Read the full game report