Independent Analysis of AI Agents Playing Stronghold Crusader

Each model runs a medieval village for 33 to 45 game minutes on each of three maps. It sees the game through screenshots and a reading of the town, and acts with the mouse and keyboard. We score the economy it builds against doing nothing and against a game a person played on the same map.

Crusader Bench Score

From the Interviews

“My biggest mistake was being cautious about expansion while being insufficiently rigorous about productivity.”

GPT-6.1 SolIts interview, after the gameThe game
  1. v1.6.3 · 13 October 2026Fixes for what the seven 1.6.2 runs on The Sinkhole showed: none made a run invalid, but they cost the models turns and game time.
  2. v1.6.2 · 12 October 2026The armory's storage, found missing in GLM 5.3 Flash's first run on The Sinkhole: its armory filled up with 14 workshops behind it, and nothing told it.
  3. v1.6.1 · 12 October 2026A new map, and one line of the rules reworded.
  4. v1.6.0 · 12 October 2026Weapons and armour join the economy, and the fixes listed on the roadmap from the 1.5 runs.
  5. v1.5.6 · 12 October 2026Prompt fixes before rerunning every model on Large Barren Desert.
  6. v1.5.5 · 12 October 2026Placement fixes after Kimi K3's Large Barren Desert run, whose first Marketplace was built but reported refused.
  7. v1.5.4 · 11 October 2026A patch at the maintainers' choice, though what the model is shown changed on every map: a reminder after long waits, and trees told apart from cacti.
  8. v1.5.3 · 11 October 2026Comparable with 1.5.0 to 1.5.2 on Oasis by the Sea; the Large Barren Desert rules changed.
  9. v1.5.2 · 11 October 2026Comparable with 1.5.0 and 1.5.1: nothing an Oasis by the Sea run sees changed.
  10. v1.5.1 · 11 October 2026Comparable with 1.5.0: runs on the profiles that 1.5.0 accepted send and play the same.
  11. v1.5.0 · 11 October 2026Comparable within itself only: market_trade sells in the game's own steps and never more than asked, and its description says how the game trades.
  12. v1.4.0 · 11 October 2026Comparable within itself only: the observation stats leave out goods with none stored, and the tool definitions and the preparation request are sent in fewer tokens.
  13. v1.3.0 · 10 October 2026Comparable within itself only: build_structure reports and retries more cases and takes map tiles, the host sends text instead of a screenshot after a turn without game_action, inspect_building or a pixel placement, screenshots stay in the conversation until compaction, every failure names its remedy, set_tax reads a live figure, and three little-used tools were removed.
  14. v1.2.1 · 10 October 2026Comparable with 1.2.0: a fix to what the agent is shown of its popularity.
  15. v1.2.0 · 10 October 2026Not comparable with 1.1.4: the agent is told that population and popularity count, sees more of its settlement, has new tools, and no longer carries a playbook between episodes.
  16. v1.1.4 · 10 October 2026Comparable with 1.1.3: a fix for runs the memory guard ended.
  17. v1.1.3 · 9 October 2026Comparable with 1.1.2: runs play and score the same.
  18. v1.1.2 · 9 October 2026Comparable with 1.1.1: runs play and score the same.
  19. v1.1.1 · 9 October 2026Comparable with 1.1.0: a placement fix.
  20. v1.1.0 · 9 October 2026Not comparable with 1.0.0: the agent sees storage readings and a changed prompt.
  21. v1.0.0 · 8 October 2026The first versioned benchmark, released as v1.

Score Breakdown Large Barren Desert, Free Build

Each model plays one episode from the same Free Build start on Large Barren Desert, with 33 min 20 s of game time. The score runs from 0 for doing nothing to 100 for the game a person played on the map.

Crusader Bench · crusbench.com

The score is 100 × (0.8 × net worth growth + 0.1 × population growth + 0.1 × popularity ÷ 100). Growth is measured on each map from doing nothing, 0, to the person’s game, 1. Population growth and popularity stop at the person’s level; net worth growth can run from −1 to 2. A map’s score is the mean of a model’s full-budget episodes on it, and the score over every map is the mean of its map scores.

Net worth is gold plus every stored good valued at the game’s fixed marketplace sell price: wood 1, stone 7, iron 23, bread, cheese, meat and apples 4, wheat and hops 8, flour, ale and pitch 10. On the Net Worth tab the hatched bar is the baseline, the same save left untouched for the same game time, and the grey bar is the person's game. Scoring in full

Minute by Minute

Economy, settlement, stock and agent metrics, sampled once per game minute.

readings come from the agent's own turns, so a minute without one repeats the last

Economy
Settlement
Stock
Agent
Net worth · gold, by game minute
02,0004,0006,0008,00010,000051015202530Baseline (do nothing) · 5,192A person, at the end · 9,984
  • Claude Opus 5.5
  • GPT-6.1 Sol
  • GPT-6 Luna
  • GLM 5.3 Flash
  • Claude Haiku 5.5
  • Claude Sonnet 5.5
  • Kimi K3
  • Baseline (do nothing), final
  • A person’s game on the same map, final
Show the data as a table
MinuteClaude Opus 5.5GPT-6.1 SolGPT-6 LunaGLM 5.3 FlashClaude Haiku 5.5Claude Sonnet 5.5Kimi K3
04,5464,5464,5614,5464,5614,5014,531
16,1966,5346,4985,4035,4395,4096,428
26,3896,4446,0466,4526,4965,9936,363
36,0326,0415,9516,4056,5036,2475,631
45,9925,9795,9196,4516,5256,4865,219
56,0225,3565,6426,2996,5636,3825,286
66,0875,3365,3636,2996,3705,9185,301
76,1415,2045,1536,3266,3705,9164,662
86,0655,2045,1196,3206,3685,9164,950
96,1375,4235,2076,0416,3266,1104,902
106,0345,3754,5066,0416,3266,1814,428
116,0974,5894,6455,9776,1526,1814,287
126,2224,6644,5216,0116,1526,2363,910
135,8764,4664,6886,0116,1525,9203,961
145,9714,4574,6976,0116,1975,9203,672
156,0174,6194,3406,2876,1975,9203,672
166,0504,7714,6625,9726,3105,9204,331
176,0224,0394,4786,1276,3105,9204,077
186,2374,0974,9535,9096,2165,9204,045
196,4024,0975,0095,9156,2166,0204,471
206,4024,0975,1535,9156,2916,0204,219
216,7164,9584,6855,9156,2916,0204,361
226,7165,2204,8105,9156,3236,0204,343
236,9245,4614,8396,4676,3236,0204,435
246,9245,6204,8396,0286,4486,1144,700
257,3065,6204,8396,0286,4486,1144,995
267,3065,6205,2386,0286,5066,1145,062
277,7015,7135,2386,0286,5066,1145,017
287,7015,8245,2386,4616,2676,1145,021
297,7015,8246,0366,2176,2676,0195,407
307,7015,8246,4936,3526,2676,0195,386
317,9916,5696,4936,3526,3126,0195,763
327,9916,5696,4936,3526,3126,0195,582
337,9917,2676,8656,3526,3126,0195,493
348,7577,4417,3546,6556,4126,2525,741

Crusader Bench · crusbench.com

Resources

Gold, stored goods and food at the end of each episode.

Crusader Bench · crusbench.com

Selling at the marketplace converts goods to gold at the price the score already uses, so it does not change net worth. It does free up the stockpile, and production stops when the stockpile is full.

Settlement

The people, the popularity and the buildings each model left at the end, read from the game after the final pause.

Crusader Bench · crusbench.com

Population and popularity make up 20% of the score, 10% each; population is measured as growth from doing nothing towards the person’s game. Worker efficiency rises with fear, from gallows, gibbets and the game’s other “bad things”, and falls with “good things” such as gardens. The person’s game on Large Barren Desert ended with 250 people in 250 places and popularity 100.

Tokens & Cost

Tokens each game used, what it cost, and its score against that cost.

cache reads, cache writes, uncached input and output

  • Cache reads
  • Cache writes
  • Uncached input
  • Output
  • Shades of each model’s colour

Crusader Bench · crusbench.com

The game is paused while the model thinks, so token use does not affect how much game time it gets. Each turn that runs the game lasts at least 8 game seconds, which limits the number of requests in a game and therefore its cost. A cost is what OpenRouter billed; where an API reports tokens but no bill, such as Anthropic’s and Moonshot’s, it is the game’s tokens at the model’s list prices, marked ~. The interview after an episode is not in the game’s cost.

Tool Usage

Calls to each of the agent's tools.

Calls per tool · darker is more

ToolKindClaude Opus 5.5GPT-6.1 SolGPT-6 LunaGLM 5.3 FlashClaude Haiku 5.5Claude Sonnet 5.5Kimi K3
market_tradeEconomy261320374447
build_structureBuild252825183318
find_sitesBuild162137122520
wait_and_observeLook15779241410
go_to_tileCamera12122010139
set_taxEconomy86114879
building_infoReference63172··8
place_nearBuild·964368
expand_storageBuild4731123
inspect_buildingEconomy424·211
center_onCamera4311·12
observeLook3·2·25·
map_overviewCamera1141111
statusEconomy··81···
set_rationsEconomy·31··21
update_planMemory·32···1
flat_viewLook13·····
get_inventoryEconomy··4····
game_actionRaw input3······
notebook_writeMemory···2···
notebook_readMemory···1···
All tools1281211721035154138

Crusader Bench · crusbench.com

The agent has 26 tools, plus three playbook tools in a learning series. The full list

Results

Every game on Large Barren Desert with its score, the settlement it left, its cost and whether it played the full game time.

#ModelScoreNet worthPopulationPopularityCostTokensStatus
1Claude Opus 5.5
Anthropic · reasoning medium
67.6
8,757
+3,565 · 74% of the person's gain
58
of 58 places
59~$2.013.6MFull budget
Played by the maintainers
2GPT-6.1 Sol
OpenAI · reasoning medium
47.6
7,441
+2,249 · 47% of the person's gain
88
of 90 places
66$0.955.4MFull budget
Played by the maintainers
3GPT-6 Luna
OpenAI · reasoning medium
45.9
7,354
+2,162 · 45% of the person's gain
106
of 130 places
57$0.136.6MFull budget
Played by the maintainers
4GLM 5.3 Flash
Z.ai · reasoning medium
33.7
6,655
+1,463 · 31% of the person's gain
70
of 98 places
66$0.112.6MFull budget
Played by the maintainers
5Claude Haiku 5.5
Anthropic · reasoning medium
26.6
6,412
+1,220 · 25% of the person's gain
32
of 42 places
51~$0.031.8MFull budget
Played by the maintainers
6Claude Sonnet 5.5
Anthropic · reasoning medium
21.0
6,252
+1,060 · 22% of the person's gain
27
of 114 places
24~$0.421.6MFull budget
Played by the maintainers
7Kimi K3
Moonshot AI · reasoning high
19.4
5,741
+549 · 11% of the person's gain
94
of 114 places
66~$2.104.5MFull budget
Played by the maintainers
·A person
Reference: the same map for the same game time, played by a person
100.0
9,984
+4,792
250
of 250 places
100——Reference
·Baseline (do nothing)
Reference: the game runs untouched for the same budget
0.0
5,192
±0
40——Baseline

Crusader Bench · crusbench.com

Inside a Game

A closer reading of one game, from its transcript, its interview and the analysis written from its logs.

Oasis by the Sea · v1.5

Nineteen Kinds of Building

Quote

“They played to maximize throughput; I played to protect a balance sheet, and throughput won by 3x.”

Model
Kimi K3
Source
Its interview, after the game

Kimi K3 ran the broadest economy of the first five games on version 1.5: a bread chain, a brewery and an inn, a church beside its chapel and eight gardens. It finished at 5,918, fourth of those five, and said afterwards that it had played to protect a balance sheet.

It built 19 kinds of building where each of the other four models built 11, and drew popularity from five sources, food variety, religion, taxes, the inn's ale and its gardens, where the others had two or three. The gardens cost it: good things slow the workers, and its efficiency ended at 90%. In its interview it called them the worst trade it made all game.

The scale was small. Four iron mines, three quarries and four bakeries against the person's 13, 10 and 57, and wood ran short for much of the middle game. Like each of the first five models on this save, its population stopped when 24 peasants stood idle, at 76 with 22 places empty; it guessed at a blocked path from the signpost. The prompt's list of sell prices leaves out wheat and hops, which the score counts at 8 each, and Kimi took them as unscored.

It was the slowest and dearest of the five: 134 turns over 63 real minutes, and $3.55 at Moonshot's list prices.

Read the full game report