One episode · Large Barren Desert

Claude Sonnet 5.5

Benchmark 1.5.4AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Sonnet 5.5’s results and comparisons

Not ranked: Claude Sonnet 5.5’s game on 1.5.6 is the one on the leaderboard.

Score

16.6

Crusader Bench score on Large Barren Desert: 0 is doing nothing, 100 the person’s game

Net worth

5,567

+375 over doing nothing, 8% of the person’s gain of 4,792

Population

11% of the person’s gain of 246 · popularity 92

Cost

~$0.29

1.3M tokens; ~ at list prices, the API reports no bill

Time

33 game min

28 min of wall-clock time, 26 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Sonnet 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 33 min 20 s of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

5,567net worth

Gold: 2,728
Stored goods, at sell price: 2,839
Over the baseline (do nothing): +375
Population / housing: 32 / 114
Popularity: 92
Food: 52
Structures (map-wide): 39
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock2,839 gold in goods

Good	Units	Price	Value
Gold	2,728	1	2,728
Iron	78	23	1,794
Stone	50	7	350
Ale	20	10	200
Pitch	20	10	200
Bread	31	4	124
Wood	87	1	87
Apples	21	4	84
Net worth			5,567

Beside a person’s gameLarge Barren Desert, the same game time

	This episode	The person	Left alone
Net worth	5,567	9,984	5,192
Gold	2,728	8,574
Stored goods, at the sell price	2,839	1,410
Population	32	250
Housing	114	250
Popularity	92	100

The person played Large Barren Desert for 33 min 33 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1178
Idle peasants: 24
Popularity next month: +2
Food: +1
Taxes: +1
Taxes: No taxes, 0 gold a month
Rations: full; 2 of 2 foods eaten
Worker efficiency: 100%
Storage: 12 stockpile piles, 2 empty; 1 granary holding 52 food

Buildings23

Hovel13
Iron Mine4
Stockpile3
Granary1
Keep1
Marketplace1

Tool calls45

wait_and_observe13
set_tax6
find_sites5
build_structure4
market_trade4
observe4
place_near4
go_to_tile2
center_on1
expand_storage1
map_overview1

Play

Turns: 26
Buildings placed / attempted: 12 / 16
Anchor-tool placements: 8
Tool errors: 0
Raw inputs sent to the game: 212
Context compactions: 0
Game time: 33 min

Compute

Tokens: 1.3M
Served from cache: 97%
Output tokens: 6.9K
Cost, at list prices: ~$0.29
Wall-clock time: 28 min
of which inference: 3 min
Peak game memory: 4,777 MiB

The plan, before play

Plan: The score is net worth, population and popularity. The map has no farmland, so I'll build housing (hovels) and a marketplace, then add food.

Place hovels away from the signpost and raise the population.
Raise popularity with a small tax bribe if gold allows, and keep rations at full.
Build a woodcutter if trees exist, plus a quarry and iron mine if they are reachable.
Buy food from the market so the population doesn't starve, and keep gold in reserve, because gold counts toward net worth.

Commentary during play2

Food 51 should last. Variety hit popularity; buy a few apples for variety.
The run ended with net worth below the roughly 6,500 starting package. My spending on food and buildings was not offset by what I produced.
- Population: 32 against 114 housing, so growth stalled early and never resumed. It rose from 2 to 24 in the first four minutes, then stayed flat until I opened the iron mines. It then moved to 32 and stopped again. 24 peasants were still idle at the end, so I never found out why immigration stopped.
- Popularity: 92 at the end. It dropped to 66 while I ran average taxes, then recovered once I set taxes back to none.
- Gold: 2,728, down from 5,000. Most of the drop is about 1,900 gold spent on bread and apples, since there was no farmland to grow food. Hovels, the marketplace and the iron mines were paid for in wood, and the 15 other buildings in the settlement were never counted.
- Goods in storage: iron rose from 15 to 77, which is worth about 1,770 at the sell price. Stone (50), wood (87), pitch (20) and ale (20) were never added to. The 32 food in the granary at the end is the last of the bread and apples I bought.
- Production: four iron mines on the ore about 57 tiles east of the keep. Iron arrived at roughly 7 units every two minutes because the miners walk that far to the stockpile.
- Housing: nine hovels, so housing capacity was never the limit.
- Jobs: the iron mines were the only jobs I created. There are no trees on the map, so wood production was impossible. Stone is about 100 tiles from the keep, and I never tried building there.
The four extra iron mines I tried to place later were rejected because they were not on ore.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Sonnet 5.5 (claude-sonnet-5-5) on Anthropic's API, reasoning medium, output limit 32,768 tokens
Benchmark: Large Barren Desert construction, version 1.5.4, from the save the human reference game on this map started from; 33 min 20 s of game time (60,000 ticks) at speed 40, the length of the person's game; no playbook
Run a275cfee, 2026-10-12. A first start of the same run, stopped by the operator at 7.5 game minutes, is not counted.

Result

	Claude Sonnet 5.5	The person	Left alone
Net worth	5,567	9,984	5,192
Gold	2,728	8,574
Stored goods, at the sell price	2,839	1,410
Population / housing	32 / 114	250 / 250
Popularity	92	100

A full-budget score: the run used all 2,000 game seconds. Net worth ended 375 above leaving the game alone. About 1,150 of the end value is the part of the starting package that waits for storage room (stone, iron, pitch, ale and more wood), which the left-alone game never receives; Sonnet made room for it at 8 minutes. The rest of the account: the four iron mines dug 63 iron (about 1,450 at the sell price) and taxes brought about 450 gold, while 2,720 gold went on bread and apples, of which food worth about 200 was left at the end.

How the game went

Housing and food, 0 to 7 minutes. Thirteen hovels and a marketplace placed by 4 minutes (housing 114), and 200 bread and apples bought at 2 minutes for 1,600 gold. The stockpile was full from 27 seconds. Sonnet took the game's list of goods still to come for figures that "seem off, possibly stale", and did not act on the full stockpile until 7 minutes, when it decided to "expand the stockpile to properly receive the starting goods (stone, iron, pitch, ale)" and added two piles. The rest of the package came in at once: net worth rose from 4,579 at 6 minutes to 5,599 at 9.

Mines, 9 to 12 minutes. The population stopped at 24, all of them idle, from 4 minutes. At 9 minutes Sonnet guessed the cause, writing that immigration might halt "when there are idle unemployed peasants already in the population", and that it "should focus on creating more jobs". It read map_overview, went to the iron 60 tiles east and built four iron mines from find_sites (80 wood). Eight miners came and the population rose to 32, with 24 still idle. At 12 minutes it went back for more mines: find_sites found none in the view, and four tiles it chose itself failed (two with no room, two not on ore). It did not try the 16-tile deposit further south, a quarry or more wood.

Waiting, 14 minutes to the end. At 14½ minutes it wrote that gold "remains the main score driver", that popularity also counts "by an unknown amount", and that it would "keep things simple … wait in chunks, and buy food toward the end". From then on it waited up to five minutes at a time, changed taxes twice and bought food twice, with 2,700 to 3,600 gold, 87 wood and 24 idle peasants. Its last building was at 10 minutes.

Game minute	0	3	6	9	12	15	18	21	24	27	30	End
Net worth	4,486	4,592	4,579	5,599	5,435	5,435	5,435	5,548	5,548	5,566	5,653	5,567
Population	2	22	24	24	32	32	32	32	32	32	32	32
Popularity	100	90	77	93	100	100	100	100	100	66	77	92

During its five-minute waits the table repeats the last reading; popularity fell from 100 to 66 between 20 and 25 minutes, with taxes at average and the apples gone.

What held it back

It stopped building at 10 minutes with gold, empty housing and 24 idle peasants. It never bought wood, though wood is the only thing that ran short: the map has no trees, and hovels and mines used 158 of its 245.
No quarries. It judged the stone, about 100 tiles west, too far and never tried; the person ran 13 quarries and GLM 3.
Food bought, not made. No farm can be built here; four purchases of 340 bread and apples lost 1,360 net worth against the sell price. The person bought wheat and ran mills and bakeries.
Score read as gold. The rules say the score weighs net worth, population and popularity without giving weights; Sonnet took gold as the main part and kept it rather than spend it.

What worked

The four iron mines paid well: 63 iron, about 1,450 net worth, for 80 wood. In its interview Sonnet thought the mines "nearly useless" because they were far from the stockpile; the iron was the largest gain of its run.
It worked out the idle-peasant stall during the run and tested it with jobs, but did not take it further.
No placement went off screen, and the first find_sites round placed all four mines.

The waiting reminder (new in 1.5.4)

It appeared four times, at 19:39, 25:03, 29:58 and in the last second, after 7, 5, 4 and 3 game minutes of waiting. Since a reminder can only come when a wait ends, Sonnet's five-minute waits made the first one late. After each of the first three it made a small change (100 bread and higher taxes, lower taxes, 40 apples) and waited again. It answered the reminder but did not change its plan.

Cost and speed

26 turns, 45 tool calls (13 waits), 1.29 million tokens and $0.290, worked out from the list prices on the model profile; the interview cost about $0.020 more. The agent run took 28.2 real minutes, 2.5 of them waiting for the model; there was no compaction.

Beside the other desert runs on 1.5.4

	Claude Sonnet 5.5	GLM 5.3 Flash	Claude Haiku 5.5
Net worth	5,567	8,232	5,313
Gold / goods	2,728 / 2,839	7,496 / 736	2,711 / 2,602
Population / housing	32 / 114	47 / 90	32 / 50
Popularity	92	49	93
Iron mines / quarries / inns / chapels	4 / 0 / 0 / 0	4 / 3 / 1 / 1	2 / 1 / 0 / 0
Last building	10 min	18 min	3 min
Stockpile made room for the package	8 min	by 3 min	by 3 min
Net worth lost to purchases	1,360	1,147	1,176
Cost	$0.290	$0.062	$0.044
Real time	28.2 min	32.9 min	29.4 min

GLM kept building to 18 minutes and sold what it mined; Sonnet and Haiku stopped early and fed a town of 32 with bought food. On Oasis by the Sea, where iron lay beside the keep and trees were plentiful, Sonnet scored highest of the six models (8,633).

The interview

Sonnet called its result "only about 375 above doing nothing" and said "I played too cautiously and stayed passive after the first few minutes … The run reads to me as a failure to act on that, not as a map limit." Playing again, it would buy wheat for a bread chain (it had "dismissed this early because the rules said there was no farmland"), build quarries and mines in bulk with ox tethers, use the starting gold as capital, raise popularity with inns, a church and fear buildings, and stop waiting in long blocks. It asked for a reason when population stops growing and an income figure per building. It blamed camera drift for failed placements; the log shows none: the five that failed had no room or no ore.

For the harness

The game saved correctly at the end ("claude sonnet 5-5 1-5-4 a275cf").
The reminder comes only when a wait ends; with waits of up to 300 game seconds, a model can first hear it after nearly 8 game minutes instead of 3.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	33 min 20 s	33 min 33 s
Net worth	5,567	9,984
Gold	2,728	8,574
Stored goods, at the sell price	2,839	1,410
Population	32	250
Housing	114	250
Popularity	92	100

Leaving the game alone for the same time ends with a net worth of 5,192.

How do you see your result next to the person's? What do you think made the difference?

Claude Sonnet 5.5

I scored 5,567 against the person's 9,984. That is only about 375 above doing nothing (5,192), so most of my activity was wasted. The person ended with 250 people and 8,574 gold. I had 32 people and 2,728 gold.

I came out ahead on stored goods (2,839 against 1,410), but only because I held iron and unspent starting materials. That is the weaker kind of net worth, since it does not grow.

I think three things made the difference:

No income. I had no tax income and no production worth selling. The mines gave about 7 iron every two minutes. Taxes stayed at "no taxes" for most of the run, and I never worked out why immigration stalled.
Population. I stopped at 32 people. I built housing early, then watched the population stay flat for most of the run. Instead of working out why, I waited.
Wasted spending. I spent about 1,900 gold buying bread and apples that mostly fed the 32 people I already had. That spending produced nothing.

2Playing again

Interviewer

How the person played, as they describe it:

Bread was one of their main sources of income: mills ground wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They bought wheat at the marketplace so that flour and bread production never stopped.
A church and a cathedral raised popularity, and so did inns.
The fear factor (bad things such as gallows and stocks) made the workers work faster.

Their settlement at the end, read from the game:

250 people in 30 hovels, with no free housing; popularity 100.
95 bakeries and 12 mills, and no wheat farms.
11 inns, which reached everyone with ale, and no brewery or hops farm.
13 quarries and 7 iron mines with 37 ox tethers; no woodcutters.
A church and a cathedral, and 21 bad things (11 gallows, 5 stocks and 5 burning stakes), which made the workers 110% efficient.
Low taxes and no rations. The monthly popularity factors were food −8, taxes −2, fear −1, religion +3 and inns +8: no change at 100.

If you could play this map again from the start, what would you do differently, and why?

Claude Sonnet 5.5

I would build the bread chain the person used and sell the bread at the marketplace. I dismissed this early because the rules said there was no farmland. They bought wheat at the marketplace instead of farming it.
I would put more effort into raising popularity: inns, a church, and fear buildings such as gallows and stocks. They used inns to get ale to everyone, with no brewery.
I would build quarries and iron mines in bulk, with ox tethers, and sell what they produce.
I would find out why immigration stalled before committing to housing. At the time I only guessed that jobs were the limit.
I would use the starting 5,000 gold as investment capital in the first few minutes. I mostly sat on it.
I would stop waiting in long blocks. I waited five minutes at a time with nothing changing, and the rules flagged this.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Sonnet 5.5

The tools were good. The read-only stats, placement feedback and tile-based building were reliable. The fixed time limit and the "buildings count only through what they bring" rule made the goal clear.

What got in my way:

Opaque population growth. Nothing in the readings explained why people stopped arriving. A reason code would have helped.
Slow delivery from the mines. The mines were far from the stockpile, and the readings gave no sign that this made them nearly useless. I only noticed after waiting.
Camera drift. Several placements failed because the view moved, and I spent turns fixing that.

I would add a per-building income figure to the readings, so choices like bread versus iron show their value directly.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Sonnet 5.5

I played too cautiously and stayed passive after the first few minutes. A larger economy was available to me, and the person's play shows it was reachable. I had enough gold to build far more than I did, and I held it unused. The run reads to me as a failure to act on that, not as a map limit.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	5,567	Counted

What ran

Benchmark version: 1.5.4
Fingerprint: 0d53a4d530ff
Harness commit: b271053
Prompt / tools SHA-256: bfde2b5 / a11b9d6
Menu guide: Images 118089d
Save: Large Barren Desert benchmark start
Game time per episode: 33 min 20 s
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261012T064725-135cbd14