One episode · Large Barren Desert

Claude Sonnet 5.5

Benchmark 1.5.6AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Sonnet 5.5’s results and comparisons

Score

21.0

Crusader Bench score on Large Barren Desert: 0 is doing nothing, 100 the person’s game

Net worth

6,252

+1,060 over doing nothing, 22% of the person’s gain of 4,792

Population

9% of the person’s gain of 246 · popularity 24

Cost

~$0.42

1.6M tokens; ~ at list prices, the API reports no bill

Time

33 game min

29 min of wall-clock time, 29 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Sonnet 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 33 min 20 s of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

6,252net worth

Gold: 3,299
Stored goods, at sell price: 2,953
Over the baseline (do nothing): +1,060
Population / housing: 27 / 114
Popularity: 24
Food: 0
Structures (map-wide): 39
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock2,953 gold in goods

Good	Units	Price	Value
Gold	3,299	1	3,299
Iron	92	23	2,116
Stone	50	7	350
Ale	20	10	200
Pitch	20	10	200
Wood	87	1	87
Net worth			6,252

Beside a person’s gameLarge Barren Desert, the same game time

	This episode	The person	Left alone
Net worth	6,252	9,984	5,192
Gold	3,299	8,574
Stored goods, at the sell price	2,953	1,410
Population	27	250
Housing	114	250
Popularity	24	100

The person played Large Barren Desert for 33 min 33 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1178
Idle peasants: 19
Popularity next month: −12
Food: −8
Taxes: −4
Taxes: Average taxes, 21 gold a month
Rations: full; no food in the granary
Worker efficiency: 100%
Storage: 12 stockpile piles, 2 empty; 1 granary holding 0 food

Buildings23

Hovel13
Iron Mine4
Stockpile3
Granary1
Keep1
Marketplace1

Tool calls54

wait_and_observe14
set_tax7
place_near6
find_sites5
observe5
market_trade4
build_structure3
go_to_tile3
expand_storage2
set_rations2
center_on1
inspect_building1
map_overview1

Play

Turns: 29
Buildings placed / attempted: 4 / 8
Anchor-tool placements: 16
Tool errors: 2
Raw inputs sent to the game: 202
Context compactions: 0
Game time: 33 min

Compute

Tokens: 1.6M
Served from cache: 95%
Output tokens: 7.8K
Cost, at list prices: ~$0.42
Wall-clock time: 29 min
of which inference: 3 min
Peak game memory: 3,200 MiB

The plan, before play

Understanding: I have 33 minutes of game time on a barren desert map. There are no farms, so net worth has to come from gold, taxes, stone and iron, plus population and popularity. I won't build any defences.

Opening plan:

Build a marketplace and several hovels. Put hovels and the marketplace more than 12 tiles from the signpost.
Set taxes to low or average once popularity allows it, to get steady gold.
Put woodcutters near any trees. If none are close, buy wood at the market.
Look at the map overview for iron to the east, about 60 tiles away. Build iron mines there, with a stockpile nearby if it helps. Quarries lie about 100 tiles west, which is too far to be worthwhile.
Keep the gold. Spend it only on things that return more than they cost.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Sonnet 5.5 (claude-sonnet-5-5) on Anthropic's API, reasoning medium, output limit 32,768 tokens
Benchmark: Large Barren Desert construction, version 1.5.6, from the save the human reference game on this map started from; 33 min 20 s of game time (60,000 ticks) at speed 40, the length of the person's game; no playbook
Run 78f13779, 2026-10-12

Result

	Claude Sonnet 5.5	The person	Left alone
Net worth	6,252	9,984	5,192
Gold	3,299	8,574
Stored goods, at the sell price	2,953	1,410
Population / housing	27 / 114	250 / 250
Popularity	24	100

A full-budget score: the run used all 2,000 game seconds. Net worth grew by 1,060 over leaving the game alone. About 1,150 of the end value is the part of the starting package that waits for storage room, which Sonnet made room for in its first minutes; the iron its four mines dug (77 units, about 1,770) fell short of the 2,320 gold spent on the 290 apples it bought and ate.

How the game went

The opening, 0 to 6 minutes. Hovels beside the keep, a marketplace, more stockpile piles as soon as the stockpile filled ("Stockpile's full, so I should expand storage to let the rest of the starting goods come in"), thirteen hovels in all (housing 114) and four iron mines on the eastern ore by about 5 minutes. A second round of mines at 6:41 found no room (two spots blocked, one not on ore). Taxes went up and down from the first seconds, with half rations for a minute at 2 minutes.

Waiting, 6 to 33 minutes. The population stopped at 32 with 24 idle by 6 minutes. Sonnet named the limit ("peasants only arrive up to 24 idle") and wrote "I should build more mines to absorb that idle labor", but built nothing more after 6 minutes. It bought 100 apples at about 4 and 11 minutes, moved taxes between none and average four more times, then waited 5 game minutes at a time from 13 minutes. The waiting reminder came at 23:02 and 28:15; after each it bought apples (50, then 40) and waited again. At 28 minutes it raised taxes to average with 10 food left and waited out the run: the food ran out, popularity fell from 82 to 24 and five people left in the last minutes.

Game minute	0	3	6	9	12	15	18	21	24	27	30	End
Net worth	4,501	6,247	5,918	6,110	6,236	5,920	5,920	6,020	6,114	6,114	6,019	6,252
Population	2	20	32	32	32	32	32	32	32	32	32	27
Popularity	100	83	77	61	57	48	48	59	70	70	82	24

During long waits the table repeats the last reading.

What held it back

It stopped building at 6 minutes with 3,000 to 4,000 gold, 87 wood, 24 idle peasants and 82 empty places, and never tried a quarry.
One food, bought late. No farm can be built here. Four purchases of 290 apples cost 2,320 gold and 1,160 net worth against the sell price, and the last one left food at 10 for the final 5 minutes.
Nothing for popularity but taxes: no inn or chapel. The end, with average taxes and no food, cost 58 points of popularity and five people.

What worked

Room for the starting package in the first minutes (in its 1.5.4 run, at 8 minutes) and four iron mines by 5 minutes, which dug 77 iron.
A second Marketplace was refused before the menu (already_built, new in 1.5.5), so the view did not jump.

Cost and speed

29 turns, 54 tool calls (14 waits), 1.61 million tokens and $0.415, worked out from the list prices on the model profile. The interview cost $0.196 more: none of its 75,920 tokens were read from the cache, most likely because the run's last request, before a 290-second final wait, came more than five minutes before it. The agent run took 28.9 real minutes, 3.3 of them waiting for the model; there was no compaction.

Beside its run on 1.5.4 and the others on 1.5.6

	Sonnet, 1.5.6	Sonnet, 1.5.4	GLM 5.3 Flash, 1.5.6	Claude Haiku 5.5, 1.5.6
Net worth	6,252	5,567	6,655	6,412
Gold / goods	3,299 / 2,953	2,728 / 2,839	4,560 / 2,095	3,428 / 2,984
Population / housing	27 / 114	32 / 114	70 / 98	32 / 42
Popularity	24	92	66	51
Iron mines / quarries	4 / 0	4 / 0	4 / 7	4 / 0
Last building	6 min	10 min	29 min	7 min
Net worth lost to purchases	1,160	1,360	2,890	1,000

Sonnet built the same town as on 1.5.4 (thirteen hovels, a marketplace and four iron mines) and ended 685 higher, about what making room for the starting package five minutes sooner is worth. It stopped building sooner than on 1.5.4 and ended with the lowest popularity of the round.

The interview

Sonnet said it "barely grew the economy" and "left the game almost idle for the last 20 minutes", that it "never created jobs" for the idle 24 and "hoarded 3,000+ gold while the clock ran". Playing again, it would buy wheat for a mill and bakery, build inns and a church, "keep jobs ahead of housing so the idle count stays under 24", and make quarries and mines productive with ox tethers. It said the rules and the playbook pushed it toward farms and woodcutters; the run had no playbook, and the desert rules say no farm can be built.

For the harness

The game saved correctly at the end ("claude sonnet 5-5 1-5-6 78f137").
The interview missed the prompt cache, most likely after the run's long final wait (more than five minutes since the last request), at about ten times the cost of its cached 1.5.4 interview.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	33 min 20 s	33 min 33 s
Net worth	6,252	9,984
Gold	3,299	8,574
Stored goods, at the sell price	2,953	1,410
Population	27	250
Housing	114	250
Popularity	24	100

Leaving the game alone for the same time ends with a net worth of 5,192.

How do you see your result next to the person's? What do you think made the difference?

Claude Sonnet 5.5

My net worth was 6,252, against 9,984 for the person. That is only about 1,000 above the 5,192 I would have had by doing nothing. In practice I barely grew the economy. I left the game almost idle for the last 20 minutes.

The main differences:

Population. I stalled at about 32 people because 24 of them sat idle. I never created jobs for them, so no new peasants arrived. The person reached 250.
Popularity. Mine ended at 24, theirs at 100. I swung taxes up and down and never built anything that raised popularity, so it fell.
Income. I bought apples with gold to feed people, which cut net worth. I never built a real income engine. Four iron mines, with a 60-tile walk to the stockpile, brought in only about 60 iron over the whole run. The person turned wheat into bread, sold it, and used inns and a church for popularity.
Waiting. I used wait_and_observe for long stretches instead of expanding.

2Playing again

Interviewer

How the person played, as they describe it:

Bread was one of their main sources of income: mills ground wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They bought wheat at the marketplace so that flour and bread production never stopped.
A church and a cathedral raised popularity, and so did inns.
The fear factor (bad things such as gallows and stocks) made the workers work faster.

Their settlement at the end, read from the game:

250 people in 30 hovels, with no free housing; popularity 100.
95 bakeries and 12 mills, and no wheat farms.
11 inns, which reached everyone with ale, and no brewery or hops farm.
13 quarries and 7 iron mines with 37 ox tethers; no woodcutters.
A church and a cathedral, and 21 bad things (11 gallows, 5 stocks and 5 burning stakes), which made the workers 110% efficient.
Low taxes and no rations. The monthly popularity factors were food −8, taxes −2, fear −1, religion +3 and inns +8: no change at 100.

If you could play this map again from the start, what would you do differently, and why?

Claude Sonnet 5.5

I would do these things:

Build a marketplace, mill and bakery early. Buy wheat at the market, mill it, bake it and sell the bread, since the person found that profitable. I would also check whether buying wheat for flour and bread gives a positive margin.
Build inns with bought ale, a church, and possibly some fear-factor buildings. These keep popularity high without needing farms.
Keep jobs ahead of housing so the idle count stays under 24 and population keeps growing. Higher population means more tax income.
Make quarries and iron mines productive with many ox tethers. I would check each mine's output early and scale what works.
Spend gold on things that grow worth. I hoarded 3,000+ gold while the clock ran.
Stop swapping taxes up and down. I would keep low taxes with high popularity and let population drive income.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Sonnet 5.5

The tools worked well. place_near, find_sites and the status readings were reliable, and the stats made problems easy to diagnose. Several things got in my way:

Many calls returned no screenshot, which led to wasted turns.
The minimum turn length was awkward.
The 8-actions-per-reply limit was never the real constraint.
The rules and playbook pushed me toward farms and woodcutters that cannot exist on this map. It would help to say outright that trading and food processing are the intended route.
The time budget is fine, but I misjudged how much the early minutes matter.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Sonnet 5.5

I played too passively and too cautiously. I stopped exploring options once a basic setup was running. Next time I would test the market margins early, with small trades. I would also keep building until the end, since the person shows growth was available throughout.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	6,252	Counted

What ran

Benchmark version: 1.5.6
Fingerprint: bb6e73f95223
Harness commit: 9c34b50
Prompt / tools SHA-256: ad97396 / 76cbfd6
Menu guide: Images 118089d
Save: Large Barren Desert benchmark start
Game time per episode: 33 min 20 s
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261012T110247-91bd9173