One episode · Large Barren Desert

Claude Haiku 5.5

Benchmark 1.6.3AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Haiku 5.5’s results and comparisons

Not part of the evaluation: the game is published, but not ranked.

Score

20.1

Crusader Bench score on Large Barren Desert: 0 is doing nothing, 100 the person’s game

Net worth

5,721

+529 over doing nothing, 11% of the person’s gain of 4,792

Population

12% of the person’s gain of 246 · popularity 100

Cost

~$0.04

2.1M tokens; ~ at list prices, the API reports no bill

Time

33 game min

30 min of wall-clock time, 40 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Haiku 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 33 min 20 s of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

5,721net worth

Gold: 2,097
Stored goods, at sell price: 3,624
Over the baseline (do nothing): +529
Population / housing: 34 / 34
Popularity: 100
Food: 2
Structures (map-wide): 43
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

No final overview was published for this episode.

Final stock3,624 gold in goods

Good	Units	Price	Value
Gold	2,097	1	2,097
Iron	63	23	1,449
Stone	138	7	966
Swords	14	30	420
Spears	23	10	230
Ale	20	10	200
Pitch	20	10	200
Bows	9	15	135
Wood	16	1	16
Meat	2	4	8
Net worth			5,721

Beside a person’s gameLarge Barren Desert, the same game time

	This episode	The person	Left alone
Net worth	5,721	9,984	5,192
Gold	2,097	8,574
Stored goods, at the sell price	3,624	1,410
Population	34	250
Housing	34	250
Popularity	100	100

The person played Large Barren Desert for 33 min 33 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1178
Idle peasants: 16
Popularity next month: +3
Taxes: +3
Taxes: Small bribe, -20 gold a month
Rations: full; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 6 empty; 1 granary holding 2 food

Buildings22

Stockpile4
Hovel3
Iron Mine3
Ox Tether2
Quarry2
Armory1
Blacksmith's Workshop1
Fletchers' Workshop1
Granary1
Keep1
Marketplace1
Poleturner's Workshop1
Woodcutter1

Tool calls78

wait_and_observe30
set_tax11
market_trade10
find_sites6
build_structure5
place_near4
observe3
go_to_tile2
status2
center_on1
expand_storage1
get_inventory1
inspect_building1
map_overview1

Play

Turns: 40
Buildings placed / attempted: 13 / 13
Anchor-tool placements: 6
Tool errors: 1
Raw inputs sent to the game: 261
Context compactions: 1
Game time: 33 min

Compute

Tokens: 2.1M
Served from cache: 95%
Output tokens: 18K
Cost, at list prices: ~$0.04
Wall-clock time: 30 min
of which inference: 4 min
Peak game memory: 3,477 MiB

The plan, before play

Understanding of the task:

The goal is to grow net worth the most over a 33-minute game-time budget. Net worth is gold plus stored goods valued at marketplace sell prices, plus buildings only through what they produce. Population and popularity count for less.
No oasis grass exists, so no farms. Food must come from elsewhere (hunting/trade via marketplace), so food is limited to what I can buy or what the granary holds, and apples/dairy/wheat are unavailable.
Stone lies about 100 tiles west and iron about 60 tiles east, so quarries and iron mines will need long hauls. Woodcutters near nearby trees are the first priority.
Housing (hovels, 8 each) gives population; popularity above 50 draws peasants. Taxes stay low at first.
Buying wood at the marketplace early (4 gold per plank, 5 units per lot) can speed construction, but only where the building pays back.

Opening plan:

Place 2–3 woodcutters next to trees near the stockpile, keeping the granary and stockpile area clear.
Build a marketplace early so I can buy planks/stone when needed and sell surplus.
Build several hovels near the keep for population growth, and check popularity and idle peasants.
Set up a quarry with an ox tether on the western stone deposit, and an iron mine on the eastern strip, once wood income is steady and a stockpile is nearby.
Keep taxes at "no taxes" until popularity is well above 50, then raise them gradually; keep rations at full.
Use status and get_inventory to check each stage, and avoid idle buildings and wasted spending.

Commentary during play20

The quarry is staffed and an ox is hauling, so stone should start arriving. The haul is long, so I'll let time pass and check the stockpile.
Stone is still at 50 and iron is creeping up, so the economy is slow to ramp. Popularity is at 100, so I can afford to raise taxes for steady gold.
Stone is now arriving (58). Food is falling, so I'll buy meat, and add a second quarry with a tether to use the idle workers.
Stone has been stuck at 58 for several minutes, so I'll check the quarry directly.
The quarries are running with oxen, so stone should start arriving. I'll top up meat and let time pass before checking again.
Stone has still not moved, and 16 peasants are idle, so jobs rather than housing are the limit. Popularity is 67, well above 50, so I'll raise taxes to gain more gold, then check the effect.
Popularity fell to 52 with a further -6 pending, which risks dropping below 50 and losing peasants. I'm reverting taxes to level 4 and topping up food.
Popularity has dropped to 47, below 50, and a peasant has already left. Setting taxes to "no taxes" gives the best popularity without a bribe.
Population has fallen to 29 and popularity is 48, still below 50. I'll try a small bribe to lift popularity above the threshold and check its cost.
The small bribe costs about 17 gold a month and lifts popularity to 52. Food is down to 20, so I'm buying meat and waiting.
Popularity is back to 60 and population to 34. I'll move to average taxes (level 5), which should settle popularity near 53 while adding gold.
Popularity is falling toward 48 at this tax level. I'll step down to low taxes (level 4) and watch it, switching back to a bribe if it drops under 50.
Food is down to 9 and popularity is 47, so I'm buying a large meat order and restoring the small bribe.
Stone and iron are now accumulating, which adds to net worth. I'll let time pass in longer steps while watching food and popularity.
Popularity is 60 with the bribe, and remaining cost is small, so I'll keep it. Food is 29, so I'm adding a meat lot and waiting.
Popularity has risen to 72, so I can afford to drop the bribe. Setting low taxes (level 4) to collect more gold, and I'll watch popularity.
Food is down to 14 again, so I'm buying a larger lot to cover the remaining nine minutes.
Popularity is sliding (65 to 57), so I'm restoring the small bribe to keep it above 50 for the last seven minutes.
Food is at 16 again, so I'm buying another large meat lot to carry the last five minutes.
Popularity is now 86 and food is 38, enough for the final three minutes. I'll let the clock run out.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Haiku 5.5 (claude-haiku-5-5) on the Anthropic API, reasoning medium, output limit 32,768 tokens
Benchmark: Large Barren Desert construction, version 1.6.3, from the save the human reference game started from; 33 min 20 s of game time (60,000 ticks) at speed 40, the length of the person's game; no playbook
Run a37e8611, series 20261014T063254-d591f30b, 2026-10-14: Haiku's second game on this map, a new series with no memory of the first, played on the latest version (the first game, series 20261012T094211-1fcf305a, played 1.5.6). It is the first valid game on 1.6.3 for this map and moved the map's round there. The map's score for Haiku is the mean of the two.

Result

	This game	First game	The person	Left alone
Net worth	5,721	6,412	9,984	5,192
Gold	2,097	3,428	8,574
Stored goods, at the sell price	3,624	2,984	1,410
Population / housing	34 / 34	32 / 42	250 / 250
Popularity	100 (+3 next month)	51	100

A full-budget score: the run used all 2,000 game seconds. Net worth grew by 529 over leaving the game alone, 11% of the person's 4,792, against 1,220 in the first game. The stored goods are 63 iron (1,449), 138 stone (966), 14 swords (420), 23 spears (230), 20 ale (200), 20 pitch (200), 9 bows (135), 16 wood (16) and 2 meat (8).

How the game went

Everything in eight minutes. A woodcutter and three hovels at 0:10, three stockpile piles at 0:46, two iron mines, a quarry and a tether at 1:02, the marketplace at 1:41, a fletcher's and a poleturner's workshop at 2:08 and 2:18, an armory at 2:44, a third iron mine and a blacksmith's workshop at 4:03, and a second quarry and tether at 7:08. That was the whole town: sixteen buildings and three stockpile piles, with three hovels for 34 people, about the population from 6 minutes to the end.

Meat and taxes, 8 to 33 minutes. No more buildings. It bought meat every two or three minutes, 325 in all, and moved taxes eleven times between average, low, none, high and a small bribe as popularity swung between 47 and 100. Of its 78 tool calls, 30 were waits. Gold ran down from 4,800 at 3 minutes to 2,097 while iron and stone and the three workshops' weapons piled up.

Game minute	0	3	6	9	12	15	18	21	24	27	30	End
Net worth	4,711	5,780	5,275	5,311	5,393	5,250	5,369	5,017	5,025	4,964	4,644	5,721
Gold	4,440	4,800	4,240	4,158	4,164	3,771	3,811	3,147	3,137	2,797	2,257	2,097
Iron	0	15	18	23	27	32	37	41	46	51	56	63
Stone	0	50	50	58	66	74	90	98	106	122	130	138
Food	48	39	48	42	25	46	10	52	15	29	42	2
Population	2	28	34	34	34	31	33	34	34	34	34	34
Popularity	100	100	100	76	55	54	47	62	66	64	84	100

Up to 30 minutes the net worth row counts gold, wood, stone, iron and food (at 4) as the recording logs them; weapons, ale and pitch are left out, so it reads low.

Beside its first game

	This game (1.6.3)	First game (1.5.6)
Net worth	5,721	6,412
Buildings	3 hovels, 3 iron mines, 2 quarries, 3 weapon workshops	4 hovels, 4 iron mines
Population / popularity	34 / 100	32 / 51
Meat bought	325	none
Cost	$0.04	$0.03

Both games stopped at about 33 people. The first kept its gold; this one spent 2,600 on meat and 400 on workshops, and ended 691 lower in net worth with 49 more points of popularity.

What held it back

Housing for 34. Three hovels all game; the population could not grow past them.
Food bought at a loss. 325 meat for 2,600 gold, worth 1,300.
Twenty-five minutes without building. The last building went up at 7:08.

Market

325 meat bought for 2,600 gold, and nothing sold: 1,300 net worth lost against the sell price (8 gold, worth 4). These are units at the market's fixed prices.

Cost and speed

40 turns, 78 tool calls (30 waits), 2.07 million tokens and $0.040 at Anthropic's list prices (preparation $0.004, play $0.035, compaction $0.002). The interview cost about $0.001 more, with 49,158 of its 51,468 tokens read from the cache. The agent run took 30.2 real minutes, 4.2 of them waiting for the model; the context was compacted once (11 seconds). All 40 requests read the prompt cache, 96.8% of the input tokens.

The interview

Haiku said "the real problem is that I held the population at 34 for most of the run": it "assumed no farms meant no food" and bought meat at a loss, its wood stayed weak, and its tax changes were reactive. Playing again, it would buy wheat for a bread chain, build hovels while popularity is above 50, add a church and an inn, and buy wood early. It found the tools good for reading and placing, and asked for more on how buying goods affects net worth and which chains pay off within the time.

For the harness

A stale map probe on the game machine. Like every game of this batch on Oasis by the Sea and Large Barren Desert, this game ran a crusader_probe built from 1.5.0's sources. It was built on 2026-10-13 at 21:28 UTC as a 1.5.0 game started. The 1.6.3 sync then copied sources that matched by hash, and Ninja found nothing to rebuild. The probe's only change since 1.5.0 is 1.5.4's check that tells trees from cacti, so map_overview counted cacti as tree groves. find_sites counts trees from the reader's tile data and was right. Everything else this game played was 1.6.3, and the game stands as played. The desert has no trees, but map_overview listed cactus groves, and find_sites searched around one at (565, 385) as "the nearest trees on the map". Haiku's one woodcutter, at (497, 387), stood where find_sites counted no trees.
The run played on this checkout's version, 1.6.3 (commit 1d0c697), and records it; as the map's first valid 1.6.3 run it moved the Large Barren Desert round to 1.6.3 (--new-round).
place_near found no free spot for an armory at 2:29; it was placed by tile at 2:44.
The game saved correctly at the end ("claude haiku 5-5 1-6-3 a37e86").

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	33 min 20 s	33 min 33 s
Net worth	5,721	9,984
Gold	2,097	8,574
Stored goods, at the sell price	3,624	1,410
Population	34	250
Housing	34	250
Popularity	100	100

Leaving the game alone for the same time ends with a net worth of 5,192.

How do you see your result next to the person's? What do you think made the difference?

Claude Haiku 5.5

My run ended well behind the person: net worth 5,721 against 9,984. The gap is mostly population and production. I finished with 34 people in 34 housing, while they had 250 people in 250 housing. Their gold was 8,574 against my 2,097, and they produced far more. My stored goods were worth more (3,624 against 1,410), but that is mostly stone and iron I mined, not a sign of a working economy. The real problem is that I held the population at 34 for most of the run. Leaving the game alone would have scored 5,192, so my actions added only about 530 net worth over that.

The main reasons, as I see them:

I never found a food source that could sustain growth. I assumed no farms meant no food and bought meat at 8 gold a unit, which sells back at 4. That loss kept the population small and drained gold every month.
My wood income stayed weak. The woodcutter had few trees, and I never built a second one or bought enough planks to expand housing and workshops.
Stone stalled at 58 for most of the run and only started rising late, so the quarry investment paid off very slowly.
Popularity management was reactive. Taxes at level 4 or 5 pushed it below 50 more than once, and I had to fall back on bribes, which cost gold each month.

2Playing again

Interviewer

How the person played, as they describe it:

Bread was one of their main sources of income: mills ground wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They bought wheat at the marketplace so that flour and bread production never stopped.
A church and a cathedral raised popularity, and so did inns.
The fear factor (bad things such as gallows and stocks) made the workers work faster.

Their settlement at the end, read from the game:

250 people in 30 hovels, with no free housing; popularity 100.
95 bakeries and 12 mills, and no wheat farms.
11 inns, which reached everyone with ale, and no brewery or hops farm.
13 quarries and 7 iron mines with 37 ox tethers; no woodcutters.
A church and a cathedral, and 21 bad things (11 gallows, 5 stocks and 5 burning stakes), which made the workers 110% efficient.
Low taxes and no rations. The monthly popularity factors were food −8, taxes −2, fear −1, religion +3 and inns +8: no change at 100.

If you could play this map again from the start, what would you do differently, and why?

Claude Haiku 5.5

The person's approach shows the mistake I made. They did not treat the lack of oasis grass as the end of food. They bought wheat at the marketplace and ran mills and bakeries, so bread became both food and a sellable product. Once bread was steady they could build hovels to 250 housing, and popularity was held up by churches, a cathedral and inns. The fear buildings then raised worker efficiency.

If I played again, I would:

Set up the bread chain early: buy wheat, build a mill and a bakery, and sell bread. Buying a single good and reselling a finished one is the only way to add value on this map.
Build hovels aggressively while popularity is above 50, and fill them with workers before buying more materials.
Put up a church early for popularity, then add an inn once ale is available. Use taxes at a steady level rather than swinging them.
Build a second woodcutter or buy planks early instead of letting wood fall toward 20.
Check net worth trends at each stage, not just gold and food, so I notice when spending is not paying back.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Haiku 5.5

The tools worked well for reading state and placing buildings, and the stats output (gold, storage, popularity factors) made it easy to tell whether things were moving. The 33-minute budget is tight for a building-heavy economy, so the early phase matters a great deal, and I lost too much time reacting to food and popularity alarms. I would like more clarity on the trade-off between buying goods and net worth, and a note on which production chains pay off within the budget, since I could not work that out in time. The screenshots were small and hard to read for building placement, so I relied on tile targets and reports more than I should have.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Haiku 5.5

The biggest lesson is that I stayed too cautious. I was waiting for conditions to be safe rather than building toward the population and production the game allows. The person's result shows that even a desert map with no farms can support a large settlement through bought grain and processing. I would start there next time.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	5,721	Counted

What ran

Benchmark version: 1.6.3
Fingerprint: 96c4682877ca
Harness commit: 1d0c697
Prompt / tools SHA-256: 3bb4a16 / e41a140
Menu guide: Images 118089d
Save: Large Barren Desert benchmark start
Game time per episode: 33 min 20 s
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261014T063254-d591f30b