One episode · The Sinkhole

Claude Haiku 5.5

Benchmark 1.7.1AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Haiku 5.5’s results and comparisons

Score

26.3

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

12,104

+9,909 over doing nothing, 25% of the person’s gain of 39,436

Population

14% of the person’s gain of 502 · popularity 48

Cost

~$0.07

3.8M tokens; ~ at list prices, the API reports no bill

Time

45 game min

37 min of wall-clock time, 68 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Haiku 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

12,104net worth

Gold: 2,935
Stored goods, at sell price: 9,169
Over the baseline (do nothing): +9,909
Population / housing: 74 / 74
Popularity: 48
Food: 212
Structures (map-wide): 101
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock9,169 gold in goods

Good	Units	Price	Value
Gold	2,935	1	2,935
Iron	325	23	7,475
Apples	212	4	848
Wood	336	1	336
Swords	6	30	180
Stone	20	7	140
Bows	8	15	120
Spears	7	10	70
Net worth			12,104

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	12,104	41,631	2,195
Gold	2,935	39,977
Stored goods, at the sell price	9,169	1,654
Population	74	506
Housing	74	506
Popularity	48	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 8
Popularity next month: −8
Food: +4
Taxes: −12
Taxes: Extortionate taxes, 103 gold a month
Rations: extra; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 1 empty; 2 granaries holding 212 food

Buildings65

Iron Mine18
Apple Orchard13
Woodcutter10
Hovel8
Stockpile4
Dairy Farm3
Fletchers' Workshop2
Granary2
Armory1
Blacksmith's Workshop1
Keep1
Marketplace1
Poleturner's Workshop1

Tool calls128

wait_and_observe43
find_sites30
build_structure24
place_near8
set_tax8
inspect_building5
market_trade3
set_rations3
building_info1
expand_storage1
map_overview1
status1

Play

Turns: 68
Buildings placed / attempted: 51 / 55
Anchor-tool placements: 8
Tool errors: 1
Raw inputs sent to the game: 433
Context compactions: 3
Game time: 45 min

Compute

Tokens: 3.8M
Served from cache: 95%
Output tokens: 27K
Cost, at list prices: ~$0.07
Wall-clock time: 37 min
of which inference: 6 min
Peak game memory: 3,547 MiB

The plan, before play

Understanding: This is a 45-minute game-time economy race on The Sinkhole map. Net worth is the main score (gold plus stored goods at sell prices, plus buildings only through what they produce), then population, then popularity. Military buildings and walls are forbidden. Buying goods lowers net worth, so spend gold only where buildings pay back.

Opening plan:

Wood first: place 2–3 woodcutters near the grove about 40 tiles down-left of the keep, using find_sites to pick spots close to trees.
Housing: build hovels (each +8 housing) near the keep so peasants arrive, keeping idle peasants low.
Food: apple orchards and dairy farms on the oasis ring near the granary so food reaches storage directly; add a well later if needed.
Stone and iron: once wood flows, place a quarry with an ox tether on the stone deposit and an iron mine on the iron ore, with stockpiles nearby (expand_storage as needed).
Keep taxes at "no taxes" until popularity and food are solid, then raise taxes for steady gold; watch storage room and idle peasants throughout.
Use a marketplace for selling surplus, and workshops feeding an armory once iron and wood are steady.
Check status every few minutes, fix idle buildings with inspect_building, and keep building through the full 45 minutes.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Haiku 5.5 (claude-haiku-5-5) on Anthropic's API, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
Run 7afa7fa4, series 20261015T132727-8abdde61, 2026-10-15: Haiku's first game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its 1.6.2 game

Result

	This game (1.7.1)	Its 1.6.2 game	The person	Left alone
Net worth	12,104	3,312	41,631	2,195
Gold	2,935	1,716	39,977
Stored goods, at the sell price	9,169	1,596	1,654
Population / housing	74 / 74	37 / 58	506 / 506	4
Popularity	48 (−8 next month)	51	99	0
Score	26.3	8.0	100	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 9,909 over leaving the game alone, 25% of the person's growth, against 1,117 (3%) in the 1.6.2 game. The score is the project website's formula (80% net worth, 10% each population and popularity); this game counts on the website's leaderboard. On the 1.6.2 round's scale it would sit between Claude Opus 5.5 (31.4) and GPT-6 Luna (25.5); GLM 5.3 Flash's 1.7.1 game scored 22.6.

Of the stored value, 7,475 is 325 iron from eighteen iron mines, 848 is 212 apples, 370 is 21 weapons, 336 is 336 planks and 140 is the starting 20 stone. The town ended with every hovel full, 8 idle peasants and no open jobs.

How the game went

Wood, apples and housing, 0 to 9 minutes. Two woodcutters and two hovels in the first turn, then apple orchards one or two at a time, two more hovels at 2:20, a first iron mine at 3:14, a dairy farm at 4:11 and a marketplace at 6:03. The population reached 32 by 4 minutes with 24 idle. Food fell from 55 to 17 by 6:38, when it set half rations, bought 20 apples at 7:35 and went back to full rations at 8:58.

Iron mines, 9 to 26 minutes. Three woodcutters at 9:12, three iron mines at 10:21, 75 planks bought at 12:00, then iron mines in twos and threes (12:45, 17:54, 23:51, 24:48) with orchards, dairy farms and two hovels beside the keep at 15:45. Taxes went from none to high at 18:56, to low at 20:11 and back to none at 25:58, as popularity fell from 100 to 63. A fletcher's workshop at 21:03 and an armory at 22:01 (one place_near against the stockpile found no room first).

The last buildings, 26 to 35 minutes. Two more hovels at 26:03 lifted housing to 74, which was full by 29 minutes. A second fletcher and a blacksmith at 28:00, a poleturner at 31:20, three more iron mines at 33:01 (eighteen in all) and a second granary at 34:27, its last building. place_near found no room against the keep for a hovel or the poleturner at 30:07, and no other hovel was tried.

Taxes, 35 to 45 minutes. With popularity at 90 it set extortionate taxes at 35:33; popularity fell to 46 by 37:23, when it took taxes off, gave a small bribe and extra rations at 38:08 and set low taxes at 38:52. Popularity came back to 78, and at 43:05 it set extortionate taxes again for the last two minutes, ending at 48 and falling. Iron kept coming in at about 15 a minute, and net worth rose from 7,219 at 35 minutes to 12,104.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	2,289	2,304	2,343	3,084	4,822	5,941	7,219	9,502	12,104
Gold	2,000	2,000	1,840	1,540	1,563	1,931	1,750	1,650	2,075	2,935
Iron	0	0	4	17	47	90	136	186	252	325
Planks	55	25	40	92	64	190	205	195	310	336
Food	55	31	48	45	59	119	172	209	239	212
Population / housing	6 / 10	32 / 42	38 / 42	42 / 42	58 / 58	58 / 58	74 / 74	74 / 74	74 / 74	74 / 74
Popularity	100	100	92	100	97	67	74	87	61	48

Each column is the last reading at or before that minute.

Beside its 1.6.2 game

The 1.6.2 game built no mine or quarry and kept 920 in bows and spears from wood; this one put its workers into eighteen iron mines and kept the iron, which is 62% of its final net worth. Both stalled on housing and taxes: 37 people there, 74 here, each time with taxes changed back and forth (14 changes there, 8 here).

What held it back

Housing. From 15 minutes on the population equalled the housing, 42, then 58, then 74; it built 8 hovels in all and none after 26:03. Food piled up meanwhile, 212 apples at the end, and the town was on extra rations for the last 7 minutes.
One food and no quarries. Thirteen apple orchards and three dairy farms, but only apples were eaten at the end (food +4 came from extra rations); no bread, ale or inn, so popularity had only rations and taxes to move it. Stone stayed at the starting 20.
Taxes as a switch. No taxes for most of the game, then extortionate taxes twice, each costing 30 or more points of popularity in a few minutes. Gold rose only from 1,540 at its lowest to 2,935.
Raw iron, not sold or worked. 325 iron counts 23 each; the four weapon workshops made 21 weapons, all kept.

What worked

Eighteen iron mines delivering about 15 iron a minute by the end, the largest part of the score.
Cheap and steady: every one of its 68 requests read the prompt cache.
Every placement reported as placed is in the game's final count of its buildings.

Market

20 apples and 75 planks bought for 460 gold, a loss of 305 net worth against the sell price (planks 4 for 1, apples 8 for 4); 100 planks sold for 100, which leaves net worth unchanged. The reply to that sale said 40 planks and 40 gold; the recording shows 100 planks and 100 gold (wood 372 to 272, gold 2,251 to 2,351), the market_trade fault of versions up to 1.7.3: the game sells 50 a click from 200 in stock and the tool expected 20, so it sold twice the 50 planks asked.

Cost and speed

68 turns, 128 tool calls (43 waits, 30 find_sites, 24 build_structure, 8 place_near, 8 set_tax), 3.80 million tokens and $0.073 at Anthropic's list prices (preparation $0.004, play $0.064, compaction $0.005), with 95.8% of input read from the cache. The interview cost about $0.001 more, with 56,156 of its 58,552 tokens read from the cache. The agent run took 36.9 real minutes, 6.5 of them waiting for the model; the context was compacted three times (31 seconds).

The interview

Haiku put the gap down to the economy's structure: its stored value was raw iron, wood and apples, and it never built bread or ale, so food variety stayed at one and held popularity back. It named taxes left at none too long and then raised too fast, and no market loop. Playing again it would build food and housing as one chain (wheat, mill and bakery by the granary), add hovels as fast as food allows, build inns, tax moderately once popularity is high, and keep raw iron and wood from filling the stockpile. On the benchmark it found the production and storage readings the most useful, placement the main time sink ("many placements failed on ground that looked free"), and the month's delay in popularity the hardest thing to judge.

For the harness

place_near found no free spot against the keep (30:07, twice) or the stockpile (21:18), each with a remedy naming find_sites; Haiku did not try another spot for a hovel after that.
No provider errors, no stalled replies, no refused calls. The game saved at the end ("claude haiku 5-5 1-7-1 7afa7f", in this folder).
market_trade read its one sale (100 planks from 372) as 40: the game sells 50 a click from 200.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	12,104	41,631
Gold	2,935	39,977
Stored goods, at the sell price	9,169	1,654
Population	74	506
Housing	74	506
Popularity	48	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Claude Haiku 5.5

My net worth was 12,104 against the person's 41,631, with 74 people against 506 and popularity 48 against 99. The gap is mostly in the economy's structure, not in how hard I worked. My stored goods (9,169) were mostly raw materials I had mined or cut and never converted: about 310 wood, 315 iron, 217 apples and 20 stone. Raw iron at 23 and wood at 1 are weak stored value. Gold was 2,935 against their 39,977, and that gap is the real difference. Their population of 506 comes from bread: wheat, mill, bakery, then food that every hovel's residents eat. My 74 population was capped by 10 hovels' worth of housing and I never built the food chain that would have justified more.

What I think made the difference, in order:

I never built a bread or ale chain, so my food was apples and a few cheese. That kept my food variety at one type for most of the run and held popularity back, so I could not tax or grow.
I built iron mines (18) and blacksmiths/workshops whose output was small (bows, a few swords). Iron was the largest part of my stored value, but it sat in the stockpile.
I spent too long at "no taxes" early. I only started taxing late, and when I did, popularity fell fast and I had to reverse it (bribe, rations, tax back down). That cost gold and population.
I never got a marketplace trade loop going. I sold only small amounts of wood, which is roughly neutral.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Claude Haiku 5.5

I would start with food and housing as one chain, not as separate tasks. Concretely: wheat farms on the oasis ring early, a mill and bakery beside the granary, and bread as the main food. Then hovels as fast as the food supply allows, and keep food variety above one type so popularity stays near 70 or more. I would build inns once ale is available so popularity has a second lever, and I would set taxes at a moderate level as soon as popularity was comfortably high, not at zero for the whole first half. I would also keep the stockpile from filling with raw iron and wood: either turn them into goods that sell for more, or put the mines and woodcutters on a smaller schedule. The person's approach shows the scale is reachable: a few hundred people is possible within 45 minutes if food and housing grow together.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Haiku 5.5

The time limit and the screenshot-plus-stats loop worked well. The stats readout, especially the per-building production numbers and storage room, was the most useful part because it told me quickly which buildings were idle or full. The bulk of my time went to placement: many placements failed on ground that looked free, and place_near often found no spot against the keep. Find_sites and tile targets were more reliable. The hardest part was judging popularity and taxes, because the effects land with a delay of about a game month, so a change looked fine for a while and then dropped sharply. A short note on expected popularity lag per tax level would help. I would also welcome a clearer readout of goods that count toward net worth and of what a building is worth per minute, so I could compare options directly.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Haiku 5.5

The run ended cleanly at the clock, and I stayed within the rules: no walls, towers, soldiers or defences were built. My main lesson is that I optimised for buildings that produced something quickly (woodcutters, iron mines, orchards) and under-invested in the chain that multiplies population and popularity. The person's result shows that population is where the value compounds, so it should drive the early plan.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	12,104	Counted

What ran

Benchmark: The Sinkhole construction
Benchmark version: 1.7.1
Fingerprint: 63921aad76e5
Harness commit: 674e7ba
Prompt / tools SHA-256: 3b0fa8a / f75ebb8
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261015T132727-8abdde61