One episode · The Sinkhole

Claude Sonnet 5.5

Benchmark 1.7.1AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Sonnet 5.5’s results and comparisons

Score

33.1

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

12,629

+10,434 over doing nothing, 26% of the person’s gain of 39,436

Population

100

19% of the person’s gain of 502 · popularity 100

Cost

~$0.71

2.6M tokens; ~ at list prices, the API reports no bill

Time

45 game min

32 min of wall-clock time, 47 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Sonnet 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

12,629net worth

Gold: 1,955
Stored goods, at sell price: 10,674
Over the baseline (do nothing): +10,434
Population / housing: 100 / 106
Popularity: 100
Food: 71
Structures (map-wide): 115
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock10,674 gold in goods

Good	Units	Price	Value
Gold	1,955	1	1,955
Iron	315	23	7,245
Flour	101	10	1,010
Bows	53	15	795
Wheat	93	8	744
Spears	41	10	410
Apples	61	4	244
Stone	20	7	140
Wood	46	1	46
Bread	10	4	40
Net worth			12,629

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	12,629	41,631	2,195
Gold	1,955	39,977
Stored goods, at the sell price	10,674	1,654
Population	100	506
Housing	106	506
Popularity	100	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: +6
Food: +5
Taxes: +1
Taxes: No taxes, 0 gold a month
Rations: extra; 2 of 2 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 0 empty; 1 granary holding 71 food

Buildings79

Iron Mine17
Hovel12
Apple Orchard11
Woodcutter10
Fletchers' Workshop5
Bakery4
Stockpile4
Armory3
Dairy Farm3
Poleturner's Workshop3
Wheat Farm3
Granary1
Keep1
Marketplace1
Mill1

Tool calls100

find_sites30
wait_and_observe20
build_structure17
place_near8
market_trade6
observe5
set_tax5
set_rations4
expand_storage2
building_info1
inspect_building1
map_overview1

Play

Turns: 47
Buildings placed / attempted: 64 / 69
Anchor-tool placements: 9
Tool errors: 1
Raw inputs sent to the game: 494
Context compactions: 1
Game time: 45 min

Compute

Tokens: 2.6M
Served from cache: 96%
Output tokens: 17K
Cost, at list prices: ~$0.71
Wall-clock time: 32 min
of which inference: 4 min
Peak game memory: 3,540 MiB

The plan, before play

Understanding: I have 45 game minutes to grow net worth, then population and popularity. I can't recruit soldiers or build defences.

Opening plan:

Build 2–3 woodcutters near trees, and hovels for housing.
Build a marketplace, then apple orchards and dairy farms on oasis grass for food.
Add wheat, a mill and a bakery, plus hops, a brewery and an inn for popularity.
Set a low tax once food and popularity are stable.
Add quarries with ox tethers, and iron mines, to produce stone and iron.
Keep expanding storage and housing, and keep building until the end.

Commentary during play3

Build more food, wood and housing.
Hovels here and a granary-friendly wait.
Food is plentiful, so I'll go back to full rations.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Sonnet 5.5 (claude-sonnet-5-5) on Anthropic's API, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
Run 1fcdaf53, series 20261015T153838-6baec514, 2026-10-15: Sonnet's first game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its 1.6.2 game

Result

	This game (1.7.1)	Its 1.6.2 game	The person	Left alone
Net worth	12,629	16,810	41,631	2,195
Gold	1,955	10,777	39,977
Stored goods, at the sell price	10,674	6,033	1,654
Population / housing	100 / 106	83 / 106	506 / 506	4
Popularity	100 (+6 next month)	68	99	0
Score	33.1	38.0	100	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 10,434 over leaving the game alone, 26% of the person's growth, against 14,615 (37%) in the 1.6.2 game. The score is the project website's formula (80% net worth, 10% each population and popularity); this game counts on the website's leaderboard. In this round so far GPT-6 Luna scored 26.7, Claude Haiku 5.5 26.3 and GLM 5.3 Flash 22.6.

Of the stored value, 7,245 is 315 iron from seventeen iron mines, 1,205 is 53 bows and 41 spears, 1,010 is 101 flour, 744 is 93 wheat and 244 is 61 apples. Gold ended below the 2,000 it started with. The town ended with 24 idle peasants, 6 empty places in its hovels and a full stockpile.

How the game went

A fast opening, 0 to 7 minutes. Three woodcutters and two hovels in the first turn, a marketplace at 0:43 and 200 planks bought at 1:00 and 1:40. Orchards, a wheat farm, a dairy farm and three more hovels by 3:29; high taxes at 3:53, low at 5:33. Two fletcher's workshops by the keep at 4:59, an armory at 5:49 (place_near against the stockpile found no room first), and at 6:52 three iron mines, a mill and a bakery.

Hunger, 7 to 19 minutes. Four hovels at 9:22, two iron mines and three more fletchers at 11:19, no taxes from 13:14, four woodcutters at 13:29. Food ran down to 8 by 13:30 with 59 people on full rations; it set half rations at 15:07 and built a bakery and orchards, then three poleturner's workshops at 16:56. Popularity fell to 45 by 20 minutes and the town shrank from 66 to 52. Full rations came back at 18:26.

Iron mines, 19 to 30 minutes. Iron mines in fours at 19:17, 23:56 and 26:23, seventeen in all, two armories added beside the first at 22:17, low taxes from 23:40 and none again from 28:59, two more bakeries at 29:15. At 30:16 expand_storage found no free ground beside the stockpile.

Waiting it out, 30 to 45 minutes. It sold 193 planks at 32:09 to make room, set half rations at 33:04, planted its last orchard at 35:47, and from 38:41 ran extra rations with no taxes, which lifted popularity from 46 to 100. As the stockpile filled again at 42:09 it sold 150 more planks. Net worth rose from 6,210 at 30 minutes to 12,629, as iron and weapons came in.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,705	1,683	2,070	3,022	4,155	6,210	7,764	10,291	12,629
Gold	2,000	1,238	1,310	1,319	1,019	1,019	1,569	1,805	1,805	1,955
Iron	0	0	0	10	36	61	109	161	229	315
Weapons	0	0	2	5	13	26	44	56	73	94
Food	55	32	18	8	39	52	33	17	127	71
Population / housing	6 / 10	35 / 50	50 / 58	59 / 90	56 / 90	67 / 90	89 / 106	97 / 106	95 / 106	100 / 106
Popularity	100	91	72	58	45	63	49	64	46	100

Each column is the last reading at or before that minute.

Beside its 1.6.2 game

The 1.6.2 game ran taxes at average to high from 17 minutes and sold iron, wheat, flour and wood as they came in, about 5,300 gold, ending with 10,777 gold. This game ran no taxes for about 26 of its 45 minutes, sold only planks, and kept its iron, grain and 94 weapons from eight workshops the 1.6.2 game did not build. It had more people (100 against 83) and full popularity, but 4,181 less net worth.

What held it back

No income. With taxes at none most of the game and nothing sold but planks, gold never rose above 2,000; the 1,500 to 1,800 gold it held for much of the run went unused, as it said itself.
Hunger at 15 minutes. Food fell to 8 with the town growing; half rations and the fall in popularity cost fourteen people and several minutes of growth.
Eight weapon workshops. Five fletchers and three poleturners made 94 bows and spears, kept at 1,205, from planks bought at four times their value early on.
A full stockpile. No room to add to it from 30 minutes; by 42 minutes it had room for 15 more iron, 3 planks and 1 flour, and it sold planks to keep the iron coming.

What worked

Seventeen iron mines, delivering about 13 iron a minute by the end.
Popularity 100 at the end, from extra rations and no taxes.
Quick and cheap per turn: 47 turns in 32 real minutes, every request read the prompt cache.
Every placement reported as placed is in the game's final count of its buildings.

Market

200 planks bought for 800 gold, a loss of 600 net worth against the sell price (4 for 1); 343 planks sold for 343, which leaves net worth unchanged.

Cost and speed

47 turns, 100 tool calls (30 find_sites, 20 waits, 17 build_structure, 8 place_near), 2.65 million tokens and $0.715 at Anthropic's list prices (preparation $0.078, play $0.606, compaction $0.030), with 96.6% of input read from the cache. The interview cost about $0.022 more, with 71,248 of its 73,997 tokens read from the cache. The agent run took 32.0 real minutes, 4.4 of them waiting for the model; the context was compacted once (16 seconds).

The interview

Sonnet put the gap down to scale and to turning goods into gold: its value was stored iron and weapons, while the person's bread, sold, "worked as a gold engine". It named 24 idle peasants blocking new arrivals, too little housing, and the early food crisis. Playing again it would build wheat, mills and bakeries at scale and sell the bread, keep hovels and jobs ahead of need, build inns and religion and spend that popularity on rations or taxes, build more iron mines earlier and sell the iron, plan stockpile space, and skip weapon workshops. On the benchmark it praised find_sites, tile targets and the stats, and named as friction calls that came back with no screenshot and a forced wait, sites near the keep running out, and a compaction handoff that lost some context. It said it would have liked to know up front that selling bread is the main route to a high score.

For the harness

No room beside the stockpile (30:16), as in GLM 5.3 Flash's 1.7.1 game: expand_storage found no free ground touching it. Since 1.7.0 the harness keeps a spot beside the stockpile when it picks spots or places by tile; two games of this round have now found none.
place_near found no room against the keep or the stockpile five times (an armory, a poleturner's workshop, hovels, a church), each with a remedy naming find_sites.
No provider errors, stalled replies or refused calls. The game saved at the end ("claude sonnet 5-5 1-7-1 1fcdaf", in this folder).

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	12,629	41,631
Gold	1,955	39,977
Stored goods, at the sell price	10,674	1,654
Population	100	506
Housing	106	506
Popularity	100	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Claude Sonnet 5.5

My net worth was 12,629 against the person's 41,631, about 30% of theirs. Population was 100 against 506. Doing nothing would have scored 2,195, so I did grow the economy, but not by much. Popularity was 100 for both of us, so that wasn't where the gap came from.

The gap came from scale and from converting goods into gold. I held about 10.7k in stored goods, mostly iron (273 units) plus bows and spears. The person held only 1.6k in goods and 40k in gold. Their bread, sold at the market, worked as a gold engine, and their population grew to 506. My population stalled around 100 for most of the run. Idle peasants sat at 24 for a long time, which blocked new arrivals. Housing was only 106. I also lost a lot of early time to starvation and to oscillating rations.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Claude Sonnet 5.5

Food first. I would build the wheat → mill → bakery chain at scale from the start, and sell the bread. I over-built apple orchards, which are worth little, and under-built wheat and bakeries. I would also keep food comfortably positive early instead of letting the granary hit 0 around minute 15.
Population. I would build hovels well ahead of need and keep jobs ahead of idle peasants so growth never stalls. Water pots and other cheap job buildings would help.
Popularity. I would build inns and ale early, plus religion, to buy popularity headroom. I would then spend that headroom on lower rations or on taxes, instead of swinging rations between half and extra.
Iron. I would build many more iron mines earlier, since they were my best earner at about 0.7 iron (about 16 gold) per minute each. I should have sold the iron for gold at the market rather than hoarding it.
Stockpile. I would plan stockpile space so it never filled. Near the end, iron deliveries were blocked by a full stockpile.
Wood. I overbuilt woodcutters for a good worth 1 each.
Weapons. I would skip the weapons workshops. They paid roughly break-even for their 100 gold cost, and the person said blacksmiths made little difference.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Sonnet 5.5

The tools were good. find_sites, tile-targeted build_structure and the free read-only reading were very helpful. The stats JSON was clear.

Friction:

Many calls returned no screenshot and a forced wait.
Sites near the keep ran out quickly.
The compaction handoff lost some context.
Because the game does not pause between actions, planning has a real cost.
The score rewards gold, but I only learned late how far a bread-selling economy could scale. It would help to know up front that bread sales are the main route to a high score.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Sonnet 5.5

I played too cautiously and too reactively. I spent a lot of time recovering from the early food crisis, and I held about 1,500 to 1,800 gold unspent for much of the run. Investing that earlier in wheat, bakeries and housing would likely have compounded. I also never tested the market price of bread against its inputs. That was a missed experiment.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	12,629	Counted

What ran

Benchmark: The Sinkhole construction
Benchmark version: 1.7.1
Fingerprint: 63921aad76e5
Harness commit: 674e7ba
Prompt / tools SHA-256: 3b0fa8a / f75ebb8
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261015T153838-6baec514