One episode · The Sinkhole

Claude Sonnet 5.5

Benchmark 1.6.2AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Sonnet 5.5’s results and comparisons

Score

38.0

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

16,810

+14,615 over doing nothing, 37% of the person’s gain of 39,436

Population

16% of the person’s gain of 502 · popularity 68

Cost

~$0.75

3.1M tokens; ~ at list prices, the API reports no bill

Time

45 game min

40 min of wall-clock time, 58 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Sonnet 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

16,810net worth

Gold: 10,777
Stored goods, at sell price: 6,033
Over the baseline (do nothing): +14,615
Population / housing: 83 / 106
Popularity: 68
Food: 176
Structures (map-wide): 96
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock6,033 gold in goods

Good	Units	Price	Value
Gold	10,777	1	10,777
Iron	154	23	3,542
Flour	78	10	780
Wheat	84	8	672
Bread	139	4	556
Wood	163	1	163
Apples	35	4	140
Stone	20	7	140
Hops	4	8	32
Cheese	2	4	8
Net worth			16,810

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	16,810	41,631	2,195
Gold	10,777	39,977
Stored goods, at the sell price	6,033	1,654
Population	83	506
Housing	106	506
Popularity	68	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: ±0
Food: +2
Taxes: −6
Religion: +2
Ale from the inn: +2
Taxes: High taxes, 82 gold a month
Rations: full; 3 of 3 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 0 empty; 1 granary holding 176 food

Buildings60

Iron Mine16
Hovel12
Apple Orchard5
Wheat Farm5
Bakery4
Stockpile4
Dairy Farm3
Woodcutter3
Brewery1
Chapel1
Granary1
Hops Farm1
Inn1
Keep1
Marketplace1
Mill1

Tool calls104

wait_and_observe19
find_sites16
build_structure13
market_trade12
observe11
set_tax11
place_near10
inspect_building4
center_on3
set_rations2
expand_storage1
game_action1
map_overview1

Play

Turns: 58
Buildings placed / attempted: 42 / 45
Anchor-tool placements: 12
Tool errors: 1
Raw inputs sent to the game: 495
Context compactions: 1
Game time: 45 min

Compute

Tokens: 3.1M
Served from cache: 96%
Output tokens: 16K
Cost, at list prices: ~$0.75
Wall-clock time: 40 min
of which inference: 5 min
Peak game memory: 3,258 MiB

The plan, before play

Plan: raise net worth through production and population.

Place 2–3 woodcutters by the trees and a few hovels. Build a marketplace if it is affordable, and set taxes once popularity allows.
Build apple orchards and wheat farms on the oasis grass for food and popularity.
Build a quarry with ox tethers, and iron mines if they are reachable. Stored stone and iron count toward net worth.
Expand storage, keep housing ahead of population, and keep building until the end.

Commentary during play2

Build an orchard, a dairy, and a hops farm, then a brewery and inn.
The 45-minute budget has run out. I'm leaving the settlement with 10,695 gold, population 83 and popularity 68. The run's own start-of-game value for the package was about 2,415.
- Stored goods at the end:
  - 152 iron and 163 wood.
  - 85 wheat and 77 flour.
  - 141 bread and 37 apples.
  - 20 stone and 4 hops.
  - 178 in the granary.
- Idle peasants: 24 stayed idle for the whole late game, and population stalled at 83 of 106 housing. Newcomers stop arriving once 24 stand idle, so I probably needed more job buildings.
- Income: most of the gold came from selling iron, which was neutral to net worth, and from taxes at levels 5–6. I switched taxes between 5 and 6 to keep popularity near 65–75.
- Storage: the stockpile was full at the end, so iron, wheat and flour deliveries had stopped. I couldn't expand it because no free ground touched it. I sold surplus at the market whenever piles filled.
- Production:
  - Iron: 13 mines were running in total, and iron was the main earner.
  - Food: three bakeries, mill, apple orchards, wheat farms and dairy farms fed the granary, with full rations from about minute 15.
  - Unused: I never built a quarry, a blacksmith or an armory.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Sonnet 5.5 on the Anthropic API, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.6.2, from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook
Run 979ef2d1, 2026-10-13

Result

	Claude Sonnet 5.5	The person	Left alone
Net worth	16,810	41,631	2,195
Gold	10,777	39,977
Stored goods, at the sell price	6,033	1,654
Population / housing	83 / 106	506 / 506	4
Popularity	68	99	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 14,615 over leaving the game alone, about 37% of the person's growth. Its gold came about half from selling what it made (5,300 gold for 160 iron, 90 wheat, 80 flour and 100 wood) and about half from taxes, at average to high from 17 minutes. Of the stored value, 3,542 is 154 iron from 16 iron mines; the rest is flour, wheat and bread from its own farms, mill and bakeries.

How the game went

The opening, 0 to 7 minutes. Three woodcutters, two hovels and a marketplace in the first 34 seconds, 200 wood bought in the first three minutes, then four wheat farms, a mill and a bakery, three iron mines at 3:42, and at 6:01 a brewery, an inn and a chapel beside the keep. Taxes went to mean at 34 seconds and came down in steps to none by 6:34; popularity fell from 100 to 60 by 5 minutes.

Mines and bread, 7 to 25 minutes. Half rations from 7:57 to 15:02, iron mines in fours at 9:51, 12:30 and 17:26, three more bakeries, more hovels, orchards and dairy farms, and two more iron mines at 24:07, its last building. Taxes rose to average at 17:26 and mean at 19:01. At 19:17 expand_storage found no free ground around the stockpile. Iron rose from 26 at 15 minutes to 102 at 25, and net worth from 2,618 to 6,386.

Selling and taxing, 25 to 45 minutes. No more building: it waited, kept taxes at average to high, and sold iron, wheat, flour and wood from 28 minutes, 100 iron at once at 33:06. Gold went from 2,295 at 25 minutes to 10,777. The stockpile had no empty pile from 35 minutes, and the status said that goods without room were no longer delivered.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,722	1,992	2,618	3,831	6,386	8,131	10,760	13,789	16,810
Gold	2,000	1,372	1,134	966	1,079	2,295	2,885	5,475	9,641	10,777
Iron	0	0	8	26	59	102	139	134	95	154
Bread / flour	0 / 0	0 / 0	7 / 20	3 / 43	0 / 66	31 / 53	49 / 67	79 / 86	112 / 56	139 / 78
Population	5	36	49	65	77	79	83	83	83	83
Popularity	100	60	56	59	83	61	75	58	74	68

During long waits a row repeats the last reading.

What held it back

No building after 24 minutes. The town stood at 83 people with 24 idle and 23 empty places from about 30 minutes, and 10,777 gold was never spent; Sonnet said so itself in the interview.
A boxed-in stockpile. It could not be extended from 19 minutes, and from 35 minutes it had no empty pile, so flour and other goods without room were not delivered.
Bread kept, not sold. Four bakeries made 139 bread that stayed in the granary at 4 each.

What worked

Second of the seven by net worth, from selling: it sold its iron and grain as they came in rather than holding them, and taxed a stable town of 83.
A food chain of its own (five wheat farms, a mill, four bakeries, orchards and dairy farms), with three foods eaten and an inn and a chapel for popularity: 68 at the end under high taxes.
Every reported placement matches the game's final count of its buildings; the one iron mine reported as possibly_rejected was indeed not built.

Cost and speed

58 turns, 104 tool calls (19 waits), 3.12 million tokens and $0.752 (preparation $0.074, play $0.650, compaction $0.028), worked out from the list prices on the model profile. The interview cost $0.026 more, with 70,699 of its 75,158 tokens read from the cache. The agent run took 40.4 real minutes, 4.9 of them waiting for the model; the context was compacted once (16 seconds).

Beside the others on 1.6.2

Scores by the project website's formula (80% net worth, 10% each population and popularity, each from doing nothing at 0 to the person's game at 1):

	Net worth	Gold / goods	Population	Popularity	Score	Cost
GPT-6.1 Sol	20,610	14,557 / 6,053	183	79	48.8	$0.995
Claude Sonnet 5.5	16,810	10,777 / 6,033	83	68	38.0	$0.752
Claude Opus 5.5	13,921	1,157 / 12,764	147	48	31.4	$2.470
GPT-6 Luna	11,177	1,764 / 9,413	108	52	25.5	$0.138
Kimi K3	9,164	2,356 / 6,808	91	52	21.1	$1.662
GLM 5.3 Flash	5,084	1,992 / 3,092	78	79	15.2	$0.114
Claude Haiku 5.5	3,312	1,716 / 1,596	37	51	8.0	$0.055

The interview

Sonnet answered all four questions in one reply under its own headings, so the harness kept it as a single answer. It put the gap down to population ("506 against 83") and said it "plateaued at 83 people and 24 idle peasants" and then "mostly waited and sold surplus". Playing again, it would treat idle peasants as a signal to build, scale the bread chain, spend gold sooner, leave room beside the stockpile and add quarries. It found tile placement, find_sites and the readings reliable; it asked for the count of unfilled jobs, earlier warning of storage limits, and benchmark strategy hints.

For the harness

expand_storage (stockpile) at 19:17 gave the no-room remedy; Sonnet said it had "no way to see why".
The game saved correctly at the end ("claude sonnet 5-5 1-6-2 979ef2").

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1All questions

Interviewer

Your result

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	16,810	41,631
Gold	10,777	39,977
Stored goods, at the sell price	6,033	1,654
Population	83	506
Housing	106	506
Popularity	68	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Playing again

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

The benchmark

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Anything else

Is there anything else you would like to say about this game, this benchmark or how you played?

Answer all 4 questions above in this one reply. Repeat each heading exactly as written above (for example "## Your result") right before its answer, so the answers can be told apart.

Claude Sonnet 5.5

Your result

The person finished at about 2.5 times my net worth: 41.6k against my 16.8k. The gap is population (506 against 83) and the gold that population produced (about 40k against 10.8k). I plateaued at 83 people and 24 idle peasants from roughly minute 15. After that I mostly waited and sold surplus. The person kept growing housing, jobs and bread output up to the last minute.

My stored goods were worth more than theirs (6.0k against 1.7k), but that was a small part of the score. The scale of the economy mattered far more than the value of what I had in storage.

What I would do differently

Treat idle peasants as a signal to build more. I saw 24 idle for about 30 minutes and only added a few more mines and farms. I should have kept adding job buildings and hovels until idle peasants stayed low.
Build far more bread capacity. The person ran dozens of bakeries and mills and sold the bread. I built three bakeries and let flour and wheat pile up. I should have scaled the whole bread chain early.
Spend gold sooner. I finished with 10k gold sitting unused. I could have put it into hovels, more production buildings, a bigger storage layout and water pots for jobs, and sold surplus whenever storage filled.
Plan storage from the start. The stockpile was boxed in and full by minute 33. I could not extend it, so deliveries of goods stopped. I should have left free ground beside the stockpile.
Use taxes more carefully and add popularity buildings. I swung between tax levels 3 and 7 and lost some popularity. Inns, religion and good food would have let me run lower taxes at higher popularity while population kept growing.
Add quarries. I never built one, although stone sells for 7. The person had ten.

The benchmark

The tools were good. Placing buildings by map tile, find_sites and the status readings were reliable, and the free reading calls let me plan without losing game time. The score is clear and honest: net worth is easy to read, and comparing against a person's game is useful.

What got in the way:

My context was compacted, so I lost track of where some buildings were, and several early turns went on checking basic facts.
Some replies ran for less than the minimum turn length, which wasted calls.
I lost track of the stockpile layout, and expand_storage only reported "no free ground" with no way to see why.
The 45-minute limit is short for a snowball economy, so early decisions matter a lot.

What I would change:

Give a short list of benchmark-level strategy hints, such as "bread chain scales population", since the winning approach is not obvious from the rules.
Show the unfilled-job count directly, so it is clear whether idle peasants mean too few jobs or too little housing.
Make storage limits more visible before they bind.

Anything else

I played too passively in the second half. A long wait_and_observe was easy to repeat when nothing seemed broken, but the lack of problems hid the lack of growth. Next time I would set explicit population and job targets every few minutes and build toward them.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	16,810	Counted

What ran

Benchmark version: 1.6.2
Fingerprint: 6cc0c0162262
Harness commit: 1e52aca
Prompt / tools SHA-256: 43ab27e / e41a140
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261013T023616-1529e12f