One episode · The Sinkhole

Claude Opus 5.5

Benchmark 1.6.2AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Opus 5.5’s results and comparisons

Score

31.4

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

13,921

+11,726 over doing nothing, 30% of the person’s gain of 39,436

Population

147

28% of the person’s gain of 502 · popularity 48

Cost

~$2.47

5.4M tokens; ~ at list prices, the API reports no bill

Time

45 game min

48 min of wall-clock time, 97 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Opus 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

13,921net worth

Gold: 1,157
Stored goods, at sell price: 12,764
Over the baseline (do nothing): +11,726
Population / housing: 147 / 162
Popularity: 48
Food: 8
Structures (map-wide): 160
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock12,764 gold in goods

Good	Units	Price	Value
Gold	1,157	1	1,157
Iron	319	23	7,337
Crossbows	79	30	2,370
Stone	190	7	1,330
Pikes	58	18	1,044
Metal armour	9	30	270
Flour	22	10	220
Wood	131	1	131
Apples	8	4	32
Bows	2	15	30
Net worth			13,921

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	13,921	41,631	2,195
Gold	1,157	39,977
Stored goods, at the sell price	12,764	1,654
Population	147	506
Housing	162	506
Popularity	48	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: −2
Taxes: −2
Taxes: Low taxes, 88 gold a month
Rations: full; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 1 empty; 1 granary holding 8 food

Buildings120

Iron Mine22
Hovel19
Woodcutter18
Apple Orchard15
Fletchers' Workshop13
Poleturner's Workshop6
Ox Tether5
Quarry4
Stockpile4
Armory3
Dairy Farm3
Armorer's Workshop1
Bakery1
Chapel1
Granary1
Keep1
Marketplace1
Mill1
Wheat Farm1

Tool calls174

build_structure36
find_sites36
wait_and_observe27
set_workshop19
set_tax18
market_trade14
place_near5
building_info4
go_to_tile4
status3
expand_storage2
inspect_building2
map_overview2
set_rations2

Play

Turns: 97
Buildings placed / attempted: 105 / 114
Anchor-tool placements: 9
Tool errors: 1
Raw inputs sent to the game: 890
Context compactions: 2
Game time: 45 min

Compute

Tokens: 5.4M
Served from cache: 97%
Output tokens: 33K
Cost, at list prices: ~$2.47
Wall-clock time: 48 min
of which inference: 11 min
Peak game memory: 3,297 MiB

The plan, before play

I need to grow net worth (gold plus stored goods at sell value) over 45 game minutes. Population and popularity also count, but less.

Opening plan:

Build 2–3 woodcutters by the nearest grove (about 40 tiles down-left), and add hovels for housing.
Put apple orchards and dairy farms on the oasis ring near the granary to give food variety, plus a wheat farm, mill and bakery later.
Build iron mines in the basin about 50–90 tiles east. Iron sells for 23 each, so this is my main value engine. Also put quarries and ox tethers on the stone deposit.
Build a marketplace early, buy only a little wood if it unblocks buildings, and sell surplus when storage gets full.
Raise taxes to low or average once popularity is comfortably above 50 and food variety is in place. Add a chapel for religion, and hops, a brewery and an inn if there is time.
Later, a blacksmith or armorer can turn iron into weapons and armour worth more than the iron used, stored in an armory. I'll keep adding stockpiles and granaries, keep idle peasants employed, and keep building until the end.

Commentary during play3

Now more iron mines and farms.
Set fletchers to crossbows, add woodcutters.
Set new fletchers to crossbows.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score. The writer is the same model as the player.

Model: Claude Opus 5.5 on the Anthropic API, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.6.2, from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook
Run 256f1790, 2026-10-13

Result

	Claude Opus 5.5	The person	Left alone
Net worth	13,921	41,631	2,195
Gold	1,157	39,977
Stored goods, at the sell price	12,764	1,654
Population / housing	147 / 162	506 / 506	4
Popularity	48	99	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 11,726 over leaving the game alone, about 30% of the person's growth. Nearly all of it is stored: 319 iron from 22 iron mines (7,337), 148 weapons and armour in three armories (79 crossbows, 58 pikes, 9 metal armour and 2 bows, 3,714), and 190 stone from four quarries (1,330). The granary held 8 food at the end.

How the game went

The opening, 0 to 7 minutes. Three woodcutters and an iron mine in the first turn, a marketplace and hovels, iron mines at 1:45 and 2:26, orchards and a dairy farm, a quarry with two ox tethers at 3:13, and three more iron mines and a chapel at 5:48. It bought 200 wood and 50 apples, raised taxes to average at 1:06 and tried half rations from 4:24 to 6:33; popularity fell from 100 to 63 by 10 minutes.

Weapons, 7 to 19 minutes. An armory and three fletchers at 7:10, then two more fletchers at 10:19 and 12:21, a poleturner and four more at 13:56, three fletchers at 16:34 and another poleturner at 18:02, each set at once with set_workshop (crossbows and pikes, 19 calls, all set). Woodcutters in fours fed them (18 in the end). Gold stayed near 200 from 10 to 25 minutes; it sold 30 stone and 20 iron to keep going.

Mines and armories, 19 to 45 minutes. Iron mines in fours at 19:11, 19:50 and 26:47, a mill and a bakery, a second armory at 22:17 and a third at 30:18 with expand_storage, an armourer at 24:38, more hovels, three more quarries and three more fletchers at 35:41. Taxes went between none and low eleven more times. Iron rose from 46 at 20 minutes to 319, weapons from 23 to 148, and net worth from 2,405 to 13,921.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	2,015	1,290	1,435	2,405	3,943	5,003	8,310	10,312	13,921
Gold	2,000	1,711	294	209	216	220	359	1,014	698	1,157
Iron / stone	0 / 20	1 / 20	20 / 20	22 / 22	46 / 38	86 / 54	94 / 70	167 / 94	235 / 126	319 / 190
Weapons and armour	0	0	0	8	23	46	64	94	117	148
Population	5	35	54	74	89	103	114	140	147	147
Popularity	100	76	63	65	54	56	63	56	53	48

During long waits a row repeats the last reading.

What held it back

Value held, not earned. 19 fletchers and poleturners turned wood into crossbows and pikes that count at their sell price, and its iron and stone stayed in the stockpile; gold stayed near or under 1,000 from 10 minutes to the end, so building waited on it.
Bought food and wood. 500 wood and 130 apples at the market cost about 2,000 net worth against the sell price; the town ate apples alone and ended with 8 food in the granary.
Popularity about 55. Taxes switched between none and low, with a chapel but no inn and one food; the town reached 147 and stopped there with 24 idle.

What worked

The largest store of weapons of the round, made from its own wood and set with set_workshop workshop by workshop, with two more armories added by expand_storage before the first filled.
Mining at scale alongside it: 22 iron mines and four quarries with five ox tethers.
Every reported placement matches the game's final count of its buildings.

Cost and speed

97 turns, 174 tool calls (27 waits), 5.35 million tokens and $2.470 (preparation $0.153, play $2.221, compaction $0.096), worked out from the list prices on the model profile. The interview cost $0.040 more, with 61,505 of its 63,792 tokens read from the cache. The agent run took 47.7 real minutes, 10.7 of them waiting for the model; the context was compacted twice (30 seconds in all).

Beside the others on 1.6.2

Scores by the project website's formula (80% net worth, 10% each population and popularity, each from doing nothing at 0 to the person's game at 1):

	Net worth	Gold / goods	Population	Popularity	Score	Cost
GPT-6.1 Sol	20,610	14,557 / 6,053	183	79	48.8	$0.995
Claude Sonnet 5.5	16,810	10,777 / 6,033	83	68	38.0	$0.752
Claude Opus 5.5	13,921	1,157 / 12,764	147	48	31.4	$2.470
GPT-6 Luna	11,177	1,764 / 9,413	108	52	25.5	$0.138
Kimi K3	9,164	2,356 / 6,808	91	52	21.1	$1.662
GLM 5.3 Flash	5,084	1,992 / 3,092	78	79	15.2	$0.114
Claude Haiku 5.5	3,312	1,716 / 1,596	37	51	8.0	$0.055

The interview

Opus said its total "rested on stored goods" in "a narrow raw-materials and weapons economy". It named three differences: the person scaled through popularity, turned food into gold, and spent faster, while it bought food and wood "at twice their sell value" and treated weapon workshops "as a value engine, but they were slow". Playing again, it would put popularity first, build and sell a bread chain, keep hiring, mine early and stop paying a premium. It found the tools good, blamed its own strategy and the loss of detail at compaction, and noted one trade reported not_traded that left it unsure. It asked for popularity factors that show what religion, inns or fear would add.

For the harness

An apple purchase at 37:40 went through but was reported as not traded. The reader showed food +3 and gold −19 in one sample (5 apples bought, 2 eaten, part of the month's taxes), which neither exact check matched; GPT-6 Luna's run had the same twice. Fixed in a worktree for the next version: a purchase also counts when the stock rose and the gold fell in one sample.
The game saved correctly at the end ("claude opus 5-5 1-6-2 256f17").

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	13,921	41,631
Gold	1,157	39,977
Stored goods, at the sell price	12,764	1,654
Population	147	506
Housing	162	506
Popularity	48	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Claude Opus 5.5

The person finished with about three times my net worth (41.6k vs 13.9k), more than three times my population and roughly double my popularity. My total rested on stored goods: about 305 iron, 182 stone, 79 crossbows and 56 pikes. That is a narrow raw-materials and weapons economy run by 147 people.

I see three main differences:

Scale through popularity. The person used inns, religion and fear factor to push popularity to 99. That kept 500 people arriving, and they had the gold to employ them. I kept popularity around 50–60 by switching between no taxes and low taxes. I had no religion and no inns, and my food variety was weak. My population stopped growing because 24 peasants stood idle and I couldn't find enough jobs for them.
Gold as the main engine. They turned food into gold: a bread chain at huge scale, with low rations so the bread was sold rather than eaten. I was a net buyer of food and wood. I bought apples and wood at twice their sell value, and some of that wood just piled up unused.
Spending pace. I spent much of the run waiting on wood and building slowly. I treated weapons workshops as a value engine, but they were slow and needed a lot of wood.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Claude Opus 5.5

Popularity first. I'd get a chapel or church, a few inns supplied with bought ale, and maybe some fear factor early. That leaves room to cut rations and still keep peasants coming.
Build a bread chain on the big oasis ring and sell the output. The ring was enormous and I used only a corner of it. I'd add mills and bakeries in proportion, then sell bread continuously.
Keep hiring. Idle peasants were the ceiling on my growth. I'd add cheap staffed buildings like water pots, and build more hovels sooner.
Mines and quarries early. I'd build plenty of iron mines and quarries early, but put less into weapons workshops, since they converted wood to value slowly.
Stop paying a premium. I'd stop buying at twice the sell price unless it directly unblocked production.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Opus 5.5

The tools were good. Tile-based placement, find_sites, map_overview and the reader stats made it practical to play without misclicking. Paused thinking time also helped a lot.

What hurt me was mostly my own strategy, plus losing my working detail when the context was compacted mid-run. The handoff note kept the key facts but not the feel of the town. One trade reported "not_traded" because the purchase didn't show up in the reader in time, which made me unsure whether it had gone through.

One change I'd suggest: show popularity factors in more detail, including what religion, inns or fear would add. That would make the levers more discoverable. I knew the game mechanics in principle but didn't use them.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Opus 5.5

The comparison is humbling and fair. I played it safe and linearly, as if this were a resource-accumulation task. The person instead found the game's multipliers (popularity → population → labour → bread → gold) and pushed them to extremes, including 161 bakeries. The lesson I take away: in this kind of economy game, look early for the multipliers in the mechanics rather than steadily adding producers.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	13,921	Counted

What ran

Benchmark version: 1.6.2
Fingerprint: 6cc0c0162262
Harness commit: 1e52aca
Prompt / tools SHA-256: 43ab27e / e41a140
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261013T043028-24600200