One episode · The Sinkhole

Claude Opus 5.5

Benchmark 1.7.1AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Opus 5.5’s results and comparisons

Score

34.7

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

15,424

+13,229 over doing nothing, 34% of the person’s gain of 39,436

Population

164

32% of the person’s gain of 502 · popularity 47

Cost

~$1.98

4.0M tokens; ~ at list prices, the API reports no bill

Time

45 game min

43 min of wall-clock time, 69 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Opus 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

15,424net worth

Gold: 2,186
Stored goods, at sell price: 13,238
Over the baseline (do nothing): +13,229
Population / housing: 164 / 202
Popularity: 47
Food: 71
Structures (map-wide): 178
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock13,238 gold in goods

Good	Units	Price	Value
Gold	2,186	1	2,186
Iron	432	23	9,936
Stone	192	7	1,344
Crossbows	43	30	1,290
Apples	71	4	284
Bows	16	15	240
Wood	144	1	144
Net worth			15,424

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	15,424	41,631	2,195
Gold	2,186	39,977
Stored goods, at the sell price	13,238	1,654
Population	164	506
Housing	202	506
Popularity	47	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: −2
Taxes: −2
Religion: ±0
Taxes: Low taxes, 98 gold a month
Rations: full; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 0 empty; 1 granary holding 71 food

Buildings138

Woodcutter29
Iron Mine27
Apple Orchard26
Hovel24
Fletchers' Workshop10
Ox Tether5
Quarry4
Stockpile4
Dairy Farm3
Armory2
Chapel1
Granary1
Keep1
Marketplace1

Tool calls124

find_sites39
build_structure29
set_tax13
market_trade11
set_workshop8
wait_and_observe8
building_info6
expand_storage2
inspect_building2
map_overview2
place_near2
center_on1
status1

Play

Turns: 69
Buildings placed / attempted: 129 / 139
Anchor-tool placements: 3
Tool errors: 2
Raw inputs sent to the game: 851
Context compactions: 2
Game time: 45 min

Compute

Tokens: 4.0M
Served from cache: 96%
Output tokens: 26K
Cost, at list prices: ~$1.98
Wall-clock time: 43 min
of which inference: 8 min
Peak game memory: 3,515 MiB

The plan, before play

My task is to grow The Sinkhole economy as far as I can in 45 game minutes. The final score counts net worth most (gold plus stored goods at sell price), then population, then popularity. Soldiers and walls are not allowed.

Opening plan:

Wood: build 2–3 woodcutters by the nearest grove, down and to the left of the keep.
Housing and food: put up 2–3 hovels away from the signpost. Build apple orchards and dairy farms on the oasis ring near the granary so the starting 55 meat doesn't run out.
Stone and iron: place quarries with ox tethers on the stone deposit, and iron mines on the ore to the east. Iron is worth 23 a unit and stone 7, so these should drive net worth.
Buying with gold: build a marketplace early. Buy wood only if it lets me put up producing buildings sooner, and sell surplus stone or iron if storage fills.
Popularity and taxes: add a chapel, then hops, a brewery and an inn. Once popularity is comfortably above 60, raise taxes to low or average for steady gold, and add hovels as idle peasants run out.
Later: if wood or iron piles up, build workshops (poleturner, blacksmith) and an armory to turn it into weapons worth more than the inputs. Expand stockpiles and granaries before they fill.
Checks: watch housing, idle peasants, food and popularity, and keep building right up to the end.

Commentary during play1

Reading the map first (free).

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Opus 5.5 (claude-opus-5-5) on Anthropic's API, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
Run 9c8871df, series 20261015T185216-6a2a0659, 2026-10-15: Opus's first game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its two 1.6.2 games

Result

	This game (1.7.1)	1.6.2, first game	1.6.2, second game	The person	Left alone
Net worth	15,424	13,921	16,193	41,631	2,195
Gold	2,186	1,157	6,261	39,977
Stored goods, at the sell price	13,238	12,764	9,932	1,654
Population / housing	164 / 202	147 / 162	144 / 162	506 / 506	4
Popularity	47 (−2 next month)	48	62	99	0
Score	34.7	31.4	37.4	100	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 13,229 over leaving the game alone, 34% of the person's growth; the score sits between its two 1.6.2 games (mean 34.4). The score is the project website's formula (80% net worth, 10% each population and popularity); this game counts on the website's leaderboard. In this round so far GPT-6.1 Sol scored 50.4 and Claude Sonnet 5.5 33.1.

Of the stored value, 9,936 is 432 iron from twenty-seven iron mines, 1,344 is 192 stone from four quarries and 1,530 is 43 crossbows and 16 bows. The town ended with 24 idle peasants, 38 empty places in its hovels and a full stockpile.

How the game went

Iron from the start, 0 to 7 minutes. A marketplace, three woodcutters, an orchard and a hovel in the first turn, 100 planks bought at 1:16 and 3:50, and four iron mines with orchards and hovels at 2:05. High taxes at 3:34; an armory at 4:31 and four fletcher's workshops at 5:01, set to crossbows.

Iron mines in fours, 7 to 20 minutes. Taxes went to average and then none by 7:30, with popularity down to 63 by 10 minutes. Iron mines in threes and fours at 7:43, 11:08, 13:54 and 17:41 (twenty-two by 19:07), a chapel, more hovels, orchards and two dairy farms; 185 more planks bought by 17:21. Gold fell to 242 by 17:42, and taxes moved between none and low every few minutes from then on, thirteen changes in all.

Quarries and more of everything, 20 to 35 minutes. Four quarries with five ox tethers from 20:12, the last five iron mines by 27:11 (twenty-seven), hovels in fours, five more fletchers at 29:38 and 30:15, orchards in fours and fives and woodcutters in fours. Food fell to 21 by 35 minutes and it bought 50 apples at 34:21. The population passed 100 at 25 minutes and 150 at 34.

A full stockpile, 35 to 45 minutes. At 35:41 expand_storage found no free ground beside the stockpile. It added five hovels and eight woodcutters and sold 460 planks between 37:02 and 43:09 to make room, but from 39:43 the stockpile had no room left for iron. Average taxes at 41:27, low at 43:02; popularity ended at 47. Net worth rose from 9,591 at 35 minutes to 15,424.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,969	1,491	1,806	2,720	5,350	7,590	9,591	12,494	15,424
Gold	2,000	1,600	821	494	242	786	1,008	820	936	2,186
Iron	0	0	14	33	77	155	226	300	384	432
Stone	20	20	20	20	20	20	44	100	140	192
Population / housing	6 / 10	23 / 34	37 / 50	59 / 66	79 / 98	101 / 106	129 / 154	150 / 162	164 / 202	164 / 202
Popularity	100	98	63	71	71	57	60	62	64	47

Each column is the last reading at or before that minute.

Beside its 1.6.2 games

The same plan as both 1.6.2 games: iron mines in numbers (27 here, 22 and 26 there), a weapons quarter, orchards for food, and taxes mostly off. This game kept more iron (432) and grew the largest town of the three (164), but its weapons quarter was smaller (ten fletchers, against thirteen fletchers and six poleturners, and sixteen poleturners), and it sold no iron, where the second 1.6.2 game sold 250 iron for gold.

What held it back

A full stockpile with no room to grow. From 35 minutes it could not add to the stockpile, and from 39:43 there was no room for iron while twenty-seven mines delivered about 16 a minute. It sold planks, worth 1, to make room, never iron or stone. Opus named this in its interview.
Popularity in the 50s and 60s. No inn or church, only one chapel, apples as the only food; taxes stayed at none or low for most of the game, so gold stayed under 1,300 from 5 minutes until the last few.
Low-value jobs. Twenty-nine woodcutters for planks worth 1, and ten fletchers that made 59 crossbows and bows in 40 minutes.
Planks bought dear, sold cheap. 385 planks bought at 4 each early and 460 sold at 1 each late.

What worked

The most iron of the round so far: twenty-seven mines and 432 iron kept, 9,936 of its net worth.
The second largest town of the round so far, 164 people.
Quick: 69 turns in 42.9 real minutes, 7.7 of them waiting for the model, and every request read the prompt cache.
Every placement reported as placed is in the game's final count of its buildings.

Market

385 planks and 50 apples bought for 1,940 gold, a loss of 1,355 net worth against the sell price (planks 4 for 1, apples 8 for 4); 460 planks sold for 460, which leaves net worth unchanged.

Cost and speed

69 turns, 124 tool calls (39 find_sites, 29 build_structure, 13 set_tax, 11 market_trade, 8 set_workshop, 8 waits), 3.96 million tokens and $1.979 at Anthropic's list prices (preparation $0.162, play $1.719, compaction $0.097), with 97.1% of input read from the cache. The interview cost about $0.039 more, with 45,677 of its 48,406 tokens read from the cache. The agent run took 42.9 real minutes; the context was compacted twice (29 seconds). From the start of one game to the next it took 45 minutes, against 57 for its first 1.6.2 game.

The interview

Opus named three differences: popularity, which it moved only with taxes and food (no inn, church or good and bad things), so it could not tax and growth stalled with 24 idle; not turning goods into gold, where the person sold bread and cut rations; and storage and scale, with a stockpile "full for the last several minutes" stalling iron, 29 woodcutters and ten slow fletchers. Playing again it would build inns with bought ale and a church first, then fear-factor buildings, spend popularity on cutting rations, build one bread chain to sell, keep building iron mines but leave open ground around the stockpile from the start and sell before it fills, and use cheap jobs such as water pots. It called the harness very good (tile targets, find_sites, the storage readout, retries) and suggested that the popularity readout list the factors that could be added, such as inns, fear and food variety, not only those in effect.

For the harness

No room beside the stockpile (35:41), the fourth game of this round to find none (after GLM 5.3 Flash's, Claude Sonnet 5.5's and GPT-6.1 Sol's). Here it cost iron: the stockpile had no room for it from 39:43.
One build_structure call at 7:36 was refused before acting: more placements than the reply had actions left (8 a reply); the message named the fix.
No provider errors or stalled replies. The game saved at the end ("claude opus 5-5 1-7-1 9c8871", in this folder).

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	15,424	41,631
Gold	2,186	39,977
Stored goods, at the sell price	13,238	1,654
Population	164	506
Housing	202	506
Popularity	47	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Claude Opus 5.5

The person finished with about 2.7 times my net worth (41,631 against 15,424), about three times my population (506 against 164) and double my popularity (99 against 47). My total came mostly from stored goods: roughly 430 iron, 180 stone and 54 weapons. Theirs was almost all gold. I think three things made the difference.

Popularity. I never built an inn, a church or any good or bad things. My only popularity levers were taxes and food, which left popularity hovering in the 50s. That meant I couldn't tax much. Late in the game, population also stalled at 164 with 24 idle, which blocks new peasants, and 40 houses stood empty.

No turning goods into gold. They took a cheap input (wheat), ran it through a scalable chain to bread and sold it, while cutting rations to none or half so the food wasn't eaten. I fed people full rations of apples and once had to buy 50 apples for 400 gold to stop starvation.

Storage and scale. My stockpile was full for the last several minutes, so iron production kept stalling. I also spent too much on low-value things: 29 woodcutters producing wood worth 1 a unit, and 10 fletchers that made about 0.2 items a minute each.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Claude Opus 5.5

Popularity first: inns stocked with bought ale, a church, then fear-factor buildings for worker efficiency once popularity had room.
Rations: spend the popularity surplus on cutting rations to none or half rather than on taxes.
One scalable chain to sell: wheat to mill to bakery to bread, built early and expanded hard, with output sold steadily so it never fills storage.
Iron, with storage planned: keep building iron mines, but leave open ground around the stockpile from the start so it can grow, and sell before it fills.
Fewer woodcutters: only enough for building. No fletchers.
Jobs to unblock growth: water pots and similar cheap jobs to keep idle peasants under 24, so population keeps growing into the housing.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Opus 5.5

What helped:

The harness is very good. Tile-target placement, find_sites, the storage readout in the stats, and the retry logic took away most of the pain of clicking in an isometric game.
Production per building and the full warning were especially useful.

What got in my way:

Mostly my own strategy. I was too passive in thinking about popularity mechanics.
Context compaction partway through cost some continuity.
One fixable point: the popularity readout listed only tax and religion. It didn't show what could be added, such as inn, fear and food-variety factors. A hint like that might have nudged me toward those levers sooner, though they are documented.

What I'd change: 45 minutes is a fair window.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Opus 5.5

The comparison is humbling and instructive. The winning approach treated popularity as a currency to spend and the marketplace as the main engine. I treated the game as "build producers and stockpile their output". Storing goods as net worth is valid, but storage caps it, and selling bread scales without that limit. Defining net worth so that gold and goods count equally made selling neutral, and I failed to exploit that.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	15,424	Counted

What ran

Benchmark: The Sinkhole construction
Benchmark version: 1.7.1
Fingerprint: 63921aad76e5
Harness commit: 674e7ba
Prompt / tools SHA-256: 3b0fa8a / f75ebb8
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261015T185216-6a2a0659