One episode · Oasis by the Sea

Claude Haiku 5.5

Benchmark 1.6.3AnthropicReasoning mediumPlayed by the maintainersFull-budget score

All of Claude Haiku 5.5’s results and comparisons

Not part of the evaluation: the game is published, but not ranked.

Score

21.8

Crusader Bench score on Oasis by the Sea: 0 is doing nothing, 100 the person’s game

Net worth

4,281

+3,061 over doing nothing, 17% of the person’s gain of 18,293

Population

26% of the person’s gain of 198 · popularity 58

Cost

~$0.05

2.6M tokens; ~ at list prices, the API reports no bill

Time

33 game min

31 min of wall-clock time, 47 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Claude Haiku 5.5
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 33 min 20 s of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

4,281net worth

Gold: 710
Stored goods, at sell price: 3,571
Over the baseline (do nothing): +3,061
Population / housing: 56 / 58
Popularity: 58
Food: 118
Structures (map-wide): 59
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock3,571 gold in goods

Good	Units	Price	Value
Gold	710	1	710
Iron	47	23	1,081
Stone	89	7	623
Wheat	62	8	496
Apples	118	4	472
Wood	349	1	349
Bows	14	15	210
Swords	7	30	210
Spears	13	10	130
Net worth			4,281

Beside a person’s gameOasis by the Sea, the same game time

	This episode	The person	Left alone
Net worth	4,281	19,513	1,220
Gold	710	18,304
Stored goods, at the sell price	3,571	1,209
Population	56	202
Housing	58	202
Popularity	58	100

The person played Oasis by the Sea for 33 min 18 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: October 1200
Idle peasants: 24
Popularity next month: +1
Taxes: +1
Taxes: No taxes, 0 gold a month
Rations: full; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 3 empty; 1 granary holding 118 food

Buildings38

Woodcutter7
Hovel6
Apple Orchard4
Stockpile4
Fletchers' Workshop2
Iron Mine2
Quarry2
Armory1
Bakery1
Blacksmith's Workshop1
Dairy Farm1
Granary1
Keep1
Marketplace1
Mill1
Ox Tether1
Poleturner's Workshop1
Wheat Farm1

Tool calls91

wait_and_observe34
build_structure18
find_sites13
set_tax7
place_near6
market_trade5
inspect_building3
expand_storage2
observe2
get_inventory1

Play

Turns: 47
Buildings placed / attempted: 27 / 29
Anchor-tool placements: 8
Tool errors: 0
Raw inputs sent to the game: 283
Context compactions: 1
Game time: 33 min

Compute

Tokens: 2.6M
Served from cache: 95%
Output tokens: 18K
Cost, at list prices: ~$0.05
Wall-clock time: 31 min
of which inference: 5 min
Peak game memory: 3,462 MiB

The plan, before play

Understanding of the task

This is an economy race on the Oasis by the Sea-1 save. I have 33 minutes of game time. The score is net worth at the end: gold plus stored goods at marketplace sell prices, with population and popularity counting for less. Recruiting soldiers and building walls, towers, gates or other defences are not allowed. Buildings count only through what they produce or house.

Opening plan

Place the required granary first (Granary, setup tray), next to the existing stockpile by the keep.
Wood: two or three woodcutters beside the western grove, with a stockpile nearby if the trip is long.
Housing: hovels as soon as wood allows, to raise population. Watch popularity and idle peasants.
Food: apple orchards and dairy farms on the oasis grass south-west of the keep. Their output goes to the granary directly.
Stone and iron: a quarry with an ox tether on the stone deposit to the west, and an iron mine on the strip north of the keep, once wood and workers allow.
Marketplace once affordable. Use it to sell surplus and, early on, to buy wood to speed construction if that pays back.
Keep taxes low (no taxes) until popularity is comfortably above 50, then raise them for steady gold. Check status and get_inventory after each build burst, and expand storage before piles fill.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Claude Haiku 5.5 (claude-haiku-5-5) on the Anthropic API, reasoning medium, output limit 32,768 tokens
Benchmark: Oasis by the Sea construction, version 1.6.3, from the save the human reference game started from; 33 min 20 s of game time (60,000 ticks) at speed 40, the length of the person's game; no playbook
Run d8be7ea7, series 20261013T230001-f3b9e432, 2026-10-13: Haiku's second game on this map, a new series with no memory of the first, played on the latest version (the first game, series 20261011T141021-5ef84787, played 1.5.0). The map's score for Haiku is the mean of the two.

Result

	This game	First game	The person	Left alone
Net worth	4,281	5,113	19,513	1,220
Gold	710	1,482	18,304
Stored goods, at the sell price	3,571	3,631	1,209
Population / housing	56 / 58	61 / 82	202 / 202
Popularity	58 (+1 next month)	67	100

A full-budget score: the run used all 2,000 game seconds. Net worth grew by 3,061 over leaving the game alone, 17% of the person's 18,293, against 3,893 in the first game. The stored goods are 47 iron (1,081), 89 stone (623), 62 wheat (496), 118 apples (472), 349 wood (349), 14 bows (210), 7 swords (210) and 13 spears (130).

How the game went

The opening, 0 to 9 minutes. Two woodcutters and two hovels at the start, the marketplace at 0:29, 50 wood bought at 0:50, an apple orchard and an iron mine at 1:13, a dairy farm at 2:23, two more hovels at 2:49, woodcutters, a quarry and an ox tether at 3:57, then a wheat farm at 6:26, a mill at 6:42 and a bakery at 8:16, with two more hovels at 7:14. Taxes went to high at 2:18 and back to low at 4:45, and popularity fell from 92 at 3 minutes to 68 at 6. It bought 30 bread at 5:17.

Weapons, 9 to 21 minutes. A second iron mine at 8:43, a fletcher's workshop at 10:20 and an armory at 11:05, a quarry and a poleturner's workshop at 13:19, apple orchards at 11:44 and 16:34, a blacksmith's workshop at 17:48 and a second fletcher at 21:09, its last workshop. It bought 15 bread at 12:39 and 30 apples at 16:53. The population reached 53 by 15 minutes and 56 by 24.

Waiting, 21 to 33 minutes. After 21:09 it placed only one more stockpile pile (24:27) and spent most replies on wait_and_observe (34 of its 91 tool calls), moving taxes between none and low. Low taxes from 24:23 took popularity under 50 at 28:10, and the population fell from 56 to 47 by 30 minutes; taxes back to none at 28:37 lifted popularity to 55, and the population returned to 56 by 32:09, with 24 idle and 2 places free. 62 wheat sat unmilled in the stockpile at the end.

Game minute	0	3	6	9	12	15	18	21	24	27	30	End
Net worth	1,345	1,191	1,284	1,526	1,583	1,809	1,729	2,116	2,143	2,569	2,928	4,281
Gold	945	817	769	935	824	781	387	497	397	562	710	710
Iron	0	0	3	6	9	15	23	28	32	36	41	47
Stone	25	25	25	33	41	49	57	65	65	73	81	89
Food	45	37	55	47	36	43	60	73	82	97	103	118
Population	3	27	38	43	47	53	54	55	56	56	47	56
Popularity	95	92	68	63	58	53	59	54	62	53	49	58

Up to 30 minutes the net worth row counts gold, wood, stone, iron and food (at 4) as the recording logs them; wheat and weapons are left out, so it reads low.

Beside its first game

	This game (1.6.3)	First game (1.5.0)
Net worth	4,281	5,113
Hovels / woodcutters / quarries / iron mines	6 / 7 / 2 / 2	9 / 9 / 5 / 2
Other	a mill and bakery, 2 fletchers, a poleturner, a blacksmith, an armory	a chapel, 3 ox tethers
Population / housing / popularity	56 / 58 / 58	61 / 82 / 67
Cost	$0.05	$0.08

The same small town both times, near 60 people with 24 idle and no taxes at the end. The first game put more into quarries, housing and a chapel; this one put its gold into weapon workshops, whose output was worth 550 at the end, and ended 832 lower.

What held it back

No growth after 15 minutes. Six hovels in all; the population was 53 at 15 minutes and 56 at the end, held by 24 idle peasants with no jobs for them.
Twelve minutes of waiting. No production building after 21:09; the last third of the game added stock but nothing that made more.
Popularity near 50. One kind of food at the end and no chapel; low taxes from 24:23 took popularity under 50 and cost 9 people for a few minutes.

Market

100 wood, 45 bread and 30 apples bought for 1,000 gold, and nothing sold: 600 net worth lost against the sell price (300 on wood at 4 gold, worth 1; 180 on bread and 120 on apples at 8, worth 4). These are units at the market's fixed prices.

Cost and speed

47 turns, 91 tool calls (34 waits), 2.58 million tokens and $0.048 at Anthropic's list prices (preparation $0.004, play $0.043, compaction $0.002). The interview cost about $0.002 more, with 72,843 of its 75,651 tokens read from the cache. The agent run took 30.9 real minutes, 4.6 of them waiting for the model; the context was compacted once (11 seconds). 46 of the 47 requests read the prompt cache (the last did not), 96.6% of the input tokens.

The interview

Haiku said "the gap is mostly population and production": it "never grew population", built a mill and bakery but "never checked that the bakery was staffed or delivering", let food and popularity reach a crisis twice, and "put gold into weapons, not into people". Playing again, it would add housing and jobs together from the start, raise popularity above 70 with a chapel, bad things and varied food, run and check the bread chain, buy wheat when short, and sell its stock. It found the reading tools the most useful part and asked for a per-building production readout and warnings when popularity is about to fall under 50 or food is falling.

For the harness

The run played on this checkout's version, 1.6.3 (commit 1d0c697), and records it; Oasis's round was already 1.6.3.
place_near found no free spot in view for a bakery at 7:26, a fletcher's workshop at 9:48 and an armory at 10:42; each was then placed by tile with build_structure. An apple orchard at 0:29 was rejected and a wheat farm at 5:28 found no room.
The game saved correctly at the end ("claude haiku 5-5 1-6-3 d8be7e").

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	33 min 20 s	33 min 18 s
Net worth	4,281	19,513
Gold	710	18,304
Stored goods, at the sell price	3,571	1,209
Population	56	202
Housing	58	202
Popularity	58	100

Leaving the game alone for the same time ends with a net worth of 1,220.

How do you see your result next to the person's? What do you think made the difference?

Claude Haiku 5.5

My net worth was 4,281 against the person's 19,513, so I finished at about a fifth of their total. The gap is mostly population and production. I ended with 56 people in 58 housing, and they had 202 people in 202 housing. My gold was 710 against their 18,304. My stored goods were worth more than theirs (3,571 against 1,209), but that was mostly a pile of unsold wood (349), apples (119), wheat (62), and iron and stone that I never turned into anything more valuable. Only about 1,000 of my 4,281 came from what the settlement produced in a way that compounds.

The main differences I can see:

I never grew population. Idle peasants sat at 22 to 24 for most of the run. I kept adding production buildings, but I did not add enough hovels or jobs to keep the population growing, and I kept my tax low, which kept popularity near 50 and at times below it.
I spent too little time on the core loop. The person's bread chain (wheat, mill, bakery, sold at the market) and their fear-factor and religion buildings pushed popularity to 100 and kept workers at 140% efficiency. I built a mill and a bakery early but never checked that the bakery was staffed or delivering, and I let bread run out.
I let food and popularity get close to crisis twice. I had to buy apples at the market to stop the granary falling, and popularity fell to 46 when I raised taxes. Each of these cost me population.
I put gold into weapons, not into people. The fletchers and armory were a reasonable value chain (bows 15 against wood 1), but they were small compared with the population base the person built.

2Playing again

Interviewer

How the person played, as they describe it:

Bread was one of their main sources of income: mills ground wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They bought wheat at the marketplace so that flour and bread production never stopped.
A church and a cathedral raised popularity.
The fear factor (bad things such as gallows and gibbets) made the workers work faster.

Their settlement at the end, read from the game:

202 people in 24 hovels, with no free housing; popularity 100.
57 bakeries, 7 mills and 10 wheat farms. The marketplace sold bread automatically.
13 iron mines and 10 quarries with 33 ox tethers, and 10 woodcutters.
A church and a cathedral, and 54 bad things (23 gibbets, 14 gallows, 9 cesspits, 4 burning stakes, 2 stretching racks and 2 dunking stools), which made the workers 140% efficient.
No inns and no good things. No taxes, and full rations of bread. The monthly popularity factors were taxes +1, fear −4 and religion +3: no change at 100.

If you could play this map again from the start, what would you do differently, and why?

Claude Haiku 5.5

I would prioritise, in this order:

Housing and jobs together, early. Add hovels steadily from the first minute, and staff each one with a job so the 24-idle buffer never builds up. Population is the base for everything else.
Get popularity above 70 early. A chapel or church, plus a few bad things (gallows, gibbet) for work efficiency, and variety of food (apples, bread, cheese). Then I can tax at a higher level without losing peasants.
Run the bread chain properly. Wheat farms on oasis, a mill and bakery next to the granary, and check each with inspect_building once built. Sell surplus bread at the market.
Buy wheat at the market when the chain is short, as the person did, to keep flour and bread running.
Sell and turn stock into gold regularly. Stockpiled wood and apples earned nothing toward the score beyond their unit value, so I should have converted them or built with them.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Haiku 5.5

The reading tools were the most useful part. get_inventory, status and the stats in each observation let me see food, storage and idle peasants quickly. find_sites and build_structure with tile targets were reliable, and inspect_building helped me confirm the mill was staffed.

What got in my way was the cost of checking things. Each screenshot and check cost turns, and I often could not see whether a building was producing until several minutes later. Stats such as gold and wood_planks sometimes lagged a purchase by a turn, and I had to check twice. Placement near the keep and stockpile often failed with "no free spot in view", which forced me to use find_sites and then place by tile. The 8-action limit per reply also meant I had to choose between building and checking.

I would like a per-building production readout (is it working, what is it waiting for) in the stats, and a clearer warning when popularity is about to drop below 50 or when food is falling. Those two alerts would have saved me the most time.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Haiku 5.5

The run taught me that in this game the economy compounds through people, not through buildings. I built many production sites and kept them small, when I should have built dense housing and kept it staffed and fed. The person's result shows what happens when population and popularity are pushed high early, and I did not do that.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	4,281	Counted

What ran

Benchmark version: 1.6.3
Fingerprint: 96c4682877ca
Harness commit: 1d0c697
Prompt / tools SHA-256: 3737ac8 / e41a140
Menu guide: Images 118089d
Save: Oasis by the Sea benchmark start
Game time per episode: 33 min 20 s
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261013T230001-f3b9e432