One episode · Oasis by the Sea

Claude Sonnet 5.5

Benchmark 1.7.3AnthropicReasoning mediumPlayed by the maintainersFull-budget score
Model
claude-sonnet-5-5
Served via
Anthropic API
Max output
32,768 tokens
Played
16 October 2026
Score
36.5
Crusader Bench score on Oasis by the Sea: 0 is doing nothing, 100 the person’s game
Net worth
7,832
+6,612 over doing nothing, 36% of the person’s gain of 18,293
Population
55
26% of the person’s gain of 198 · popularity 50
Cost
~$0.41
1.5M tokens; ~ at list prices, the API reports no bill
Time
33 game min
23 min of wall-clock time, 26 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

  • Claude Sonnet 5.5
  • Baseline (do nothing), final
  • A person's game, final

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Videos and images stream from the dataset. Game footage © Firefly Studios, shared for non-commercial research.

Episode 1 · 33 min 20 s of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

7,832net worth
Gold
1,053
Stored goods, at sell price
6,779
Over the baseline (do nothing)
+6,612
Population / housing
55 / 90
Popularity
50
Food
183
Structures (map-wide)
74
Attempt
1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock6,779 gold in goods

GoodUnitsPriceValue
Gold1,05311,053
Iron224235,152
Apples1834732
Wood4801480
Stone257175
Bows815120
Crossbows430120
Net worth7,832

Beside a person’s gameOasis by the Sea, the same game time

This episodeThe personLeft alone
Net worth7,83219,5131,220
Gold1,05318,304
Stored goods, at the sell price6,7791,209
Population55202
Housing90202
Popularity50100

The person played Oasis by the Sea for 33 min 18 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game
October 1200
Idle peasants
16
Popularity next month
+1
Taxes
+1
Taxes
No taxes, 0 gold a month
Rations
full; 1 of 1 foods eaten
Worker efficiency
100%
Storage
24 stockpile piles, 8 empty; 1 granary holding 183 food

Buildings49

  • Hovel10
  • Iron Mine10
  • Woodcutter10
  • Apple Orchard7
  • Stockpile6
  • Fletchers' Workshop2
  • Armory1
  • Granary1
  • Keep1
  • Marketplace1

Tool calls59

  • find_sites14
  • wait_and_observe11
  • build_structure10
  • set_tax7
  • observe4
  • place_near4
  • building_info3
  • expand_storage2
  • market_trade2
  • inspect_building1
  • set_workshop1

Play

Turns
26
Buildings placed / attempted
37 / 40
Anchor-tool placements
9
Tool errors
1
Raw inputs sent to the game
279
Context compactions
0
Game time
33 min

Compute

Tokens
1.5M
Served from cache
95%
Output tokens
7.7K
Cost, at list prices
~$0.41
Wall-clock time
23 min
of which inference
3 min
Peak game memory
3,607 MiB

The plan, before play

Understanding: I have to grow net worth (gold plus stored goods), then population and popularity, on the Oasis by the Sea map in about 33 game minutes. I can't build defences or recruit.

Opening plan:

  1. Place the granary near the keep.
  2. Build 2–3 woodcutters by the trees.
  3. Build hovels for housing.
  4. Build apple orchards and a wheat farm on the oasis grass for food.
  5. Build a marketplace, then iron mines on the strip north of the keep. Iron sells for 23 each.
  6. Add quarries and ox tethers.
  7. Raise taxes once popularity allows.
  8. Keep expanding and keep an eye on storage.

Commentary during play3

  1. Add two more mines, then wait.

  2. Final stretch: set taxes for gold at the very end (popularity buffer 84).

  3. Popularity is dropping fast; set tax back to avoid the penalty.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (drafted by Sonnet 5.5, checked by Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score. The drafter is the same model as the player.

  • Model: Claude Sonnet 5.5 (claude-sonnet-5-5) on the Anthropic API, reasoning medium, output limit 32,768 tokens
  • Benchmark: Oasis by the Sea construction, version 1.7.3 (commit b967c65), from the save the human reference game started from; 33 min 20 s of game time (60,000 ticks) at speed 40, the length of the person's game; no playbook. Weapons and armour in the armory count toward net worth at the rules' prices, as since 1.6.0.
  • Run 1fefeb89, series 20261016T192138-297b5b78, 2026-10-16: Sonnet's weapons-counted game of the 1.7.3 batch, a fresh series with no memory of any other game. Its no-arms game (run 1d0a2c40, series 20261016T194728-2f79fee8) was played right after.

Result

This game (1.7.3) Its 1.7.3 no-arms game Its 1.6.3 game Its 1.5.0 game The person Left alone
Net worth 7,832 6,567 10,107 8,633 19,513 1,220
Gold 1,053 1,165 1,324 1,147 18,304
Stored goods, at the sell price 6,779 5,402 8,783 7,486 1,209
Population / housing 55 / 90 43 / 82 26 / 98 60 / 74 202 / 202
Popularity 50 (+1 next month) 60 (+3) 49 (+1) 84 (+2) 100
Idle peasants 16 15 0 24
Score 36.5 31.4 44.9 43.6 100

A full-budget score: the run used all 2,000 game seconds. Net worth grew by 6,612 over leaving the game alone, 36% of the person's 18,293. With 55 people and popularity 50 the score is 36.5 by the project website's formula (80% net worth, 10% each population and popularity, the net-worth share clipped to −1..2); this game counts on the website's leaderboard. It is 5.1 above its no-arms game and 8.4 and 7.1 below its 1.6.3 and 1.5.0 games.

The stored goods are 224 iron (5,152), 183 apples (732), 480 wood (480), 25 stone (175), 8 bows (120) and 4 crossbows (120). Iron is 66% of net worth. The town ended with 35 empty places in its hovels and one food eaten.

How the game went

Times are the game clock (game time used) when a call began. The game is paused while the model replies, so a reply costs no game time; a build or buy call keeps it running for 20 to 117 seconds while it clicks.

The opening, 0 to 7 minutes. The first call, at 0:07, tried to place a granary and was rejected ("Needs to be placed adjacent to the granary": the save has one). Three woodcutters followed at 0:42 and two hovels at 1:22. At 2:11 one call placed an iron mine and the marketplace; both apple orchards in it failed for want of wood, and the call took 86 s. At 4:09 the 26 people filled the 26 places, 21 of them idle with no open job. Planks were bought at 4:11, and at 4:58 two orchards, two iron mines and two woodcutters went up in a call that ran to 6:55. At 6:57 the granary held 8 food for 26 people ("Food is low"); two stockpiles came at 7:00.

Mines and housing, 7 to 15 minutes. Four orchards at 7:41, two hovels at 8:34, high taxes at 9:00, four iron mines and two hovels at 9:19. A warning at 10:13 said popularity had fallen from 98 to 84; taxes went to low at 10:15 and none at 12:42. At 11:34 it placed the seventh orchard and eighth mine, one site each. The 12:41 reading was the first with 24 idle. It placed five woodcutters at 12:58 and bought 100 planks at 13:46.

Weapons and the last mines, 15 to 19 minutes. One call from 14:53 to 16:20 placed the armory, two fletcher's workshops and four hovels; gold fell from 626 to 426. At 18:10 its reasoning called bows too slow next to mining and ore sites exhausted; it set average taxes, and a search around tile (435, 335) at 18:21 found two ore sites. Its two mines at 18:30 made ten, the last production buildings of the game.

Waiting, 19 to 33 minutes. It built nothing more but three stockpiles (23:50); the last 14 min 21 s went on six waits of 110 to 280 s and tax changes. The host noted the waiting at 23:48 and 28:27. At 28:29 it set high taxes, writing "set taxes for gold at the very end (popularity buffer 84)". Warnings ended the next two waits (30:02, popularity 84 to 66; 31:22, 66 to 48); it waited again after the first and set taxes to none at 31:24. Popularity was under 50 from about 31:13, people left, and the town ended at 55.

Game minute 0 3 6 9 12 15 18 21 24 27 30 End
Net worth 1,408 1,335 1,141 1,210 1,785 2,197 2,706 3,597 4,487 5,347 6,389 7,832
Gold 988 1,000 780 780 966 626 426 644 644 644 796 1,053
Iron 0 0 2 7 18 42 68 92 125 156 189 224
Stone 25 25 25 25 25 25 25 25 25 25 25 25
Wood 45 4 96 54 22 142 177 214 281 360 415 480
Food 50 39 11 10 52 72 91 112 128 145 164 183
Population 4 20 26 28 49 57 59 63 63 63 63 55
Popularity 95 100 100 100 75 77 83 68 74 81 67 50

Each column is the last reading at or before that minute. Up to 30 minutes the net worth row counts gold, wood, stone, iron and food (at 4) as the recording logs them; weapons are left out, so it reads low.

Weapons

It built an armory and two fletcher's workshops, no other workshop and no armour. At 11:34 its reasoning weighed fletchers (bows 15) against poleturners and blacksmiths (swords 30) and chose fletchers on margin. A place_near for the armory found no free spot beside the stockpile at 12:12 and one for the fletchers was 11 wood short at 12:27; the call at 14:53 placed them. No weapon was in the armory at 17:34; at 17:36 inspect_building showed a fletcher working on bows, and at 17:52 one was set to crossbows. Three bows were in store at 20:07, and from 23:48 the readout showed a bow and a crossbow every 3 minutes, 0.3 items a workshop a minute. The end value is 8 bows and 4 crossbows, 240 at the rules' prices: 3.1% of net worth and about 1.0 of the 36.5 points (35.4 with weapons worth nothing).

The cost, from prompt/game-mechanics.md: 200 gold and 40 wood for the workshops, 5 wood for the armory and 28 in the items (bows 2 wood, crossbows 3): 200 gold and 73 wood. The town ended with 480 unspent wood, so wood counts at its sell price, 1: the weapons returned 240 for 273, about 33 less than keeping both (252 less with wood at the 4 gold a plank cost it). A workshop made about 7 gold a minute; an iron mine, 24 (1.0 to 1.1 iron at 23). The two jobs let two more peasants arrive (below), 0.1 point.

The same 200 gold and 45 wood could have stayed as gold, which counts 1:1 (gold sat at 644 for 8 minutes); paid for a chapel (250 gold), which its no-arms game built (religion +2, popularity 60); or gone to buildings that cost only wood and that it never built: wheat farm, mill and bakery (15, 20 and 10 wood), or a quarry (20) with tethers (5). Its iron-site searches found 3 sites at 2:11, 4 at 9:13, 1 at 11:34 and 2 at 18:21, and none came after that.

Beside its other games and the other models

This game (1.7.3) No-arms game (1.7.3) 1.6.3 game 1.5.0 game
Iron mines 10 10 12 10
Last production building placed 18:30 17:10 8:53 12:10
Food 7 orchards 4 orchards, dairy 6 orchards, wheat, mill, bakery 4 orchards, dairy, 2 wheat, mill, 2 bakeries
Weapons / chapel armory, 2 fletchers / none none / chapel none / none none / none
First planks bought 4:11 3:57 1:10 0:36
Planks bought 155 115 250 290

Against the 1.6.3 game this one ended 2,275 lower: iron −1,748 (224 against 300), flour −390, gold −271, apples −268, wood +162, weapons +240. The iron gap follows the mines: twelve by 8:53 there; here three at the 4:58 call, seven at 9:19 and ten at 18:30. Its marketplace went up at 2:11 (0:25 in the 1.5.0 game). Against its no-arms game it ended 1,265 higher: apples +396, wood +373, iron +368, weapons +240, gold −112. So weapons were 240 of that gap, and they worked against the 1.6.3 fall.

In the 1.7.3 batch so far, 36.5 is the best of four finished games (Haiku 23.0 with weapons, 26.9 without; Sonnet 31.4 without). Weapons mattered unevenly: 240 of Sonnet's net worth, 810 (19%) of Haiku's 4,163, whose weapons game scored below its no-arms game. Among the 1.6.3 round's second games on this map (Gemini 68.0, Sol 54.6, Fable 45.2, Sonnet 44.9, Opus 28.9 down to GLM 15.3), 36.5 would stand fifth; the versions differ. The first round's best was Opus at 52.0.

What held it back

  • No new production from 18:30. In the 14 min 21 s after that call ended it added nothing that makes goods, with 461 to 1,053 gold and 141 to 480 wood in hand. Population is employed plus 24 here: 59 with 24 idle at 16:23 (35 employed, the workers its buildings need), 63 with 24 at 20:07 (39, after two mines), and the same at 31:22. The rules say new peasants stop arriving at 24 idle; its reasoning at 17:34 read it as "new ones just join the idle pool".
  • The last tax raise. From 28:29 to 31:24 taxes at 62 gold a month took gold from 644 to 1,053 (+409, 1.8 points) and cost popularity 84 to 50 (3.4 points against holding at 84) and 8 people (0.4): about −2 net, 38.5 against 36.5. It had raised taxes three times; the first two dips healed at +1 a month, the last had no time.
  • A slow start. 21 idle at 4:09 with 1,000 gold and 40 wood, three woodcutters until 4:58. Its 155 planks (620 gold) lost 465 of net worth, 2.0 points. It meant to buy 40 at 13:46: its reasoning says "buy 40 wood for 160 gold", the call asked 20 lots of 5. It ended with 480 wood.
  • One food, no bread, no stone. Its plan listed a wheat farm; none of its 23 reasoning blocks mentions wheat, flour, a mill, a bakery, a church or religion. Stone stayed at 25; it never asked find_sites for a quarry (at 9:00 it weighed "a distant stone quarry" and chose mining).
  • A popularity warning not acted on. Taxes were cut within 2 s of the warnings at 10:13, 20:07 and 31:22; after 30:02 it waited again and lost 18 more points.

What worked

  • Ten iron mines. 224 iron (5,152) was 66% of net worth, at 0.9 to 1.1 iron a mine a minute in every readout from 16:23.
  • Food and storage held. Seven orchards fed up to 63 people on full rations (food was 8 once, at 6:57); stockpiles were expanded at 7:00 and at 23:50 with 2 of 12 piles empty.
  • Every placement reported as placed is in the final count. Five calls had one that did not go through (the granary, two orchards at 2:11, a hovel at 8:34, the armory at 12:12, the fletchers at 12:27). An iron mine, a fletcher and a hovel were placed 5, 1 and 4 tiles from their targets, and the result said so. No call was refused; the only tool error was the last wait, at the deadline.

Market

155 planks (at 4, worth 1) bought for 620 gold and nothing sold: 465 net worth lost against the sell price. The first buy, at 4:11, got 55 of 80 asked (the tool stopped for storage room); the 100 at 13:46 came just before the call that used 69 wood, 45 of it on weapon buildings. The two took 105 game seconds.

Cost and speed

26 turns, 59 tool calls (14 find_sites, 11 waits, 10 build_structure, 7 set_tax, 17 others), 1.51 million tokens and $0.409 at the profile's prices ($2 per million input, $10 output, $0.10 cache read, $2.50 cache write): preparation $0.078, play $0.330, no compaction. The run took 23.3 real minutes, 2.7 of them waiting for the model. All 26 requests read the prompt cache (96.9% of input tokens). Waits took about 960 of the 2,000 game seconds, build_structure 570. The interview cost about $0.022, with 75,628 of its 77,918 tokens read from the cache.

The interview

Sonnet put its result at about 40% of the person's, its growth "linear" against their compounding, and named four causes: no bread chain, population stalled at 55 to 63 with 24 idle, no stable tax, and wasted time. Playing again it would build bread first, a church and fear buildings, housing and jobs and quarries, and skip the fletchers ("close to worthless"). It found the tools good, said some replies cost "1–2 minutes" of game time it had not meant, and said open_jobs: 0 with 24 idle confused it.

Against the game, its figures match (7,832 and 19,513, 1,220, 55 and 202, popularity 50 and 100, no bread chain, stone skipped, fletchers 0.3 items a minute at 100 gold). Not matching: it said its goods were mostly wood and iron, but wood is 7% of them and iron 76%; the 40% is of net worth, the score is 36.5; the minutes were build calls (86, 117 and 87 s), not replies, which use none; open_jobs: 0 was right, since employed matched the workers its buildings need at 12:41, 16:23 and 20:07, so there was no open job to fill, and housing (90) was already ahead; and "close to worthless" is about break-even here. The last result of each of the 26 turns carried the game clock, but no call reports its own duration.

For the harness

  • The granary line. The rules' Start section says to place the required granary; the save has one (docs/benchmark.md), so the first call of this game and of the no-arms game was rejected; here it cost 20 game seconds.
  • Lots against units. market_trade takes lots of 5, as its text says; a 40-wood intent became 100 planks. The slip is the model's, but a units argument would remove it.
  • Waiting warnings and notes worked as built: the model acted on three of four popularity warnings; after the host's two notes its next calls were a stockpile (23:50) and a tax change (28:29), not new production.
  • The reader's memory watchdog forced one collection at about 22:18 (a 20 s stall); the game ran on. It saved correctly at the end ("claude sonnet 5-5 1-7-3 1fefeb").

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

You The person
Game time played 33 min 20 s 33 min 18 s
Net worth 7,832 19,513
Gold 1,053 18,304
Stored goods, at the sell price 6,779 1,209
Population 55 202
Housing 90 202
Popularity 50 100

Leaving the game alone for the same time ends with a net worth of 1,220.

How do you see your result next to the person's? What do you think made the difference?

Claude Sonnet 5.5

My net worth was 7,832 against the person's 19,513, so I finished at about 40% of their score. Doing nothing would have given 1,220, so I did grow the economy. My growth was linear and theirs compounded. Our final stocks show the difference. They held 18.3k gold and almost no goods. I held 1k gold and 6.8k in goods, mostly wood and iron at 1 and 23 per unit. Their population was 202 against my 55, and their popularity was 100 against my 50.

I think four things made the difference:

  • Gold income. They turned bread into gold at scale. I built no bread chain at all.
  • Population. I stalled at 55–63 people. Idle peasants sat at 24 for most of the run, which is the arrival cap, so I was at the limit of what my jobs supported. I never built enough job-heavy buildings to pull more people in.
  • Popularity. I never found a stable way to tax. I swung between tax levels and ended at the 50 mark. They reached 100 with a church, a cathedral and fear buildings.
  • Wasted time. Some of my replies cost real game time. Several early replies burned 1–2 minutes each, and at the end I waited in long blocks with nothing to do.

2Playing again

Interviewer

How the person played, as they describe it:

  • Bread was one of their main sources of income: mills ground wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
  • They bought wheat at the marketplace so that flour and bread production never stopped.
  • A church and a cathedral raised popularity.
  • The fear factor (bad things such as gallows and gibbets) made the workers work faster.

Their settlement at the end, read from the game:

  • 202 people in 24 hovels, with no free housing; popularity 100.
  • 57 bakeries, 7 mills and 10 wheat farms. The marketplace sold bread automatically.
  • 13 iron mines and 10 quarries with 33 ox tethers, and 10 woodcutters.
  • A church and a cathedral, and 54 bad things (23 gibbets, 14 gallows, 9 cesspits, 4 burning stakes, 2 stretching racks and 2 dunking stools), which made the workers 140% efficient.
  • No inns and no good things. No taxes, and full rations of bread. The monthly popularity factors were taxes +1, fear −4 and religion +3: no change at 100.

If you could play this map again from the start, what would you do differently, and why?

Claude Sonnet 5.5

I would change these things:

  1. Build the bread chain first. Build wheat farms, mills and bakeries early, put a marketplace on automatic sale, and scale up from there. The person's result shows bread is the best compounding income on this map.
  2. Make gold the early goal. Net worth was mostly gold in the winning run, so I would expand income before stockpiling raw goods.
  3. Add a church or cathedral, and use fear buildings early. That would let me keep popularity high without giving up taxes or work speed.
  4. Raise worker efficiency. I would test the person's fear-factor buildings, which gave 140% efficiency.
  5. Build housing ahead of need and fill every open job. Idle peasants at 24 meant arrivals had stopped.
  6. Add quarries and ox tethers. I skipped stone entirely, and at 7 per unit it is a good raw good.
  7. Skip the fletchers. They produced about 0.3 items per minute each and cost 100 gold apiece. That was close to worthless.
  8. Stop ending a reply on a long wait without acting.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Claude Sonnet 5.5

The tools were good. Tile-based placement, find_sites and the text-only observations let me act quickly. The stats were easy to read and the warnings were useful.

Three things got in my way:

  • The reply timing was opaque to me. Some replies advanced 1–2 minutes of game time without my intending it. I wish the tool results had shown the game time elapsed.
  • The open_jobs: 0 reading with 24 idle peasants confused me. I could not tell whether jobs were truly full or whether the peasants were not taking them.
  • The 33-minute limit is short for a compounding economy, so early decisions matter a lot.

The scoring is clear and fair. Comparing against a human run is useful because it shows how far off I was.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Claude Sonnet 5.5

I was too passive in the second half. I had spare wood, iron and idle workers but did not find a way to turn them into growth, and I should have researched the bread chain right away. The run was a useful lesson in compounding versus linear growth.

Record

Attempts and Reproducibility

Every attempt1

EpisodeAttemptOutcomeNet worthCounted
11Full-budget score7,832Counted

What ran

Benchmark
Oasis by the Sea construction
Benchmark version
1.7.3
Fingerprint
903c3c00f84a
Harness commit
b967c65
Prompt / tools SHA-256
5bfa08c / f75ebb8
Menu guide
Images 118089d
Save
Oasis by the Sea benchmark start
Game time per episode
33 min 20 s
Playbook between episodes
No
Minimum turn / default wait
8 s / 5 s
Observation after acting
Text unless a screenshot is needed
Screenshot width
1,280 px
Context budget
120,000 tokens
Real-time limit
360 min
Submitted by
hesamation
All files in the dataset 20261016T192138-297b5b78