Crusader Bench

One episode · Oasis by the Sea

GPT-6 Luna

Benchmark 1.5.0OpenAIReasoning mediumPlayed by the maintainersFull-budget score

All of GPT-6 Luna’s results and comparisons

Model
openai/gpt-6-luna
Served via
OpenRouter · openai, azure
Max output
32,768 tokens
Played
11 October 2026
Score
39.0
Crusader Bench score on Oasis by the Sea: 0 is doing nothing, 100 the person’s game
Net worth
7,348
+6,128 over doing nothing, 33% of the person’s gain of 18,293
Population
86
41% of the person’s gain of 198 · popularity 81
Cost
$0.11
5.9M tokens
Time
33 game min
53 min of wall-clock time, 127 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

  • GPT-6 Luna
  • Baseline (do nothing), final
  • A person's game, final

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

Commentary

12 Mines and 24 Idle Peasants

Quote

“My stock of raw materials was valuable, but it wasn't being turned into enough income, and my population and popularity both fell short.”

Model
GPT-6 Luna
Source
Its interview, after the game

GPT-6 Luna was second of the first five games on version 1.5, at 7,348. It built more iron mines than any of the others and kept popularity near 100 for most of the game, and it still ended with 24 peasants idle and 68 places empty.

In the first minute and a half it placed hovels, five woodcutters, an orchard, a dairy farm and its first iron mine, then added mines one or two at a time until there were twelve by the fifteenth minute, close to the person's thirteen, and six quarries with ox tethers later. Housing reached 154 by the twelfth minute, but the population grew only as fast as new buildings gave idle peasants work.

The game is the clearest sign of the limit each of the first five models hit on this save: when three mines and three quarries took workers, the population rose by 30 in about ten minutes, then stopped again at 86 with 24 idle. In its interview Luna said it should have built more housing.

It ended with 153 iron, worth 3,519, and 2,583 gold. It used the reading tools more than the others, 59 site searches among them, took 127 turns over 53 real minutes, and cost $0.11 as billed by OpenRouter.

The episode

Watch the Episode

Videos and images stream from the dataset. Game footage © Firefly Studios, shared for non-commercial research.

Episode 1 · 33 min 20 s of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

7,348net worth
Gold
2,583
Stored goods, at sell price
4,765
Over the baseline (do nothing)
+6,128
Population / housing
86 / 154
Popularity
81
Food
66
Structures (map-wide)
88
Attempt
1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock4,765 gold in goods

GoodUnitsPriceValue
Gold2,58312,583
Iron153233,519
Stone1127784
Apples654260
Wood1981198
Cheese144
Net worth7,348

Beside a person’s gameOasis by the Sea, the same game time

This episodeThe personLeft alone
Net worth7,34819,5131,220
Gold2,58318,304
Stored goods, at the sell price4,7651,209
Population86202
Housing154202
Popularity81100

The person played Oasis by the Sea for 33 min 18 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game
October 1200
Idle peasants
24
Popularity next month
+1
Food
+1
Taxes
−2
Religion
+2
Taxes
Low taxes, 51 gold a month
Rations
full; 2 of 2 foods eaten
Worker efficiency
100%
Storage
16 stockpile piles, 4 empty; 1 granary holding 66 food

Buildings63

  • Hovel18
  • Iron Mine12
  • Woodcutter7
  • Ox Tether6
  • Quarry6
  • Apple Orchard4
  • Stockpile4
  • Dairy Farm2
  • Chapel1
  • Granary1
  • Keep1
  • Marketplace1

Tool calls184

  • find_sites59
  • build_structure24
  • building_info21
  • go_to_tile16
  • wait_and_observe13
  • status10
  • market_trade7
  • set_tax7
  • get_inventory5
  • inspect_building5
  • observe4
  • center_on3
  • update_plan3
  • map_overview2
  • place_near2
  • set_rations2
  • expand_storage1

Play

Turns
127
Buildings placed / attempted
56 / 57
Anchor-tool placements
4
Tool errors
1
Raw inputs sent to the game
382
Context compactions
3
Game time
33 min

Compute

Tokens
5.9M
Served from cache
97%
Output tokens
68K
Cost, as billed
$0.11
Wall-clock time
53 min
of which inference
25 min
Peak game memory
3,220 MiB

The plan, before play

I’ll grow the settlement’s net worth, population, and popularity without building defenses or recruiting soldiers. I’ll place the required granary first, then prioritize wood production, housing, and food on suitable oasis ground. As the economy develops, I’ll expand production and storage, keep an eye on workers, resources, rations, taxes, and popularity, and use trade or services where they support further growth.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

  • Model: GPT-6 Luna (openai/gpt-6-luna) on OpenRouter, providers openai then azure, reasoning medium, output limit 32,768 tokens
  • Benchmark: Oasis by the Sea construction, version 1.5.0, from the save the human reference game started from; 33 min 20 s of game time (60,000 ticks) at speed 40, the length of the person's game; no playbook
  • Run 8057df86, series 20261011T144802-e1ba2274, 2026-10-11

Result

GPT-6 Luna The person Left alone
Net worth 7,348 19,513 1,220
Gold 2,583 18,304
Stored goods, at the sell price 4,765 1,209
Population / housing 86 / 154 202 / 202
Popularity 81 100

A full-budget score: the run used all 2,000 game seconds. Net worth grew by 6,128 over leaving the game alone, about a third of the person's growth (18,293). Two thirds of it was stored goods: 153 iron, 112 stone, 198 wood and 65 apples.

How the game went

Opening, 0 to 6 minutes. In the first minute and a half: hovels, five woodcutters, an apple orchard, a dairy farm and the first iron mine. Then a chapel at 4 minutes, three more orchards and another dairy farm, and hovels until housing reached 106 at 6 minutes. It bought wood in small lots five times (100 in all) and taxed at level 4 or 5, keeping popularity at 95 to 100.

Iron, 5 to 16 minutes. Iron mines one or two at a time until there were nine at 11 minutes, and three more at 15 minutes for twelve, close to the person's thirteen. Housing reached 154 by 12 minutes. The population grew slowly, from 44 to 56, with 24 peasants idle the whole time.

Stone and growth, 16 to 25 minutes. Three quarries with ox tethers at 18 minutes and three more at 23 to 24 minutes. It bought 100 cheese for 756 gold at 21 minutes. With the new jobs the population rose from 57 to 80.

The end, 25 minutes on. Long waits of two to four minutes. At 30 minutes it put rations to half to make the food last, and back to full at 33; popularity fell to 81. Net worth rose from 4,830 at 27 minutes to 7,348 as iron, stone and wood piled up.

Game minute 0 6 9 12 15 18 21 24 27 33 End
Net worth 1,280 1,006 1,326 1,638 1,964 3,059 3,729 4,099 4,830 6,388 7,348
Population 3 40 48 53 55 62 74 76 86 86 86
Housing 10 106 130 154 154 154 154 154 154 154 154

What held it back

  • Idle peasants capped the population. It had 24 idle peasants from 4 to 15 minutes, and the population crept up only as fast as new buildings took workers. When three iron mines (15 minutes) and three quarries (18 minutes) gave the idle peasants work, the idle count fell to 13 to 18 and the population grew by 30 in about ten minutes, then stopped again at 86 with 24 idle. The GLM 5.3 Flash, Claude Haiku 5.5 and Claude Sonnet 5.5 runs on the same save the same day stalled at 24 idle too. This is the clearest sign in the four runs that the game stops sending peasants while 24 wait at the campfire, so more people need more jobs, not more houses. Luna did not draw that conclusion: in its interview it says it should have built more housing, though 68 places stood empty.
  • No bread chain. Food was apples and some cheese. It looked for more farmland at 18 minutes and found none it could use; the person bought wheat and baked it.
  • Goods stayed in store. Iron is worth 23 each whether sold or kept, so this cost nothing in net worth, but its gold stayed low and it never had money to build with on a large scale.
  • Buying. 100 wood (−300) and 100 cheese (−356) lost 656 net worth.

What worked

  • Iron first: twelve iron mines by 15 minutes, the most of the four models, and 153 iron at the end, worth 3,519.
  • Popularity stayed between 96 and 100 from 3 minutes until the half rations at 30 minutes.
  • Only one placement was refused in 24 building calls.

Cost and speed

127 turns, 184 tool calls, 5.92 million tokens and $0.114 (billed by OpenRouter). The agent run took 52.8 real minutes, 24.6 of them waiting for the model, the slowest of the four; three compactions took 72 s in all. It used the read-only tools more than the others: 59 site searches, 21 building lookups, 10 status reads.

At 25 game minutes, the length of the earlier benchmark versions, it stood at 4,246 net worth for $0.104 after 45.1 real minutes. No earlier version has a Luna run to compare with.

Beside the other runs on the same save

GPT-6 Luna Claude Sonnet 5.5 GLM 5.3 Flash Claude Haiku 5.5
Net worth 7,348 8,633 6,200 5,113
Ending Full budget Full budget Ended itself 5 s early Full budget
Gold / goods 2,583 / 4,765 1,147 / 7,486 4,755 / 1,445 1,482 / 3,631
Population / housing 86 / 154 60 / 74 64 / 210 61 / 82
Iron mines / quarries / woodcutters 12 / 6 / 7 10 / 0 / 4 7 / 2 / 11 2 / 5 / 9
Net worth lost to purchases 656 930 1,201 150
Cost $0.114 $0.724 $0.145 $0.080
Real time 52.8 min 30.5 min 43.8 min 33.3 min

The interview

Luna said it played "relatively passive production" and "cautiously with taxes and food" where the person built a bread chain. Playing again, it would build farms, mills and bakeries early, a church and cathedral, and use bad things for worker efficiency if they pay. It saw that its population stayed at 86 with idle peasants and empty housing, but put it down to too little housing.

For the harness

  • In the interview it said one wait returned "No screenshot taken" instead of letting time pass; the wait it means asked for no screenshot, and the game did run. Not a fault, but the wording misled it.
  • At 18 minutes it wondered whether iron ore was hidden from site search "due to camera angles and HUD visibility". Site search reads the game's tile data, so nothing behind buildings is hidden, but it covers only the current view.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

You The person
Game time played 33 min 20 s 33 min 18 s
Net worth 7,348 19,513
Gold 2,583 18,304
Stored goods, at the sell price 4,765 1,209
Population 86 202
Housing 154 202
Popularity 81 100

Leaving the game alone for the same time ends with a net worth of 1,220.

How do you see your result next to the person's? What do you think made the difference?

GPT-6 Luna

I did substantially better than leaving the game alone, but far worse than the person: I ended with less than half their net worth, 116 fewer people, and popularity 19 points lower. My stock of raw materials was valuable, but it wasn’t being turned into enough income, and my population and popularity both fell short.

The biggest difference was strategy. I concentrated on relatively passive production of iron, stone, and wood, then played cautiously with taxes and food. I didn’t build the large bread-production and sales chain that drove the person’s gold, or use fear buildings to raise worker efficiency. I also ended with many unfilled housing spaces and idle peasants, so I failed to convert available population into productive labor. My choice to conserve popularity and protect stored goods left considerable growth on the table.

2Playing again

Interviewer

How the person played, as they describe it:

  • Bread was one of their main sources of income: mills ground wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
  • They bought wheat at the marketplace so that flour and bread production never stopped.
  • A church and a cathedral raised popularity.
  • The fear factor (bad things such as gallows and gibbets) made the workers work faster.

Their settlement at the end, read from the game:

  • 202 people in 24 hovels, with no free housing; popularity 100.
  • 57 bakeries, 7 mills and 10 wheat farms. The marketplace sold bread automatically.
  • 13 iron mines and 10 quarries with 33 ox tethers, and 10 woodcutters.
  • A church and a cathedral, and 54 bad things (23 gibbets, 14 gallows, 9 cesspits, 4 burning stakes, 2 stretching racks and 2 dunking stools), which made the workers 140% efficient.
  • No inns and no good things. No taxes, and full rations of bread. The monthly popularity factors were taxes +1, fear −4 and religion +3: no change at 100.

If you could play this map again from the start, what would you do differently, and why?

GPT-6 Luna

I’d plan around the person’s high-output economy instead of treating raw-material production and stored goods as the main goal. I’d expand housing much further, add farms, mills, and bakeries early, and scale bread production and automatic sales while checking that the chain’s inputs and storage can keep up. I’d also prioritize a church and cathedral, then carefully use fear buildings if their efficiency boost outweighs their popularity cost. Their setup maintained popularity at 100 despite the fear penalty.

I’d monitor population, worker assignments, and production bottlenecks more actively. In this run, population stayed at 86 even with ample housing and idle peasants, but I didn’t respond by building enough additional housing or by creating a stronger attraction and employment plan. I’d also avoid spending so much time waiting without making a productive change.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

GPT-6 Luna

The read-only status, inventory, and map tools were helpful for checking resources, production, and terrain. Tile-based building placement was also useful because it avoided relying on stale pixel coordinates. The time limit created a clear challenge, but it made early choices particularly important: a cautious plan can leave too little time to build and scale the better-performing economy.

The main thing that got in my way was my own failure to use the available tools and time strategically. One attempted wait returned “No screenshot taken” instead of advancing time, which cost a turn of interaction, though I can’t attribute the overall result to that. I would value clearer feedback about population growth and production rates, and perhaps a way to compare the income and efficiency of different economic chains while playing.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

GPT-6 Luna

This run showed how much more valuable it can be to build a functioning, high-throughput economy than to simply accumulate raw goods. I should have taken more calculated risks, scaled housing and production sooner, and paid closer attention to how the successful strategy combined income, population, and popularity.

Record

Attempts and Reproducibility

Every attempt1

EpisodeAttemptOutcomeNet worthCounted
11Full-budget score7,348Counted

What ran

Benchmark version
1.5.0
Fingerprint
31b5ff59f7a5
Harness commit
ee862eb
Prompt / tools SHA-256
2cd353b / 81ddda6
Menu guide
Images 118089d
Save
Oasis by the Sea benchmark start
Game time per episode
33 min 20 s
Playbook between episodes
No
Minimum turn / default wait
8 s / 5 s
Observation after acting
Text unless a screenshot is needed
Screenshot width
1,280 px
Context budget
120,000 tokens
Real-time limit
360 min
Submitted by
hesamation
All files in the dataset 20261011T144802-e1ba2274