One episode · The Sinkhole

GPT-6 Luna

Benchmark 1.7.1OpenAIReasoning mediumPlayed by the maintainersFull-budget score

All of GPT-6 Luna’s results and comparisons

Score

26.7

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

11,136

+8,941 over doing nothing, 23% of the person’s gain of 39,436

Population

105

20% of the person’s gain of 502 · popularity 66

Cost

$0.16

7.4M tokens

Time

45 game min

1 h 29 min of wall-clock time, 155 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

GPT-6 Luna
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

11,136net worth

Gold: 4,198
Stored goods, at sell price: 6,938
Over the baseline (do nothing): +8,941
Population / housing: 105 / 122
Popularity: 66
Food: 25
Structures (map-wide): 115
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock6,938 gold in goods

Good	Units	Price	Value
Gold	4,198	1	4,198
Iron	257	23	5,911
Crossbows	17	30	510
Bows	12	15	180
Stone	20	7	140
Cheese	25	4	100
Wood	49	1	49
Hops	6	8	48
Net worth			11,136

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	11,136	41,631	2,195
Gold	4,198	39,977
Stored goods, at the sell price	6,938	1,654
Population	105	506
Housing	122	506
Popularity	66	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: ±0
Food: +4
Taxes: −6
Religion: ±0
Ale from the inn: +2
Taxes: High taxes, 104 gold a month
Rations: extra; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 6 empty; 1 granary holding 25 food

Buildings79

Iron Mine24
Hovel14
Woodcutter9
Apple Orchard8
Dairy Farm7
Fletchers' Workshop5
Stockpile4
Armory1
Brewery1
Chapel1
Granary1
Hops Farm1
Inn1
Keep1
Marketplace1

Tool calls235

find_sites79
build_structure43
building_info32
set_tax17
set_rations10
wait_and_observe9
get_inventory8
status8
inspect_building6
market_trade5
place_near5
set_workshop4
map_overview3
update_plan3
observe2
notification_history1

Play

Turns: 155
Buildings placed / attempted: 69 / 75
Anchor-tool placements: 4
Tool errors: 0
Raw inputs sent to the game: 600
Context compactions: 5
Game time: 45 min

Compute

Tokens: 7.4M
Served from cache: 96%
Output tokens: 118K
Cost, as billed: $0.16
Wall-clock time: 1 h 29 min
of which inference: 47 min
Peak game memory: 3,513 MiB

The plan, before play

I’ll grow net worth through productive buildings, stored surplus, and taxes while maintaining popularity and adding housing for workers. I’ll first assess the live map and resources, then prioritize wood, housing, and food; expand into the nearby stone and iron deposits; and add workshops, storage, and services as workers and supplies allow. I’ll avoid military spending, monitor production and popularity, and keep improving the economy through the final minute.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: GPT-6 Luna (openai/gpt-6-luna) on OpenRouter, providers openai then azure and no others, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
Run ddbc11eb, series 20261015T140724-fab00866, 2026-10-15: Luna's first game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its 1.6.2 game

Result

	This game (1.7.1)	Its 1.6.2 game	The person	Left alone
Net worth	11,136	11,177	41,631	2,195
Gold	4,198	1,764	39,977
Stored goods, at the sell price	6,938	9,413	1,654
Population / housing	105 / 122	108 / 114	506 / 506	4
Popularity	66	52	99	0
Score	26.7	25.5	100	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 8,941 over leaving the game alone, 23% of the person's growth, almost exactly as in its 1.6.2 game. The score is the project website's formula (80% net worth, 10% each population and popularity); this game counts on the website's leaderboard. In this round so far Claude Haiku 5.5 scored 26.3 and GLM 5.3 Flash 22.6.

Of the stored value, 5,911 is 257 iron from twenty-four iron mines, 690 is 17 crossbows and 12 bows, 140 is the starting 20 stone and 100 is 25 cheese. The town ended with 24 idle peasants, no open jobs and 17 empty places in its hovels.

How the game went

A fast start and a tax misstep, 0 to 10 minutes. Two woodcutters, three hovels and a marketplace in the first turn, then orchards, dairy farms and a first iron mine at 2:24 after buying planks. It set mean taxes at 1:51 and took them off at 3:39, but popularity was down from 100 to 71 by 5 minutes. An armory and a fletcher's workshop beside it at 7:02. Food ran low, 14 at 10 minutes, and it moved rations between half and full six times by 16 minutes.

Popularity bought with bribes, 10 to 20 minutes. A hops farm, a brewery and an inn by 11:49; large and small bribes at 10:33, 12:33 and 15:01 alternated with low taxes; more dairy farms and orchards at 15:16, so food recovered to 78 by 20 minutes. From 17:08 it added an iron mine every turn or two, with woodcutters and hovels between.

Iron mines, 20 to 45 minutes. Twenty-four iron mines in all, the last at 42:25, five fletcher's workshops (four set to crossbows with set_workshop), a chapel at 26:00, and a hovel every few minutes; taxes changed nine more times, mostly between low and average. Population grew from 57 to 105 as hovels and mines opened. At the end it bought 50 cheese (43:12), set double then extra rations (44:01) and high taxes (44:30); food ended at 25. Net worth rose from 4,982 at 30 minutes to 11,136, mostly as iron.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	2,355	2,394	2,303	2,952	3,950	4,982	6,317	8,546	11,136
Gold	2,000	2,078	2,055	1,692	1,886	2,312	2,578	2,977	3,708	4,198
Iron	0	0	4	10	21	41	71	112	177	257
Food	55	27	14	50	78	95	85	69	30	25
Population / housing	6 / 10	32 / 34	38 / 50	48 / 50	57 / 66	65 / 74	76 / 90	84 / 90	98 / 106	105 / 122
Popularity	100	71	67	69	67	69	71	63	61	66

Each column is the last reading at or before that minute.

Beside its 1.6.2 game

Nearly the same game: iron mines (24 against 27), hovels (14 against 13), a few fletchers, and almost the same net worth and population. This one kept more gold and less iron (257 against 335) and ran popularity higher with bribes, an inn and, at the end, extra rations.

What held it back

24 idle peasants. From 3 minutes on, more than half of all readings showed 24 idle peasants and no open jobs, so the population grew only as fast as new workplaces opened, about one iron mine a turn. Luna named this in its interview.
Taxes and rations on a short leash. 17 tax changes and 10 ration changes; popularity spent most of the game between 61 and 71, which left taxes mostly low.
Iron kept raw. 257 iron at 23 each is most of the stored value; the five fletchers made 29 crossbows and bows, all kept.

What worked

A steady iron engine: twenty-four mines, iron rising about 15 a minute by the end.
Gold grew all game, from its lowest of 1,629 at 16 minutes to 4,198, against 1,764 in its 1.6.2 game.
Every placement reported as placed is in the game's final count of its buildings; the seven not placed (three dairy farms, two hovels, an iron mine short of planks, a hops farm off screen) were not built.

Market

30 planks and 50 cheese bought for 520 gold, a loss of 290 net worth against the sell price (planks 4 for 1, cheese 8 for 4); nothing sold.

Cost and speed

155 turns, 235 tool calls (79 find_sites, 43 build_structure, 32 building_info, 17 set_tax, 10 set_rations, 9 waits), 7.39 million tokens and $0.162 billed by OpenRouter (preparation $0.003, play $0.149, compaction $0.010), with 96.7% of input read from the cache; every request went to OpenAI. The interview cost $0.001 more, with 26,399 of its 29,612 tokens read from the cache. The agent run took 89.1 real minutes, the longest of the round so far, 47.3 of them waiting for the model; the context was compacted five times (177 seconds).

The interview

Luna said it kept "optimizing a modest production economy" instead of turning food and popularity into a larger, faster-growing town, and that its extra rations and high taxes at the end were not used deliberately. Playing again it would follow the person: inns with bought ale early, fewer rations with the food sold, a bread chain, housing as food and popularity allow, and jobs such as water pots so idle peasants do not block growth, with workshops second. It found the map, placement, production and inventory readings valuable and called its weakness strategic; it suggested making the effect of selling food, and of popularity, rations and housing, easier to compare in the status.

For the harness

No provider errors, stalled replies or refused calls. The game saved at the end ("gpt-6 luna 1-7-1 ddbc11", in this folder).
The slowest game of the round so far in real time: 155 turns at about 18 seconds of model time each, 79 find_sites and 32 building_info calls, and five compactions.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	11,136	41,631
Gold	4,198	39,977
Stored goods, at the sell price	6,938	1,654
Population	105	506
Housing	122	506
Popularity	66	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

GPT-6 Luna

I made substantial progress beyond leaving the settlement alone, but fell far short of the person’s result. I ended with a useful productive base and considerable stored goods, but too little gold, housing, and population. The main difference was that I kept optimizing a modest production economy rather than turning surplus food and popularity into a much larger, faster-growing settlement. I also kept extra rations and level-6 taxes late in the run, instead of using popularity and food more deliberately to support growth and preserve income.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

GPT-6 Luna

I would plan around the person’s strategy from the beginning: establish inns and secure ale early to raise popularity, then reduce rations and sell food surplus rather than consuming it. I’d build a bread chain and expand housing as food and popularity support more arrivals, while using jobs such as water pots to keep idle peasants from blocking further growth. I’d still expand iron production, but I’d treat workshops as secondary unless production data showed that converting resources was profitable.

I’d monitor housing, idle peasants, food, and popularity more actively, and make tax and ration changes according to the settlement’s current capacity. In my run, I had 24 idle peasants and no open jobs, yet only 122 housing and 105 people. I should have invested much more of the available time in housing and the food-and-popularity loop that could support a far larger population. The person’s low taxes, religion, fear factor, and inns also provided a much stronger popularity balance than my reliance on extra rations.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

GPT-6 Luna

The tools and readings were valuable: map and terrain tools, placement checks, production stats, and inventory readings made it possible to build and verify an economy without guessing from screenshots alone. The fixed time budget and clear prohibition on military spending also made the objective straightforward.

The main weakness in my play was strategic, not a lack of information: I didn’t identify early enough that population growth and food monetization were the central opportunity. For a benchmark like this, it could help to make the scoring implications of selling food and the effects of popularity, rations, and housing especially easy to compare in the live status display. The time limit creates a useful test of prioritization, but it also rewards learning and exploiting the compounding growth loop early.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

GPT-6 Luna

The result makes clear that a working economy is not necessarily a strong one: my mines, farms, and workshops generated substantial stored value, but that value could not match the gold and population produced by a faster-growing settlement. In a replay I’d aim to scale the population and sell surplus food earlier, rather than spending so much of the run accumulating goods.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	11,136	Counted

What ran

Benchmark: The Sinkhole construction
Benchmark version: 1.7.1
Fingerprint: 63921aad76e5
Harness commit: 674e7ba
Prompt / tools SHA-256: 3b0fa8a / f75ebb8
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261015T140724-fab00866