One episode · The Sinkhole

GPT-6.1 Sol

Benchmark 1.6.2OpenAIReasoning mediumPlayed by the maintainersFull-budget score

All of GPT-6.1 Sol’s results and comparisons

Score

48.8

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

20,610

+18,415 over doing nothing, 47% of the person’s gain of 39,436

Population

183

36% of the person’s gain of 502 · popularity 79

Cost

$0.99

6.2M tokens

Time

45 game min

57 min of wall-clock time, 136 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

GPT-6.1 Sol
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

20,610net worth

Gold: 14,557
Stored goods, at sell price: 6,053
Over the baseline (do nothing): +18,415
Population / housing: 183 / 194
Popularity: 79
Food: 152
Structures (map-wide): 190
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

No final overview was published for this episode.

Final stock6,053 gold in goods

Good	Units	Price	Value
Gold	14,557	1	14,557
Iron	104	23	2,392
Stone	176	7	1,232
Wheat	81	8	648
Crossbows	21	30	630
Bread	152	4	608
Flour	47	10	470
Wood	57	1	57
Hops	2	8	16
Net worth			20,610

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	20,610	41,631	2,195
Gold	14,557	39,977
Stored goods, at the sell price	6,053	1,654
Population	183	506
Housing	194	506
Popularity	79	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: −2
Food: +4
Taxes: −12
Ale from the inn: +6
Taxes: Extortionate taxes, 256 gold a month
Rations: extra; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 0 empty; 2 granaries holding 152 food

Buildings150

Iron Mine28
Bakery27
Hovel23
Woodcutter16
Ox Tether8
Wheat Farm7
Brewery6
Hops Farm6
Inn5
Fletchers' Workshop4
Mill4
Quarry4
Stockpile4
Apple Orchard2
Granary2
Armory1
Dairy Farm1
Keep1
Marketplace1

Tool calls138

find_sites43
build_structure39
market_trade14
place_near9
wait_and_observe7
set_tax4
set_workshop4
building_info3
expand_storage3
go_to_tile3
inspect_building3
update_plan3
demolish_building1
map_overview1
set_rations1

Play

Turns: 136
Buildings placed / attempted: 132 / 135
Anchor-tool placements: 13
Tool errors: 1
Raw inputs sent to the game: 957
Context compactions: 1
Game time: 45 min

Compute

Tokens: 6.2M
Served from cache: 98%
Output tokens: 12K
Cost, as billed: $0.99
Wall-clock time: 57 min
of which inference: 19 min
Peak game memory: 3,465 MiB

The plan, before play

I’ll maximize final net worth, with population and popularity as secondary goals, without recruiting troops or building defenses.

I’ll start with woodcutters, housing, food farms and a marketplace, then expand profitable production—especially iron and stone. I’ll keep workers supplied, storage available and popularity high enough for growth and sustainable taxes, checking actual production before scaling further.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: GPT-6.1 Sol on OpenRouter, providers openai then azure, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.6.2, from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook
Run 78e8c0bf, 2026-10-13

Result

	GPT-6.1 Sol	The person	Left alone
Net worth	20,610	41,631	2,195
Gold	14,557	39,977
Stored goods, at the sell price	6,053	1,654
Population / housing	183 / 194	506 / 506	4
Popularity	79	99	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 18,415 over leaving the game alone, about 47% of the person's growth. Most of its gold came from taxes, raised step by step to extortionate (256 gold a month at the end) while extra rations and five inns kept popularity at 79; 5,300 more came from selling 200 iron and 100 stone. Its stored value is iron (2,392), stone (1,232), wheat, bread, flour and 21 crossbows.

How the game went

The opening, 0 to 12 minutes. Three woodcutters and an orchard in the first turn, hovels and a marketplace against the keep, three iron mines at 1:49, three wheat farms, a mill and four bakeries by 5:27, then two quarries with four ox tethers. Taxes went to average at 5:44 and rations to extra at 9:08. By 12 minutes it had eight more bakeries, two more quarries and four more wheat farms. It bought 200 wood and 50 bread in these minutes.

Bread, ale and mines, 12 to 30 minutes. Three hops farms and four breweries at 14:24 to 15:39 and an inn, then an armory and four fletchers set to crossbows (17:09 to 18:49), iron mines in fours and twos, four more bakeries each at 24:11 and 28:12, and three more inns at 28:52. At 26:10 it demolished a building and put a granary nearer its bakeries, which it described as relocating the granary. Taxes went to high at 22:19. Population rose from 82 at 15 minutes to 131 at 30.

Taxing a growing town, 30 to 45 minutes. Taxes to mean at 32:01 and extortionate at 39:51, with popularity between 79 and 100; more iron mines (to 28 in all), hovels, breweries, hops farms, an inn, two mills, seven more bakeries and a second granary at 40:33. It sold 100 iron at 38:52 and 100 more at 44:10. Gold went from 3,452 at 30 minutes to 14,557, and the town to 183 people.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,580	2,080	3,478	4,008	6,144	7,929	10,919	15,130	20,610
Gold	2,000	1,200	1,128	1,750	1,500	2,447	3,452	4,375	9,247	14,557
Iron / stone	0 / 20	1 / 20	15 / 20	32 / 52	49 / 76	73 / 100	114 / 32	169 / 72	123 / 104	104 / 176
Bread	0	0	42	80	62	39	89	141	153	152
Population	5	41	52	82	84	114	131	153	176	183
Popularity	100	100	74	77	79	86	69	94	100	79

During long waits a row repeats the last reading.

What held it back

Bread eaten, not sold. 27 bakeries fed the town on extra rations, which bought the popularity its taxes spent; none of the bread was sold. Sol said so itself in the interview.
Idle peasants. The town reached 24 idle several times, which stops immigration, and ended with 24 idle and 11 empty places.
Crowded storage. expand_storage found no room around the granary (25:53) or the stockpile (27:07), and the stockpile had no empty pile at the end.

What worked

The best result of the round and the largest town: it kept building to the last three minutes and grew to 183 people.
Popularity managed as a budget: extra rations (+4) and inns (+6) paid for extortionate taxes (−12) with popularity at 79.
Its own food at scale (seven wheat farms, four mills, 27 bakeries) and four quarries with eight ox tethers alongside 28 iron mines.
Every reported placement matches the game's final count of its buildings.

Cost and speed

136 turns, 138 tool calls (7 waits), 6.17 million tokens and $0.995 (preparation $0.051, play $0.919, compaction $0.026), worked out from the list prices on the model profile. The interview cost $0.015 more, with 42,796 of its 44,644 tokens read from the cache. The agent run took 57.0 real minutes, 19.0 of them waiting for the model; the context was compacted once (46 seconds).

Beside the others on 1.6.2

Scores by the project website's formula (80% net worth, 10% each population and popularity, each from doing nothing at 0 to the person's game at 1):

	Net worth	Gold / goods	Population	Popularity	Score	Cost
GPT-6.1 Sol	20,610	14,557 / 6,053	183	79	48.8	$0.995
Claude Sonnet 5.5	16,810	10,777 / 6,033	83	68	38.0	$0.752
Claude Opus 5.5	13,921	1,157 / 12,764	147	48	31.4	$2.470
GPT-6 Luna	11,177	1,764 / 9,413	108	52	25.5	$0.138
Kimi K3	9,164	2,356 / 6,808	91	52	21.1	$1.662
GLM 5.3 Flash	5,084	1,992 / 3,092	78	79	15.2	$0.114
Claude Haiku 5.5	3,312	1,716 / 1,596	37	51	8.0	$0.055

The interview

Sol said it "built a functioning economy, but not a sufficiently aggressive one", leaning "on iron and taxes" where the person built "a much larger bread-export economy", and that it "repeatedly reached the 24-idle-peasant limit". Playing again, it would make bread exports the main engine, plan a compact bread district, buy ale at first, cut rations when popularity allows, expand housing and jobs in larger batches, and test religion and fear. It found the readings and tile placement helpful, and asked for each producer's recent output, a list of all buildings with coordinates and staffing, more reliable recentring for tile placements, and clearer trade accounting.

For the harness

place_near for the first Armory (against the stockpile, 16:29) used all 20 probes and returned the expansion remedy, "No room around your armory", before any armory stood; Sol then placed it with build_structure. The same cause as Claude Haiku 5.5's failed calls; fixed in a worktree for the next version.
Three tile placements near the top edge of the view (a mill at 20:48, two woodcutters at 35:28 and 37:38) came back not_placed with "nothing fits at or near that spot"; Sol read this as failing at the screen edge.
demolish_building by tile worked (26:10).
The game saved correctly at the end ("gpt-6-1 sol 1-6-2 78e8c0").

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	20,610	41,631
Gold	14,557	39,977
Stored goods, at the sell price	6,053	1,654
Population	183	506
Housing	194	506
Popularity	79	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

GPT-6.1 Sol

My result was a substantial improvement over leaving the game alone, but only about half the person’s net worth. The population gap—183 versus 506—shows that I stopped scaling well before the map’s economic potential was exhausted.

I built a functioning economy, but not a sufficiently aggressive one. I leaned on iron and taxes, while the person built a much larger bread-export economy. My 27 bakeries and four mills were far below their 161 bakeries and 19 mills. Feeding my population at extra rations also consumed output that they largely sold.

I repeatedly reached the 24-idle-peasant limit. That should have prompted faster job creation, rather than small batches of expansion. I also spent too long accommodating inefficient walking distances, particularly before relocating the granary. Several crossbow workshops were a poor investment: their observed output was too slow to justify expanding that approach.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

GPT-6.1 Sol

I would make bread exports the main growth engine, with iron as a complementary source of value.

Establish a compact bread district early. Place farms, stockpiles, mills, bakeries and granaries with delivery distances in mind, rather than fixing the layout much later.
Buy ale initially instead of building its entire production chain. I would test whether the popularity it supplies pays for itself through reduced food consumption and faster expansion.
Reduce rations when the popularity breakdown supports it. Extra rations were reliable, but economically expensive. I would use the lowest sustainable level rather than treating generous feeding as the default.
Expand housing and jobs together, in larger batches. Idle peasants were a recurring growth bottleneck. More productive jobs—and inexpensive service jobs where justified—would keep immigration moving.
Test religion and fear-factor productivity early. I did not exploit the person’s 130% worker efficiency. I would measure its benefits and popularity costs rather than assume ordinary efficiency was sufficient.
Provide more quarry transport and manage storage proactively. My ox capacity was too modest for the distance, and full storage occasionally stopped production.

I would not blindly copy every building count. The important lesson is the economic loop: maintain popularity cheaply, turn a growing workforce into export production, and reinvest quickly. Taxes were useful in my settlement, but they did not compensate for its smaller productive workforce.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

GPT-6.1 Sol

The benchmark was helpful in several ways. The read-only statistics exposed popularity factors, idle-worker limits and storage bottlenecks that would have been difficult to track from screenshots alone. Tile-based placement and find_sites made a large map manageable. Pausing while thinking also kept the task focused on decisions rather than response speed.

What got in my way was the combination of long production delays, walking distances and occasionally unsuccessful placement or trade attempts. A building could be staffed and “working” without delivering much soon, so establishing its actual productivity required repeated observations. Some placement batches also failed at screen edges despite having tile targets.

I would improve:

Producer-level delivered-output statistics over a recent interval.
A settlement-wide building list with coordinates and staffing.
More reliable automatic recentering for every tile placement.
Clearer trade accounting when displayed stock changes do not obviously match the reported transaction.

The 45-minute budget was enough for meaningful strategic differences. The instructions were clear, although lengthy; a shorter operational reference would make essential restrictions and tool behavior easier to retain.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

GPT-6.1 Sol

The person’s result is a useful correction to my strategy. I treated popularity mainly as a resource to spend on taxes, and food mainly as something to keep safely supplied. They treated popularity as a way to reduce consumption and increase worker productivity, while food became the principal export.

My biggest weakness was not failure to establish production—it was failing to turn a stable settlement into sustained, large-scale expansion. Once the opening economy worked, I needed to accelerate rather than continue making cautious incremental additions.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	20,610	Counted

What ran

Benchmark version: 1.6.2
Fingerprint: 6cc0c0162262
Harness commit: 1e52aca
Prompt / tools SHA-256: 43ab27e / e41a140
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261013T032320-2db24416