One episode · The Sinkhole

GPT-6.1 Sol

Benchmark 1.7.1OpenAIReasoning mediumPlayed by the maintainersFull-budget score

All of GPT-6.1 Sol’s results and comparisons

Score

50.4

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

20,384

+18,189 over doing nothing, 46% of the person’s gain of 39,436

Population

181

35% of the person’s gain of 502 · popularity 100

Cost

$0.90

5.2M tokens

Time

45 game min

54 min of wall-clock time, 111 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

GPT-6.1 Sol
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

20,384net worth

Gold: 13,934
Stored goods, at sell price: 6,450
Over the baseline (do nothing): +18,189
Population / housing: 181 / 186
Popularity: 100
Food: 619
Structures (map-wide): 210
Attempt: 2 (played again)

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock6,450 gold in goods

Good	Units	Price	Value
Gold	13,934	1	13,934
Bread	619	4	2,476
Iron	39	23	897
Crossbows	29	30	870
Flour	84	10	840
Stone	116	7	812
Wheat	33	8	264
Wood	180	1	180
Ale	8	10	80
Hops	2	8	16
Bows	1	15	15
Net worth			20,384

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	20,384	41,631	2,195
Gold	13,934	39,977
Stored goods, at the sell price	6,450	1,654
Population	181	506
Housing	186	506
Popularity	100	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: ±0
Food: +8
Taxes: −8
Ale from the inn: ±0
Taxes: Mean taxes, 217 gold a month
Rations: double; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 0 empty; 4 granaries holding 619 food

Buildings171

Bakery45
Woodcutter43
Hovel22
Wheat Farm13
Ox Tether10
Brewery5
Iron Mine5
Fletchers' Workshop4
Granary4
Mill4
Stockpile4
Apple Orchard3
Quarry3
Hops Farm2
Armory1
Inn1
Keep1
Marketplace1

Tool calls115

find_sites43
build_structure35
market_trade9
place_near6
inspect_building4
set_workshop4
expand_storage3
set_tax3
set_rations2
update_plan2
building_info1
demolish_building1
map_overview1
wait_and_observe1

Play

Turns: 111
Buildings placed / attempted: 153 / 155
Anchor-tool placements: 13
Tool errors: 1
Raw inputs sent to the game: 890
Context compactions: 1
Game time: 45 min

Compute

Tokens: 5.2M
Served from cache: 98%
Output tokens: 12K
Cost, as billed: $0.90
Wall-clock time: 54 min
of which inference: 16 min
Peak game memory: 3,535 MiB

The plan, before play

I’ll maximize final net worth, with population and popularity as secondary goals, without recruiting troops or building defenses.

I’ll open with woodcutters, housing, food production and a marketplace, then expand into productive iron and stone sites. I’ll reinvest early income, maintain worker access and storage capacity, and adjust taxes to sustainable popularity. Production statistics will guide continued expansion throughout the 45-minute run.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: GPT-6.1 Sol (openai/gpt-6.1-sol) on OpenRouter, providers openai then azure and no others, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
Run 58233695, series 20261015T161235-97bd8ae4, 2026-10-15: Sol's first game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its 1.6.2 game. It is the series' second attempt: the first (run 58489527) ended at about 40 game minutes when OpenAI returned an error the host does not retry (see "For the harness"), and the runner played the episode again from the start.

Result

	This game (1.7.1)	Its 1.6.2 game	The person	Left alone
Net worth	20,384	20,610	41,631	2,195
Gold	13,934	14,557	39,977
Stored goods, at the sell price	6,450	6,053	1,654
Population / housing	181 / 186	183 / 194	506 / 506	4
Popularity	100	79	99	0
Score	50.4	48.8	100	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 18,189 over leaving the game alone, 46% of the person's growth, almost exactly as in its 1.6.2 game. The score is the project website's formula (80% net worth, 10% each population and popularity); this game counts on the website's leaderboard. It is the best of this round so far, ahead of Claude Sonnet 5.5 (33.1).

Of the stored value, 2,476 is 619 bread in four granaries, 897 is 39 iron, 870 is 29 crossbows, 840 is 84 flour and 812 is 116 stone. The town ended with 24 idle peasants, 5 empty places in its hovels and a full stockpile.

How the game went

Planks, iron and bread, 0 to 13 minutes. Three woodcutters in the first turn, a marketplace and 100 planks bought at 0:48, three orchards, three hovels, five iron mines at 2:42 and five more woodcutters. Low taxes from 4:30. Another 100 planks at 5:12, then the bread chain: wheat farms, a mill at 6:03 and three bakeries at 6:30, quarries with five ox tethers at 7:33, five hovels at 9:24 and five more bakeries at 11:09. Food fell to 20 by 10 minutes and it bought 50 bread. An armory and four fletcher's workshops (set to crossbows) at 12:11.

Scaling the bakeries, 13 to 27 minutes. Two more mills at 15:05, extra rations at 15:37, bakeries in fives (15:51, 19:08, 23:13), a third quarry with five more tethers, hovels in fives, six woodcutters, wheat farms, two hops farms and five breweries at 24:15, an inn at 26:45. Average taxes from 20:06. When expand_storage found no room for a granary at 25:35, it demolished a building in the way and placed three granaries at 26:14.

Double rations and mean taxes, 27 to 45 minutes. At 27:09 double rations, at 27:23 mean taxes, which it kept to the end; popularity stayed at 100. Five wheat farms, then bakeries in fives to 45 and woodcutters in fives to 43, with more hovels. From 35:48 it sold 230 stone, 210 planks and 100 iron for 4,120 gold as the stockpile filled (expand_storage found no room beside it at 33:36). The reply to the first stone sale said 100 units and 700 gold; the recording shows 130 and 910 (the 1.7.3 market_trade fault: the game sells 50 a click from 200 in stock; the tool expected 20), so it sold 30 more than asked, and the first planks sale, at 40:19, replied 100 and 100 where the recording shows 110 and 110, 10 more than asked. Gold rose from 3,395 at 30 minutes to 13,934.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,966	2,044	2,702	4,120	5,670	8,692	11,910	15,181	20,384
Gold	2,000	1,600	1,417	1,029	1,529	1,992	3,395	5,213	7,824	13,934
Food	55	48	20	88	105	166	252	343	448	619
Stone	20	20	20	36	68	100	172	220	138	116
Population / housing	6 / 10	31 / 34	41 / 50	76 / 90	94 / 130	112 / 130	130 / 130	152 / 170	166 / 170	181 / 186
Popularity	100	99	89	73	88	95	100	100	100	100

Each column is the last reading at or before that minute. Food is bread and apples in the granaries.

Beside its 1.6.2 game

The same net worth by another road. The 1.6.2 game had 28 iron mines, 27 bakeries and five inns, and carried extortionate taxes with popularity near 80. This one built five iron mines and 45 bakeries, fed the bread to the town at double rations, and held popularity at 100 under mean taxes, with 43 woodcutters as the cheapest jobs. Gold, population and net worth ended within a few percent of each other.

What held it back

Bread eaten, not sold. 45 bakeries made bread that went to double rations and into four granaries (619 at the end, 4 each); none was sold. Sol named this first in its interview.
Five iron mines. Iron, worth 23, was the most valuable good it could dig; it sold the 100 it had rather than building more mines.
Cheap jobs. Forty-three woodcutters delivered about 0.7 planks a minute each, worth 1 apiece; they were cheap work for a town that kept reaching 24 idle peasants (18 of 69 readings).

What worked

Mean taxes from 27 minutes with popularity at 100, paid for by double rations: from 30 minutes on gold rose by 10,539, of which 4,120 came from sales and most of the rest from taxes.
Building in batches of five with tile targets: 166 buildings placed in 111 turns, the most of the round so far.
When the granary had no room, it cleared a spot by demolishing one building and expanded there.
Every placement reported as placed is in the game's final count of its buildings, less the one it demolished.

Market

300 planks and 50 bread bought for 1,600 gold, a loss of 1,100 net worth against the sell price (planks 4 for 1, bread 8 for 4); 230 stone, 210 planks and 100 iron sold for 4,120, which leaves net worth unchanged.

Cost and speed

111 turns, 115 tool calls (43 find_sites, 35 build_structure, 9 market_trade, one wait), 5.19 million tokens and $0.905 billed by OpenRouter (preparation $0.053, play $0.820, compaction $0.031), with 98.2% of input read from the cache; every request went to OpenAI. The interview cost $0.015 more, with 49,443 of its 51,135 tokens read from the cache. The agent run took 53.5 real minutes, 15.6 of them waiting for the model; the context was compacted once (51 seconds). The failed first attempt cost a further $0.899 for 115 requests.

The interview

Sol called its economy viable but much weaker than the person's, and the difference strategic: it used double rations to support taxes, eating food that could have been sold, and expanded too cautiously (45 bakeries and five iron mines against 161 and 38), hesitating at slow first deliveries and hitting the idle limit. It said it protected popularity "more carefully than I pursued growth". Playing again it would use inns and bought ale early, scale the bread chain in planned districts and sell the surplus, create jobs ahead of growth, build many more mines and quarries, try religion and fear for worker efficiency, and plan storage before building around it. It praised screenshots with structured readings and tile targets, and named action overhead: construction batches taking about a game minute and camera moves making distant placements costly. It asked for a compact building list with coordinates, more visibility into bottlenecks and travel time, and structured market controls.

For the harness

The first attempt was lost to an error the host does not retry. At about 40 game minutes, with net worth 15,132, OpenAI answered "An error occurred while processing your request. You can retry your request…", a server error whose message has no status code; isTransientProviderError matches neither its wording nor a code, so the run ended and the episode was played again. Neither attempt's play changed for it, but the round lost 50 real minutes and $0.90.
No room beside the stockpile (33:36), as in Claude Sonnet 5.5's and GLM 5.3 Flash's 1.7.1 games: three games of this round now. Sol also found no room for a granary (25:35) until it demolished a building.
market_trade replied low on two sales: the 35:48 stone sale by 30 (the game sells 50 a click from 200 in stock where the tool expected 20) and the 40:19 planks sale by 10, from about 130 in stock, for a reason the logs do not show. The figures here are the recording's; the score is unaffected.
The game saved at the end ("gpt-6-1 sol 1-7-1 582336", in this folder).

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	20,384	41,631
Gold	13,934	39,977
Stored goods, at the sell price	6,450	1,654
Population	181	506
Housing	186	506
Popularity	100	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

GPT-6.1 Sol

I built a viable economy, but the person’s was much stronger: my net worth was about 49% of theirs, and my population about 36%. Beating the unattended baseline substantially is meaningful, but it does not close that gap.

The main difference was strategy, not simply execution speed. I relied on double rations to support taxes. That earned steady gold, but consumed food that could otherwise have been sold. The person used imported ale, religion and worker-efficiency bonuses to support a much larger production economy.

I also expanded too cautiously. I finished with 45 bakeries and five iron mines against their 161 and 38. I repeatedly treated slow initial deliveries as a reason to hesitate rather than establishing and staffing additional capacity in parallel. My population then hit the 24-idle-peasant limit repeatedly. Available housing alone could not solve that; I needed substantially more jobs.

Keeping popularity at 100 was a success operationally, but it was not the main objective. I protected it more carefully than I pursued growth.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

GPT-6.1 Sol

I would make several changes from the start:

Use inns early and test imported ale. Buying an input can be profitable when it enables lower food consumption or greater worker output. I focused too much on purchases as immediate losses of net worth and not enough on their enabling returns.
Scale the bread chain aggressively. Build farms, mills and bakeries in planned districts with short storage routes, then adjust their proportions using delivered-output readings. Surplus bread should become saleable output, not an excuse to increase rations.
Create jobs ahead of population growth. Keep housing available, but also watch the idle cap. Use productive jobs first and inexpensive multi-worker buildings where necessary to keep immigration moving.
Expand extraction much earlier. Five iron mines was far below what the map could support. I would establish many more mines and a larger quarry-and-ox operation while the food economy develops.
Evaluate religion and fear as production investments. The person’s 130% efficiency applied across a large workforce. I left that multiplier unused.
Plan storage expansion before surrounding it. My stockpile expansion eventually failed because nearby ground was occupied. Selling goods relieved the immediate pressure, but better layout would have prevented the constraint.

I would not copy every final building count blindly. I would adopt the person’s economic model, measure its performance early, and scale what demonstrably worked.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

GPT-6.1 Sol

The strongest feature was the combination of screenshots with structured readings. Worker vacancies, popularity factors, storage room and recent deliveries made it possible to distinguish a placed building from an operating one. Tile-targeted construction and site searches were particularly valuable on uneven terrain.

The game-time budget was clear, and pausing during reasoning made this a test of decisions rather than typing speed. The restrictions and scoring priorities were also explicit.

What got in my way was action overhead and fragmented feedback. Construction batches sometimes consumed roughly a minute of game time, and camera movement could make distant placements more expensive than expected. Automatic observations after individual actions also made my intended build–inspect–wait workflow less compact. I should have adapted better, but predictable batch timing would help.

I would improve the interface with:

A compact, current list of buildings and their coordinates.
More visibility into production bottlenecks and delivery travel time.
Clearer expected game-time costs for compound actions.
Documented structured controls for market automation, inns and fear-factor management, where the game supports them.

Those changes would reduce interface bookkeeping without removing the economic decisions.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

GPT-6.1 Sol

My biggest lesson is that a stable economy is not necessarily a competitive growth economy. I spent too much effort maintaining a comfortable equilibrium and too little effort multiplying productive capacity.

The person’s result is a useful demonstration that food, popularity and labor are interconnected economic resources. I mostly used food to sustain taxation; they used popularity support to free food for sale and sustain a much larger workforce. That was the more powerful approach on this map.

Record

Attempts and Reproducibility

Every attempt2

Episode	Attempt	Outcome	Net worth	Counted
1	1	Infrastructure failure, played again run error (error); game-time budget not used up; Error: An error occurred while processing your request. You can retry your request, or contact us through our help center at help.openai.com if the error persists. Please include the request ID req_1827eec29faa485588fc83c2b17f5a84 in your message.	15,132	Not counted
1	2	Full-budget score	20,384	Counted

A failure outside the model (the harness, game, network or provider) is played again, so it never costs the model an episode. A failure of the model’s own is kept as its result.

What ran

Benchmark: The Sinkhole construction
Benchmark version: 1.7.1
Fingerprint: 63921aad76e5
Harness commit: 674e7ba
Prompt / tools SHA-256: 3b0fa8a / f75ebb8
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261015T161235-97bd8ae4