One episode · The Sinkhole

Gemini 4 Argon

Benchmark 1.7.1GoogleReasoning mediumPlayed by the maintainersFull-budget score

All of Gemini 4 Argon’s results and comparisons

Score

52.4

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

23,189

+20,994 over doing nothing, 53% of the person’s gain of 39,436

Population

215

42% of the person’s gain of 502 · popularity 56

Cost

$1.45

3.0M tokens

Time

45 game min

1 h 1 min of wall-clock time, 85 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Gemini 4 Argon
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

23,189net worth

Gold: 15,929
Stored goods, at sell price: 7,260
Over the baseline (do nothing): +20,994
Population / housing: 215 / 250
Popularity: 56
Food: 419
Structures (map-wide): 235
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock7,260 gold in goods

Good	Units	Price	Value
Gold	15,929	1	15,929
Iron	139	23	3,197
Bread	419	4	1,676
Stone	208	7	1,456
Crossbows	8	30	240
Spears	21	10	210
Metal armour	6	30	180
Pikes	7	18	126
Ale	6	10	60
Wood	45	1	45
Wheat	5	8	40
Bows	2	15	30
Net worth			23,189

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	23,189	41,631	2,195
Gold	15,929	39,977
Stored goods, at the sell price	7,260	1,654
Population	215	506
Housing	250	506
Popularity	56	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 19
Popularity next month: ±0
Food: +8
Taxes: −8
Taxes: Mean taxes, 258 gold a month
Rations: double; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 4 empty; 3 granaries holding 419 food

Buildings196

Bakery57
Hovel30
Woodcutter30
Iron Mine19
Wheat Farm15
Apple Orchard7
Mill7
Ox Tether7
Stockpile4
Armorer's Workshop3
Granary3
Quarry3
Brewery2
Fletchers' Workshop2
Poleturner's Workshop2
Armory1
Dairy Farm1
Hops Farm1
Keep1
Marketplace1

Tool calls221

find_sites134
build_structure28
building_info24
market_trade14
set_tax7
set_rations4
set_workshop4
update_plan3
expand_storage1
map_overview1
wait_and_observe1

Play

Turns: 85
Buildings placed / attempted: 188 / 195
Anchor-tool placements: 2
Tool errors: 1
Raw inputs sent to the game: 1,008
Context compactions: 2
Game time: 45 min

Compute

Tokens: 3.0M
Served from cache: 92%
Output tokens: 71K
Cost, as billed: $1.45
Wall-clock time: 1 h 1 min
of which inference: 19 min
Peak game memory: 3,519 MiB

The plan, before play

Understanding of the Task

Goal: Grow the economy on The Sinkhole Free Build map within 45 minutes of game time (2700 game seconds).
Evaluation: Judged by final settlement state—Net Worth (gold + stored goods valued at marketplace sell price) is the primary metric, followed by population and popularity.
Restrictions: No recruiting soldiers or building walls, towers, gates, or other defenses. Never press P, Escape, or Space.
Map & Starting State: Granary and 4 stockpiles are already placed beside the keep. Starting resources are 2,000 gold, 55 wood, 20 stone, and 55 meat, with 5/10 population. Oasis grass rings the basin around the keep; tree groves lie ~40 tiles SW; iron ore lies 50–90 tiles east inside the basin; stone deposit lies ~85 tiles NE.

Opening Plan

Survey & Bootstrap (Turn 1): Use map_overview and find_sites while paused (zero game-time cost) to locate optimal spots for Woodcutters, Marketplace, Hovels, Farms, Iron Mines, and Quarries. Place a Marketplace, initial Woodcutters near trees, Hovels (out of the way of the keep), and initial Apple Orchards/Wheat Farms.
Accelerate Expansion via Market: Use starting gold to buy wood at the Marketplace so we can immediately build out a massive raw-material and food infrastructure (Quarries + Ox Tethers, Iron Mines, Woodcutters, Hovels) without waiting for slow initial wood deliveries.
Scale High-Value Production & Bread/Ale Chains:
- Expand quarries and iron mines rapidly (stone sells/values at 7, iron at 23).
- Build Wheat Farms, Mills, and Bakeries near the keep/granary to support high rations, food variety, and large food surpluses.
- Add Hops Farms, Breweries, Inns, and Religion/Weapon workshops (Crossbows/Maces/Swords/Metal Armor) to maximize net worth and popularity (allowing high taxes and steady population growth).
Continuous Monitoring: Expand stockpiles, granaries, and armories before they fill, keep idle peasants below 24 by adding jobs + housing together, and tune taxes/rations to maximize gold and net worth up to the final second.

Commentary during play2

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Gemini 4 Argon (google/gemini-4-argon) on OpenRouter, provider Google AI Studio only, reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
Run 7a88d5fe, series 20261015T204651-fca746fe, 2026-10-15: Gemini's first valid game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its 1.6.2 game. An earlier series that evening (20261015T194933-304ab830) never started play: the game would not launch after the host's computer had slept.

Result

	This game (1.7.1)	Its 1.6.2 game	The person	Left alone
Net worth	23,189	30,996	41,631	2,195
Gold	15,929	25,454	39,977
Stored goods, at the sell price	7,260	5,542	1,654
Population / housing	215 / 250	246 / 274	506 / 506	4
Popularity	56	62	99	0
Score	52.4	69.4	100	0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 20,994 over leaving the game alone, 53% of the person's growth, against 28,801 (73%) in the 1.6.2 game. The score is the project website's formula (80% net worth, 10% each population and popularity); this game counts on the website's leaderboard. It is the best of this round, ahead of GPT-6.1 Sol (50.4).

Of the stored value, 3,197 is 139 iron, 1,676 is 419 bread in three granaries, 1,456 is 208 stone and about 800 is weapons and armour. The town ended with 19 idle peasants and 35 empty places in its hovels.

How the game went

Planks, quarries and the first bakeries, 0 to 10 minutes. A marketplace, three woodcutters and three hovels in the first turn, 100 planks bought at 0:46 and again at 3:09, three iron mines at 1:35, and three quarries with four ox tethers at 3:53. Low taxes from 5:40, apples and cheese bought for food, orchards and wheat farms, a mill and three more iron mines at 7:15, and four bakeries at 8:48. Gold was down to 793 by 10 minutes, and it set half rations at 10:13.

The bread chain, 10 to 25 minutes. Four iron mines at 10:53; full rations and no taxes from 12:03 (gold reached its lowest, 349, at about 14 minutes), then low taxes again. Bakeries in fives at 12:37, 18:55 and 22:01, more iron mines and mills, extra rations from 18:48 and average taxes from 20:15, two more granaries at 20:22, and at 24:07 an armory with two fletcher's and three armourer's workshops and a brewery.

Double rations and mean taxes, 25 to 45 minutes. At 25:23 double rations and at 25:29 mean taxes, which it kept to the end but for extortionate taxes from 41:30 to 44:16. Bakeries went up in fives to eights from 30:04 to 42:47, fifty-seven in all, with woodcutters, hovels, wheat farms and mills. It sold 100 stone at 29:46, 200 iron at 31:14 and 36:59 and 20 metal armour at 41:13. Gold rose from 2,390 at 30 minutes to 15,929, and the town from 148 to 215 people.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,943	1,294	1,800	3,705	5,417	8,919	12,703	17,327	23,189
Gold	2,000	1,600	793	509	677	1,457	2,390	7,243	12,137	15,929
Bread	0	0	0	18	148	139	216	143	190	419
Iron	0	0	8	26	63	106	165	122	81	139
Population / housing	6 / 10	33 / 50	61 / 66	80 / 90	101 / 122	120 / 162	148 / 178	169 / 194	196 / 218	215 / 250
Popularity	100	100	89	63	57	66	72	74	77	56

Each column is the last reading at or before that minute.

Beside its 1.6.2 game

The same bread economy, at the same size: 57 bakeries against 56, double rations both times. The 1.6.2 game also built eight inns with eleven breweries and seven hops farms, and with that popularity ran extortionate to cruel taxes (344 gold a month at the end) on 30 iron mines. This game built no inn, two breweries and one hops farm, so double rations alone balanced its taxes, which stayed at mean (258 a month at the end); it had 19 iron mines and sold 200 iron. The difference is mostly gold: 15,929 against 25,454.

What held it back

No inns. Popularity rested on double rations (+8) against mean taxes (−8), with nothing left over to tax harder; when it tried extortionate taxes at 41:30, popularity fell from 77 to 56 by the end. Gemini named inns with bought ale as its first change.
Bread eaten. Double rations for 200 people ate much of what 57 bakeries made; none was sold, and 419 bread was left at 4 each.
Fewer mines. 19 iron mines and 3 quarries against 30 and 4 in its 1.6.2 game; it sold 200 iron for gold rather than keep it.
Real time. 85 turns in 61 real minutes: Google AI Studio refused 26 requests with 429 (each retried with the game paused), and it called find_sites 134 times. Gemini named the time each batch of placements took, with the camera moving between distant areas, as a limit on its pace.

What worked

The largest economy of the round: 57 bakeries, 215 people, and the best net worth and score.
Mean taxes on double rations from 25 minutes: gold rose by 13,539 in the last 15 minutes, about 5,200 of it from selling iron and armour and most of the rest from taxes.
Every placement reported as placed is in the game's final count of its buildings; the seven not placed were not built.

Market

500 planks, 50 apples, 25 cheese and 40 meat bought for 2,920 gold, a loss of 1,960 net worth against the sell price (planks 4 for 1, food 8 for 4); 100 stone, 200 iron and 20 metal armour sold for 5,900, which leaves net worth unchanged.

Cost and speed

85 turns, 221 tool calls (134 find_sites, 28 build_structure, 24 building_info, 14 market_trade), 3.04 million tokens and $1.447 billed by OpenRouter (preparation $0.044, play $1.338, compaction $0.064), with 92.6% of input read from the cache; every request went to Google AI Studio. The interview cost $0.027 more, with 67,438 of its 71,468 tokens read from the cache. The agent run took 61.2 real minutes, 18.8 of them waiting for the model; the context was compacted twice (50 seconds).

The interview

Gemini said double rations to carry mean and extortionate taxes fed its peasants much of its bread, where inns with ale could have bought the same popularity and let it cut rations and keep the bread; it named the pace of placing buildings and the unused fear factor as the other two differences. Playing again it would build inns with bought ale instead of double rations, go all in on the bread chain and on stone and iron early, add religion and bad things for worker efficiency, and batch each turn's placements in one camera view. It called the benchmark "one of the best-designed game benchmarks" it had met, praising read-only tools before the first action, find_sites by ground and the production and storage readings. It asked for build_structure to place an armory the way expand_storage does, since the camera jumps to the existing one, and for larger batches when all placements are in view.

For the harness

26 refusals (429) from Google AI Studio, each retried with the game paused; no game time lost.
An armory placed by tile in a batch at 39:40 was not placed; Gemini reports the camera jumping to the existing armory when one is chosen (camera_moved).
No stalled replies or refused calls; the game saved at the end ("gemini 4 argon 1-7-1 7a88d5", in this folder). The stockpile kept room (4 empty piles at the end).

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	23,189	41,631
Gold	15,929	39,977
Stored goods, at the sell price	7,260	1,654
Population	215	506
Housing	250	506
Popularity	56	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Gemini 4 Argon

Reaching 23,189 net worth (up roughly 10x from the starting state) with 215 population, 57 bakeries, 15 wheat farms, 19 iron mines, and 30 woodcutters felt like a solid industrial buildup, especially once bread production crossed 140 loaves per minute in the final third of the run. However, seeing the human player's 41,631 net worth and 506 population makes clear where my economic model was sub-optimal.

Three big differences stand out:

Rations vs. Inns for Popularity: I ran double rations (+8 popularity) to offset Mean/Extortionate taxes (-8 to -12 popularity). That meant my 200+ peasants were eating a massive share of my bread and apples every minute. Each loaf of bread eaten is 4 gold of net worth lost, whereas buying or supplying ale to Inns provides up to +8 popularity much more cheaply, allowing rations to be cut to none or half so nearly 100% of baked bread turns into pure net worth.
Scaling Pace & Action Throughput: Because each turn of up to 8 placements takes game time (especially whenever the camera had to move between distant zones like the stone/iron deposits in the northeast and the farmland in the southwest), I topped out at 215 population and 57 bakeries, whereas the human reached 506 population and 161 bakeries.
Fear Factor (+30% Worker Efficiency): I didn't use bad things (fear factor) at all. At 130% worker efficiency across 161 bakeries, 38 iron mines, and 10 quarries, every worker produces 30% more goods per minute, which compounds dramatically over 45 minutes.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Gemini 4 Argon

If I played this map again from the start, I would change my strategy in five concrete ways:

Use Inns + Purchased Ale Instead of Double Rations: Instead of feeding the population double rations to support taxes, I would build Inns as population grows and buy ale at the marketplace (or brew it) to get +6 to +8 popularity from ale. That would let me run none or half rations and keep almost every loaf of bread my bakeries produce, which is worth far more than the tax income double rations enabled.
Go All-In on the Bread Chain & Iron/Stone Early: Bread has an extraordinary return on wood invested (1 wheat farm + mills + ~5 bakeries turns 15–70 wood into a huge stream of 4-gold loaves with zero gold construction cost). I would clear out any early apple orchards once wheat farms need their oasis grass, pack all 30+ wheat farms onto the oasis ring, and build 150+ bakeries on the plain ground near the keep.
Exploit All Stone & Iron Deposits: The human fit 10 quarries (with 29 ox tethers) and 38 iron mines, whereas I only had 3 quarries and 19 iron mines because I didn't probe the full extent of the northeastern stone and eastern iron deposits early enough.
Add Religion + Negative Fear Factor (Bad Things): Once basic bread production is humming, building a Church/Cathedral (+3 popularity) plus Inns (+6 to +8 popularity) creates enough popularity cushion to place punishment structures for -5 fear factor (+25% to +30% worker efficiency) while still keeping popularity above 50.
Strict Camera Batching: Moving the camera mid-call in build_structure costs valuable game seconds. I would dedicate entire 8-action turns to a single screen view—for example, placing 8 bakeries in one view, then 8 hovels in one view, then 8 iron mines in one view—so almost every turn runs close to the minimum 8-second floor.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Gemini 4 Argon

This is one of the best-designed game benchmarks I have encountered:

What helped most:
- Making read-only tools (find_sites, map_overview, status, get_inventory) pause-free before the first action in a turn is a fantastic design choice. It allows genuine spatial planning without penalizing the model's thinking or querying time.
- find_sites grouping non-overlapping sites by terrain (plain, oasis_grass, stone, iron_ore) and checking worker walking access prevented the classic RTS agent failure mode of walling in its own keep or placing farms on rocky ground.
- The production and storage breakdowns in the observation stats made bottleneck diagnosis (e.g., seeing wheat pile up while flour hit zero, or seeing empty_piles drop) immediate and quantitative.
What got in the way / what I would change:
- Choosing Stockpile, Granary, or Armory in build_structure causes the game itself to snap the camera to the existing storage building, which occasionally caused retryStopped: "camera_moved" when trying to place an Armory by tile in the middle of a multi-building batch. Having build_structure automatically handle Armory the same way expand_storage does (or reminding the agent to always use expand_storage for armories too) would make that smoother.
- The 8-action-per-turn cap is fair, though when scaling to 500+ population and 300+ buildings in 45 minutes, even 8 buildings per minimum turn length creates a mild ceiling on how fast an agent can place 100+ 1x1 bad things or 160 bakeries compared to a fast human clicking rapidly. Allowing slightly larger same-view batches (e.g. 12 placements if all are in the current camera view) would let agents test their true macro ceiling.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Gemini 4 Argon

Playing Stronghold Crusader through this harness was genuinely fun. The tension between short-term net worth (not wasting gold on wood purchases or expensive workshops late in the run) and long-term compounding (keeping idle peasants below 24 so immigration never stalls, balancing wheat -> flour -> bread ratios, and managing stockpile piles via marketplace sales) captures the core economic puzzle of the game remarkably well. Seeing the human's "Inns + no rations + 130% fear-factor efficiency + 161 bakeries" build after the match is also a great touch—it immediately illuminates the deeper mechanics of the game.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	23,189	Counted

What ran

Benchmark: The Sinkhole construction
Benchmark version: 1.7.1
Fingerprint: 63921aad76e5
Harness commit: 674e7ba
Prompt / tools SHA-256: 3b0fa8a / f75ebb8
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261015T204651-fca746fe