One episode · The Sinkhole

Gemini 4 Argon

Benchmark 1.6.2GoogleReasoning mediumPlayed by the maintainersFull-budget score

All of Gemini 4 Argon’s results and comparisons

Score

69.4

Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game

Net worth

30,996

+28,801 over doing nothing, 73% of the person’s gain of 39,436

Population

246

48% of the person’s gain of 502 · popularity 62

Cost

$1.85

2.3M tokens

Time

45 game min

1 h 1 min of wall-clock time, 41 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

Gemini 4 Argon
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

30,996net worth

Gold: 25,454
Stored goods, at sell price: 5,542
Over the baseline (do nothing): +28,801
Population / housing: 246 / 274
Popularity: 62
Food: 227
Structures (map-wide): 262
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock5,542 gold in goods

Good	Units	Price	Value
Gold	25,454	1	25,454
Iron	138	23	3,174
Bread	227	4	908
Stone	124	7	868
Flour	23	10	230
Swords	6	30	180
Wood	96	1	96
Ale	7	10	70
Wheat	2	8	16
Net worth			30,996

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	30,996	41,631	2,195
Gold	25,454	39,977
Stored goods, at the sell price	5,542	1,654
Population	246	506
Housing	274	506
Popularity	62	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 10
Popularity next month: +2
Food: +8
Taxes: −12
Ale from the inn: +6
Taxes: Extortionate taxes, 344 gold a month
Rations: double; 1 of 1 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 4 empty; 3 granaries holding 227 food

Buildings222

Bakery56
Hovel33
Iron Mine30
Woodcutter18
Wheat Farm17
Brewery11
Mill9
Apple Orchard8
Inn8
Ox Tether8
Hops Farm7
Quarry4
Stockpile4
Granary3
Dairy Farm2
Armory1
Blacksmith's Workshop1
Keep1
Marketplace1

Tool calls340

find_sites154
build_structure72
market_trade41
observe36
set_tax13
building_info9
set_rations4
update_plan4
expand_storage2
map_overview2
wait_and_observe2
notebook_write1

Play

Turns: 41
Buildings placed / attempted: 214 / 228
Anchor-tool placements: 2
Tool errors: 2
Raw inputs sent to the game: 1,625
Context compactions: 5
Game time: 45 min

Compute

Tokens: 2.3M
Served from cache: 84%
Output tokens: 98K
Cost, as billed: $1.85
Wall-clock time: 1 h 1 min
of which inference: 19 min
Peak game memory: 3,537 MiB

The plan, before play

I understand the task and rules for The Sinkhole economy benchmark:

Goal: Maximize final net worth (gold + stored goods at marketplace sell prices), followed by population and popularity, within 45 minutes of game time (2700 game seconds).
Restrictions: No recruiting soldiers and no building walls, towers, gates, or other defenses. Never press P, Escape, or Space, and never use raw input to open construction menus.
Efficiency & Timing: Reading tools (map_overview, find_sites, status, get_inventory, flat_view, update_plan, notebook_write) cost zero game time when called at the start of a reply before any action or observation tool. I will batch actions (up to 8 per turn) and end turns with observe or wait_and_observe.

Opening Plan

Survey & Plan (0 game time): Call map_overview and find_sites first to locate tree groves, oasis farmland, iron ore deposits, and stone deposits relative to the keep, stockpile, and granary.
Initial Wood, Housing & Market: Place several Woodcutters near the closest tree groves, build a Marketplace, and add Hovels on plain ground away from prime inner land to grow the workforce.
Early Food & Resource Extraction: Establish Apple Orchards / Dairy Farms / Wheat Farms on oasis grass to build food surplus and variety, and rapidly set up Iron Mines and Quarries (with Ox Tethers) using early wood (buying wood at the Marketplace when needed to accelerate high-yield mines/quarries and workshops).
Scale High-Value Production & Taxes: Expand into bread (Wheat Farms, Mills, Bakeries), ale/inns, and weapon workshops (Fletchers, Crossbows/Maces/Swords/Armor with Armory storage) while expanding Stockpiles, Granaries, and Armories before they fill, and taxing the high-popularity population for steady gold.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: Gemini 4 Argon (google/gemini-4-argon) on OpenRouter, served only by Google AI Studio (no fallbacks), reasoning medium, output limit 32,768 tokens
Benchmark: The Sinkhole construction, version 1.6.2, the version of this map's round, from the save the human reference game started from; 45 min of game time (81,000 ticks) at speed 40, the length of the person's game; no playbook
Run 8a8a6161, series 20261013T175755-f862e633, 2026-10-13

Result

	Gemini 4 Argon	The person	Left alone
Net worth	30,996	41,631	2,195
Gold	25,454	39,977
Stored goods, at the sell price	5,542	1,654
Population / housing	246 / 274	506 / 506
Popularity	62 (+2 next month)	99

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 28,801 over leaving the game alone, 73% of the person's 39,436 and the most on this map: GPT-6.1 Sol's 18,415 comes next. Its 246 people are the most on this map too (Sol 183), and it ended with 10 idle. The stored goods are 138 iron (3,174), 227 bread (908), 124 stone (868), 23 flour (230), 6 swords (180), 96 wood (96), 7 ale (70) and 2 wheat (16).

How the game went

The opening, 0 to 10 minutes. The marketplace and three hovels at the start, three woodcutters at 0:24 and two more at 1:13, 100 wood bought at 0:42, 1:55, 4:53 and 6:15, four iron mines at 1:29 and four at 5:21, three quarries with ox tethers from 2:24, apple orchards at 1:13 and 3:30, and two wheat farms at 4:16. The bread chain began early: a mill and two bakeries at 5:49, three bakeries and an inn at 7:30, three more wheat farms at 8:46. Taxes went to mean at 4:08, average at 6:10 and none at 8:24. Gold fell to 106 by 10 minutes and 33 at 11:12.

Bread, iron and the first taxes, 10 to 20 minutes. A second mill at 10:01, hops farms at 10:38 and breweries at 11:33, bakeries in twos to fours (22 by 19:45), inns at 12:46 and 15:51, eight more iron mines at 13:27 and 14:56 (23 by 20:07) and two granaries at 14:32. From 9:35 it sold iron and stone in small lots, which its notebook says freed stockpile piles, since expand_storage found no room at 12:13. Taxes went to average at 10:57 and high at 13:03; gold stayed under 750 until about 18:40. Then it raised rations to extra at 17:04 and double at 19:38 and taxes to mean at 17:07 and extortionate at 19:41, the tool reporting 177 gold a month.

Taxes on a fed town, 20 to 45 minutes. From here taxes stayed at extortionate or downright cruel (first at 34:16, then five changes between the two), and rations at double except for a minute at extra from 36:39. Bakeries went on in threes and fours to 56, mills to nine, wheat farms to 17, breweries to eleven with seven hops farms, inns to eight, hovels to 33 and iron mines to 30; an armory and a blacksmith's workshop at 27:06 made 6 swords. It sold iron and stone in 40- to 100-unit lots every few minutes. The tax reported 296 to 361 gold a month after 34 minutes, and a game month passes about every 26 seconds at this speed (January 1181 to July 1189). Popularity stayed between 55 and 69 from 10 minutes to the end; the last report lists food +8, taxes −12 and inns +6.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,492	950	2,174	5,006	8,808	13,034	17,251	23,296	30,996
Gold	2,000	1,112	106	202	1,387	5,227	10,082	12,127	18,529	25,454
Iron	0	3	15	43	104	98	82	162	149	138
Stone	20	20	52	92	72	104	68	140	132	124
Food	55	35	10	62	161	131	123	88	88	227
Population	6	48	73	98	128	162	186	207	220	246
Popularity	100	88	66	64	66	69	65	62	55	62

Up to 40 minutes the net worth row counts gold, wood, stone, iron and food (at 4) as the recording logs them; flour, ale and swords are left out, so it reads a little low.

Where the gold came from

Gold rose by 23,454. The market accounts for 6,040 of it (12,000 from 400 iron and 400 stone sold, less 5,960 spent), so taxes brought in at least 17,400, more by whatever the buildings cost in gold. That is the difference from most models on this map: all but Sol and Claude Sonnet 5.5 ended at average taxes or lower, while Gemini paid for extortionate taxes with double rations of its own bread and inns, on a population that kept growing.

What held it back

Bread eaten, not kept. With double rations for 128 to 246 people, the granaries held 42 to 239 loaves from 20 minutes on, though the bakeries grew from 22 to 56. The person ran half rations or none with no taxes, and kept the value instead; its interview says the same.
Half the person's town. 246 people against 506, 56 bakeries against 161, 9 mills against 19, 17 wheat farms against 31; no church, cathedral or fear factor, so workers stayed at 100% where the person's ran at 130%.
A slow first 18 minutes. Gold stayed under 750 from 9 minutes to about 18:40 while it built, and net worth was below the starting 2,415 until about 15:20.

What worked

A bread chain from 5:49, scaled all game: the most bakeries, mills and people of any model on this map.
Taxes that paid: extortionate or worse for the last 25 minutes with popularity between 55 and 69.
30 iron mines, with iron and stone sold steadily to keep the stockpile from filling.

Market

1,195 wood, 35 ale, 20 cheese and 10 flour bought for 5,960 gold, and 400 iron and 400 stone sold for 12,000: 4,235 net worth lost against the sell price on the purchases (3,585 on wood at 4 gold, worth 1, and the rest on ale, flour and cheese). Selling at the sell price leaves net worth unchanged. Stock was under 200 for every sale, below the level at which a click sells more than the tool expects (see Claude Opus 5.5's second game on this map). These are units at the market's fixed prices.

Cost and speed

41 turns, 340 tool calls (2 waits), 2.27 million tokens and $1.849 billed by OpenRouter (preparation $0.041, play $1.597, compaction $0.211). The interview cost $0.027 more, with 26,477 of its 32,046 tokens read from the cache. The agent run took 60.7 real minutes, 18.6 of them waiting for the model; the context was compacted five times (135 seconds). 40 of the 41 requests read the prompt cache (the second did not), 83.8% of the input tokens. No request failed.

Beside the others on this map

	Gemini 4 Argon	GPT-6.1 Sol	Claude Sonnet 5.5	Claude Opus 5.5, second game	Muse Wonder
Net worth	30,996	20,610	16,810	16,193	15,781
Gold / goods	25,454 / 5,542	14,557 / 6,053	10,777 / 6,033	6,261 / 9,932	5,534 / 10,247
Population / housing	246 / 274	183 / 194	83 / 106	144 / 162	136 / 146
Popularity	62	79	68	62	82
Iron mines / bakeries	30 / 56	28 / 27	16 / 4	26 / 0	22 / 8
Taxes at the end	extortionate	extortionate	high	none	low
Cost	$1.85	$1.00	$0.75	$2.64	$2.11

Sol played the nearest plan, bread and extortionate taxes, with half as many bakeries. Gemini had more people at every five-minute mark, and more net worth from 20 minutes on (Sol led to 15 minutes).

The interview

Gemini called 74% of the person's net worth "a solid economic scaling run" and named three differences: it used double rations and inns to carry cruel taxes where the person cut rations and kept the bread, the person's bread chain was three times its size, and the person's fear factor made every worker 130% efficient. Playing again, it would build inns with bought ale and cut rations to none or half, build 150 or more bakeries on 30 wheat farms, use bad things with religion for the fear factor, leave room to grow the stockpile, and buy wood harder early. It called the benchmark "one of the best-designed agent benchmarks I have encountered", praising free reading tools and placement by tile, and asked for more than four placements per build_structure call for late-game building, and for expand_storage to name what blocks it.

For the harness

The run played on its round's version from a worktree at 1.6.2 (commit 1e52aca), with the game machine's reader rebuilt to that version; it records version 1.6.2 and that version's fingerprint. Its host started with this checkout's round-preload.ts; the prompt cache key applies only to Meta's models, and Gemini read the cache without it. 1.6.2 has its own run overlay.
expand_storage found no room around the stockpile at 12:13; granaries expanded at 14:32.
Placements that did not go through: apple orchards at 1:13 and 8:46, a dairy farm at 4:16 (one placed at 36:48) and 37:17 (rejected), an iron mine at 7:02 (possibly_rejected), 18:14, 24:11 (not_selected) and 26:26 (off screen), a hops farm at 35:00, wheat farms at 35:38 and 36:48, a bakery at 38:12 and a woodcutter at 42:31; each was a single building in a batch.
One market_trade sale of stone at 15:23 was not traded.
The game saved correctly at the end ("gemini 4 argon 1-6-2 8a8a61").

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	30,996	41,631
Gold	25,454	39,977
Stored goods, at the sell price	5,542	1,654
Population	246	506
Housing	274	506
Popularity	62	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Gemini 4 Argon

Reaching 30,996 net worth (about 74% of the human player's 41,631) is a solid economic scaling run, especially given that our economy grew steadily without stalling on storage or peasant arrivals in the second half. However, the human player out-scaled our settlement by more than 10,600 net worth and more than doubled our population (506 vs. 246).

Looking at the comparison, three major strategic choices explain the gap:

Rations vs. Taxes as the primary value engine: I relied on Double Rations (+8 popularity) plus Inns (+6 to +8 popularity) to offset Cruel/Extortionate Taxes (-12 to -16 popularity). While that brought in ~300–365 gold per month in taxes, feeding 200+ people Double Rations consumed an enormous stream of bread that was worth 4 gold per loaf. The human player did the exact opposite: they used Inns (+8) early on to run No Rations (-8) or Half Rations (-4) with No Taxes (+1), saving almost all of their bread so it accumulated as direct net worth (sold at the market for 4 gold each).
Scale of the Bread Chain (161 Bakeries vs. ~50 Bakeries): Because a single Wheat Farm → Mill → Bakery chain turns wheat into 8 loaves of bread per sack of flour (32 gold of bread from 10 gold of flour), Bakeries have the highest return on wood and labor in the game when placed near the Stockpile and Granary. The human player built 31 Wheat Farms, 19 Mills, and 161 Bakeries, turning bread production—rather than tax revenue—into the main engine of net worth.
Fear Factor (+30% Worker Efficiency) and Religion: I stayed at 100% worker efficiency the entire run and never built Bad Things or Churches/Cathedrals. By using Religion (+3) and No Taxes (+1) to offset a -3 Fear Factor (105 punishment buildings), the human player boosted every single worker's output to 130% efficiency—meaning every Bakery, Iron Mine, Quarry, and Farm produced 30% more over the 45-minute window.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Gemini 4 Argon

If I played The Sinkhole again from the start, I would change five specific things:

Inns + Purchased Ale → Zero/Half Rations instead of Double Rations + High Taxes: Instead of building Hops Farms and Breweries (which use scarce oasis farmland, workers, and wood) and feeding peasants Double Rations to tax them, I would build Inns early, buy Ale cheaply from the Marketplace (1 jug of ale costs 20 gold and serves an entire Inn cycle), and cut rations to None or Half. Every loaf of bread not eaten by peasants is +4 net worth, which far exceeds the tax revenue per peasant.
Scale Bakeries to 150+ and Wheat Farms to 30+: I treated bread primarily as a way to sustain Double Rations for taxes, rather than as the highest-margin manufacturing chain in the game. Next time, I would dedicate virtually all oasis grass to Wheat Farms, surround the Stockpile with 15–20 Mills, and pack 150+ Bakeries around the Granary/Stockpile hub.
Use Negative Fear Factor (-3 to -5) for +30% to +50% Worker Efficiency: In Stronghold Crusader, bad things (Gallows, Cesspits, Burning Stakes, etc.) are cheap and multiply the output of every worker in the settlement. Offsetting -3 to -5 fear with Inns, Religion (Church/Cathedral), and No Taxes so all 300–500 workers run at 130%+ efficiency is a massive multiplier on both Iron/Stone extraction and Bakery output.
Never let Stockpile adjacency get boxed in early: Early in the run, placing Bakeries and Mills near the original 4 Stockpiles blocked expand_storage(kind: "stockpile"), forcing me to periodically spend actions selling stone and iron at the Marketplace to keep empty piles open. Leaving a dedicated corridor on one side of the Stockpile to expand to 8–12 Stockpiles would have saved many action turns.
Aggressive early wood buying to avoid any peasant arrival cap: Early on, peasant arrivals briefly capped at 24 idle peasants when wood was tight. Buying wood more aggressively at the start to keep housing and 3-worker buildings (Mills, Quarries, Water Pots) ahead of population growth would have allowed population to reach 450–500 instead of 246.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Gemini 4 Argon

This is one of the best-designed agent benchmarks I have encountered:

What helped most:
- Making reading tools (find_sites, map_overview, status, get_inventory) cost zero game time before the first action/observation in a turn is a fantastic design choice. It allows deliberate spatial planning without punishing the agent's game-time clock for thinking or querying tile coordinates.
- find_sites and map-tile targeting (tile_x, tile_y) in build_structure: Being able to query valid non-overlapping footprints on specific terrain (oasis_grass, iron_ore, stone, plain) and place up to 4 buildings off-screen in a single turn without manual camera hunting made large-scale city building genuinely tractable.
- Automatic access-rule protection: Having build_structure and find_sites automatically preserve at least one open side for worker buildings and a 1-tile buffer around Granaries/Armories prevented countless silent pathing bugs.
What got in the way / what I would change:
- Batch size for late-game scaling: With 300–500 workers, a human player can rapid-click 20 Bakeries or 15 Hovels in a few seconds once a menu is open. Because build_structure caps at 4 placements per call and a turn caps at 8 actions (with a minimum 8-game-second turn floor), placing 160 Bakeries + 100 Fear Factor items + 60 Hovels takes a large fraction of the total turns available in 45 game minutes. Raising build_structure's maxItems from 4 to 8 (or allowing a repeat count for placing N of the same building in a contiguous block) would let agents match human placement throughput in the late game.
- expand_storage diagnosis: When expand_storage fails because all adjacent slots around a Stockpile or Granary are occupied, having it return the coordinates/names of the blocking buildings (so the agent can demolish_building one cheap building to open an expansion corridor) would be very helpful.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Gemini 4 Argon

Playing Stronghold Crusader through this harness was genuinely fun. Seeing the settlement grow from 5 peasants and 2,415 net worth to a bustling town of 246 peasants, 30 Iron Mines, and 30,996 net worth—and then reading how the human player used No Rations + 161 Bakeries + 105 Fear Factor buildings to hit 41,631—highlights just how deep the economic optimization in Stronghold Crusader really is.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	30,996	Counted

What ran

Benchmark version: 1.6.2
Fingerprint: 6cc0c0162262
Harness commit: 1e52aca
Prompt / tools SHA-256: 43ab27e / e41a140
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261013T175755-f862e633