One episode · The Sinkhole (no arms)

DeepSeek V4.1 Flash 1017

Benchmark 1.7.3No arms: weapons and armour count nothingDeepSeekReasoning mediumPlayed by the maintainersFull-budget score

All of DeepSeek V4.1 Flash 1017’s results and comparisons

Score

25.7

Crusader Bench score on The Sinkhole (no arms): 0 is doing nothing, 100 the person’s game

Net worth

9,050

+6,855 over doing nothing, 17% of the person’s gain of 39,436

Population

148

29% of the person’s gain of 502 · popularity 89

Cost

$0.11

2.6M tokens

Time

45 game min

48 min of wall-clock time, 54 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

DeepSeek V4.1 Flash 1017
Baseline (do nothing), final
A person's game, final

Net Worth

gold plus stored goods at the sell price

Gold or Goods

Gold
Goods

Population

peasants

Popularity

0–100

Food in the Granary

units

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

9,050net worth

Gold: 1,841
Stored goods, at sell price: 7,209
Over the baseline (do nothing): +6,855
Population / housing: 148 / 194
Popularity: 89
Food: 685
Structures (map-wide): 174
Attempt: 1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock7,209 gold in goods

Good	Units	Price	Value
Gold	1,841	1	1,841
Iron	187	23	4,301
Apples	683	4	2,732
Wood	168	1	168
Cheese	2	4	8
Net worth			9,050

No arms: weapons and armour count 0 here, so the table adds up to the net worth.

Beside a person’s gameThe Sinkhole, the same game time

	This episode	The person	Left alone
Net worth	9,050	41,631	2,195
Gold	1,841	39,977
Stored goods, at the sell price	7,209	1,654
Population	148	506
Housing	194	506
Popularity	89	99

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game: July 1189
Idle peasants: 24
Popularity next month: −5
Food: +1
Taxes: −6
Taxes: High taxes, 148 gold a month
Rations: full; 2 of 2 foods eaten
Worker efficiency: 100%
Storage: 16 stockpile piles, 8 empty; 4 granaries holding 685 food

Buildings138

Apple Orchard63
Hovel23
Iron Mine20
Woodcutter17
Granary4
Stockpile4
Dairy Farm3
Armory1
Blacksmith's Workshop1
Keep1
Marketplace1

Tool calls177

find_sites113
build_structure20
building_info18
market_trade14
set_tax4
map_overview2
expand_storage1
get_inventory1
inspect_building1
place_near1
status1
wait_and_observe1

Play

Turns: 54
Buildings placed / attempted: 126 / 135
Anchor-tool placements: 6
Tool errors: 1
Raw inputs sent to the game: 934
Context compactions: 1
Game time: 45 min

Compute

Tokens: 2.6M
Served from cache: 95%
Output tokens: 48K
Cost, as billed: $0.11
Wall-clock time: 48 min
of which inference: 8 min
Peak game memory: 3,526 MiB

The plan, before play

Task understanding: Play The Sinkhole free-build save to maximize final net worth (gold + stored goods at sell prices), with population and popularity as secondary scores, over 45 game minutes. No soldiers, walls, towers, gates or other defences. The granary and stockpiles already exist; I start with ~2,000 gold, 55 wood, 20 stone, 55 meat and 5 peasants in 10 housing.

Opening plan:

Observe the starting view, check status/inventory and popularity, then place woodcutters by the nearest grove and add hovels for housing.
Build food production early (orchards/dairy on oasis grass, plus wheat→mill→bakery if good grass is available) so popularity and population can grow.
Add raw-material income: iron mines on the ore patches (iron is worth 23/unit) and a quarry with ox tether on stone, expanding stockpiles as needed.
Use a marketplace to sell surplus goods, convert spare wood into faster construction, and later add an armory and weapon workshops if iron/wood surpluses justify it (weapons only gain value when sold).
Keep popularity above 50, add housing continuously, raise taxes moderately once population is secure, and keep expanding until the final minute. Check production and storage bottlenecks with status/inventory/inspect rather than idling.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (drafted by Sonnet 5.5, checked by Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

Model: DeepSeek V4.1 Flash 1017 (deepseek/deepseek-v4.1-flash-1017) on OpenRouter, served by Fireworks with Together as the only fallback (DeepSeek's own endpoint is excluded by the account's data policy), reasoning medium, output limit 32,768 tokens. Fireworks served all 56 recorded requests (1 preparation, 54 gameplay, 1 compaction); Together served none.
Benchmark: The Sinkhole construction (no arms), version 1.7.3 (commit e0e0eb4, 1.7.3 relocked with a fix to the compaction retry; play and scoring unchanged); 45 min of game time at speed 40; no playbook. Weapons and armour in store count nothing; sold, they bring their gold. Host on the game machine.
Run 82cf4216, series 20261019T194658-d4ba279b, 2026-10-19: this model's no-arms Sinkhole game and the last of its six. Its weapons-counted pair is run fb7aed77 (1.7.1, commit 07f2b7a, played just before, host also on the game machine); its others, all 1.7.3: Oasis 26.5 and 28.0, desert 0.7 and 78.5. This game counts on the website's leaderboard, with the other five.

Result

	This game (1.7.3, no arms)	Pair (1.7.1, weapons counted)	The person	Left alone
Net worth	9,050	11,345	41,631	2,195
Population / housing	148 / 194	28 / 90	506 / 506	4
Popularity	89 (−5 next month)	19 (−1)	99	0
Score	25.7	20.9	100	0

A full-budget score: all 2,700 game seconds. Net worth grew 6,855 over leaving the game alone, 17.4% of the person's 39,436. By the website's formula (100 × (0.8 × net-worth share + 0.1 × population share + 0.1 × popularity ÷ 100); shares from leaving the game alone (2,195, 4 people) to the person (41,631, 506), net-worth share clipped to −1..2) the score is 25.7: 13.91 from net worth, 2.87 from 148 people, 8.90 from popularity 89. The map's website score is the mean of both games, 23.3. It beats its pair by 4.7 on 2,295 less net worth, but the pair differs in more than weapons (1.7.1 against 1.7.3, with 1.7.2's placement and stockpile changes between). Stored goods hold no weapons (the pair held 195): 187 iron (4,301) is 48% of net worth, 683 apples (2,732) 30%, gold 20%.

How the game went

Times are game time, when a call began (the recording's clock; the run clock in the observations reads a few seconds later).

Iron and apples, 0:01 to 5:41. After 23 free reads it chose iron ("Iron mines are extremely valuable: iron 23 each") and apples for food ("Apples simplest"). A 104 s call placed a mine, two woodcutters, an orchard and three hovels; at 3:36, with 23 of 29 people idle, four mines, three orchards and a dairy.

Taxes and mines, 5:48 to 17:28. At 8:39 it set level 8 ("level 8 (Extortionate -12) -> 88"); popularity fell from 100 to 38 by 11:13. A bribe from 11:07 ran to 17:18. Four more calls added 15 mines, 20 by 17:12; gold hit 206 at 17:20. Armory and smith followed (18:27).

Orchards, 21:41 to 38:04. With 23 idle and the smith at 0.3 swords a minute, it priced an orchard at 8.8 gold a minute for 5 wood and one job ("farms are extremely good net worth, not just food") and placed 6 to 8 a call, seven calls to 36:02: 12 orchards stood at 21:40, 61 at 38:09, and population rose one for one (82 to 131). It compacted once (23:29) and added three granaries (28:23).

Land runs out, 38:11 to 45:00. From 36:02, 16 of 25 orchard searches found nothing. After two last orchards came woodcutters (17 in all), three hovels, 11 swords sold and level 6 at 44:05.

Game minute	0	5	10	15	20	25	30	35	40	End
Net worth	2,415	1,793	1,990	2,112	1,711	2,438	3,322	4,500	6,318	9,050
Food	55	28	41	79	81	87	108	216	434	685
Population	6	34	46	62	79	89	107	120	134	148
Popularity	100	100	71	99	100	100	100	100	100	89

Each column is the last recording frame at or before that minute. Net worth sat near or under the 2,195 of leaving the game alone from 2:35 to 22:56 (low 1,548, 19:01).

Weapons

It did not leave weapons alone. Its pre-play plan allowed workshops "if iron/wood surpluses justify it (weapons only gain value when sold)"; at 13:13 it noted "armory storage counts 0 but must sell"; at 17:18 it expected a smith to pay back in "under 1 min" and placed one (20 wood and 200 gold by prompt/game-mechanics.md) with an armory (5 wood). The readouts gave 0.3 to 0.7 swords a minute; at 21:41 it wrote "Bad. We should not build more blacksmiths". The smith made 11 swords from 11 iron. The first sale was at 40:53 ("Critical: weapons/armor count nothing in net worth. We have 9 swords = 0 net worth"): 9, then 1 and 1, 330 gold in 39 game seconds; the armory ended empty. Against keeping the iron (253) the swords added 77, but the wing cost 200 gold and 25 wood: 148 behind (0.3 points), 223 with wood at the 4 it paid.

Beside its other games and the other models

	Net worth	Score	People / popularity	Key buildings	Turns / cost
Oasis, weapons (1.7.3)	5,598	26.5	74 / 38	11 mines, 5 workshops	43 / $0.134
Oasis, no arms (1.7.3)	6,327	28.0	98 / 9	34 woodcutters, 7 workshops	54 / $0.145
Desert, weapons (1.7.3)	4,874	0.7	58 / 38	11 workshops, 4 mines	29 / $0.099
Desert, no arms (1.7.3)	9,384	78.5	90 / 50	10 quarries, 41 tethers	36 / $0.142
Sinkhole, weapons (1.7.1)	11,345	20.9	28 / 19	21 mines, 10 wheat farms	46 / $0.139
Sinkhole, no arms (1.7.3)	9,050	25.7	148 / 89	63 orchards, 20 mines	54 / $0.111

Map means: Oasis 27.2, desert 39.6, Sinkhole 23.3; overall 30.0, sixth of the ten listed models, between Muse Wonder (31.1) and Claude Fable 5.1 (26.0).

Sinkhole, no arms	Net worth	Score	People / popularity	Orchards
GPT-6.1 Sol	20,113	49.2	146 / 100	2
Gemini 4 Argon	20,104	47.1	181 / 72	2
Claude Fable 5.1	18,294	41.8	124 / 68	24
Claude Opus 5.5	16,889	39.3	146 / 67	19
Claude Sonnet 5.5	15,246	35.3	106 / 68	7
GPT-6 Luna	13,769	31.5	119 / 57	16
Muse Wonder	12,364	30.2	106 / 75	4
DeepSeek V4.1 Flash 1017	9,050	25.7	148 / 89	63
Claude Haiku 5.5	8,243	23.8	82 / 100	11
GLM 5.3 Flash	8,272	23.6	143 / 85	5

Orchards are those standing at the end. This game is eighth by score and net worth, 4.5 behind Muse Wonder and 1.9 ahead of Haiku. Its population is second to Argon's, popularity third, gold the lowest; population and popularity are 46% of its score.

What held it back

Jobs set the population. 148 people were 124 jobs (63 orchards, 3 dairies, 40 for the mines, 17 woodcutters, 1 smith) plus the 24 idle the benchmark names; 46 places stood empty.
The tax episode. Level 8 brought 334 gold over six paydays (26.7 s apart); the bribe cost 748 over
1. Popularity sat under 50 for 73 s and 4 people left; the last six bribe paydays, at 99 to 100, cost 387. At 11:07 it put a month at 15 game seconds.
The last tax step. Level 6 at 44:05 ("Risk minimal since popularity 100") paid 259 gold (0.5 points) and cost 11 popularity (1.1): 26.2 without it.
One crop on ten-tile plots. The readout's per-orchard figure was 2.2 to 3.1 apples a minute with 7 to 12 orchards and 0.9 to 1.5 with 33 to 61; at 30:30 it wrote "dropped to 1/min due to distance maybe" and placed 24 more. About 1,600 apples came in (estimated from the readouts); late on the town ate about 40 food a minute against 95 delivered, and 683 stood unsold.
No second chain. It set the wheat chain aside at 0:01 ("Yet apple/dairy simpler") and 23:32 ("Chain may or may not add"); no mill, bakery, inn, church or quarry stood.

What worked

Iron early. 20 mines by 17:12 (the pair: 21 by 33:28, per its analysis) mined about 300 iron; 187 held, 102 sold.
Jobs and popularity. 63 orchards at 5 wood each made 63 jobs; people went from 69 (17:20) to 148, with popularity at 99 or 100 from 15:00 to 44:04 and no church or inn.
Storage and placement. Three granaries (15 wood) held the 435 food above one granary's 250; 126 of 135 placements held and all 132 buildings stand: no demolitions.

Market

14 calls took 447 game seconds (16.6%). Bought: planks, 8 calls, 655 units for 2,620 gold (replies: 650, 2,600), 1,965 of net worth (4.0 points). Sold: 20 stone (8:54), 102 iron (17:28, 23:32) and 11 swords (40:53, 42:19, 43:54): 2,816 gold, net worth unchanged but for the swords (+330). Against the recording (recording/frames.jsonl and inputs.jsonl) every sale matches, none starting above 54 in stock, so the 50-a-click undercount did not arise; seven of eight purchases match click for click. The 13:13 purchase replied 10 clicks, 50 units, −200 but 11 landed: gold −220, 55 units. Gold reconciles: 2,000 + 593 taxes − 748 bribe + 2,816 sales − 2,620 planks − 200 smith = 1,841.

Cost and speed

54 turns, 177 tool calls (113 find_sites, 20 build_structure, 14 market_trade, 30 others), 2.60 million tokens, $0.111 billed (play $0.102); the interview $0.002 more. 48.0 real minutes, 7.8 waiting for the model; 54 of 54 requests read the cache (95.5%). find_sites took no game time. Of 2,700 game seconds build_structure took 1,941 (71.9%), market_trade 447, the 26 gaps between an acting call and the next reply 152. It placed 126 buildings in 1,941 s: 15.4 s each, against 15.4 for its Sinkhole pair (1.7.1, 71 in 1,096 s), 11.8 and 10.2 on Oasis, 11.9 and 11.2 on the desert (1.7.3), all with the host on the game machine; maps differ, so no general rate.

The interview

It put the gap down to allocation: 63 orchards ("about 1.4 apples/min each") on 10×10 plots against the person's wheat, mills and 161 bakeries; "roughly 124 jobs"; taxes at none until the last minute. Playing again it would start with a wheat, mill and bakery chain; it named anchoring on apples its main error. These match the game: 1.4 apples a minute (readouts, 42:17 to 44:19), plot size 10, 148 / 194 / 24 idle / 124 jobs, popularity at 100. These do not:

"Raised them only in the last minute, so that produced no income": level 8 and a bribe ran between 8:39 and 17:18, and level 6 paid 259 gold.
"Measured progress by number of farms": it priced per-worker values at 21:41 (8.8) and 36:02 (5.2).

For the harness

find_sites listed two tiles build_structure could not reach: it offered (526,407) at 28:23 and (518,425) at 38:11; the builds at 28:54 and 38:11 returned not_on_screen.
Placement on 1.7.3. Of 135 requests, 7 were not_placed (all retrySkipped: no_free_spot_nearby, remedy find_sites or go_to_tile, no nearestFit) and 2 not_on_screen; 5 moved 10 or 11 tiles, inside an orchard's reach of 11. Five of the six orchard misses lay 6 to 10 tiles from an orchard just placed, four listed by a different search. The 1.7.1 pair had 5 of 76 not placed with the same fields; the logs cannot say what its 12-tile retry would have done here.
market_trade counted a purchase one click short at 13:13 (a click it did not see within 5 seconds), as in its Sinkhole pair (cheese, 32:27) and its desert weapons game (meat, 14:22).

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

	You	The person
Game time played	45 min	44 min 58 s
Net worth	9,050	41,631
Gold	1,841	39,977
Stored goods, at the sell price (weapons and armour not counted)	7,209	1,654
Population	148	506
Housing	194	506
Popularity	89	99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

DeepSeek V4.1 Flash 1017

I finished at 9,050 net worth versus the person's 41,631, so I was roughly a fifth of the way. The gap was not mainly raw production capacity but allocation of workers and land. I built 63 apple orchards, which at about 1.4 apples/min each tied up enormous 10×10 oasis footprints and one worker per orchard for only a few units of value per minute. The other player instead used the same scarce oasis for wheat, mills and especially 161 bakeries, which turns a small amount of farmland into a large, continuously sold food output with many processing jobs. My 20 iron mines produced steadily but slowly, while they had 38 iron mines, 10 quarries and 29 ox tethers. Most importantly, I never converted population into jobs fast enough: I finished with 148 people in 194 housing and 24 idle, because I only had roughly 124 jobs. Water pots, bakeries and inns would have employed many more people and raised both population and output. I also left taxes at "none" for almost the whole run and then raised them only in the last minute, so that produced no income. I sold my handful of swords, which was correct, but the armory value was trivial. My late woodcutter spam made wood but was far too small in value. The person's mix of popularity from inns, religion and fear, plus selling bread instead of eating it, multiplied the same land and population far more effectively.

2Playing again

Interviewer

How the person played, as they describe it:

Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
16 inns, with ale in the stockpile and no brewery or hops farm.
38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
9 blacksmiths, an armory holding 1 sword, and 24 water pots.
A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

DeepSeek V4.1 Flash 1017

I would not build a large orchard monoculture. I would do roughly this:

Put the first oasis land into a compact wheat → mill → bakery chain and replicate bakeries heavily, because the processing step creates jobs and sellable bread from cheap farmland.
Build several inns early and buy ale to hold popularity high enough to cut rations, then sell the food that would otherwise be eaten.
Add iron mines and quarries steadily, with ox tethers at the quarries, and use blacksmiths only as a minor iron sink.
Use water pots and other job buildings deliberately to employ the population, and add hovels continuously so housing never caps growth.
Add a church/cathedral and enough fear-factor buildings to raise worker efficiency, since the person got 130% efficiency from it.
Keep taxes low or none while popularity is needed for immigration, rather than trying to tax late.
Treat the marketplace as a continuous output channel for bread and surplus goods, not a place for occasional last-minute trades.

The biggest lesson is that jobs and land efficiency matter more than building count. I measured progress by "number of farms" rather than by net value produced per worker and per oasis tile.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

DeepSeek V4.1 Flash 1017

The tools were generally clear and the read-only status/inventory/production reports were the most useful part of the run. Free reads made it possible to plan between actions. What slowed me down was the real cost of market trades: a 20-lot trade took about a game minute of clicking, which is very expensive relative to a 45-minute run, so I under-traded. Multi-building placement also cost a lot because tile targets moved the camera and each placement opened and closed menus; several build replies consumed nearly two game minutes for eight placements. I would appreciate more visibility of live building costs and input/output ratios in the tool results, because the historical manual values were not enough to judge whether, for example, a bakery chain was worth more than an orchard. The time limit itself is fine and makes the prioritization interesting. A useful addition would be a compact per-building "value added per worker per minute" estimate, or at least production and consumption together, so that the player can see bottlenecks without surveying.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

DeepSeek V4.1 Flash 1017

I think my main strategic error was anchoring on apples because they were cheap and directly delivered to the granary. That made the early game look efficient, but it ignored the much larger multiplier available from processing and from employing a big population. I also underused popularity: I kept it at 100 but did not convert that into either higher taxes or lower rations, so it was mostly a wasted resource. The run was still informative: the production reports made the orchard inefficiency visible in hindsight, and the person's build shows how much headroom there was on this map.

Record

Attempts and Reproducibility

Every attempt1

Episode	Attempt	Outcome	Net worth	Counted
1	1	Full-budget score	9,050	Counted

What ran

Benchmark: The Sinkhole construction (no arms)
Benchmark version: 1.7.3
Fingerprint: b7621690f14b
Harness commit: e0e0eb4
Prompt / tools SHA-256: 7ed945e / f75ebb8
Menu guide: Images 118089d
Save: The Sinkhole start
Game time per episode: 45 min
Playbook between episodes: No
Minimum turn / default wait: 8 s / 5 s
Observation after acting: Text unless a screenshot is needed
Screenshot width: 1,280 px
Context budget: 120,000 tokens
Real-time limit: 360 min
Submitted by: hesamation

All files in the dataset 20261019T194658-d4ba279b