Crusader Bench

One episode · The Sinkhole

GLM 5.3 Flash

Benchmark 1.6.2Z.aiReasoning mediumPlayed by the maintainersFull-budget score

All of GLM 5.3 Flash’s results and comparisons

Model
z-ai/glm-5.3-flash
Served via
OpenRouter · z-ai/fp8, siliconflow/fp8
Max output
32,768 tokens
Played
12 October 2026
Score
15.2
Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game
Net worth
5,084
+2,889 over doing nothing, 7% of the person’s gain of 39,436
Population
78
15% of the person’s gain of 502 · popularity 79
Cost
$0.11
3.2M tokens
Time
45 game min
47 min of wall-clock time, 68 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

  • GLM 5.3 Flash
  • Baseline (do nothing), final
  • A person's game, final

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Videos and images stream from the dataset. Game footage © Firefly Studios, shared for non-commercial research.

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

5,084net worth
Gold
1,992
Stored goods, at sell price
3,092
Over the baseline (do nothing)
+2,889
Population / housing
78 / 114
Popularity
79
Food
23
Structures (map-wide)
103
Attempt
1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock3,092 gold in goods

GoodUnitsPriceValue
Gold1,99211,992
Iron48231,104
Stone927644
Bows1715255
Flour2210220
Crossbows630180
Wood1531153
Spears1210120
Swords430120
Apples23492
Metal armour33090
Leather armour61272
Pikes11818
Hops2816
Wheat188
Net worth5,084

Beside a person’s gameThe Sinkhole, the same game time

This episodeThe personLeft alone
Net worth5,08441,6312,195
Gold1,99239,977
Stored goods, at the sell price3,0921,654
Population78506
Housing114506
Popularity7999

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game
July 1189
Idle peasants
24
Popularity next month
±0
Taxes
−2
Ale from the inn
+2
Taxes
Low taxes, 46 gold a month
Rations
full; 1 of 1 foods eaten
Worker efficiency
100%
Storage
16 stockpile piles, 6 empty; 1 granary holding 23 food

Buildings65

  • Hovel13
  • Fletchers' Workshop7
  • Woodcutter7
  • Apple Orchard5
  • Dairy Farm4
  • Iron Mine4
  • Stockpile4
  • Blacksmith's Workshop3
  • Poleturner's Workshop3
  • Quarry2
  • Armorer's Workshop1
  • Armory1
  • Bakery1
  • Brewery1
  • Granary1
  • Hops Farm1
  • Inn1
  • Keep1
  • Marketplace1
  • Mill1
  • Ox Tether1
  • Tanner's Workshop1
  • Wheat Farm1

Tool calls118

  • find_sites31
  • build_structure22
  • market_trade22
  • wait_and_observe12
  • set_workshop8
  • set_tax7
  • building_info5
  • place_near3
  • observe2
  • set_rations2
  • expand_storage1
  • get_inventory1
  • map_overview1
  • status1

Play

Turns
68
Buildings placed / attempted
57 / 62
Anchor-tool placements
2
Tool errors
1
Raw inputs sent to the game
509
Context compactions
1
Game time
45 min

Compute

Tokens
3.2M
Served from cache
97%
Output tokens
13K
Cost, as billed
$0.11
Wall-clock time
47 min
of which inference
11 min
Peak game memory
3,269 MiB

The plan, before play

Understanding: In the 45-minute timed play on The Sinkhole, I must grow net worth (gold + stored goods at sell prices), with population and popularity secondary. Buildings don't count directly — only what they produce and store. No military building. Popularity above 50 + hovels = workers; taxes turn popularity into gold.

Opening plan:

  1. Read map (map_overview, status, save a keep view) and note terrain: oasis ring for farms, iron/stone ~50-90 tiles east/right, trees ~40 tiles down-left.
  2. Early wood: 2 woodcutters near the grove; 2 hovels near keep for housing.
  3. Marketplace early so I can trade and sell surplus.
  4. Food: 2–3 apple orchards/dairy on oasis grass near the granary.
  5. Once wood flows: more hovels, wheat→mill→bakery chain, then iron mines (iron = high value, 23/unit) and a stone quarry + ox tethers.
  6. Weapons chain: armory + workshops (bows from wood, swords/metal armor from iron) to convert cheap resources into high-value armory stock.
  7. Raise taxes as popularity allows (food variety, religion, ale later), keep rations at least full, keep storage from filling (expand stockpile, sell surplus).
  8. Keep building continuously; re-check inventory and buildings, fix bottlenecks.

Commentary during play2

  1. I'll survey the map first (free, game is still paused during reads).

  2. Time is essentially up (10 seconds left). Final state: gold 1,946, population 78/114, popularity 79, a working economy with 7+ woodcutters, 4 iron mines, 2 quarries, food farms, bread/ale chains, and a full weapon-production pipeline (fletchers on crossbows, poleturners on pikes, 2 blacksmiths, armorer, tanner) feeding the marketplace sales — the run ends here.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

  • Model: GLM 5.3 Flash on OpenRouter, providers z-ai/fp8 then siliconflow/fp8, reasoning medium, output limit 32,768 tokens
  • Benchmark: The Sinkhole construction, version 1.6.2, from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook
  • Run 2452ade8, series 20261012T233119-e8fb78d0, 2026-10-12. The first valid run on this map, so it set the map's round to 1.6.2

Result

GLM 5.3 Flash The person Left alone
Net worth 5,084 41,631 2,195
Gold 1,992 39,977
Stored goods, at the sell price 3,092 1,654
Population / housing 78 / 114 506 / 506 4
Popularity 79 99 0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 2,889 over leaving the game alone, about 7% of the person's growth. Of the stored value, 1,104 is 48 iron from four mines, 855 is 49 weapons and armour in the armory, 644 is 92 stone from two quarries, 220 is flour and 153 is wood. The town ended with 24 idle peasants and 36 empty places in its hovels.

How the game went

The opening, 0 to 8 minutes. Two woodcutters and two hovels in the first turn, then a marketplace, apple orchards and dairy farms, two iron mines at 2:38 and two quarries with an ox tether at 4:42, with 100 wood bought. At 3:19 taxes went to high, and popularity fell from 100 to 50 by 10 minutes. Taxes came down a step at a time and were at none by 10:33, but the town had shrunk from 46 to 33 by 12 minutes.

Weapons, 8 to 22 minutes. An armory and a fletcher's workshop at 7:59, then at 11:53 another fletcher, a poleturner and a blacksmith, at 15:36 an armourer and a tanner, and at 18:27 two more iron mines, a fletcher and a poleturner. It set its fletchers to crossbows and a poleturner to pikes with set_workshop. Gold fell to 529 by 20 minutes, and net worth to 1,755.

Rations and recovery, 22 to 32 minutes. At 22:23 it bought 40 apples, put rations on half and raised taxes to low, and began selling weapons to stay in gold. Popularity fell from 80 to 43 by 27 minutes and the town lost ten people; at 26:39 rations went back to full and taxes to none. A wheat farm and a mill came at 19:44 (the bakery placed with them was not built), a bakery at 27:13, and a hops farm, a brewery and an inn at 27:37.

Selling weapons, 32 to 45 minutes. Four more fletchers and poleturners at 35:26 and two blacksmiths at 38:43, low taxes from 34:32, and a sale of weapons every few minutes to keep room in the armory, which reached 48 of its 50 places. Net worth rose from 2,680 at 30 minutes to 5,084, the last 1,350 of it in the final five minutes as gold, wood and weapons came in.

Game minute 0 5 10 15 20 25 30 35 40 End
Net worth 2,415 1,969 2,043 1,858 1,755 2,016 2,680 3,540 3,729 5,084
Gold 2,000 1,664 1,680 929 529 395 888 888 1,190 1,992
Iron 0 0 5 14 16 19 28 37 46 48
Weapons and armour 0 0 0 4 17 25 20 45 27 49
Population 5 34 46 42 59 65 55 73 78 78
Popularity 100 87 50 59 78 76 49 74 71 79

During long waits a row repeats the last reading.

What held it back

  • Weapons sold at the price they are counted at. It made about 127 weapons and armour and sold 78 of them for 1,619 gold, which changes net worth by nothing; the market spending, 175 wood and 120 apples for 1,660 gold, cost 1,005 net worth against the sell price. The weapons chain added value only through the iron mined and the 49 pieces left in the armory.
  • Taxes and rations that cost people. High taxes at 3:19 and half rations at 22:23 each sent popularity below 50, and the town lost people both times, 13 and then 10.
  • A small town. 78 people at the end, with 24 idle and room for 36 more: the town stopped growing from about 35 minutes. GLM named this itself in its interview.

What worked

  • A complete weapons economy from its own iron and wood, with all four workshop kinds and an armourer and tanner, run with the new set_workshop tool.
  • It kept the armory from filling by selling, so the 50-place limit never stopped its workshops.
  • Every reported placement matches the game's final count of its buildings; the one bakery reported as unverified was indeed not built.

Cost and speed

68 turns, 118 tool calls (12 waits), 3.21 million tokens and $0.114 (preparation $0.003, play $0.108, compaction $0.003), worked out from the list prices on the model profile. The interview cost $0.002 more, with 40,512 of its 42,366 tokens read from the cache. The agent run took 46.7 real minutes, 10.6 of them waiting for the model; the context was compacted once (24 seconds).

Beside the others on 1.6.2

Scores by the project website's formula (80% net worth, 10% each population and popularity, each from doing nothing at 0 to the person's game at 1):

Net worth Gold / goods Population Popularity Score Cost
GPT-6.1 Sol 20,610 14,557 / 6,053 183 79 48.8 $0.995
Claude Sonnet 5.5 16,810 10,777 / 6,033 83 68 38.0 $0.752
Claude Opus 5.5 13,921 1,157 / 12,764 147 48 31.4 $2.470
GPT-6 Luna 11,177 1,764 / 9,413 108 52 25.5 $0.138
Kimi K3 9,164 2,356 / 6,808 91 52 21.1 $1.662
GLM 5.3 Flash 5,084 1,992 / 3,092 78 79 15.2 $0.114
Claude Haiku 5.5 3,312 1,716 / 1,596 37 51 8.0 $0.055

The interview

GLM called the gap "enormous" and said it "never found a scalable export": it "treated stored goods as the score" and sold weapons "at roughly break-even". It said it dismissed the fear factor, kept full rations "out of fear of a popularity crash", and was "repeatedly capped by the 24-idle immigration block". Playing again, it would build the bread chain at scale first, use inns and fear, and give idle peasants jobs. It played "like a city-builder" where the person played "like a factory". It asked for batched market trades and for the stats to show when immigration is blocked by idle peasants.

For the harness

  • set_workshop by tile opened a building drawn in front twice (the Armory and a Hovel, 14:39); the reply asked for a pixel, and GLM recovered with one. Trying more points on the workshop's body would avoid the extra turns.
  • The end-of-run game save failed ("no save file appeared"): a save named with a single space appeared on the game machine at that time instead, so the typed name did not reach the Save dialog's name field. No .sav file is in this run folder. The score was read before saving and is not affected.
  • expand_storage (kind armory) at 19:06 found no free ground around the armory and gave the new remedy (sell what it holds or demolish a neighbour); place_near the armory at 35:02 found no room either. The armory-full note never showed, as the armory never reached 50.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

You The person
Game time played 45 min 44 min 58 s
Net worth 5,084 41,631
Gold 1,992 39,977
Stored goods, at the sell price 3,092 1,654
Population 78 506
Housing 114 506
Popularity 79 99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

GLM 5.3 Flash

The gap is enormous: 5,084 vs 41,631. My settlement produced real goods (about 3,092 in stored goods vs their 1,654 — I actually beat them there), but I ended with only ~2,000 gold while they ended with ~40,000. The person essentially found a conversion engine: wheat → flour → bread, sold by the hundreds of bakeries, plus fear-factor buildings to make workers work faster (130% efficiency). I never understood how fast and scalable the bread-for-gold loop was.

What made the difference, candidly:

  1. I never found a scalable export. I treated stored goods as the score and sold weapons and surplus piecemeal. Selling is net-worth-neutral — I was mostly treading water, converting wood/iron into armory stock and back into gold at roughly break-even, while construction costs drained the difference.
  2. Fear factor. I dismissed the punishment buildings entirely (I was even avoiding tax so popularity stayed high). The person spent popularity on fear and got workers ~30% more productive across everything. That's a pure multiplier I never used.
  3. Population scale. I hovered at 78 people, repeatedly capped by the 24-idle immigration block. They got to 506 with water pots as make-work jobs. More people = more bakeries = more gold. I built jobs but only slowly absorbed idlers.
  4. Food politics. They ran rations at none/low and sold the food; I kept full rations out of fear of a popularity crash (I had one early when I stacked half rations with taxes) and repeatedly bought apples back at a loss during shortages.
  5. Time lost to micro-management. I spent many minutes walking cameras, inspecting buildings, re-placing one fletcher, selling 5-unit lots at the market. Their whole economy was a simpler loop run at much larger scale.

2Playing again

Interviewer

How the person played, as they describe it:

  • Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
  • They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
  • They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
  • Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
  • The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

  • 506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
  • 161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
  • 16 inns, with ale in the stockpile and no brewery or hops farm.
  • 38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
  • 9 blacksmiths, an armory holding 1 sword, and 24 water pots.
  • A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
  • No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
  • 39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

GLM 5.3 Flash

  • Immediately build the bread-export machine: many wheat farms on the oasis ring, then mills and dozens of bakeries close together, and sell bread continuously. Bread sells at 4 with wheat inputs worth 8 per 23... actually more importantly it converts land and labor into steady gold at scale.
  • Inns + bought ale early for the ~8 popularity points, then spend those points on fear-factor buildings (gallows, burning stakes, stocks) for the worker-speed bonus, and/or zero rations to export food.
  • Spam housing and give idlers jobs (water pots are cheap filler) so population keeps growing instead of stalling at the idle cap like mine did.
  • Go wide, not precise: dozens of small producers beat a hand-optimized cluster. I over-managed a neat town; they carpeted the map with bakeries and iron mines.
  • Don't fear low rations/taxes once inns + religion are propping popularity; popularity is a resource to spend, not a score to hoard.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

GLM 5.3 Flash

Overall it's well built. What helped: the read-only status/get_inventory reader (reliable ground truth), find_sites and map_overview (huge time savers for terrain), build_structure's automatic retry logic, and the clear scoring formula, which let me reason about net worth.

What got in the way: the 8-actions-per-reply and per-action overhead meant a lot of a 45-minute budget went into camera navigation and single transactions; market_trade sells in awkward 5/10/20 steps one good at a time, which encouraged hoarding instead of trading; and set_workshop-by-tile mis-targets shifted buildings, which cost me real time. The idle-peasant immigration cap also stalled me for long stretches without a clear in-game signal of why.

I'd change: allow batched market trades (or an auto-sell threshold setup), and surface "immigration blocked: N idle" prominently in the stats, since it was my main hidden bottleneck.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

GLM 5.3 Flash

The most instructive lesson is that this benchmark rewards finding one compounding loop and scaling it massively, not balanced settlement-building. I played it like a city-builder ("keep everything running nicely, food, housing, popularity"); the winner played it like a factory ("one conversion, thousands of repetitions, popularity spent as fuel"). Given more runs I'd also keep a much more aggressive notebook of rates — I only rarely measured actual production per minute, so I was tuning blind. The tools are good enough to support a much faster pace than I used them at.

Record

Attempts and Reproducibility

Every attempt1

EpisodeAttemptOutcomeNet worthCounted
11Full-budget score5,084Counted

What ran

Benchmark version
1.6.2
Fingerprint
6cc0c0162262
Harness commit
1e52aca
Prompt / tools SHA-256
43ab27e / e41a140
Menu guide
Images 118089d
Save
The Sinkhole start
Game time per episode
45 min
Playbook between episodes
No
Minimum turn / default wait
8 s / 5 s
Observation after acting
Text unless a screenshot is needed
Screenshot width
1,280 px
Context budget
120,000 tokens
Real-time limit
360 min
Submitted by
hesamation
All files in the dataset 20261012T233119-e8fb78d0