Crusader Bench

One episode · Oasis by the Sea

GLM 5.3 Flash

Benchmark 1.5.0Z.aiReasoning mediumPlayed by the maintainersFull-budget score

All of GLM 5.3 Flash’s results and comparisons

Model
z-ai/glm-5.3-flash
Served via
OpenRouter · z-ai/fp8, siliconflow/fp8
Max output
32,768 tokens
Played
11 October 2026
Score
33.8
Crusader Bench score on Oasis by the Sea: 0 is doing nothing, 100 the person’s game
Net worth
6,200
+4,980 over doing nothing, 27% of the person’s gain of 18,293
Population
64
30% of the person’s gain of 198 · popularity 90
Cost
$0.15
4.1M tokens
Time
33 game min
44 min of wall-clock time, 89 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

  • GLM 5.3 Flash
  • Baseline (do nothing), final
  • A person's game, final

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

Commentary

Five Seconds Short

Quote

“Literally just calling tools until the clock ends would have gained real gold — ending early on a passive watch was pure waste.”

Model
GLM 5.3 Flash
Source
Its interview, after the game

GLM 5.3 Flash turned iron into 4,755 gold, the most gold of the first five games on version 1.5, and finished at 6,200. With 32 game seconds left it decided there was nothing more to do and answered four times without a tool call, which ended its game five seconds early.

It built seven iron mines on the ore strip, five dairy farms, two quarries and 25 hovels, with housing at 210 by the fourteenth minute, and bought 150 wood at the start. The last building went up at 16 minutes. From then on it traded: every couple of minutes it sold iron, stone and surplus wood and bought bread, and moved the tax between levels 3 and 5 to hold popularity between 67 and 90. It sold 146 iron for 3,498 gold.

Its purchases cost it most: 290 bread for 2,269 gold lost 1,109 net worth, since bread costs about twice its sell price, and the wood another 450. The population stopped at 64 from the seventeenth minute with 146 places empty; it had seen the 24 idle peasants at 8 minutes and kept building hovels anyway.

The host warned it each time that four replies without a tool call end the game. The fourth came at 1,994.7 of 2,000 game seconds, and the settlement was read from the game as at the end of any game. The game cost $0.15 as billed by OpenRouter.

The episode

Watch the Episode

Videos and images stream from the dataset. Game footage © Firefly Studios, shared for non-commercial research.

Episode 1 · 33 min 20 s of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

6,200net worth
Gold
4,755
Stored goods, at sell price
1,445
Over the baseline (do nothing)
+4,980
Population / housing
64 / 210
Popularity
90
Food
59
Structures (map-wide)
74
Attempt
1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock1,445 gold in goods

GoodUnitsPriceValue
Gold4,75514,755
Iron2923667
Stone427294
Wood2481248
Bread474188
Cheese12448
Net worth6,200

Beside a person’s gameOasis by the Sea, the same game time

This episodeThe personLeft alone
Net worth6,20019,5131,220
Gold4,75518,304
Stored goods, at the sell price1,4451,209
Population64202
Housing210202
Popularity90100

The person played Oasis by the Sea for 33 min 18 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game
September 1200
Idle peasants
24
Popularity next month
+1
Food
+1
Taxes
−2
Religion
+2
Taxes
Low taxes, 38 gold a month
Rations
full; 2 of 2 foods eaten
Worker efficiency
100%
Storage
8 stockpile piles, 0 empty; 1 granary holding 59 food

Buildings59

  • Hovel25
  • Woodcutter11
  • Iron Mine7
  • Dairy Farm5
  • Ox Tether2
  • Quarry2
  • Stockpile2
  • Apple Orchard1
  • Chapel1
  • Granary1
  • Keep1
  • Marketplace1

Tool calls117

  • find_sites32
  • market_trade28
  • build_structure20
  • wait_and_observe14
  • go_to_tile13
  • set_tax5
  • building_info1
  • expand_storage1
  • inspect_building1
  • map_overview1
  • observe1

Play

Turns
89
Buildings placed / attempted
54 / 65
Anchor-tool placements
0
Tool errors
0
Raw inputs sent to the game
533
Context compactions
1
Game time
33 min

Compute

Tokens
4.1M
Served from cache
97%
Output tokens
15K
Cost, as billed
$0.15
Wall-clock time
44 min
of which inference
17 min
Peak game memory
3,243 MiB

The plan, before play

I understand the task: grow the Oasis by the Sea free-build economy over ~33 minutes of game time, maximizing net worth (gold + stored goods at sell prices), population and popularity. No military buildings or defenses.

Opening plan:

  1. Immediately place the granary (required), near the keep facing the oasis grass.
  2. Woodcutters by the tree grove to the west; check terrain with map_overview/find_sites.
  3. Hovels for housing; keep popularity above 50 so peasants arrive.
  4. Farms (apple/dairy first — direct food) on the oasis grass, grouped tightly.
  5. Add marketplace early to enable trading and check prices.
  6. Later: wheat→mill→bakery chain, iron mine + quarry (iron is worth 23) with ox tether, tax at sustainable level, ale chain if oasis grass allows, expand storage before it fills.
  7. Regularly check status/popularity/storage and keep building until the run ends.

I'll batch actions, read first (free), and end each reply with an observation.

Commentary during play5

  1. The granary already exists (stats show 1 granary). Moving on: hovels and woodcutters.

  2. The run is essentially complete — 32 seconds left. Final state: gold 4717, population 64/210 housing, popularity 88 (rising), iron 25 (worth ~575), stone 42, wood 236, food 60 with rations full. The settlement ended with steady iron/stone production, positive popularity, and no stockpile or granary blockage.

  3. Gold 4755, popularity 89, iron at 26 — the run ends in ~23 seconds with the strongest state yet. Nothing further to change; the economy is running cleanly with rising popularity, steady iron and stone production, and food buffered.

  4. Final seconds of the run — popularity 90 and rising, gold 4755, iron 27, all production flowing. The settlement is left in its best state; nothing to change.

  5. Run complete — 6 seconds left. Final state: gold 4755, iron 29, stone 42, wood 248, food 59 with full rations, population 64 with popularity 90 (factors food +1, tax −2, religion +2). Net worth roughly: 4755 gold + 29×23 (667) + 42×7 (294) + 248×1 + food ~482 ≈ 6250, up from the ~1425 starting package, with high popularity and a healthy, growing population.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, interview.md); it is a reading of the run, not part of its score.

  • Model: GLM 5.3 Flash (z-ai/glm-5.3-flash) on OpenRouter, providers z-ai/fp8 then siliconflow/fp8, reasoning medium, output limit 32,768 tokens
  • Benchmark: Oasis by the Sea construction, version 1.5.0, from the save the human reference game started from; 33 min 20 s of game time (60,000 ticks) at speed 40, the length of the person's game; no playbook
  • Run 8f5c6169, series 20261011T132054-9bb4d8ae, 2026-10-11

Result

GLM 5.3 Flash The person Left alone
Net worth 6,200 19,513 1,220
Gold 4,755 18,304
Stored goods, at the sell price 1,445 1,209
Population / housing 64 / 210 202 / 202
Popularity 90 100

Net worth grew by 4,980 over leaving the game alone, about a third of the person's growth (18,293).

The run is a model failure, not a full-budget score: with 32 game seconds left GLM decided there was nothing more to do and answered four times in a row without calling a tool. The host warned it each time that four such replies end the run, and the fourth ended it at 1,994.7 of 2,000 game seconds. The net worth was read from the game afterwards, so it is the settlement's real value.

How the game went

Opening, 0 to 3 minutes. A marketplace, one apple orchard, four hovels and woodcutters, and 150 wood bought at the marketplace. One granary placement was refused.

Building out, 3 to 16 minutes. Seven iron mines on the ore strip near the keep, five dairy farms, two quarries with two ox tethers, a chapel at 8 minutes, and hovels until housing reached 210 at 14 minutes (25 hovels in all). Spending took net worth down from 1,265 to 723 at 9 minutes, with 117 gold. The last building went up at 16 minutes.

Trading, 16 minutes to the end. No more buildings. Every couple of minutes it sold iron, stone and surplus wood and bought bread, and it moved the tax between levels 3 and 5 five times to keep popularity between 67 and 90. Net worth rose steadily, about 240 a minute from 12 minutes on, almost all of it as gold.

Game minute 0 6 9 12 15 18 21 24 27 30 End
Net worth 1,265 951 723 1,141 1,606 2,283 3,071 3,677 4,192 5,136 6,200
Population 3 45 53 54 58 64 64 64 64 64 64
Housing 10 90 138 170 210 210 210 210 210 210 210

What held it back

  • The population stopped at 64 from 17 minutes, with 146 empty places. It stopped growing when 24 peasants stood idle, and stayed there; Claude Haiku 5.5's run on the same save the same day stalled at 24 idle too. That fits a limit on how many peasants wait at the campfire: more people seem to come only when the idle ones get work. GLM saw the 24 idle peasants at 8 minutes ("More peasants won't produce more without jobs") but went on building hovels and put the slow growth down to immigration. The person ended with 202 people, working in 57 bakeries, 13 iron mines, 10 quarries and more.
  • It stopped building at 16 minutes with 24 idle workers and 90 to 250 wood in store, for the last 17 minutes of the game.
  • Farmland ran out. It found room for one orchard and five dairy farms and decided bread was out of reach without wheat land. The person bought wheat at the marketplace and milled and baked it.
  • Food was bought, not grown. It bought 290 bread for 2,269 gold, which lost 1,109 net worth (bread costs about twice its sell price), and 150 wood for 600 gold (−450). All its trades together lost 1,201 net worth; selling iron, stone and wood is roughly neutral, since goods already count at their sell price.

What worked

  • No tool call returned an error, though 11 placements were not placed. It used the marketplace's selling steps correctly in its own planning ("sells 5 below 50 stock, 10 from 50, 20 from 100").
  • Iron was its main product: 146 iron sold for 3,498 gold, with 29 more in store at the end.
  • Popularity never fell below 67 and ended at 90.

Cost and speed

89 turns, 117 tool calls, 4.12 million tokens and $0.145 (billed by OpenRouter). The agent run took 43.8 real minutes, 16.9 of them waiting for the model; one compaction took 34 s.

At 25 game minutes, the length of the earlier benchmark versions, it stood at 4,201 net worth for $0.125 and 35.3 real minutes. The six published GLM episodes of versions 1.0 and 1.1 ended between 2,249 and 5,495 (4,167 on average); the published 1.1 episode 1 cost $0.205 and took 41.1 minutes.

The interview

GLM called its passive ending "self-inflicted" and the gap to the person "stark". Playing again, it would put population first (religion, no taxes), run a wheat to flour to bread chain from bought wheat, and use bad things for worker efficiency; it said it had wrongly avoided buying because purchases lower net worth.

For the harness

  • The rule that ends a run after four replies without tools turned a nearly complete game into a model failure 5 seconds before the end.
  • Site search covers only the current view, and placing on a tile off screen fails: two placements failed with not_on_screen.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

You The person
Game time played 33 min 15 s 33 min 18 s
Net worth 6,200 19,513
Gold 4,755 18,304
Stored goods, at the sell price 1,445 1,209
Population 64 202
Housing 210 202
Popularity 90 100

Your run ended before its game time was used up (No tool calls in 4 replies in a row), so this is not a full-budget result.

Leaving the game alone for the same time ends with a net worth of 1,220.

How do you see your result next to the person's? What do you think made the difference?

GLM 5.3 Flash

The gap is stark: 6,200 vs 19,513 net worth, and 64 vs 202 population. I ended the run passively — three replies with no tool calls while waiting for the clock to run out — which directly cost me the last few minutes of income and was self-inflicted. The deeper difference is strategic: I treated "net worth" as stockpiled iron and a slow tax drip, while the person treated the economy as a conversion machine — buy wheat (23/unit), mill it, bake it into bread, sell it (bread sells 8?) — no wait, they ran the chain from bought wheat to bread and it was profitably self-sustaining at scale, because 57 bakeries × worker efficiency compounded. They also scaled population to 202, which multiplies every production chain, whereas I hit ~64 and concluded immigration was "the cap" without ever testing what would move it (they got 100 popularity via religion and used fear to offset it while working at 140% efficiency).

2Playing again

Interviewer

How the person played, as they describe it:

  • Bread was one of their main sources of income: mills ground wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
  • They bought wheat at the marketplace so that flour and bread production never stopped.
  • A church and a cathedral raised popularity.
  • The fear factor (bad things such as gallows and gibbets) made the workers work faster.

Their settlement at the end, read from the game:

  • 202 people in 24 hovels, with no free housing; popularity 100.
  • 57 bakeries, 7 mills and 10 wheat farms. The marketplace sold bread automatically.
  • 13 iron mines and 10 quarries with 33 ox tethers, and 10 woodcutters.
  • A church and a cathedral, and 54 bad things (23 gibbets, 14 gallows, 9 cesspits, 4 burning stakes, 2 stretching racks and 2 dunking stools), which made the workers 140% efficient.
  • No inns and no good things. No taxes, and full rations of bread. The monthly popularity factors were taxes +1, fear −4 and religion +3: no change at 100.

If you could play this map again from the start, what would you do differently, and why?

GLM 5.3 Flash

Three big changes:

  1. Population first. I wrongly concluded arrival rate was fixed. The person got 202 people by keeping popularity at 100 (church + cathedral, no taxes) — more workers means more of everything. I'd build religion early and spam hovels from minute one.
  2. Buy-into-value-added chains. I avoided buying goods because "buying lowers net worth." That's true per unit, but buying wheat at 23 and selling bread made from it is profitable per unit of labor, and labor is what idle peasants and high efficiency give you. I'd run a large wheat→flour→bread operation continuously.
  3. Fear factor. I dismissed bad things as irrelevant ("buildings count only through what they bring") — but they bring +40% worker efficiency, which multiplies everything. 54 gibbets for −4 popularity, offset by religion, was the key trick I never considered.

I'd also not sell raw iron — I suspect at scale, feeding iron into nothing was fine, but the person's gold came overwhelmingly from bread volume, not raw material sales.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

GLM 5.3 Flash

The tools are genuinely good: tile-targeted placement with automatic retry, the reader-backed status, find_sites and the placement guardrails (open-side checks) removed most pixel-hunting pain. What got in my way:

  • The marketplace sell-lot granularity ("sells 20 a click above 100") produced repeated partly_traded surprises; I'd want an explicit "sell X units" API.
  • I could never verify why immigration was slow — the reader doesn't expose arrival rate or what drives it, so I "explained" my stagnation with a plausible-but-wrong hypothesis and stopped pushing on it. Better observability into popularity/immigration dynamics would have helped a lot.
  • The scoring rubric ("buildings count only through what they bring") actively steered me away from the fear-factor trick, which was probably the single largest lever. A hint that efficiency multipliers exist would have changed my whole build.
  • The 33-minute budget is tight for a chain-economy build order; most of my early game went into hovels and woodcutters that mattered little in the end.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

GLM 5.3 Flash

My biggest failure was mode: I optimized a stable small economy (sell iron, keep popularity, avoid losses) instead of a compounding one (maximize workers, maximize throughput, accept per-unit losses where volume wins). By the time I saw the trend it was month 20 of 33. Also, literally just calling tools until the clock ends would have gained real gold — ending early on a passive watch was pure waste. A useful general lesson for agentic play: re-examine "verified" conclusions (like my immigration-cap theory) whenever a score gap suggests they might be wrong.

Record

Attempts and Reproducibility

Every attempt1

EpisodeAttemptOutcomeNet worthCounted
11Full-budget score6,200Counted

What ran

Benchmark version
1.5.0
Fingerprint
31b5ff59f7a5
Harness commit
ee862eb
Prompt / tools SHA-256
2cd353b / 81ddda6
Menu guide
Images 118089d
Save
Oasis by the Sea benchmark start
Game time per episode
33 min 20 s
Playbook between episodes
No
Minimum turn / default wait
8 s / 5 s
Observation after acting
Text unless a screenshot is needed
Screenshot width
1,280 px
Context budget
120,000 tokens
Real-time limit
360 min
Submitted by
hesamation
All files in the dataset 20261011T132054-9bb4d8ae