One episode · The Sinkhole

Muse Wonder

Benchmark 1.7.1MetaReasoning mediumPlayed by the maintainersFull-budget score
Model
meta/muse-wonder
Served via
OpenRouter · meta
Max output
32,768 tokens
Played
15 October 2026
Score
33.0
Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game
Net worth
14,473
+12,278 over doing nothing, 31% of the person’s gain of 39,436
Population
125
24% of the person’s gain of 502 · popularity 57
Cost
$2.16
4.0M tokens
Time
45 game min
48 min of wall-clock time, 75 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

  • Muse Wonder
  • Baseline (do nothing), final
  • A person's game, final

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Videos and images stream from the dataset. Game footage © Firefly Studios, shared for non-commercial research.

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

14,473net worth
Gold
4,917
Stored goods, at sell price
9,556
Over the baseline (do nothing)
+12,278
Population / housing
125 / 130
Popularity
57
Food
456
Structures (map-wide)
140
Attempt
1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock9,556 gold in goods

GoodUnitsPriceValue
Gold4,91714,917
Iron222235,106
Bread30441,216
Flour120101,200
Crossbows39301,170
Apples1474588
Stone207140
Wood92192
Hops3824
Cheese5420
Net worth14,473

Beside a person’s gameThe Sinkhole, the same game time

This episodeThe personLeft alone
Net worth14,47341,6312,195
Gold4,91739,977
Stored goods, at the sell price9,5561,654
Population125506
Housing130506
Popularity5799

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game
July 1189
Idle peasants
24
Popularity next month
−1
Food
+2
Taxes
−4
Religion
+1
Ale from the inn
±0
Taxes
Average taxes, 100 gold a month
Rations
full; 3 of 3 foods eaten
Worker efficiency
100%
Storage
16 stockpile piles, 3 empty; 2 granaries holding 456 food

Buildings104

  • Bakery18
  • Iron Mine17
  • Hovel15
  • Woodcutter13
  • Apple Orchard9
  • Fletchers' Workshop7
  • Wheat Farm5
  • Dairy Farm4
  • Stockpile4
  • Granary2
  • Mill2
  • Armory1
  • Brewery1
  • Chapel1
  • Church1
  • Hops Farm1
  • Inn1
  • Keep1
  • Marketplace1

Tool calls140

  • find_sites55
  • building_info25
  • build_structure24
  • wait_and_observe10
  • set_workshop7
  • set_tax6
  • inspect_building5
  • map_overview2
  • update_plan2
  • expand_storage1
  • market_trade1
  • notebook_write1
  • status1

Play

Turns
75
Buildings placed / attempted
97 / 97
Anchor-tool placements
1
Tool errors
1
Raw inputs sent to the game
586
Context compactions
1
Game time
45 min

Compute

Tokens
4.0M
Served from cache
95%
Output tokens
29K
Cost, as billed
$2.16
Wall-clock time
48 min
of which inference
12 min
Peak game memory
3,560 MiB

The plan, before play

Task: Grow the economy of The Sinkhole Free Build save for 45 game minutes. Final score is net worth first (gold + stored goods at sell prices), then population and popularity. No soldiers, walls, towers or defences.

Starting state: Jan 1181, 5/10 housing, granary + 4 stockpiles by keep, 2000 gold, 55 wood, 20 stone, 55 meat. Keep on inner edge of oasis ring around a basin; iron ore 50-90 tiles east, large stone ~85 tiles, tree grove ~40 tiles SW, more groves along ring.

Opening plan:

  1. Read stats/map first (free reads): check status, inventory, map_overview, find_sites for woodcutters, farms, quarry, iron.
  2. Build 2-3 woodcutters near trees, 2 hovels for housing, apple/dairy farms on oasis grass near granary for early food. Keep buildings off oasis except farms; leave open sides and routes to keep/storage.
  3. Add marketplace early to trade surplus and buy wood if it accelerates building; set low/no tax while popularity builds.
  4. Expand to quarry + ox tethers on stone and iron mines on ore (high sell values 7/23), plus wheat->mill->bakery chain and hops->brewery->inn for food variety/ale popularity.
  5. Grow population with hovels ahead of jobs (watch 24-idle cap), add chapel/inn, then raise taxes for gold. Build armory + fletcher/blacksmith/armorer workshops to convert wood/iron into high-value weapons (bows 15, crossbows/swords/maces/metal armour 30).
  6. Continuously monitor storage piles, expand stockpile/granary/armory before full, sell surplus, and keep expanding production until time runs out — growth to the last minute counts.

Commentary during play0

The model played this episode without writing commentary between its tool calls.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

  • Model: Muse Wonder (meta/muse-wonder) on OpenRouter, provider Meta only, reasoning medium, output limit 32,768 tokens
  • Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
  • Run b387c248, series 20261015T195630-5d6c7dc4, 2026-10-15: Muse's first valid game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its 1.6.2 game. An earlier series that evening (20261015T193710-4ffd4860) does not count: the host's computer went to sleep 4 game minutes in, and the replay could not launch the game.

Result

This game (1.7.1) Its 1.6.2 game The person Left alone
Net worth 14,473 15,781 41,631 2,195
Gold 4,917 5,534 39,977
Stored goods, at the sell price 9,556 10,247 1,654
Population / housing 125 / 130 136 / 146 506 / 506 4
Popularity 57 (−1 next month) 82 99 0
Score 33.0 38.4 100 0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 12,278 over leaving the game alone, 31% of the person's growth, against 13,586 (34%) in the 1.6.2 game. The score is the project website's formula (80% net worth, 10% each population and popularity); this game counts on the website's leaderboard. In this round so far GPT-6.1 Sol scored 50.4, Claude Opus 5.5 34.7 and Claude Sonnet 5.5 33.1.

Of the stored value, 5,106 is 222 iron from seventeen iron mines, 1,216 is 304 bread, 1,200 is 120 flour, 1,170 is 39 crossbows and 588 is 147 apples. The town ended with 24 idle peasants, 456 food in two granaries and 5 empty places in its hovels.

How the game went

Wood, orchards and hovels, 0 to 10 minutes. Three woodcutters, two orchards, two hovels and a marketplace in the first turn, four more woodcutters and two hovels at 3:23, then two iron mines, a dairy farm, a wheat farm and orchards at 6:31 and a mill at 8:03. It bought 50 apples at 9:23 and built an armory with three fletcher's workshops at 9:55, set to crossbows. No taxes; popularity stayed at 100, and from 10 minutes the town had 24 idle peasants.

A bit of everything, 10 to 30 minutes. A first bakery and two iron mines at 11:11, more fletchers (seven in all by 21:22), three iron mines at 15:37, average taxes from 16:44, a hops farm and a brewery, a chapel at 19:47, low taxes from 21:08, a second mill at 22:47, a church and two bakeries at 27:00 and an inn at 28:35. Popularity slid from 100 to 61 by 30 minutes; the population grew from 48 to 96.

Bakeries, 30 to 45 minutes. High taxes at 31:55, and from 32:03 bakeries in twos and threes, eighteen by 42:24, with an iron mine every turn or two, seventeen in all. A second granary at 37:43. Taxes moved between average and high to the end. On full rations, food piled up from 157 at 35 minutes to 456. Nothing was sold all game; net worth rose from 4,960 at 30 minutes to 14,473.

Game minute 0 5 10 15 20 25 30 35 40 End
Net worth 2,415 2,328 2,066 1,933 2,710 3,436 4,960 7,745 10,558 14,473
Gold 2,000 2,000 1,600 1,300 1,444 1,465 1,500 2,543 3,572 4,917
Iron 0 0 1 7 23 43 74 118 161 222
Food 55 38 54 44 37 58 98 157 255 456
Population / housing 6 / 10 28 / 42 48 / 50 56 / 58 73 / 98 83 / 106 96 / 106 108 / 114 116 / 130 125 / 130
Popularity 100 100 100 100 78 62 61 77 69 57

Each column is the last reading at or before that minute.

Beside its 1.6.2 game

The same mixed plan: iron mines, a crossbow quarter of seven fletchers, bakeries, a chapel and a church. The 1.6.2 game had more iron mines (22 against 17) and four quarries with ox tethers, and held popularity at 82 with low taxes; this one built more bakeries (18 against 8) and an inn, but ran average and high taxes and ended at 57. Both kept most of their value as goods.

What held it back

  • Nothing sold. Bread, flour and apples piled up (456 food at the end on full rations, and 120 flour), and iron went to the stockpile. It made one purchase all game and no sale; Muse named selling and reinvesting as its main miss.
  • Bakeries late. The first bakery came at 11:11 and most of the eighteen after 32 minutes; in the last three game minutes they delivered 183 bread, worth 732 at the sell price.
  • Slow crossbows. Seven fletcher's workshops made 39 crossbows in about 35 minutes; at the end they delivered about 0.1 a workshop a minute.
  • 24 idle peasants in 23 of 45 readings, with no open jobs.

What worked

  • No failed placement at all: every one of its 24 build_structure calls placed what it asked for.
  • Seventeen iron mines and 222 iron kept, the largest part of its net worth.
  • The prompt cache held: 74 of 75 requests read it, with Meta's cache key in place.

Market

50 apples bought for 400 gold, a loss of 200 net worth against the sell price (8 for 4); nothing sold.

Cost and speed

75 turns, 140 tool calls (55 find_sites, 25 building_info, 24 build_structure, 10 waits, 7 set_workshop), 3.96 million tokens and $2.155 billed by OpenRouter (preparation $0.098, play $2.007, compaction $0.051), with 95.2% of input read from the cache. The interview cost $0.054 more, with 71,668 of its 74,464 tokens read from the cache. The agent run took 48.2 real minutes, 12.0 of them waiting for the model; the context was compacted once (19 seconds). The series that did not count cost about $0.27 more.

The interview

Muse called the difference scale: a working mixed economy of farms, mines, crossbows, bakeries, religion and an inn, never turned into "a single high-throughput system". It said it was too protective of food and popularity, kept full rations, leaned on taxes, expanded step by step instead of using gold and the market, spent on slow crossbows, and took the fear factor for decoration. Playing again it would build around bread, population and selling from the start, buy ale for inns, try religion with the fear factor, add cheap jobs against the idle cap, cut rations once popularity allows, build more mines and quarries, and use the market to reinvest. It praised the statistics, find_sites, tile targets and map_overview, and asked for more of the game's numbers: worker efficiency and its causes, religion and inn coverage, ale and food use, and expected output per worker. It also said "selling leaves net worth unchanged", while true, undersells what gold can buy.

For the harness

  • No provider errors, stalled replies or refused calls; the game saved at the end ("muse wonder 1-7-1 b387c2", in this folder).
  • The series before it was lost to the host computer sleeping on battery: the bridge timed out 4 game minutes in, and the game's Proton session on the game machine never exited, so Steam ignored the next launches ("The game window did not appear"). game-session.py status looks only for the game process, not for a launch still holding the app; once that session was ended, the game launched at once.

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

You The person
Game time played 45 min 44 min 58 s
Net worth 14,473 41,631
Gold 4,917 39,977
Stored goods, at the sell price 9,556 1,654
Population 125 506
Housing 130 506
Popularity 57 99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Muse Wonder

My settlement was much smaller: 125 people instead of 506, 57 popularity instead of 99, and net worth 14,473 instead of 41,631. The composition was also different. I finished with 4,917 gold and a large amount of value tied up in stored iron, flour, bread and other goods, while the person finished with almost 40,000 gold and relatively little storage.

The main difference was scale. I built a functioning mixed economy—food farms, 17 iron mines, crossbow workshops, bakeries, religion and an inn—but I never turned it into a single high-throughput system. The person built an enormous bread industry: 31 wheat farms, 19 mills and 161 bakeries, supported by hundreds of additional workers. Their production system must have generated saleable output far beyond what my 18 bakeries produced.

I was also too protective of food and popularity. I kept full rations, worried about food stocks and used taxes as a major gold source. That limited the amount I felt able to push population and industry. The person treated food primarily as a commercial product rather than something that had to accumulate for the population. Cutting rations to none allowed nearly all bread to be sold while purchased ale, religion and other popularity sources kept people satisfied.

I did recognize that bakeries were profitable, iron was valuable and crossbows converted wood cheaply, but I expanded incrementally and often waited for resources instead of using gold and the marketplace aggressively to accelerate growth. I also spent time and money diversifying into crossbows, whose production turned out to be slow. The person concentrated much more heavily on the best scalable chain and only used blacksmiths as a minor sideline.

Finally, I missed the importance of worker efficiency. Their fear-factor system made workers 130% efficient, effectively multiplying the entire economy. I avoided those buildings because they appeared to be popularity-reducing decorations rather than a central economic mechanism, and I did not investigate how religion and inns could offset them.

2Playing again

Interviewer

How the person played, as they describe it:

  • Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
  • They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
  • They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
  • Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
  • The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

  • 506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
  • 161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
  • 16 inns, with ale in the stockpile and no brewery or hops farm.
  • 38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
  • 9 blacksmiths, an armory holding 1 sword, and 24 water pots.
  • A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
  • No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
  • 39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Muse Wonder

I would build around bread, population growth and marketplace sales from the beginning.

First, I would establish woodcutters, wheat farms, mills and bakeries much earlier, and keep expanding that chain as the main engine. Apple orchards could support the earliest population, but wheat and bread would become the priority because bread is reproducible, scalable and easily sold.

Second, I would use inns and purchased ale as a popularity tool rather than waiting for a hops farm and brewery. My hops chain produced very little ale, so my inn’s benefit was inconsistent. Buying ale early would be a worthwhile investment if it raised popularity enough to support ration cuts, faster immigration and eventually fear-factor bonuses.

Third, I would investigate religion plus fear factor early. If church/cathedral coverage and inns could offset the popularity loss, running workers at 130% efficiency would be one of the biggest available gains. I incorrectly treated those buildings mainly as punitive decorations.

Fourth, I would aim for a much larger population. That means building hovels earlier and creating large numbers of simple jobs so the 24-idle immigration barrier does not throttle growth. Their water pots are a useful example: even if a service building does not directly produce saleable goods, employing three people can unlock more immigration and support the wider economy.

Fifth, I would keep rations at half or none once popularity could sustain it. I spent too much of the run feeding a smaller population full rations and preserving several food types for variety. If bread can be sold instead, food production becomes gold production rather than an inventory that grows passively.

Sixth, I would build many more iron mines and quarries, because the person demonstrated that the map can support far more extraction than I attempted. I would continue using those resources, but with greater emphasis on selling output and reinvesting the proceeds into bakeries, mines, housing and worker-employment buildings.

Seventh, I would use the marketplace more actively. Selling output does not by itself change net worth, but converting stored production into gold gives flexibility to buy ale, wood or other bottlenecks and expand the production base. My final inventory was valuable, but a smaller settlement producing continuously for 45 minutes was worth less than a much larger settlement that had repeatedly converted output into expansion.

I would also stop treating taxes as indispensable. Tax income helped me, but it constrained popularity and population. The winning approach suggests that selling a large industrial surplus can dwarf tax income while keeping taxes at none.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Muse Wonder

The strongest tools were the read-only statistics, find_sites, and tile-based build_structure. Being able to place buildings without manually navigating every menu made a large economy feasible, and the production, storage and popularity breakdowns gave useful feedback. map_overview was also valuable for understanding the ring, oasis and resource locations.

The main difficulty was learning hidden or poorly quantified mechanics through limited observations. For example, I could see that a chapel initially produced little religion benefit, a church suddenly improved it, and the factor later fluctuated, but the interface did not clearly show priest coverage or explain the thresholds. The same applied to inn coverage, ale consumption, food-variety changes and the potentially major effect of fear factor on worker efficiency. I had building descriptions, but not enough numerical information to compare strategies confidently.

The eight-action limit and the time consumed by camera-moving placements also encouraged batching and made experimentation expensive. That is understandable in a real-game benchmark, but it meant I spent a significant part of the run managing construction logistics rather than testing economic assumptions.

The instructions helped by explaining storage, placement and basic production chains. However, the statement that selling leaves net worth unchanged is technically correct at the moment of sale, while potentially understating the strategic importance of liquidity. Gold obtained through sales can be reinvested quickly into more productive capacity, and that compounding effect proved decisive. Likewise, the guidance framed popularity largely through rations, taxes, food variety, religion and ale, but did not highlight fear-factor efficiency as a major economic lever.

I would improve the benchmark by exposing a few more mechanical values in building panels or statistics: worker efficiency and its causes, religion and inn coverage percentages, ale consumption, effective food consumption, and expected output per worker. That would reward planning and interpretation rather than requiring prior game-specific knowledge.

The 45-minute limit was a good test of prioritization. It was long enough to build a substantial economy but short enough that early misconceptions could not be fully corrected.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Muse Wonder

I played too much like I was preserving a safe settlement and too little like I was trying to maximize a timed economic race. Several of my decisions were locally sensible—keeping food stocked, maintaining popularity, avoiding unnecessary purchases and ending with valuable inventories—but those choices limited compounding growth.

The human result is a useful reminder that end-state net worth does not simply reward having valuable goods; it rewards building the largest productive system that can repeatedly turn labour and land into value before time expires. My economy worked. It simply did not scale aggressively enough.

Record

Attempts and Reproducibility

Every attempt1

EpisodeAttemptOutcomeNet worthCounted
11Full-budget score14,473Counted

What ran

Benchmark
The Sinkhole construction
Benchmark version
1.7.1
Fingerprint
63921aad76e5
Harness commit
674e7ba
Prompt / tools SHA-256
3b0fa8a / f75ebb8
Menu guide
Images 118089d
Save
The Sinkhole start
Game time per episode
45 min
Playbook between episodes
No
Minimum turn / default wait
8 s / 5 s
Observation after acting
Text unless a screenshot is needed
Screenshot width
1,280 px
Context budget
120,000 tokens
Real-time limit
360 min
Submitted by
hesamation
All files in the dataset 20261015T195630-5d6c7dc4