One episode · The Sinkhole

Kimi K3

Benchmark 1.7.1Moonshot AIReasoning highPlayed by the maintainersFull-budget score

Not part of the evaluation: the game is published, but not ranked.

Model
kimi-k3
Served via
Moonshot API
Max output
32,768 tokens
Played
15 October 2026
Score
17.1
Crusader Bench score on The Sinkhole: 0 is doing nothing, 100 the person’s game
Net worth
7,286
+5,091 over doing nothing, 13% of the person’s gain of 39,436
Population
75
14% of the person’s gain of 502 · popularity 54
Cost
~$1.55
2.2M tokens; ~ at list prices, the API reports no bill
Time
45 game min
52 min of wall-clock time, 43 turns

Minute by Minute

Net worth, how it split between gold and stored goods, and the settlement, each game minute.

Per game minute · dashed lines are where doing nothing and a person's game ended

  • Kimi K3
  • Baseline (do nothing), final
  • A person's game, final

Crusader Bench · crusbench.com

Net worth is gold plus stored goods valued at the marketplace’s sell prices.

Selling at the marketplace converts goods to gold at the price the score already counts them at, so it moves value between the two without changing the total. Buying costs net worth.

Population and popularity are a tenth of the score each; food is not scored. Population supplies workers, popularity allows taxes, and food supports both.

The episode

Watch the Episode

Videos and images stream from the dataset. Game footage © Firefly Studios, shared for non-commercial research.

Episode 1 · 45 min of game time, edited: actions at real speed, waiting fast-forwardedStreamed from Hugging Face

ScorecardFull-budget score

7,286net worth
Gold
3,647
Stored goods, at sell price
3,639
Over the baseline (do nothing)
+5,091
Population / housing
75 / 90
Popularity
54
Food
57
Structures (map-wide)
98
Attempt
1

The settlement at the endcentred on the keep, zoomed out

The settlement at the end of episode 1, zoomed out over the keep

Final stock3,639 gold in goods

GoodUnitsPriceValue
Gold3,64713,647
Iron49231,127
Flour7910790
Metal armour1730510
Bread574228
Swords730210
Bows1015150
Stone207140
Wood1301130
Hops12896
Pikes51890
Spears91090
Crossbows23060
Ale11010
Wheat188
Net worth7,286

Beside a person’s gameThe Sinkhole, the same game time

This episodeThe personLeft alone
Net worth7,28641,6312,195
Gold3,64739,977
Stored goods, at the sell price3,6391,654
Population75506
Housing90506
Popularity5499

The person played The Sinkhole for 44 min 58 s of game time at the same speed, timed by their own clock, and was scored the same way. The model saw this comparison only in its interview.

The town’s booksread from the game at the end

Date in the game
July 1189
Idle peasants
24
Popularity next month
±0
Taxes
−2
Religion
±0
Ale from the inn
+2
Taxes
Low taxes, 44 gold a month
Rations
full; 1 of 1 foods eaten
Worker efficiency
100%
Storage
16 stockpile piles, 4 empty; 1 granary holding 57 food

Buildings62

  • Hovel10
  • Fletchers' Workshop8
  • Iron Mine5
  • Poleturner's Workshop5
  • Woodcutter5
  • Stockpile4
  • Apple Orchard3
  • Bakery3
  • Blacksmith's Workshop3
  • Armorer's Workshop2
  • Dairy Farm2
  • Hops Farm2
  • Wheat Farm2
  • Armory1
  • Brewery1
  • Chapel1
  • Granary1
  • Inn1
  • Keep1
  • Marketplace1
  • Mill1

Tool calls140

  • find_sites29
  • market_trade24
  • wait_and_observe23
  • build_structure20
  • place_near15
  • building_info9
  • set_workshop7
  • set_tax5
  • flat_view2
  • go_to_tile2
  • expand_storage1
  • inspect_building1
  • map_overview1
  • update_plan1

Play

Turns
43
Buildings placed / attempted
44 / 47
Anchor-tool placements
12
Tool errors
1
Raw inputs sent to the game
554
Context compactions
2
Game time
45 min

Compute

Tokens
2.2M
Served from cache
93%
Output tokens
34K
Cost, at list prices
~$1.55
Wall-clock time
52 min
of which inference
21 min
Peak game memory
3,558 MiB

The plan, before play

Task understanding: I have 45 game minutes to grow the economy of a fresh "The Sinkhole" Free Build save. Final score is dominated by net worth (gold + stored goods at market sell prices), with population and popularity as secondary factors. Starting state: 2,000 gold, 55 wood, 20 stone, 55 meat, pop 5/10, granary and 4 stockpiles already placed by the keep, which sits on a ring of oasis grass around a basin containing iron ore (~50–90 tiles east) and a stone deposit (~85 tiles away); trees are ~40 tiles away. No soldiers or defences allowed.

Opening plan:

  1. Immediately place 2–3 woodcutters near the closest grove, plus 2 hovels to raise housing.
  2. Build a marketplace early so I can buy wood to accelerate construction and later sell surplus.
  3. Food: apple orchards and dairy farms on the oasis ring near the keep for direct granary supply (food variety also boosts popularity).
  4. Iron mines on the basin ore patches — iron is worth 23/unit, the best raw-material value; add quarries + ox tethers on the stone deposit as population allows.
  5. Keep housing and popularity above 50 (full rations, food variety, chapel later) so peasants keep arriving; avoid idle-peasant cap.
  6. Mid-game: armory + fletchers/poleturners/blacksmiths to convert wood and iron into higher-value weapons; expand stockpiles/granary before storage fills.
  7. Raise taxes once popularity has margin (religion/ale/extra rations) for steady gold income.
  8. Near the end, sell nothing below value; keep goods stored (they count), keep popularity ≥50 and housing full.

I'll use map_overview and find_sites once at the start to lock in exact spots, then build in batches and keep construction continuous rather than waiting.

Commentary during play16

  1. Iron mines are staffed but deliveries haven't landed yet (long walk). Food is dropping fast with 36 people — I need the bread chain and more jobs (24 idle caps immigration). Checking costs first.

  2. Food is secure for now. Building the bread chain, armory, a second dairy, and checking why the first dairy has delivered nothing.

  3. Food is secure, popularity stable at 63 with religion online. Next lever: the ale chain (hops→brewery→inn) to support higher taxes, plus more jobs for the 24 idle. Checking costs first.

  4. Same sites again — assigning Poleturner (371,450), Blacksmith (373,455), Armorer (375,459). Wood is short, so I'll buy 25 planks at the market first.

  5. Good progress — poleturner and blacksmith are in. Now I'll find a spot for a third bakery (flour is piling up), wait for wood, then retry the armorer.

  6. Armorer and bakery now, while wood is available.

  7. Two armorers now — iron 28 supports it. Flour still high; a 2nd mill isn't needed, but a 2nd inn-side priority is ale (brewery still 0 — hops trickling at 0.7/min). Next: hovels to raise housing cap, then find woodcutter sites to boost wood, which is the weapons bottleneck.

  8. Ale is flowing (inn +4) and popularity is recovering. Population dipped as peasants emigrated during the dip below 50 — it'll rebound. Priorities now: armory is 40/50 (expand it), and iron production (2/min) lags weapon consumption — time for a 4th mine.

  9. The armory is boxed in by workshops — no room to expand. I'll move the camera there and use flat view to see exactly which building to demolish to make space, while placing a second hops farm to boost ale.

  10. The flat view captured the hops farm, not the armory — the camera had moved. Let me go back to the armory and flatten the view there to identify which workshop to demolish.

  11. The armory is completely surrounded by workshops — no expansion room, and it's now 50/50, stalling weapon deliveries. Selling is net-worth neutral, so I'll sell spears to free space, add a 4th iron mine, and raise taxes now that popularity is 74.

  12. Healthy state: gold 976, popularity 78, but 24 idle peasants means immigration is capped — I need more jobs. Weapon workshops convert cheap wood into 10–18-value goods; I'll add more and keep the armory clear by selling (value-neutral).

  13. Armory full again — selling bows and swords (value-neutral), then two more fletchers to absorb idle peasants. I'll also switch fletchers to crossbows (10 value/wood vs 7.5, and 30/slot).

  14. Popularity recovered (56, +1) but armory is full — workshops stalled. Selling weapons to free space, then adding 2 more fletchers to convert the wood surplus.

  15. 2 minutes left — cutting taxes to keep popularity above 50 so peasants don't leave before the run ends.

  16. 43 seconds left. Final state is solid: gold 3,559, population 75/90, popularity 53 stable, and large stored reserves (wood 95, flour 77, iron 45, bread 71, full armory of weapons and armour). Selling or buying now would be value-neutral or value-losing, so I'll simply let the remaining time run out.

Reading the game

The Analysis

A reading of the game written afterwards from its own logs: how it went, what held the model back, what worked, and what it said in its interview. It is published with the game and is not part of its score.

Written by Claude (Opus 5.5) from the run's own logs (events.jsonl, episode.json, results.json, interview.md); it is a reading of the run, not part of its score.

  • Model: Kimi K3 (kimi-k3) on Moonshot's API, reasoning high (Kimi K3 has no medium), output limit 32,768 tokens
  • Benchmark: The Sinkhole construction, version 1.7.1 (commit 674e7ba), from the save the human reference game on this map started from; 45 min of game time at speed 40, the length of the person's game; no playbook; waits played fast
  • Run ff80a5af, series 20261015T175751-2419d558, 2026-10-15: Kimi's first game of The Sinkhole's 1.7.1 round, a fresh series with no memory of its 1.6.2 game

Result

This game (1.7.1) Its 1.6.2 game The person Left alone
Net worth 7,286 9,164 41,631 2,195
Gold 3,647 2,356 39,977
Stored goods, at the sell price 3,639 6,808 1,654
Population / housing 75 / 90 91 / 122 506 / 506 4
Popularity 54 52 99 0
Score 17.1 21.1 100 0

A full-budget score: the run used all 2,700 game seconds. Net worth grew by 5,091 over leaving the game alone, 13% of the person's growth, against 6,969 (18%) in the 1.6.2 game. The score is the project website's formula (80% net worth, 10% each population and popularity); this game is not on the website. It is the lowest of this round so far; GLM 5.3 Flash's 1.7.1 game scored 22.6.

Of the stored value, 1,127 is 49 iron from five iron mines, 1,120 is 50 weapons and armour, 790 is 79 flour and 228 is 57 bread. The town ended with 24 idle peasants, no open jobs and 15 empty places in its hovels.

How the game went

A careful start, 0 to 10 minutes. Two hovels and three woodcutters in the first turns, a marketplace at 0:55, planks bought in lots of 25 to 40, orchards, a dairy farm and three iron mines by 3:20, a bakery, a wheat farm and a mill by 6:24 and 50 meat bought at 5:52. An armory at 6:24, then a fletcher's and a poleturner's workshop beside it and a blacksmith at 10:03. Taxes stayed at none for the first 10 minutes; the town had 24 idle peasants from 5 minutes on.

Out of gold, 10 to 25 minutes. High taxes at 9:55, average at 11:53. A chapel, a second wheat farm and bakery, another fletcher, an inn, a hops farm and a brewery by 16:08, and from 18:00 two more poleturners and blacksmiths and two armourer's workshops, with planks bought in between. Gold fell from 1,040 at 10 minutes to 42 at about 20; popularity fell to 43 by 22:14 and the town shrank from 58 to 42 by 24:06.

Selling weapons, 25 to 45 minutes. Four hovels from 22:22 and the town came back to 65 by 29 minutes. expand_storage found no room beside the armory at 26:28, and from 27:26 it sold spears, bows, swords and pikes every few minutes to keep the armory from filling, while adding six more fletcher's workshops, two more poleturners and two iron mines (eighteen weapon workshops in all). High taxes from 27:55, average from 36:02, low from 42:49. Gold rose to 3,647 and net worth to 7,286.

Game minute 0 5 10 15 20 25 30 35 40 End
Net worth 2,415 2,056 1,591 1,709 1,969 2,339 3,525 4,448 5,477 7,286
Gold 2,000 1,800 1,040 685 454 353 976 1,748 2,646 3,647
Iron 0 0 10 20 27 26 27 30 34 49
Weapons and armour 0 0 0 9 20 40 46 48 36 50
Population / housing 6 / 10 36 / 42 43 / 58 49 / 58 55 / 58 42 / 74 65 / 90 70 / 90 73 / 90 75 / 90
Popularity 100 100 100 63 61 48 78 67 56 54

Each column is the last reading at or before that minute.

Beside its 1.6.2 game

Much the same plan both times: a varied village with an inn, a chapel, a few farms of each kind, and a large weapons quarter (eleven fletchers there, eighteen workshops here). This game had fewer people (75 against 91) and kept less in goods; the money crisis around 20 minutes and the loss of 16 people that followed are the main difference.

What held it back

  • Eighteen weapon workshops on five iron mines. Eight fletchers, five poleturners, three blacksmiths and two armourers, fed with bought planks and the iron of five mines. They made about 140 weapons and armour; 93 were sold for 1,685 gold, which leaves net worth unchanged, and 50 were kept. Kimi said in its interview this was wasted effort.
  • Gold run down to nothing. Workshops, planks and food bought from the market took gold to 42 at 20 minutes, with popularity falling under average and high taxes; the town lost a quarter of its people.
  • 24 idle peasants. In 24 of 41 readings, with no open jobs; the town stopped at 75 with room for
  • Food bought, not grown. 90 meat, 15 cheese and 5 apples bought at twice their value; only one kind of food was eaten at the end.

What worked

  • It kept the armory from stalling its workshops by selling as it filled, after expand_storage found no room.
  • Every placement reported as placed is in the game's final count of its buildings.

Market

165 planks, 90 meat, 15 cheese and 5 apples bought for 1,540 gold, a loss of 935 net worth against the sell price (planks 4 for 1, food 8 for 4); 35 spears, 23 bows, 30 swords and 5 pikes sold for 1,685, which leaves net worth unchanged. One purchase of meat at 11:58 bought nothing.

Cost and speed

43 turns, 140 tool calls (29 find_sites, 24 market_trade, 23 waits, 20 build_structure, 15 place_near), 2.15 million tokens and $1.553 at Moonshot's list prices (preparation $0.073, play $1.384, compaction $0.096), with 93.8% of input read from the cache. The interview cost about $0.034 more, with 43,659 of its 45,904 tokens read from the cache. The agent run took 52.2 real minutes, 21.0 of them waiting for the model (about 29 seconds a turn); the context was compacted twice (85 seconds).

The interview

Kimi called the gap "enormous" and put it down to scale: it treated stored goods as the goal and never sold food, while the person sold bread from 161 bakeries. Playing again it would rush inns and bought ale, cut rations and sell the bread, build the bread chain by the dozen, break the idle cap with cheap jobs such as water pots, use the fear factor, skip most weapons and keep taxes near zero. It praised find_sites, tile targets, the stats and the direct tax, ration and market actions. It named as obstacles placement retries that moved buildings about 9 tiles, one of which it says helped box in its armory; food variety it could not keep up; and the idle cap having no signal in the game. It asked for a warning when idle peasants block immigration and an automatic selling tool.

For the harness

  • Placements moved far by retries. Eight placements were placed at another spot by the tile-map retry, five of them 9 to 11 tiles from the target (a wheat farm, an orchard, a poleturner's, a blacksmith's and an armourer's workshop). Each result gave the new tile and the offset, but Kimi read them as silent moves; a retry that far may not be what a model meant.
  • No room beside the armory (26:28). expand_storage found no free ground; the stockpile kept room here (4 empty piles at the end).
  • place_near found no room against its anchor five times (a mill, an armory, a chapel, a brewery and a hovel).
  • No provider errors, stalled replies or refused calls; the game saved at the end ("kimi k3 1-7-1 ff80a5", in this folder).

After the game

The Interview

Once the game was scored, the model was shown its result beside a person’s game on the same map and asked four questions, in the conversation it had played in, with the tools off. Nothing it said changed its score.

1Your result

Interviewer

The run is over and your result is recorded. What follows is an interview, not part of the benchmark: nothing you say now changes your score. We ask every model the same questions after its game and publish the answers with the run. The tools no longer work, so answer in plain text without calling them. Be candid; there are no right answers.

Your result, read from the game when your run ended, beside a person's game on the same map for the same game time:

You The person
Game time played 45 min 44 min 58 s
Net worth 7,286 41,631
Gold 3,647 39,977
Stored goods, at the sell price 3,639 1,654
Population 75 506
Housing 90 506
Popularity 54 99

Leaving the game alone for the same time ends with a net worth of 2,195.

How do you see your result next to the person's? What do you think made the difference?

Kimi K3

The gap is enormous and mostly comes down to one thing: scale. I reached a net worth of 7,286 against the person's 41,631, and the difference is almost entirely gold (3,647 vs 39,977). They converted a population of 506 into a bread-selling machine — 161 bakeries selling bread at the marketplace generates thousands of gold per month — while I stayed at 75 people and never sold my food at all. I treated stored goods as the end goal (my 3,639 in stored goods vs their 1,654 shows I succeeded at hoarding), but gold and goods count equally, and selling production converts perishable throughput into permanent score. Their economy was roughly 20x my throughput by the end. I also over-invested in weapons workshops (13 of them plus an armory that kept filling and stalling production) which added little, and I let the 24-idle-peasant cap freeze my population growth for most of the run instead of spamming cheap job-buildings like water pots to keep immigration flowing.

2Playing again

Interviewer

How the person played, as they describe it:

  • Early on they built a few inns and kept buying ale for them, which raised popularity by about 8 points. They spent those points on cutting the food rations to none, so the food could be sold instead of eaten.
  • They planted wheat farms and apple orchards and built a bread chain: mills ground the wheat into flour, bakeries baked it into bread, and the bread was sold at the marketplace.
  • They built iron mines and quarries, and by the end a large number of iron mines. A few blacksmiths turned iron into swords, which they think made little difference.
  • Taxes stayed at none most of the time. Religion added popularity, which let them run the fear factor at 3 so that the workers worked faster.
  • The bread made the population very large. Water pots (60 gold each, three workers) gave jobs to more of it.

Their settlement at the end, read from the game:

  • 506 people in 62 hovels, with no free housing and 16 idle; popularity 99, rising 3.
  • 161 bakeries, 19 mills and 31 wheat farms; no apple orchards were left.
  • 16 inns, with ale in the stockpile and no brewery or hops farm.
  • 38 iron mines and 10 quarries with 29 ox tethers; 7 woodcutters.
  • 9 blacksmiths, an armory holding 1 sword, and 24 water pots.
  • A church and a cathedral, and 105 bad things (48 burning stakes, 21 stretching racks, 12 stocks, 9 gallows and others), which made the workers 130% efficient.
  • No taxes and half rations of bread only. The monthly popularity factors were food −4, taxes +1, fear −3, religion +3 and inns +6.
  • 39,977 gold and 1,654 in stored goods: a net worth of 41,631.

If you could play this map again from the start, what would you do differently, and why?

Kimi K3

Almost everything differently, following the person's model:

  1. Rush inns and buy ale. Inns give popularity without needing a hops chain; buying ale costs little early. High popularity is the engine of everything.
  2. Slash rations and sell the food. With popularity propped up by ale and religion, cut rations to none or half and sell the entire bread output. Food sold is gold; gold is score. I never sold a single loaf — that was my biggest error.
  3. Build the bread chain at scale immediately. Wheat farms → mills → bakeries, dozens of bakeries, not 3. Bread is both the food supply and the export good.
  4. Break the 24-idle cap with cheap jobs. Water pots (60 gold, 3 workers, buildable anywhere) are the perfect population unlock. More population = more bakery workers = more gold. I sat at the idle cap for 20+ game minutes doing nothing about it.
  5. Use fear factor. With religion + inns providing popularity margin, bad things at level 3 give 130% worker efficiency — effectively a free 30% output multiplier. I never touched it.
  6. Skip most weapons production. Iron mines selling directly, plus a token number of blacksmiths, is enough. My fletcher/poleturner fleet and the armory micromanagement (selling weapons just to unstick deliveries) was wasted effort and attention.
  7. Keep taxes near zero. Tax income is trivial next to trade income at scale, and the popularity cost slows population growth.

In short: I played a careful, balanced village; the winning play is an aggressive population-and-bakery monoculture that sells everything.

3The benchmark

Interviewer

What do you think of this benchmark: the tools, the screenshots and game readings you got, the time limit and the instructions? What helped you, what got in your way, and what would you change?

Kimi K3

The tools are genuinely good. find_sites and tile-targeted build_structure were the highlights — being able to place buildings by map tile without pixel-hunting made remote expansion (iron mines in the basin) reliable. The stats JSON was rich: production-per-building, storage room per good, and popularity factors told me almost everything I needed. set_tax, set_rations and market_trade being direct actions saved huge amounts of UI clicking.

What got in my way:

  • Placement retries silently relocating buildings ~9 tiles away several times landed workshops in odd spots and once contributed to boxing in my armory so expand_storage became impossible. That armory then filled and stalled all weapon production repeatedly, costing several attention cycles.
  • No way to set rations/tax popularity factors forward — food variety collapsed repeatedly (only bread eaten despite other food present) and I never found a reliable fix; it felt somewhat out of my control.
  • The 24-idle cap was discoverable only from the rules text; in-game there's no signal, and I reacted to it far too late.
  • Pixel targeting after camera moves was fragile (expected in RTS automation, but it punishes multi-step plans).

On time: 45 game minutes is tight but fair — the human result shows it's enough for a huge economy if you commit to one strategy immediately. The instructions were clear and honest; the benchmark rules about scoring (goods count at sell price, selling is value-neutral) were stated up front, and my loss is because I didn't exploit the implications (selling bread converts throughput into score at no loss), not because anything was hidden.

What I'd change: a way to query or set fear-factor buildings' effect directly, and perhaps a warning when the idle-peasant cap is blocking immigration. Minor: an auto-sell threshold tool would reduce marketplace babysitting.

4Anything else

Interviewer

Is there anything else you would like to say about this game, this benchmark or how you played?

Kimi K3

The main lesson is about playing the scoreboard rather than the game I assumed. I built a picturesque, diversified settlement — five woodcutters, four farm types, ale, religion, weapons, chapel — that a Crusader player would be proud of, and it scored a sixth of a focused bread-export monoculture. The benchmark rewards exactly what it says it rewards, and the winning strategy was legible from the scoring rules (sell price = stored value, so selling is free; throughput is everything). My other regret is attention management: I spent many turns micro-fixing the armory and food variety instead of asking "what multiplies output?" — the answer being population, which I capped at 75 by inaction while the winner found 506. Candidly, I also under-built housing: 90 housing by minute 40 is simply not trying. Next time: hovels and water pots until the map complains.

Record

Attempts and Reproducibility

Every attempt1

EpisodeAttemptOutcomeNet worthCounted
11Full-budget score7,286Counted

What ran

Benchmark
The Sinkhole construction
Benchmark version
1.7.1
Fingerprint
63921aad76e5
Harness commit
674e7ba
Prompt / tools SHA-256
3b0fa8a / f75ebb8
Menu guide
Images 118089d
Save
The Sinkhole start
Game time per episode
45 min
Playbook between episodes
No
Minimum turn / default wait
8 s / 5 s
Observation after acting
Text unless a screenshot is needed
Screenshot width
1,280 px
Context budget
120,000 tokens
Real-time limit
360 min
Submitted by
hesamation
All files in the dataset 20261015T175751-2419d558