Publishing Runs

Steps

pip install datasets                      # Parquet tables and the Hugging Face client
npm run episodes -- --record ...          # a series of episodes, recorded (see The benchmark)
npm run package -- <series id> --submitter <hugging face user> [--tier official]
npm run upload -- harness/runtime/publish/v1.0/<benchmark>/<model>/<series id>        # dry run
npm run upload -- harness/runtime/publish/v1.0/<benchmark>/<model>/<series id> --yes  # upload
npm run upload -- --evaluation --yes      # after editing tools/dataset/evaluation.json

1. Package. npm run package takes learning series ids (harness/runtime/series/<id>) or run folders, checks them, and stages each one with its tables and a manifest.json. A bundle with problems is still staged so you can look at it; the problems are printed and listed in the manifest.
2. Review. Watch the videos and read manifest.json: what the scrub changed, and the problems.
3. Upload. npm run upload prints what it would send. With --yes it makes one commit per bundle; with --pr it opens a pull request instead, which is how anyone without write access to the dataset submits runs. --tier official is for runs made by the maintainers; everyone else's are community. --replace replaces a bundle already in the dataset, for one packaged again after a packaging change.
4. Count it. A published series is not ranked until its id is in tools/dataset/evaluation.json, which --evaluation uploads to the dataset's root. The upload refuses a list that names a series the dataset does not have. Leave a series out to publish it without ranking it: an early run, a test, or one set aside.

The token is HF_TOKEN (environment or .env), else the login from hf auth login. The dataset is --repo ORG/NAME or HF_DATASET_REPO.

What Can Be Published

Packaging records a problem, and the upload refuses the bundle, unless:

• the series is completed and every episode has a result (below);
• every episode's result is a full-budget score (valid in its episode.json: completed, ended by the game-time budget, final pause confirmed, scored from the final reading; invalid says why not) or a run the model itself ended (outcome model_failure, published with valid false);
• every episode has video.mp4 (record with --record);
• the runs' code was committed (harness.dirty is false) and every commit is on origin/main as last fetched, so anyone can see the code that ran;
• the runs carry a benchmark version (below);
• all episodes of a series share the benchmark minor version and preparation guide, the prompt and tool hashes, the model and its settings, and the run settings;
• the model endpoint is not a local address, and --submitter gives a Hugging Face user name;
• the scrub (below) found nothing to report.

The upload also refuses a bundle without a version, and one whose files changed after packaging: every file's size and SHA-256 must match manifest.json. It replaces a bundle already in the dataset only with --replace, which deletes the old folder in the same commit, so files the new bundle no longer has go too. The old files stay in the dataset's history.

Versions and Attempts

The host stamps every run with the benchmark version its code belongs to (run.json benchmark.version, from benchmark-versions.json; see Versions), MAJOR.MINOR.PATCH. It is null when the code's fingerprint is not a listed version; such runs are staged under unversioned/ and cannot be uploaded. Runs compare within one minor version, so bundles are staged under v<major>.<minor>/ and the exact version is in the dataset_version column; the episodes of a series may differ only in the patch.

An episode of a series can run more than once: a stop or an infrastructure failure is run again. The series keeps an episode's last attempt that is a full-budget score or a model failure as its result (harness/server/series.ts), and that is the run published; earlier attempts are not published and are not problems. An episode without a result yet, or a series that is running or paused, is. The series folder's runner.log is not published.

Scrubbing

Every text file a bundle publishes (.json, .jsonl, .md) is scrubbed by tools/dataset/scrub.py:

• Replaced with placeholders: the game machine's SSH target and address (<game-host>) and source path (<game-root>) from .env, and this machine's home directory (<home>) and host name (<hostname>). Only values with a /, ., @ or : are replaced; a plain word such as a short host name could be ordinary text, so it is reported instead.
• Dropped: pid and windowId (the game's process and window on the game machine), system and graphics in memory samples (machine-wide memory, the graphics driver and PCI device), and directory fields holding an absolute path (the recorder logs where it kept its frames).
• Reported, which blocks the upload: the API keys in .env and saved model profiles, other key-shaped strings (sk-, hf_, GitHub and AWS keys, bearer tokens, private keys), the local and game-machine user names, home and user runtime paths, IP addresses other than loopback, and email addresses. Findings name the file, line and kind, never the value.

Screenshots are base64 inside the logs; strings of 512 or more base64 characters, and data:image URLs, are skipped, so letters inside an image are never a finding. Lines that need no change are copied byte for byte. If text the video is drawn from (run.json, episode.json, events.jsonl) was replaced, the video is rendered again from the scrubbed files and the original recording/ (JPEGs, or the frames.mp4 they were compacted into); without recording/ that is a problem.

Smaller Than the Run

After the scrub, packaging makes each episode smaller (tools/dataset/media.py). The run folder keeps the originals.

• Images leave events.jsonl. Every image the log holds as base64 (the screenshots, the preparation message's reference images and the final frames) is written once to images/<hash>.webp: WebP at quality 70 and full size, or the original bytes when those are smaller. An image block {"type": "image", "data": …} becomes {"type": "image", "mimeType": …, "path": "images/…"}, and any other field that held an image (the final frames' frame.image) holds the path. A tool's screenshot appears in a run's own events.jsonl under the tool's result and again at the start and end of the message that carries it, but the host keeps the image data in the result only; the other two already hold a small {"type": "image", "mimeType": …, "omitted": "see tool_execution_end"} in its place, so there is only one copy to turn into a file (runs recorded before this, with all three in full, still end up as one file: the same bytes hash the same).
• Token deltas are dropped. message_update events stream a reply as it is written; the reply's message_end holds all of it.
• The video is 720p, scaled from the run's 1080p render (after re-rendering it, when the scrub changed the logs it is drawn from) with the renderer's encoder settings.

manifest.json counts what this did under slimmed.

The Dataset

v<major>.<minor>/<benchmark>/<model ID with "/" as "--">/<series id>/
  series.parquet  series.json  playbook-after-episode-<n>.md  manifest.json
  episode-<n>/
    episode.parquet  timeline.parquet  final-overview.jpg  video.mp4
    run.json  inputs.json  episode.json  events.jsonl  logs.jsonl  notifications.jsonl
    memory.json  notebook.md  playbook.md  interview.md  interview.json  results.json  results.md
    analysis.md
    images/<hash>.webp

A single run that is not part of a series is staged as run-<start time>-<id>/episode-1/. Not published: recording/ (the raw frames, or the video they were compacted into; the rendered video replaces either), checkpoint.json (the conversation again) and controls-guide.html.

The dataset card defines three tables, read by the Hub's viewer and by load_dataset:

Column lists are in the dataset card. The numbers in series and episodes come from the same code as npm run report (harness/server/run-report.ts), so the two always agree.

The timeline is read from events.jsonl: the stats in each host observation (stamped with the budget clock), reader samples in tool results (stamped with a game tick, placed on the budget clock from the final reading) and the final reading (from episode.json when the log's final reading came back empty). The grid follows the budget. Row m is the latest reading at or before m game minutes; row 0 is the first reading and the last row the final one. Readings come from the agent's turns, so a minute without a fresh reading repeats the one before, and sample_game_seconds shows when it was taken. Net worth uses the marketplace sell prices in harness/server/market-prices.ts.

The Website

The project website is a static site built from this dataset and from this repository's ROADMAP.md, CHANGELOG.md and docs/, each read at its newest commit when the site is built. Its leaderboards and charts use only the series in evaluation.json, and of a model's series on one map only those on the newest patch; the others keep their run pages. A Vercel deploy hook rebuilds it. These call it:

• This repository. .github/workflows/notify-website.yml calls the hook on a push to main that changes docs/, ROADMAP.md or CHANGELOG.md, and by hand from the Actions tab. It reads the hook's URL from the Actions secret WEBSITE_DEPLOY_HOOK and does nothing in a fork.
• The upload. npm run upload -- --yes to hesamation/crusader-arena-runs calls it once, after its last commit, so an upload appears on the site a minute or two later and a build never sees half of one. It uses WEBSITE_DEPLOY_HOOK from .env when it is set, and otherwise runs the workflow above with the GitHub CLI (gh, signed in with the workflow scope). A dry run, a pull request (--pr) and an upload to another dataset do not. The dataset has no webhook: one fires on every commit and again on each of the Hub's own Parquet conversions, about ten builds an upload.
• The website's own repository, on every push.

The hook can only rebuild the website's main; it cannot change its code or content. If its URL leaks, make a new hook in Vercel and put it in the secret, and in .env where it is set.

Checks

python3 -m unittest discover -s tools/dataset

The tests build a small series and check the layout, the tables, the timeline, the scrub, the images and token deltas leaving the logs, the video scaled to 720p, the problems that block an upload, and the upload itself, replacing a bundle included, against a fake Hub client. One test runs npm run report's code on a made-up run to check the fields packaging reads.