Docs · Operate

Reproduce a run

Every number on the leaderboard ships with a bundle. These commands rebuild the site and replay the harness from a clean checkout.

Set up

Python 3.12 or newer and uv. The pinned Mnemosyne revision is installed as a dependency.

uv sync --locked --extra test

Rebuild this website

The site is rendered from retained, verified inputs. Every digest and signature is checked before a page is written.

uv run --locked python build_site.py
python -m http.server 8792 --directory site

Run and verify a bundle

The smoke suite is a redistributable self-test. It is never publishable, but it exercises the same bundle and verification path as a real run.

uv run --locked mneme eval-public --suite smoke --out-dir out/smoke
uv run --locked mneme eval-public --verify-bundle out/smoke
uv run --locked mneme eval-public --reproduce-bundle out/smoke --out-dir out/smoke-again

What a bundle contains

traces.jsonlWhat was stored, retrieved and answered for each question

metrics.jsonEvery metric with its interval method and seed

config.jsonThe exact run configuration

build.jsonEnvironment contract and system seam

judge.jsonReader and judge disclosure, or none

reproduce.shThe one command that regenerates the number

Large evaluation datasets stay outside the repository and are identified by SHA-256; obtain the approved dataset before reproducing a run.