Experiments and run records¶
This is the part of NeoRx that exists because of a failure, and the failure is worth stating plainly, because the design only makes sense against it.
Why the runner owns persistence¶
A manuscript audit of this codebase found that no reported number in four papers could be traced to an execution artifact. The defects were all in scaffolding rather than algorithm code:
benchmark_paper.pyproduced the seven-disease results behind four tables and persisted nothing — nojson.dump, nowrite_text. Every number existed only in terminal scrollback._bench_paper2.pysaved only after all twelve environment × agent cells finished. It timed out on the fourth, so 881 seconds of real measurement was lost and later retyped by hand into another script as a literal dict. That transcribed −619.0 is what a published table reports; the one surviving raw file gives −587.8, from a different run._gen_figures.pyheld the F₁ values as hardcoded literals, so a published figure was a transcription of a table rather than a rendering of data.results/andreports/were both gitignored, so even what was saved was invisible to version control.
The common thread: a number could exist without a record of the run that produced it. This subsystem makes that impossible going forward.
It does not make the published numbers reproducible retroactively.
Nothing can; a v0.1.0-paper tag preserves the state the manuscripts
describe.
The enforcement model¶
The runner owns persistence. An experiment is a function taking a
RunRecord; it never opens a file and never decides where results go.
There is no code path by which an experiment reports a result without a
record.
This is deliberately stronger than a context manager an author opts into. A voluntary mechanism is exactly what the earlier state already had available and did not use.
>>> import tempfile
>>> from pathlib import Path
>>> from neorx.experiments.registry import experiment, run_experiment
>>>
>>> @experiment(name="docs-demo", help="Two cells, one record.")
... def demo(record):
... for n in (1, 2):
... record.append_row({"cell": n, "value": n * 0.5})
...
>>> with tempfile.TemporaryDirectory() as tmp:
... record = run_experiment("docs-demo", runs_dir=Path(tmp))
... status = record.status
... rows = record.rows()
... files = sorted(p.name for p in record.path.iterdir())
>>> status
'complete'
>>> rows
[{'cell': 1, 'value': 0.5}, {'cell': 2, 'value': 1.0}]
>>> files
['env.json', 'inputs', 'record.json', 'rows.jsonl']
What a record contains¶
runs/2026-09-03-neorx-7disease-a3f9c1/
record.json summary, status, citable flag, rows digest
rows.jsonl appended per cell as it completes
env.json code SHA, dirty flag, python, deps, platform, snapshots
inputs/ frozen API responses (vcrpy cassettes)
Run directories default to not tracked, so a routine git add -A
cannot sweep in an ad-hoc or smoke run. A record that actually backs a
claim is committed deliberately with git add -f runs/<run-id>, and the
check_committed_records_are_citable gate then holds it to being
citable.
>>> import json
>>> with tempfile.TemporaryDirectory() as tmp:
... record = run_experiment("docs-demo", runs_dir=Path(tmp))
... env = json.loads((record.path / "env.json").read_text())
... summary = json.loads((record.path / "record.json").read_text())
>>> sorted(env)
['deps', 'git_dirty', 'git_sha', 'platform', 'python', 'snapshots',
'snapshots_mismatched', 'snapshots_uncited', 'started_utc']
>>> sorted(summary)
['citable', 'experiment', 'finalised_utc', 'n_rows', 'rows_sha256',
'run_id', 'size_bytes', 'status']
Three properties matter more than the format.
Rows append as they complete¶
rows.jsonl gains a line the moment each cell finishes. This is the
direct fix for the lost-881-seconds failure: a crash now costs the
current cell, not the run. record.json is written at finalisation.
A dirty tree marks a run non-citable¶
env.json records the code SHA and whether the working tree carried
uncommitted changes. A run from a dirty tree cannot be reproduced by
anyone, so record.json sets citable: false.
>>> with tempfile.TemporaryDirectory() as tmp:
... record = run_experiment("docs-demo", runs_dir=Path(tmp))
... env = json.loads((record.path / "env.json").read_text())
... # citable requires status 'complete' AND a clean tree
... env["git_dirty"] is False or record.citable is False
True
Provenance that cannot be established is a failure, not a blank field:
>>> from neorx.experiments import ProvenanceError
>>> issubclass(ProvenanceError, RuntimeError)
True
A run whose git SHA or dirty state cannot be determined raises rather than recording a fabricated SHA or a false "clean".
A record has a size ceiling¶
250 MB, from a measurement: ~7 MB per disease on a live seven-disease
run, so seven diseases is about 49 MB. The original spec estimate was
"5–6 MB per seven-disease run" — roughly eight times too low, which
would have meant hitting refusal after ~25 minutes of live API cost was
already sunk. --allow-large overrides it deliberately.
>>> from neorx.experiments import RecordTooLargeError, RunIDCollisionError
>>> issubclass(RecordTooLargeError, RuntimeError)
True
>>> issubclass(RunIDCollisionError, RuntimeError)
True
A run ID collision refuses to write into an existing run's directory rather than merging into it.
Snapshot citations¶
If an experiment reads pinned snapshots, the runner cites exactly what was read — from that store's own manifest.
>>> from neorx.experiments.registry import experiment
>>> @experiment(name="docs-pinned", reads_snapshots=True,
... help="Declares that its numbers come from extracts.")
... def pinned(record):
... store = record.snapshot_store(Path("snapshots"))
... # ... read through `store`, never a SnapshotStore built directly
...
>>> from neorx.experiments.registry import get_experiment
>>> get_experiment("docs-pinned").reads_snapshots
True
Two refusals, both of which this project has actually hit:
- A read with no manifest entry. The extracts were there and the manifest describing them was not, so the run produced numbers from inputs it cannot identify. This is the failure found in four papers.
reads_snapshotsdeclared but no reads recorded. The experiment opened a store some way other thanrecord.snapshot_store, so nothing it read was tracked.
What the check deliberately does not accept is "some citation exists". An earlier version did, and a run over the 18.06 release completed while citing only a 25.06 entry that happened to be in the manifest.
Each citation is the whole manifest entry, not a digest. A digest pins which bytes an extract was, not what they meant — and the manifest that says what they meant is replaced in place when an extract is rebuilt, so a record keeping only the digest would, months later, cite something nothing can interpret.
Replay¶
A record can be re-executed against its frozen inputs:
The claim the subsystem rests on is that anyone can reproduce a run's numbers without reaching the network. A replay that differs is not a failure of the tool — it names exactly which values moved, which is what a reviewer needs.
HTTP traffic is frozen with vcrpy, which patches at the connection level.
That is not a stylistic choice: the source clients call module-level
requests.get/post at thirteen sites and construct a Session per
call, so a transport adapter mounted on a Session would intercept
nothing.
Volatile fields¶
Wall-clock timings differ on every replay. Each experiment declares them, and they are excluded from the IDENTICAL/DIFFERS verdict but still reported:
>>> import sys
>>> sys.path.insert(0, ".") # the repository's own experiments/
>>> import experiments # registers the definitions
>>> from neorx.experiments.registry import get_experiment
>>> get_experiment("neorx-7disease").volatile_fields
('wall_clock_s', 't_graph', 't_identify')
>>> get_experiment("neorx-7disease").captures_http
True
A gate must never swallow information, only decide what does and does not fail a verdict.
The registered experiments¶
>>> from neorx.experiments.registry import list_experiments
>>> this_repository = {"neorx-7disease", "causalbiorl-bench", "genmol-eval",
... "dated-build", "corpus-census", "figures"}
>>> this_repository <= {d.name for d in list_experiments()}
True
(A subset check rather than an exact list: the registry is process-global, so anything else registered in the same interpreter — this page's own examples above, the test suite's fixtures — would be in it too.)
| Experiment | Measures | Freezes HTTP | Reads snapshots |
|---|---|---|---|
neorx-7disease |
Causal target benchmark across 7 diseases | yes | no |
causalbiorl-bench |
RL agent comparison across control envs | no | no |
genmol-eval |
GenMol generation quality metrics | no | no |
dated-build |
Graphs from pinned releases; exclusions and timing | no | yes |
corpus-census |
Per-release disease counts | no | yes |
figures |
Renders manuscript figures from a run record | no | no |
figures is the one that closes the hardcoded-literals hole: it takes a
run ID and renders from that record's rows, so a figure is a rendering of
data by construction.
The definitions are not in the wheel
neorx.experiments — the runner — ships, so an installing user can
record their own runs. The experiment definitions live in the
repository's experiments/ package and are deliberately not
shipped: they are this project's experiments, not a library feature.
On a clean install neorx exp list says so by name, rather than
listing zero experiments — an empty list looks like an answer, and
the honest answer is that the definitions live somewhere the
installation cannot see.
Using it¶
neorx exp list
neorx exp run neorx-7disease
neorx exp show 2026-09-03-neorx-7disease-a3f9c1
neorx exp figure --from 2026-09-03-neorx-7disease-a3f9c1
neorx exp replay 2026-09-03-neorx-7disease-a3f9c1
neorx exp prune 2026-09-03-neorx-7disease-a3f9c1
prune drops a run's frozen inputs and keeps its recorded results, for
when the cassettes are no longer worth the disk. A pruned run is marked
unreplayable and says so when you try.
The CI gates¶
neorx.experiments.gates holds the checks that make "no number without a
record" enforceable:
| Gate | Closes |
|---|---|
find_hardcoded_metrics |
A figure script holding metric values as literals |
check_records_wellformed |
A record whose summary disagrees with its rows |
check_cited_runs |
A manuscript citing a run that does not exist or nobody can reproduce |
check_committed_records_are_citable |
A non-citable record committed anyway, adding weight while backing no claim |
check_record_integrity |
rows.jsonl edited after the fact, by recomputing the SHA-256 stored at finalise |
check_no_fabricated_statistics |
The specific fabricated-statistic names this project has previously shipped |
What these gates do not claim
find_hardcoded_metrics matches bare float literals and
metric-shaped string literals. A metric assembled by computation —
545 / 1000, round(x, 3), an f-string, a dict lookup — is
invisible to an AST literal check by construction, not by an
oversight fixable there. A clean run means "no metric was typed in as
a literal"; it does not mean "no computed metric literal exists".
check_record_integrity detects accidental edits and drift. It does
not detect a determined forger who edits rows.jsonl and
rewrites rows_sha256 to match. That would require signing, and this
gate is not that.
Both limits are written into the module's own docstring, where someone reading a green gate will find them.