Skip to content

Versioned snapshots

A dated run must read what that date would have read. Otherwise the analysis is using hindsight — and not the obvious kind, in the predictor, but the subtler kind in the sampling frame, where a gene is scored because a later source connected it to something.

The snapshot layer exists so that a run over a 2018 release reads 2018.

Derived extracts, not mirrors

Upstream releases are streamed, reduced to a compact Parquet extract, and discarded. What persists is four columns of gene–disease evidence and a table of directed regulatory interactions — enough to rebuild the causal subgraph as it would have looked at a past date, and small enough that a decade of releases costs less disk than one of them.

>>> from neorx.snapshots.schema import ASSOCIATION_COLUMNS, INTERACTION_COLUMNS
>>> list(ASSOCIATION_COLUMNS)
['target_id', 'target_symbol', 'disease_id', 'datatype', 'score']
>>> list(INTERACTION_COLUMNS)
['source_symbol', 'target_symbol', 'is_directed', 'consensus_direction',
 'is_stimulation', 'is_inhibition', 'primary_sources', 'references']

Three Open Targets format eras feed this: a flat gzipped JSON dump in 2018, an ETL-era Parquet tree in 2021, and the current output layout. Each gets its own reader, and each reader's job is to emit exactly those columns. Downstream code never learns which era a row came from — that is the whole value of a derived extract over a mirror. Format drift is absorbed once, at the edge, instead of leaking into every consumer.

The per-era field mappings are recorded in OpenTargets release layouts, characterised against the live archive rather than inferred.

Only genetic evidence reaches the frame

The candidate frame for a dated run is drawn from the pinned release alone, and only from the datatypes that confer causal admissibility:

>>> from neorx.snapshots.schema import GENETIC_DATATYPES
>>> sorted(GENETIC_DATATYPES)
['genetic_association', 'somatic_mutation']

That is the same set as GENETIC_EVIDENCE_CLASSES, and for the same reason. A gene supported only by literature co-mention is not in the frame, because such an edge is not admissible as a causal arrow and the gene has nothing for identification to reason about.

Adding a name to GENETIC_DATATYPES silently widens both the frame and what the backdoor criterion may reason over.

The same principle applies one level up, to the disease corpus. Selecting diseases because they look well studied today and then analysing them as of 2018 is the same leak, so both criteria are applied to the pinned release.

Building one

neorx snapshot build opentargets 21.11
neorx snapshot build omnipath 20211113-20220114
neorx snapshot list

build fetches the release, reduces it to the canonical schema, writes the Parquet extract, and writes the manifest entry beside it.

The manifest

A release identifier alone does not identify an extract. Two extracts of Open Targets 18.06 produced by different extractor code are different inputs to every number computed downstream, so the extractor version is recorded next to the release id, and the SHA-256 digest is taken over the extract actually written rather than over the upstream file.

>>> from neorx.snapshots.manifest import SnapshotEntry
>>> from neorx.snapshots.schema import EXTRACTOR_VERSION
>>> EXTRACTOR_VERSION
2
>>> entry = SnapshotEntry(
...     source="opentargets", release="21.11",
...     url="https://ftp.ebi.ac.uk/pub/databases/opentargets/platform/21.11/",
...     sha256="0" * 64, extractor_version=EXTRACTOR_VERSION, rows=1234,
... )
>>> entry.source, entry.release, entry.rows
('opentargets', '21.11', 1234)

EXTRACTOR_VERSION is bumped when a reader's output changes meaning for the same input. Version 2 exists because the OmniPath archive reader now filters to human–human interactions: 49% of the 2018 dump is mouse and rat (242,338 and 73,989 rows of 644,845), and nothing downstream marked them. An extract built before that change and one built after are different inputs for the same release, and the manifest has to be able to say which you have.

Entries are keyed by (source, release), so re-running an extraction replaces its entry rather than appending a second one.

What a reader had to do, that the rows no longer show

>>> from neorx.snapshots.schema import ExtractNotes
>>> notes = ExtractNotes(consensus_direction_synthesised=True,
...                      non_human_rows_dropped=316327)
>>> notes.consensus_direction_synthesised
True
>>> notes.non_human_rows_dropped
316327

Both facts are recorded rather than defaulted away, because both change what the extract means and neither is visible in the extract itself. consensus_direction_synthesised says the 2018 archive had no such column and is_directed was used in its place — a weaker criterion than later eras publish. And a non_human_rows_dropped of zero on a real OmniPath archive means the filter did not fire, not that the dump was human-only.

Reading them back

>>> import tempfile
>>> from pathlib import Path
>>> from neorx.snapshots.reader import SnapshotStore, SnapshotMissingError
>>> with tempfile.TemporaryDirectory() as tmp:
...     store = SnapshotStore(Path(tmp), on_read=None)
...     store.has("opentargets", "21.11")
...     try:
...         store.associations("21.11")
...     except SnapshotMissingError as exc:
...         print("refused:", "will not fall back to live data" in str(exc))
False
refused: True

A missing snapshot raises rather than returning an empty frame. An empty frame reads downstream as "this release recorded no evidence for that disease", which is a finding. A missing extract is a configuration mistake. The two must not be confusable, and the error names the exact neorx snapshot build command that fixes it.

on_read is how a run record learns what a run actually read, so it can cite exactly that — from this store's own manifest, rather than from a second path that might name a different store. It has no default, deliberately: when it defaulted to None an experiment could read one extract through RunRecord.snapshot_store and another through a store it built itself, and finish complete and citable while citing only the first.

Resolving a date

SourceResolver maps a source plus an optional date to a reader. With no date, every source resolves to the live client it always used, so interactive single-disease use is unaffected by any of this.

>>> from neorx.snapshots.resolver import PINNED_SOURCES
>>> sorted(PINNED_SOURCES)
['omnipath', 'opentargets']

Only those two are pinned, because only those two reach the causal subgraph. Sources contributing associational edges stay live even on a dated run: identification filters those edges out before it runs, so pinning them would buy rigour against a leak that does not exist.

A dated run that asks for a source with no snapshot at that date is refused by name:

>>> from neorx.snapshots.resolver import SourceResolver, UnpinnedSourceError
>>> from neorx.snapshots.releases import release_for
>>> import tempfile
>>> from pathlib import Path
>>> with tempfile.TemporaryDirectory() as tmp:
...     store = SnapshotStore(Path(tmp), on_read=None)
...     resolver = SourceResolver(store=store, release_for=release_for, live={})
...     try:
...         resolver.resolve("opentargets", "2021-11")
...     except UnpinnedSourceError as exc:
...         print("21.11" in str(exc), "fall back to live data" in str(exc))
True True

The refusal matters more than it looks. A dated run that quietly mixed in current data would produce exactly the anachronistic graph this layer exists to prevent, and would do it invisibly.

A limitation recorded rather than smoothed over

The pinned release table pairs each date's Open Targets release with the OmniPath dump whose window contains it. For 2018 and 2021 the two are within days of each other. For 2025 they are not:

>>> from neorx.snapshots.releases import TIME_POINTS
>>> TIME_POINTS["2025-06"]["opentargets"]
'25.06'
>>> TIME_POINTS["2025-06"]["omnipath_data_as_of"]
'2023-07-28'

OmniPath published no refresh between 2023-07-28 and 2025-08-13, so the 2025 time point pairs a 2025 association release with regulatory data roughly 23 months older. omnipath_data_as_of is recorded separately from the archive name precisely so that this is visible, and it belongs in the limitations section of anything that reports a 2025 result.

Every entry in the table was verified against the live archive on 2026-09-06 by listing it, not by reading a spec.