The data
Data sources
Everything this project reads or produces, with provenance and how to regenerate it. The
machine-readable registry is src/exaproof/io/datasets.py (KNOWN); this file is the prose
account. Written for the move to a new machine — the verification commands at the bottom are
what "the data survived the copy" means.
Read §2f Table A first, and treat it as the only inventory. This file grew by appending a dated block each time the buckets changed under us, so its sections run 1, 2, 2b, 2e, 2f, 2c, 2d, 3 — and one of them is marked "SUPERSEDED the same day" by a block below it. §2f, "The full inventory", is the reconciled state: every brain we have a name for, what exists for it, and which names do not resolve. The blocks before it are the discoveries that produced it, kept for their provenance, and any of them may be a snapshot of a state that lasted hours.
This is not a style complaint. A stale row in an earlier block hid a whole brain from an analysis for the life of that analysis — 609281 was listed as "gold only" while its store held 737,085 fragments — and four RESULTS sections carried a caveat about a missing labelling cache that was never missing. If a row here disagrees with §2f, §2f wins; if you are about to write "brain X has no Y", check the bucket rather than this file.
The live check is
exaproof data status, and the search commands areexaproof data index-s3.
1. Remote sources
s3://aind-open-data — the public bucket, and the only one anything is redistributed from
AWS Open Data Program, anonymous read (no credentials needed; the code reads via
fsspec/s3fs with anon=True). Three kinds of object per exaSPIM subject:
| kind | key pattern | format |
|---|---|---|
| imagery | <fusion_prefix>/fused.zarr |
OME-Zarr (zarr v2), 5D TCZYX, 7-level pyramid, blosc-zstd, 256³ chunks; level 0 at 0.748 × 0.748 × 1.0 µm/voxel |
| gold tracings | <reconstructions_prefix>/… |
SWC, µm, two layouts (below) |
| automated fragments | <fusion_prefix>/seg/<seg_key> |
zip of SWC skeleton fragments from FFN (653158: 1.48 M fragments) |
Subject registry (from KNOWN; prefixes abbreviated — full strings live in the code).
This table and Table A in §2f disagree in four rows, and Table A is the later one. The
imagerycolumn here predates the upstream deletions and still reads ✓ for 653158, 653980 and 648434, where only the denoised twin reads (§2b); 609281 reads "stitched only" here and "nothing readable" in §2b; 721332 is "—" here and "raw" in Table A, and the code agrees with this table (live_for_subjectraiseshas no fused volume in s3); and 685221's archive is 7.4 GB here against 22 MB in Table A, a factor of 336 that nobody has resolved — do not quote either without checking the bucket. Left as a disagreement rather than silently picked, which is the convention for everything else in this repository.
| subject | imagery | gold arbors | fragments archive | end-to-end? |
|---|---|---|---|---|
| 653158 | ✓ | 45 (Complete_raw) |
seg.zip (2.0 GB) |
✓ |
| 653980 | ✓ | 21 (Complete_raw) |
swcs-processed.zip |
✓ |
| 648434 | ✓ | 12 (specimen_space_reconstructions) |
648434_swcs.zip (283 MB) |
✓ |
| 609281 | stitched only | 25 (Complete_raw) |
seg.zip → 737,085 fragments ingested |
✓ |
| 685222 | ✓ | 38 (specimen_space) |
corrected-swcs.zip 4.1 GB (other bucket) |
✓ 2026-09-18, coverage 0.816 |
| 789202 | ✓ | 14 (gold/, other bucket) |
original_swcs.zip 305 MB (other bucket) |
✓ 2026-09-18, coverage 0.921 |
| 685221 | ✓ | 60 (specimen_space) |
corrected-processed-swcs-s3.zip 7.4 GB (other bucket) |
✓ 2026-09-18, coverage 0.889 |
| 651324 | ✓ | 13 (Complete_raw) |
— | imagery+gold only |
| 721332 | — | 22 (specimen_space) |
— | gold only |
Seven subjects support the split/merge task (END_TO_END_SUBJECTS); 685221, the seventh, finished
aligning at 19:59 PDT on 2026-09-18. Three of them were found by widening a search rather than by any measurement, and all three
had been recorded here as unusable:
609281's row said "gold only" and was wrong for the life of the project. Its fragment store was ingested from
raw/609281/seg.zip— 737,085 fragments, 41.9 M nodes — andmeta.jsonhad said so since. The table was never updated, so a fourth end-to-end brain sat unused until §226 went looking. It is also the hardest of the four: 41.7 % of its gold cable has no fragment, the highest pendant share (44.1 %) and the fewest simple spans (17.9 %).685221 and 685222 were wrong the same way, for a different reason. Both were verified on 2026-09-14 as having no fragment store and no seg archive — in
aind-open-data. Both have one inaind-msma-morphology-data(7.4 GB and 4.1 GB). With 60 and 38 gold arbors they are the largest tracing sets in the corpus after 653158's 45, and the check that excluded them was scoped to one bucket.789202 needed both buckets at once: imagery from
aind-open-data, fragments and all 14 tracings from the other. See §2f for the full inventory of what exists where.
Only 651324 and 721332 are genuinely gold-only. Every brain-held-out result computed before 2026-09-18 was computed over four brains.
Gold layouts: Complete_raw is one SWC per (neuron, compartment, annotator) with occasional
consensus files (three of 653158's carry out-of-frame OFFSET headers and are rejected by a
bounds guard — RESULTS §1); specimen_space_reconstructions is one file per neuron/part.
Node types arrive as 0 and are stamped from the filename compartment at load.
The owners' own lists (received 2026-09-18) are checked in under src/exaproof/io/manifests/: the fused
imagery URL of every exaSPIM subject in the bucket (81, against KNOWN's five with imagery) and the segmentation
dataset produced for eight further subjects. The README there says how they line up with KNOWN. The CSV's
segmentation names identify a volume, not a location; the outputs computed over those volumes are in
aind-msma-morphology-data and are now ingested (§2e).
Known data quirks, all handled in code and documented in RESULTS: per-subject seg_key names
are not uniform; 653158 N016/N036/N049 consensus files sit tens of mm outside the volume;
648434/653980 segmentations under-cover gold cable (44.1 % / 54.0 % at 8 µm vs 653158's
86.8 % — RESULTS §24), which bounds everything downstream. The two brains added on 2026-09-18 sit
above both: 789202 reads 0.921 (n=14, min 0.797) — the best in the corpus — and 685222 reads
0.816 (n=38, min 0.564, worst N004), and 685221 reads 0.889 (n=60, min 0.794 N027, max 0.943 N010) — the
largest tracing set in the corpus, second only to 789202 in coverage.
653158 has two pairs of gold arbors that trace the same fibres (E406, scratchpad/e406_label_audit.py).
N015 and N016 (somata ~260 µm apart, different annotators) share 21.9 mm of path within 3 µm, 21 % of N016's cable
in 381 separate stretches; N003 and N067 (same annotator) share 27.8 mm, 6 % of N067's. Either the axons run together
below the resolution limit or a tracing followed the other neuron; the image cannot say which. Consequences: 46 % of
653158's merge instances come from these two pairs, and 39 % of all its instances sit on a shared path, where no
image check can find a merge. 653980's largest shared path is 0.8 mm and 648434's 0.07 mm. The split keeps N015
and N016 together in dev; N067 is a test arbor and N003 a training arbor, but only 7 of the 150 standard test gaps
lie within 3 µm of a training, calibration or dev tracing.
s3://aind-msma-morphology-data — readable, internal, in use since 2026-09-18
Anonymously readable but not part of the AWS Open Data Program and carrying no license. This project treated it as off-limits until 2026-09-18, when the maintainer lifted that: "you may use the unlicensed data." Reading, ingesting and training on it are allowed; redistributing or publishing it outward is not, and any number drawn from it must say so — the bucket a figure came from is part of the figure.
What is read from it, as of 2026-09-18: the seven newer brains' fragment archives under
agrim-whole-brain-predictions/ (§2e — 122.8 M nodes, the largest corpus this project has), registered as
extras["seg_url"] in KNOWN and fetched by the same fetch_seg_zip that handles aind-open-data. A written
index of the bucket's structure lives at registry/s3_aind-msma-morphology-data.txt (exaproof data index-s3),
so searching it costs a grep rather than a few dozen round trips.
2. Local tree (EXAPROOF_DATA, default /home/ubuntu/data/exaproof)
Every directory, what it holds, and how to regenerate it. "Derived" means losing it costs only compute; sizes are as of the sandbox-04 copy.
| dir | size | contents | regenerate with |
|---|---|---|---|
raw/<subject>/ |
3.6 GB | downloaded gold SWCs + fragment zip per subject (653158 2.0 GB, 653980 1.4 GB, 648434 311 MB). Imagery is never local — fused.zarr streams from S3 |
exaproof data fetch --subject S |
fragments/<subject>/ |
2.4 GB | columnar fragment store (xyz/radius/parent/node_frag as mmapped .npy) + spatial grid indices + end_nodes |
exaproof data ingest --subject S |
fragments/<subject><suffix>/ |
~ base + grown | augmented store: the delivered fragments plus cable generated for them (exaproof grow --suffix g). Same layout as fragments/, with meta.json carrying base_subject and n_base_fragments so generated cable is always separable from delivered. Exists so alignment, mining and the session measure the difference without knowing anything new (RESULTS §58) |
exaproof grow --subject S <neuron> --suffix g |
labellings/ |
150 MB | gold arbors aligned to fragments (cache keyed by tol/spacing) | exaproof gt align / first get_labellings() call |
decisions/<subject>/<tag>/ |
3 MB | mined labelled join corpora (653158 v3: 116k; 653980 v1: 20k; 648434 v1: 32k) |
exaproof decisions mine |
features/<subject>/<tag>_<hash>/ |
84 MB | feature tables, keyed by extractor version hash (97c5e30fac current) |
exaproof train rebuilds on miss |
slabs/<subject>/<tag>/ |
579 MB | oriented gap-voxel slabs (float16), incremental caches (v3 + wide) — expensive: ~56 MB/s S3 streaming to rebuild |
exaproof encode |
models/<subject>/<name>/ |
40 MB | decision heads + calibration + OOD + split metadata. Key names: 653158 a10 (main), askdyn (question ranker), slab_enc (CPU encoder); 653980/648434 native |
exaproof train, exaproof asker, exaproof encode |
runs/ |
18 MB | benchmark outputs (session/reach/triage/… JSON) — the numbers RESULTS.md cites | the corresponding exaproof command |
cache/ |
73 GB | fsspec chunk cache of streamed imagery. Disposable; refills on demand (copying it saves ~a day of warm-up) | |
learned/653158/ |
<1 MB | LLM feature-proposal loop ledger | exaproof learn |
somas/<subject>.csv (+ .source) |
~3 MB | detected somas per brain; the .source sidecar names the bucket and run a file came from (the imagery asset has none) |
io.somas.for_subject fetches on first use |
viewer/kept/<subject>/<neuron>.jsonl |
small | cable people kept in the viewer ("keep this cable"), one record per keep with who and when. Not derived and not regenerable: back it up like viewer/users.db |
written by the viewer |
runs/e463_walks/, runs/e465_paths/, runs/e465b_paths/, runs/e468_paths/ |
small | per-stretch records of the write-back experiments (E463: each walk's skeleton with probability and distances to tracing and seed; E465/b, E468: each kept path). E464's analyses read them | the experiment runners in scratchpad/ |
registry/<subject>.json |
<1 MB | per-subject ingest metadata (bounds, counts) | written by ingest |
ROI training corpora live under runs/roi_corpus_<subject>[_ms]/: one N<arbor>.npz of cubes per arbor plus
_seed and _objlabel companions (roi-corpus, then roi-seeds, roi-objlabels). Built so far: 653158 (native and
_ms, plus _ms3 with a level-4 channel, §288), 653980c _ms, 648434 _ms, and 609281 _ms2 (2026-09-15: 18
train+calib arbors, context 2:4 + 3:8, no touch cubes — a third brain for pooled training). 685221 _ms
(2026-09-18, E466): the 40 training arbors of scratchpad/e466_split.json, ms_100's recipe, plus compartment, seed
and object-label companions. roi-seeds takes --split now, or falls back to the corpus's own shards when the
subject has no decision model.
609281's imagery reaches the uint16 ceiling and cubes are stored float16, where 65535 casts to
inf: one such voxel survives median/IQR normalisation into the network.extract_cubenow clips at 65504 andnormalise_cubesanitises any older corpus. The first 609281 attempt (roi_corpus_609281_ms, one shard) predates the fix and is not used. The other brains' cubes peak at 1,900–17,200, far below the ceiling.
Provenance chain: S3 → raw → fragments → labellings → decisions → features/slabs →
models → runs. Only raw touches the network for anything but imagery; everything after is
deterministic given the code version (feature tables carry the extractor hash for exactly this
reason).
2b. The imagery metadata is kept locally (2026-09-17)
s3://aind-open-data/exaSPIM_653980_.../fused.zarr lost its .zattrs, .zgroup and every <level>/.zarray some
time on 2026-09-17 — the chunk keys are all still there and read fine, but zarr cannot open the group, so the brain
became unreadable mid-experiment (E421's first run died on it; E419 had finished an hour earlier only because its
process already held the volume open). The bucket is public and not ours to fix.
ImageSource.open now falls back to a saved copy of the metadata under imagery_meta/<hash of url>/, and
imagery.snapshot_metadata(url) takes that copy from any volume that still has it. 653158, 648434, 609281 and
651324 were snapshotted from their sources; 653980's was recovered from this machine's zarr chunk cache and verified
against the shape of the fused-denoised.zarr twin in the same prefix — (1, 1, 23146, 29005, 62654), 8 levels,
128³ chunks, <u2 — and then by reading imagery at a known coordinate.
A second copy is checked into the repository at src/exaproof/imagery_meta/ (50 KB, one directory per
volume, each with a SOURCE file naming it) and local_metadata falls back to it, so a fresh checkout on
another machine can open these volumes even if the bucket loses more of them.
The raw volume of 653980 exists only through that copy (2026-09-18) — SUPERSEDED the same day, see the table below
The release owner explained the deletion: 653980's fusion was denoised and the raw volume removed, with the
denoised image published beside it as fused-denoised.zarr in the same prefix. The chunks of the raw volume were
never deleted — only its metadata — so fused.zarr still reads through the saved copy, and that copy is now the
only thing that makes the baseline for every prior 653980 measurement readable. Do not delete
src/exaproof/imagery_meta/.
The two volumes sit on the same grid: same shape (1, 1, 23146, 29005, 62654), same 0.748 × 0.748 × 1.0 µm
voxels, same eight levels, same dtype. Coordinates, skeletons and the atlas registration are unaffected; only the
pixels differ. Measured on 60 gold nodes of N002: cable peak 393 → 389 (×0.99), background 74 → 75 (×1.01),
background noise sd 6.0 → 4.2 (×0.69), contrast 5.31 → 5.19. Whether the models care is E427.
Select one with ImageSource.for_subject(subject, variant="denoised"), or EXAPROOF_VOLUME=denoised for a whole
run. Both volumes' metadata is snapshotted.
Snapshot a new subject's metadata the first time it is opened. A few hundred bytes of JSON is the only part of a remote volume that cannot be recovered by retrying.
653158's and 648434's raw volumes are GONE — not just their metadata (2026-09-18)
This is worse than the 653980 case above and it failed silently. list-objects-v2 under
exaSPIM_653158_.../fused.zarr/ and exaSPIM_648434_.../fused.zarr/ returns zero objects: the chunks
themselves have been removed, not only the metadata. Eight gold-cable points per brain read 0/8 non-zero at two
resolution levels while the fused-denoised.zarr twins in the same prefixes read 5/5.
| brain | raw fused.zarr |
fused-denoised.zarr |
what to use |
|---|---|---|---|
| 653158 (primary, 45 gold arbors) | gone | present | EXAPROOF_VOLUME=denoised — there is nothing else |
| 648434 | gone | present | EXAPROOF_VOLUME=denoised — there is nothing else |
| 653980 | gone (was metadata-only earlier the same day) | present | EXAPROOF_VOLUME=denoised — there is nothing else |
| 609281 | gone (checked 21:30 PDT 2026-09-18) | none published | nothing readable — only a ktx octree and one low-resolution TIFF remain |
And 609281 has no readable imagery at all (checked 21:30 PDT 2026-09-18). list-objects-v2 under its
stitched_…/fused.zarr/ returns zero objects. What remains in that prefix is a MouseLight-style octree/ (ktx/
tiles plus one 134 MB default.0.tif), which this project does not read, and there is no denoised twin. The viewer
reports reads: false. Found by the viewer's metadata route (/ng/volume/): the saved metadata served fine, and
every redirected chunk came back 404. Any 609281 figure that reads imagery must be dated before the deletion or
re-read from the octree. Figures from skeletons and fragments alone are unaffected.
653980 went the same way later on 2026-09-18 (checked 17:00 PDT): list-objects-v2 under
exaSPIM_653980_.../fused.zarr/ returns zero objects, and 24 fragment-node points read blank while the denoised twin
reads. The section above — "the chunks of the raw volume were never deleted" — was true when written and is not now.
exaproof view no longer needs the variable for a tool a person looks at: ImageSource.live_for_subject moves to the
denoised twin when the default reads blank at cable and the twin reads, and the browser's Neuroglancer URL follows it
(zarr_url_for checks the default's .zattrs once). Measurements still get exactly the volume they ask for
(for_subject is unchanged), so a blank read there stays a loud failure rather than a quiet substitution.
Why it was silent, and the guard that now stops it. The saved-metadata fallback above resolves the shape from
our copy; with the objects gone, every chunk 404s and zarr fills zeros. A volume of zeros never raises — it
answers every question with "nothing here", which from a segmenter looks like an answer. ImageSource.reads_data
and .check_alive turn it back into a loud failure, exaproof view shouts at startup, and any corpus build must
check before and after (see scratchpad/e440_denoised_corpus.sh). Hand them points where there IS cable — gold
nodes — because dark background is common and only a lit point proves the volume is alive.
Two consequences for anyone reading old numbers. Every stored 653158 and 648434 figure was measured on imagery
that can no longer be reproduced, so §337's rule binds hard: no new figure may be compared with an old one unless
both were read on the same volume. And runs/roi_corpus_653158_ms (37 GB, 39 arbors) is now the only copy of
raw-imagery training cubes for 653158 that will ever exist — do not delete it. runs/roi_corpus_653158_msd is
the denoised replacement (E440).
There is also a deliberate variant="blank" — the real volume with every read zeroed, same grid, same seeds.
It exists because the accident above turned out to be a useful control: a cue that claims to read the image is
scored against it (§346, §347).
2e. The eight newer brains, surveyed (2026-09-18)
The owners' manifests (src/exaproof/io/manifests/) name 81 subjects with fused imagery and 8 with a segmentation
dataset. Those 8 are the interesting ones — imagery plus a segmentation is what this tool needs — so every prefix
under s3://aind-open-data carrying their subject number was listed and opened and read. Every "readable"
below is a verified read of real voxels, not an inference from a listing, because listing was misleading in both
directions here.
| subject | volume that reads | levels · shape (z,y,x) | somas (CCF) | CCF align |
|---|---|---|---|---|
| 709221 | fused-denoised.zarr — the raw one is gone |
8 · 21186×40386×66437 | 6,321 | ✓ |
| 718162 | fused.zarr and fused-denoised.zarr |
7 · 17411×38903×62516 | 3,718 | ✓ |
| 789202 | fused.zarr |
7 · 21558×41022×66869 | 92 | ✓ |
| 794491 | fused.zarr in a different asset (_processed_2025-11-08_09-50-12) |
7 · 19587×40078×64486 | 649 | ✓ |
| 794492 | fused.zarr |
7 · 20685×40538×65967 | 724 | ✓ |
| 794493 | fused.zarr |
7 · 20991×39179×64507 | 1,077 | ✓ |
| 794495 | fused.zarr |
7 · 22973×40789×67530 | 119 | ✓ |
| 802449 | fused.zarr |
7 · 16901×38042×62544 | 57 | ✓ |
789202 has no fused-denoised.zarr. EXAPROOF_VOLUME=denoised and variant="denoised" fall back, with a warning, to its raw fused.zarr, so every 789202 row run "denoised" read raw (§809). Before row 299, variant="raw" was not a raw switch: it asked for fused-raw.zarr and fell back to the registry's default, which for 685221 is the denoised volume. It now opens the raw fused.zarr and raises when that is gone; strict=True makes any variant raise. 685221's raw fused.zarr beside its denoised volume is deleted (fused.zarr.deleted, 2026-09-10). Its other fusions with a fused.zarr (flatfield-correction_2025-02-20…_fusion_2025-02-20_19-56-18: 3,626×3,679×7,859 at a stated 0.748 µm; fusion_2024-05-02_20-16-08: 5.984 µm) are not on the denoised grid (row 294).
721332 is the one gold brain with both volumes on one grid (row 297, §810; registered in datasets.KNOWN by row 299, raw fused.zarr as its default, variant="denoised" its twin, the 22 gold fetched; raw and denoised give identical separation verdicts there, §811): exaSPIM_721332_2025-04-05_14-48-36_flatfield-correction_2025-04-24_20-06-12_fusion_2025-05-06_17-30-15/ holds fused.zarr and fused-denoised.zarr (25,187 × 52,320 × 62,708 at 0.748 µm, 8 levels), and its 22 gold neurons are specimen-space SWCs in exaSPIM_721332_2025-04-05_14-48-36_reconstructions_2026-08-28_23-04-54/specimen_space_reconstructions/swc/ (41 MB) that land on its fibres. datasets.KNOWN["721332"] still has fusion=None. Every other gold brain has one volume. EXASPIM25's block crops are raw on the denoised grid for 653158, 650010 and 651324.
No segmentation exists for any of them in this bucket. segmentation_datasets.csv names pipeline runs
(raw.unet_r3_ckpt_425250_seedfix_overlapfix, mean40.stddev105.mask…ffn.mt_0.1); the immediate children of all
42 top-level assets for these subjects were listed and none contains a seg/ or anything matching. Where they
are stored has to come from the owners. Until then these brains can be looked at and not proofread: the
worklist is built from fragments, and fragments come from a segmentation.
What is genuinely new and usable: somas. soma_detection/aligned/soma_locations_ccf.swc gives 57 to 6,321
detected somas per brain in CCF space. Every reconstruction starts at a soma and until now the seed had to be
given by hand. ccf_alignment/ also carries the full forward and inverse SyN warps (~1.8 GB each), not only the
per-node displacement SWCs of §2c.
What is NOT usable, and was nearly recorded as if it were. Four subjects have denoised/probability/ and
denoised/mask/ directories, and a first pass through this section listed them as a per-voxel probability volume
this project could use. They are empty shells — 794492's probability/SPIM.ome.zarr is a single 24-byte
.zgroup and nothing else; 789202's is empty. Listing a directory is not reading it, which is the same mistake in
the same day as §343's blank volume. Where real denoised output does exist it is per-tile
(denoised/SPIM.ome.zarr on 789202's and 802449's second assets), not a fused volume on the imagery grid.
Two traps, both of which silently produce the wrong data.
- The newest asset is not the one with the data. Each subject has two to seven
_processed_assets; four of the eight keep their somas in an earlier asset than their latest, and 794491's live imagery is in a different asset from the one the manifest names. Picking "the last prefix that matches" — whatdiscoverdid before it read the manifest — gets the wrong one. - The manifest can name a volume that has been deleted. 709221's and 794491's manifest entries both point at
fused.zarrprefixes with zero objects. The bucket marks these explicitly —fused.zarr.deleted,fused.n5.deleted,denoised_full_res.deleted— so a deletion is detectable rather than something to be discovered by reading zeros (§2b). Any code that resolves a volume should look for a.deletedsibling and for afused-denoised.zarrtwin before giving up.
The segmentations, found (2026-09-18)
segmentation_datasets.csv names a run per subject and no object under aind-open-data matches, because the
outputs live in s3://aind-msma-morphology-data — which this project had been told to treat as internal until
the maintainer lifted that on 2026-09-18 ("you may use the unlicensed data"). Read and ingest yes; do not
redistribute it, and say which bucket a number came from.
They are under agrim-whole-brain-predictions/<subject>/<run>/, and each run's metadata.json carries the
segmentation_id that the CSV names, which is what ties the two together:
{"brain_id": "709221", "segmentation_id": "202508_...mean50.mask.136168199.no_omitted_20k.ffn.mt_0.1",
"min_fragment_size": "40.0um", "node_spacing": 4, "use_somas": true, "search_radius": "30.0um",
"model_name": "GraphNeuralNet-20250331-v2-0.8577.pth", "accept_threshold": 0.825}
So these are not raw segmentations but a graph-neural-net proposal pass over one: fragments above 40 µm, up to three proposals per leaf within 30 µm, accepted at 0.825. That is the same decision this project's chooser makes, already made by another model — which makes them a comparison, not only a corpus.
Seven of the eight are ingestable, and all seven now are. An earlier revision of this section said only four
were, because three of them nest the archive one directory deeper — …/<run>/corrected-swcs/corrected-swcs.zip
rather than …/<run>/corrected-swcs.zip — and a listing of the run prefix shows that as a bare corrected-swcs/
next to connections.txt, which reads like a directory of loose files. It is one zip. Descend before concluding
a thing is absent.
| subject | run taken as primary | archive | fragments | nodes |
|---|---|---|---|---|
| 709221 | …855_mean50.mask…ffn.mt_0.1_20250829_020215 |
3.84 GB (nested) | 2,947,651 | 61,488,872 |
| 789202 | …raw.jin_masked_mean40_stddev105_less_aggressive_seedfix_20260304_004343 |
68 MB | 19,075 | 1,137,366 |
| 794491 | …mean40.stddev105.mask…ffn.mt_0.1_20251117_054446 |
1.18 GB (nested) | 595,890 | 19,487,832 |
| 794492 | …jin_masked_mean40_stddev105_less_aggressive_20260209_190101 |
500 MB | 203,823 | 8,285,727 |
| 794493 | …mean40.stddev105.mask…ffn.mt_0.1_20251117_053518 |
500 MB (nested) | 241,445 | 8,265,318 |
| 794495 | …raw.unet_r3_ckpt_87750_aggressive_20260307_210855 |
478 MB | 258,723 | 7,989,421 |
| 802449 | …jin_masked_mean40_stddev105_20260308_032531 |
985 MB | 308,370 | 16,106,818 |
| 718162 | — | nothing published under this prefix | — | — |
That is 122.8 million nodes across seven brains, against the 10 brains this project had — and 709221 alone, at 2.9 million fragments, is larger than the other six together. Its ingest took 366 s and 9 GB of resident memory; the other six took 9-25 s each.
Which run, when there are several. 802449 publishes 24 and 794495 eight. The rule taken here is the newest
corrected_<subject>_… run, which is the pipeline's own output at its current settings; original_* are the
same segmentation before the GNN's proposals were applied, and corrected_nongnn_* / corrected_heads4_* are
ablations of the accepting model. The first pick for 802449 was corrected_nongnn_… — chosen by date alone,
and by name the one run with the GNN switched off. It has been replaced, and the ablations are kept in the
registry as extras["seg_variants"], because a segmentation and the same segmentation without the joiner is a
matched pair, which is the cleanest measurement of what an accepting model contributes that this project has
access to.
The CSV names a segmentation, not a run, and mostly not a published one. segmentation_datasets.csv gives
e.g. 802449,raw.unet_r3_ckpt_425250_seedfix_overlapfix; the runs published for 802449 are all over
jin_masked_mean40_stddev105. Only 794491 and 794493 have a run whose metadata.json segmentation_id equals
the CSV's string. The newer runs (789202, 794492, 794495, 802449) carry metadata_graph.json +
metadata_ml.json instead of metadata.json, and no segmentation_id at all — the segmentation is named only
in the run's directory name. So the CSV identifies the volume each group intends, and the volume itself is not
in either bucket. segmentation/exaSPIM_<subject>_zarr/ exists for 28 other subjects and for none of these
eight (checked against the written index, exaproof data index-s3). Despite the _zarr in the name,
those prefixes hold no voxels. A recursive listing on 2026-09-24 found only SWC archives and sidecars
under segmentation/: 919 objects, split into 401 .zip, 299 .swc and 209 .json, with no array chunk
or .zarray. The FFN run directories inside them, e.g. exaSPIM_685221_zarr/…ffn.mt_0.1/, hold
swcs_0.zip–swcs_15.zip (§2d).
The accepting model is named too, and differs by run: VisionHGAT-20260205-40-0.8723.pth at threshold 0.8 for
789202 and 794495, VisionHGAT-20260205-31-0.8617.pth at 0.7 for 794492, VisionHGAT-20260301-58-0.8692.pth
at 0.8 for 802449 — so the four newer brains are not a single homogeneous corpus, and a cross-brain reading has
to say which acceptor produced each. Every newer run also ships proposal_predictions.json: another model's
probability for each proposal it considered, on the same decision this project's chooser makes.
Also in that bucket and worth knowing: segmentation/794495_it2_split_corrected_distance_search/ (299 SWCs with
JSON sidecars), exaspim_soma_detection/, training-data/, training_blocks/, and merge-site collections for
685221 and 719654.
789202 has manual tracings, and is the fifth end-to-end brain (2026-09-18)
The same bucket holds an annotator's working tree, anna.grim/, and in it
groundtruth_tracings/789202/world/ — 14 manually traced neurons, one file per (neuron, annotator),
named N005-789202-SP.swc in the same convention Complete_raw uses without the per-neuron directory. That is
8.06 metres of gold cable (259–978 mm per arbor, 6.9 M nodes), and with the fragments above it makes 789202
the first subject since the original four to support the end-to-end split/merge task — and the first assembled
from two buckets: imagery from aind-open-data, fragments and gold from aind-msma-morphology-data.
The frames agree, which was checked and not assumed — the rule that BIL's voxel-index SWCs taught this project. Thirteen of the fourteen arbors sit a median 2.5 to 4.8 µm from the nearest fragment node, so the tracings are already in the fragments' world micrometres and need no scaling. N023 reads 72.8 µm, which looked like a frame problem and is not one: aligned, it covers 0.411 of its cable against a median of 0.715, so it is simply the worst-segmented arbor and its distance distribution is bimodal — 41 % of nodes within 8 µm, the rest far.
Aligned coverage puts it FIRST in the corpus — but only for the right segmentation. Three segmentations of this brain exist and they do not agree:
| registered as | archive | fragments / nodes | median coverage @ 8 µm |
|---|---|---|---|
789202 |
the CSV's raw.unet_r3_ckpt_425250…, pre-proofreading |
60,622 / 4.52 M | 0.921 (min 0.797) |
789202final |
the same after the proofreading pipeline | 75,914 / 4.37 M | measured in E449 |
789202wbp |
agrim-whole-brain-predictions' 68 MB output |
19,075 / 1.14 M | 0.715 (min 0.411) |
0.921 beats 653158's 0.868 and every other brain. The first ingest here made 789202wbp canonical — it was
the archive the whole-brain-predictions survey found — and recorded 0.715 as the brain's coverage. That archive
holds a quarter of the cable. The owners' CSV named the better segmentation all along, and the lesson is the
one §353 records: the file you find first is not evidence about the brain.
It is the only one of the eight with tracings: groundtruth_tracings/ contains 789202 and nothing else.
Three more collections in anna.grim/ are worth a later look, recorded here so the search is not repeated:
| prefix | what it is |
|---|---|
merge_detection/<subject>/result_<segmentation>_<date>/ |
detected_sites.zip + fragments.zip (~250 MB) for 789202, 794491, 794492, 794493 — the pre-correction fragments beside the sites a detector flagged, which is the matched pair for studying merges |
image-carveouts/802449/whole-brain-N001…N039/ |
input.zarr + mask.zarr + swcs_extracted/, one whole-brain-sized mask.zarr per gold neuron. It is a binary mask: every chunk sampled on 2026-09-24 across N001 and N036 holds only 0 and 1. It is likely drawn from the tracing and is not segmenter output. It could supervise a segmenter rather than a skeleton model. image-carveouts/754612/blocks/block_000/ has the same layout at 1024³, with a malformed store: chunk files are written in both /- and .-separated form, and some lie outside the declared shape |
automated_proofreading/<subject>/<segmentation>/ |
789202 and 794495, under the segmentation names segmentation_datasets.csv gives — this is where the CSV's names finally resolve |
A written index of that prefix (depth 6) is how these were found; regenerate with
exaproof data index-s3 --bucket aind-msma-morphology-data --root anna.grim/ --depth 6.
2f. The full inventory (2026-09-18)
Both buckets were walked and the listings written to registry/s3_*.txt and registry/wbp_survey.json, so the
two tables below are regenerated offline in a second (exaproof data inventory) rather than by another few dozen
aws s3 ls calls. Three fixed-name assumptions had each hidden data before this: an archive nested one
directory deeper, a gold directory under a layout name the loader did not know, and a run whose archive is called
corrected-processed-swcs-s3.zip rather than corrected-swcs.zip. Every search here matches by shape — any
.zip in the run, any *.swc in the directory — not by an expected name.
Table A — every brain we have a name for, and what exists for it
Built offline from the written indexes (exaproof data index-s3, registry/s3_*.txt) plus
registry/wbp_survey.json. imagery is what the owners' exaspim_image_prefixes.json entry resolves to;
denoised only means the raw fused volume it names is gone and fused-denoised.zarr sits beside it.
somas: yes = the imagery asset's soma_detection/; bucket = the production detector's runs in the
morphology bucket (exaspim_soma_detection/<subject>/results_*/, 122 brains, 51 of them only there), which
exaproof.io.somas.for_subject falls back to, newest run first. That source was added 2026-09-18 evening. Until
then this column said "—" for every end-to-end brain.
685221's archive was 22 MB in this table and 7.4 GB in §1's registry, a factor of 336. Settled 2026-09-19 against the local artefact: raw/685221/seg.zip is 7,444,560,935 bytes — 7.4 GB decimal, 6.9 GiB — which is §1's figure, so this table was wrong and now reads 7.4 GB. For reference the tracings beside it are 596 MB (specimen_space_reconstructions), so 22 MB was not a mislabelled tracings size either; it appears to predate the archive being identified.
| brain | imagery | tracings | segmentation archive | somas | CCF | status here |
|---|---|---|---|---|---|---|
| 609281 | — | manual (open-data) | own seg/ |
bucket | yes | end-to-end |
| 613814 | denoised only | — | — | bucket | yes | — |
| 648434 | denoised only | manual (open-data) | 291 MB + own seg/ |
bucket | yes | end-to-end |
| 650010 | denoised only | — | — | bucket | yes | — |
| 650011 | denoised only | — | — | bucket | — | — |
| 651324 | denoised only | manual (open-data) | — | — | yes | registered |
| 651895 | denoised only | — | — | bucket | yes | — |
| 652440 | denoised only | — | 5,321 MB | yes + bucket | yes | — |
| 652779 | denoised only | — | — | yes + bucket | yes | — |
| 653158 | denoised only | manual (open-data) | 3,420 MB + own seg/ |
bucket | yes | end-to-end |
| 653159 | denoised only | — | 999 MB | bucket | yes | — |
| 653980 | denoised only | manual (open-data) | own seg/ |
— | yes | end-to-end |
| 657676 | denoised only | — | 3,923 MB | bucket | yes | — |
| 658735 | denoised only | — | 14,204 MB | bucket | yes | — |
| 665081 | denoised only | — | 2,672 MB | bucket | yes | — |
| 667996 | denoised only | — | 1,112 MB | bucket | yes | — |
| 667997 | denoised only | — | — | yes + bucket | yes | — |
| 671477 | raw | — | — | — | — | — |
| 674185 | denoised only | — | 0 MB | bucket | yes | — |
| 674190 | raw | — | — | — | yes | — |
| 674191 | denoised only | — | — | bucket | yes | — |
| 675375 | denoised only | — | — | bucket | yes | — |
| 676009 | denoised only | — | — | — | yes | — |
| 681465 | denoised only | — | — | bucket | yes | — |
| 685221 | denoised only | manual (open-data) | 7.4 GB | bucket | yes | registered |
| 685222 | denoised only | manual (open-data) | 4,143 MB | bucket | yes | registered |
| 686951 | denoised only | — | — | yes + bucket | yes | — |
| 686955 | raw | — | — | bucket | yes | — |
| 697837 | denoised only | — | — | yes + bucket | yes | — |
| 703071 | denoised only | — | — | yes + bucket | yes | — |
| 704522 | denoised only | — | 11,008 MB | yes + bucket | yes | — |
| 704523 | — | — | 6,416 MB | bucket | yes | — |
| 706301 | none | — | 1,069 MB | bucket | yes | — |
| 708365 | denoised only | — | — | bucket | yes | — |
| 708369 | denoised only | — | 0 MB | bucket | yes | — |
| 708373 | none | — | — | bucket | yes | — |
| 709203 | denoised only | — | 5,166 MB | bucket | — | — |
| 709221 | denoised only | — | 3,842 MB | yes + bucket | yes | registered |
| 709222 | denoised only | — | 51 MB | — | yes | — |
| 709393 | denoised only | — | — | bucket | yes | — |
| 713600 | denoised only | — | — | bucket | yes | — |
| 713601 | denoised only | — | — | bucket | yes | — |
| 715345 | raw | — | 883 MB | bucket | yes | — |
| 715346 | raw | — | — | bucket | yes | — |
| 715347 | raw | — | 2,529 MB | bucket | — | — |
| 717445 | raw | — | 1,089 MB | bucket | — | — |
| 717614 | raw | — | — | — | — | — |
| 718162 | raw | — | — | yes + bucket | yes | registered |
| 718168 | raw | — | — | bucket | yes | — |
| 719654 | raw | — | 2,038 MB | bucket | yes | — |
| 720164 | none | — | — | yes + bucket | yes | — |
| 720165 | raw | — | — | bucket | — | — |
| 721332 | raw | manual (open-data) | — | bucket | yes | registered |
| 721828 | raw | — | 0 MB | yes + bucket | yes | — |
| 721830 | raw | — | — | yes + bucket | — | — |
| 730223 | raw | — | 34 MB | yes + bucket | yes | — |
| 730902 | raw | — | 313 MB | bucket | yes | — |
| 733420 | raw | — | — | yes + bucket | yes | — |
| 737563 | raw | — | 281 MB | bucket | yes | — |
| 737826 | raw | — | — | yes + bucket | yes | — |
| 744783 | raw | — | 59 MB | yes + bucket | yes | — |
| 747807 | raw | — | 2,893 MB | yes + bucket | yes | — |
| 747820 | raw | — | 678 MB | yes + bucket | yes | — |
| 750318 | raw | — | 3,159 MB | yes + bucket | yes | — |
| 751473 | raw | — | 20 MB | — | yes | — |
| 754610 | raw | — | 5 MB | — | — | — |
| 754611 | raw | — | 465 MB | yes + bucket | yes | — |
| 754612 | raw | — | 0 MB | bucket | yes | — |
| 754613 | raw | — | 555 MB | bucket | yes | — |
| 763259 | raw | — | — | bucket | yes | — |
| 776235 | raw | — | — | yes + bucket | yes | — |
| 776241 | raw | — | — | yes + bucket | yes | — |
| 784666 | raw | — | — | yes + bucket | yes | — |
| 784802 | raw | — | — | yes + bucket | yes | — |
| 784896 | raw | — | — | yes + bucket | yes | — |
| 785688 | raw | — | 269 MB | yes + bucket | yes | — |
| 789202 | raw | manual (morph) | 68 MB | yes + bucket | yes | end-to-end |
| 794491 | other sibling | — | 1,182 MB | yes + bucket | yes | registered |
| 794492 | raw | — | 500 MB | yes + bucket | yes | registered |
| 794493 | raw | — | 500 MB | yes + bucket | yes | registered |
| 794495 | raw | — | 478 MB | yes + bucket | yes | registered |
| 802449 | raw | — | 553 MB | yes + bucket | yes | registered |
| 802450 | raw | — | — | yes + bucket | yes | — |
| 820994 | — | — | 499 MB | yes + bucket | yes | — |
84 brains named.
| can we use it? | count | brains |
|---|---|---|
| tracings + segmentation (end-to-end reachable) | 7 | 609281, 648434, 653158, 653980, 685221, 685222, 789202 |
| segmentation, no tracings | 38 | 652440, 653159, 657676, 658735, 665081, 667996, 674185, 704522, 704523, 706301, 708369, 709203, 709221, 709222, 715345, 715347, 717445, 719654, 721828, 730223, 730902, 737563, 744783, 747807, 747820, 750318, 751473, 754610, 754611, 754612, 754613, 785688, 794491, 794492, 794493, 794495, 802449, 820994 |
| tracings, no segmentation | 2 | 651324, 721332 |
| neither | 37 | 37 brains, imagery only |
All 7 are in use: 653158, 653980, 648434, 609281, 789202, 685221 and 685222. (This line said "5 of the 7" until 2026-09-19 and was already wrong when written -- 685221's ROI corpus is described in this file's own section 2, and FINDINGS carries results on it.)
Table B — the names we were given, and which ones resolve
Both manifests arrived from the data owners on 2026-09-18. Every entry was probed.
| list | entries | resolve as given | resolve to a sibling | nothing there |
|---|---|---|---|---|
exaspim_image_prefixes.json |
81 | 42 | 36 | 3 |
segmentation_datasets.csv |
8 | see below | — | — |
The 36 that resolve only to a sibling are one thing, not thirty-six: the raw fused.zarr the manifest
names has been deleted and fused-denoised.zarr published in its place. §343 recorded this for 653158 and
648434 as an incident; it is bucket policy. Any code that opens the manifest's URL reads nothing, silently.
| brain | the manifest's URL ends in | what actually reads |
|---|---|---|
| 613814 | fused.zarr |
fused-denoised.zarr |
| 648434 | fused.zarr |
fused-denoised.zarr |
| 650010 | fused.zarr |
fused-denoised.zarr |
| 650011 | fused.zarr |
fused-denoised.zarr |
| 651324 | fused.zarr |
fused-denoised.zarr |
| 651895 | fused.zarr |
fused-denoised.zarr |
| …and 30 more | fused.zarr |
fused-denoised.zarr |
Three have nothing readable in the prefix at all — not a rename, an empty prefix:
| brain | manifest URL |
|---|---|
| 706301 | s3://aind-open-data/exaSPIM_706301_2024-04-23_11-24-24_fusion_2024-05-21_00-00-03/fused.zarr/ |
| 708373 | s3://aind-open-data/exaSPIM_708373_2024-04-02_19-49-38_fusion_2024-09-03_19-00-37/fused.zarr/ |
| 720164 | s3://aind-open-data/exaSPIM_720164_2025-07-07_17-55-45/SPIM.ome.zarr/tile_000004_ch_488.zarr/ |
The segmentation CSV. All eight subjects it names have outputs published, but the CSV names a segmentation (a volume) and the published objects are runs over one. Where the two can be matched:
| brain | CSV names | is that segmentation published? |
|---|---|---|
| 709221 | 20251001_mean40.stddev105.mask.136168199.no_omitted_20k.ffn.mt_0.1 |
no — the published runs are over 202508_…mean50…, a different segmentation |
| 718162 | raw.jin_masked_mean40_stddev105_less_aggressive_seedfix |
no run published at all under this brain |
| 789202 | raw.unet_r3_ckpt_425250_seedfix_overlapfix_rempm |
yes — anna.grim/automated_proofreading/789202/raw.unet_r3_ckpt_425250_seedfix_overlapfix_rempm/ |
| 794491 | mean40.stddev105.mask.136168199.no_omitted_20k.ffn.mt_0.1 |
yes — the run's metadata.json segmentation_id matches exactly |
| 794492 | denoised.unet_r3_ckpt_155250 |
no — published run is over jin_masked_mean40_stddev105_less_aggressive |
| 794493 | mean40.stddev105.mask.136168199.no_omitted_20k.ffn.mt_0.1 |
yes — segmentation_id matches exactly |
| 794495 | raw.unet_449_792_202_splits_and_merges_831600 |
no — seven runs, none over raw.unet_449_792_202_splits_and_merges_831600 |
| 802449 | raw.unet_r3_ckpt_425250_seedfix_overlapfix |
no — 22 runs, all over jin_masked_mean40_stddev105 |
So three of eight CSV names resolve to a published segmentation (789202, 794491, 794493); the other
five name a volume nobody has published, while the brain does have outputs over a different one. The
segmentation/exaSPIM_<subject>_zarr/ prefixes exist for 28 other subjects and for none of these eight.
They hold skeletons, not segmentation volumes (§2d), so connections.txt cannot be resolved into cable.
What this changes
agrim-whole-brain-predictionsholds 43 brains, not the eight the CSV lists, and every one of them has an ingestable archive (68 MB to 14.2 GB). The CSV is a list of eight segmentations someone cared about, not an inventory of what exists.- INVENTORIED 2026-09-19 (
scratchpad/e516_wbp_inventory.out): all 43 have imagery, and 38 of them have no tracings. Every prediction brain resolves to anexaSPIM_<subject>asset onaind-open-data(220 exaSPIM brains have imagery there in total). Five of the 43 are brains this project already traces. The 38 untraced ones carry 245 GB of skeletons, from 708369 and 730223 at under 0.1 GB to 706301 at 29.8 GB. What they can and cannot supply. The ROI corpus samples omission gaps and terminals, and both need gold: a gap is cable a person traced that the segmenter missed, and a terminal is where an arbor ends rather than where the segmenter stopped. So these 38 brains cannot supply the supervision the ROI model exists for. What they can supply is IDENTITY — this neurite against the one beside it — which is the measured cross-brain failure (§364: recall holds on unseen brains, selectivity falls 0.084–0.113) and what §441 showed setting what every write-back lever buys. The trap to design around: the segmenter breaks rather than merges, so labelling its unclaimed voxels as background would train our model to reproduce its misses. The corpus format already carries the fix (0 bg, 1 neurite, 3 hard background, 255 ignore): seed the fragment as 1, other fragments as 3, every unclaimed BRIGHT voxel as 255, and only dim voxels as 0 — thresholded by E171's measurement that missed cable sits at the 89th–97th percentile of its own background. Centre cubes on fragment interiors, never ends, because a fragment end is usually the segmenter's error rather than a terminal. GoalD4-multibrain-roi. - Seven brains are end-to-end reachable and five are in use. The two unused are 685221 (60 gold arbors)
and 685222 (38) — the largest gold sets in the corpus after 653158's 45, both previously recorded here as
"gold only, no fragments", which was true of
aind-open-dataand false of the other bucket. - 653158 and 648434 have a second, independent segmentation (3.4 GB and 291 MB) of brains this project has measured for a year. 648434's 44.1 % coverage bound is the tightest constraint in the corpus and has never been tested against an alternative segmentation of the same brain.
- Only two brains have tracings and no segmentation anywhere: 651324 and 721332.
2c. The atlas (CCF), and the registration that reaches it
What exists, and where (listed and read 2026-10-16). Every gold brain's reconstruction release in
aind-open-data publishes its tracings a second time in CCF micrometres, axes (AP, DV, LR) — checked on 648434
N012 against the release's own per-node allenId: the soma reads SUT in both, and 46.6 % of its nodes match as
written against 0.7 % with DV flipped (most of the rest lie behind the atlas volume's 13.2 mm, where the release
writes root and the annotation has nothing). exaproof.io.ccf.registration(subject) pairs each gold node with its own atlas position and caches
the pairs (and the neuron each came from) under ccf/registration_<subject>.npz:
| form | brains (files) | what it is | paired by |
|---|---|---|---|
<reconstructions>/Complete_annotated/<N>-<subject>-<annotator>.json |
609281 (26), 651324 (13), 653158 (42), 653980 (21) | the annotator's own Complete_raw files: axon = the soma then the axon file, dendrite = the dendrite file, in file order, with a per-node allenId |
node for node, only where every parent pointer agrees (pair_annotated); a part edited since the release, branch by branch |
<reconstructions_2026-05-06…>/ccf_space_reconstructions/{swc,json}/<N>-<subject>.swc |
648434 (12), 685221 (60), 685222 (38), 721332 (22) | the same neuron after the owners' refinement (snapped to the image, A*-rerouted, re-sampled — 648434 N012 has 2,720 nodes here and 3,091 there) and transform through the exaSPIM template; topology kept, node order and count not | branch by branch (pair_tracings); validated below |
721332's local gold has not been fetched, so it builds on first use after exaproof data fetch --subject 721332.
The 2026 brains have four reconstruction releases each (2026-03-27, -04-15, -05-06, -08-28); the 2026-05-06 one is
the one KNOWN names and the one the local gold was fetched from (648434 N012 byte-identical). The raw transforms also
exist — CCF_Atlas_Registration/{affine_transforms.mat, warp_transforms.nii.gz} in 648434's and 653158's fusion
prefix, CCF_Atlas_Registration_10/ (a 5.4 GB 10 µm warp) in 653158's and 653980's, none in the fusion prefixes
KNOWN names for 685221 and 685222 —
and are not needed: the CCF-space tracings are the transform already applied. (The "CCF" column of §2f's table means
such a folder exists, not that a registration was built.) In aind-msma-morphology-data the only CCF-space tracings
are cameron.arshadi/{snapshot-test,nmcp-snapshot-test}/ccf_space_reconstructions/ (55 neurons of 685221, NMCP
export tests of November 2025); nothing there is CCF-space for 653980 or 789202 (the 2026-09 indexes under
registry/, and live listings of anna.grim/groundtruth_tracings/).
Per brain, as built (scratchpad/ccf_internal_verify.py brains, …_brains.out). The checks need no truth:
neighbours — pairs of nodes of different neurons within 10 µm of each other in the brain, and how far apart
the registration puts them in the atlas (a correspondence must keep them close; consistency, which registration
applies, refuses a median over 50 µm); cable ratio — each gold arbor's length over its length mapped through its
own neuron's pairs, which should sit at the brain's linear scale; soma — the gold soma's region against the
allenId the release itself gives that soma.
| brain | source | gold neurons | pairs | neighbours: pairs · atlas p50 / p90 µm · over 50 µm | cable ratio p50 (range) | worst arbor covered | somas in annotation · region = release's |
|---|---|---|---|---|---|---|---|
| 653158 | Complete_annotated, node for node |
45 (42 annotated; N016, N036, N049 have no JSON) | 305,184 | 1,123 · 1.8 / 4.0 · 0 | 2.31 (2.23–2.40) | 1.00 of the 42 | 42/42 · 42/42 |
| 653980 | Complete_annotated (N022's axon, edited since, by branch) |
21 | 206,589 | 237 · 3.2 / 4.3 · 0 | 2.35 (2.19–2.42) | 0.95 | 21/21 · 20 (21 nested) |
| 609281 | Complete_annotated, node for node |
25 | 137,381 | 420 · 3.4 / 5.4 · 0 | 1.84 (1.73–1.97) | 1.00 | 25/25 · 23 |
| 651324 | Complete_annotated, node for node |
13 | 250,191 | 29,975 · 2.3 / 4.4 · 0 | 2.12 (2.02–2.24) | 1.00 | 13/13 · 13 |
| 648434 | ccf_space_reconstructions, by branch |
12 | 142,814 | 14 · 15.6 / 61.4 · 2 | 2.41 (2.31–2.47) | 0.90 | 12/12 · 11 |
| 685221 | ccf_space_reconstructions, by branch |
60 | 2,415,714 | 28,075 · 1.6 / 3.5 · 4 | 2.76 (2.67–2.87) | 0.86 | 60/60 · 54 (58 nested) |
| 685222 | ccf_space_reconstructions, by branch |
38 | 1,635,413 | 2,810 · 2.3 / 4.4 · 0 | 2.67 (2.51–2.79) | 0.85 | 38/38 · 32 (33 nested) |
"Worst arbor covered" is the share of the least-covered arbor's nodes within 15 µm of its own neuron's pairs (a branch
the matcher could not settle is left out, not guessed). Every mapped soma sits exactly on the release's own soma
(0.0 µm on 648434, 685221 and 685222); where the region still differs it is the annotation, not the pairing — the
release reads its allenId on the 10 µm volume, io.ccf on the 25 µm one, and the misses are neighbours (P against
MV or PB, scp against MRN).
The linear scale is not 3: expanded micrometres per atlas micrometre, measured from the pairs, is 2.1–2.4 on
653158, 653980, 651324 and 648434, 1.84 on 609281 and 2.7–2.8 on 685221 and 685222. (653158's own expansion() read
1.56 under the old file-order pairing; it reads 2.27 now.)
How good the branch-by-branch pairing is, against truth (…_synthetic.out). The node-for-node brains supply
truth: each gold arbor of 653158 and 653980 is given a "release-style" copy made from its own true CCF tracing (every
branch re-sampled at 8 µm, 1 µm jitter, children shuffled, and in a second run one terminal twig removed), and
pair_tracings pairs the gold against it. Without a lost twig, 99.9 % (653158) and 99.7 % (653980) of nodes are
paired, the per-arbor median node lands 1.4 and 1.6 µm from its own atlas position, 90 of 304,851 (0.03 %) and 3,731 of 199,049
(1.9 %) are off by more than 25 µm and 1 and 1,249 (0.6 %) by more than 50 µm, and none of the 41,904 matched branch
ends is off by more than 25 µm — no branch is paired with the wrong one. With one twig removed from every arbor,
99.5 % and 96.2 % are paired (653980 N019 loses 488 of its 838 branches to a twig lost near the soma) and 8 of 41,284 branch
ends are off by more than 25 µm, twig merges taken the wrong way. The error that remains sits inside long unbranched
stretches whose registration changes scale along them (653980 N031: CCF µm per local µm goes 0.42 → 0.90 → 0.28 along
one 4.5 mm axon). On the real 2026 brains it is the few neighbour pairs above 50 µm in the table: two nodes on
648434 N020's 23 mm axon, 73-75 µm from where N023 puts the same place, and one spot where 685221 N012 and N027 run
together, ~60 µm.
789202 has none. Its 14 tracings are anna.grim/groundtruth_tracings/789202/world/ in the morphology bucket, with
no CCF copy beside them (world/ is the only folder), and aind-open-data has no reconstructions release for it.
What its processed asset (exaSPIM_789202_2026-02-03_15-09-59_processed_2026-02-15_23-47-18/ccf_alignment/) does
hold: the brain → exaSPIM-template SyN (789202_to_exaSPIM_SyN_{0GenericAffine.mat, 1Warp.nii.gz, 1InverseWarp.nii.gz}, 1.8 GB each), registration_metadata/ (the resampled and moved images), and
ccf_anno_to_sample/ccf_anno_in_sample_space.zarr — the CCF annotation warped into the brain's own grid (level 0
at 8 × 5.984 × 5.984 µm, z y x, origin 0, i.e. multiscale level 3 of the imagery). The template → CCF half of the
chain (reg_exaspim_template_to_ccf_25um_v1.4, named in soma_detection/register_soma_process.log) is in neither
bucket's index. Two ways to a registration: (1) the owners run their exaspim-swc-transform +
aligned-swc-processing on these 14 tracings (what produced every ccf_space_reconstructions above), after which
registration("789202") works unchanged; (2) the template → CCF transforms are found and the chain is applied to the
gold nodes (needs ANTs or a NIfTI reader; neither is installed), checked on the asset's 92 somas, which carry both
xyz_raw and xyz_ccf_auto. Region lookups alone need neither: read ccf_anno_in_sample_space.zarr at a point's
own micrometres — it gives the 92 somas the same structure as the CCF annotation at their xyz_ccf_auto for 80 (82
equal or nested), and is not yet wired into io.ccf.
What this replaced. Until 2026-10-16 registration read 653158's
CCF_Atlas_Registration/displacement_swcs/ (one zip per neuron, one two-node SWC per node of the release's own
*_joined.swc) and paired file _i with gold node i. The paragraphs below are the record of that and of its first
correction.
s3://aind-open-data/exaSPIM_653158_.../CCF_Atlas_Registration/displacement_swcsholds one zip per traced neuron, and inside it one two-node SWC per gold node giving that node's position in CCF micrometres. The file index is the gold node index — checked node-for-node, 7,879 files against 7,879 nodes for N006 — so pairing them with the local gold gives a dense correspondence (257,670 nodes for 653158) without touching the 854 MB displacement field.exaproof.io.ccf.registration(subject)builds and caches it underccf/registration_<subject>.npz.CORRECTED 2026-10-16 (K5,
scratchpad/k5_denominator.out): the file index is NOT the local node index. The counts agree (31 of 45 neurons pair), the order does not: paired by index, two neurons' nodes within 10 µm of each other locally land more than 200 µm apart in CCF in 1,067 of 1,068 cases, and a gold arbor comes out ten times longer in the atlas than in the expanded brain. Each neuron's CCF points are its own arbor as an unordered cloud, soexaproof.io.ccf.rematched_registration(subject)re-pairs them by shape (cached asccf/registration_<subject>_rematched.npz): the same pairs then agree to a median 9.4 µm, and the gold somas move 0.6-6.4 mm, from isocortex (21 of 31 as released) to thalamus (24 of 31).registration()is unchanged, and every region split read through it on 653158 (agentic.scoreboard.Regions, the viewer's place lookup) is wrong until its caller switches.653980 has no
displacement_swcs— onlyaffine_transforms.matandwarp_transforms.nii.gz— so a confirmation on that brain needs the warp applied, which has not been done.
SUPERSEDED 2026-10-16 by Complete_annotated (above): the same tracings node for node, so nothing is inferred.
Against it the file-order cache was off by a median 1,179 µm per gold node (somas 3,775 µm; no node within 25 µm;
…_old.out), and the shape re-pairing of PR #78/#79 agrees with it to a median 36.7 µm (p90 66) — of which the two
published registrations' own difference is a part: N006's displacement_swcs cloud and its Complete_annotated
cloud lie a median 15.6 µm apart, nearest point to nearest point. Caches now carry a version, so the file-order one
(no version) is rebuilt on first use, never read; rematched_registration returns registration.
| file | where from | licence |
|---|---|---|
ccf/annotation_25.nrrd |
download.alleninstitute.org/.../annotation/ccf_2017/annotation_25.nrrd |
Allen Institute, research use with attribution |
ccf/structures.json |
api.brain-map.org structure graph 1 |
Allen Institute, same |
ccf/registration_<subject>.npz |
built from the public releases above (Complete_annotated, ccf_space_reconstructions) |
— |
ccf/depth_{pia,tract}_25_v1.npy |
built from the annotation (exaproof ops depth-maps --build, 3.7 min, ~2 GB peak; E1224, RESULTS §1049): uint16 CCF µm to the nearest voxel outside the brain and to the nearest fiber tracts voxel, read by op-read-depth-layer |
derived from the annotation; same |
Downloaded to the data root on first use and never redistributed from here.
Whole-brain affines for the agent's CCF read (agentic/ccf_affine.json, 2026-11-10, E1215). Three brains that had no
map now have one. 721332: registration("721332") builds from its reconstructions_2026-05-06 / 2026-08-28
ccf_space_reconstructions (494,011 pairs over 22 neurons, 587 s; the cache is ccf/registration_721332.npz); held out
one neuron at a time the affine errs a median 189.8 um (p90 377.5, leaf region agreement 0.586), the worst registration
brain. 794495, 820994: the CCF-registered soma list (soma_detection/soma_locations.csv with xyz_ccf_auto) sits in
an EARLIER processed asset than the imagery (_processed_2026-01-29_22-02-08, _processed_2026-05-20_16-45-09, beside
ccf_alignment/), so io.somas had cached the morphology bucket's list, which has the same rows and no CCF column;
io.somas._published now tries the subject's other _processed_ assets before the bucket, and the two caches were
replaced (the old files are kept as somas/<id>.csv.morph.bak and somas/morph_lists_2026-11-10/). Held out one soma at
a time: 794495 93.4 um (p90 209.2, agreement 0.714, 119 somas), 820994 89.1 um (p90 156.2, 0.742, 124 somas). Built by
context_ops.build_ccf_affine; outputs scratchpad/e1215/ccf_affine_new.json.
Tile seams (io/tile_seams.py, 2026-11-10, E1215). The fused volume's tiles and their aligned blocks, from the XML
the fusion ran on (fusion/processing.json's s3_xml_path, else tile_alignment/bigstitcher_affine.split.xml,
tile_alignment/ip_split_affine_alignment/bigstitcher_affine.split.xml, tile_alignment/rhapso/..., else the
session's _alignment_ assets), accepted only when the views' bounding box matches the fused level-0 shape within
16 voxels (all built maps: within 1.8). Built for 709221, 718162, 721332 (from _alignment_2025-04-29_13-08-50), 789202,
794491, 794492, 794495, 802449, 820994 and 841260; the fusion mode is recorded only on 794495 (LOWEST_VIEWID_WINS) and
820994 (AVG_BLEND). No map: 685221 and 685222 (none of the 55 / 9 XMLs in their _alignment_ assets spans the
fusion_2024-07-22 / 2024-07-29 shape; closest 583 / 134 voxels off: the registration those fusions ran on is not
published), 794493 (its XMLs are 1,718+ voxels off), and the 2023 fusions 609281, 648434, 651324, 653158, 653980 (no
alignment XML in any of their assets). RESULTS 1031's site B5 (794495 N008, "the fibre stops at a vertical tile seam")
reads 0.8 um from the tile 1 / tile 5 seam; a random point of the volume is within 10 um of a tile seam 0.6 % of the time.
2d. What the upstream segmentation is, and what is not published (2026-09-18)
The fragments this project joins come from a flood-filling network pipeline: the Allen Institute runs FFN-based segmentation with GNN post-processing on exaSPIM in collaboration with Google Applied Sciences (platform page). An FFN's own output is a voxel segmentation — a recurrent 3D convolutional network that "directly produces individual segments from a raw image", one object at a time, by moving a field of view to the frontier of what it has accepted (Januszewski et al. 2016, Nature Methods 2018).
Those voxels are not published. Searched on 2026-09-18 across the whole aind-open-data bucket — 69,395
top-level prefixes, of which 922 are exaSPIM assets — no prefix anywhere matches segment, ffn, flood or
mask. The exaSPIM collection publishes only these kinds:
| assets | kind |
|---|---|
| 337 | raw acquisition |
| 192 | alignment |
| 168 | processed |
| 56 | fusion |
| 48 | flatfield-correction |
| 20 | reconstructions |
| 2 each | fused, stitched, raw-fusion |
The only segmentation output is <fusion>/seg/*.zip, which is SWC skeletons. The precomputed/ and
precomputed_neuroglancer/ folders under the reconstructions prefix look promising and are not it: they are the
gold tracings as Neuroglancer skeletons.
Nor are they in aind-msma-morphology-data (re-searched 2026-09-24). Every store there that looks like
a voxel volume was opened and turned out to be something else:
| prefix | what it actually is |
|---|---|
segmentation/exaSPIM_<subject>_zarr/ (28 subjects) |
SWC archives only, despite the name (§2e) |
685221-precomputed-test/ |
its info declares a uint64 segmentation at 1 µm, but no scale data was ever written; only skeleton/ and segment_properties/ exist |
anna.grim/image-carveouts/{802449,754612}/…/mask.zarr |
binary 0/1 masks, likely drawn from gold tracings (§2e) |
EXASPIM25/{train,test}/<brain>/…/label_mask.n5 |
ground-truth voxel labels, not FFN output. metadata.json records an SNT Dijkstra fill grown from the human tracing (fill_method: dijkstra, fill_threshold: 0.135) |
exaspim_whole_brain_masks/exaspim_<subject>_medsam2/ |
whole-brain tissue masks from MedSAM2 on a downsampled image |
The EXASPIM23/ prefix in the 2026-09-18 index has since been deleted, and EXASPIM25/ replaces it. By its
README it is a segmentation benchmark: 1024³ blocks (some smaller), each with an input.n5 image, a uint8
label_mask.n5 and gold swcs/, at 0.748 × 0.748 × 1.0 µm. Counted: 60 labelled blocks, split into train
653158 (23), 609281 (11), J141R (10), 650010 (6), 613814 (4) and 651324 (2), and test 650010 (2) and 651324 (2).
The README says 61. This is the only voxel ground truth for brains this project measures (653158, 650010).
It would let our ROI/flood field be scored voxel for voxel. It does not provide an FFN mask to compare against.
Read with exaproof.io.exaspim25 (2026-09-24). fetch(brain) copies labels, SWCs and metadata, never images,
to external/exaspim25/. blocks(brain) reads them. The labels are instances: label k + 1 is
SNT_Data-000k.swc. A block voxel is chunk_origin + local on our level-0 grid.
This was verified on both 653158 and 650010: the block's input.n5 phase-correlates with our fused-denoised at zero
shift, and every SWC node lands on its label. 653158's blocks overlap: its 23 blocks form 16 groups
(overlap_groups). Scored first in §704 (the flood tracer).
Only three of the six brains can be scored on our volumes (checked 2026-09-25, e680_voxel_panel.py --verify,
which phase-correlates each block's input.n5 with our volume at two SWC sites per block):
| brain | blocks | on our grid? | why not |
|---|---|---|---|
| 653158 | 23 | yes | — |
| 650010 | 8 | yes (r 0.71–1.00, zero shift) | — |
| 651324 | 4 | yes (r 0.86–1.00, zero shift) | — but its train blocks are not labelled k + 1: block_000 is k + 2, block_001 2k + 1. Read the label from the fill: exaspim25.label_of |
| 609281 | 11 | no | cut from single raw tiles (image_url = tile_x_…), voxel_spacing in µm, block z at a 0.748 µm pitch. A sub-study placed all 11 on the fused octree (per-block offsets, r 0.8–0.95; not in the repository), but 609281's fused.zarr is deleted and reads zeros, so no model can run there |
| 613814 | 4 | no | labelled on the 2023-03-18 acquisition, whose fused.n5 is gone (octree only); the 2023-04-15 volume ImageSource opens is another acquisition (r ≈ 0) |
| J141R | 10 | no | no volume in aind-open-data; raw-tile coordinates |
Block.on_fused_grid is False for the raw-tile blocks.
Consequences, which are load-bearing for several results here.
- Calibre cannot be read from the corpus: every fragment node carries radius 1.0 (§244). Thickness has to be
measured from the imagery (
features.caliber) or from our own model's object (roi.flood.calibre). - There is no upstream voxel mask to compare our field against, so "our segmenter does better locally than the FFN" cannot be tested on equal terms — only against the skeletons, which is a different comparison.
- The FFN variant, its training data and its thresholds are, as far as can be found, unpublished for this application: Viren Jain's publication list has no light-sheet or expansion-microscopy work, and the exaSPIM pipeline is described only in Allen Institute material, not in a paper.
Worth asking the data owners, since the voxels demonstrably existed (the skeletons were derived from them): whether the FFN segmentation volumes were retained internally.
A detail that fell out of the same search
reconstructions/precomputed/segment_properties/info names 42 neurons with the annotator in the label —
N001-653158-AW, N006-653158-consensus, N011-653158-PL, N070-653158-DT, N015-653158-SK. That independently
confirms §334 (the gold is one person's tracing, and which person varies) and shows there are more annotators than
AW and PG: at least PL, DT and SK.
3. Verifying a copy (what was run on sandbox-04)
export EXAPROOF_DATA=/home/ubuntu/data/exaproof
python3 -m pytest -q # 105 tests
exaproof data status # per-subject presence table
# the end-to-end reproduction — expect median cov@60min 0.162:
exaproof session --subject 653158 --model a10 --threshold 0.60 \
--long-radius 250 --long-cone-deg 45 --asker askdyn
sandbox-04 (172.30.3.53, NVIDIA L4) verified 2026-08-27: full tree copied with sizes matching
the source, including the same-day 653980/648434 artifacts. The cache/ copy came across too,
so no S3 warm-up is needed there.
Imaging: exaSPIM versus MouseLight, and the coordinate space every external source must be converted into
Our coordinates are EXPANDED micrometres. exaSPIM images tissue after ~3× physical expansion, samples it at
0.748 × 0.748 × 1.0 µm in the expanded specimen, and the gold arbors were traced on that image (Janelia Workstation;
# ORIGINAL_SOURCE Janelia Workstation Large Volume Viewer). Verified 2026-09-16: gold arbors span up to 43 × 17 × 25
mm, against a native mouse brain of ~13 × 8 × 11 mm. Every external corpus below is in TISSUE (native) micrometres,
so any distance, curvature or rendered morphology taken from one must be scaled by the expansion factor (~3, the
published value; 648434's 43 mm span suggests ≥ 3.2 for that brain) before it is compared with ours. Angles are
scale-free; lengths are not.
| exaSPIM (this project) | MouseLight (Janelia) | |
|---|---|---|
| modality | axially swept light sheet, 3× expanded tissue (Glaser et al., eLife) | serial two-photon tomography, point-scanning, vibratome block-face (Economo et al., eLife) |
| sampling, specimen | 0.748 × 0.748 × 1.0 µm (expanded) | 0.30 × 0.30 × 0.99 µm (tissue) |
| sampling, tissue units | ~0.25 × 0.25 × 0.33 µm | 0.30 × 0.30 × 0.99 µm |
| optical resolution, tissue units | ~0.5 µm lateral, ~1.0 µm axial | 0.45 µm lateral, 1.33 µm axial |
| label | sparse Cre-dependent AAV, GFP; one channel in our data | sparse Cre-dependent AAV, eGFP / tdTomato; two channels |
| clearing | hydrogel expansion + delipidation (dilutes the fluorophore) | DMSO / sorbitol refractive-index matching (n = 1.468), gelatin embedding |
| typical artefacts | tile seams, refractive-index inhomogeneity at depth, uint16 saturation (609281) | a section boundary every ~250 µm in z, tile seams, uint16 saturation (65,520) |
| automated fragments | FFN skeletons | machine segmentation SWCs (segmentation/) |
| coordinates | expanded µm, image space | image µm in the imagery bucket; CCF µm in the NeuronBrowser export |
Effective resolution in tissue is similar; the sampling and the physics are not. MouseLight is sampled 3× coarser along z, its PSF is a two-photon point-scan's rather than a light sheet's, and its noise and background come from a different contrast mechanism. To put a MouseLight patch on our grid, resample by ~1.2× in x and y and ~3× in z (tissue → expanded).
MouseLight imagery on AWS Open Data — CC BY 4.0
s3://janelia-mouselight-imagery (anonymous; registry). Licence
Creative Commons Attribution 4.0 International; required citation: "Mouse Brain Anatomy: MouseLight Imagery was
accessed on [DATE] from https://registry.opendata.aws/janelia-mouselight." (first accessed 2026-09-16). Contents:
Horta KTX octrees and low-resolution TIFFs for 11 brains; machine-segmentation SWCs; consensus tracings; and, for
brain 2018-08-01, a full-resolution carveout, carveouts/2018-08-01/fluorescence-near-consensus.n5/ volume-rechunked — uint16, gzip N5, 33,792 × 25,600 × 13,312 × 2 voxels (x, y, z, channel), blocks
128 × 128 × 64, stored only near the 133 traced neurons (all green channel), with the tracings in the same image
space (transform.txt: origin and level-0 voxel size in nm, divided by 2^(nl−1) for full resolution). Verified: a
traced node reads 65,520 in channel 0 against a block median of 12,339.
Gold QC — stretches traced into two neurons (rows 244, 245, 2026-10-24)
Seven of the eight gold brains have axon traced into two gold neurons along one path (685221 113.9 mm, 651324 99.5 mm,
685222 10.0 mm, 653158 8.2 mm, 789202 1.3 mm; 609281 and 653980 under 1 mm); the held-out 648434 has none. The per-brain
table, the renders, what it means for scoring and training, and the questions for the annotators are in
docs/GOLD_QC.md (repository file); the opt-in filter is exaproof.gt.shared.SharedGold.
Leakage register — released datasets that contain OUR brains (row 229, 2026-10-23)
Allen Institute papers release data from the mice this project scores. Before any released morphology set or image
set is used as a reference, a prior, training data or a benchmark, check its subject ids here (docs/METHOD.md rule
148). The KB carries the same ids as each source's or paper's data_subjects; the #234 guard drops those items for
those brains.
| release | what it is | subjects | verdict | evidence |
|---|---|---|---|---|
su2026: Su et al. 2026, Nature, doi 10.1038/s41586-026-11026-0. Code: AllenNeuralDynamics/aind-LC-morphologies. Data: Code Ocean asset LC_Snapshots 12f47d79-…; public copies at s3://aind-open-data/exaSPIM_<subject>_…_reconstructions_2026-05-06_*/ccf_space_reconstructions |
132 LC-NE whole-axon reconstructions, exaSPIM, CCF | 648434 (12), 685221 (60), 685222 (38), 721332 (22) | OUR GOLD on all four. Never a prior or reference for these brains; usable held out by brain for the others (120 neurons for 648434) | The notebooks print "LC Neurons: 132 Mice: 4" with those samples. The neuron ids are ours (N004/N011/N020/N023/N024-648434). The snapshot …LC-manuscript_2026-05-06_10-44-00 matches our reconstructions folders of 2026-05-06 13:28–13:48. Somas equal the paper's table to 0.1 µm. CCF axon length is 1–3 % under the paper's. <data>/external/su2026/PROVENANCE.txt |
| grim2025: Grim, Chandrashekar, Sümbül, arXiv 2501.01022 (supervoxel loss) | its "new public 3-D light-microscopy mouse-brain dataset" is s3://aind-msma-morphology-data/EXASPIM25 (paper footnote 3), which this repo reads as external/exaspim25 (§2d). Voxel labels are SNT fills grown from the human tracings |
653158 (23 blocks), 609281 (11), J141R (10), 650010 (8), 613814 (4), 651324 (4); none from 648434 | OVERLAPS 653158, 609281, 651324 (gold brains) and 650010, 613814 (registry subjects). Independent of 648434. Its labels are our tracings' fills: never a held-out test for those brains | Paper §Experiments and footnote 3 (<data>/external/leakcheck/grim2025.pdf); block counts from the bucket index (§2d) |
glaser2025: Glaser et al., eLife 2025, the exaSPIM paper (papers/exaspim_2025_elife.pdf) |
exaSPIM imaging datasets | exaSPIM_708373, exaSPIM_671477 (mouse), Z13288-QN22-26-036 (macaque), H17.24.006-CX55-31B (human) | INDEPENDENT of every gold brain, 648434 included. Our registry has no gold for 708373 or 671477 (Table A) | Its "Data availability" statement. None of our nine ids appears anywhere in the text |
Our own models are not held out everywhere (row 287, E906, 2026-10-25). The deployed instance model
inst_s9_ho648434_bndffn is held out of 648434 ONLY. Its e726.json lists its training brains as 653158, 653980, 789202,
685222 and 685221, with 178 soma blocks; 685221 alone contributes 60 soma blocks (e716). Any read of it on 685221, 685222,
789202, 653158 or 653980 is in-sample, and has to be labelled so. A held-out read on the LC brains needs a model retrained
with both 685221 and 685222 left out. Done by row 300 (E919, 2026-10-25): inst_s9_ho648434-685221-685222_bndffn is held out of all three LC-NE brains (267 blocks of
653158, 653980, 789202, 609281, J141R, 613814, 651324; e726.json carries leak_check), 434 s for the same 1,500 steps; its first
reads are RESULTS §813. It is a weaker model (80 of 178 soma blocks), and the deployed model stays in-sample on 685221 and 685222. The fragment stores are not ours (row 287 provenance): 685221 is a GNN-corrected
FFN whole-brain run, 685222 a GNN-corrected run, 653158 / 653980 / 648434 FFN skeletons from the fusion asset, and
789202 a U-Net run.
Independent LC-NE sources (subjects not ours), checked for row 229:
- The Allen Mouse Brain Connectivity Atlas has 5 Dbh-Cre_KH212 injections including LC: 519735413, 480689656, 159510205, 159942862 and 287029894. They are downloadable as 25 µm projection density (
external/allen_connectivity/), under Allen Institute terms of use. Row 229's E849b read them as a tract field: NULL-HARM. - schwarz2015 (TRIO) and kebschull2016 (MAPseq on LC) release region-level projection data, not single-neuron paths.
- mckinney2023 (Dryad) has local slice morphologies only.
- MouseLight has 0 somas in LC (1 within 200 µm, AA1505).
External morphology corpora — licences and attribution
Neither of these is imagery. Both are skeletons only, which is what makes them usable: a branching angle is cheap to store, and §216 shows one half of it is a property of the neuron rather than of the instrument that recorded it. Each carries a licence that permits reuse with attribution, and that attribution is recorded here because the licence requires it and because a corpus whose provenance is not written down is a corpus nobody can check.
| corpus | path | licence | what it is |
|---|---|---|---|
| CORAL | external/coral_swc (32 neurons, 87 MB) |
CC BY 4.0 | Whole-brain light-microscopy tracings, hand-annotated. §135's branch-angle prior comes from these. |
| MouseLight | external/mouselight_swc (1,651 neurons, 340 MB) |
CC BY-NC 4.0 — non-commercial; use in this repository was authorised by the repo owner on 2026-09-15 | Janelia whole-brain serial two-photon tomography single-neuron reconstructions with complete axonal arbors (an earlier version of this row said light-sheet; that was wrong). The only external corpus with whole-brain axons and public full-resolution imagery (section above). Fetched 2026-09-16 from ml-neuronbrowser.janelia.org (GraphQL listing, POST /export format 0; scratchpad/fetch_mouselight.py). Coordinates are TISSUE micrometres in CCF space (verified: node step median 8.6 µm, arbor extent ~3.4 mm) — multiply by the expansion factor (~3) before comparing any length with exaSPIM's expanded coordinates, node types 1/2/3 = soma/axon/dendrite; each file's header carries its own DOI. Nothing derived from it is redistributed, and no commercial use. |
| MICrONS | external/microns_swc (300 neurons, 129 MB) |
CC BY 4.0 | minnie65 v661 skeletons from the IARPA MICrONS cortical mm³ electron microscopy volume, machine-derived from a volumetric segmentation at 1.5 µm. Retrieved 2026-09-13 from s3://bossdb-open-data/iarpa_microns/minnie/minnie65/skeletons/v661/skeletons/, selected by file size (the 300 largest). The bulk v661 set is documented as automatically reconstructed with little to no human proofreading, so the selection was checked against meta/ax_proof_v661_sids.csv: 270 of the 300 are axon-proofread, and the other 30 carry no axon bifurcations at all (§216). Cite per the MICrONS Explorer citation policy. |
What they are for, and what they are not for. MICrONS is not training data for this segmenter: electron microscopy has no fluorescent label and an unrelated contrast model, and no downsampling makes it look like light-sheet. It is an independent morphology corpus, and §135's entire prior rested on 32 CORAL neurons — MICrONS supplies 33,555 axonal bifurcations against CORAL's 4,899, from a reconstruction process with none of the same failure modes. §216 uses exactly that independence: sibling divergence agrees to 7.2° across both of these and all three exaSPIM brains, while the continuation turn does not agree at all and tracks how the skeleton was made.
Sources checked and their status, so nobody re-opens a closed question
| source | licence | status |
|---|---|---|
| Brain Image Library (BIL) — BICCN whole-brain fMOST single-neuron SWCs (Zeng lab, Allen Institute / Luo) | CC BY-SA 4.0, re-confirmed verbatim 2026-09-14 on brainimagelibrary.org/download.html: "BIL Data is distributed under the Creative Commons Attribution-ShareAlike 4.0 International License." Attribution required; derived datasets carry CC BY-SA. |
RECOVERED AND IN USE (2026-09-14, ~12:25 PDT). download.brainimagelibrary.org answers again (the morning outage was the data tier; doi. still returns 500). /biccn/zeng/luo/fMOST/cells/mouseID_*/1/*.swc holds 5,094 complete reconstructions across 32 mice, standard SWC with compartment types (radii all 1.0, like ours) — but coordinates are anisotropic voxel indices of the fMOST volume, not µm (lateral ≈ 0.30–0.35 µm, axial ≈ 1 µm; one mouse's terafly grid is 35,000 × 22,793 × 10,553): scale before any metric use (RESULTS §269 note, E347). A sample of up to 10 per mouse — 311 SWCs, 477 MB — is at /home/ubuntu/data/exaproof/external/bil_fmost_swc/ with PROVENANCE.txt. First use: §236-style branch-angle prior comparison (E339) — read in §269: continuation turn p75 26.4° (pipeline-dependent, as §216), sibling p5 16.8° (the one corpus whose tail departs). Attribution: Brain Image Library, BICCN fMOST reconstructions, CC BY-SA 4.0; nothing derived from it is redistributed. |
| SELMA3D | challenge data, MICCAI 2024 | Light-sheet, but blood vessels, c-Fos cells, Aβ plaques and nuclei — not neurites. Wrong structure for a morphology prior. |
| Allen Institute Cell Types Database | public API, no download | Used (§236). api.brain-map.org/api/v2/data/query.json?criteria=model::NeuronReconstruction publishes average_bifurcation_angle_local for 701 reconstructions / 19,426 bifurcations — a fourth modality (patch-clamp slice) for two HTTP calls and no bulk transfer. Median 74.5°, inside the 72.2–83.3 range the other five corpora span. |
| Allen Institute full-morphology (fMOST, CCFv3-registered) | not checked | A different release from the row above — whole-brain fMOST rather than slice. Probed 2026-09-14: api.brain-map.org is up and answering (HTTP 200), and ApiCellTypesSpecimenDetail filtered on nr__reconstruction_type = 'full' returns 367 rows — but those are the patch-seq/slice reconstructions of the row above, not the whole-brain fMOST release. The fMOST release is not served by this API and its access path is still unexplored. |
| Janelia MouseLight | see the licensed-corpora table above | NOW IN USE (2026-09-15, owner-authorised): 1,651 SWCs under external/mouselight_swc. The NC term stands — internal research only, nothing redistributed, no commercial use. |
The trap in that row, recorded because it would have caught me. Janelia's general open-science page states "CC BY 4.0 terms for Janelia data" with a link to the CC BY 4.0 deed — and hedges it with "Most (but not all) of such resources". MouseLight is one of the "not all": its own terms page carries the NC variant. Read the project's page, not the institution's.
| NeuroMorpho.org (v2 API) | per-archive; site terms require citation | Reachable, not used. neuromorpho.org/api/neuron?page=0&size=1 answers HTTP 200 with JSON. It aggregates hundreds of archives with per-neuron provenance and per-archive terms, so it is a place to find corpora rather than a corpus, and any use has to resolve the licence of the specific archive drawn from. Noted as a path, not opened. |
s3://aind-msma-morphology-datais a different matter and is covered above: anonymously readable, not part of the AWS Open Data Program, and carrying no licence. It is internal, and nothing is built on it or redistributed from it.