← Exaproof  ·  all docs  ·  understand it  ·  README.md

The repository

exaproof

Automated tracing and proofreading of neurons in exaSPIM whole-brain light microscopy.

Everything is documented at https://trace.scidex.ai/docs — the same files as below, rendered on request so a page cannot drift from what the project maintains, arranged by what you want:

level pages
use it the tracing desk — every layer, button and key in the viewer (illustrated at /manual)
understand it the models · the design · the data · this file
check it what is settled · every experiment · how claims are checked · what is running
run it every command · running an experiment · layout

GOAL.md (repository file) is what the project is trying to do. This file is how to run the code.


What this is

Brain 653158 publishes 1,479,239 automated fragment SWCs next to 45 complete gold tracings and a fused 21605 × 28886 × 54049 OME-Zarr volume, all in one micrometre world frame. That combination — automated segmentation, ground truth, and imagery on the same brain — makes an end-to-end split/merge task the proposals believed did not exist, and it makes real decision-level supervision minable rather than synthetic.

exaproof uses it to reconstruct one neuron at a time from a soma seed: it grows through the fragment graph, cuts suspected merges before it joins anything, resolves competing reconnections as a global assignment under endpoint exclusivity, buys image evidence only where that evidence can change the answer, and hands back a ranked, clustered worklist of what it declined to decide — with a certified bound on how often the things it did decide are wrong.

Install

pip install -e .                          # the core; pyproject.toml is the authority on versions
pip install -e '.[torch,llm,viz,dev]'     # the extras, each needed only by the commands named below
export EXAPROOF_DATA=~/data/exaproof      # defaults to this anyway

The core needs numpy, scipy, scikit-learn, lightgbm, zarr, s3fs, fsspec, networkx, typer, rich, pyyaml and tqdm. The extras are imported only when a command needs them: torch by anything that trains or runs a network (the roi-* commands, grow, field-cache, encode, encode-pooled, bridge-train, restore-train, and any command given --roi-model or a rescorer); llm by learn improve without --offline; viz for matplotlib, by the commands that draw figures (the viewer itself runs on the standard library); dev for pytest. smoke and simulate drive a headless Chromium through Playwright (src/exaproof/viz/browser.py), which pyproject.toml does not list. The AWS CLI is used for the bulk downloads, anonymously. What data fetch takes from s3://aind-open-data is CC-BY-4.0; DATA.md §1 has the licence of every other source, and one of them is read but never redistributed.

Getting the data

exaproof data fetch   --subject 653158     # gold tracings (23 MB) + fragments (2.0 GB)
exaproof data ingest  --subject 653158     # -> columnar store + spatial index, ~1 min
exaproof data verify  --subject 653158     # confirms one coordinate frame, against voxels
exaproof data status

data verify is not a formality. It samples gold nodes and measures image intensity against the local background; on 653158 gold nodes sit at a median 16× background with 100 % above 1.5×, which is how the frame was established rather than assumed.

Seven subjects have fragments and gold and imagery — 653158, 653980, 648434, 609281, 789202, 685221 and 685222 (io.datasets.END_TO_END_SUBJECTS, the authoritative list; DATA.md §2f has what each one holds) — so brain-held-out evaluation uses real fragments, not injected errors.

Measuring the baseline

exaproof gt align    --subject 653158        # project gold onto fragments (cached)
exaproof gt baseline --subject 653158        # the number to beat, with a tolerance sweep

The raw automated segmentation on 653158, 45 arbors, 8.09 m of cable, at 8 µm tolerance:

value published 794495
edge accuracy 76.6 % 73.78 %
normalized ERL 0.0201 0.0095
splits / neuron 819 571
merges / neuron (50-node threshold) 11.3 6.9
omitted edges 13.6 % 1.36 %

The alignment tolerance is a reported parameter, never a hidden constant — gt baseline prints the full sweep, because at 5 µm the same segmentation reads 21 % omitted and at 12 µm it reads 10 %, and neither number means anything without the other.

Mining supervision, and training

exaproof decisions mine --subject 653158 --tag v3
exaproof train          --subject 653158 --tag v3 --name default --alpha 0.01 \
                        --image-neurons 6

decisions mine walks each gold tracing over the real fragments: where it crosses from fragment A to fragment B, a human implicitly decided those two are the same neuron. Every other candidate the proposer generates at that site and is not taken is a hard negative. On 653158 that is 116,677 labelled reconnection decisions — 40,444 positive, 76,233 negative — over 24,709 true fragment adjacencies at 97.4 % proposal recall.

Proposal recall is printed because it is a hard ceiling: no classifier can join what was never offered. It started at 0.289 — the generator only proposed endpoint-to-endpoint joins, while a third of true adjacencies are endpoint-to-interior branch attachments that such a generator cannot express at all. That single fix moved the ceiling more than any model change could have.

train fits two heads — geometry-only for screening, geometry-plus-voxels for rescoring — and certifies an operating point rather than tuning a threshold: given a tolerated false-accept rate α, the accept threshold comes from a Clopper–Pearson bound on the fraction of accepted decisions that are wrong. With ~570 reconnections per arbor (GOAL.md §1; this line said ~800 until 2026-09-19), a 1 % false-accept rate manufactures as many merges as the raw segmentation already contains, so precision on accepts is the constraint that matters and it is the one that gets a guarantee.

When the requested α cannot be certified, the run says so and reports the tightest α that can be. That is not a hypothetical: on 653158 geometry and priors alone top out at α = 0.061, against a problem that needs ~1 %.

Adding image evidence was the obvious way to close that, and a matched control — same decisions, same split, differing only in whether the image columns are present — says it mostly does not: the certifiable floor moves 0.0675 → 0.0638, about 5 % of the ~6× needed. Twelve hand-designed patch statistics are not a substitute for a learned representation over the patch. RESULTS.md §5–§7.

Proofreading a neuron

exaproof proofread N001 --subject 653158 --minutes 55

Writes reconstruction.swc, worklist.json (what it would not decide), journal.json (every edit with its evidence, replayable and revocable) and result.json (per-round telemetry, wall-clock, image reads).

Two strategies, selectable with --mode. sequential grows the neuron a frontier at a time: detect and cut merges in the regions that changed → regenerate proposals there → score with geometry → buy image evidence only inside the uncertainty band → accept under the risk bound → defer the rest. global collects candidates over the whole neighbourhood without committing, then solves one maximum-weight forest under endpoint exclusivity, acyclicity and a cable budget. RESULTS.md §8 compares them: global buys precision at the cost of recall rather than beating the trade-off, and both percolate below an accept threshold of ~0.3. Merges are cut before splits are joined because ERL gives a merged label run length 0 at full weight: joining clean fragments into a contaminated segment scores worse than doing nothing, which exaproof bench exploits reproduces on demand.

Looking at it — the proofreading interface

This section is how to launch it, share it, and what it records. The per-control reference — every layer, button and key, what to trust and the known gotchas — is docs/VIEWER.md. Keep new UI detail there and launch/deployment detail here; the two have drifted into parallel descriptions of one interface before.

exaproof view N001 --subject 653158 --port 9000        # reconstructs, then serves at :9000

With the calibrated chooser and the segmenter's field, which is the configuration the measurements support:

exaproof view N001 --subject 653158 --port 9000 \
    --choice-model scratchpad/e184_model_from_648434_context.json --roi-model ms_100

Serves the gold tracing, the local fragments and the reconstruction as Neuroglancer precomputed skeleton layers over the S3 imagery, draws every deferred decision as a line between the two endpoints it is asking about, and puts a clustered, ranked worklist beside the viewer with keyboard accept / reject / cut / skip.

The walker's fork questions. Record them by passing a recorder as grow_cable's ask (it answers nothing, so the walk is unchanged), then add them to the list as fork rows; exaproof.viz.simulate.answer_forks answers them from the gold over the same API:

from exaproof.roi.forks import ForkRecorder
rec = ForkRecorder(subject="653980", neuron="N001", walk="w0")   # set rec.walk before each walk
grow_cable(image, est, seed, direction, ask=rec, ask_margin=0.05)
rec.save("scratchpad/forks_N001.json")
exaproof view N001 --subject 653980 --forks scratchpad/forks_N001.json   # 1-9/n look, Enter answers, q none

Sharing it with other people. Add --auth and every route needs a signed-in account, and every logged decision carries whose it is (src/exaproof/viz/auth.py):

exaproof users add ann@alleninstitute.org --name "Ann"      # prompts for a password
exaproof users list
EXAPROOF_SIGNUP_CODE=$(openssl rand -hex 6) \
    exaproof view N006 --subject 653158 --host 0.0.0.0 --port 9000 --auth ...

With EXAPROOF_SIGNUP_CODE set, people create their own password account with that code on the sign-in page; without it, only exaproof users add and Google sign-in create accounts. Google sign-in appears when GOOGLE_OAUTH_CLIENT_ID, GOOGLE_OAUTH_CLIENT_SECRET and GOOGLE_OAUTH_REDIRECT_URI (https://<host>/login/google/callback) are set, and EXAPROOF_ALLOWED_EMAILS (a comma-separated list of addresses or @domain suffixes) limits who may use it. Sessions are cookies over 30 days; /account shows a person their own decisions and changes their password.

A public URL needs HTTPS, because the embedded Neuroglancer is served over HTTPS and browsers block its fetches to an http:// source (localhost is the exception, which is why the tunnel below works). The deployment here is nginx on sandbox-02 terminating TLS and proxying to this machine: scratchpad/ holds the config as trace.scidex.ai.conf, live on sandbox-02 as /etc/nginx/conf.d/trace.scidex.ai.conf since 2026-09-17. The viewer needs --host 0.0.0.0 so the proxy can reach it, and it sets the session cookie Secure whenever X-Forwarded-Proto says the browser used HTTPS.

The viewer behind it runs as the systemd unit deploy/exaproof-viewer.service, which restarts it after a crash, an earlyoom kill or a reboot (its header has the install commands). A 502 from trace.scidex.ai means that unit is down: systemctl status exaproof-viewer, journalctl -u exaproof-viewer. Change the neuron or flags in the unit, not by launching a second viewer on port 9000.

Before deploying a viewer change, run both gates against a viewer started from the branch on a spare port (view N006 --subject 653158 --port 9017 --no-image --log-dir <scratch> is enough for layout): exaproof smoke (every page, tab and tool works) and exaproof shots --url http://127.0.0.1:9017 --out <dir> (nothing overlaps, clips or scrolls sideways at four screen sizes), and pytest tests -m browser (seven tests, ~30 s; the fast suite leaves them out, and on 2026-10-15 one of them was the only check that saw /api/desk return 500 on a server with no brain loaded, shipped a day earlier). Smoke cannot see layout: on 2026-10-15 a Help tab and a Demo link had pushed the tab row 70 px past the side panel, and the whole panel scrolled sideways on the live site for two deploys while smoke passed. A --no-image viewer cannot show what only imagery triggers: the blank-imagery warning scrolled the live page 150 px sideways at 1024 px (now a browser test). After a deploy, shots the live site too — and know that smoke there walks the subject switches and leaves the shared desk on its last brain (609281, whose imagery is gone): reload 653158/N006 afterwards (sudo systemctl restart exaproof-viewer does).

Reaching it from your laptop. The server binds the host machine; open an SSH tunnel and use localhost (the embedded viewer is HTTPS and the local skeleton sources are HTTP — browsers allow that for localhost but block it for a bare public IP):

ssh -L 9000:localhost:9000 <host>      # then open http://localhost:9000

The default bind, 127.0.0.1, is what the tunnel reaches; --host 0.0.0.0 is for the proxy above or for another machine on the network. The imagery streams from public S3, so no credentials are needed in the browser.

What you are looking at. Layers (toggle in Neuroglancer's top bar):

layer colour meaning
exaSPIM greyscale the raw light-microscopy imagery (window adjustable in the shader)
reconstruction orange what automation has built for this neuron so far
gold green the reference tracing, when one exists (absent for a fresh seed)
fragments grey the local FFN fragments available to join
deferred decisions yellow line a join/gap the model would not commit — endpoint A↔B
suspected merges red point a place the model thinks two neurites were wrongly fused
tube tree default (rows 265/272), tube tree (row N) blue / red / yellow a gold-free tube-tree run of record (viz/tree_runs.py TREE_RUNS): its tree and attached pieces by node ownership at 8 µm — own, foreign, off-gold (pieces lighter). Off until switched on in the Gold tab, whose "show in 3D" button opens the whole neuron beside the gold with the run's recall@8 and own share

Selecting a worklist item recentres the camera on that decision and rotates it so the gap lies across the view (both stubs and the dark stretch between them visible at once, not edge-on).

The order is the point. The worklist is ranked and re-ranks after every decision: a suspected merge outranks a join (a false merge corrupts ~20× the cable); a path review outranks a single join; larger un-worked clusters come first; decisions on longer fragments (more cable at stake) and ones the model is least sure about float up; once you answer in a neighbourhood, its remaining questions get a bonus so you finish it while its imagery is loaded; and with --choice-model, each item is multiplied by the model's expected yield — its own probability that there is anything there to join at all.

That last term is the one that is calibrated rather than tuned, and it is there because the others answer the wrong question: they ask where an answer would be most informative, while an hour of annotator time is spent on what is most productive. Measured over a session's scanned ends, and after the barren-question rule has already fired, ordering by it roughly doubles the productive share of the first hundred-odd questions (RESULTS.md §122). A caution that belongs next to it: the same ordering inside the automated session does not reliably help — it reverses on one of three brains (§125), because there an ordering also changes how much work exists. A worklist is built once and only re-scored, so that failure mode does not apply here, but the two numbers are about different things and should not be quoted for each other.

Deciding. Keys (mouse buttons mirror them):

key action
j / k next / previous item
a accept — the two endpoints are the same neurite; commit the join
r reject — they are different neurites; leave them apart
c cut — this is a merge; separate the fused neurites
s skip — undecided, come back later
1–9 on an item with a proposed chain, accept the first n hops; on an end with competing continuations and no chain, look at alternative n
n with --choice-model, cycle the competing continuations at this end — each is framed along its own gap with the others still drawn
Enter join the framed competitor
q none of these continues it

Competing continuations (--choice-model <fit.json>). A path review asks yes-or-no about the model's single best chain. The decision you are actually making at a stalled end is which of these, if any — so with a fitted choice model the item carries every nomination the scan made, each with a calibrated probability, and P(none of these) on the same scale, because they come out of one softmax over the set. All of them are drawn at once. That matters most on the common case: roughly three quarters of the questions a session asks are barren, and the model's own answer to "is there anything here" is the fastest way to dispatch one.

Two honest details you will see. If some competitors have since been joined into the reconstruction they are dropped and the header says so — the remaining probabilities then no longer sum to one and are deliberately not rescaled, because a rescaled softmax is a number that looks like the model's and is not. And opening an item re-scores it against the reconstruction as it stands, since an end with nothing to join early can have something later.

With --roi-model as well, the alternatives are ordered by the chooser's probability combined with the segmenter's join level — the level at which its predicted field connects the end to each candidate — which picks the right continuation more often on three brains (FINDINGS.md §4). Each alternative shows "joins at 0.85" and its rank under the chooser alone; when the order changes, a line under the header says why. When the chooser is already confident (top probability ≥ 0.65) the prediction is skipped, because it never changed such a pick, and the panel says so.

A pick is logged as human_choice rather than human_path: confirming the model's own proposal and overruling its ordering are different acts, and the second is the more informative. The log also records the chooser's own rank of the pick.

The agent's checklist. With --playbook <file> (fitted by exaproof playbook-fit on another brain's stall ends; it needs --choice-model), each end also shows what an annotation agent would do — nothing here, join, or needs a person — as a numbered checklist: what it read, how sure it became, and the named reasons the case is hard (for example "two candidates at 7 and 9 µm" or "three fibres run alongside within 12 µm"). Its verdict is logged beside yours. On real stall ends it closes about a quarter as empty at 0.97 precision and does not yet join reliably (FINDINGS.md §6), so treat its join suggestions as suggestions.

Check this branch. Most stall ends that lie off the neuron sit on long fragments the automatic segmentation merged across two cells (FINDINGS.md §5). The check this branch button (or branch in the Ask tab) says how far along its fragment an end is from where the reconstruction entered it, and marks in magenta the most merge-like places on that stretch, which point 2–3× closer to where a foreign branch leaves than chance. It does not decide.

Tabs beside the worklist.

  • Decide: the worklist and the tools above.
  • Ask: type help, explain, view, next, progress, conflicts, gold on / gold off or agent start / agent stop. The system answers these itself. Other questions are written to runs/<subject>_<neuron>/assistant_inbox.jsonl, where a Claude session watching the file can answer them (POST /api/answer); the page says a question is waiting until then. Comments (with tags such as crossing, faint or gold looks wrong) are saved with the decision and added to the run's hard-case registry.
  • History: every logged event, newest first, with the model's and the agent's numbers.
  • Gold: switch the gold tracing and each annotator's own tracing on and off. It lists the places where two annotators disagree, and where two gold neurons trace the same fibre (gt.annotators), each with a button to go there.
  • Agent (needs --playbook): the agent works the open stall ends in the background. It closes the ones it is sure are empty, and joins only if you allow it. The list can be ordered with the agent's least-sure decisions first, and your answers refit its calibration.

Why the system thinks this. Under the viewer, each item shows the evidence the decision loop gathered (exaproof.adaptive): what it measured, what each measurement said, and when it declined to look further. The look buttons render the same views a looking agent uses (plane, field of view, contrast), and each look is logged beside the decision. The look and merge-check panels also say what the image shows at that point, in one line with the numbers behind it (exaproof.adaptive.scene): part of the view outside the imaged volume, a cell body, dense labelled neuropil, low contrast, or faint cable on the fragment itself.

Every human action is appended to runs/<subject>_<neuron>/decisions.jsonl with the model's probability, epistemic/OOD uncertainty, cable-at-stake, and the fragments involved. That is deliberate: light microscopy has no CAVE-style edit history, so capturing decisions from the tool is the only mechanism by which the corpus grows past what can be mined from existing gold — and it is what turns the simulated-human session numbers (VALIDATION.md §5) into real ones.

A timed row — one per answered question, written by /api/decision (yes or no on a deferred join or merge), /api/choose (j: which competing continuation at a stalled end, or none) and /api/path (k: how many hops of a proposed chain were accepted) — carries, beside index, action, item and user:

field what it is
dwell_ms wall clock from the item being opened to the answer. It keeps this meaning whatever else is logged; the fields below are subtracted by the analyst, never here
away_ms, idle_ms of that: time with the tab hidden, and time with no pointer, key or wheel event for longer than 45 s
score_wait_ms of that: time spent waiting on the server — from the request for the item's view to the later of the view being answered and the item's scoring (competitors, join levels, the agent's checklist) first becoming available. 0-ish when the background agent prepared the item before it was opened; null when the server never saw the item opened (a decision posted without a view). METHOD rule 33: a dwell read whole measures the machine's load that afternoon, so scratchpad/e176_measured_costs.py reports medians net of this beside the raw ones
view_toggles how many times v (the 90° roll) was pressed on this item — the one difficulty signal the interface collects for free
session_id one short id per sitting at one neuron, minted when the server starts and again when the picker switches neuron; every row, timed or not, carries it. Two sittings at one neuron share a file, and the first decision of each is orientation, not a decision

overruled_decline and decline_p mark an answered question the skip rule had hidden; p_model, p_none, model_rank, chooser_rank and agent sit beside a choice so the disagreement with the model is on the row.

Switching neurons. The brain/neuron picker at the top of the worklist lets you start from any neuron without relaunching: choose a subject and neuron and press Load — the server reconstructs it in place (~½–2 min; the brain's fragment store and model are cached, so switches after the first are quicker) and repopulates the worklist. exaproof precompute --subject <id> builds those reconstructions ahead of time with the viewer's own settings, so a switch is instant. The catalog lists only neurons that actually reconstruct (653158: 45, 653980: 21, 648434: 12, 609281: 25). The neuron on the launch command is just the starting point.

Notes: --no-image skips only the server-side imagery reads; the browser still streams the imagery.

Evaluating

exaproof session --subject 653158 --model a10 --threshold 0.60

This is the only command denominated in what the one-hour target is actually about: human decisions per neuron. The system grows, stalls at a join it will not commit, asks one question, and continues. Watch it when changing anything — batch precision and recall can both improve while this does not.

--threshold -1 runs at the operating point the risk bound certified instead of 0.60; session --help says why 0.60 is nonetheless the default. --inward and --propose-from-generated change which questions exist, so a run with either is not comparable with stored arms without it — each says so in its help. --skip-barren fits the barren-question skip rule on past session logs; certify its threshold per brain with exaproof skip-certify first.

Add --long-radius 250 to let the human be asked about continuations across segmentation omissions (automation never commits beyond the proposal radius; a directional cone gates the wide scans). Over the dev split at threshold 0.60: median final coverage 0.495 (0.766 ungated), median coverage within a 60-minute human budget, benchmark cov60 (current value on /leaderboard), which rose over the run before the frontier fix and long-range questions. The remaining gap is question targeting and automation share — RESULTS.md §23 has the decomposition. The one-hour budget (415 joins in 120 questions) needs a leverage of 3.46.

exaproof eval  --subject 653158 --split dev
exaproof bench exploits          # the three metric exploits, reproduced as tests
exaproof bench splits            # neuron-level splits and the held-out brains

eval runs the agent on held-out neurons and reports the full panel plus join precision and recall against real gold adjacencies, deferred count, wall-clock, and image cost. The headline is human decisions per neuron, not accuracy: the goal is one neuron per hour, and at ~30 s of human attention per decision an hour buys ~120 of them.

eval also reports the number the goal is actually about: the ratio of unaided human minutes to human minutes with the system, both counted the same way (sites collapsed per endpoint, clustered into vignettes).

These are the first end-to-end measurements (RESULTS.md §9); later work changed the picture substantially — see FINDINGS.md §1.

Median saving 0.97×, on every neuron tested, at every operating point tested. The system costs human time rather than saving it. Median join precision 0.556, median recall within reach 0.110; it removes the 13–52 joins it gets right and adds every site it defers at that was not a real join. RESULTS.md §9.

The per-neuron binding constraint turns out to be reach, not precision: only 24 % of an arbor's true joins are ever collected by breadth-first exploration, because connected growth from a soma makes end-to-end recall the probability of an unbroken chain (q^k). A best-first frontier is ~50× more efficient per fragment and holds precision at 0.73 instead of 0.016 (RESULTS.md §9.3) — and still does not survive the chain. The same model scores recall 0.545 at precision 0.846 on the mined corpus and delivers r = 0.11 end to end.

The accounting closes in one inequality: with recall r, the best achievable saving is 1/(1−r). At the measured r = 0.11 the ceiling is 1.12×; reaching 60 minutes from a 102-minute baseline needs r ≥ 0.41, and the proposals' 120-decision budget needs r ≥ 0.71. Recall cannot be bought by lowering the threshold, because the component percolates from r ≈ 0.13. Both coordinates have to move, precision by two orders of magnitude. RESULTS.md §9.3.

Proofreader-Time Equivalent is reported as a band across plausible coefficients rather than a point score, because no public exaSPIM timing calibration exists. Note the coefficients are not the 20× cable-damage ratio of a merge to a split — a merge may take one cut to fix, while chasing a dim terminal branch can take minutes. Metric damage and human cost are different quantities and the panel keeps both visible.

Learning from its own mistakes

exaproof learn mistakes --subject 653158
exaproof learn improve  --subject 653158 --rounds 1

mistakes clusters the system's false accepts and false rejects in feature space and describes what distinguishes each cluster. improve hands those clusters, the feature registry and the performance table to Claude, which writes new feature functions; each one is statically screened, sandboxed, validated on a real batch, rejected if it duplicates an existing column, then measured — promoted only if it improves the dev split and survives the validation split. Every attempt lands in a ledger with its measured effect, accepted or not.

--offline swaps the LLM for a built-in catalogue of physically-motivated candidates, so the acceptance machinery — which is where the discipline lives — runs without credentials.

The model is not the classifier here, and that is on evidence: ConnectomeBench measured zero-shot frontier models at 61–63 % on merge identification, below a fine-tuned ResNet-50. Its job is reading a failure description and writing a candidate explanation as code. Whether that explanation is right is decided by measurement.

The agentic tracer: tracing whole neurons from the soma

exaproof view N006 --subject 653158 --from-scratch            # the soma's fragment alone; every join is made on the list
exaproof agentic bench --out DIR --arms person,rules+desk+person --reach all --truth-reach-um 250 \
    --neurons "653158:N006,N007"                              # trace from the soma under each tier stack, scored against gold
exaproof agentic fit-desk --corpus DIR                         # the desk model, one per brain with that brain held out
exaproof agentic report DIR                                    # the arm x brain table
exaproof agentic replay DIR/person/653158_N006.json --page demo.html   # a 3D replay (also served at /demo)
exaproof shots --url http://127.0.0.1:9000 --out /tmp/shots   # the viewer at four screen sizes, layout faults measured

exaproof.agentic works the viewer's own list through the same answer methods a person's keys call, deciding each question with a stack of tiers, cheapest first, each deciding only what it is sure of: named rules; the desk model (three calibrated gradient-boosted heads — P(nothing to join here), P(this candidate), P(this hop) — fitted on gold-answered questions from other brains); the ROI tier (refresh_alternatives: the multiscale ms_100 field rerank and the playbook's "nothing here"); Claude (claude -p or the Anthropic SDK, batched, the dossier in words plus the desk model's readings, escalating when unsure); and finally the person (live) or the gold oracle at the harness's prices (benchmark). When the list runs dry, reach sources propose what the list cannot: attachments from outside fragments (the session's inward pass, every place offered), long questions at dead ends (250 µm, a 90° cone about the end's heading, since §720) and long attachments; --walk adds the ROI walker on the dying frontier. Every step records the dossier, the verdict with its tier, rule, confidence and rationale, the gold's answer and the standing against the gold, which is also the replay.

Live, the Agent tab runs the same stack beside the person (/api/agentic): it never takes the decision the person has open, leaves what it is not sure of on the list marked agent left this for you with its reason, logs every decision as an agentic_decision row, and waits for the person's answers before carrying on. Use view --log-dir for any test or demo viewer, so its rows stay out of the people's decision corpus.

Measured (registered, from the soma, the gold as the person, 133 neurons of four brains no desk model saw): closing at the desk model's 0.90 bar takes 97 % of the person's minutes and a median 0.146 of each arbor (§712); holding those questions for last instead keeps the arbor (Δfinal 0.000), leaves cov@60 where it was (+0.0005), and where it reaches the person-alone coverage (109 of 133) gets there in a median 0.77 of the minutes (§715) — and a person who loads a neuron from the soma gets that order on the list itself (held questions at the back, badged; the list refills from the reach sources when it runs dry; agentic bench --person-desk traces on that desk); the walker on stalled traces reaches nothing new (§713). So every stack the Agent tab offers holds — the desk model (deskhold), the segmenter's checklist (roihold) and Claude (llmhold) — and none runs the rules tier, which made all 7 agent wrong joins in §715 from 40 decisions in 190 neurons. rules+desk+roi+gold stays on the tab, labelled, as the closing comparison. The live desk model is <data>/agentic/models/desk_e685.pkl (desk_model.DEPLOYED), the set §712 and §715 measured.

The scoreboard: auto@soma, the metric of record. exaproof agentic score traces every gold neuron from its soma with nobody answering (default arm desk+auto: the desk tier joins at P ≥ 0.95, auto leaves everything else; desk models held out of each brain; the bench's flags), then scores each trace (agentic.scoreboard). It reports the share of the gold arbor in the final soma-connected reconstruction, with its precision (wrong joins, cable off the gold). It splits the whole arbor into joined / reachable but not joined / beyond 250 µm gaps / unsegmented, by compartment (the gold's node types), by distance from the soma, and by CCF region where the brain has a registration (only 653158's is public). The headline is the worst brain's median. Each run writes scoreboard/<commit>.json and appends a line to scoreboard/history.jsonl with the change since the last scored commit, which is the Measure step of the loop in AGENTS.md §1. Run it from a worktree at origin/main, so the commit it names is on main, and commit what it writes. --no-run --out DIR re-scores traces already on disk.

The knowledge base: priors, heuristics, context, the wiki, the literature

What should inform a tracing decision is kept in one PostgreSQL database with pgvector (exaproof, on this machine's postgresql@16-main; spec docs/KNOWLEDGE.md). It holds:

  • entries: priors, heuristics, contextual heuristics, context, prompts and cases;
  • wiki pages;
  • papers;
  • the cell-type databases and atlases;
  • the tool registry: every model, scanner, tracer and matcher a tracer can call at a decision, with where it is called, what it costs and what it was measured to be worth (banked where it was found not to help), scoped to the assembly situations it serves.

Each is an item with a stable id, a scope (brains, CCF regions, compartments, cell types, decision kinds), a source, evidence and a status (proposed → measured → adopted, or banked with the reason). Every save is a revision.

On the site, /kb lists and searches everything, and /wiki, /lit, /types and /tools narrow it. /kb/<id> shows an item with its links both ways, its revisions with diffs, and its comments. Anyone signed in, except a share-link guest, edits there; an admin, or an attached measurement, adopts.

exaproof kb init      # tables and the vector extension, where missing (idempotent)
exaproof kb seed      # 150 verified papers, 8 sources, the LLM prompt, CLOSED.md banked, census priors,
                      # the named rules, a page per brain and region, the tool registry; numbers computed
                      # from the censuses, never typed
exaproof kb seed --only tool   # one group (here the tool registry), without the minutes-long region census
exaproof kb export    # the git snapshot, knowledge/*.yaml (current revisions); `kb import` rebuilds from it
exaproof kb ls --kind prior -q gap ; exaproof kb show prior-arbor-split-from-soma
exaproof kb records-bench <data>/agentic/e688_H   # every agent decision of a bench run, joined to the gold
exaproof kb records-desk                          # every answer a person gave on the desk, with their reasons
exaproof kb records                               # what the reasoning records hold (also /kb/records on the site)

The database is dumped hourly by Reprise and nightly by scratchpad/backup_s3.sh (full history; what, where and how to restore: docs/BACKUPS.md (repository file)). Tests use a throwaway schema (tests/test_knowledge.py, marker postgres, skipped where the database does not answer).

Tracing from raw imagery

exaproof trace N001 --subject 653158 --seconds 120

The "from scratch, no fragments" experiment, scoped honestly: whole-arbor from-scratch tracing on CPU will not beat the published fragments, and the design says so up front. What the same machinery is good for is local — a minimal-path bottleneck across a 25 µm gap is among the strongest evidence available to the split classifier, and 10-54 % of gold cable has no fragment (§193, per brain; this line said ~13 % from a single early brain) within tolerance at all, which no amount of reconnection can reach. Results report coverage and precision, because a tracer that wanders scores well on one and badly on the other.

The benchmark that decides the goal

exaproof train --group          # endpoint-level head, beside the pairwise top-1 it must beat

The one number that governs whether unattended tracing is possible: recall at the precision a false merge tolerates. A merge fuses two arbors, so a ~400-join arbor needs precision ~0.998; today's recall there is under 1 %. Any new feature, model, or heuristic should be judged on this column first — batch precision and recall can both improve while it does not. RESULTS.md §15 records the endpoint-head experiment that produced it, and why group context did not help.

exaproof encode                 # learned encoder over raw gap voxels (matched control)
exaproof ceiling                # how often do two experts agree? the benchmark's own limit
exaproof repair                 # can a merge be found again after it is made?
exaproof triage                 # can a human find committed merges by reviewing the doubtful end?
exaproof reach                  # the hard ceiling on connected growth: soma-reachable cable vs jump range
exaproof audit                  # are the mined negatives actually negative?

Before trusting any ceiling this system reports, check the labels it was measured against. exaproof audit finds every negative pairing two fragments from the same arbor and asks whether they are adjacent on the tracing with nothing between them. On 653158 the answer is zero out of 5,938 — the negatives are distant branches of one cell, median separation ~230 µm. RESULTS.md §16 records it, including the shortcut bug that first made the labels look broken.

See DATA.md for every data source with provenance and regeneration commands. Where each line of work stands is exaproof goals, not a document; ROADMAP.md holds only the current session's handoff — what is running and what to do with each verdict.

Extending it

Adding a feature is one decorated function:

from exaproof.features import CandidateBatch, feature

@feature("my_signal", group="geometry", cost="cheap",
         outputs=("my_a", "my_b"), description="what it is for")
def my_signal(b: CandidateBatch):
    return np.stack([..., ...], axis=1)

It is picked up automatically by the model, the feature cache, exaproof features list, and the learning loop. The cost class is load-bearing, not documentation: cheap and prior features are computed for every candidate, image and expensive ones only where the agent decides the evidence is worth buying, and an uncomputed feature is NaN — missing, not zero — which the gradient-boosted default handles natively.

Other extension points, all behind the same interfaces: models/base.py (Estimator) for a different classifier, agent/merge.py for merge detectors, candidates/conflict.py for the assignment objective, learn/propose.py (Proposer) for a different proposal source.

Every command

Generated from exaproof --help and each command's own --help on 2026-09-21: the command, the first sentence of its help, and where it is explained. Seventy commands — 47 at the top level and 23 under eight sub-apps. Flags are not repeated here: exaproof <command> --help is their reference, and the flag text there carries the measurement each default rests on, so it cannot drift from the code.

Top level, in the order --help lists them:

command what it does explained in
train Fit the decision model and certify its operating point. training; MODELS.md §2
proofread Reconstruct one neuron and emit a ranked worklist of what it could not decide. proofreading
eval Run the agent on held-out gold neurons and report the full panel. evaluating
smoke The release gate: drive every page, tab, tool and subject switch in a browser and list the blockers. the viewer; viz/smoke.py
simulate Work through decisions in a real browser the way a person does, and report what the workflow costs. GOAL.md (repository file) §5 G; viz/simulate.py
field-cache Precompute the model's probability field along one neuron's cable, for the viewer to serve instantly. VIEWER.md §4, paint: field
precompute Build the viewer's reconstructions ahead of time, so opening any of these neurons is instant. the viewer
view Launch the Neuroglancer proofreading viewer for one neuron. the viewer; VIEWER.md
shots Screenshots of the viewer at four screen sizes, with the layout faults measured in each. the agentic tracer; viz/shots.py
agentic bench / report / fit-desk / replay Trace neurons from the soma under tier stacks and score them; the table; the desk model; 3D replays. the agentic tracer
audit Check the mined negatives against the tracing they came from. the benchmark
review-report Turn a proofreader decisions.jsonl into annotator metrics — the real-human measurement. VALIDATION.md §5
asker Train the does-this-end-continue head that ranks the session's questions. models/continuation.py; session --asker
reach The hard ceiling on connected growth: soma-reachable cable vs allowed jump distance. the benchmark; FINDINGS.md §1
bridge Can the imagery bridge an omission gap the fragments cannot? GOAL.md (repository file) §5 C; bench/bridge.py
bridge-corpus Slab corpus for the gap verifier: per mined gap, the nominated decoys plus the true re-entry pair, labelled. MODELS.md §6; bench/bridge.py
bridge-train Fine-tune the certified encoder into a gap verifier on the mined-pair corpus. MODELS.md §6
roi-corpus ROI training cubes for the local segmentation model. MODELS.md §1; FINDINGS.md §2
synth-corpus Render CORAL geometry through this instrument's measured optics into a training corpus. GOAL.md (repository file) §5 E; synth/
roi-train Train the ROI U-Net and measure what matters: gap connectivity vs decoy connectivity. MODELS.md §1; FINDINGS.md §2
restore-train Train the restoration U-Net (E052 Stage 1) on bright corpus cubes. E052_restoration.md (repository file)
adaptive-eval Run the adaptive decision loop over a holdout and score it. FINDINGS.md §6; adaptive/
merge-weights Fit the merge-window score's weights on contact-centred pairs, for the viewer's scanner. MODELS.md §4; FINDINGS.md §5
look-session Build a blind session for a looking agent: shuffled sites in a manifest, answers sealed. FINDINGS.md §6; adaptive/looks.py
look Render views of sites in a blind session, and record that they were looked at. FINDINGS.md §6
look-verdict Record a looker's verdict and its reasoning. FINDINGS.md §6
look-score Unseal a blind session: the looker against the model on the same sites, and the traces. FINDINGS.md §6
adaptive-extend Ask stored traces a new question: rebuild their sites and append probes, verifying the match. adaptive/harness.py
scene-calibrate Measure a brain's own thresholds for the viewer's image line. the viewer; adaptive/scene.py
playbook-fit Fit the stall-end playbook's per-step calibrators and action thresholds, and save them. MODELS.md §7; the viewer
adaptive-heuristics Learn readable rules from recorded traces, and say what readability costs. FINDINGS.md §6; view --heuristics
roi-panel Re-panel a trained ROI model on a chosen threshold grid, without retraining it. MODELS.md §1; roi/eval.py
roi-compartments Add compartment labels (unknown, axon, dendrite) to an existing ROI corpus. roi/corpus.py
roi-seeds Add the seed channel to an existing ROI corpus: the entry object rasterised. roi/corpus.py
roi-objlabels Object-conditioned labels over an existing corpus's saved centres. roi/corpus.py
roi-mine Cubes at the training gaps a model fails to connect, as a new mining round. roi/corpus.py
triage If merges cannot be prevented, can a human find them cheaply? the benchmark
encode Train the learned encoder over raw gap voxels, against geometry as a matched control. the benchmark; MODELS.md §6
encode-pooled Train one encoder on every brain's slabs and save a deployable rescorer per brain. MODELS.md §6
endpoint-bench Endpoint-level recall at tight precision, cheap vs fused. the benchmark
disputes One evidence dossier per join the annotators disagreed about. GOAL.md (repository file) §5 H; VIEWER.md §6
ceiling How often do two experts agree on the same join? The benchmark's own ceiling. the benchmark
repair Fuse two gold arbors with a real merge and try to find the bad edge again. the benchmark
session Simulate a proofreading session and report the goal's own metric. evaluating; METHOD.md
trace Trace from the raw imagery alone, with no fragments, and score it against gold. tracing
grow Generate cable where the segmentation produced none, from stalled fragment ends. FINDINGS.md §9; GOAL.md (repository file) §5 B
skip-certify Price a barren-question skip threshold for one brain, on both metrics. evaluating; GOAL.md (repository file) §5 A
goals Where every goal stands, checked against the records it cites. AGENTS.md (repository file) §2, §5
record Read one result out of the record without opening the record. AGENTS.md (repository file) §3, §5; docs/INDEX.md

| exaproof status | Read the leaderboard (--path alt file). Per-brain recall, TBS F1, benchmarks. | AGENTS.md (repository file) §2, §5; exaproof goals |

data — fetch and ingest exaSPIM data:

command what it does explained in
data status What is present locally, per subject. the data; DATA.md §2
data fetch Download the gold tracings and the automated fragment archive for a subject. the data; DATA.md §1
data ingest Flatten the fragment archive into the columnar store and build spatial indices. the data
data inventory What exists for every named brain — imagery, tracings, segmentation, somas — from the written indexes. DATA.md §2f
data index-s3 Write a depth-limited listing of a bucket to a file, so later searches are local. DATA.md §1; io/s3_index.py
data verify Check that gold tracings, fragments and imagery share one coordinate frame. the data

gt — gold-to-fragment alignment and the raw baseline:

command what it does explained in
gt align Project every gold arbor onto the fragment set and cache the result. the baseline
gt baseline The raw automated segmentation scored against gold — the number to beat. the baseline

docs — compaction docs, read-only audit / lint:

command what it does explained in
docs audit Audit docs (memory, KB, links); write TSV + audit_summary.md. --repo . --out docs/METHOD.md; AGENTS.md (repository file) §5
docs lint Report bare numbers in prose (SPEC_COMPACTION §6), never edits. Pass files... docs/METHOD.md §6

decisions — mine labelled reconnection decisions:

command what it does explained in
decisions mine Mine labelled reconnection decisions from gold tracings over the real fragments. training
decisions summary Describe a mined corpus. training
decisions augment Fold session ask logs into a decision corpus — the retrain-on-live-mistakes loop. gt/session_supervision.py

features — inspect the feature registry:

command what it does explained in
features add-image Add image evidence to an existing feature cache, for more neurons. training
features list Every registered feature, its cost class, and what it is for. extending

learn — learn from the system's own mistakes:

command what it does explained in
learn mistakes Cluster the system's own errors and describe what distinguishes each cluster. learning; FINDINGS.md §9d
learn improve Propose new features from the mistake report, measure them, keep what helps. learning; FINDINGS.md §9d

users — accounts for the proofreading viewer (view --auth):

command what it does explained in
users add Create a viewer account. the viewer; VIEWER.md §9
users list Every viewer account. VIEWER.md §9
users password Set or reset an account's password. VIEWER.md §9
users reset Issue a one-time link with which someone chooses a new password themselves. VIEWER.md §9
users remove Delete an account (its logged decisions keep the email they were recorded with). VIEWER.md §9

viz — serve-side pieces of the proofreading viewer:

command what it does explained in
viz fetch-ng Mirror the Neuroglancer client so view can serve it from its own origin. VIEWER.md; view --ng-base

bench — benchmark utilities and adversarial checks:

command what it does explained in
bench exploits Reproduce the three known metric exploits, so a harness change cannot reopen one. evaluating; bench/exploits.py
bench splits Show the neuron-level train/calib/dev/test split, and the brains available held out. evaluating; METHOD.md §2

Tools (PYTHONPATH=src; python tools/X.py --help): bench_freeze freezes a benchmark (--id, --items, --md5, --dry-run); brief writes a brief (new); ledger_split splits ledgers (results|rows); memory_lint lint (dir); viewer_gates deploy gates (--dry-run, --port).

Layout

One line per package, from each package's own docstring; ls src/exaproof is the check.

src/exaproof/
  cli.py          every command above; `exaproof --help` is its index
  config.py       paths and run configuration; `EXAPROOF_DATA` is the data root
  goals.py        `exaproof goals`: goals.yaml validated against the records it cites
  record.py       `exaproof record`: retrieval over RESULTS.md and VALIDATION.md without opening them
  io/             SWC, fragments, gold, OME-Zarr imagery (`EXAPROOF_VOLUME`), S3 indexes, the brain inventory, somas, MouseLight, CCF
    manifests/    the data owners' location lists, checked in as received (its README says what each maps)
  imagery_meta/   each volume's zarr metadata, kept here because the sources have lost theirs (its README)
  gt/             gold->fragment alignment; the decision miner; annotator agreement, disputes, gaps, session supervision
  metrics/        the panel (ERL, edge accuracy, merges at 0/10/50 nodes)
  features/       the registry: geometry, context, image, priors, slabs, rays, texture, tracts, calibre
  models/         estimator, calibration, conformal risk control, OOD; the choice, continuation, rescoring and encoder heads (docs/MODELS.md)
  candidates/     proposal generation, conflict graph, exact assignment
  agent/          journalled state, merge detection, the sequential and global loops, the result cache
  bench/          end-to-end evaluation, the simulated annotator and session, reach, repair, triage, bridge, exploits
  roi/            the local segmenter: corpus, training, tiled prediction, panels, flood tracer, growth, restoration, field cache
  synth/          rendered training cubes: known geometry through the instrument's measured optics and noise
  adaptive/       per-decision evidence loops, traces, rendered looks, the playbook, merge scan, scene line, branch check
  viz/            precomputed skeletons, Neuroglancer state, the proofreading server, accounts, the docs site, browser gates
  learn/          mistake mining, LLM proposals, sandbox, acceptance loop
  agentic/        the agentic tracer: dossiers, tiers (rules, desk model, ROI, Claude, person), reach and walker sources, live worker, bench, replays
  tracing/        minimal paths and the from-scratch tracer

Tests

PYTHONPATH=src pytest tests -m "not data and not browser and not heavy and not server and not node"   # ESSENTIAL suite (default, tests/essential_suite.toml; docs/TESTING.md); add --suite extended|all for the deferred rest
PYTHONPATH=src pytest tests                                                                          # everything: ~2 min, ~6 GB peak

Markers are registered in pyproject.toml with --strict-markers: data (reads real data under /home/ubuntu/data), torch, browser (playwright/Chromium), node, server (a live viewer on an ephemeral port), heavy (over 2 GB or 30 s alone), slow. The essential suite is what to run beside an experiment; the full suite is itself a heavy job on this machine (AGENTS.md section 6).

In a worktree, PYTHONPATH=src is not optional. The editable install points at the main checkout's src/, so a bare pytest in a worktree runs that worktree's tests against the main checkout's code: a change you just made is neither exercised nor able to fail the suite, and a green run means nothing about it. Found 2026-09-19, when a new server method passed review and then raised AttributeError under the test that was supposed to cover it.

Concentrated on the traps that would otherwise be invisible in the final numbers: the SWC OFFSET convention, point-to-segment distance (fragment nodes are ~10 µm apart, so nearest node is not nearest fragment), merge-threshold semantics, exactness of the assignment solver against brute force, the risk bound holding, and the sandbox refusing dangerous code.

Full measured results, including every bug that looked like a model failure and was not, are in RESULTS.md — reached with exaproof record <n>, not opened.

Running an experiment

Each rule below has cost a run; AGENTS.md (repository file) §6 has the incidents and docs/METHOD.md the rules for the measurement itself.

  • Work in a worktree under ~/worktrees/agentic_proofreading/, never on main. A worktree has no .venv, so every run and every test there is PYTHONPATH=src /home/ubuntu/git/agentic_proofreading/.venv/bin/python -m exaproof.cli … (or -m pytest); a bare exaproof runs the main checkout's source. A job launched from a worktree writes its outputs there.
  • Set EXAPROOF_VOLUME=denoised. The raw fused.zarr is gone upstream for several brains and reads as zeros rather than failing (DATA.md §2b). io/imagery.py reads the variable; the driver below exports it.
  • Gate on memory, not on the GPU. RAM is the binding resource and an OOM kill leaves no traceback. scratchpad/mem_gate.sh (sourced) provides mem_ok [GB], which admits when at least GB is available and memory pressure was low over the last minute — or over the last ten seconds with swap-in quiet — and heavy_running, which counts running jobs matching its HEAVY_SCRIPTS list: add a heavy script there when you write it. scratchpad/slots.sh is the older slot gate for parallel roi-train.
  • Launch through tools/launch_run.sh (every run is capped, logged and tracked in OpenResearch, orx runs <project>); a queue is one such run whose command is the driver, detached: tools/launch_run.sh run --prereg ... --tag ... --minutes N --mem-gb N -- 'bash scratchpad/run_queued.sh <exp> <script.py> [--after-pid PID] [--marker FILE] [--gb N] [--max-heavy N]'. It waits for the PID or the marker, then for mem_ok and fewer than N heavy jobs, runs the script with its output in scratchpad/<exp>_read.out, and records every phase in scratchpad/<exp>_queue.state.json. Its header is the full contract; its own PID is what the next queue passes to --after-pid.
  • See what is running with pgrep -af 'bash scratchpad/' and .venv/bin/python scratchpad/queue_state.py, which reads those state files. Never wait on pgrep -f PATTERN in a loop: it matches itself.
  • A scratchpad script that needs a sibling loads it with import _load; m = _load.load("e618_walk_close") (scratchpad/_load.py): once per process, with this checkout's src on the path.
  • When it lands, follow AGENTS.md §5: a RESULTS.md section, goals.yaml in the same commit, and the outputs the section rests on committed by name.

Limits, stated rather than buried

  • One GPU. The development host has a single NVIDIA L4 (23 GB) and 30 GB of RAM; two segmenter trainings at once saturate it.
  • 10–54 % of gold cable has no fragment within tolerance, depending on the brain (10 % on 653158, 54 % on 648434). That is a ceiling on any fragment-based method, and it is why the local segmenter and tracing/ exist.
  • Coordinates are 3× expanded. exaSPIM images expanded tissue, so every micrometre in this repository is an expanded micrometre. External data in tissue units must be scaled first.
  • Ground truth is uncertain to ~9 % on axons. Two experts tracing the same exaSPIM axon differ by 8.9 % of cable on average, 16.6 % at worst. No improvement smaller than that against a single annotator's tracing is meaningful, and the harness reports cluster-bootstrap intervals with the neuron as the resampling unit for exactly this reason.
  • PTE is uncalibrated. Reported as a band. Calibrating it needs instrumented proofreading sessions, which the viewer's decision log is designed to produce.
  • Three gold consensus files in 653158 are in the wrong coordinate frame (N016, N036, N049 — offsets tens of millimetres outside the volume). The loader detects and reports this and falls back to the per-annotator file; it is a data issue worth reporting upstream.