September 2026 submission — zarr-pyramid-audit

The 60-second case

A valid Zarr header can still be a lie. The confirmed PHerc0814 pyramid opens normally, but all six levels have no chunks and silently read as zeros. That failure can waste training runs or invalidate downstream segmentation and ink work without throwing an error.

Find

957 S3 roots audited in about five minutes; one live structural corruption case independently re-confirmed on September 30.

Prove

A direct zarr-python read returned all zeros without error; a populated control returned 64,498 nonzero voxels.

Prevent

zpa-gate turns the finding into a publish-time CI failure: defective root exit 1, populated sibling exit 0, nonexistent path exit 1.

Evidence is committed and reproducible; the page links every headline claim to a repo artifact below.

Post-deadline update · 2026-10-01

Added after the September 30 deadline. Everything else on this page is the submission as frozen on 2026-09-30, apart from the dated corrections marked in place. This panel lists what was added the next day; each item links to the committed evidence.

tifxyz surfaces are now audited. New zpa-tifxyz checks Volume Cartographer surface patches (meta.json + x/y/z.tif) with strict range reads. All 1,539 surfaces in the S3 bucket are integrity PASS; the content tier ran on 1,391 (148 over the 32 MiB cap are reported as a coverage gap). It found 161 surfaces with points at negative coordinates, outside any CT volume, and 30 whose meta.json bbox disagrees with the stored coordinates, all low review findings (artifacts/2026-10-01-s3-tifxyz/). Triangular meshes are not audited here.
Compared with the tools people already use. On the live PHerc0814 defect the OME-NGFF validator (ome-zarr-models 1.7) accepts the pyramid and zarr-python 3.1.6 reads a level-5 window as 249,856 voxels, all zero, without an error; the gate rejects it. On the fixture corpus, defects flagged: zarr-python 8 / 26, the validator 14 / 26, the ZPA header audit 25 / 26, with the chunk probe 26 / 26. The corpus was built around ZPA's failure classes, so it is read per defect class, not as a score (artifacts/2026-10-01-baseline-comparison/).
OME-NGFF conformance re-audit. All 957 S3 roots re-audited with the new spec-conformance checks: 0 conformance findings, every root declares OME-NGFF 0.4, and the 2026-09-29 findings reproduce row for row (956 clean, 1 defective) (artifacts/2026-10-01-s3-conformance/).
ScrolIQ now fails closed. Since scrollq#55, scrollq-health takes integrity from this repo's versioned report: an unreadable level or an absent root is DO NOT TRAIN instead of falling through to the quality score. Live negative control: an absent root reads DO NOT TRAIN (ScrolIQ evidence).
A versioned contract. Audit and tifxyz reports share one JSON contract, schema_version 1.2.0, with published schemas, a recommended verdict mapping (FAIL / UNKNOWN → DO NOT TRAIN, WARN → CAUTION), and committed golden outputs for 67 fixture cases (docs/INTEGRATION.md, CHANGELOG.md).

Reviewer map, criterion by criterion: docs/SUBMISSION.md · live dashboard

(1) Which scroll data did you work on?

The entire open-data surface, on both public stores:

S3 bucket (s3://vesuvius-challenge-open-data): 957 Zarr pyramid roots discovered by full bucket crawl, spanning PHerc0814, PHercParis4, PHerc186 and community uploads.
dl.ash2txt.org: the 241-root set behind villa issues #1755–#1760, re-audited for a 20-day regression check.
Deep dives: the defective PHerc0814 surface volume (villa #1892), the known silent-zero volumes, and 64 same-named volumes present in both stores.

(2) How does it increase the probability of reading the scrolls?

Ink detection, segmentation, registration and interactive viewing commonly consume reduced pyramid levels rather than repeatedly reading the full-resolution volume. A level whose header is valid but whose chunks are missing can return plausible all-zero voxels with no error — silent corruption that burns compute and can mislead downstream analysis. This contribution does not merely report that risk; it turns the check into a reproducible pre-publication gate:

Finds live corruption before anyone trains on it. The audit independently confirms the header-only PHerc0814 surface volume (villa #1892, all 6 levels header-only, reads silently return zeros) is still live in the bucket — and names its two populated sibling renders as safe alternatives. The new chunk-content probe then caught a second live case the header audit cannot see: 7 of 8 levels of other/dev/meshes/20231022170900-ome.zarr decode to all zeros while L0 holds real mesh data.
Stops new corruption at the door. zpa-gate is a publish-time metadata gate that fails closed on structural defects (high severity by default), with GitHub Actions annotations for CI.
Proves defects don't self-heal. The 20-day regression found zero of 18 defective pyramids repaired — so catching defects before publish is the only point of leverage, which is exactly what the gate provides.

The fix ships with the repo. zpa-gate audits roots before they publish and fails closed — the gate can block this defect class before publication when used. Proven end-to-end 2026-10-01 (artifacts/2026-10-01-gate-proof/): rejects the defective PHerc0814 root (exit 1), passes its populated sibling (exit 0), and rejects a path that does not exist (exit 1).
Correction (2026-10-01): the 2026-09-30 proof's "clean sibling" path does not exist in the bucket; the gate passed it because a missing root was only low severity. The gate now fails closed on missing roots, and the proof was re-run with a real sibling (erratum in artifacts/2026-09-30-gate-proof/).

(3) What does it enable that was not possible before?

Full-bucket S3 coverage added to this auditor. The upstream tool only covered dl.ash2txt.org; this fork adds anonymous S3 discovery/auditing and has now run across 957/957 discovered roots header-only in ~5 minutes. This is the evidence base for the live PHerc0814 finding.
Zarr v3 support. v3 bare arrays were misreported as NOT_A_ZARR_GROUP; now correctly classified as BARE_ARRAY with shape/chunks/dtype. All 70 v3 pyramids audited clean.
Chunk-content probing (zpa-scan-chunks): closes the next silent-corruption class — chunks that exist but decode to all fill_value — via sampled decode (v2/v3, raw/Blosc/sharded, and now volcomp: all 64 scroll volumes on dl.ash2txt.org decoded with a vendored libvolcomp, 384 levels probed, 0 present-but-empty).
Mirror-fidelity evidence. The 64 same-named volumes are format migrations, not copies (S3: v2/128³/raw; dl: v3/1024³-sharded/volcomp, quantization noise only) — reproducibility must pin the store, not just the name.
A public, living data-health dashboard generated from audit artifacts, so anyone can see bucket health at a glance.

Distinct from the August 2026 awarded scroll-data-audit work. The official winners page describes that project as defect-hunting across the open-data catalog and volume-cartographer with catalog metadata checks and VC3D fixes. This submission complements it at a different layer: Zarr pyramid structure, chunk presence/content, mirror fidelity, and a CI gate that can reject a bad published root before downstream use.

Official August 2026 winners →

Progress Prize rubric fit

The current Progress Prize asks for a specific problem and demonstrated solution, comprehensive documentation, and modular integration using standard community formats. The submission maps directly to those criteria:

Problem + solution	Silent structural/content corruption in multiscale Zarrs; demonstrated on live data, with `zpa-audit`, `zpa-scan-chunks`, and fail-closed `zpa-gate`.
Significant advantage	Header-only corpus audit is fast enough for continuous use; sampled content probing closes the “chunks exist but are empty” gap; the gate converts diagnosis into prevention.
Documentation	README usage, machine-readable artifacts, reproducibility manifests, known-defect records, and a public dashboard generated from committed evidence.
Technical integration	Accepts Zarr v2/v3 over HTTP/S3, emits stable JSONL/CSV/summary outputs, and exposes console commands suitable for CI or orchestration with the wider Vesuvius toolchain.

Criteria source: Vesuvius Challenge open prizes (September 30, 2026 snapshot).

Update (2026-10-01): technical integration now also covers tifxyz surfaces and a versioned JSON report contract; see the post-deadline update.

(4) What evidence have you provided?

claim	evidence (in repo)
957 S3 roots, 956 clean, 1 defective	`artifacts/2026-09-29-s3/` — full audit JSONL + summary
Defective pyramid reads silently return zeros	`artifacts/2026-09-29-s3/SILENT_ZEROS.md` — zarr-python proof vs populated control (64,498 nonzero voxels). Re-confirmed still live 2026-09-30 ~22:25 UTC (`artifacts/2026-09-30-s3-reverify/`): 6 high-severity `LEVEL_NO_CHUNKS` findings, identical signature — unrepaired for the full September window.
Defect isolated to one pipeline run (1 of 19 copies)	`SILENT_ZEROS.md` — 18 clean siblings named
Zero defects repaired in 20 days	`artifacts/2026-09-29-dl-regression/` — strict diff: 0 fixed, 0 new
Mirrors are migrations, not copies	`artifacts/2026-09-29-s3/MIRROR_FIDELITY.md` — byte-level comparison
Present-but-empty chunks: found live	`artifacts/2026-09-30-dl-v2-probe/` — 128 roots / 748 levels content-probed (1,795 chunks with real data): 7 levels of `other/dev/meshes/20231022170900-ome.zarr` decode to all zeros while L0 holds mesh content (verified: 12/12 L1 chunks empty, downscaled counterpart of populated L0 region empty, byte-identical zero-blobs). Not observed in S3 (788/788 populated) or the 64 volcomp scroll volumes (548 decoded, 0 present-but-empty).
Gate rejects the defective root, passes clean ones	`bin/gate.py` — demonstrated in README and in the end-to-end proof below

Novelty is claimed only for the tooling and the integration; the underlying defects (#1892, #1755–#1760) are credited to their reporters and independently confirmed here.

Reproduce the headline result

The end-to-end proof uses the same public S3 bucket: the defective root and a populated sibling render of the same segment. After installing the repo, these two commands should disagree by design:

# defect: expected exit 1 zpa-gate --base s3://vesuvius-challenge-open-data \ --root "PHerc0814/segments/20260226123353-auto_grown_20260226123353106/surface-volumes/1.129um-0.22m-59keV-volume-20260521123630-L1.zarr" \ --fail-on high # populated sibling: expected exit 0 zpa-gate --base s3://vesuvius-challenge-open-data \ --root "PHerc0814/segments/20260226123353-auto_grown_20260226123353106/surface-volumes/2.399um-0.22m-78keV-volume-20260309142202.zarr" \ --fail-on high

Correction (2026-10-01): this block originally named …-20260521123630.zarr as the clean sibling. That path does not exist, and the gate now rejects it (exit 1, GATE_ROOT_ABSENT) instead of passing it.

Committed logs and README → · original 2026-09-30 proof + erratum → · same-day re-verification →

Why this matters for the 2027 Grand Prize pipeline

This project is infrastructure, not an unrolling claim. The 2027 Grand Prize requires a reproducible pipeline that can segment/unroll an entire scroll and then render legible ink. Those stages depend on trustworthy volumes, surfaces and multiscale reads. zarr-pyramid-audit supplies a cheap integrity boundary before expensive segmentation or ink inference and a machine-readable trail for reproducing exactly which store/root was used.

That trail is consumed downstream: ScrolIQ's scroliq-provenance gate requires a zarr_audit record for the exact eligible CT volume (tool name, audit-manifest SHA-256, and a root naming that volume), and its Grand Prize probe protocol makes this audit Stage A.

2027 Grand Prize requirements → · how this fits with ScrolIQ →

Team

Solo contributor (Sven) + Muse (AI assistant). Upstream tool by sgsllc-jr, MIT-licensed; all upstream credit retained in the fork.

How this fits with ScrolIQ

Two repos, one question: should anyone spend GPU or expert time on this volume? This submission answers whether the data is what it claims to be; ScrolIQ answers what is worth doing with it.

zarr-pyramid-audit · this submission

Don't train on lies. Integrity: header-only pyramid audit, sampled chunk-content probes, and a fail-closed publish gate. A high-severity finding means do not train, do not publish. Says nothing about scan quality.

ScrolIQ · companion

Find the bottleneck. A diagnostic layer for the Challenge's 2026 open problems: a 0–100 scan-health triage score and “label next” coverage join, plus a diagnostic passport, spatial scan map, mesh / winding / ink audits, and 2027 Grand Prize recto-coverage and provenance gates. Its score is not readability or prize readiness.

repo · live survey · its September writeup

One verdict per volume. scrollq-health runs this audit alongside ScrolIQ's score: a high-severity integrity finding is DO NOT TRAIN; a medium finding, or a pass with quality unscorable or below 40, is CAUTION; otherwise TRAIN. Verified on live data: DO NOT TRAIN on the defective PHerc0814 pyramid (its quality is unscorable, so the verdict comes from this audit alone), TRAIN on a healthy PHerc0813 dl volume, CAUTION on the v2 dev mesh (ScrolIQ evidence; rules from its health.py, checked 2026-09-30).
Update (2026-10-01): the rules above are as of 2026-09-30. Since scrollq#55, missing integrity evidence (an unreadable level, an absent root) is also DO NOT TRAIN; the three verdicts above are unchanged (re-run).
Shared foundation. ScrolIQ imports this package's HTTP/S3 store, header parser, audit, and vendored libvolcomp decoder, and declares zarr-pyramid-audit as a dependency. The dependency runs one way: this repo never imports ScrolIQ.
Shared discovery data. ScrolIQ's label-coverage join reads the discover_zarr.roots.jsonl that zpa-discover produces for the S3 bucket.

ScrolIQ is spelled with a capital I (as in Mesh IQ) and was previously ScrollQ; its package and scrollq-* commands keep the old name for compatibility. · Full breakdown with the verdict table →

Submission status

Frozen public snapshot. Evidence and same-day re-verification are frozen on September 30, 2026, the day of the official Progress Prize deadline (11:59 PM Pacific); submission itself is completed through the Challenge's official form. The evidence on this page stays frozen at that date — the live dashboard carries the current evidence. Additions made after the deadline are confined to dated corrections and the clearly marked 2026-10-01 update.