Measurement-pipeline audit — 2026-10-06

Follow-ups (all closed unless noted): 1 (streaming replay and normalize, both chunk-by-chunk with measured RSS), 2 (used-dir refusal, extended to init), 3 (schema gate: check, normalize, replay, reconstruct), 4 (reconstruct manifest gate, gap classification and counting, sequence-break counting), 5 (failure policy documented and kept current; behavioral unification closed via replay_normalized — null rows replay as skips, raw ≡ normalized proven), 6 (--snapshot refuses foreign files with venue guidance; no memory-mapped Bybit format), 7 (normalize idempotency documented), 8 (normalized-replay engine with three tests), 10 (live-book error counting, then applied-only semantics), 11 (all ten CLI formats pinned with golden tests), 12 (fail-closed sides, canonical enforcement at validation). Post-audit hardening with README rows: decimal range on all tables, corrupt prints and sizes never fill, BookTop gap-breaking with snapshot healing, verify path validation, non-UTF8 skips, fee-error naming, gap-number recovery, seeded fuzzing, partition-by-instrument, index-count verification, undecodable-abort coverage, trade price range. Still open: 9 (documented). The report below is otherwise unchanged evidence of what the audit found.

The completed half (astra-types, astra-book, astra-record, astra-normalize, astra-replay, ops/soak.sh) run to the throttle: full suite, lints, panic-surface scan, and a line-by-line review of every production path. Three bugs fixed same-day (below). Everything else is reported as found — including the things that are fine, so the next reader does not re-audit them.

Method and baseline

cargo fmt --all -- --check: clean.
cargo clippy --workspace --all-targets -- -D warnings: zero warnings (one caught mid-audit in the auditor's own fix — dead backoff doubling — and removed, not silenced).
cargo test --workspace: 145 passed, 0 failed, 0 ignored (17 book + 16 normalize + 86 record + 10 replay + 15 types + 1 capture saturation counted in record). Four tests are new in this audit.
Panic-surface scan: every .unwrap()/.expect() in production code inspected. All remaining ones are in test modules except one provably-safe site (store.rs, chunk framing — the slice length is guarded two lines above; flagged as hygiene, not a bug).
Not covered by this audit: live venue traffic (futures endpoints are geo-blocked from this machine, same as ever), the 72h soak, a second machine, and wall-clock performance beyond the recorded live numbers.

Fixed in this audit

1. Continuity checks panicked on u64::MAX update IDs. capture.rs sequence tracker, check.rs offline audit, and astra-book reconstructor all computed previous + 1 on venue-supplied u64 IDs. In debug builds (every cargo test / cargo run invocation) that expression panics on overflow — proven with a standalone reproduction (exit 101, attempt to add with overflow). Trigger: two frames where the first carries u/U near u64::MAX. Real venues will never send that; a hostile local feed (--url pointed at an attack server) could crash the recorder, the audit, and reconstruction alike — exactly the class this project claims to reject ("all rejected, none panicked"). All three sites now use checked_add; at saturation no forward jump is representable, so no gap is reported and nothing panics. Regression tests at all three sites.

2. Reconnect backoff accumulated to 30 seconds across long soaks. backoff doubled per reconnect with no reset, saturating at 30s. Over 72h with occasional blips, every late-soak reconnect would wait up to 30s — manufacturing holes. Worse, the doubling was load-bearing nothing: a failed reconnect attempt ends the capture outright, so there was no retry stream to back off across. Replaced with a fixed 250ms wait (lossless-first: a blip costs a quarter-second hole, always), dead RECONNECT_MAX_BACKOFF removed. Existing reconnect tests green.

3. check trusted the manifest; replay and normalize did not. replay and normalize refuse captures whose frames_written disagrees with the chunks on disk, but check — the tool the soak is judged with — never compared them. A doctored manifest (or a capture killed before finish() rewrote it) audited healthy. check now reports manifest_mismatch (claimed vs actual venue frames, unhealthy when set) and prints stop_reason (previously invisible — a -9'd capture looked identical to a clean one). CLI prints both lines. Test helper corrected to write venue-only frame counts, matching production.

Open findings (ranked by importance)

Soak-scale memory: only check streams. check reads chunk by chunk; replay and normalize read_all the entire capture into memory. A 4-stream 72h soak holds millions of frames with multi-KB payloads — single-digit GB RAM on the VPS that runs it. check will be fine there; normalize/replay of a full soak on a 4 GB box may not be. Recommendation: stream replay/normalize per chunk before the soak, or size the judging box accordingly. Not fixed: needs design, not a patch.
Reopen semantics are half-true. ChunkWriter continues the chunk sequence in an existing dir, but Session::open rewrites the manifest (new capture id, zeroed counts) and restarts seq at 0. Re-capturing into the same dir produces seq collisions across old/new chunks (detectable via seq_breaks, but still). Recommendation: refuse to capture into a non-empty dir, or implement real resume. Not changed: all live paths use fresh dirs; changing CLI behavior unilaterally risks unknown workflows.
No schema-version gate anywhere. Capture writes schema_version; no tool reads it back. A format change would be silently misread. Recommendation: refuse mismatched versions in read_manifest paths.
reconstruct is the least strict reader. It never opens the manifest (no frame-count check, no stop-reason visibility), never checks seq continuity, and skips SYNTHETIC markers without counting connection gaps or breaking the book on them — it relies on the reconstructor's own update-ID discontinuity to break, which works for sequence jumps but under-reports connection gaps in the summary. Recommendation: share check's prelude (manifest + seq walk) before reconstructing.
Fail-loud inconsistency on bad data. An undecodable gap marker aborts normalize/replay entirely, is counted by check, and is skipped-then-possibly-broken-on by reconstruct; an invalid diff aborts compare but is counted by reconstruct. Each choice is defensible alone; together they mean the "same" corruption produces three different outcomes depending on the subcommand. Recommendation: one documented policy table.
--snapshot bootstrap is Binance-only. Both reconstruct and verify hardcode Venue::Binance + BookSnapshot parsing. Harmless today (Bybit bootstraps in-band) and documented in the README, but a confusing error for anyone holding a Bybit snapshot file.
Normalize is not idempotent into one output dir. Re-running into the same directory appends part-00001… next to part-00000…, duplicating rows. The cross-machine procedure already mandates fresh dirs, so this bites only the careless — document-it territory.
Null-row vs skip divergence. An unparseable venue frame becomes a null-field row in Parquet but counts as skipped_unparseable (no event) in replay. A replay-over-normalized-data would therefore see a null trade row where a raw replay sees nothing. Small, real, worth one test when normalized-replay exists.
Tail holes have no gap records. The final disconnect (including reconnect_failed, reconnect_exhausted, and --max-reconnects 0) ends the capture with no trailing gap marker — only the manifest's stop_reason covers it. Correct (no post-hole data exists to corrupt), but the soak judge reads gap lines, so tail behavior must always be read off the manifest, never the stream.
Live-book errors are invisible during capture. Failed book applies are neither counted nor logged — only book_updates increments on success. A capture whose live book silently stopped tracking looks identical in the output to a healthy one. One counter would fix it.
Zero automated coverage on the operator surface. Both CLIs print ~15-line reports and ops/soak.sh judges Gate 1, but all test targets for both main.rs binaries report 0 tests. A renamed field or broken verdict line (which soak.sh check parses!) would pass CI. The verdict-format contract deserves at least a golden-output test.
Unvalidated trade sides and unbounded gap reasons. Bybit S values pass through unvalidated; Coinbase sides are case-normalized but never checked against Buy/Sell; tungstenite error strings go verbatim into gap payloads. Live data is clean; hostile data gets stored verbatim. Low priority by design (raw is raw), noted for completeness.
Benign by inspection (no action): duplicate/overlapping spans apply twice harmlessly (set semantics); round_div rounds wrong only for negative divisors, unreachable in prod (sole production divisor is the constant +2 in mid()); store.rs framing unwrap is guard-proven safe; Timestamp::now saturates at i64::MAX; Bybit subscribe confirmations consume seq as venue frames by design (documented).

Claim spot-check

The README's live numbers (601-frame capture, 300/300 top-of-book, latency tables) cannot be re-observed from this machine (no venue route), but every code path that produces them was reviewed: counters increment in the right places, frames_written excludes synthetic records consistently between writer/manifest/replay/normalize — and now check too. One methodological note for the record: capture latency is measured from socket-read return to record-appended (processing latency, not venue-to-host), which matches the documented definition. Nothing in the verification record was found false; two layout-table rows were stale ("design only, not implemented") and were fixed in an earlier push.