Normalized event schema (v1)

Status: implemented for book_diff, trade, and top_of_book (astra-normalize). funding and liquidation remain reserved table names only. The spec below is the format the normalizer targets and the replay engine builds against — one decided format instead of three invented ones.

Why this document comes before code

Every downstream claim depends on this schema: deterministic replay needs byte-identical normalized output, the audit trail needs provenance, and research needs queryable columns. Designing it now forces the decisions while they are still cheap. Changing it later means migrating stored data.

Sources

The schema is derived from what the recorder already captures, not imagined:

• CaptureRecord: seq, instrument (venue + market type + symbol), channel, ts_socket, optional ts_exchange, raw payload, CaptureFlags
• Binance depthUpdate: U/u update ids, b/a levels, event time E
• Bybit orderbook: u version, b/a levels, type snapshot/delta
• Bybit publicTrade, allLiquidation; Coinbase matches, ticker
• GapMarker: synthetic records marking holes with reasons

Anything the current capture cannot produce is marked as such below rather than specified wishfully.

Tables

One table per channel family. A single unified table would force nullable columns for everything and make every query pay for every channel's shape.

book_diff

One row per venue book-update event.

trade

One row per venue trade print. Aggregated messages (Bybit publicTrade bundles) expand to one row per trade, distinguished by print_index — decided because a bundle is a transport artifact, not a market event. An empty bundle (zero prints) yields zero rows: no events, nothing to placeholder — unlike an unparseable frame, which keeps a null row so seq accounting survives.

top_of_book

One row per top-of-book update (Binance bookTicker, Coinbase ticker).

funding and liquidation

Specified only as reserved table names. No parser exists, and Binance funding/liquidation are not live-verified (geo-blocked). Columns will be defined from the first real payloads, not from documentation.

Fixed rules

• Prices and quantities are fixed-point decimals (scale 10^8, matching Fixed). No floats in any table, for the same reason as in code.
• Timestamps are int64 nanoseconds since epoch. The three-clock system (exchange, socket, ready) is what makes latency measurable later.
• Raw payloads are not stored in normalized tables. The raw chunks remain the source of truth; normalization must be re-derivable from them or it is not a normalization.
• Synthetic gap markers become rows with synthetic = true and null market fields, so a replay over normalized data sees the same holes as a replay over raw data.

Partitioning and files

dataset/
└── venue=binance/
    └── market_type=spot/
        └── symbol=BTC_USDT/
            └── channel=book_diff/
                └── date=2026-09-30/
                    ├── part-00000.parquet
                    └── ...

Hive-style partitions (venue, market_type, symbol, channel, date). Symbols use _ instead of / because / is a path separator. One writer per partition per day; files are immutable once closed, mirroring chunk semantics. Since streaming, each touched chunk appends one row group per partition file — so the same records under different chunkings share file layout and logical rows but package row groups differently (read-back equality, proven by test).

Rerunning into the same output directory does not overwrite: the writer numbers the next part-NNNNN.parquet after existing files, so a rerun duplicates every row. Idempotency comes from fresh output directories, not from the writer — the cross-machine procedure already mandates them.

Validation rules

Every normalized batch must satisfy these before it is accepted:

timestamps within [capture start, capture end + tolerance]   [deferred: tolerance undecided]
prices > 0 where present                                     [enforced]
quantities >= 0 where present                                [enforced]
update ids continuous within the batch unless a synthetic gap row intervenes   [not enforced: continuity is the capture layer's job, checked there]
no duplicate (capture_id, seq, print_index) triples          [enforced]
schema version recorded in file metadata, equal to the writer's version   [enforced]

A batch that fails validation is rejected whole, never partially accepted.

Open questions (undecided, deliberately)

• The tolerance on timestamp validation (clock skew between venue and local).
• Whether ts_ready belongs in the table or in sidecar metadata.
• Compression codec and row-group sizing (measure on real data first).

Decided since v1 (was open, now closed)

• Deterministic normalization: byte-identical files for the same capture twice (CI-enforced), read-back equality across chunkings, no wall-clock reads, fixed-point only. Designed with the replay engine, as required.

What this document is not

Not a migration plan, not a query engine, not a promise of Parquet files by any date. It is the format that future code must target, so that when the normalizer is built, its output is already specified.