Segment Format: indexed segments

This document has four parts. The v8 format is specified first and is authoritative for what is on disk: it is the one version this build writes and reads, and the format acceptance tests are written against its tables. The historical v7 layout follows — the format the v0.24.x line wrote, kept byte-precise because the migrator’s frozen v7 decoder reads it and because v8 carries its records region (blocks, directory-entry layout, record encoding) forward unchanged. The historical v6 layout comes next — the format every release before v0.24.0 wrote, read by the migrator’s frozen v6 decoder; its content-index table remains normative for v8, carried through v7 unchanged. The original v0.3 design proposal is retained at the bottom as history. The proposal does not describe the format that shipped — it predates the implementation and differs in the layout, the section list, and the compatibility story. Read the first section for the current format, the middle two when working on the migrator, and the last only for the reasoning.

Format v8

Status: the shipped format. This section describes what is on disk and is the contract the implementation is held to — the format acceptance tests (tests/segment_format_acceptance.rs) are written against these tables, and a divergence between spec and code is resolved by deliberately amending one of them, never by letting them drift.

v8 is the second bump under the policy below and pays point 3 the same way v7 did: it ships inside the automatic, resumable migrator, whose frozen v6 and v7 decoders are the only code outside this section’s contract that still reads an older layout. The one readable version is 8; a v6 or v7 file is something the store converts at open, never something a query sees.

Two things changed from v7, and they are two halves of one decision about the metadata sections. The records region did not change: v8 carries v7’s compression blocks, block-directory entry layout, and record encoding ((key id, value digest) pairs, payload as the value-text authority) forward byte-for-byte, and the payload blob format (TRZBLOB1) is untouched — the v7 section below remains normative for all of that, exactly as v7 kept v6’s index tables.

First: every metadata section is integrity-protected. This is a defect fix, reproduced against v0.24.2 before it was designed away: flipping one byte of a trace id inside a sealed segment’s trace-index section was accepted at reopen, and get_trace for that trace answered ZERO rows while a full scan still decoded the span — a silent empty answer from exclusion metadata nothing guarded. Every index section is exclusion metadata in that sense: an absent trace entry, a missing attribute posting, a cleared content-row bit, or a doctored header word each prunes work — and therefore rows — without decoding a record, and v7 checksummed none of them (the per-block CRCs guard record bytes only; the v0.24.2 timestamp-range validation guards exactly the two range words and nothing else). In v8 the header carries its own CRC-32 and every metadata section starts with a checksummed wrapper, both verified by BOTH opens before the segment serves anything. The corruption contract this enforces: index or header corruption is an error naming the section, never a shrunken answer. tests/index_integrity.rs pins it with the reproduced mutation and a sweep of every section and header byte, and was shown red against the unguarded behavior (checksum comparisons disabled) per the testing standard.

Checksums guard corruption, not forgery — an adversary who can edit the file can recompute them. The record-anchored timestamp validation that closed the v0.24.1 pruning forgeries (v7’s amendment #5, specified in the v7 section below) therefore carries into v8 unchanged and still runs at both opens: the declared range is proved against the first and last records themselves, and the acceptance tests drive their forgeries PAST the recomputed checksums to keep those deeper checks covered.

Second: the index sections earn their bytes. Measured on the v7 run in benchmarks/storage.md (records-region table), the records region is 45–57% of settled segment bytes by corpus — meaning the index sections are most of the rest, now that the records themselves are compressed. v7 stored them raw and fixed-width: 8 bytes per posting, 4-byte length prefixes, 4-byte counts. v8 stores the record-offset, trace, and attribute bodies as LEB128 varints with posting lists delta-coded, and every wrapped section body MAY additionally be LZ4-compressed (the same pinned lz4_flex, block format) when that is strictly smaller — with the content index deliberately excepted from compression, because its bit-sliced rows are read from disk by byte range at query time. Attribute-value digests keep their full 16 bytes: the 128-bit collision argument is load-bearing and is not traded for size. The savings are recorded where measurements live — benchmarks/storage.md, regenerated by storage-bench on the canonical corpora — rather than promised here as a percentage.

The v8 header

Little-endian, fixed offsets, header length = 132:

Offset	Size	Field
0	8	magic, TRAZASEG
8	2	format version (8)
10	2	header length (132)
12	4	codec id: 0 = raw, 1 = LZ4 block format
16	8	record count
24	8	records offset
32	8	records length, as stored (compressed)
40	8	record-offset index offset
48	8	record-offset index length
56	8	trace index offset
64	8	trace index length
72	8	attribute index offset
80	8	minimum record timestamp
88	8	maximum record timestamp
96	8	content index offset
104	8	block directory offset
112	8	block directory length
120	8	records logical length (uncompressed)
128	4	header CRC-32 over bytes [0, 128)

Offsets 0 through 127 are v7’s fields at v7’s positions; the CRC word is appended. It is verified at parse, after the magic, version, and header-length words (so a future format is still a version refusal, not a checksum one) and before any other field is believed — every offset, length, count, codec id, and timestamp word below it steers section reads, negative pruning, or allocation sizes, and none was covered by anything else. Section order and the contiguity rule are v7’s: header, records (stored), block directory, record-offset index, trace index, attribute index, content index to EOF; every section starts where the previous ends, the attribute index is bounded by the content offset, trailing bytes are refused. Two new structural checks ride on the count words: the record count must fit the logical region at the record framing’s 24-byte minimum before it sizes any allocation, and a section shorter than its wrapper is refused outright.

The segment-level timestamp-range validation — fences cross-checked at open, both bounds proved against the first and last RECORDS, out-of-range reads refused — is carried from v7 verbatim; see amendment #5 in the v7 section.

The section wrapper

Every metadata section — block directory, record-offset index, trace index, attribute index, content index — begins with a 16-byte wrapper:

Offset	Size	Field
0	4	CRC-32 (IEEE/gzip polynomial) over bytes 4..end of section
4	4	flags: bit 0 = body is LZ4-compressed; all other bits must be zero
8	8	body length, uncompressed
16	—	stored body

Validation, all of it mandatory and run by both opens before the section is believed:

• The checksum is verified first, over the flags, the length word, and the stored body together — a flipped byte anywhere in the section is a Corrupt refusal NAMING the section, never a shifted index.
• An unknown flag bit is refused by name, exactly as an unknown codec id is: it means “the wrong decoder”, not “damaged bytes”.
• A raw body’s declared length must equal its stored length.
• A compressed body’s declared length is bounded by LZ4’s ~255x expansion ceiling of the stored bytes BEFORE it sizes the decompress allocation — the same rule the block reader and the blob decoder apply, because a forged length must be Corrupt, never a gigantic allocation that aborts the process — and the decoded length must then equal the declared one exactly.
• The content index’s body must be stored raw (flag 0). Its bit-sliced rows are read from disk by byte range at query time, so a compressed body could not be row-addressed; the wrapper checksum still covers the rows, which is why the file-backed open now reads the whole content section once (rows are CRC-verified, then dropped — only the prologue, summary, and page table stay resident). Rows read AFTER open are verified through the page table below, so the record path and the content path carry the same read-time guarantee.
• All varints in the wrapped bodies must be canonically encoded — a multi-byte varint whose final byte contributes no bits (0x82 0x00 for 2) is refused. The byte→value map stays injective, which the byte-identical re-encode premise (the acceptance tests, the migration’s idempotent re-hash) rests on.

The writer compresses a compressible section’s body exactly when the compressed form is strictly smaller, mirroring the records region’s raw-passthrough rule, so an incompressible section costs its raw bytes plus the 16-byte wrapper and never more. Under codec 0 every section body is stored raw, keeping “uncompressed segment” a codec choice rather than a format variant.

The index section bodies

All integers below are LEB128 varints (seven value bits per byte, low bits first, high bit marking continuation). A decoder refuses a varint that overflows 64 bits or continues past its tenth byte. A posting list is:

varint  posting count            (bounded by the record count before allocation)
varint  first offset, absolute
varint  × (count − 1)            strictly positive gap to each next offset

Postings are logical record offsets in record order, so the sequence is strictly ascending in honest bytes; a zero gap is refused. The lists keep their u64 logical currency — only the encoding changed.

Record-offset index body: exactly record count varints — the first record’s absolute logical offset, then the gap to each next offset (which is the previous record’s encoded length, restating v6’s consecutive-offsets rule). The body must hold nothing else. Ordering and logical-bound validation are v7’s, unchanged.

Trace index body:

varint                entry count (bounded by the remaining bytes)
per entry, in sorted trace-id order:
  varint + bytes      trace id, length-prefixed
  posting list        the trace's record offsets

Attribute index body:

varint                key count (dictionary, sorted key order)
  varint + bytes      each attribute key name, length-prefixed
varint                entry count
per entry, in (key id, digest) order:
  varint              key id (must index the dictionary)
  16 bytes            digest of (key, value), see src/hash.rs
  posting list        the pair's record offsets

The semantics of every probe are unchanged from v7: a digest probe returns CANDIDATES, the record’s own pair list prefilters them, and the store’s payload-derived verification remains the authority (v7’s amendment #2, which carries forward whole).

Block directory body is byte-for-byte the v7 encoding — 32-byte entries — inside the wrapper; its table in the v7 section remains normative. One bookkeeping consequence: the header’s directory-length word now counts the wrapper (and any compression), so the entry count comes from the unwrapped body’s length, and an empty segment’s directory section is exactly one wrapper with an empty body.

Content index body keeps the v6-era prologue, summary filter, and bit-sliced row layout (the tables in the v6 section remain normative for those) with ONE insertion between the summary and the rows: a page-checksum table, one CRC-32 per 4096-byte page of the rows region (CONTENT_PAGE_BYTES), the last page short:

[ 32-byte prologue        ]  unchanged fields, v6 layout
[ summary filter          ]  summary_bits / 8 bytes, resident
[ page-checksum table     ]  u32 × ceil(rows_len / 4096), resident
[ bit-sliced rows         ]  block_bits × row_bytes bytes, on disk

The table exists because the rows are the one exclusion surface read from disk for the segment’s LIFE: record blocks are CRC-checked on every read, but rows were verified only by the wrapper at open, so corruption arriving after open cleared admit bits and silently under-included content matches until the segment was reopened. The wrapper authenticates the table at open; the table then checks every row read. All pages overlapping the row are fetched and verified against their resident CRCs before their bytes are used; a mismatch is Corrupt naming the matrix. Rows wider than one page can require more than two page reads. The resident table costs four bytes per page, including the final partial page: about 0.1% of a matrix made of full pages. Matrix size itself depends on the record count and content. Geometry validation is mandatory before any of this sizes an allocation: summary_bits and block_bits must be powers of two AT OR BELOW the encoder’s own ceilings (32 KiB and 8 KiB of filter respectively), and every derived length — rows, table, section — is computed with checked arithmetic and must account for the section exactly.

Migration: v6 and v7 → v8

The contract is the one v7 shipped with, extended to a second source format: automatic at first open, resumable, and never woven into the read path. Store::open converts a store holding any v6 or v7 segment before serving anything; the frozen v6 decoder (copied at 5f23172) and the frozen v7 decoder (copied at 016eab8, the last commit whose live reader spoke v7) live only inside src/migration.rs.

• Same path, temp + fsync + rename. Each legacy segment is decoded by the decoder its version word names, its records re-derived from their payloads through the same span → record derivation ingest uses, and re-encoded as v8 onto the same file name — name preservation is invariant 1, not hygiene. Rollup sidecars are rebuilt with fresh bindings in the same step, exactly as before.
• The record STREAM is the completeness authority, never the legacy offset index. Both legacy formats store their record-offset index and record count unchecksummed, so an internally consistent shortened index — one offset removed, the count decremented, later sections shifted, every record block untouched — used to migrate a SUBSET of the intact records and publish a freshly checksummed v8 store over the loss (independent migration review, the release’s P1). Both frozen decoders therefore walk the framed records SEQUENTIALLY over the validated record bytes (v7’s after per-block CRC verification and inflation) and require exact, zero-based, gap-free coverage: every record must start where the previous ended AND where the offset index says, the final record must end exactly at the region’s end, the discovered count must equal the declared one, and — for v7 — every block boundary must land on a discovered record start. Any disagreement refuses the whole open by file name with the legacy bytes untouched; nothing lossy is ever published. The legacy QUERY indexes (trace, attribute, content) are deliberately never read and never an authority: a damaged one is discarded and rebuilt from the record stream — repair, not laundering.
• Blobs are already the current format. v8 did not change the blob format, so the blob pass is unchanged from v7’s three-way rule: a file passing full TRZBLOB1 validation is left byte-for-byte alone (every v7-written blob lands here and a v7 store’s blob pass finds nothing to do), a raw file whose bytes SHA-256 to its name is a v6 blob and is rewritten, and a file matching neither is refused per file, never laundered.
• Pins get the same passes plus the manifest-digest rewrite, and resume re-validates every pin without trusting the version-word trigger — a digest-moved pinned segment is re-accepted only after an eager v8 open that checks every wrapper and block checksum and decodes every record.
• Completion is a checkpoint that declares segment_format: 8 in its manifest and is a FULL re-hash, forbidden from carrying any digest forward, publishing folded_through unchanged — all for the reasons the v7 contract states, which carry verbatim.
• The trigger rule at open: any segment (live or pinned) declaring v6 or v7 starts a full migration; all segments v8 but a manifest not declaring 8 — which includes every v0.24.x store’s segment_format: 7 — re-runs the idempotent blob pass, re-validates every pin, and checkpoints. The WAL is neither read nor rewritten; frames after folded_through replay against the migrated store, and a pending erasure settles after migration.

Costs and the rollback boundary, stated. Migration duration is proportional to store size — every record is decoded and re-encoded once, plus once more per pin — and nothing is served until it completes. Peak memory is several multiples of a segment’s LOGICAL (uncompressed) record bytes, which no on-disk cap bounds for a v7 source; the module doc in src/migration.rs states what overlaps and why the estimate is workload-dependent rather than a promised ceiling. It is one-way from the first converted file: a v8 segment is not readable by any v0.24.x build, so after a mid-migration interruption the store holds mixed legacy/v8 files that only a v8-capable build can finish, and the store is down (data-safe, not availability-safe) until that open completes. The operator’s rollback boundary is therefore BEFORE the first v8 open: take a backup by one of the two procedures in the durability guide — stop the server and copy the directory, or take a filesystem snapshot atomic across the whole directory — and keep it until the migrated store has been verified (/v1/verify reporting intact: true after the completion checkpoint). Every conversion is atomic onto its own name, so ENOSPC mid-migration fails the open loudly at the file it starved and the next open resumes from exactly that file; free space to check beforehand is the live store’s rewrite plus one full copy per pre-existing pin, because rewriting pinned files breaks their hard links — the cheap-pin promise stops holding for pins that pre-date the migration, exactly as it did across v6 → v7 (backup guide). Releasing pins before migrating and re-taking them after remains the cheap way out.

Determinism: encoder output is format bytes

The rule carries from v7 and now covers the section wrappers too: encoding the same records twice yields identical bytes, compressed sections included, so the codec crate stays pinned to an exact version and upgrading it is a deliberate re-baseline. Varints are written minimally by the encoder; the decoder accepts any in-bounds encoding, so determinism is a property of the writer, exactly as it is for the compressor.

Acceptance gates

Gates 1 through 4 of the v7 spec carry forward, re-based on v8 and enforced by the test suite:

1. Round trip: encode/decode equality on randomized corpora, raw passthrough and oversized records included.
2. Derivation invariant: stored (key id, digest) lists re-derived from parsed payloads, byte for byte.
3. Collision safety: both layers, forged postings and planted pairs.
4. Migration: v6 stores (downgraded fixtures plus the committed pr50 bytes) and v7 stores (the committed tests/fixtures/v8-migration baseline bytes, written by v0.24.2 at 016eab8) open into query-identical answers, same names, verified manifests and pins, with SIGKILL resume proven on the crash matrix and mid-migration v7 state proven to resume. Completeness is gated too: an internally consistent shortened legacy offset index (first, middle, or last offset omitted; an offset nudged into a record; trailing unindexed record bytes; live and pinned alike) refuses by file name with the legacy bytes untouched — never a lossy conversion — while a damaged legacy QUERY index migrates cleanly, rebuilt from the record stream.

v8 adds a gate of its own:

5. Integrity: single-byte mutations anywhere in the header or any metadata section — including the reproduced trace-index defect — must refuse at open or serve the span, never answer clean-and-empty; forged wrapper lengths and content geometries must be refused before any allocation and never panic; malformed or non-canonical varints, inflated posting counts, and zero deltas are refusals by name; and content-matrix damage arriving AFTER open surfaces as Corrupt at the query that touches it, never as fewer matches (tests/index_integrity.rs, and the hardening tests in the acceptance suite).

The two measurement gates keep their v7 definitions and harness enforcement — storage-bench asserts settled amplification at or below 1.0x on generic and llm before writing its record, and bench asserts the trace-lookup and attribute-filter tripwires (0.75 ms / 6 ms p50) before writing its own. Both were re-run on v8 and passed: their published records in benchmarks/ are v8 runs — settled amplification 0.23x on generic and 0.16x on llm (storage.md), trace-lookup p50 0.577 ms and attribute-filter p50 3.657 ms on the canonical corpus (canonical-corpus.md) — and the paired v7-vs-v8 before/after, with its run-variance honesty notes, is recorded in storage-v8-comparison.md.

---

Format v7 (historical — the migration source)

Status: historical. This is the layout the v0.24.x line wrote. No shipped build writes it any more, and the only code that reads it is the migrator’s FROZEN v7 decoder (src/migration.rs, copied from src/segment.rs at 016eab8) — this section is that decoder’s reference, kept byte-precise for exactly that reason. Most of it remains live beyond the migrator: the records region — the compression blocks, the block-directory entry layout, the record encoding with its digest pairs, the timestamp-range validation (amendment #5), and the two-layer candidate verification (amendment #2) — is carried into v8 unchanged and stays normative there, as do the payload-blob format and the erasure semantics. What v8 replaced is only the header length (a CRC word was appended), the raw index-section encodings below (now varint bodies inside checksummed wrappers), and the migration contract (superseded by the v6-and-v7 → v8 section above). The section was written while v7 was the shipped format and keeps that voice, its labeled amendments included; read “the shipped format” here as “the format v0.24.x shipped”.

v7 was the first bump made under the policy below, and it pays point 3 in full: it shipped with an automatic, resumable migrator, and the v6 decoder lives inside that migrator rather than surviving in the query path.

Three things changed from v6 — the records region, the payload store, and the header that describes them. The four index sections and the record encoding’s framing keep their v6 byte layout: the byte tables for the attribute and content indexes live in the historical v6 section and remain normative for v7, carried over unchanged. The WAL, the annotation/eval/tombstone logs, and the rollup sidecar formats are untouched.

The motivation is measured, at 65652a2 on the storage-bench corpora — a v6-era run whose figures are cited here. benchmarks/storage.md now records the v7 run that superseded it (its records-region section restates these numbers as the retired motivation); the full v6-era record lives in that file’s git history:

• The records region is where the bytes are: 89.6% of segment bytes on generic, 95.3% on llm, 91.9% on pinned-context, corpus-wide, and no individual segment file above 95.3%. Measured from the kept corpora of that run (TRAZA_STORAGE_BENCH_KEEP=1 leaves the data directories on disk; each file’s header gives its records region at bytes 24 and 32).
• A third of the LLM corpus’s record bytes are a structural double-store: every indexed attribute value is held once inside the span’s JSON payload and again, verbatim, in the record’s own key/value list. Counting the value text alone — key names and length prefixes excluded — the duplicated text is 36.1% of llm’s records region, 25.6% of pinned-context’s, and 7.4% of generic’s. The double-store grows with exactly the workload this store exists for.
• Compressing the records region is worth a multiple, not a percentage. The projection storage-comparison published at the time — labeled a projection there and here — puts block-wise zstd -3 at a 6.30x/11.59x/9.27x segment shrink across the three corpora. (It lives in that file’s history before 94e7f86; the page now records where v7’s and v8’s savings came from.) v7 uses LZ4, which trades ratio for speed and safety of implementation, and cuts blocks at 128 KiB where the projection was computed at 64 KiB — so those numbers are an optimistic reference measured under different knobs, not a forecast of v7’s. The acceptance gates below, not the projection, are what the implementation must meet.

v7 records: digests instead of value text

A v7 record is:

Field	Size
timestamp	u64
trace-id length	u32
attribute count	u32
payload length	u32
reserved, zero	u32
trace id	trace-id length bytes
attribute pairs	attribute count × (key id u32 + value digest 16 bytes)
payload	payload length bytes

The framing is v6’s; what changes is the attribute list. v6 stored each pair as two length-prefixed strings — 4 + key + 4 + value bytes, with the value in its canonical JSON form. v7 stores a fixed 20 bytes per pair: the key id indexes the attribute-index section’s key dictionary, and the digest is the same 128-bit (key, value) digest the attribute index posts under (src/hash.rs). Pairs are written in ascending key-id order, which is the order v6’s BTreeMap iteration already produced, so encoding stays deterministic.

The value text is not lost; it was never only here. The payload is the store’s own serialization of the span, and the v6 key/value list was derived from that same span at seal time (span_to_record in src/lib.rs): user attributes minus NUL-prefixed keys, their values canonicalized to JSON text, plus the reserved service/name/tenant entries carrying their raw text — only user attributes are JSON-canonicalized, and the derivation, that asymmetry included, is the definition. That derivation becomes a format invariant: for every v7 record, applying it to the parsed payload must reproduce the stored (key id, digest) list exactly. An acceptance test pins it by re-deriving the pairs from every payload in a randomized corpus — it is what makes the digest list droppable text rather than lost data.

Candidate verification is two layers, and the parse authority lives in the store. This paragraph is deliberate amendment #2 to the original specification, made under this document’s own drift rule: the spec as first written said segment-layer confirmation “parses the payload, applies the same derivation to the queried key, and compares the result.” The implementation deliberately does not do that at the segment layer, and the contract now says what it does do:

• The segment layer confirms by digest pair. Segment::query_attribute and the record_carries_attribute prefilter compare a candidate against the (key id, digest) pairs the record ITSELF carries — a 20-byte compare, no parse. This discards every forged or corrupt index posting (an entry pointing at a record that never held the pair), and it preserves the economics the original text wanted: the payload parse is paid only on digest matches. What it cannot see through, by construction, is a true 128-bit collision — a colliding pair sits in the record bytes exactly as an honest one does. Segment::query_attribute on its own is therefore digest-confirmed, not collision-safe, and its doc comment says so; nothing in the serving path uses it bare.
• The store layer holds the authority. Every result the STORE returns is verified against the value derived from the record’s parsed payload — span_matches and its relatives in src/lib.rs — under the same derivation span_to_record defines: canonical JSON for a user attribute, raw text for the reserved service/name/tenant keys, which is already the form the probe side builds its needles in. No digest forgery can satisfy that comparison, so a collision costs a wasted decode and can never produce a wrong row.

The semantics do not move: an index still only narrows a filter. The forged-collision acceptance test proves each layer at its own job — the posting-only forgery is discarded by the segment layer’s digest compare, and a full forgery (the colliding pair planted in both the record bytes and the index, the shape a real collision has) is admitted by the segment layer and rejected by the store’s payload-derived verification. The parse cost was already paid by every true match (results are returned as spans), so the parse authority costs extra only on false candidates, which digest probing makes rare.

The pairs stay in the record rather than vanishing entirely for three reasons. A full scan can reject a record against an attribute predicate on a 20-byte compare before parsing its payload — false positives possible, false negatives not, so the payload parse remains the authority. A segment rewrite can rebuild the attribute index from records without parsing every payload. And the record remains self-describing against its index entries, which is what erasure’s segment verification walks.

The records region: record-aligned compressed blocks

The records region is carved into blocks. Not the content index’s blocks — those are 128 records; to keep the two apart this section always says compression block. A compression block:

• targets 128 KiB of uncompressed record bytes: the writer appends whole records until adding the next record would cross 128 KiB, then cuts the block. A block always holds at least one record, so a single record larger than 128 KiB becomes a block by itself. No record ever spans two blocks; a reader may treat a record that would as corrupt.
• is independently compressed with the codec the header names — LZ4 block format (not the LZ4 frame format), as lz4_flex’s block API produces it, with no length prefix of its own: the directory carries the lengths.
• carries a raw-passthrough flag for incompressible blocks: if the compressed output is not strictly smaller than the input, the block is stored as its raw bytes and flagged. This bounds the worst case at raw size plus the directory, and it is why already-compressed payload text cannot make v7 larger than v6’s records were.

The stored region is the concatenation of the blocks’ stored bytes — compressed or raw — with no per-block framing inside the region; the block directory is the only framing. A segment whose header declares codec 0 (raw) is carved and directed identically, with every block stored raw; logical and physical offsets then coincide, and what the segment keeps is the per-block CRC and the directory’s timestamp fence. That uniformity is deliberate: there is one reader shape, and writing uncompressed segments is a codec choice, not a format variant. The raw codec is format-supported but not operator-exposed in v0.24.0 — a deliberate ruling, not an oversight: every segment the store writes is encoded under LZ4 (encode_with hard-codes it), segment::encode_with_codec is the library surface that writes codec 0, the acceptance tests exercise it, and the reader accepts both. An operator flag is deferred until someone asks for one; if it ships, it belongs to configuration, which names flags so this document does not have to. An earlier draft of this sentence asserted the flag as if it existed — amended here under the drift rule.

One derived constraint: the raw flag lives in the directory’s stored-length word (below), so one record’s encoding must be smaller than 2^31 bytes. That is a new bound, not a restatement of an old one: v6 caps individual fields at u32 — trace id, payload, attribute count (encode_record in src/segment.rs) — and nothing in it bounds the record as a whole, so a v6 record assembled from several large fields can legally exceed 2^31. (The server’s 64 MiB request cap keeps ingest far below it, but that is a server limit, not a store one.) The v7 encoder rejects an oversized record with the TooLarge error class, naming the record by trace id and timestamp, and the migrator inherits the rule: a v6 record whose v7 encoding reaches the bound fails migration as a named Store::open error — the encoder’s TooLarge wrapped with the one thing the encoder cannot know, which segment file — never a silent skip. No plausible span comes near 2 GiB in one record; the point is that if one ever does, the failure is loud and actionable rather than a record quietly missing from the migrated store.

The block directory

A new section, uncompressed, holding one 32-byte entry per compression block, in block order:

Field	Size	Meaning
logical start	u64	offset of the block’s first byte in the uncompressed records region
stored offset	u64	offset of the block’s first stored byte, relative to the records section start
stored length	u32	bytes as stored; bit 31 set = raw passthrough, remaining 31 bits are the length
crc32	u32	CRC-32 (the IEEE/gzip polynomial) over the stored bytes exactly as they appear in the file
min timestamp	u64	timestamp of the block’s first record

Validation at open, all of it mandatory: logical starts strictly increasing from 0; stored offsets strictly increasing from 0; the masked stored lengths sum to the header’s records length; the last block’s logical extent ends at the header’s records logical length; entry count × 32 equals the section length. Every block’s logical extent is bounded before it can size an allocation: a block holding more than one record spans at most the 128 KiB carving target, a single-record block stays below 2^31 (the record bound), and a compressed block’s extent cannot exceed LZ4’s ~255x expansion ceiling of its stored bytes. The bounds exist because the extents derive from header and directory words no checksum covers, and the reader allocates a block’s extent before decompressing into it — a forged length must be Corrupt, never a gigantic allocation that aborts the process. The CRC is checked on every block read, before decode — it is why the field exists, and a mismatch is Corrupt naming the block. A block’s decoded size must equal its logical extent; anything else is Corrupt too.

The min-timestamp column is verified against the records, not trusted. It is derived metadata the CRCs do not cover, and it steers the window search without a decode, so a corrupt-but-still-sorted fence could otherwise shift a window bound silently — returning out-of-window rows or dropping in-window ones. Two checks close it, both mandatory: every decoded block’s first record must carry exactly its directory entry’s min timestamp (the byte-resident open decodes every block eagerly, making this an open-time check on that path), and the window search verifies each landed bound against the records on both sides of it — at most two timestamp reads, usually inside the block the search just decoded. Either disagreement is Corrupt, never a shifted answer. The spec’s first draft omitted this column from the mandatory list; amended here under the drift rule after review demonstrated the silent-wrong-window consequence.

The header’s timestamp range is validated too, not trusted. (Deliberate amendment #5, under the drift rule, after external review of v0.24.1 demonstrated the silent-omission consequence; the validation was then grounded in record bytes after second-round review demonstrated that a fence-only check could be colluded with.) The two range words at header offsets 80 and 88 carry no checksum, and may_contain_timestamps prunes whole segments on them before a single segment byte is read — so the format as first shipped let sixteen edited bytes hide decodable records from every time-bounded query: a false-negative predicate, which the corruption contract (an error or a conservative scan, never a silent omission) forbids. Three checks close it:

• Both opens, against the directory. A non-empty segment’s declared minimum must EQUAL the first directory fence — a fence is its block’s first record’s timestamp, and records are timestamp-sorted — and the declared maximum must sit at or above the LAST fence (fences carry minima only), which through the fences’ own ordering also forces min ≤ max. An empty segment must declare exactly the canonical empty range the encoder writes — minimum u64::MAX, maximum 0 — and nothing else. Any violation is Corrupt. On its own this is only a consistency check between two pieces of unchecksummed metadata — the fences are derived data too, so a forgery can move a fence and the header word together — which is why the next check exists.
• Both opens, against the records. The bounds are then proved against record bytes. The eager, byte-resident open decodes every record anyway and requires BOTH declared bounds to equal the observed extremes. The lazy open decodes exactly two blocks: the FIRST — whose fence-vs-first-record check (above) turns the pinned minimum into the actual first record’s timestamp — and the block holding the LAST record, whose timestamp is the true maximum (records are timestamp-sorted) and must EQUAL the declared maximum. That costs the lazy open two bounded block decodes, which it drops from its cache afterwards so a freshly opened segment still holds zero payload bytes.
• At read time, on every backing. Any record decode — and any fence-adjacent timestamp read — that observes a timestamp outside the declared range is Corrupt: defense in depth behind the open-time checks, for damage that arrives after open.

So: both opens validate both bounds exactly, and there is no lazy/eager asymmetry in what they prove. A metadata-only forgery of the range — any combination of the header words and the fences — is refused at open, because the bounds are anchored to the first and last records themselves; a forgery that survives must therefore alter record bytes as well, and the block crc32 catches that on the very decode the validation performs. No forgery of the range can cause a wrong row to be SERVED either: every served record passes the read-time range check.

Posting lists keep their u64 currency as LOGICAL offsets into the uncompressed records region — the record-offset index, the trace index, and the attribute index are byte-for-byte v6 encodings whose values simply mean “offset before compression”. The translation lives entirely in the reader: binary-search the directory’s logical starts for the containing block, read its stored bytes, verify, decode, and index into the decoded buffer at logical − logical start. A record’s byte length is still the gap to the next entry in the record-offset index — the last record ending at the records logical length — which is v6’s consecutive-offsets rule restated in logical bytes. Whether a reader caches decoded blocks is an implementation choice the format does not constrain; the format guarantees only that one block decode suffices for any record.

CRC-32 over the gzip polynomial is already in the crate: the WAL frames every record under exactly this polynomial. The directory shares that implementation — hoisted into src/crc.rs, where the WAL, the directory, and the blob header all reach it, not written a second time — and it costs no dependency.

Because records are timestamp-sorted (the v6 invariant, unchanged), a block’s first record carries its minimum timestamp, and the directory’s min-timestamp column is a sorted array. A window disjoint from the segment never gets this far in either version: the header’s timestamp range answers it (may_contain_timestamps in src/segment.rs), exactly as in v6. For a window that does overlap, ordinal_range_for_window’s binary search paid log2(n) 8-byte reads into the v6 record region; the v7 probe narrows to a block through the resident directory — no read, no decode — and pays at most a boundary-block decode to place the exact bound.

The directory is read at open and held resident: 32 bytes per 128 KiB of uncompressed records, so a segment with 256 MiB of uncompressed record bytes carries a 64 KiB directory — arithmetic from the constants, not a measurement. That is the resident price the FORMAT imposes. The reader adds one implementation cost on top: a small decoded-block cache per open segment (BLOCK_CACHE_SLOTS in src/segment.rs, four blocks — nominally 512 KiB once queries have touched the segment, bounded by the largest record rather than by 128 KiB when single records exceed the carving target, and retained for the segment’s life). That residency is counted by Store::resident_payload_bytes rather than hidden; an earlier draft of this paragraph said “nothing else new is held in memory”, which was true of the format and false of the reader, and is amended here.

The v7 header

Little-endian, fixed offsets, header length = 128:

Offset	Size	Field
0	8	magic, TRAZASEG
8	2	format version (7)
10	2	header length (128)
12	4	codec id: 0 = raw, 1 = LZ4 block format
16	8	record count
24	8	records offset
32	8	records length, as stored (compressed)
40	8	record-offset index offset
48	8	record-offset index length
56	8	trace index offset
64	8	trace index length
72	8	attribute index offset
80	8	minimum record timestamp
88	8	maximum record timestamp
96	8	content index offset
104	8	block directory offset
112	8	block directory length
120	8	records logical length (uncompressed)

Offsets 16 through 103 are v6’s fields at v6’s positions; the codec id takes the u32 that v6 reserved as zero, and three u64s are appended. The records logical length exists because the directory entry has no room for the last block’s uncompressed size and because record-offset validation needs the logical bound (offset < logical length, exactly as v6 checked against records_len).

Section order in the file: header, records (stored), block directory, record-offset index, trace index, attribute index, content index to EOF. The v6 contiguity rule extends to the new section: every section starts where the previous ends, the attribute index is still bounded by the content offset, the content index still runs to EOF, and trailing bytes are still refused. An empty segment has empty records, an empty directory, logical length zero, the canonical empty timestamp range (minimum u64::MAX, maximum 0 — mandatory since amendment #5 above), and otherwise encodes as v6-empty did.

The codec id is parameterization, not a version. A reader refuses an unknown codec id with an error that names it — the same shape as the version refusal, and for the same reason: decoding bytes under the wrong codec produces garbage, not errors. Adding a codec (the evidence-gated zstd cold tier, if it ever clears its gate — see dependencies) is configuration plus a new id, not a format bump: v7 readers built after it read both, and the cost is confined to stores that opt in.

What stays raw, and why

The header and all four index sections — record offsets, trace, attribute, content — are stored uncompressed. The reader parses them at open and probes them per query: posting lists are probed by digest and intersected, content rows are read by exact byte range, the offset table is the scan path. Compressing them would put a decode in front of every probe, and it would buy little — the argument storage-comparison made at the time (before 94e7f86): past compressed records, the work is in the indexes, not the payload. After v7 the indexes are roughly half of segment bytes — measured at 43–55% by corpus, the complement of the records shares in storage.md’s records-region table; the projection’s tables had put them at 55–75%, an overestimate — and shrinking them is future index-format work, not a codec knob. That work is v8’s; storage-comparison now records where its savings come from.

The WAL and the rollup sidecars also stay raw, deliberately: erasure verification runs byte-level occurrence scans over both (see administration), and those scans work precisely because the bytes on disk are the literal bytes. The same goes for annotations.jsonl, evals.jsonl, and tombstones.jsonl. Compression is confined to the two places the measurement says the bytes are: segment records and payload blobs.

Payload blobs

A v7 blob in payloads/<aa>/<sha256>.bin is a 24-byte header followed by the stored body:

Offset	Size	Field
0	8	magic, TRZBLOB1
8	4	codec id: 0 = raw, 1 = LZ4 block format
12	8	uncompressed length
20	4	crc32 over the stored body bytes
24	—	body

The writer compresses each blob independently and falls back to codec 0 when compression does not strictly shrink it. There is no random access inside a blob: blobs are consumed whole — GET /v1/payloads/… streams the decoded bytes, dedup compares nothing but names — so one codec unit per file is the right shape and block carving would be waste.

The uncompressed-length word is bounded before it is believed. The crc32 covers the stored body only, so the header’s length field is protected by nothing — and the decoder allocates the declared length before inflating into it, which would turn one flipped header byte into an allocation-failure abort on the read path. A declared length beyond LZ4’s ~255x expansion ceiling of the body is Corrupt, naming the file, checked before any allocation; the decoded length must then still equal the declared one exactly.

The content address does not move: it is the SHA-256 of the UNCOMPRESSED bytes. Dedup therefore behaves exactly as v6’s — an identical pinned context still collapses to one file under the same name — and erasure’s literal reference needles (sha256/<hex> strings in spans, logs, and eval records) are unchanged bytes matching unchanged names. What does change: the file’s bytes no longer hash to its name, so any check that verified a blob by hashing the file (sha256_file) must decode first; blob verification in v7 is header parse, CRC check, decode, then SHA-256 of the decoded bytes against the name.

Migration: v6 → v7

The contract, per the policy: automatic at first open, resumable, and never woven into the read path. Store::open on a v0.24.0 build converts a v6 store before serving anything; the v6 decoder exists only inside the migrator module.

• Same path, temp + fsync + rename. Each v6 segment is decoded, its records re-derived through the same span → record derivation ingest uses (parse payload, derive attributes and content text — both are recoverable from the payload, which is what makes this a re-encode and not a lossy copy), and re-encoded as v7 onto the same file name. Name preservation is not hygiene; segment path order IS recency order, and a migrator that assigned fresh ids would reorder last-write-wins. The content index is rebuilt under the store’s current content-index configuration, exactly as compaction already behaves.
• Rollup sidecars are rebound in the same step. The sidecar’s binding includes the segment’s byte length, which the rewrite changes, so every existing sidecar becomes self-invalidating — correct, but a rebuild storm at first query. The migrator instead computes the rollup from the records it already has decoded and writes a freshly bound sidecar beside each migrated segment. The write order of sidecar against segment rename is deliberately unspecified, so a crash in that window can leave a sidecar bound to a segment that no longer matches it; resume does not repair this, because it does not need to — a stale-bound sidecar self-invalidates and rebuilds on first use, a performance cost, never a wrong answer.
• Blobs are rewritten in the same pass, onto the same names, by the same temp + fsync + rename discipline, so that the reader accepts exactly one format of each — no header-sniffing in the serving path, ever. The blob pass classifies each file by a three-way rule, never by its magic alone. A file that passes full v7 validation — magic, CRC, decode, SHA-256 of the decoded bytes against the name — is already migrated and is left alone. A file that fails that, but whose raw bytes SHA-256 to the file name, is a v6 blob and is migrated; a v6 blob whose content merely happens to begin with the magic bytes lands here, which is why magic alone was never the test. A file that fails both is Corrupt, named per file, and the migrator refuses to rewrite it — rewriting would launder bytes that match neither format into a validly framed v7 blob. That refusal fails the whole open, deliberately: the store serves nothing until it holds one format, so there is no “serve around the bad blob” state to fall back to. The pass collects every unrecognized file before refusing, so one report covers them all, and the next open after the operator acts resumes the migration from where it stopped — a pending erasure settles after that open, not before.
• Pinned backups are migrated the same way. A pin is a hard-link farm that keeps pre-rewrite inodes alive, so after the live pass every pin still holds v6 bytes — bytes a v0.24.0 restore could not read. The migrator runs the same two passes inside each pins/<label> and then rewrites the pin’s state-manifest.json digests to match, because restore verifies a pin against those digests before installing it. One crash window needs its own rule: between a pin’s file pass and its manifest rewrite, every file in the pin already reads as v7, so the version-word trigger below sees nothing left to do — and a pin whose manifest still carries v6 digests fails every restore. Resume therefore does not trust the trigger for pins: every migration resume re-validates each pin’s files against that pin’s manifest digests, and where the files validate as v7 but the digests disagree, it redoes the manifest rewrite. “Validate” is load-bearing and means the WHOLE file: a digest-moved pinned segment is accepted only after an eager open that checks every block CRC and decodes every record — the lazy open reads the header and index sections plus only the two blocks its bounds validation decodes (amendment #5), and trusting it would launder a bit-flipped pinned middle block into a manifest that then verifies clean over garbage. A digest-moved manifested file the migrator cannot validate in any format — neither a segment nor a content-addressed blob — refuses the resume by name, for the same reason. The rewrite is idempotent, so re-validating a finished pin costs a hash pass and changes nothing. A pin already copied off-host is outside the data directory and outside this contract: it restores only under a build that reads its format, which the backup guide already says of every backup.
• Completion is recorded in the manifest, by a checkpoint that re-hashes everything. The migration’s final act is a checkpoint whose manifest declares the store format; every later checkpoint carries the declaration forward. This checkpoint is exempt from the incremental rule ordinary checkpoints live by — the backup guide lets a checkpoint carry segment digests over from the previous manifest because segments are immutable, and migration has just violated that premise wholesale: every segment and blob was rewritten onto its same name. A carried-over digest is therefore a digest of bytes that no longer exist, and a completion checkpoint that carried one forward would publish a generation that fails verification everywhere — verify-at-pin fails, and restore is impossible. The completion checkpoint is a full re-hash and is forbidden from carrying any digest forward. For the same reason, the live generation’s manifest is expectedly stale from migration start until that checkpoint: it describes the pre-migration bytes, and nothing during migration consults it. The checkpoint publishes folded_through unchanged from the prior generation — it folds nothing, because the migrator does not replay the WAL — and frames after folded_through replay normally against the migrated store afterward. The trigger rule at open: any segment (live or pinned) declaring v6 starts a full migration; all segments v7 but no manifest declaration re-runs the idempotent blob pass, re-validates every pin as above, and then checkpoints. Both passes are resumable by construction — every file conversion is atomic, a v7 segment is recognized by its version word, a v7 blob by validation — so a crash at any point re-runs from where it stopped, converting only what remains.
• Migration runs before WAL replay and before any maintenance; the WAL is format-independent of the segments — binary-framed JSON batches under its own TRZWAL02 magic (src/wal.rs) — and is neither read nor rewritten by the migrator. A pending erasure resumes after migration, against the migrated store.

Costs, stated: migration duration is proportional to store size — every record and every blob is decoded and re-encoded once, plus once more per pin. There is a single writer and there are no reads during migration; the store serves nothing until it holds one format. Pins also cost disk, permanently: rewriting a pinned file allocates a new inode, and the migrator does not re-link identical outputs across pins or against the live store — detecting that two independently produced rewrites are byte-identical and re-linking them is machinery this design declines to build, so after migration every pre-existing pin is a full independent copy of its generation. Two consequences an operator plans around: migration requires free space for the live store’s rewrite plus one full copy per pre-existing pin, and the cheap-pin promise — the backup guide’s “a pin costs almost no disk” — stops holding for pins that pre-date the migration. The free space is the operator’s to check, not the store’s. This section originally demanded an up-front probe (“refused with a named error if absent, never discovered at 90%”); the implementation deliberately amends that sentence — per this file’s own drift rule — because the standard library has no portable free-space call, the libc it would take is a dependency the ledger has not admitted, and running dry mid-migration is not the hazard the sentence assumed: every conversion is atomic onto its own name, so ENOSPC fails the open loudly at the write it starved and the next open after space is freed resumes from exactly that file. The probe would buy earlier notice of an already-safe condition at the price of a dependency; if the ledger ever admits libc for other reasons, it is worth adding then. One consequence the removed probe DID buy is owed an honest sentence: refusal before the first rewrite left a store the previous build could still serve, whereas migration is one-way from the first converted file — after a mid-migration ENOSPC the store holds mixed v6/v7 files that only a v0.24.0+ build can finish, and since nothing serves during migration, the store is down until disk is freed. Data-safe, not availability-safe: an operator short on disk checks free space, or releases pins, BEFORE the first v0.24.0 open. The backup guide’s cheap-pin sentence was v6-era text this spec did not edit; the implementing PR amends backup.md alongside this section. The cheap way out is procedural: release pins before migrating and re-take them after, when the disk cannot afford the copies — a fresh pin of the migrated store hard-links v7 files and costs almost nothing again. Un-pinned historical generations stop being verifiable once their files are rewritten, exactly as they do after any compaction; the post-migration checkpoint is the first generation that describes the v7 store. An operator who wants a fallback takes a backup first, by one of the two procedures in the durability guide — the same sentence every migration section in this file has ever said.

Erasure and verification under v7

Erasure’s segment domain was never a byte scan, and v7 does not make it one. v6 verifies segments through their indexes: an index-driven walk selects candidate records and decodes exactly those (subject_keys_in_segment in src/lib.rs); the byte-level occurrence scan (count_occurrences) runs over the WAL and the rollup sidecars only. v7 keeps that shape — the walk now reads each selected record through its block’s decode, and the semantics do not move. What compression changes is only how emphatically the rationale holds: a raw byte scan of a segment file proved nothing against v6’s binary encoding, and against LZ4 output “I hex-grepped the file and found nothing” is not even a scan of the record bytes — which is why the receipt names the domain as index-verified rather than scanned. The WAL and rollup-sidecar byte scans are unchanged because those file formats are unchanged: the migrator rewrites every sidecar’s contents, but the rewritten bytes are the same raw format the scan reads; only the WAL file itself is untouched. Blob checks decode before hashing, as above. A settle receipt taken before migration cites generations whose digests stop verifying when migration rewrites their files — the loss every un-pinned historical generation takes, and no worse: the finding is re-derivable by re-running verification against the migrated store, and a pin taken at settle time preserves the cited generation if an audit needs the original bytes. The in-place segment rewrite that a purge performs writes v7 through the ordinary encoder; nothing about the erasure contract’s ordering, its receipt semantics, or the tombstone log moves.

Determinism: compressor output is format bytes

The v6 rule that encoding the same records twice yields identical bytes is kept, and it now covers the compressor: the codec’s output bytes are format bytes under the acceptance tests. Reproducible merges and the byte-comparing format tests depend on it. Therefore the codec crate is pinned to an exact version (= in Cargo.toml), and upgrading it is a deliberate re-baseline: a commit that bumps the pin, regenerates the golden bytes, and says so — never a routine dependency bump. The pin is about encoder stability only; any correct LZ4 decoder reads any valid stream, so readability never depends on the pinned version. The dependency itself is argued in internals/dependencies.md.

Acceptance gates

The implementation is held to these, each with an executable oracle:

1. Round trip: encode/decode equality on randomized corpora, including blocks that trip the raw-passthrough flag and records larger than one block.
2. Derivation invariant: for every record in a randomized corpus, the (key id, digest) list re-derived from the parsed payload equals the stored list byte for byte.
3. Collision safety: the forged-collision tests carry over and cover both layers of amendment #2 — a forged index posting is discarded by the segment layer’s digest compare, and a collision planted in the record bytes AND the index (the shape a real collision has) is rejected by the store’s payload-derived verification. A fabricated collision must cost a wasted decode, never a wrong row.
4. Migration: a v6 store (segments, blobs, sidecars, pins, non-empty WAL) opened by v0.24.0 serves query-identical results, preserves segment names, and survives kill -9 at arbitrary points mid-migration with a clean resume — the crash-point harness the durability tests already use. The migrated store must also prove its manifest: the post-migration generation verifies clean (/v1/verify reports intact: true) and a pin taken immediately after migration passes its verify-at-pin. That half of the gate is what catches a completion checkpoint publishing digests of bytes the migration replaced.
5. Storage: settled amplification at or below 1.0x on the generic and llm corpora, measured by storage-bench and recorded in benchmarks/storage.md.
6. Latency: measured, stated, and bounded — not hidden inside a relative percentage. (Deliberate amendment #3: this gate originally demanded p95 within 10% of v6. The profiled floor is structural and the amendment records it rather than shaving the benchmark to pass: a v7 point read pays one 128 KiB block read + CRC + inflate — measured ~+11 µs on a trace lookup (49 µs vs 38 µs, in-process A/B over a settled million-span store) — where v6 paid a handful of ~500-byte reads, and a limit-100 attribute filter at sparse candidate spacing inflates ~38 blocks (~1.23 ms vs 0.64 ms on the same A/B). A relative gate on a double-digit-microsecond operation measures machine noise, not the format.) The gate as amended: on the canonical corpus, interleaved against the v6 baseline on the same machine in the same session, trace-lookup p50 ≤ 0.75 ms and attribute-filter p50 ≤ 6 ms absolute (generous tripwires over the measured medians, in the house style — tripwires, not oracles), ingest throughput not below the v6 baseline (measured +22%: compression writes fewer bytes), and the measured medians of both sides published beside each other in benchmarks with the block-decode cost stated plainly. (Deliberate amendment #4, to the publication clause only: the files under benchmarks are harness-written records of canonical runs, and no harness on this branch can produce a v6-side run — the v6 writer is gone, and the only v6 code left is the migrator’s frozen decoder — so a benchmarks/ file for the interleaved medians would be hand-written prose wearing a harness record’s clothes. The record of the interleaved v6/v7 medians is amendment #3 above — 49 µs vs 38 µs on trace lookup, ~1.23 ms vs 0.64 ms on the limit-100 attribute filter, with the block-decode cost stated there — and this amendment names that text as the publication the clause requires.)

Gates 1 through 4 are enforced by the test suite. Gates 5 and 6 are measurement gates, and they landed. storage-bench asserts gate 5 after measuring and before writing, and refuses to write benchmarks/storage.md on a miss; bench asserts gate 6’s two latency tripwires the same way against benchmarks/canonical-corpus.md. Both gate and publish on canonical runs only — default corpora, default configuration; an experimental knob prints its table and leaves the record alone, un-gated, because the gates are defined on the canonical configuration. Gate 6’s other two clauses are recorded rather than harness-asserted: the ingest clause rests on the interleaved A/B’s measured +22% above, and the publication clause is satisfied by amendment #3, per amendment #4. The recorded runs are published: settled amplification 0.41x on generic and 0.23x on llm (gate 5’s bound is 1.0x), and trace-lookup p50 0.425 ms with attribute-filter p50 2.974 ms on the canonical corpus (gate 6’s tripwires are 0.75 ms and 6 ms). Nothing in this section is a projection any more except what a sentence explicitly labels as one — the zstd reference numbers; the recorded runs in benchmarks/ are the published claims.

---

Format v6 (historical — the migration source)

Status: historical. This is the layout every release before v0.24.0 wrote. No shipped build writes it any more, and the only code that reads it is the migrator’s FROZEN v6 decoder (src/migration.rs, copied from src/segment.rs at 5f23172) — this section is that decoder’s reference, kept byte-precise for exactly that reason. Two parts of it remain live beyond the migrator: the attribute index and content index tables below are carried into v7 unchanged and stay normative there, and the versioning/policy subsections at the end apply to the format as a whole.

A v6 segment is one file named segment-<20-digit id>.seg, written temp + fsync + atomic rename. It is a fixed 104-byte header followed by five contiguous sections, in this order, with no gaps and nothing trailing:

[ 104-byte header ]
[ records         ]  encoded records, ascending timestamp order
[ record offsets  ]  u64 per record, relative to the record region
[ trace index     ]  trace id -> record offsets
[ attribute index ]  (key id, value digest) -> record offsets
[ content index   ]  word filters per 128-record block; runs to EOF

Header fields, little-endian, at fixed byte offsets:

Offset	Size	Field
0	8	magic, TRAZASEG
8	2	format version (6)
10	2	header length (104)
12	4	reserved, zero
16	8	record count
24	8	records offset
32	8	records length
40	8	record-offset index offset
48	8	record-offset index length
56	8	trace index offset
64	8	trace index length
72	8	attribute index offset
80	8	minimum record timestamp
88	8	maximum record timestamp
96	8	content index offset

Neither the attribute index nor the content index stores its own length. The attribute index is bounded by where the content index begins; the content index runs to EOF. The reader rejects a segment whose sections are not contiguous from the end of the header, whose record-offset index is not exactly record_count * 8 bytes, or whose sections exceed the file.

Each record is: timestamp u64, trace-id length u32, attribute count u32, payload length u32, reserved u32, then the trace id, then that many length-prefixed key/value pairs, then the opaque payload. Records carry attribute value text; the index does not.

Record order is an invariant, not a convention

Records are stored in ascending timestamp order, and encode_with establishes that itself — it stable-sorts by timestamp rather than trusting the caller. The sort is by timestamp alone, so a caller’s finer tie-break survives it and already-ordered input encodes to byte-identical output.

The enforcement is load-bearing because Segment::ordinal_range_for_window binary-searches the record region to turn a time window into a contiguous ordinal range, which is what lets a windowed query decode a slice instead of the whole segment. A binary search over unordered records does not fail loudly; it returns the wrong records. Every writer in the store already sorted before encoding, so nothing changed for them — what changed is that a future one cannot silently break the search. Pinned by records_are_stored_in_ascending_timestamp_order_whatever_order_they_arrive_in in tests/segment_format_acceptance.rs, which encodes a shuffled corpus and cross-checks the range search against a brute-force filter at every bound.

The attribute index

u32                     key count
  u32 + bytes           each attribute key name, length-prefixed
u32                     entry count
  u32                   key id (index into the dictionary above)
  16 bytes              digest of (key, value), see src/hash.rs
  u32                   posting count
  u64 × posting count   record offsets, ascending

Entries are written in (key id, digest) order, and the dictionary in sorted key order, so encoding the same records twice produces identical bytes.

Keys are interned because they are a schema — tens of them, repeated on every span, and an operator needs their names to read a cost report. Values are replaced by a digest because they are data: unbounded in size, and for LLM traffic they are the corpus. Through v3 the index stored value text, and a store of prompts therefore held every prompt in RAM for the life of the segment. See capacity for what that cost.

A digest probe returns candidates, not matches. Every caller must check a decoded record against the filter before returning it; Segment::attribute_candidate_offsets is named to make that unmissable, and tests/segment_format_acceptance.rs forges a collision to prove the check is load-bearing. A 128-bit digest will not collide naturally, which is exactly why the safety argument cannot rest on a passing query test.

The content index

Answers “which spans mention this word” without storing the words.

u32   reserved, zero
u32   records per block (128)
u32   block count (0 means: no content index, see below)
u32   hashes per token (4)
u64   summary filter size in bits
u64   block filter size in bits
[ summary filter    ]  one Bloom over every token in the segment
[ bit-sliced blocks ]  one ROW per bit position, one BIT per block

Records are grouped into blocks of 128. Each block has a Bloom filter over the distinct tokens of its records’ text, and those filters are stored transposed: row p holds bit p of every block’s filter.

That transposition is the whole design. Stored per block, testing one word against a segment means reading every block’s entire bitmap — hundreds of kilobytes to look at a handful of bits. Stored as rows, the same test is one read of block_count bits, so a two-word query reads tens of bytes per segment. Only the summary filter is held resident, capped at 32 KiB, which is why the content index’s memory cost scales with segment count and not with how much text a store holds.

Tokens are maximal runs of ASCII alphanumerics, lowercased, truncated at 40 bytes; the tokenizer and the bit derivation are format constants pinned by test in src/content.rs.

A block filter admits candidates, never answers. Bloom filters have false positives, and a block is 128 records wide, so most admitted records will not match; content::Query::matches against the decoded span decides. This also fixes the feature’s semantics: content search is WORD search, because a word index cannot soundly over-approximate a substring search. Searching refund against a span reading refunds were issued would be skipped by the filter while a substring match would have returned it — a wrong answer, not a slow one. See src/content.rs for the full argument.

block_count = 0 means the index is absent, not empty. A segment whose records carry no indexable text, or one written with --no-content-index, must be SCANNED by a content query. Reading absence as “holds nothing” would make content search silently return no rows. Note that the section is always present: “indexed nothing” is stated inside it rather than by its absence, which is what lets the header field bound the attribute index unconditionally.

Segment::open reads only the header, the three index sections, and the content index’s prologue and summary filter into memory; record payloads and the bit-sliced block rows stay on disk and are read by exact byte range on demand. That is what makes stores larger than RAM serveable.

Versioning

There is exactly one readable version. A file declaring any other is refused, including a JSONL segment from 0.3.x, which carries no magic at all.

It was not always one. The format grew by appending header fields behind if version >= N gates, and each gate turned a field into an Option that every reader downstream had to treat as “unknown, therefore assume the worst”: a segment whose timestamp range could not be read had to be scanned by every time-bounded query, and a second attribute-index decoder existed solely to read the encoding that predated digests. Those branches were removed and the header fields became plain values, so the pruning path no longer carries a case where it cannot prune.

Versions 1 through 7 are spent. 1 was JSONL. 2 was written by v0.16 and v0.17, 3 by v0.18 and v0.19. 4 and 5 were never released — they existed only on unreleased main, so no tag writes them and no tag reads them. 6 was written by every release after v0.19 and before v0.24.0, and 7 by the v0.24.x line. None of the seven is written now, and none opens for serving: 2 through 5 are refused outright, and a 6 or a 7 is read exactly once — by the migrator, which converts it before anything is served. The README’s pre-1.0 terms permit an on-disk break between 0.x versions, and the 2-through-5 cut was one: Store::open refuses such a segment and names it, never advising deletion.

Migrating between formats

Take a backup by one of the two procedures in the durability guide: stop the server and copy the directory, or take a filesystem snapshot that is atomic across the whole directory. Copying a live directory file by file is not safe — an in-flight flush can change the segment set between files — and that guide is the single source of truth for it. Then read the copy with the build that wrote it.

A span export is a reasonable way to move a dataset, but it is not a migration of the store:

Part of the store	In GET /v1/export?
Spans	yes — every one, as of the instant the export began. It pins a snapshot, and that snapshot copies the write buffer, so spans not yet sealed into a segment are included
Offloaded attribute values	no — left as {"$payload": "sha256/…"} references; the bytes stay in payloads/
Annotations	no — a separate surface (/v1/annotations) the export does not touch

The design document gives the underlying reason: a span export “cannot pin [annotations and payload bytes] at all” — there is no consistent point across the store’s independent recovery domains for it to pin. Closing that is what the generation/checkpoint boundary is designed for.

So export-and-reingest is a complete migration only for a store with no offloaded payloads and no annotations. Otherwise: back up, and read the backup with a build that can read it.

Which build? For a v6 or v7 store: none — no manual step exists. Any current build migrates both automatically at first open, per the Migration section of the v8 spec above; back up first, open, done.

For anything older, not “the release that wrote this segment”. A store accumulates segments in whichever format was current when each was sealed, so one directory can hold several formats at once, and a release that reads the oldest of them cannot read the newest. Formats 4 and 5 were never tagged at all.

One commit reads every indexed format this project retired WITHOUT a migrator — 2 through 5 — and that is what the error names: cf40bea (MIN_READABLE_VERSION 2, VERSION 5), exposed as traza::LEGACY_SEGMENT_READER. Build it, point it at the backup, and export from there. Format 1 was JSONL; it is refused separately and needs 0.3.x.

The policy from here

This cut is a one-time exception, taken while the store holds no data anyone depends on. “Every layout change makes all prior files unreadable” is not a policy — it treats stored telemetry as disposable, and a datastore does not get to do that.

The rule from v6 onward is three planes, kept apart on purpose:

1. Runtime reads exactly one canonical format. No if version >= N, no Option field standing in for “unknown”, no compatibility branch in the query path. That is what this change bought, and it is the part to preserve.
2. Version numbers stay monotonic. An identifier written by a release is never reused for a different layout.
3. A format bump ships with a migrator — an explicit, resumable conversion from the previous format into the current one, run offline or at startup, never woven into the read path. Reading an old format is code that has to exist somewhere; the win is that it lives there rather than in every query.

Point 3 is the part the v6 cut did not pay for, and that was worth stating rather than hiding: a migrator from v2/v3/v5 would have meant resurrecting precisely the decoders just deleted, to serve stores that did not exist. The debt was declined once, on the last occasion it could be declined cheaply — and v6 is where it started being paid: the v6 → v7 bump shipped with exactly the migrator point 3 demands, and the v7 → v8 bump pays it again, both decoders frozen inside one migrator (the Migration section above is its contract).

---

Original design proposal (v0.3, historical)

Everything below is the proposal as written before implementation. It is kept for the reasoning behind the design, not as a description of the format. Where it disagrees with the sections above — notably the trailing-footer arrangement, the JSONL payload, and readable v1 segments — the sections above are correct.

Problem (measured)

The engine keeps every span materialized in memory (Segment { spans: Vec<Span> }, ~450 B and ~10 heap allocations per span) and answers every query by walking all of them. Measured on the bundled benchmark:

Metric	1M spans	10M spans
Trace lookup p50	14.2 ms	145.6 ms
Attribute filter p50	66.5 ms	4,395 ms
Server RSS	~0.5 GB	~5 GB

Two independent causes: no index (every query touches all N spans) and a pointer-chasing row layout (per-span visit cost rises ~7x once the corpus outgrows the CPU caches). Both are fixed by this format.

Design

A v2 segment is ONE file, written with the existing temp + fsync + atomic rename discipline, laid out as:

[ JSONL span payload, byte-identical to v1 ]
[ index block ]
[ fixed-size footer: index offset, index length, span count,
  min_start_time_ns, max_start_time_ns, format version, magic ]

The index block contains, all offsets relative to payload start:

1. Trace index: sorted (trace_id, line byte offset, line length) entries — binary-searchable; a trace’s spans are found without touching other lines.
2. Posting lists: for service, name, and each attribute key/value pair: the sorted list of line offsets whose span matches. Dictionary of strings once per segment.
3. Time bounds are in the footer for whole-segment skipping.

Encoding: length-prefixed strings and little-endian u64s — no new dependencies (serde_json for the payload lines only).

Memory rule

Open no longer materializes spans. A loaded segment holds:

• the payload bytes (mmap if straightforward with std-only constraints, otherwise a Vec<u8> read once), and
• the parsed index (small: a few percent of payload).

Span structs are parsed on demand from payload lines, only for spans a query actually returns. The write buffer (unflushed spans) is unchanged.

Read paths

• get_trace: binary-search the trace index in every non-skipped segment, parse only that trace’s lines. Plus the write buffer as today.
• query: intersect posting lists for indexed predicates (service, name, attributes) and time-skip segments via footer bounds; parse candidates and re-verify EVERY predicate against the parsed span (an index accelerates a filter, it never changes its semantics). Predicates with no usable index (e.g., min_duration alone) fall back to a full scan of the contiguous payload bytes — never to materialized structs.
• stats, persisted_segment_spans: derive from footers/payload parse.

Compatibility

• v1 segments (no magic in the trailing footer position) remain readable via the legacy path: parse the JSONL fully at open, build the same index structures in memory, then treat identically. First compaction that rewrites a v1 segment writes v2.
• The lock discipline (writer before segments), crash recovery (orphan temp removal, duplicate-span healing at open), TTL semantics, and the wire contract are unchanged. Duplicate healing must work across v1/v2.

Acceptance criteria

Blocking, each with an executable oracle:

1. ./ci.sh green: fmt, clippy, full test suite.
2. All existing storage and server_on_engine tests pass unmodified (they pin crash recovery, lock discipline, healing, and the wire contract).
3. New tests prove: a trace lookup parses only the target trace’s lines (instrumentable via a parse counter or by construction); v1 segments load and serve correctly; a v2 file survives kill-and-reopen; index-served results equal full-scan results on a randomized corpus (extend the existing naive-reference test to cover the indexed path).
4. Memory: after open of a flushed corpus, resident span structs are zero — verifiable by API (no Vec<Span> in Segment) and a stats/heap assertion.
5. cargo run --release --bin bench (1M canonical) regenerated: all three gates PASS with trace-by-id p95 under 5 ms.

Advisory (report, do not gate on machine-dependent numbers):

6. TRAZA_BENCH_SPANS=10000000 run: trace lookup p50 < 1 ms, attribute filter p50 < 100 ms, server RSS under 1.5 GB. Record actuals in BENCHMARKS-10M.md style notes (do not overwrite canonical-corpus.md).

Non-goals

Columnar payload encoding, compression, OTLP, auth, replication — all stay on the roadmap. The JSONL payload stays byte-identical to v1 precisely so this change is index + residency only.