Capacity and performance

Provenance, stated per source rather than as a blanket claim.

• The 64 MiB index-memory matrix and the compaction transient come from index-memory.md and its raw index-memory.json, generated by the harness itself and carrying the commit it was built from.
• Ingest figures come from canonical-corpus.md and ingest.md.
• Several figures on this page have no committed record, and are marked where they appear: the 10M-span / 846 MiB row, the 256-1024 MiB scaling series, the offloaded-vs-inline comparison, the seed-corpus figure, and the per-key breakdowns. They were measured with the harness in this repository but their raw results were not committed at the time. Treat them as reported observations, not as reproducible records, until they are re-run and recorded.
• The 29 GB projection is an extrapolation, marked as such where it is used.
• The ingest figures predate the admission measurement pass. Ingest now serializes every span once more before admitting it, to enforce the admission caps and to charge usage bytes. The pass runs on the request’s own thread, outside every engine lock. The committed ingest records were measured without it and have not been re-run since. An interleaved before/after run on a heavily loaded host could not separate the two from run-to-run noise — which is not the same as showing the pass costs nothing.

Nothing is carried over from a version that no longer exists, and where a figure’s configuration is not recorded, this page says so rather than guessing.

Run the benchmarks on your own hardware. These were measured on one machine; the shapes generalize, the absolute numbers do not. See benchmarking.

Sources:

• canonical-corpus.md — rewritten by cargo run --release --bin bench, the 1,000,000-span canonical corpus
• ingest.md — rewritten by cargo run --release --bin ingest-bench, the ingest matrix
• CHANGELOG.md — the 10M and 100M scaling runs, recorded with the release that measured them

The canonical corpus

From canonical-corpus.md: 1,000,000 spans over the real HTTP path, on macOS/aarch64 with 10 hardware threads.

Metric	Measured	Project target	Result
Sustained batched HTTP ingest	65,749 spans/s	≥ 50,000 spans/s	PASS
Trace-by-id p95	1.280 ms	< 50 ms	PASS
Attribute-filtered query p95	5.443 ms	< 300 ms	PASS

Query	p50	p95	p99	samples
Trace by ID	0.577 ms	1.280 ms	2.094 ms	200
Attribute filter	3.657 ms	5.443 ms	6.468 ms	100

This is the v8-format record. Run-to-run spread on the shared benchmark laptop is comparable to the v7 → v8 movement in the tails — the paired before/after runs and their honesty notes are in storage-v8-comparison.md.

The ingest row measures the bench’s compacting configuration (fan-out 4), so compaction runs concurrently with the flood. ingest.md’s 82,805 spans/s single-client engine-limit row is the same span shape without that contention — quiet machine, compaction off, and a server rate whose client JSON encoding sits outside the timed loop, where this row’s sits inside it — so the two rates are not like-for-like beyond that. That corpus occupied 66,854,329 bytes on disk across 65 segments at measurement time — roughly 67 bytes per span for this benchmark’s span shape, on the v8 format (the same corpus measured ~125 bytes per span on v7). Your bytes-per-span depends entirely on your attribute volume, so treat it as a method rather than a constant: ingest a representative sample and read bytes_on_disk from GET /v1/stats.

Note that JSON generation sits inside the timed ingest loop, so the ingest rate includes client serialization and loopback HTTP overhead. It is a floor on what the server can do, not a ceiling.

Ingest throughput and concurrency

From ingest.md: 1,000,000 spans per run, 1,000 spans per batch, wal durability, median of 5 runs, on macOS/aarch64 (Apple M1 Max, 10 hardware threads). This matrix is the 2026-07 record on the v6-format build and has not been re-run since; the v8 canonical record above is the current single-configuration ingest measurement.

Scenario	Concurrency	Median spans/s
HTTP JSON, keep-alive, wal	1	82,805
HTTP JSON, keep-alive, wal	4	155,685
HTTP JSON, keep-alive, wal	8	193,188
HTTP JSON, keep-alive, wal	16	208,973
Direct engine (no HTTP), wal	8	193,637
Direct engine (no HTTP), buffered	8	313,201

Two readings matter more than the peak:

HTTP is not the constraint. The direct-engine number at concurrency 8 (193,637) and the HTTP JSON number at the same concurrency (193,188) are the same figure. Removing HTTP entirely — no socket, no parsing, no protocol — changes nothing. Whatever limits ingest is inside the engine.

The writer lock was the ceiling, and sealing was most of it. Sealing is now off the lock. The stage breakdown in ingest.md put the work performed while holding the writer lock at 88% of the run’s wall clock at the default --flush-spans 10,000 — and 74% of that was one thing: writing the segment. Encoding it, fsyncing it, renaming it and reopening it all happen on a private vector no other thread can reach, so none of it needed the lock. The seal now drains the buffer under a short lock, writes with nothing held, and publishes under a short lock.

Measured before and after with the two builds alternated round-robin on a loaded host (1-minute load average 8.9-12.8; read the ratio, not the level — see load conditions), 1M spans at concurrency 8, median of four rounds:

--profile (--flush-spans)	Before	After	Change
throughput (30,000)	162,763	222,683	+37%
balanced (10,000)	116,612	176,004	+51%
latency (5,000)	83,400	180,331	+116%

The operational consequence is that --flush-spans is no longer a throughput knob. It used to span a 2x range, because sealing more often meant stopping every ingesting thread more often; latency and balanced now measure within 3% of each other. Tune it for the tail latency and buffer memory you want, not to buy throughput. writer lock wait — the contention signal — fell 62% at balanced over the same 1,000 batches.

The lock is still busy; wal write is now what it is busy with. Sealing fell from 74% of in-lock work to 10-15%, and appending to the log rose to 78-85% of it. If ingest is your bottleneck now, the thing to look at is the log device, not the engine: traza_wal_write_ns_sum and traza_wal_fsync_ns_sum against wall clock, and the group-commit ratio. --wal-commit-window-us and faster storage are the levers there; a bigger --flush-spans no longer is.

Two things this changed that an operator should know:

• The write buffer can exceed --flush-spans while a seal is in flight, because ingest no longer waits for one. It is bounded: past four times the threshold an ingesting thread waits for the seal to publish. Size memory for that bound, not for the threshold.
• --flush-wal-bytes (64 MiB by default) now governs the log’s real size. A seal that empties the buffer still discards the whole log, so a quiet store behaves as before; under sustained ingest the log runs up to that bound between reclamations rather than being emptied on every seal. Restart replay is bounded by this setting. Lower it if you need faster restarts; the cost is reclaiming more often, and reclamation is the one part of a seal that still takes the writer lock.

Against the 250,000 spans/s target: --profile throughput at concurrency 16 measured a median of 250,453 spans/s (min 122,768, max 261,215), and a second run 261,782. The medians clear the target; the minima do not, and the spread is contention on a machine that never went idle. The target is therefore recorded as not yet confirmed — re-measure on an idle machine, not as met. What is established is that the shortfall was never a property of the design; most of it was one default.

Durability is the biggest single lever, which is exactly what the mode names promise: buffered at 313,201 against wal at 193,637 in the same direct-engine scenario.

Protobuf versus JSON. The controlled comparison now exists: holding the route fixed at /v1/traces, optimized OTLP protobuf decodes at 479 ns/span against optimized OTLP JSON at 1,275 ns/span — 2.7x faster — on payloads 2.9x smaller (117 vs 342 bytes/span). Protobuf was never slower as a wire format; it was slower as one implementation of one, and that implementation is fixed. This barely moves end-to-end throughput, because decode is ~1.9% of ingest cost and the writer lock reabsorbs the difference above concurrency 1. Prefer protobuf for wire size and CPU; use whichever your exporter speaks.

Memory

Payloads are not resident. Indexes are, and their cost tracks cardinality — how many distinct values a segment holds — not the size of those values. The second clause is new in segment format v4. Through v3 the attribute index was keyed on the value text itself, and for LLM traffic the index therefore was the corpus; that regime, and what it cost, is recorded below because operators running stores written before v4 still need to recognize it.

The payload half is true, structural, and asserted by the test suite. An open segment holds a file handle, its record-offset table and its two decoded indexes. Record payloads are read from disk by exact byte range on demand and are never retained — not even by the flush that just wrote them. Store::resident_persisted_span_structs() and Store::resident_payload_bytes() are both zero after open and after flush.

Note that resident_payload_bytes() deliberately excludes indexes, so it reports zero however large they get: it is not a memory-safety signal on its own. Store::resident_index_bytes() and Store::resident_attribute_index_entries() report the other half. Both are documented as approximate — exact allocator accounting needs a dependency this crate does not take.

The cost model

Since v4 the attribute index interns keys in a per-segment dictionary and stores a 128-bit digest in place of each value:

resident ≈ 20 bytes × distinct (key, value) pairs per segment  (index entries)
         + 8 bytes × spans × indexed attributes                (postings)
         + 8 bytes × spans                                     (record offsets)
         + the trace index                                     (still keyed on text)

Every term is now bounded by span count and attribute cardinality. None of them is bounded by attribute value size, which is the property that was missing. Measured end to end, an index entry costs roughly 150 bytes of RSS including postings, hash-table slack and allocator overhead.

Deduplication is still per segment. A value that recurs across segments is counted once per segment, so repetition within a segment is what saves an entry. This matters far less than it used to: an entry is 20 bytes rather than a prompt.

For what conventional tracing indexes — service, operation name, http.method, db.system, a deployment tier: tens to thousands of short distinct values — the postings floor was always the whole cost, and v4 does not change it. Measured under v3: 10,000,000 spans, six indexed attributes, between 1 and 50 distinct values each, 8.4 GiB on disk across 977 segments — 846 MiB RSS on open, with the per-key breakdown flat at ~80 MB, which is 10,000,000 × 8 bytes of postings and nothing else. Span size never entered into it, and under v4 it still does not.

The supersede prefilter adds one more per-segment term. Resolving last-write-wins consults a per-segment set holding the FNV-1a hash of every primary key the segment stores — eight bytes of payload per distinct key, roughly sixteen of RSS with hash-table slack. It is built lazily from the rollup sidecar the first time a query asks whether that segment could supersede a candidate, so an idle store pays nothing, and a queried store converges on every segment but the oldest holding its set for the segment’s lifetime.

What this cost, measured

One indexed text attribute, --flush-spans 10000, 256 MiB of attribute text ingested, all values distinct — the corpus shape that broke the v3 model. Same machine, same command, the commit before and after the change:

Cell		v3 (text-keyed)	v4 (digest-keyed)
2 KiB × 131,072	RSS on open	391 MiB	21.6 MiB
	Approx. resident index	279 MiB	16.5 MiB
	Attribute index on disk	262 MiB	6.0 MiB
	Segment bytes on disk	825 MiB	569 MiB
512 B × 524,288	RSS on open	439 MiB	77.5 MiB
	Approx. resident index	348 MiB	65.3 MiB
	Attribute index on disk	282 MiB	24.0 MiB
	Segment bytes on disk	997 MiB	740 MiB

Both cells hold the same 256 MiB of text. That the 512 B cell still costs 3.6× the 2 KiB cell is the model working as designed, not a residual defect: it holds four times as many spans, so four times as many index entries and postings. Cost tracks cardinality now, and cardinality is what differs.

These two cells are spot measurements, not a matrix. They were taken one run each at load average 18-22 on a machine that was not idle. RSS has proved insensitive to that load (three repeats of one cell previously landed within 0.2 MiB), and the effect here is 5-18×, far outside any plausible noise. But the rest of the matrix below — every 10%-distinct row, 8 KiB, 64 KiB, clustering, --payload-threshold-bytes — has not been re-measured since v4 and is marked accordingly.

The text-keyed index figures, kept as the baseline they were measured under

Every row in this section describes the superseded text-keyed attribute index. No readable segment uses it any more — the format collapsed to a single version and the decoder that read it was removed — so these costs describe nothing the store can produce today. They are kept because they are what the recorded figures elsewhere in this repository were measured against, and a comparison with no baseline is not a comparison.

One indexed text attribute, --flush-spans 10000, 256 MiB of text per row:

Value size	Distinct values	Segments	Approx. resident index	RSS on open
512 B	524,288 (100%)	52	348 MiB	439 MiB
512 B	52,428 (10%)	52	348 MiB	439 MiB
2 KiB	131,072 (100%)	13	279 MiB	391 MiB
2 KiB	13,107 (10%)	13	279 MiB	391 MiB
8 KiB	32,768 (100%)	4	262 MiB	448 MiB
8 KiB	3,276 (10%)	4	96 MiB	156 MiB
64 KiB	4,096 (100%)	1	257 MiB	580 MiB
64 KiB	409 (10%)	1	26 MiB	60 MiB

Under v3 this scaled linearly with text, which was the whole problem. 2 KiB values, all distinct: 256 / 512 / 1024 MiB of text cost 391 / 759 / 1498 MiB RSS, a least-squares fit of RSS ≈ 1.44 × (indexed text) + 22 MiB. Extrapolating it, 10M spans each carrying one 2 KiB prompt was 20 GB of text and roughly 29 GB of RSS — a store that would not open on a 32 GB machine. The same extrapolation against the v4 measurement above gives roughly 1.6 GB.

Two v3 mitigations are now largely obsolete, and are recorded here only so that existing tuning can be unwound rather than cargo-culted:

• Temporal clustering (confining repeats of a value to one segment) cut the 2 KiB / 10%-distinct cell from 391 MiB to 50 MiB. It still helps under v4, by the same per-segment-dedup mechanism, but the amount it can save is now 20 bytes per avoided entry.
• --payload-threshold-bytes, lowered to a few kilobytes, took prompt text out of the index entirely: 391 MiB → 81 MiB on the 2 KiB cell. It cost an fsynced file per distinct value, which took that run from 8 seconds to roughly 25 minutes, and an extra read to display a value. Under v4 this is no longer a memory tactic — the index costs the same either way — so set the threshold on the merits of on-disk size and record width alone.

Not indexing the attribute at all remains available: a user attribute whose key begins with a NUL byte is stored verbatim in the span and skipped by the index entirely.

Content search

?content=refund finds spans by the words in their text. It is served by a per-segment word filter (introduced in segment format v5), and its cost profile is worth understanding because it is very good in one regime and does nothing in another.

Re-measured on the v8 build (2026-09). cargo run --release --bin content-bench runs the same corpus twice — once with the index and once with --no-content-index — so the comparison is two measurements rather than one measurement and a memory. 100,000 spans carrying a prompt and a completion of 60 words each, 72.6 MiB of text, 50 segments, no compaction, on a loaded machine, paired against the identical run on the v7 build:

Query	Matches	v8 with index	v8 without	v7 with index
A word in one span	1	0.570 ms	484.873 ms	0.555 ms
Two words, one span	1	0.613 ms	513.076 ms	0.556 ms
A word in ~every span	97,777	394.502 ms	421.073 ms	412.495 ms
A word in no span	0	0.011 ms	490.379 ms	0.005 ms

The indexed store’s segment bytes fell 46.9 → 41.5 MiB from v7 to v8 on this corpus (index overhead over the no-index store: 0.7%), while the resident index measured 27.08 MiB on both builds — the v8 change is on-disk encoding, and it makes no RAM claim. Selective lookups stay sub-millisecond with v8’s per-page checksum verification on the row-read path. These short runs on a loaded machine do not establish performance equivalence.

The v5-era record, kept for the regime analysis below. 200,000 spans, 145 MiB of text, 100 segments, load average 7-10 — a larger corpus than the v8 run above, so the columns are not comparable across the two tables:

Query	Matches	With index	Without	Speedup
A word in one span	1	1.48 ms	1,258 ms	849x
Two words, one span	1	0.74 ms	1,156 ms	1,554x
A word in ~every span	195,526	1,156 ms	1,120 ms	1.0x
A word in no span	0	0.008 ms	1,163 ms	146,000x

Read the third row as carefully as the first. When nearly everything matches, the index cannot help and does not pretend to — it costs about 3% and saves nothing, because the work is decoding and returning 195,526 spans, not finding them. The index buys selectivity; a query with none gets none.

The fourth row is the common case in practice and the cheapest: a word no segment holds is rejected from the resident summary filters alone, without opening a single file.

Latency is flat in corpus size, which is the property that matters. The same cells at 50,000 spans measured 0.72 ms for the rare term against 285 ms scanning. Four times the corpus took the scan from 285 ms to 1,258 ms and the indexed query from 0.72 ms to 1.48 ms.

What it costs:

	50,000 spans	200,000 spans
Segment bytes added	+0.1%	+0.1%
Resident filters	0.05 MiB	0.20 MiB
Resident per segment	~2 KiB	~2 KiB

The resident cost scales with segment count, not with text, because only the per-segment summary filter stays in RAM — the per-block filters are read from disk a row at a time. It is capped at 32 KiB per segment, so even a store with a thousand segments cannot spend more than 32 MiB on it.

The ingest cost is not resolved by these runs and should not be quoted. The 50,000-span cell measured 0.80 s with the index against 0.62 s without; the 200,000-span cell measured 3.26 s against 3.33 s — the opposite sign. Both are single runs on a machine at load average 7-10, and the honest reading is that seal-time tokenization is somewhere between free and about 30%, which is too wide to publish. Re-measure on an idle machine before relying on it.

Two operational signals, both on /v1/metrics:

• traza_segments_pruned_by_content_total against traza_segments_examined_total — how much of the store the summary filters are eliminating. If this goes to zero, content search has degraded to a scan and results will look identical while latency does not.
• traza_records_admitted_by_content_total against the rows actually returned — the block filters’ real selectivity. A ratio that climbs means the filters are saturating.

--no-content-index turns the index off. Content search still works; it scans.

What v4 does not fix

The trace index is still keyed on trace-id text. It is now the largest text-keyed structure a segment holds. This is bounded — a trace id is 32 hex characters, not a prompt — so it scales with span count rather than with payload size, but it is a real term and it is why the 512 B cell’s floor is not lower than it is.

Compaction’s transient is untouched. A merge materializes its inputs, so its peak tracks the segment-size cap and not the index. Measured on the 2 KiB cell, peak RSS during merge was 1,587 MiB under v3 and 1,362 MiB under v4: the merge got slightly cheaper because the index it rewrites is smaller, and it is still three orders of magnitude above the steady state. Size a host for the merge, not for the steady state.

The recorded figures, and what is wrong with them

Corpus	Configuration	Peak RSS
10M spans (~6 GB on disk)	Not recorded alongside the figure	0.25 GB
100M spans (~55 GB on disk)	Compaction disabled	0.43 GB
100M spans	Default compaction, 256 MiB cap	2.0 GB
100M spans	1 GiB segment cap	6.7 GB

The 100M rows come from the 0.16.0 changelog entry and were sampled during the run at 20-second intervals.

Treat every row as unusable without its corpus’s attribute cardinality, which is the variable none of them records and the one that dominates. All of them were measured on the benchmark corpus in src/bin/bench.rs, whose spans carry five indexed attributes with at most 100 short distinct values each — the cheap regime described above, and not a shape any LLM workload has.

The 10M row does not survive its own arithmetic: five indexed attributes over 10M spans is 400 MB of postings before a single value byte, plus 80 MB of record offsets and a trace index for 1M traces. 0.25 GB is below that floor, so it cannot describe a fully opened 10M-span store. It needs re-measuring; the closest controlled figure here is the 846 MiB above.

For contrast, on a corpus with a realistic LLM shape — the project’s own seed generator, 641,805 spans and 4.9 GB on disk, 6% of the 10M corpus — RSS on open is 527 MB, already twice the recorded 10M figure. Its largest single line item is gen_ai.input.messages: 48,069 distinct values holding 141 MiB resident, more than every conventional attribute in the store combined.

Note the direction of the compaction rows: RSS rises sharply with compaction. A merge materializes its inputs, so the working set tracks the segment-size cap rather than the index size.

An earlier revision of this page gave that transient as “peaked at 721 MiB” and “1053 MiB”. Both were wrong, and understated. The benchmark took one RSS sample after compact_segments() returned — by which point the merge has freed its working set — so it measured the trough and published it as a peak. RSS is now sampled every 20 ms by a background thread for the duration of the merge. From index-memory.json, every configuration whose merge completed:

Steady-state open	Peak during merge	Settled after
10 MiB	1,204 MiB	788 MiB
14 MiB	1,271 MiB	796 MiB
116 MiB	1,292 MiB	928 MiB
116 MiB	1,426 MiB	1,008 MiB
116 MiB	1,601 MiB	1,102 MiB

A store serving in 10 MiB needs over a gigabyte to compact itself. Size a host for the merge, not for the steady state. The transient is not O(indexes) by anybody’s definition.

Read the individual figures as approximate. They were taken at load average 36-45 on a machine that was not idle, one run per configuration, and an earlier run of the same matrix produced peaks differing by up to 8% — those earlier figures are deliberately not quoted here, because the record that held them has been superseded and citing a number with no live record is the habit this page is trying to break. What is stable across both runs, and is the point, is the order of magnitude: a merge costs roughly a gigabyte and a half regardless of how small the store it is merging.

These 5 rows are the only compaction measurements on this page, and they come from a matrix in which 21 of 26 configurations merged nothing at all. That had two causes, and only one of them was a defect:

• A partial tail segment masked the qualifying run behind it, so a corpus that ended mid-segment declined to merge however many full segments it held. Fixed by #21, and verified after the merge: 45,000 spans (four full segments plus a 5,000-span tail) now merges 3 of 5 where the same shape previously merged nothing.
• The remaining non-merging cells held three full-size segments, below the default fanout of 4. Declining to merge those is the policy working. The matrix simply used cell sizes that did not reach the threshold.

So the table is still narrow — it describes merges on 512-byte-value corpora and nothing wider — but because of how the cells were sized, not because compaction refuses wider ones. Re-running the matrix with span counts that reach fanout is what closes this, and it has not been done yet.

Reproducing all of this

cargo run --release --bin index-mem-bench -- --matrix runs the corpus matrix; --help lists the axes. It reads process RSS from the OS rather than from resident_payload_bytes(), and it measures the reopen and compaction points in a fresh child process, because after an ingest the allocator is still holding the write buffer’s pages and the parent’s RSS answers a different question. Figures above were taken on an Apple M1 Max (10 cores, 32 GB) under a sustained background load average of 10-16; RSS proved insensitive to that, with three repeats of one cell landing within 0.2 MiB of each other. Ingest timings from those runs are not reported, because that load makes them meaningless.

Filtered search and compaction

A filtered query costs one index probe per segment, so its latency tracks the number of segments rather than the size of the corpus. That is what compaction exists to bound, and the effect is the largest single performance lever in the system.

At 10M spans, uncompacted against default compaction (this table and the 100M one below are single-run records from before format v8; they have not been re-run on v8, whose 1M-span canonical record is above):

Metric	Uncompacted	Default compaction
Attribute filter p50	14.8 ms	2.4 ms
Attribute filter p95	33.4 ms	4.1 ms
Attribute filter p99	220 ms	14.1 ms
Trace lookup p99	4.65 ms	2.28 ms

At 100M spans (~55 GB on disk), all three columns from the same harness, differing only in --compaction-fanout and --compaction-max-segment-bytes:

100M spans	Uncompacted	Default (256 MiB cap)	1 GiB cap
Attribute filter p50	155.5 ms	9.8 ms	2.3 ms
Attribute filter p95	747.3 ms	27.1 ms	9.3 ms
Attribute filter p99	1664.6 ms	72.9 ms	22.2 ms
Trace lookup p99	7.72 ms	1.82 ms	0.99 ms
Segments	~10,100 †	~380 †	~100–125 ‡
Peak RSS	0.43 GB	2.0 GB	6.7 GB ‡
Sustained ingest	59,025/s	40,894/s	31,267/s

† Extrapolated from mid-run samples, not measured directly; the benchmark deletes its data directory on exit. ‡ Sampled directly during the run at 20-second intervals. The segment count oscillates between about 97 and 125 over the last five minutes of ingest as merges create and retire segments, and peak RSS is the maximum of those samples — a shorter-lived merge spike between samples could exceed it.

Read honestly: compaction is worth roughly 16–28x on filtered search at the default cap, and raising the cap to 1 GiB is worth roughly another 3–4x on top. At a 1 GiB cap, historical filtered-search p99 is 22.2 ms at 100M spans, versus 72.9 ms at the 256 MiB default. These are not current-format acceptance results: the canonical gates apply to trace p95 < 50 ms and filter p95 < 300 ms.

That win is paid for in memory and ingest: peak RSS 2.0 → 6.7 GB, and sustained ingest a further 24% lower (40,894 → 31,267 spans/s). Raising the cap is the right trade for a large store that is read more than written, and the wrong one for a memory-constrained host. Both are one flag apart.

These are single-run measurements on one machine at 100M spans. Nothing above that size has been measured, and segment count still grows with the corpus, so the same tail returns at a large enough store. The structural answer remains a cross-segment attribute routing index, which is not built.

Trace lookup

Limited queries decode only the records they return, which is why trace lookup stays fast as the corpus grows: p99 2.094 ms at 1M on the v8 format (canonical-corpus.md), 2.28 ms at 10M and 1.82 ms at 100M under default compaction (both pre-v8 runs, below). It is not entirely scale-independent — the 100M uncompacted column shows 7.72 ms — because a lookup still probes each segment’s trace index, so it tracks segment count too, just far less steeply than a filtered search.

LLM aggregation, and what a restart costs

GET /v1/stats/llm and GET /v1/sessions are served from a per-segment rollup: the model, provider, service, day and session counters for everything that segment holds. Segments are immutable, so a rollup is computed once and then reused.

The rollup is held in memory AND persisted beside its segment as segment-<id>.rollup. The persisted copy is what a restart lives on. Process memory is empty after a restart, so with only the in-memory cache the first aggregation had to decode the whole corpus — measured at 3.8 s cold against 127 ms warm on a one-million-span store, a ratio that scales with bytes on disk rather than with what the query asked for. Reading the sidecars instead puts the same query at ~290 ms cold.

Two things follow for sizing:

• Budget the sidecars in absolute bytes — roughly 30 MB per million spans on the query-bench corpus — not as a stable share of the store. The dominant term is eight bytes per span for the supersede prefilter, plus the per-session trace sets, so sidecar bytes scale with span count and session/trace cardinality rather than with span size. Sidecars stay raw because erasure scans require literal bytes. In the paired 1M-span runs, v7 measured 27.7 MB of sidecars and 11.6% overhead, while v8 measured 29.8 MB and 19.5% overhead; see query.md. Here overhead means sidecar bytes divided by non-sidecar bytes, not their share of the total directory. Segment layouts differed between runs, so neither the sidecar totals nor the overhead are fixed sizing constants.
• Compaction no longer costs the aggregations a rebuild. A merge replaces several segments with one, which kills the inputs’ cached rollups and publishes an output that has none — so the next aggregation used to pay to re-establish what the merge had just taken away. Measured while compaction ran under sustained ingest, the worst single query was 2.6 s. The merge now hands its outputs’ rollups straight to the cache, under the same lock that publishes the segments, because it already had to build them to write their sidecars; the worst query is 210 ms and the median is indistinguishable from a settled store. TTL expiry does the same when it rewrites a partially-expired segment. Watch traza_segment_merges_total and traza_segments_merged_away_total to see compaction at all — segment count cannot show it, because seals raise it while merges lower it.
• A sidecar is derived and disposable. It is rebuilt on demand whenever it is absent, truncated, corrupt, bound to a different segment, or written under an older analytics schema. Deleting every .rollup in a data directory is safe; it costs one rebuild per segment.

Aggregations do not stall ingest. The fold pins the segment list — a pointer copy per segment — and releases the lock before it decodes anything, because a pinned Arc<Segment> keeps its own file descriptor and stays readable even if a merge or expiry unlinks it underneath. Holding that lock across the scan meant every analytics request stalled every writer for as long as it ran, and the Overview screen starts several at once. Watch traza_segments_lock_wait against traza_analytics_fold: a long fold beside a short wait is the intended shape.

The TTL sweep no longer re-reads the corpus. It runs once a minute over every segment and used to JSON-decode all of them to discover that nothing had aged out. The rollup sidecar’s prologue carries the segment’s span END range under its own checksum, so a segment wholly newer than the cutoff is skipped and one wholly older is retired without being read; only a segment straddling the cutoff is decoded. traza_expiry_segments_skipped_total against traza_expiry_segments_decoded_total is the sweep’s real cost — nothing about a correct sweep otherwise says whether it read one segment or all of them. A missing or damaged sidecar reads as “I do not know” and falls back to decoding, so retention is never decided on an unverified byte.

A maintenance timer does not fill the rollup cache. The payload sweep asks about every segment in the corpus, including ones no aggregation has ever touched. It now reads sidecars without caching them, and compact_expired skips the reference set entirely when the store has no payloads/ directory — which is most stores, and where the whole walk was in service of an empty answer. Store::cached_rollup_count() is the hook that says what is actually resident.

Time-windowed queries are the common dashboard case and they behave differently. A segment fully inside the window is answered from its rollup; a segment straddling the boundary is decoded, though only across the window’s own record range, which the reader locates by binary search over the ascending record timestamps. Concurrent ingest is what makes this matter: with several clients writing at once, segments interleave in time and no segment is fully inside any window, so every one of them takes the decoding path. cargo run --release --bin query-bench measures exactly that axis — TRAZA_QUERY_BENCH_THREADS sets the number of concurrent ingest clients.

Sizing guidance

There is no single “spans per node” answer, because the binding constraint moves with the workload. Work through it in this order:

1. Disk. Ingest a representative sample and measure the whole data directory, including WAL, payloads, sidecars, manifests, and pins. bytes_on_disk reports persisted segment bytes only. Budget headroom for superseded versions (they persist until compaction rewrites their segment) and for a merge’s output existing alongside its inputs. Upgrading a v6/v7 store: the first open under v0.25.0 migrates it to format v8 in place — automatically, and one-way from the first converted file — so take a backup first and budget the migration’s own transient disk (the live rewrite plus one full copy per pre-existing pin). The authoritative contract, including the rollback boundary, is the format document’s migration section.
2. Filtered-search latency. This is what degrades first as a store grows, and it tracks segment count. Watch segment_count and the --compaction-max-segment-bytes trade above.
3. Memory. Budget corpus-dependent resident indexes and caches, plus transient compaction and migration allocations. Choose a compaction cap that leaves room for both steady-state memory and a merge.
4. Ingest rate. Measure your workload with the intended durability, batching, compaction, and concurrency. Historical peak rates are not current capacity guarantees. Per-batch costs are paid once per request.
5. File descriptors. One per segment. Fine with compaction on; a real limit with it off. Rollup sidecars do not add to this — they are read once and closed, not held open.

What is not measured

Stated plainly, because absence of a number is not a claim of good behaviour:

• Anything above 100M spans on a single node.
• The 10M and 100M compaction tables and the memory matrices on format v8 — they were measured on earlier formats and have not been re-run; the 1M-span canonical record and the content-search pairing are the v8 measurements on this page.
• Query performance under concurrent read load — the query percentiles above were sampled without competing readers.
• Sustained mixed read/write workloads.
• Behaviour on spinning disks or network-attached storage; all measurements are on local SSD.
• Protobuf versus JSON ingest cost (being re-measured, see above).

If you need one of these, measure it. The benchmarks are in the repository precisely so you can.