Testing

The standard: a test must be shown to fail

A test that has never failed is a claim, not evidence.

Before a test is considered to guard a behaviour, break the behaviour and watch the test go red. Revert, watch it go green. If it stays green against broken code, it does not guard anything — no matter how thorough its assertions look.

This is not aspirational. Several files in tests/ were written or rewritten specifically because they passed against broken code:

• tests/segment_format_acceptance.rs built its own “deliberately independent” byte fixture. Because that fixture was only ever compared against itself and never fed to the real reader, it drifted into a layout Traza has never written — a u32 header length at offset 12, three sections instead of four, a .trz2 extension — and passed continuously while asserting nothing about the engine. Feeding it to Segment::from_bytes failed with Corrupt("invalid v2 header length"). The same blind spot hid a magic-bytes disagreement: the encoder wrote TRAZAV2 while the test and the format doc expected TRAZASEG, and the two never met. The file now parses bytes from the real segment::encode, round-trips them through the real reader, and pins the magic to the real constant so they cannot drift again.

• The same rewrite removed a reopen_persistence case that wrote bytes and read them back asserting equality — a test of fs, not of Traza.

• reads_never_miss_committed_spans does not catch the visibility regression an unlocked seal can introduce, and this was verified rather than assumed. Its writer is single-threaded and flush() is synchronous end to end, so the window never opens. When sealing moved off the writer lock, the drain was deliberately changed to remove the spans instead of copying them — the exact v0.17 bug — and the entire pre-existing suite passed, all 140 tests, including the SIGKILL durability set. Only the new tests/seal_concurrency.rs::reads_see_every_acknowledged_span_while_a_seal_is_in_flight failed, immediately and on every run. A correctness suite cannot see a transient visibility hole unless something is reading during the hole.

• tests/seal_concurrency.rs::a_seal_holds_an_engine_lock_for_a_small_fraction_of_its_work is the shape to copy when asserting that work moved. Query results are identical whether or not the seal holds the writer lock, so no assertion over data can distinguish them; the test reads traza_segment_seal_locked against traza_segment_seal instead. Putting the write back under the lock takes that ratio from ~0.05% to ~44%. If you are claiming a performance property, assert it through an observable — a metric, a counter, a lock-hold measurement — never through query results.

• The same test is also the cautionary tale about what an observable MEASURES. traza_segment_seal_locked originally covered every critical section of a seal, including the post-publish reconcile — and the reconcile truncates the write-ahead log under the writer lock, so it is an fsync: ~12 ms against a ~4 us drain. It set the ratio almost single-handedly, parking it at roughly 23% against a 25% threshold. The test consequently failed whenever the machine was busy and would have PASSED with the segment write back under the lock, which is the one thing it existed to catch. The fix was not a looser threshold but a truer metric: the reconcile moved to traza_segment_seal_reconcile, and the ratio the test asserts on became ~0.05% with a 1% threshold. A metric that aggregates two costs with different shapes cannot bound either of them. If an assertion sits close to its threshold, suspect the measurement before the tolerance.

• tests/fold_concurrency.rs::an_analytics_fold_does_not_hold_the_segments_lock is the third variation, and the trap is subtler again. The obvious test for “a reader must not hold the segments lock” is to time a flush() racing a fold and compare the two. It flakes, because a seal’s wall clock is dominated by its segment write and its write-ahead-log fsync — neither of which has anything to do with that lock — so on a busy machine the comparison measures the storage device and blames the fold. The fix was to assert on traza_segments_lock_wait, which times exactly the thing under test and nothing else. Assert on the quantity itself, not on a proxy that merely contains it.

The lesson generalizes. When you add a test:

• Mutate the code it guards, not the test. Flip a comparison, drop a lock, skip an fsync, return early. Confirm red.
• Beware self-consistent fixtures. If your expected value and your actual value are computed the same way, you are testing arithmetic. Feed fixtures through the real reader.
• Concurrency bugs need concurrent tests. Invariants 2, 6, and 9 in invariants all have failure modes a sequential test cannot reach. A synchronous flush() never opens the window a real flush opens.
• Prefer a real oracle. SIGKILL for durability. An independent encoder for a wire format. Opening the data directory with a second Store for “the server owns no storage”.

Layout

Unit tests live inline in the module they cover, in the usual #[cfg(test)] style — currently in auth, media, metrics, payload, seed, semconv, ui, and wal.

Integration tests live in tests/, one file per concern. Files that exercise the HTTP contract spawn the real traza-server binary and drive it over a socket; files that exercise the engine use traza::Store directly.

File	Tests	Covers
storage.rs	42	The engine end to end: persistence, crash recovery, filter correctness, ordering, TTL compaction (including that expiry reaches the write-ahead log, so an expired span cannot return across a restart), log-corruption refusal, the flush bounds that keep the log finite, pinned snapshot views, the retryability of a failed expiry under injected I/O faults, concurrency, second-open rejection (under any spelling of the directory), stale-lock reclamation — a dead or impossible PID, and our own PID left by an earlier process, as a restarted container’s PID 1 finds it
scenarios.rs	14	Derived views over the deliberately messy seed corpus — three attribute dialects, agent trees, multimodal payloads, linked retries, fan-out. Guards rollups that are right on a tidy fixture and wrong on real data
keepalive.rs	14	Persistent connections. Not “is it faster” but “does the server ever leave bytes on a socket it intends to reuse” — request smuggling is the risk. Asserts the Connection header and the behaviour, because a response that says keep-alive while the server closes desynchronizes any client that believes it. The answers given before the auth gate (probes, dashboard shell and assets, the missing-build 404) each consume exactly their own request, a declared body included, one at a time and pipelined; a HEAD response carries no body
payloads_annotations.rs	10	Payload offloading, content addressing, annotations, and streaming export
ingest_hardening.rs	10	Adversarial input: lying Content-Length, oversized headers, silent peers, degenerate ids at the library boundary, query-parameter extremes
query_deadline.rs	12	The --query-deadline-ms compute budget: an over-budget scan answers 400 and is counted, the default never touches a legitimate query, 0 disables, the fold, session, and trace-lookup paths enforce the same budget, as do a rollup rebuilt for a segment with no sidecar (on the payload probe and the analytics fold) and the sweep of buffered spans on every query path, MCP surfaces exhaustion as a tool error carrying the same narrowing guidance from every tool (proven on one that leans on the shared conversion), an export under a 1 ms budget still streams to its complete trailer, and TRAZA_TEST_PANIC=0 reads as disabled
panic_guard.rs	2	The connection-slot guard: three handler panics on a two-slot server leave it serving, with the slots released and the panics counted; the panic route exists only under the TRAZA_TEST_PANIC startup latch
generations.rs	6	The generation boundary from outside the engine: an existing directory adopted at first open with its data intact and idempotently on the second, a checkpoint a reopen resolves through, a published deletion that stays deleted, and backup-by-pin then restore round-tripping spans, annotations and payload bytes together. Its sharpest test reconstructs the crash the stamp exists for — the log’s pre-checkpoint bytes written back over a committed generation — and was shown to fail when the fold rule is removed
durability.rs	12	The acknowledgement contract, proven by SIGKILL — no unwinding, no destructors, no flush on the way out. Each mode held to exactly what it claims, including that buffered is verified lossy rather than accidentally durable. Two kills are aimed at a specific window rather than a stopwatch: one lands inside a seal, one inside a grouped compaction merge, waiting for the journal on disk that says one is in flight. A third aims at the checkpoint’s commit point through three named on-disk signals — the manifest directory appearing, .CURRENT.tmp existing between the staged write and its rename, and CURRENT changing — because those windows are single fsyncs wide. The staged-CURRENT signal was added after a slower CI runner reached a window a fast machine never did: aiming at a state beats depending on the machine being slow enough to stumble into it
analytics.rs	18	Sessions and LLM aggregation across the write buffer, sealed segments, window boundaries, and reopen; numeric strings, explicit totals and saturating overflow; the rollup cache and its sidecars (surviving a compaction supersede, rejected when bound to another segment or untrustworthy, never outliving their segment); partial expiry reflected in the aggregates, segments ruled out by expiry without decoding, sweeps that do not fill the rollup cache, and a sparse rewrite costing one probe rather than a walk
object_storage.rs	18	The archive (feature object-storage only): local-store oracle equality (LWW across segments, tenants, sessions, cursors, content, payloads), manifest-last and read-back-verified publication ordering, partial-upload invisibility and quiescent-acknowledged cleanup, corruption/truncation/malformed-manifest refusal, manifest↔generation binding against omission/substitution, one publication winner per name, permanent tombstones with cross-client never-reuse and interrupted-delete resume, swept-publisher abort, identity and expected-digest guards, cache bounds and cache-tamper refusal, fresh-reader-from-remote-only, foreign-async-runtime safety, budget expiry on every bounded path, whole-domain restore (spans, annotations, payloads, evals, settled tombstones, exact ids), mid-restore target races, bounded timeouts, and positive range-read I/O savings
object_storage_s3.rs	1	The same round trip against a REAL S3-compatible service; #[ignore]d, run explicitly with TRAZA_TEST_S3_ENDPOINT/TRAZA_TEST_S3_BUCKET set — invoked without them it FAILS rather than silently passing
tenancy.rs	16	Tenant identity in the key: scoped reads and fences, per-tenant expiry, scoped payload fetch under pending erasures, and the bound probe’s sidecar heal — a sidecar-less store pays one rebuild, writes the sidecar back, and a fresh process pays none
compaction.rs	18	Size-tiered compaction bounding segment count without changing a single answer. Every test checks the data as well as the count — losing the newest version of a re-ingested span would be far worse than the slow search it fixes. Three are concurrent: reads and ingest must not wait out a merge, a flush landing mid-merge must still supersede merged content, and a store that never stops ingesting must still get compacted — compaction that declines whenever a seal is in flight is starved by its own correctness guard
server_on_engine.rs	8	The real server over its real wire contract, plus opening its data directory with traza::Store and comparing — the oracle for “the server has no private store”
segment_format_acceptance.rs	28	The on-disk format, hand-parsed at fixed offsets from bytes the real encoder produced, round-tripped through the real reader — v8’s section wrappers, varint bodies and content page-checksum table included. The forgery tests recompute the checksums they can, so the record-anchored checks behind the integrity layer stay covered; the hardening tests drive malformed geometry, varints, and decompress bombs at the reader; a post-open content-row mutation must surface as Corrupt at the query that reads it. Each test emits a JSON evidence record
migration.rs	13	Acceptance gate 4: v6 stores (downgraded fixtures plus committed pr50 bytes) and v7 stores (the committed fixtures/v8-migration baseline bytes) migrating to v8 with identical answers, same names, verified manifests and pins, SIGKILL resume on the crash matrix, deterministic mid-migration resume — and the completeness gate: internally consistent shortened legacy offset indexes (first/middle/last omitted, offset into a record, trailing record bytes, live and pinned) refuse by name rather than migrate a subset, while damaged legacy QUERY indexes migrate cleanly, rebuilt from the record stream
index_integrity.rs	4	The v8 corruption contract at store level: the reproduced v7 defect (one flipped trace-index byte reopening into a silent empty get_trace), a mutation sweep over every metadata section and every header byte, and content-row damage — each must refuse naming the corruption or serve the span, never answer clean-and-empty. Shown red with the wrapper/header checksum comparisons disabled
auth.rs	4	The bearer-auth matrix at process level: loopback open by default; 401/403/200 across ingest, OTLP, and flush
dashboard.rs	4	The SPA served from disk, asset content types, path-traversal refusal, shell loading without credentials while the API stays gated, and a missing build degrading to a helpful 404
openllmetry_conformance.rs	4	Traceloop conventions through both ingest surfaces landing in sessions and rollups with no client-side renaming. The regression guard for the gap where an OpenLLMetry app stored fine but registered zero tokens, cost, calls, or sessions
otlp_protobuf.rs	4	Binary protobuf against the real server, using the test’s own protobuf encoder — an independent implementation, so agreement is evidence about the format rather than a shared bug — plus one logical request sent as OTLP JSON and as protobuf coming out as identical spans, and span links round-tripping on the native path
otlp_conformance.rs	2	OTLP/HTTP JSON driving the real binary end to end
mcp.rs	42	The MCP surface. The in-process half drives mcp::Server against a real Store with no listener anywhere in the process — which is the assertion that tool handlers call the engine rather than looping back through HTTP. The rest spawns the real binary for the transport rules: origin refusal, protocol-version refusal, 202 for a notification, 405 for GET, and a ro token reaching read tools that the REST method rule would refuse outright
credentials.rs	11	Runtime credentials: bootstrapping from open mode (the first must be an unbound admin), a request admitted in open mode authenticated again when its body lands after the first credential, default deny under /v1/admin/, backups needing admin, revocation surviving kill -9, caller-supplied tokens, expiry, a non-loopback start accepting a credential log as configured auth, cross-site minting refused in open mode, and backups and restores never carrying credentials.jsonl
quotas.rs	9	Tenant limits and usage: limits documents replaced whole and surviving a crash, monthly span and byte quotas refusing a whole batch with 429 and Retry-After (bytes charged as sent, before offload, and each survivor of an erasure-suppressed duplicate charged its own size), the ingest pause never blocking reads, exports or erasure, usage scoped by binding and written every five seconds, and the runtime TTL override at store level
admission_caps.rs	5	The admission caps refusing a whole batch with 400 naming the cap and the span, the defaults applying with no flags, 0 disabling a cap, the span byte cap measured after offload, and a bad cap value refusing startup
gzip_ingest.rs	5	Content-Encoding: gzip on both ingest routes, the decompressed-size cap (413), malformed members (400), and 415 for any other coding or route
health_whoami.rs	4	/healthz and /readyz answered before the auth gate, and /v1/whoami for open mode, unbound and bound credentials
guard.rs	12	Run fuses against the real server: each limit (cost, steps, repeats, error streak) stopping the run that reaches it, a warn at 80% announced and counted once, a retried batch changing no count — the error streak included — a batch moving cost from one span to another stopping nothing, live or after a restart, a stop that sticks through later spans until reset, a reset that cannot refill a spent budget and a grant that raises it, the kill switch, a hypothetical check that trips nothing, tenant and bound-token isolation (a ro token may check, not trip), the event feed’s cursor, long-poll wake-up and gap across a restart, the Traza-Fuse header on OTLP JSON and protobuf with a hostile run id unable to forge a header, a crash restart rebuilding usage and keeping kill switches and policies, --no-fuses (MCP included), MCP check_run_guard, the three route paths whose prefix and suffix overlap answering 404, and an over-long run id refused (the Claude Code hooks are claude_code_hooks.rs’s). Each property was shown red by breaking the code that provides it; the bounds, parking, erasure and rebuild races, batch ordering and listing index are proven by unit tests in src/guard.rs, and a batch’s live verdict agreeing with a replay of the store — over random batches of upserts, for every limit but duration and the error streak — by unit tests in src/guard/state.rs
replay.rs	14	Tapes across recording conventions, bounded payload resolution, the OpenAI and Anthropic SDK request shapes replayed with zero model calls (streamed and not, x-api-key included), strict divergence refused without consuming the step, an over-long request model unable to amplify a stream, answer text that reads as JSON served as text, an uncaptured tool result refused rather than served as null, system instructions kept beside offloaded messages, tools/next, tenant isolation, per-tenant replay shares, erasure withdrawing a replay, and the MCP run tools over HTTP
diff.rs	3	Run diffing: identical runs align with no divergence, each divergence kind is found where it happens, and malformed or missing runs are refused
recall.rs	4	Recall ranking (successes first, then matches, then recency), the outcome filter, skeleton folding and its cap, and refusals that stay inside a tenant
ledger.rs	11	The ledger chain growing with checkpoints and every head recomputing independently, tampering named by seq, anchors accepted only for the head they name, erasure entries carrying a salted digest never the subject, evidence packs (usage row, runtime retention and the age of the chain walk included) with bound tenants seeing only themselves, a bound tenant’s pack on a long chain stopping at the query deadline and resuming its walk, pins carrying the chain, the offline verify --ledger exit codes (an earlier erasure not standing in for a later one a restore undid, a settled erasure missing its entry), and the guide’s Python scripts run verbatim. A pack’s per-phase budgets, its indexes by time, the cross-check it keeps between requests, the file check it runs off the maintenance lock, unfinished work surviving the 60-second reuse window, a new manifest taken in off that lock a budget at a time, expiry and a checkpoint landing mid-check not reported as damage, an append-only log’s digest surviving an append and checkpoint before every retry, and discrepancies past the per-kind cap counted rather than named are proven by unit tests in src/ledger.rs
flight_recorder.rs	3	The seams between those features: a quota refusal decided before the run guard sees a batch, an admitted batch both charged and judged, a runtime-minted credential driving a replay through x-api-key (and losing it on revocation), and every unbound-only surface refusing a bound credential with required named
llm_semantics.rs	3	Every query recipe in docs/llm-semantics.md, through both ingest paths, and OpenInference keys feeding the LLM rollups. The documentation’s executable oracle
fate.rs	7	The Fate Map over the behaviour corpus, whose outcomes are exact: every branch’s counts and Wilson interval, the fatal fork and hotspot, foresight for live runs (each verdict), a predictive fuse stopping a doomed run at the check and never on ingest, tenant scoping and bound credentials, bounds and refusals, erasure dropping cached maps, and MCP foresee_run inside the telemetry frame
steer.rs	15	Nudges delivered once through the pre-call check, a block vetoing one step (and its batch), nudges surviving a crash, bounded, tenant-scoped and erased with their run; the live list, its window and its stream; run topology; max_handoff_cycles by policy, flag, restart and reset; --no-fuses; and the MCP steering tools
claude_code_hooks.rs	7	The Claude Code hook pack run as Claude Code runs it: the exact JSON each hook prints and its exit code, fail-open versus TRAZA_FAIL_CLOSED=1, recall framed as untrusted telemetry, tenant isolation, no redirect followed with the token, and the installer’s merge, re-run and uninstall
import.rs	9	traza-server import claude-code and codex over the synthetic fixtures: runs, steps, tokens, costs and outcomes; idempotent re-imports; importing into a data directory; credential checks; --watch following appends and resuming from its state; and Agent Blame over HTTP, the CLI and MCP, scoped to the repository, per bound credential
xray.rs	8	The context X-Ray’s provenance, staleness, duplicates and compaction loss, the Cache Autopsy naming the volatile token and pricing it, offloaded prompts resolved, refusals, tenancy, the predecessor following the conversation, and MCP xray_step
radar.rs	6	The change radar end to end over OTLP (build attributes kept and held outside the caps), bound credentials and tenancy, refusals and the run cap, the query deadline, and MCP what_changed
tql.rs	8	Trajectory search naming the matched steps, filters and limits, parse errors naming the column, explain, the detector library and its findings, a bound credential seeing only its tenant, hostile queries and data staying bounded, and a heavy regex stopping at the deadline
fork.rs	12	Replay edits and live points, strict matching up to the edit, the fork listing diffed against the parent, golden runs and their un-marking, bounded divergence explanations, long runs rewound, and tenancy
cross_feature.rs	3	v0.28’s features on each other’s output: an imported Claude Code history answering the Fate Map, the X-Ray (each call’s whole conversation rebuilt from recorded deltas), the Cache Autopsy priced with cache rates, a TQL query and detector, Agent Blame, run topology and the change radar; and the live view’s foresight read from memory, per tenant, and off with the Fate Map cache

The client libraries carry suites of their own, in their own languages: sdk/python/tests (unittest) and sdk/typescript/test (node:test, run against the compiled package). Each spawns the real traza-server — $TRAZA_SERVER_BIN, else target/release/traza-server — once open and once with TRAZA_TOKENS, and drives every client method against its responses. What the real server never sends — redirects, HTML pages, truncated bodies — comes from a canned-answer stub beside each suite; see client libraries.

Note two rows. docs/llm-semantics.md is tested — if you change a recipe there, change the test. And tests/mcp.rs guards claims the MCP guide makes in prose: that a truncated result always says so, that stored span text never reaches a tool description or an error message, that the three time forms resolve to one window, and that failure shares are computed against the total rather than the page.

Running the suite

cargo test                     # everything
cargo test --test storage      # one integration file
cargo test --release --test durability   # process-level tests are faster in release

Tests that spawn the server need target/release/traza-server (or the debug equivalent) to exist; ./ci.sh builds it before testing.

Process-level tests bind --port 0 and read the actual port from the server’s stderr announcement, so they do not collide when run in parallel. Each uses its own temporary data directory, because a data directory has exactly one writer.

The gate

./ci.sh is the gate, and GitHub Actions runs exactly it on Linux and macOS (ci.yml) — the same script, no drift. It must be green before a change is finished. It runs, in order:

1. Source hygiene — no tracked source file may contain a literal NUL byte. A NUL makes git treat a file as binary: its diff disappears from review and grep and blame stop working on it. This actually happened, which is why it is a gate.
2. cargo fmt -- --check
3. cargo clippy --all-targets --all-features -- -D warnings
4. cargo build --release
5. cargo test, then cargo test --features object-storage
6. The script tests (scripts/test-*.py) and the Rust notices check (scripts/generate-rust-notices.py --check). Most cover the release tooling; test-demos.py covers the demos: their agents refuse a redirect rather than carry a token to wherever it points, and step 7 runs every demo, against the binary just built, from a path with a space in it
7. The demos — every directory under examples/ that has a run.sh, at its documented smoke setting, against the release binary (TRAZA_BIN is set, never inherited from the caller’s shell), each run through a symlink to the checkout whose path has a space in it; each asserts its own claims, so a change that breaks a surface a demo shows fails here
8. The client libraries: scripts/check-sdk-metadata.py (their versions match Cargo.toml, their LICENSE and NOTICE copies match the root’s), then their suites against the release binary. The Python suite always runs (python3 is already required); the TypeScript suite is skipped with a message when Node 20+ with npm is absent
9. cd ui && npm ci && npm audit --omit=dev --audit-level=high && npm test && npm run build

The dashboard is a first-class part of the product: a UI that does not build, or whose vitest suite fails, does not merge green. TRAZA_SKIP_UI=1 skips step 9 for a deliberate Rust-only run — not for a merge.

The benchmarks are not part of ci.sh; run them when a change could plausibly move performance. See benchmarking.