Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[0.28.0] - 2026-10-18

The foresight release. 0.27.0 let the recording act on one run at a time; this release reads every run together. Traza now learns from every run an agent made which paths end well, and sees failure coming: the Fate Map counts each path an agent’s completed runs took and how often it ended in success, with an interval and the number of runs behind it, and foresight walks a live run’s path down that map, so a run that has started down a path that has mostly failed is called likely_failure before it has failed. A predictive fuse can stop it there, at the pre-call check, before the next step is paid for. Around that: steering (the runs live now, a person’s nudge or veto delivered at the agent’s next check, how a run’s agents handed work around), zero-setup recorders for Claude Code and Codex with Agent Blame for the lines their edits introduced, a context X-Ray of what a model saw at one step and a Cache Autopsy of why its prompt cache broke, a change radar for what changed in an agent and whether it hurt, TQL, a regular-expression language over a run’s steps with nine built-in detectors, and fork and rewind for any recorded run, with tape tests on top. Six new dashboard screens and three new demos (twelve in all) show it. Traza still never calls or embeds a model: every verdict here is counting and textbook statistics over stored spans, and every answer says how much history it stands on. There is no storage-format change. The Claude Code hook moved, the run guard’s memory budget now also holds behaviour paths, OTLP ingest keeps five resource attributes it used to drop, and a rollback to 0.27.0 after using a new fuse limit needs a step first: read Changed and Upgrade before deploying over an existing setup.

Added

• Foresight: the Fate Map and predictive fuses. docs/guide/foresight.md.

  • A run’s steps become behaviour tokens kind:name (the step signature without its input hash), consecutive repeats folded into one token by bucket (x, x×2, x×3-5, x×6+) so a loop is one shape. A path holds at most 64 tokens. Runs are compared only with runs of the same agent in the same tenant; traza.task (else session.goal) splits an agent into task cohorts.
  • GET /v1/fate/cohorts lists agents and tasks with outcome counts and base rate. GET /v1/fate/map?agent= is the tree: every node with its visits, outcomes, success rate and 95% Wilson interval, lift against the base rate, entropy of what runs did next, summed cost and duration, and the newest run of each outcome ready for /v1/diff. Fatal forks are children at least 25 points below their siblings’ pooled rate, both sides with at least min_support runs, at Fisher exact p < 0.05 — judged on the exact gap, never the four-place figure shown; hotspots are nodes with at least one bit of entropy. Pruning is always reported. The same spans make the same path on every surface: steps that start at the same instant are ordered by span id, then signature, then agent, in the map, the run route and the live view alike. Building and rendering a map, cached or not, is linear in its tree and spends the query budget, answering 400 when it runs out. Every map scores itself: each completed run predicted at several depths from the half of the cohort it is not in (two-fold cross-validation), with Brier skill against the base rate and a reliability table, in calibration.
  • GET /v1/fate/runs/{run} is foresight for one run: its path walked down its agent’s map over the last 30 days, a verdict (on_track, at_risk, likely_failure, novel, insufficient_history), the sentence behind it, what earlier runs did next with their rates, and the best of them as recommended_next. A live run’s last step is still open, so it is pooled with every run that repeated that step at least as often; a completed run is left out of the statistics it is judged by.
  • Predictive fuses: a fuse policy may set min_success_odds, with foresight_min_support (default 30) and foresight_after_steps (default 3). When the upper bound of the run’s interval — computed from the counts, not the rounded figure — is below the odds, the policy’s action applies, with the reason foresight: runs that took this path succeeded 5 of 50 times (≤21.4%). A check that names its next tool is judged with that step appended. It is evaluated on the pre-call check and verdict queries only — never on ingest, and not in the GET /v1/fuses/runs listing — reads only a cached map and never builds one there, and is not latched: a prediction made again at every question, lifted by a reset with a grant of lower odds. traza-server builds requested maps on a background thread, two at a time, each under --fate-build-seconds (default 120), halving a window that runs out of time and saying so. New flags --fate-max-runs, --fate-max-nodes, --fate-cache-seconds, --fate-tenant-maps and --fate-build-seconds; metrics traza_fate_builds_total and traza_fate_cached_builds. Any erasure drops every cached map.
  • MCP tool foresee_run, which fits --mcp-max-result-bytes whole: past it, it drops fields in a fixed order (next steps first, the path’s end last), says which in its headline and in truncated, and refuses with a short error, quoting nothing recorded, when even the run’s identity will not fit. And seed --behaviours, a corpus whose ground truth is known exactly.
  • What it cannot do: a fatal fork is correlation, not cause; paths ignore inputs; undeclared outcomes are inferred from the last span; the fork tests are not corrected for how many nodes were tested; maps are per process. See limitations.

• Steer: live runs, nudges and run topology. docs/guide/steer.md.

  • GET /v1/runs/live lists the runs with activity in the last --live-window-seconds (default 900) — current step, steps, tokens, cost, errors, the guard’s verdict, pending nudges and the run’s foresight — from the run guard’s memory and the cached Fate Map. GET /v1/runs/live/stream is the same view as server-sent events: a snapshot, then a row each time a run changes (coalesced to one batch per 250 ms), event: removed when a run the client holds leaves the view (evicted by the guard, or expired out of the window), a heartbeat after 15 quiet seconds, at most 64 streams per server and 8 per scope. A listing, or a stream update, reads its runs’ foresight under one query budget: rows it did not reach say foresight_deferred, and a run whose read failed rests for 60 seconds.
  • Nudges: POST /v1/runs/{run}/nudges (rw) queues advice (words for the agent) or block (a veto of its next step) for the run’s next pre-call check, which delivers it at most once in a new nudges field. A block answers stop for that check and every check of the run in the two seconds after, without tripping the fuse. A check naming no next step, the verdict route and the MCP tools never consume nudges. Nudges are durable in steer.jsonl, 1 to 4,096 bytes, expire after ttl_seconds (default an hour, at most seven days), 32 pending per run, bounded per tenant and in all, and erased with their run’s session, trace or tenant. A block whose two-second hold is in force is never evicted to make room (at most 32 holding per run); a budget full of them refuses a new nudge with 409 for those two seconds. GET /v1/runs/{run}/nudges lists them with their status. There is deliberately no MCP tool to read or queue one.
  • GET /v1/runs/{run}/topology: a run’s agents, the delegations between them, the handoff sequence and its cycles — control returning to an agent straight from the agent it handed off to. A new fuse limit, max_handoff_cycles (global: --fuse-max-handoff-cycles), stops agents bouncing a task between them, counted exactly as the topology counts it, since the last reset. Agents and services are told apart by their whole names; one longer than 160 characters is shown as its first 160, …# and 16 hex digits of a digest of the whole.
  • MCP tools list_live_runs and run_topology, whose headline is the server’s own counts, with run ids and tenants inside the telemetry frame.
  • The Claude Code hook pack, examples/claude-code/: one standard-library script and an installer that merges it into .claude/settings.json. PreToolUse asks the pre-call check, denies on a stop or a block nudge, and relays advice as additionalContext without ever answering allow, so the user’s own permission prompt is untouched; a stop’s recorded reasons reach Claude inside the untrusted-telemetry block, cut whole rows at a time; SessionStart adds a summary of earlier runs from /v1/recall inside the untrusted-telemetry block; an optional PostToolUse recorder. It fails open unless TRAZA_FAIL_CLOSED=1. The hook contract was checked against Claude Code 2.1.263’s reference.
  • What it cannot do: a lost check answer loses its nudge; an agent that never checks never hears one; a veto covers two seconds, not a turn; a nudge’s author is a credential, not a person; the live view is per process. See limitations.

• Recorders for Claude Code and Codex, and Agent Blame. docs/guide/recorders.md.

  • traza-server import claude-code and traza-server import codex turn the transcripts those tools already write (~/.claude/projects, ~/.codex/sessions, including Codex’s .jsonl.zst) into runs: every session a run, every model response an llm step with its usage, every tool call a tool step, every sub-agent an agent span. No SDK, exporter or proxy. Ids derive from the transcript’s own identity, so importing twice leaves one copy. Unknown records are counted and skipped, never fatal. Over --url (with --tenant, checked against /v1/whoami before anything is read) or straight into a --data-dir; --since, --project-filter, --dry-run, --json.
  • --watch follows live sessions within about a second, resuming from a per-destination state file (~/.traza/import-state.json) so a restart sends nothing twice, and outlives an unreachable server. A file is skipped only when its size and its stamp (modification time, and device and inode on Unix) are both unchanged, so a transcript rewritten to the same length is read again; a sub-agent’s .meta.json that appears or changes later re-sends its transcript, placed under its parent call. A Codex rollout found while its first line is still being written waits for the next poll, so a sub-agent keeps its parent’s session. What an importer remembers is bounded: 16,384 response digests, 4,096 closed turns, 256 MiB of a turn’s held lines.
  • A tool step’s status stays unset until its result arrives; an importer no longer calls a call ok that may not have finished.
  • Compressed rollouts are read by Traza’s own Zstandard decoder (RFC 8878, no new dependency), checksum verified, capped at 2 GiB compressed and decompressed; a sub-agent meta file at 64 KiB. A file over its cap is refused, named and counted in files_failed.
  • Agent Blame: GET /v1/blame?path=&line=, traza-server blame FILE:LINE and the MCP tool blame_line name the earliest tool step whose edit introduced a line — its run, the prompt that started the turn, the model, the step on the tape — read from Edit, MultiEdit, Write, NotebookEdit, Anthropic’s text-editor tool and Codex patches, whitespace-normalised. An edit whose step failed (old_string not found, a patch that did not apply) wrote nothing and is never a match; each match says outcome — applied, or unconfirmed for a call with no recorded result, listed after every applied one and printed as “attempted”. The CLI scopes itself to the repository and its worktrees (--anywhere lifts that). Measured on one developer’s real history, before failed edits stopped counting (so an upper bound): 0.33 s to answer that no agent wrote a line, and a median of 0.44 s for a line it found, over 77,002 spans; 3.2 s and 2.9 s over 970,972 spans with a Codex history added.
  • What it cannot do: the formats are undocumented and change with their tools; a transcript records no system prompt or tool definitions; blame cannot see lines a shell command wrote. See limits.

• Context X-Ray and Cache Autopsy. docs/guide/xray.md.

  • GET /v1/xray/{trace_id}/{span_id}: one model call’s context window as chunks (system, user, assistant, tool call, tool result, tool definition) with estimated tokens, each traced to the step that introduced it; stale tool results (unreferenced for --xray-stale-steps, default 20 — a heuristic); duplicates; and what the conversation’s previous call had that this one lost (dropped_since_previous, instructions first) — the compaction-loss detector.
  • GET /v1/cache/{run}: every model call against its predecessor in the same conversation, with a cause (stable, volatile_content, reordered, ttl_expired, new_context, unknown), the first chunk that changed and the bytes that changed in it — usually one volatile token, a timestamp or an id — the tokens the cache could have served and did not, priced, and culprits with advice from a lookup table. Disagreements with the provider’s own cache counters are listed.
  • Conversations recorded as deltas (gen_ai.conversation.id and traza.input.delta, which the recorders set) are rebuilt call by call, so an imported session’s X-Ray shows the whole conversation. A conversation is identified by a SHA-256 of its whole id.
  • What could not be read is never evidence. A context past 2,000 chunks is cut (chunks_truncated, on each cache verdict and on the autopsy), and a verdict the unread part could change is unknown with nothing counted lost; a chunk missing from a context that was cut, or that held an offloaded value left unresolved (a request, tool definitions, an earlier answer), is unverified — not proven lost — rather than a lost instruction.
  • MCP tool xray_step.
  • What it cannot do: per-chunk tokens are estimates; the provider’s cache rules (minimum length, breakpoints, TTL refresh) are not modelled; byte exactness is the instrumentation’s. See limits.

• Cache rates in the pricing file. A derived cost knew nothing about cached-prompt discounts. A model’s entry may now name cache_read_per_mtok and cache_write_per_mtok, and every derived cost then counts cache reads and writes at their own rates under the provider’s convention — on top of the prompt for Anthropic, Bedrock and Claude on Vertex, carved out of it for OpenAI, Azure OpenAI and Gemini on Vertex — instead of understating a cache-heavy Anthropic call and overstating a cache-heavy OpenAI one. Without the keys nothing changes. See model pricing.

• Change radar: what changed? docs/guide/radar.md. GET /v1/changes?agent= (and the MCP tool what_changed) returns a ranked timeline of change points in an agent’s system prompt, tool catalog, requested and returned model, sampling parameters, service and SDK versions, and in its behaviour with no configuration change (a shift in its top-level paths, chosen on half the runs and tested on the other half). A value is a change point when it takes over and runs it played no part in proposing confirm it, found in one pass linear in the runs and under the query deadline. Sampling parameters are identified by their typed JSON (1 is not "1"), and a run carrying more than four values of a dimension is left out of that dimension’s detection rather than mistaken for a change. Each change carries error rate, success rate, cost, steps, duration and tool mix on both sides, with two-proportion z-tests (Fisher’s exact test when a rate’s table is sparse) and Welch t-tests under Holm’s correction, and a verdict (regression, improvement, neutral, insufficient_data below 10 runs a side) in which a metric takes part only when its own test had 10 observations a side; regressions come first. ?run= places one run on the timeline, ?environment= filters by deployment.environment, and --radar-max-runs (default 20,000) bounds an analysis. What it cannot do: correlation in time, not cause; it lists co-changes it cannot separate, assumes independent runs, and is noisier below about 200 runs. See limitations.

• TQL and the detector library. docs/guide/tql.md. A pattern language over a run’s steps: atoms (tool(bash), llm(gpt-4o), agent(...), step, and derived edit and test), predicates in brackets ([status=error], [cost>0.5], [args~"pytest"], [file~"test_"], …), quantifiers, adjacency ->, eventually ..>, alternation, grouping and captures that must match (@f), compiled to an NFA run by a Pike VM, with a linear-time, std-only regex subset for ~. Parse errors name the column, and compiling is bounded in work as well as size. GET /v1/trajectories/search?q= (newest runs first, under the query deadline, continued exactly — runs tied on time included — with the opaque cursor an answer’s next_cursor gives), GET /v1/trajectories/explain, GET /v1/detectors, GET /v1/detectors/findings, and the MCP tool find_behavior. An answer where a matcher cap may have hidden a match counts those runs in runs_inexact and is truncated, never an exact “no match”. Nine detectors: test_tampering, give_up_loop, hallucinated_tool, edit_thrash, ping_pong, retry_without_change, context_blowup, tool_error_streak and unverified_finish; findings compare how the matching runs ended against the rest. What it cannot do: edit and test are heuristics over tool names and arguments, failure means span status, there is no index, and “never” over a whole run cannot be written. See limits.

• Fork and rewind, tape tests and golden runs. docs/guide/fork.md.

  • POST /v1/replays takes edits (up to 64: a different model answer or tool result at a step) and live_after (after that step every call answers 409 replay_live, so the harness switches to the real provider). Strict matching holds up to the first edit and is off after it. A fork answers fork_id and the traza.fork.* attributes its spans carry; a live point alone is a rewind, and a run past 20,000 spans can still be rewound within what was read. A repeated divergence is answered from the difference already worked out.
  • GET /v1/forks?run= lists a run’s forks, each diffed against its parent; GET /v1/goldens the runs marked as baselines, decided from the newest 10,000 golden markers under one query budget, with annotations_examined counting every one in scope. MCP tool list_forks.
  • What it cannot do: Traza never goes live itself — your harness does; an edited answer has an invented id and zero usage; a fork is a replay, in memory for an hour. See limits.

• Client libraries. docs/guide/clients.md. Python: traza.fork(run, at_step, response=… | tool_result=…), a context manager that points the provider SDKs at a loopback switch which forwards to the replay until it answers replay_live and to the real provider after, and tags the spans started in its context; forks, goldens, mark_golden; traza.testing tape tests; the pytest-traza plugin (@pytest.mark.traza_tape("run"), a pytest11 entry point; the package still depends on nothing); and a traza command — traza test replays every golden run into JUnit XML, keeping the last 64 KiB of an agent’s output as it is written, never the whole, plus traza golden add, traza forks and traza fork. A tool the recording lacks running live moves no model call off the recording: only the model endpoint’s own replay_live does, in both libraries.

• traza forward, in the Python package, lets the tools that speak plain HTTP only — traza-server import --watch, traza-server blame, the MCP bridge — reach a server over TLS, Traza Cloud included: traza forward --upstream https://api.trazadb.com, then --url http://127.0.0.1:8089. It listens on loopback only, verifies the upstream’s certificate against the system’s authorities with no way to turn that off, passes method, path, query and end-to-end headers through, streams bodies both ways, caps a request body at 64 MiB, refuses redirects, and bounds every wait. Standard library only. The importer’s and the bridge’s refusal of an https:// URL now name it. See the recorders guide. TypeScript: client.fork() returning a Fork with a switching fetch for the provider SDKs and a span processor, forks, goldens, markGolden, and replayTest() for vitest, jest or node:test. Both hand the nudges a check delivered to the caller (Verdict.nudges and on_nudges, verdict.nudges and onNudges).

• Dashboard: six screens, and steering from the Run view. docs/guide/trace-browser.md.

  • Live (#/live): the runs working now over the live stream, read with fetch so the token never enters a URL — current step with its elapsed time, spend counting up, the guard chip and foresight against the base rate — ranked so runs that need a person come first, with nudge, kill-switch and approve controls (a grant can lower min_success_odds, the only way to lift a foresight stop). It reconnects with backoff and rebuilds from a fresh snapshot, drops a row the moment the server says its run left the view, says foresight deferred for a row whose read the listing’s budget did not reach, says so when the server has no live view, and marks a run whose named step has waited as held only after reading that run’s verdict, which never takes its nudges. An address naming no screen now opens Live when runs are working, the Overview otherwise.
  • Foresight (#/foresight): the Fate Map drawn as a tree with ribbons sized by runs and coloured against the base rate, fatal forks and hotspots labelled in words, intervals on hover, example runs and a success-versus-failure diff, the map’s calibration in words, and a run lit on its map with its verdict — only where it was judged.
  • X-Ray (#/xray/<run>): the Cache Autopsy timeline and culprits, and one call’s context as a stacked bar by kind and introducing step, with what was dropped since the previous call — a loss from a context not read whole marked “not proven lost”, never counted.
  • Changes (parameters shown as typed JSON; a delta marked only when its own test had 10 observations a side), Behaviors (a TQL box checked by the server as you type, pages continued by the server’s cursor, and the detectors with their intervals) and Blame (an edit with no recorded result shown as an attempt, after the applied ones).
  • The Run view gains a Foresight panel, an Agents card with the topology drawn, a Nudges card with the composer, Fork from here on any model or tool step — an editor on the recorded answer, then the environment as shell-safe lines with the key as TRAZA_REPLAY_KEY, a traza.fork snippet and a traza fork command — and a Forks card with each fork’s deltas and its diff.
  • Stored text is rendered as text everywhere; every new screen reflows at phone width without a horizontal page scroll.

• Three demos: foresight, coding-agents and mission-control, twelve in the tour now. foresight draws the Fate Map with every interval, recomputes the fatal fork’s Wilson interval and Fisher test from its counts, calls a live run likely_failure before it has an outcome and stops it at the check while a healthy run goes through. coding-agents imports four synthetic Claude Code and Codex sessions, prices each sub-agent, finds a cache that expired and a /compact that dropped an instruction, catches a test edited to pass, blames three lines and follows a live transcript with --watch. mission-control runs simulated agents that check before every step and obey nudges, checks seven claims (a run held by its history, a nudge heard after an approval, the kill switch, a ping-pong stopped at three cycles, a fork), then keeps them running for the Live screen until Ctrl-C. Each asserts its own claims, and ci.sh runs all twelve, pointing the three new ones at the binary it just built (and foresight at the seed built beside it).

Changed

• The Claude Code hook moved. examples/guard/claude-code-hook.sh is gone; the hook pack replaces it, with an installer, nudges and a SessionStart recall. Its fail-closed switch is TRAZA_FAIL_CLOSED=1 (it was TRAZA_FUSE_FAIL_CLOSED), and it never answers allow. The guard demo runs the pack in its seventh beat.
• OTLP ingest keeps five resource attributes. service.version, deployment.environment.name, deployment.environment, telemetry.sdk.name and telemetry.sdk.version on a resource are copied onto each of its spans as attributes, beneath any the span sets itself, never pushing a span over its admission caps. They are stored, searchable and what the change radar reads. Every other resource attribute is still dropped. See ingest.
• The run guard keeps every live run’s behaviour path — its earliest 192 steps, 48 bytes each — for live foresight and the predictive fuses, counted against the same 1,000,000-record budget as span contributions. Step-heavy runs are therefore evicted somewhat sooner than in 0.27.0 (they rebuild from the store when next asked about). --fate-cache-seconds 0 stops keeping paths; an embedder sets FuseConfig::live_paths = false, after which only runs a predictive policy could govern keep one.
• POST /v1/fuses/check answers carry nudges (an empty list when none), and foresight when the policy sets min_success_odds; the verdict document of GET /v1/fuses/runs/{run} carries foresight too. A check that names its next step now delivers pending nudges and counts as activity for the live view: give that credential only to the agent, and leave next out of monitoring checks.
• The MCP surface grows from sixteen tools to twenty-four: eight reads (foresee_run, list_live_runs, run_topology, blame_line, xray_step, what_changed, find_behavior, list_forks), so an ro token is offered twenty-two. See the MCP guide.
• max_repeats counts within a window of the run’s latest steps. A step fingerprint repeats when it occurs that often among the run’s last repeat_window_steps steps (default 50, at most 1,000; a policy field, a grant field and --fuse-repeat-window-steps for the global policy), not over the whole run — so the global default no longer warns on every long interactive session (one real session took the same screenshot 32 times over three days), while a tight loop still trips. A warning clears once the loop leaves the window; a stop latches where a window first filled, live and in a rebuild alike, whatever order the spans arrived in. The reason now reads … repeated 4 times in the last 50 steps, and usage.top_repeat carries window_steps. Each span contribution the guard holds grew from 40 to 48 bytes (its start, to rebuild the window in step order), and each run keeps at most its latest 1,000 steps, 24 bytes each, for the count. See run fuses.
• TQL’s edit_thrash means thrash, and context_blowup stays inside one conversation. On coding agents edit_thrash fired on any five edits of a file in ten steps — what writing a feature looks like (24 of 57 real runs, 23 of them successful) — and context_blowup compared a sub-agent’s small first call with the main thread’s next one (half of 905 real hits). edit_thrash now needs, within 30 steps, one file’s edits undoing and redoing each other (text written, removed, written again), the file rewritten whole back to an earlier version, or three edits each followed by a failing test with no pass between; it is medium, low in a run that succeeded (severity_if_succeeded in GET /v1/detectors; findings rate a detector’s row by its runs and give each example its own severity). context_blowup compares calls of one gen_ai.conversation.id, else of one agent, and only calls naming neither with each other. Three TQL fields say so: conversation, and an edit’s added and removed text, read as Agent Blame reads edits and whitespace-normalised. See TQL.
• A session’s turns are its prompts where a recorder marks them. GET /v1/sessions and /v1/sessions/{id} carry turn_count — the turn spans traza-server import writes, one per user prompt, else the session’s traces — and turns_from (prompts or traces). The rollup sidecars carry the prompt count, so they move to format 6 and are rebuilt once from their segments.
• A tape step’s tokens carry context: the context the engine reads (input plus cache read and write for a provider whose input leaves the cache out, input alone for one whose input includes it, null when that cannot be told), so a cached Anthropic call no longer reads as an input of 2. Both client libraries read it.
• The Cache Autopsy gains a cause, partial_hit, and by_cause lists it. A call past the assumed TTL whose provider read only part of the predecessor’s reported context — a long-lived system prompt kept warm while the conversation after it expired — was stable with nothing lost when it read at least half. It now counts what was not served as lost; only a read of all of it is stable. See the X-Ray guide.
• Library surface. Config gains fate: fate::FateConfig, steer: steer::SteerConfig and xray_stale_steps: usize; guard::FuseConfig gains live_paths: bool; guard::Limits gains min_success_odds, foresight_min_support, foresight_after_steps, max_handoff_cycles and repeat_window_steps; guard::TopRepeat gains window_steps; tql::Detector gains severity_if_succeeded and tql::RunMatch severity; tape::StepTokens gains context; analytics::SessionSummary gains turn_count and turns_from; xray::Cause gains PartialHit; pricing::Rate gains cache_read_per_mtok and cache_write_per_mtok. A struct literal must add them, and a match on Cause the new variant; ..Default::default() need not. New public modules: blame, fate, fork, import, radar, steer, tql, xray, zstd.
• GET /v1/annotations and experiment scores spend the query budget. Their walk of the annotation index is checked against --query-deadline-ms and answers 400 past it, like every other read; it could not run out before. With a limit the walk holds at most that many annotations, the newest, and annotations a pending erasure withholds no longer take places within it.
• traza-server --help lists fifty-six flags, ten of them new: --fate-max-runs, --fate-max-nodes, --fate-cache-seconds, --fate-tenant-maps, --fate-build-seconds, --fuse-max-handoff-cycles, --fuse-repeat-window-steps, --live-window-seconds, --xray-stale-steps and --radar-max-runs; and two subcommands, import and blame. See configuration.
• Docs: the ingest guide said other resource attributes were “merged beneath span attributes”. They never were; it now says which reach the span.
• Version metadata: the crate, the UI and both client libraries move to 0.28.0. The MSRV stays 1.81, and a default build still has three direct dependencies. No performance figure in this release is new; the published ingest throughput predates both 0.27.0’s admission pass and the behaviour path above.

Fixed

• An evidence pack finishes on a busy store. A manifest too long for one request’s budget never finished when the store checkpointed between retries: each retry started over on the new generation, met the next checkpoint and started over again — and a store with any damaged file never got its pack, because confirming the damage let the pack move on to the newest generation and begin again. A pack now pins the snapshot and generation its manifest load began on and answers for that pair until it is whole (its ledger.head says which), and a problem stands while the path still holds the file the digest read, with the length and modification time the pinned manifest records. Confirming file problems takes at most 256 per hold of the maintenance lock, with the budget checked between holds, and replaced file checks are freed 4,096 files at a time between budget checks, never under the lock. See the ledger guide.
• A replay call is judged from when its request arrived. Time spent reading, parsing and digesting a request before the replay registry saw it did not count toward the replay’s expiry, so a request that arrived a moment before expiry could be served after it. The expiry clock now starts as soon as the request head is read.
• A diff explanation quotes a long configuration value in bounded time. Quoting eighty characters of a differing parameter scanned the whole value first, so one long string held the explanation past the deadline (600 ms for 32 MiB in a debug build). The quote is now written a character at a time and stops at the cap.
• An annotation query naming both an experiment and a trace returned the experiment’s annotations from every trace: the trace was ignored when the experiment’s index was walked. Every filter now applies whichever index is walked.
• @trazadb/sdk writes JSON the way JSON.stringify does. It re-read an array’s length on every element, so a toJSON hook that grew its own array ran until the heap was exhausted, before any request timer existed; and it read every member up front, so a hook’s change to a later member went out differently. Both now match JSON.stringify.
• Dashboard. A credential minted while the tab switched to another token (a handoff link followed before the mint answered) no longer shows its secret over the new session, and a secret on display no longer outlives the switch. Leaving the Live tail (or any streaming screen) no longer throws an unhandled “signal is aborted” error. A session nothing priced — a coding-agent transcript, with no pricing table — shows “—” for its cost per turn, as for its cost, not 0.00000.
• Foresight on a long session keeps its outcome. GET /v1/fate/runs/{run} (and MCP foresee_run) read a run of more than 20,000 spans from its earliest 20,000 only, so a finished coding-agent session that long came back "outcome": "unknown" with a step count capped at what was read — while the same answer said the run had been left out of its map as a completed one. The run is now read whole: its path still comes from its earliest 20,000 spans (partial), but its outcome and steps cover every span, as the map’s fold always did.
• import --since selects sessions that ran recently, not files that were touched. Claude Code appends untimestamped bookkeeping (the last prompt, a title, a mode) to transcripts it has not run for months, which moves their modification time: on one real machine a third of the transcripts modified in the last two weeks had not recorded a turn in five to thirteen weeks, so --since 14d imported 57 sessions of which 49 had no span in the last 30 days — and every 30-day surface (the Fate Map, TQL, findings) then saw only the 8 that did. A Claude Code file now counts as recent by its newest timestamped record, found by reading backwards from its end (at most 8 MiB, past bookkeeping and an unfinished line); a file whose records cannot tell, and a Codex rollout (which timestamps every record it writes), are still judged by when they were modified.
• TQL searches, detector findings and recall stop hashing every step. Each read every candidate run’s steps with their request and response SHA-256s — over a coding agent’s runs, a digest of every model call’s whole conversation — though only a pattern naming digest or fingerprint reads one, and recall reads neither. On a real store of 57 coding-agent sessions (89,777 spans) that was most of the time: a search took 4.8–12.5 s and the whole detector library 9.6 s. A step’s digest is now computed when a pattern first reads it, and recall summarises runs from their step signatures alone: searches take 2.4–3.6 s, findings 4.1 s and recall 1.2–3.4 s on the same store (release build, a laptop).
• The Cache Autopsy believes a provider that served the prefix. An intact prefix past the assumed five-minute TTL was always ttl_expired, its whole prefix counted lost, even when the provider reported reading it from cache — counted as a disagreement but kept in every total. Claude Code runs longer caches: on one real session 24 of its 25 expired verdicts reported a cache read, 15 of them of the whole prefix, and its “9.2 million tokens lost” were two-thirds served. A call that reports reading at least half of what its predecessor reported as its context is now stable, nothing lost, its explanation saying the cache outlived the assumed TTL; a smaller read (a long-lived system prompt alone) leaves the verdict standing, and no structural verdict yields to a reported count.
• The X-Ray reads a long session’s latest calls. A run read its first 20,000 spans, and a call past them was X-rayed alone — no step index, no predecessor and, for Claude Code and Codex transcripts (recorded as deltas), no context at all: exactly the latest calls of the sessions worth X-raying. Such a call is now read with the 20,000 spans that end at it: its indices count the whole run, and a conversation already under way before those spans is rebuilt from its first compaction inside them (a delta call before that says its conversation began before the read). xray_step reaches any step of a run, not only those in its first read.
• An imported session’s goal is what the person asked. A harness that opens a Claude Code session’s first prompt with a <system-reminder> block (a worktree launcher, a workflow) made that boilerplate the session’s session.goal — the text every run surface shows, and the value the Fate Map hashes into a task, so every such session joined one task. The goal now skips leading reminder blocks, and a prompt that is nothing else sets none.
• Dashboard: a fork stays in the tenant it was drafted in. Back and forward to the same run in another tenant kept the fork editor open, and creating it applied the first tenant’s edited answer to the second tenant’s step of that index. The editor now closes when the tape’s scope changes, and a creation still pending never lands in a later draft.
• Dashboard: Live’s clock and figures. A run frame for a run the page did not hold — left out of a cut snapshot, or let go past the page’s 1,000 rows, and sent again because a nudge or a verdict moved it — was read as the server’s clock, which could move it back by minutes and suppress waiting runs and held-run probes; it now only bounds the estimate. Letting runs go past 1,000 now marks every figure partial (1,000+), as a cut snapshot does, until the server drops those runs or a snapshot covers every run. The held-run probe forgets runs that left the view, cancels reads about them, and keeps at most four reads in flight.
• Dashboard: Find these runs finds the node’s runs. Its query matched later repeats and other cases of a name (×3-5 found runs that searched six times; Search found search). It now matches exact step signatures and holds the last token to its repeat bucket, and is not offered where two branches of the map read the same (a tool literally named x×2 beside x called twice).
• Dashboard: Foresight, Changes and X-Ray. Naming another run kept the last run’s verdict, odds and path in the banner under the new name until its own answer arrived, and a map still drawn from another window lent the banner its skill; both wait for their own answers now. A Changes timeline marker jumped to another tenant’s card when two tenants’ agents share a name. A context of thousands of chunks no longer clips its last chunks off the X-Ray’s bar: the bar scrolls.
• The Mission Control demo abandons a run whose step the check keeps refusing. A stop on the next step alone left the verdict read saying continue, which the agent took for a release, so it never abandoned and could fill the fleet. One deadline now covers the refused step, and a release counts only when the check lets the step go.
• Foresight for one run answers to the query deadline. foresee read every span of a run, computed its outcome over all of them and judged it without consulting the deadline after the read, so a long run could answer well past --query-deadline-ms. The read now checks the deadline every 1,024 spans and around the outcome, and the answer is checked once more before it is returned; a predictive fuse’s read that runs out of time says so instead of judging.
• Dashboard: a session’s turns are its prompts. The Sessions list and a session’s page counted traces as turns, so every imported Claude Code or Codex session — one trace however many prompts it held — showed one turn and a cost per turn equal to its whole cost, and the “longest” and “worst cost/turn” sorts ranked by the same count. Both now read turn_count, and each count says on hover which it is; a session’s per-trace table heads its rows “trace” when its turns are prompts.
• Dashboard: a tape step’s input counts what it read from the cache. The run tape showed Anthropic’s input_tokens, which leaves cached tokens out, so a call that read 40,000 tokens from the cache read “2 / 555”. A step now shows its whole input — the server’s context figure, under the engine’s provider rule — with its parts on hover. Where the provider’s convention is not known the reported input stands, and the hover says why.
• Dashboard: the Fate Map tells apart two branches that read the same. Siblings with one text (a tool literally named x×2 beside x called twice) shared a React key: React warned of duplicate keys, clicking the second opened the first, and folding one folded both. Each branch is now keyed by its place among its siblings; a fatal fork or hotspot entry opens the branch whose figures it gives, and a link naming a path by its text still opens the first.
• Dashboard: the Run screen’s foresight waits for its own run. Moving to the same run in another tenant kept the other tenant’s verdict, odds and path in the foresight panel until the new answer arrived. Each answer is now kept with the run, tenant and credential it was asked for and shown only beside that run; a late answer for the earlier scope changes nothing.
• Dashboard: a Fate Map still drawn opens its own tenant’s runs. Switching Foresight to another tenant kept the first tenant’s map on screen while the second’s loaded, but its example runs and “Diff success vs failure” opened the first tenant’s run ids under the second tenant. They now carry the tenant the drawn map was read in.

Upgrade

• No migration. Segments, the write-ahead log, generation manifests and every file 0.27.0 wrote are read as they are; the derived analytics sidecars (*.rollup) move to format 6 and are rebuilt once, from their segments, on first read. One new file appears beside them when it is first needed: steer.jsonl, the nudge log, which like fuses.jsonl lives outside generations, so a backup does not carry it. traza-server import keeps its progress on the importing machine, in ~/.traza/import-state.json.
• Move a 0.27.0 Claude Code hook to the pack: remove the old claude-code-hook.sh entries from .claude/settings.json and run python3 examples/claude-code/install.py, renaming TRAZA_FUSE_FAIL_CLOSED to TRAZA_FAIL_CLOSED if you set it.
• Leave next out of checks that are not the agent’s own. A monitoring script that sends next to POST /v1/fuses/check would take the run’s nudges; without it the check is a verdict read and takes nothing.
• Repeat limits count within 50 steps. A policy that relied on max_repeats counting a fingerprint over the whole run sets repeat_window_steps (up to 1,000); a monitor that matched the reason text exactly sees … repeated N times in the last W steps. edit_thrash and context_blowup find fewer, better runs, and the Cache Autopsy may report partial_hit where it said stable: findings and saved queries compared across the upgrade will differ.
• Expect five more attributes on OTLP spans whose resource sets them. A consumer that compares a span’s attribute set exactly will see them.
• Embedders must add the new Config, FuseConfig, Limits and Rate fields to struct literals.
• Rolling back to 0.27.0 reads the same data, ignores steer.jsonl and the import state, and stops delivering nudges. But it refuses to start on a fuses.jsonl that holds any policy, reset or grant naming min_success_odds, foresight_min_support, foresight_after_steps, max_handoff_cycles or repeat_window_steps — deleting the policy is not enough, since the log keeps the record until it is next rewritten. Either remove the lines that name them from fuses.jsonl while no server holds the directory, or start 0.27.0 with --no-fuses, under which it neither reads nor writes the file and no policy, kill switch or grant applies.

[0.27.0] - 2026-10-17

The flight recorder release. Traza already kept what an agent did; this release lets the recording act, while a run is going and after it is over. A run fuse holds a run to limits on cost, tokens, steps, repeated calls and failure streaks, answers every ingest acknowledgement with a verdict, and offers a pre-call check that refuses a step before it starts. A recorded run replays through the agent’s own OpenAI or Anthropic SDK with zero model calls, and two runs diff to the step where they parted. A hash-chained ledger over every published generation and erasure makes committed history tamper-evident against an anchor kept outside the store, and an evidence pack puts it in one digest for a notary. Underneath is what a shared, multi-tenant deployment needs: runtime credentials, tenant quotas and usage counters, admission caps, gzip bodies and health probes. Python and TypeScript clients, three new demos (nine in all), and five new dashboard screens round it out. There is no storage-format change. Several authorization defaults tighten, and the data-directory lock behaves differently in containers: read Changed and Upgrade before deploying over an existing setup.

Added

• Steps and runs. Every span is now also read as a step of a run: its step kind (llm, tool, agent, retrieval, embedding, other, from gen_ai.operation.name, openinference.span.kind or traceloop.span.kind), its tool, agent and tool-call id, its request and response as stored with a canonical-JSON SHA-256 digest of each, and a kind:name#hash fingerprint that two steps share exactly when they made the same call with the same input. A run is the span’s session id (session.id, then gen_ai.conversation.id, then OpenLLMetry’s association properties), else trace:<trace_id>, within its tenant. Only llm, tool and agent spans count as steps. The OpenInference keys Arize Phoenix emits (llm.model_name, llm.provider, llm.system, llm.token_count.*, openinference.span.kind) are recognized for analytics too, ranked after every existing spelling so no span’s previous answer changes; OpenInference cache counters are deliberately not read, since its prompt count already includes them. See LLM semantics § Steps and runs.

• Guard: run fuses. docs/guide/guard.md. Eight optional limits per run (max_cost_usd, max_tokens, max_steps, max_llm_calls, max_tool_calls, max_duration_seconds, max_repeats, max_error_streak) give it a verdict: continue, warn at 80% of any limit, and at 100% the policy’s action, where stop is sticky until reset. A fuse is arithmetic over recorded spans; nothing calls a model.

  • Policies per tenant (POST /v1/fuses/policies, GET, and POST /v1/fuses/policies/{id}/delete): matched by the service of the run’s earliest-starting span, else the tenant’s default, else the global policy set by --fuse-max-cost-usd, --fuse-max-tokens, --fuse-max-steps, --fuse-max-duration-seconds, --fuse-max-repeats (default 12), --fuse-max-error-streak and --fuse-action (default warn). Limits do not inherit between policies. Out of the box the global policy warns at 12 repeats and stops nothing.
  • The acknowledgement carries the verdict. POST /v1/spans answers a fuses field and a Traza-Fuse: stop; run=… header naming the most severe verdict in the batch; POST /v1/traces carries the header alone, since OTLP’s response has no place for it. The spans are stored either way: a verdict is about the run’s next step, never a refusal of the batch.
  • POST /v1/fuses/check is the pre-call gate, and the only surface that can refuse a step before it runs: the run’s recorded usage plus one hypothetical step with an optional est_cost_usd, est_tokens and tool. It changes nothing and needs only the ro scope.
  • The kill switch (POST /v1/fuses/runs/{run}/trip), reset with an optional grant of absolute higher limits (POST …/reset; a reset never refills cumulative usage), a run’s verdict document (GET /v1/fuses/runs/{run}), runs by verdict (GET /v1/fuses/runs), and every transition — warn, stop, manual_stop, reset — by cursor from a 4,096-event ring with a long-poll of up to 30 s (GET /v1/fuses/events).
  • Live state is a bounded cache folded on the ingest path — --fuse-max-runs (default 100,000) runs, 1,000,000 span contributions, six-hour idle eviction, stopped runs parked rather than forgotten — and a run this process did not follow from its first step is rebuilt from the store on the first question about it. Policies, kill switches, resets and grants persist in fuses.jsonl, outside generations: a backup does not carry it and a restore does not touch it. --no-fuses turns the observer off entirely. Metrics: traza_fuse_trips_total, traza_fuse_warnings_total, traza_fuse_live_runs.
  • examples/guard/claude-code-hook.sh: a Claude Code PreToolUse gate that exits 2 with the reasons on a stop, and an optional PostToolUse recorder that gives the step and repeat limits something to count. It fails open unless TRAZA_FUSE_FAIL_CLOSED=1.
  • What it cannot do, stated in the guide: a post-ack stop is at least one step late, more with exporter batching; a step in flight is never interrupted; the check sees recorded usage only, so parallel steps can all pass it; across batches, arrival order counts; and concurrent re-sends that move usage between steps of one run can latch a stop on a total the store never held. See limitations.

• Replay: tapes, the replay proxy, diffs and recall. docs/guide/replay.md.

  • GET /v1/tapes/{run}: a run’s steps in start order with requests, responses, digests, fingerprints and token counts, read from OTel GenAI, OpenLLMetry, OpenInference and native layouts into one message shape. ?resolve=true inlines offloaded payloads up to --tape-max-bytes (default 8 MiB) as a prefix, and says resolved_partial when it stops.
  • POST /v1/replays snapshots a run (in memory, for one hour) and returns base URLs for an OpenAI-compatible …/openai/v1/chat/completions and an Anthropic-compatible …/anthropic/v1/messages. Set OPENAI_BASE_URL or ANTHROPIC_BASE_URL to one and the agent’s own code runs again with every model answer served from the recording: zero model calls, and no outbound request of any kind. "stream": true answers as server-sent events rebuilt from the stored text. Strict mode, the default, refuses a request whose normalised messages are not the recorded step’s with 409 replay_divergence, naming the step, without consuming it, and with x-should-retry: false so the SDKs do not retry; loose mode serves in order and records whether each call matched. GET /v1/replays/{id} says what was served. Any valid credential, ro included, may create and use a replay, and on the routes under /v1/replays/ only, the token may arrive as x-api-key, so ANTHROPIC_API_KEY can hold it.
  • POST /v1/replays/{id}/tools/next hands a harness the next recorded tool result; otherwise tools run live, with their real side effects.
  • GET /v1/diff?a=&b= aligns two runs by longest common subsequence over kind:name signatures, classifies each aligned pair as same or changed by input, decision or config (unaligned steps are structure), and explains the first divergence, with cost, token, duration and error deltas.
  • GET /v1/recall?q= finds earlier runs whose recorded text holds every word of the query, successful runs first, each with a skeleton of what it did.
  • Bounds: a run of more than 20,000 spans is refused for replay (409) rather than replayed as its beginning; each tenant’s share of the registry holds 64 live replays and 64 MiB, the registry 1,024 and 512 MiB, past which creation is 429 with Retry-After; a served response is at most 64 MiB. An erasure withdraws any replay holding what it covers (410 replay_erased). Only chat completions and messages are replayed, streams are not the provider’s bytes, and strict mode compares messages only, not the model or parameters. See limits.

• Prove: the ledger and evidence packs. docs/guide/ledger.md.

  • ledger.jsonl in the data directory: an append-only SHA-256 hash chain with one entry per published generation (checkpoint, carrying the manifest’s SHA-256), per settled erasure (erasure, carrying a SHA-256 of the subject, never a trace, span or session id or a payload reference — though a tenant subject’s tenant field is the tenant’s name — and a random salt, so another tenant shown a stub of the entry cannot recompute its head), and per external anchor, each head committing to the one before: sha256("traza-ledger/1\n" + prev + "\n" + canonical_json(entry without head)). The guide’s ten-line standard-library Python script recomputes every head, and tests/ledger.rs runs it verbatim from the guide.
  • GET /v1/ledger (paged; unbound credentials), GET /v1/ledger/head (any ro, bound included), GET /v1/ledger/verify (unbound: recomputes every head and link, checks the latest checkpoint names the live manifest, re-digests every manifested file, and checks ledgered erasures against the tombstone log in both directions), POST /v1/ledger/anchors (unbound admin: stores a notary’s countersignature over an earlier head, 409 unless target_head is the head recorded at target_seq), and GET /v1/evidence (any ro; a bound credential gets its own tenant’s pack, with other tenants’ erasure entries as stubs it cannot recompute).
  • traza-server verify --ledger [--data-dir DIR] [--json]: the same verification offline and read-only, against a directory no live server owns. Exits 0 intact, 2 broken, 1 when it cannot verify.
  • A backup pin carries pins/<label>/ledger.jsonl, ending on the pinned checkpoint. The erasure receipt gains a ledger domain, reported retained-by-design.
  • What it does not prove, stated before anything else in the guide: the chain has no secret, so history is tamper-evident only against an anchor you keep outside the store; anchor signatures are stored, never checked; at_ns is the server’s clock; restoring a pin taken before an erasure undoes it, and verification then names the entry. The ledger never rotates: a checkpoint entry is about 305 bytes, about 88 KB a day at the five-minute checkpoint cadence. See limitations.

• Platform.

  • Runtime credentials: POST /v1/admin/credentials mints a token (trz_ and 64 hex digits, shown once) or registers a supplied high-entropy one, with an optional label, tenant binding and expiry; GET /v1/admin/credentials lists them and POST /v1/admin/credentials/{id}/revoke revokes one, durably, from the next request. Kept as SHA-256 digests in credentials.jsonl (mode 0600, fsynced per write, outside generations and pins), beside TRAZA_TOKENS entries, which list as source: "env". An open store is closed by minting its first credential — which must be an unbound admin — and stays closed across restarts; revoking every credential locks the store rather than reopening it. See administration § Runtime credentials.
  • Tenant limits: POST /v1/admin/tenants/{tenant}/limits (and GET; _default names the default tenant) replaces a tenant’s monthly_spans, monthly_bytes, ingest_paused and ttl_seconds, durable in limits.json. The runtime TTL outranks --tenant-ttl and --ttl-seconds and applies at the next minutely retention pass, with no restart.
  • Usage counters: per tenant, lifetime and current-UTC-month spans, bytes (each span’s native JSON before payload offload), batches and refused spans, at GET /v1/usage in O(tenants); a bound credential sees its own row. They are written to usage.json every five seconds by a thread of their own and on every flush, so a crash undercounts by about five seconds’ worth and never overcounts. /v1/metrics gains traza_tenant_spans_ingested_total{tenant} and traza_quota_rejected_total{tenant}.
  • Quota enforcement: a batch that would take its tenant past a monthly limit, or any batch of a paused tenant, is refused whole with 429, Retry-After (the seconds to next month, 60 for a pause) and a body naming the limit, the month’s usage and when it resets — on native and OTLP ingest alike. Reads, exports and erasure are never blocked. The check reserves nothing, so batches admitted concurrently can overshoot a limit by what was in flight. See administration § Quotas and usage.
  • Admission caps, on by default: --max-span-attributes (1024), --max-attribute-key-bytes (256), --max-span-events (1024), --max-span-bytes (4 MiB, measured as stored, after payload offload) and --max-batch-spans (100,000). A violating batch is refused whole with 400 naming the cap and the span, before anything — a payload file included — is written. 0 disables each.
  • Gzip request bodies: Content-Encoding: gzip (or x-gzip) on POST /v1/spans and POST /v1/traces, what an OTLP exporter sends with compression on. The inflate is the crate’s own, so this adds no dependency; every member’s CRC-32 and length are checked, and the decompressed size is held to the 64 MiB body cap (413 past it). Any other coding, or gzip on another route, is 415.
  • GET /healthz ({"status":"ok","version":"0.27.0"}) and GET /readyz, answered before the auth gate and left out of the request metrics; GET /v1/whoami tells any caller what its own credential is.
  • Operator hardening: in open mode, admin operations refuse cross-site browser requests (see Changed); a request admitted in open mode whose body arrives after the first credential was minted is authenticated again and answered 401; of two bootstraps racing, exactly one creates a credential; a failed append to the credential or fuse log is truncated back out instead of being left for the next record to merge with; and a non-loopback bind counts a runtime credential already in the data directory as configured authentication.

• MCP: four read tools over runs. check_run_guard (not listed under --no-fuses), get_run_tape, compare_runs and recall_runs, with stored text kept inside the untrusted-telemetry block as everywhere on that surface. There is deliberately no replay tool: letting the model under test create one would let stored telemetry choose which recording its next run is fed. Sixteen tools in all: fourteen reads and the two gated writers. See the MCP guide.

• Dashboard: five screens, and a tenant-aware page. docs/guide/trace-browser.md.

  • Guard (runs by verdict, usage against limits, reasons and evidence, the kill switch, reset with a grant, policies and the global default), Run (the tape with request and response previews and payload loading, a replay with both SDKs’ environment lines, compare-with), Diff (aligned steps, the first divergence explained, totals and delta), Ledger (head, verification, entries, anchors, and an evidence pack saved as the server’s exact bytes) and Credentials (mint with the token shown once, list, revoke). Sessions and traces link to their run.
  • The page asks GET /v1/whoami before anything that depends on it. A tenant-bound credential never calls an operator surface — the API client refuses those paths in the browser — reads its pulse and counts from its own GET /v1/usage row, and is not offered Server, Store or Credentials. After a 401 every poll stands down until a token is set, and a 403 names the grant the server said was missing.
  • A token can be handed over in the URL fragment (#token=…, or a route’s token= parameter), taken before the first render and on fragment navigation, and removed from history with history.replaceState.
  • <meta name="traza-api-base"> in the built index.html prefixes every request, stream, MCP call and copyable snippet, for a proxy that mounts the store under a path; see deployment.

• Client libraries in sdk/: traza for Python 3.9+ (standard library only, typed dataclasses with py.typed, clean under mypy --strict) and @trazadb/sdk for TypeScript (Node 20+, Deno, Bun, edge runtimes and same-origin pages; no runtime dependencies; a lossless JSON reader, so every *_ns arrives as a bigint). Both cover whoami and usage; the fuse check and a guard that raises FuseTripped before a stopped step starts; the other fuse routes and the event feed; tapes; replays with absolute base URLs and an env() for the provider SDKs; diff; recall; the ledger head; and evidence packs with digest verification — plus the OTLP exporter environment. They are not instrumentation, they refuse redirects rather than carry a token wherever a 3xx points, and they install from this repository; neither is published to a package registry. Both carry the engine’s version, and scripts/check-sdk-metadata.py fails the gate on a mismatch. See docs/guide/clients.md.

• Three demos: guard, replay and prove, nine in the tour now. guard stops a looping agent between steps, refuses its next step at the gate, trips a healthy run, approves more budget with a grant and runs the Claude Code hook; replay records a run, replays it through OpenAI-wire agent code with the stand-in model switched off, has a changed prompt refused at step 0, and diffs it; prove recomputes every ledger head with hashlib, records an anchor, ledgers an erasure by digest, and has verify --ledger name one flipped byte and exit 2. Each runs in seconds and asserts its own claims. ci.sh runs all nine from a checkout path containing a space, and points the three new ones at the binary it just built rather than one inherited from the caller’s shell.

Changed

• /v1/backups/* needs an unbound admin credential. A pin holds every byte of its generation on disk until it is released, so taking one is a capacity decision rather than a write. An rw token now gets 403 {"error":"forbidden","required":"admin"}, a bound one "required":"unbound". The pin still carries no credentials.jsonl, limits.json or usage.json, and a restore leaves them as they are.
• Everything under /v1/admin/ is unbound-admin only, by prefix. The gate refuses a bound or non-admin credential there before any route is consulted, including paths that do not exist yet, so a route added under the prefix later is denied before anyone remembers to list it. The new GET /v1/ledger, GET /v1/ledger/verify and POST /v1/ledger/anchors join the store-global operator surfaces a bound credential is refused.
• Every 403 from the auth gate carries "required" — "rw", "admin" or "unbound", the capability the credential lacked: {"error":"forbidden","required":"rw"} where the body used to be {"error":"forbidden"}, and the foreign-tenant refusal gains "required":"unbound". The error strings are unchanged. A 401 now also covers expired and revoked tokens, with the same body as an unknown one.
• In open mode, admin operations refuse cross-site browser requests. With no credential configured, POST /v1/erasures, eval version tombstones, /v1/backups/*, POST /v1/ledger/anchors and every path under /v1/admin/ answer a request whose Origin is not a loopback page with 403 {"error":"origin not allowed: …","next":"…"}; without that, any page the operator visited could mint itself an admin credential with a plain cross-site POST. Requests with no Origin (curl, scripts) and pages served from loopback — the dashboard on its default bind — are unaffected, and once a credential exists the rule no longer applies. A dashboard served to a browser under another hostname cannot mint credentials on an open store; configure one first.
• A POST with a body under /v1/admin/ must declare Content-Type: application/json, or it is 415. An HTML form cannot send that type, and a cross-site script cannot without a CORS preflight this server never approves.
• Admission caps are on by default, for embedders too (Config defaults to Some for each). A span with more than 1,024 attributes or events, a key over 256 bytes, a stored size over 4 MiB, or a batch of more than 100,000 spans — all accepted before — is refused with 400, and nothing from its batch is stored.
• Run fuses are on by default. Every acknowledged batch feeds the observer, which follows runs from their first step and holds no state for ordinary tracing traffic. The default global policy only warns (12 repeats), so no run is stopped unless someone configures a stop; ingest responses can now carry a fuses field and a Traza-Fuse header. --no-fuses (for an embedder, config.fuses.enabled = false) restores ingest that does no fuse work.
• The retention pass runs every minute whether or not a flag configured a window, returning at once when none applies, so a runtime ttl_seconds takes effect without a restart.
• The data-directory lock reclaims a lock left by an earlier process with the opener’s PID. A lock naming the opening process’s own PID is now stale unless this process already holds the directory (a process-wide registry of canonical paths decides, so a second open within one process is still refused under any spelling of the path). On Linux, liveness is read from /proc/<pid>, so it works in the FROM scratch release image, which has no kill; elsewhere kill -0; an owner neither can check is still treated as live. This changes what two containers sharing one volume do: each is PID 1 in its own namespace, and where the second used to refuse the first’s lock — it named PID 1, its own PID, which read as live — it now reclaims it, and both write one directory. Stop the old container before the new one starts (in Kubernetes, strategy: Recreate, not a rolling update). See deployment § One writer per data directory.
• Library surface. Admission is no longer Copy and gains signals: Vec<guard::Signal>, one per run in the batch whose verdict is not continue. Config gains max_span_attributes, max_attribute_key_bytes, max_span_events, max_span_bytes, max_batch_spans (each Option<usize>) and fuses: guard::FuseConfig; a Config built by struct literal must add them, ..Config::default() need not. Error gains QuotaExceeded(usage::QuotaRefusal) and Unauthenticated, so an exhaustive match needs two more arms. New public modules: caps, credentials, diff, guard, gzip, ledger, limits, platform, recall, replay, run, tape, usage.
• The MCP surface grows from twelve tools to sixteen. An ro token is now offered fourteen read tools where it was offered ten; the two writers are still listed only to an rw token on a server started with their flag.
• The non-loopback refusal names the new way out: configure TRAZA_TOKENS, create a runtime credential (POST /v1/admin/credentials on a loopback start), or pass --allow-unauthenticated-non-loopback explicitly. Running open on a network is now documented for what it is: the first client to mint a credential owns the store, persistently.
• Ingest serializes every span once more before admitting it, to enforce the admission caps and to charge usage bytes, on the request’s own thread and outside every engine lock. The published ingest throughput records predate this pass and have not been re-run; an interleaved before/after run on a heavily loaded host could not separate its cost from run-to-run noise, which is not the same as showing it costs nothing. See capacity. No performance figure in this release is new.
• Version metadata: the crate, the UI and both client libraries move to 0.27.0. The MSRV stays 1.81, and a default build still has three direct dependencies.

Fixed

• A restarted container opens its store again. The release image runs the server as PID 1, which ignores the SIGTERM of docker stop and docker restart, so SIGKILL left the lock behind naming PID 1 — the PID the restarted server had again, which read as a live owner — and the store refused to open on every start until the lock was removed by hand. Stale locks of other PIDs were not reclaimed in that image either, because liveness shelled out to a kill binary the FROM scratch image does not have. Both are covered by the lock change above.
• Requests answered before the auth gate are consumed before they are answered. The dashboard shell and assets, the missing-build 404 and the new probes never drained their request from the connection’s buffer, so on a kept-alive connection the same request was parsed and answered again and again, up to the 100,000-requests-per-connection limit, and the client’s next request on that connection read a stale copy of the last answer. A HEAD response also carried its body, which a client reads as the start of the next response; it now sends the head only.
• Dashboard. The theme follows the operating system’s light or dark preference until the toggle is used: the pre-paint script compared the stored "dark" with dark and never matched, and the first visit stored light, overruling the preference for good. Dialogs close on Escape, keep Tab inside and return focus to what opened them; the rail and header controls are real buttons; the time ruler fits its ticks to its width, so labels no longer overprint; the hardcoded “1 node” is gone. Below 900px the rail becomes a drawer, and below 700px grids reflow and wide tables scroll inside their cards rather than widening the page.
• tests/object_storage_transfer.rs gives each test its own source directory. The name was the process id and the clock, which ticks in microseconds on macOS, so two parallel tests could open the same directory and one failed AlreadyOpen — 4 runs in 40, enough to break the object-storage leg of ci.sh.
• Links to storage-comparison’s removed projection section in docs/segment-format.md and docs/internals/dependencies.md point at where the savings come from, and the sentences around them say the projection lived on that page before 94e7f86 rather than that it still does.

Upgrade

• No migration. Segments, the write-ahead log and generation manifests are written exactly as 0.26.0-preview.1 wrote them, so its data directory opens as it is. New files appear beside them as they are first needed: ledger.jsonl (at the first open, with a genesis entry and the live generation’s checkpoint), fuses.jsonl, credentials.jsonl, limits.json and usage.json. All of them live outside generations; a pin carries only the ledger’s copy.
• Move backup jobs to an unbound admin token, and anything that calls /v1/admin/* with a bound or non-admin one. A client that compares a 403 body exactly must accept the required key.
• Check producers against the admission caps. A batch that used to land and now gets 400 names the cap and the span; raise that cap, or set it to 0 to disable it.
• Embedders must add the new Config fields to struct literals, two arms to exhaustive Error matches, and stop relying on Admission being Copy.
• Containers: a docker restart no longer needs the lock removed by hand, but two containers must not mount one data volume at the same time — nothing refuses the second any more.
• Rolling back to 0.26.0-preview.1 reads the same data, but that binary knows nothing of the new files. It ignores credentials.jsonl, so a store closed only by runtime credentials serves open again on loopback (and refuses a non-loopback bind); keep TRAZA_TOKENS set if you may roll back. Fuse policies, tenant limits and quotas likewise stop applying until 0.27.0 runs again.

[0.26.0-preview.1] - 2026-09-06

Preview: an object-storage snapshot archive behind the new, default-off object-storage cargo feature and the feature-gated traza-object binary. A verified pin publishes to S3-compatible storage as an explicitly named, immutable snapshot — manifest written last under a conditional create, every object read back and SHA-256-verified before it is named — and reads back three ways: queryable in place by chunk-verified range reads (exact engine semantics: last-write-wins across segments, tenant scoping, content search, cursors, payload retrieval), verified full restore into a fresh directory, and snapshot administration (list, inspect, verify, delete, cleanup).

Added

• object-storage feature: traza::object_storage (Rust API) and traza-object (CLI) for publish/query/restore/delete against S3-compatible backends via object_store (env credentials, TLS verifying by default, bounded per-request and per-transfer budgets with retries, plain HTTP only by explicit opt-in), plus injectable in-memory and fault backends for tests.
• Deletion is fenced by permanent snapshot tombstones: a deleted (or swept-after-abandonment) snapshot id is never reusable, which is what makes delete and cleanup safe against concurrent or resumed publishers across machines without a lock service. Sweeping an abandoned upload requires an explicit publisher-quiescent acknowledgment, retires the id, and the publisher re-checks the tombstone and its own upload marker around its commit.
• Every remote manifest is bound to the generation manifest it archived before a query is served — an omitted or substituted segment fails the open instead of shrinking answers — and the archived tombstone log is strictly re-parsed for pending erasures. Store identity is checked on every read, verify, delete and cleanup; publish reports a manifest_sha256 to retain externally and enforce via --expect-manifest-sha, which is the stated trust root (the bucket is not assumed Byzantine-proof).
• Source pins are validated before anything they name is opened: canonical file naming, symlink refusal, and refusal of unmanifested or live-store files, so a stale manifest can never silently archive an older subset.
• segment::RangeSource and Segment::open_from_source: local file opens and remote opens now share one implementation, so a remote segment gets the full v8 validation story (header CRC, section checksums, timestamp grounding, per-read block/content-page CRCs) by construction. Remote reads additionally verify a per-chunk SHA-256 table from the snapshot manifest, including cache hits, under one global bounded chunk cache.
• Store::pin_for_object_archive (feature-gated): pin + verify + refuse pending erasures — the pin publication requires and independently re-checks from the pin’s own tombstone log.
• Operations guide: docs/operations/object-storage.md, stating the scope honestly — archives are historical immutable state, not tiering; local TTL/erasure never reaches published snapshots (delete the snapshots, and mind bucket versioning/object-lock); remote querying saves the archive copy’s disk, not the live store’s.

Changed

• pin_generation now checkpoints and links under one maintenance/seal acquisition, closing the between-locks gap in which compaction could replace a just-manifested file and fail the pin.
• Version metadata: crate and UI move to 0.26.0-preview.1. The standalone MSRV stays 1.81; the object-storage feature’s resolved toolchain floor is recorded in package.metadata.traza.object-storage-rust-version.

Fixed

• Incremental SHA-256 preserves a buffered partial block across small or empty updates. Object verification now remains correct across arbitrary network chunk boundaries; one-shot digests and the storage format are unchanged.

[0.25.0] - 2026-09-06

A lossless storage-efficiency release: format v8 compacts persisted indexes while preserving spans, payloads, query semantics, and retention settings.

Measured on the three bundled synthetic corpora (paired same-machine before/after runs, identical ingested bytes; not a forecast for arbitrary traffic): whole-data-directory size fell 44.2% on generic (132,663,519 → 73,968,447 bytes for 1M service-trace spans), 31.4% on llm (107,692,288 → 73,909,976 bytes for 200k LLM-call spans), and 38.1% on pinned-context (4,283,487 → 2,650,416 bytes for 10k long-context spans). The full normalized export of all 1.21M baseline spans is byte-identical after migration, and every migrated store verifies intact. Evidence, latency pairs, and reproduction commands: docs/benchmarks/storage-v8-comparison.md.

Changed

• Record offsets and trace/attribute postings use delta varints. Metadata sections use LZ4 when it produces fewer bytes, with raw passthrough otherwise. Record compression and content-addressed payloads retain their existing codecs.
• Segment headers and all metadata used to exclude query results carry integrity checks. Content-index pages are also checked when read after the segment opens.
• Format v6 and v7 stores migrate automatically to v8, including pins and their manifests. Migration preserves filenames and WAL fold boundaries and proves complete coverage of legacy records before replacing a segment.

Fixed

• Corrupted persisted query indexes can no longer silently hide matching spans at open. Malformed geometry and noncanonical varints are refused.
• Legacy offset metadata that omits intact records is refused instead of publishing an apparently healthy but incomplete migrated store.

Upgrade

Migration changes files in place through atomic replacements and resumes after interruption. It is one-way from the first converted file. Stop the old server and retain a complete cold backup plus a tested compatible binary before opening existing data with this release. Downgrade requires restoring that backup; replacing only the executable is insufficient. See the format and migration contract.

0.24.2 - 2026-08-31

The follow-up an external review of the v0.24.1 line earned: the query deadline now reaches the sidecar rebuilds and buffered sweeps it missed, the v7 header’s timestamp range is validated against real records instead of trusted, and the filter benchmark refuses to publish a latency for an answer it did not verify. Every finding was reproduced red before its fix.

Fixed

• The query deadline reaches the rollup sidecar rebuild. External review of v0.24.1 found two query paths that check the budget per segment and then rebuild a missing rollup sidecar INSIDE the segment with no check at all — the scoped payload fetch’s reachability probe and the analytics fold’s fast path. A 50k-span rebuild ran to completion under a 1 ms budget (~1 s of decode) and answered as though the probe had been cheap. The rebuild’s record-decode loop now runs under the calling request’s budget at the standard cadence, with a final check before it returns; second-round review found the sibling — the supersede prefilter’s key-hash rebuild, reachable per emitted span on every query and fold — and it is threaded the same way. The maintenance TTL sweep passes no budget, exactly as the deadline contract exempts it, and the shadowed-tail scan never rebuilds at all (a segment without a sidecar ends that scan).
• Buffered spans no longer escape the query deadline. The same review found every buffered-span sweep unchecked: an unflushed 20k-span corpus answered complete results under a zero budget on the unlimited query, fold, session-union, and analytics-rollup paths, and the limited query built its whole buffer source before the merge’s first check. Every buffer sweep on a query path now checks at the standard cadence, and every budgeted query path performs one final check before returning a completed answer — a request past its budget is refused, never rewarded for finishing late. The deadline’s cost statement in docs/configuration.md is amended to match.
• The segment header’s timestamp range is validated, never trusted. The same review demonstrated the harm: the two range words carry no checksum, may_contain_timestamps pruned whole segments on them before any validation, and editing sixteen header bytes silently hid a record the store could still decode — the false-negative predicate the corruption contract forbids. Both opens now prove both declared bounds against record bytes: the eager open against every record, the lazy open by decoding the first block (whose fence-vs-first-record check pins the declared min to the actual first record) and reading the last record’s timestamp — the true max, records being timestamp-sorted — which must equal the declared max. Second-round review forced that grounding: pinning the min to the first FENCE alone could be colluded with, since the fence is unchecksummed metadata too — raising the header word and the fence together still silently hid the head window. A metadata-only forgery of the range is now refused at open on either bound; a forgery must alter record bytes plus the block crc32 to get past it, which the validation’s own decode catches. An empty segment must declare the canonical empty range, and any record read outside the declared range still fails loud as corrupt. Deliberate amendment #5 in docs/segment-format.md states the validation and its cost — two bounded block decodes per lazy open, dropped from the cache after.
• The canonical benchmark’s filter gate gained a semantic oracle. Review also noted the bench recorded filter latency on any 200 with parseable JSON — a wrong or empty answer would have been timed and published as a healthy number. Each filter sample now asserts the exact span count the corpus construction defines, the requested benchmark.group value on every returned span, and the exact expected span ids, before its latency is recorded; a wrong answer aborts the run under the same refuse-to-publish rule as the gates.

0.24.1 - 2026-08-31

Fixed

• The compaction keeps-up test’s tick deadline is sized for v7 merges. The bound that proves a tick merges the backlog it found and returns — rather than merging what arrives for as long as the writes do — was twenty seconds, sized before v7 made every merge re-encode lz4 blocks. The v0.24.0 tag run watched a starved shared runner push one legitimate tick past it and fail release preflight, so the v0.24.0 tag exists but published no artifacts on any channel; this release is the v7 line’s first shipped artifact. The tripwire is two minutes now, and the comment says which failure the clock is for.

0.24.0 - 2026-08-31

The storage release: segment format v7. The same store, at less than half the disk — measured by gates the benchmarks now enforce on themselves before they will write a number down. Records stop storing every indexed attribute value twice, the records region compresses into 128 KiB lz4 blocks behind a validated directory, payload blobs compress under an unchanged content address, and a v6 store migrates automatically at first open: resumable, same names, pins included, proven by an aimed-SIGKILL crash matrix. Settled amplification lands at 0.41x of ingested bytes on service traces and 0.23x on LLM calls, against 1.81x and 2.07x before; ingest is faster because compression writes fewer bytes; a point read pays one block decode, priced and bounded in the format’s own acceptance gates. lz4_flex is the third direct dependency, carried by the ledger in docs/internals/dependencies.md, and it raises the minimum supported Rust to 1.81.

Changed

• Segment format v7: the records region is compressed, and attribute value text stops being stored twice. A v6 store is migrated automatically at first open — no command, no flag; back up first, open, done. The migration is resumable by construction (every file conversion is atomic onto its own name), survives kill -9 at arbitrary points, converts pinned backups and rewrites their manifests, and completes with a full-re-hash checkpoint that declares the format. What the format buys: records carry a fixed 20-byte (key id, value digest) pair instead of each attribute’s key and value text (the text lives once, in the payload, from which the pair list is provably re-derivable), the records region is carved into 128 KiB record-aligned LZ4 blocks with per-block CRC-32 and a resident block directory, and payload blobs gain a TRZBLOB1 header with the same codec-or-raw rule while keeping their uncompressed-bytes content address, so dedup and erasure needles are unchanged. The claims are published: settled amplification measured 0.41x on generic and 0.23x on llm (v6 shipped at 1.81x and 2.07x), recorded in benchmarks/storage.md by a run that asserts acceptance gate 5 — amplification at or below 1.0x on both corpora — before it will write, and the canonical latency benchmark asserts gate 6’s tripwires the same way — trace-lookup p50 at or below 0.75 ms and attribute-filter p50 at or below 6 ms (measured 0.425 ms and 2.974 ms) — before writing benchmarks/canonical-corpus.md; both gates are defined in segment-format.md, and a run that misses one exits non-zero with nothing written. Adversarial-review hardening shipped inside the same change: every header- or directory-declared length is bounded before it can size an allocation (a flipped high bit used to abort the process at open instead of erroring), the block directory’s min-timestamp fences are verified against the records they describe (a sorted-but-corrupt fence used to shift time windows silently), and a digest-moved pinned segment is re-accepted on migration resume only after a full validation of every block. The shared CRC-32 (src/crc.rs) is computed slice-by-16 from compile-time tables: it now fronts every block read, and read-path profiling showed the earlier byte-at-a-time fold as the single largest term in trace-lookup and attribute-filter latency — the sliced form keeps the checksum a rounding error beside the LZ4 inflate it precedes, with bit-identical output.
  • Library surface: segment::Codec and segment::encode_with_codec are new (the raw codec is format-supported; store-written segments are fixed at LZ4 in this release), segment::Error::TooLarge carries a String naming the offending field or record instead of a &'static str, Segment::resident_bytes / Store::resident_payload_bytes now count the decoded-block cache (zero at open and flush, bounded by four blocks per open segment afterwards), segment::BlockWalk (via Segment::walk) is the request-scoped decoded-block memo every in-order decode loop reads through — it bounds a walk’s inflations to one per block even when a concurrent scan thrashes the shared cache — and Store::open may perform the v6 → v7 migration before returning.

Added

• lz4_flex =0.14.0 — the third direct dependency, and the first since serde/serde_json. LZ4 block compression for segment records and payload blobs; pinned to an exact version because compressor output bytes are format bytes under the acceptance tests (upgrading the pin is a deliberate re-baseline, never a routine bump). Built with default-features = false and the safe encode/decode features only, so the crate-wide #![forbid(unsafe_code)] posture keeps its meaning. It raises the minimum supported Rust from 1.75 to 1.81, recorded in the ledger. The written justification lives in docs/internals/dependencies.md; the lockfile grows from twelve packages to thirteen.

0.23.0 - 2026-08-26

The release where the production obligations land, and the first artifact to carry them: everything between v0.22.1 and here ships for the first time, including the 0.22.2 section below, whose tag was never cut. Generations and checkpoints with hot backup and one-flag restore; erasure with a receipt that re-checks every domain by name; tenant identity in the primary key; the eval entity model; attribution that names the failing step; a compute deadline on every query and a connection slot no panic can leak; and a six-demo tour that CI holds green. The format pays for it once: segment formats 1–5 are spent, and a pre-generation store migrates on first open, automatically.

Added

• --query-deadline-ms (default 30000, 0 disables): a wall-clock budget one query or aggregation may spend inside the engine, checked at segment boundaries and every few thousand decoded records. Past it the request answers 400 {"error":"query deadline exceeded: ..."} with no rows — the refusal that stops one runaway scan (an unindexable content search, a whole-corpus fold) from holding a handler thread for minutes, and never a partial aggregate, because a partial answer would look complete. A 200 still means what it meant — every answer served is still the whole answer — and durability is untouched: ingest, the tail, exports (bounded by the socket timeout), and erasure work are all exempt. The worst measured legitimate query in this repo is ~3 s, so the default never touches measured use. traza_query_deadline_exceeded_total counts the HTTP refusals; an MCP tool that exhausts the budget surfaces the refusal in its tool result instead, uncounted.

  • Library surface: Config::query_deadline is a new field (None — the unbounded behaviour every embedder had — by default; a Config built by struct literal must add it, ..Config::default() need not), and Error::DeadlineExceeded is a new variant of the exhaustive Error enum, so a downstream exhaustive match must add an arm.

• A demo tour in examples/ — six scripted proofs, one claim each. Five new demos join mcp-demo, and examples/README.md indexes the tour: swarm streams a simulated agent platform into a live dashboard in real time; crash SIGKILLs the server mid-ingest and proves every acknowledged span survived, then corrupts a byte, watches /v1/verify name the file, and restores from a hot backup; needle ingests a million spans over real HTTP and times the answers; incident investigates a runaway session over MCP — diagnose_session, a defused prompt injection, promote_failures_to_dataset, and an experiment diff proving a fix; vanish erases one tenant’s session next to an identical twin tenant and walks the receipt, including the backup pin it refuses to ignore. Every number each demo prints is measured during the run — a timed request, a counter, a field from a live response — and each script asserts its own claims and exits non-zero when one fails, which is why ci.sh now runs all six at their smoke settings as merge gates.

  • Demos refuse to run against a port another server already holds: readiness is the child’s own startup line, not a probe that a squatter can answer. This guard also landed in mcp-demo.

• Attribution: the server names the failing step, instead of the reader doing it. A new diagnose_session MCP tool answers why did this run fail — the outcome, the step the failure is attributed to, and the repetition behind it — with the evidence for each claim travelling beside it. It replaces the two prompts that used to teach a model to read a trace and judge its shape by eye (“repeated sibling names are a retry storm, deep chains are a loop”), on the surface’s own rule that a prompt wanting a branch is a tool.

  • Every rule fires on telemetry that already exists. An earlier design rested on a session.outcome attribute and a relation: "retry-of" link — both conventions Traza invented and nothing else emits — which would have been a closed loop: seed the convention, detect it, pass CI, and answer cause: null on every real store. A declared convention now raises confidence and is never a precondition. The discriminators are error density, serial fraction, self-similar depth and context growth.
  • Context growth is read from context_tokens, not prompt tokens. gen_ai.usage.cache_read_input_tokens and cache_creation_input_tokens are recognized, because Anthropic reports input_tokens as the uncached remainder: a growing conversation reports a prompt count that falls as its cache warms, so reading the wrong field inverts the signal on the configuration long-running agents actually use. Where the arithmetic is unknown the trend is reported as unreadable rather than guessed.
  • It refuses to say more than the data supports. A shape is classified only when a discriminator agrees; otherwise it is reported as inconclusive with the missing signal named. Ordinary iteration is reported as ordinary so a reader can see it was examined and set aside. Absent token data can never satisfy a healthy classification. The strongest test is silence: the analysis finds no fault anywhere in the bundled corpus of healthy workloads.
  • Session outcome is first-class and provenance-tagged: declared when a span said so (session.outcome, session.goal — Traza extensions, labelled as honestly as llm.cost_usd), derived from the run’s own spans otherwise, and unknown — never rendered as success — for a run still in flight. A run that failed a call and recovered is a success that reports its errors, not a failure.

• promote_failures_to_dataset closes the loop into the M4 eval entities: a session’s attributed failures become a regression dataset version, each example carrying its own copy plus provenance back to the span. Re-promoting is idempotent. Gated by an rw token and a new --mcp-promote switch, separate from --mcp-annotations because an annotation dies with its span while a promoted example deliberately outlives its source. The caller names a session, never a span: the promoted set is re-derived server-side from the diagnosis, so text injected into the telemetry can change whether a promotion happens but never what it copies — proven by a test that plants exactly that instruction.

• A runaway agent scenario in the seed corpus: an agent whose search tool fails from the fourth turn on, so every later reflection appends the failure to its context and asks again. Its steps are siblings under one root, the shape real frameworks emit, and it carries no declared outcome and no retry link — so finding it proves the analysis works on what a real pipeline sends.

• Tenant identity in the primary key. Span identity is now (tenant, trace_id, span_id) — everywhere: the write buffer’s index, WAL replay, segment supersede resolution, compaction’s last-write-wins merge, the tail ring’s veils, cursors, the analytics key hashes, annotations and erasure. Two tenants sharing a trace id can no longer silently upsert over each other, which is the whole reason this ships now: keys cannot be retrofitted after the format freeze. The default tenant is the empty string and is never serialized, so a single-tenant store writes byte-identical WAL frames, segment records and annotation lines to what it wrote before tenancy existed — no segment or WAL format bump, proven by a serialization-guard test. Tenant scoping reaches every surface the roadmap names:

  • Credentials: TRAZA_TOKENS entries take an optional binding, scope@tenant:token. A bound credential ingests, queries, tails, exports, annotates and erases exactly its own tenant; naming another tenant is a 403, and the store-global operator surfaces (/v1/stats, /v1/metrics, /v1/verify, checkpoint/flush/backups) refuse bound tokens outright. OTLP exporters select a tenant with the traza.tenant resource attribute; MCP tools are scoped by the same binding.
  • Sessions are (tenant, session_id) — the same session.id under two tenants is two sessions, in the rollup sidecar (format v3, self-healing rebuild), the session list, and group_by=session rows.
  • Retention takes per-tenant windows: --tenant-ttl TENANT=SECONDS, repeatable. A tenant’s cutoff is its override, else the global TTL, else never — and the segment retire-whole fast path only runs when a global TTL covers everyone, because a whole-segment decision taken from configured windows alone would delete an unswept tenant’s data with the segment.
  • Quota accounting: GET /v1/tenants reports per-tenant spans, traces, serialized bytes and offloaded payload bytes from one exact fold. Accounting, deliberately not enforcement.
  • Erasure: trace/span/session subjects carry a tenant (empty = the default tenant, never “all”), and a new tenant subject erases everything a tenant owns — spans, annotations, scores, datasets, experiments — with the same barrier, ordering, and receipt discipline as M3. Reference-aware payload deletion now spans tenants: a blob two tenants share survives one tenant’s erasure and the receipt names why.

• The eval entity model — identity only, no workflow. The addressing the product thesis requires, and nothing else: Dataset (stable id, name, tenant), DatasetVersion (immutable, content-addressed manifest of (example_id, digest) pairs with a parent version for lineage and the promotion’s provenance — re-POSTing identical content IS the same version), Example (stable id across versions; input, optional expected output, split label, provenance back to the source span; bodies carry $payload references that count as live for the TTL sweep and reference-aware erasure, so a promoted copy is real for offloaded content), Experiment (stable id, one dataset version, config metadata), Run (the experiment→trace link, appended by the external harness), and Score — an annotation whose addressing was generalized to a typed subject (trace / span / session / experiment example), so a score addresses the (experiment, example, span) tuple with every existing annotation field preserved. All of it lives in evals.jsonl, a new manifested append-only recovery domain with the annotation log’s torn-tail healing and pin-by-copy discipline. Deletion semantics were settled up front and are enforced by test: erasing source traces never corrupts a dataset version (the receipt’s new eval-records domain REPORTS surviving copies and turns inconclusive — purging a curated copy is a deliberate second act); erasing a payload leaves dangling addresses in example bodies, reported retained-by-design without losing conclusiveness; a dataset-version tombstone is logical deletion with defined effects (410 with the tombstone, dependent experiments keep working and say why, new experiments refused); a tenant erasure takes the tenant’s eval records inside the barrier, and ids are never reused past the rewrite (a counter record floors the allocators). Score distributions (/summary) and experiment-over-experiment diffs (/diff) come from ordinary reads with per-(example, name) last-write-wins dedup. The whole loop — promote failing traces, run externally, record runs and scores, read distributions and diffs — runs end to end in CI against the built binary, and survives kill -9.

• One recovery domain: generations and checkpoints. Query-visible state lived in several independent recovery domains — the write-ahead log and buffer, segments, annotations.jsonl, payloads/ — each with its own durability rule and its own idea of “now”, and nothing named a state they all agreed on. That is why backup, export, retention and deletion were four mechanisms rather than one. A generation is that agreed state: a manifest naming every load-bearing file with its SHA-256 digest, plus the log position it folded through. CURRENT names the live one, and moving it — a staged rename made durable by a directory fsync — is the single commit point for a checkpoint, a restore, or a published deletion.

  • Backup runs against a live server: POST /v1/backups/{label} checkpoints, hard-links the manifested files into pins/{label}, and verifies every digest before reporting success. Hard links share inodes, so a pin costs almost no disk and holds its bytes even after compaction unlinks the originals — the copy proceeds at its own pace, and POST /v1/backups/{label}/release frees it. The pinned set carries spans, annotations and payload bytes together, which is what a span export never could.
  • Restore is install: traza-server --restore DIR (or Store::restore) verifies the backup before swapping anything, and commits at one CURRENT rename, so a failed or interrupted restore leaves the prior store rather than a blend.
  • GET /v1/verify re-digests the live generation and names each discrepancy, so recovery can distinguish damage it may ignore from damage that changes what the store contains by asking rather than inferring it from whether parsing happened to succeed.
  • POST /v1/checkpoint, plus a checkpoint every five minutes from the maintenance thread. Cheap by construction: segments are immutable, so a checkpoint carries their digests over from the previous manifest and hashes only what was written since.

• Log frames carry the generation they belong to. The framing gains an eight-byte magic and a (epoch, sequence) stamp — a 24-byte header, up from 8. CURRENT and the log are separate filesystem objects, so no rename can make “the new generation is live” and “the frames folded into it are gone” one event; a crash between them meets a log still holding folded frames, and for a checkpoint that published a deletion, replaying them resurrects exactly what was deleted. Recovery now replays a frame only when its stamp is strictly after the live generation’s folded_through, which demotes log reclamation from correctness to housekeeping. Sequences are monotonic for the life of the store — a rewrite keeps counting rather than restarting — because a stamp landing at or before a recorded fold point would be discarded by the next replay as though it had been folded.

  tests/generations.rs proves it against the crash state itself: the log’s pre-checkpoint bytes are written back over a committed generation, and the test fails when the stamp rule is removed.

• Existing data directories are adopted at first open. Nothing moves — segments stay exactly where they are, because their paths are load-bearing — and generation one is published over the files already there, converting a pre-generation log to stamped framing on the way. One-way, resumable, and committed by the CURRENT rename, so a crash mid-adoption leaves a directory the next open finishes.

• Targeted deletion, with a receipt. POST /v1/erasures erases a subject — a trace, a span, a session (resolved across every recognized session key), or one offloaded payload — and the new traza-server verify --erasure (or GET /v1/erasures/{id}/verify) then proves it: every domain the subject’s bytes could inhabit, checked by name, with the result of each. TTL removes by age; this removes because someone is entitled to have it gone, and the receipt is the difference.

  • The tombstone log (tombstones.jsonl) is a new manifested, append-only recovery domain beside the annotation log. The intent record is fsynced before anything is removed, so from that moment the subject is invisible to every read path — search, lookups, sessions, analytics, annotations, payload fetches, export, the live tail — even before the rewrites run, and a crash mid-purge leaves a pending erasure the next open masks and the maintenance tick finishes. Idempotent to resume: the purge re-verifies rather than re-damages. (The settle record’s counts are therefore the settling pass’s counts; the receipt, not the tallies, is the authority on absence.)
  • A pending erasure is an admission barrier, not just a veil. Covered spans are dropped at ingest — acknowledged, never stored, counted in traza_erasure_spans_suppressed_total — and the barrier is total: it runs BEFORE payload offloading, so a suppressed span leaves no orphan payload bytes behind (a review probe planted exactly that orphan — the file of a span that never entered the store, invisible to record and receipt alike); an oversized value whose content hash is itself under erasure offloads directly to its redacted marker instead of recreating the file being deleted; and annotations addressed to a covered subject are dropped at admission the same way. The barrier holds at the BEGIN transition, not just in the steady pending state: admissions hold an erasure gate in read mode from their mask load through their store mutation — the offload’s file writes included — and begin/settle move the mask only under its write mode, so a batch or annotation is wholly before an erasure or wholly after it, never astride it. A one-shot mask check could not say that, and review was right to test the transition. The cut is exact: every span acknowledged before settled_unix_ns is erased or was never stored, under concurrent writers hammering the subject (a review probe demonstrated 92 pre-settle survivors before the barrier existed; tests/erasure.rs keeps the probes as regression tests, plus a transition stress test that erases in a loop under payload-writing and annotating threads and then audits the payload directory itself for orphans). The ingest response tells the truth about it — accepted counts what was stored, suppressed appears when nonzero, and the OTLP surfaces answer with partialSuccess.rejectedSpans (hand-encoded for the protobuf path, because an empty response is a claim of full success).
  • Nothing rewrites a manifested file after the checkpoint the settle cites. The confirm purge and annotation drops moved BEFORE the checkpoint; the settle append itself rides the append-only allowance every manifest grants the tombstone log. A review probe caught the old order making the settle’s own generation fail verification (annotations.jsonl: digest mismatch) the moment a concurrent annotation landed; the ordering is now barrier → purge → confirm → checkpoint → settle → lift, and the suite asserts the cited generation verifies clean, including under concurrent writers.
  • The purge reaches every domain. Buffer and write-ahead log rewritten to the survivors (the TTL discipline: a deletion a restart undoes is not a deletion); segments rewritten in place, superseded versions of an erased key included — the purge tests every physical record, not the visible one; annotations addressed to erased spans dropped; payload files deleted reference-aware, because content addressing means one file can back spans outside the subject — those bytes are retained and the receipt names the reason. A payload subject additionally rewrites every referencing span to drop its inline preview: the preview is content too. Publication is a checkpoint — the deletion is durable when CURRENT moves, and tests/erasure.rs proves it by killing the server after its 200.
  • The receipt is a verification, not a claim. Its result is computed from the walk, never from the settle record. Matches are classified against the erase record’s resolved keys under one rule for every domain: an erased key found live again is a re-delivery and fails the receipt; a fresh key under the same identifiers is new activity, reported without failing it — an erasure is a barrier, not a ban. Pins are checked and named: a backup pinned before the erasure still holds the bytes in its hard-link farm, and the receipt says which pin to release. What remains afterwards is stated rather than hidden: the tombstone record itself keeps the subject’s identifiers and content hashes — never the erased text — as the record the receipt verifies against. Where a check is over-approximate by construction (the byte-level occurrence scans), its findings never blend into the verdict: the receipt carries a separate conclusive flag, and the subcommand exits 0 erased-and-conclusive, 3 erased-but-inconclusive, 2 not erased.
  • Pins no longer share the append-only logs’ inodes. A pin hard-links the immutable files and copies annotations.jsonl and tombstones.jsonl at their manifested length. Hard-linking them let every later append edit the “backup” in place — a pin taken before an erasure inherited the erasure’s settle record through the shared inode, so restoring it produced a store that recorded a deletion it did not contain.
  • Erasure requires a new admin token scope. TRAZA_TOKENS grows admin: alongside rw:/ro:; POST /v1/erasures refuses plain rw with a 403. Every collector holds a write token, and a credential minted to produce telemetry must not be the credential that destroys it. Payload subjects are canonicalized to lowercase hex before anything is resolved or recorded — an uppercase hash previously matched no stored reference and produced a green receipt over untouched content. Redaction markers are no longer counted as live payload references anywhere (protection sets, receipts): a marker records that content is gone.
  • No MCP tool for any of it, deliberately. Deletion is an HTTP verb behind the admin scope; the agent-facing surface stays read-only, so stored adversarial text has no destructive tool to actuate.

• Derived LLM cost from a configured pricing table, --pricing FILE. OpenTelemetry defines no cost attribute, so a span carries one only if its pipeline metered it — and most do not, which left stores that knew the model and both token counts reporting $0.00 on every cost surface. Rates are USD per million tokens, keyed by exact model name or a prefix* pattern (longest match wins; an exact name beats every pattern; a bare "*" is a default). There is no built-in table and will not be one: prices move on the vendor’s schedule, and self-hosted models have no public rate.

  • A metered llm.cost_usd always wins, and a span reporting only a total token count stays unpriced rather than being split by an assumed input/output ratio.
  • Estimates are reported as estimates, by count rather than by amount. Every cost-bearing row — /v1/stats/llm, /v1/sessions, and each bucket of /v1/stats/series — carries cost_derived_usd beside cost_usd plus cost_metered_calls, cost_derived_calls and cost_unpriced_calls. The dollars cannot carry provenance on their own: a zero-rate model is priced and adds nothing, a call nothing could price also adds nothing, and neither is distinguishable from a genuine measurement of zero. cost_derived_calls > 0 means the total is an estimate; cost_unpriced_calls > 0 means it is an undercount. The dashboard and MCP both mark an estimate ~, show — rather than 0.0000 when nothing could be priced, and state the breakdown — and analyze_cost no longer claims cost is exact when a rate table contributed to it.
  • Rollup sidecars record the fingerprint of the table they were folded under (format v4), so editing the rates invalidates exactly the cached counters that would now be wrong instead of reporting last month’s prices from a sealed segment forever. An empty table fingerprints to zero, so a store that prices nothing binds its sidecars exactly as before. A malformed pricing file refuses startup rather than being ignored.

Changed

• The erasure receipt’s failing verdict is not-erased, not incomplete. A receipt over a pinned backup reported result: "incomplete" beside conclusive: true — semantically right (the walk conclusively found the pin holding the bytes) but reading as a contradiction, because “incomplete” invites attaching to the verification when it describes the erasure. The value now names its subject: conclusively not-erased and erased-but-inconclusive both read as the claims they are, the wire value matches the prose the docs and the verify usage text already used (2 not erased), and the exit codes are unchanged.

• Ingest rejects an inadmissible tenant with 400, on both surfaces. A tenant is identity: lowercase [a-z0-9][a-z0-9._-], at most 64 bytes, or empty for the default. A misconfigured traza.tenant resource attribute fails the whole OTLP export loudly rather than being silently dropped — partialSuccess is for data the server chose to suppress, not for a defect the client must fix.

• Cursor tokens carry a version byte. The ordering key changed shape (tenant joined it), and a pre-tenancy token must parse as invalid, never as a plausible wrong position. Live cursors from before an upgrade get a 400, which is what a stale cursor always deserved.

• The span identity key on the wire and on disk is $tenant, not tenant. A span’s top-level namespace is open — unknown fields survive in the round trip — so a bare tenant is client data and must stay client data. The $ sigil (as in $payload) is the format discriminator: bytes written before tenancy cannot carry $tenant, so a store read after an upgrade keeps its bare tenant values as data rather than promoting one to an identity no query selects and no erasure names. This replaces an earlier decode-time normalization that tried to tell legacy data from identity by a value’s shape — which a valid-looking legacy value ("acme") slipped through. The ?tenant= read filter and the erasure subject’s tenant field are closed namespaces and keep the plain name — but they, along with an annotation’s and a dataset’s tenant, now also accept $tenant as an alias, so a client that learned the span’s spelling cannot silently misroute a score, a dataset, or — worst of all — an erasure to the default tenant. The span alone rejects a bare tenant as identity, because the span alone has the open namespace that makes it data.

• Append-only files are digested and verified over their recorded prefix, exactly. digest_engine recorded an append-only log’s length from one instant and its hash from another (sha256 over the live file), and verify_against hashed the whole file whenever the lengths happened to match — so an annotation or settle record appended mid-walk made a freshly published generation fail its own verification with a digest mismatch that was growth, not damage. Both sides now hash exactly the recorded length, which is what the checkpoint’s concurrency story always claimed. Found by the erasure transition stress test; the race predates erasure and was reachable by any annotation landing inside a checkpoint’s digest walk.

• wal_bytes counts replayable work, not file bytes. The log’s constant preamble is excluded, so a reclaimed log measures zero. Both the flush_wal_bytes bound and the statistic mean the same thing — what a restart would redo — and a constant is neither replayed nor reclaimable.

• The checkpoint’s commit point is proven by SIGKILL, not only deterministically. tests/durability.rs kills a real server mid-checkpoint at three named on-disk signals — the manifest directory appearing, .CURRENT.tmp existing between the staged write and its rename, and CURRENT itself changing — because those windows are single fsyncs wide and a stagger alone lands wherever the machine favours. Recovery must then answer with exactly one state: every acknowledged span present, the hot key at the newest acknowledged version, the live generation verifying clean over GET /v1/verify, a complete manifest behind whatever CURRENT names, and nothing staged surviving the sweep. Publishing CURRENT before the manifest is durable fails it.

  The staged-CURRENT signal exists because a slower CI runner reached that window when a fast machine never did — and a test that only covers a state when the machine happens to be slow is a test that passes for the wrong reason. Aiming at it beats depending on the weather.

  Recorded because the opposite would be a claim rather than evidence: the other ordering rule — reclaiming folded frames only after CURRENT is durable — was mutated too and did not fail. A checkpoint seals the buffer first, so the frames it reclaims are already in a durable segment, and recovery loads segments by walking the directory rather than by reading the manifest. That rule is defensive under today’s engine and becomes load-bearing the moment segment loading is manifest-driven, which is what a replicated snapshot install wants. Invariant 12 now says so.

• Checkpointing is never a side effect of a primitive. It seals the write buffer, and expiry must not decide when to seal: expire_before deletes from every domain and stops, exactly as before. The deletion is durable when its domains are durable and published by the next checkpoint — one maintenance interval away, or immediately when a backup asks.

Fixed

• The bound-token payload probe now heals the sidecar it rebuilds. The reachability check behind scoped GET /v1/payloads/{ref}, MCP get_payload, and dataset-version promotion read rollup sidecars directly, so a segment missing one — a crash, a copied data directory, a pricing change invalidating every binding — was decoded again on every probe, forever. The probe now reads rollups through the same cache-then-sidecar-then-rebuild path every aggregation uses: the decode is paid once, the sidecar is written back, and the new traza_rollup_builds_total counter says whether heals are landing.

• A panicking handler permanently consumed a --max-connections slot. The live-connection count was decremented at the end of the handler closure — a line a panic never reaches — so each handler panic leaked one slot, and enough of them walked the server to a permanent 503 with nothing left in the logs but old panics. The slot is now claimed by an RAII guard whose Drop releases it on return, unwind, and spawn failure alike, and each unwind is counted in the new traza_http_handler_panics_total (any nonzero value is a bug report waiting to be filed).

• Two documentation claims that were not true of the code. The OTLP mapping table said “other resource and scope attributes” were merged beneath span attributes. Scope attributes are; resource attributes are discarded — only service.name and traza.tenant are read — so deployment.environment, service.version, host.name and k8s.* never reach the store and cannot be filtered on. The table now says so, names the workaround (copy them onto spans in an SDK processor or a Collector transform), and an OTLP conformance assertion pins the behaviour so the page cannot drift from it again. Separately, the MCP tool table billed record_annotation as “Score this trace”; it writes a trace/span annotation and cannot address an (experiment, example) pair, which is what a score is. The MCP guide now states that and that the eval entities are HTTP-only.

• A promotion could read across a tenant boundary. promote_failures_to_dataset pinned the tenant it wrote but not the tenants it read: an unbound credential resolves a session across every tenant, so naming one tenant’s session copied its span attributes and provenance into a default-tenant dataset — and a promoted example deliberately outlives its source, so erasing that tenant afterwards left the copy standing, permanently outside the reach of the erasure that should own it. Every read the tool makes is now scoped to the tenant being written (Some("") being the default tenant named, not the absence of a scope). Diagnosis is unchanged: an operator may still read any session, and simply cannot copy one out of its tenant.

• Prompt-cache counters were recognized under one spelling. OpenLLMetry moved them to dotted segment names (gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens) to match OTel GenAI, and only the older underscore forms were read — so on a cached Anthropic agent, the exact configuration this field exists for, the detector fell back to the uncached remainder and reported a growing context as ordinary iteration. Both spellings are accepted now, dotted first.

• Link attributes were outside every payload walker. A $payload reference copied into a span’s links[].attributes — an ordinary thing for an SDK to do, since the wire contract stores what a client sends — was counted by no reference sweep, redacted by no erasure, and invisible to the verify predicate. So a blob a surviving span still referenced could be deleted, and verify --erasure could report erased and conclusive over content still on disk and still readable. All five walkers now share one iterator naming every attribute map a span carries; the duplicate collector that made the gap exist twice is gone. Links are also bounded at ingest for the first time, with its own test. The two erasure guards are mutation-proven: with the link arm removed from the iterator, one test finds the whole secret sitting inline in the link and the other finds the payload deleted while a live reference remained.

• top_failures and slowest_spans advertised a row limit whose maximum (100) and default (20) were both wrong — they apply 50 and 10 — so a caller obeying the schema was silently given a fraction of what it asked for.

• The claim in the data-model and LLM-semantics guides that a link’s relation attribute “keeps link semantics queryable” was false — no query path reaches link attributes. The docs now say what is true: links are traversed inside a diagnosis, and are not filterable.

• A pending whole-tenant erasure withholds its payload bytes from a scoped fetch. A tenant erasure discovers its span-held references only as its purge walks them, so the mask’s payload_files set fills after the mask already hides the tenant. A GET /v1/payloads under a bound credential now returns nothing the moment its tenant is masked — whether or not that exact reference has been enumerated — closing a window in which a planted crash state served the bytes an erasure was seconds from deleting. Freshly discovered references are also folded into the live mask as they are noted, so the operator path stops serving them mid-purge too.

• The MCP describe_store “nothing to search yet” note follows the caller’s own usage, not the store total. For a tenant-bound caller the note was keyed on the store’s total_records, so it appeared only on a globally empty store and vanished the instant any other tenant ingested a span — a co-tenant presence oracle across the isolation boundary the rest of the overview respects. It now reads the bound tenant’s own row.

• An explicit default-tenant scope reaches its own sealed payload. The default (empty) tenant carries no attribute-index posting — that is what keeps single-tenant stores byte-identical — so a payload-reachability probe that trusted the posting found nothing for a Some("") scope once the span sealed. The probe now scans the records and lets the decoded tenant decide for the default tenant, exactly as select_probe already did.

• Resolving last-write-wins no longer re-reads the store to prove keys were never replaced. Every query-side supersede probe — the limited merge, the unlimited scan, and the fold behind the aggregation routes — now consults each newer segment’s own key-hash set before paying an exact probe, the prefilter the analytics fold’s exact path already ran; that fold now gates per segment too, so a key rewritten eleven segments later costs one probe rather than a walk across the ten between. On a store carrying superseded versions — a crash-recovered store before its first compaction is the canonical case — queries cost matches × segments × trace-width decodes, which is how a recovery query blew a 30-second deadline in CI. The new traza_supersede_probes_total counter is the observable: roughly one probe per superseded version actually held.

• A pre-$tenant rollup sidecar is rebuilt, never believed. Reserving $tenant changed what a bare tenant field decodes to, and a sidecar’s key hashes are evidence only under the decoding they were computed with — trusting one across that boundary treated a stale membership miss as proof of absence and resurrected a superseded span. The rollup SCHEMA_VERSION is bumped, so the first read after upgrading rebuilds each segment’s sidecar once; tests/fixtures/pr50-tenant-identity, sealed by the pre-reservation build, is the corpus that fails if a future decoding change forgets the bump.

• Native ingest accepts an event’s timestamp under the OTLP name. A span’s timestamps accepted start_time_unix_nano; its events accepted only timestamp_ns, so a client spelling both the way OTLP spells them had its entire batch rejected with a 400 naming a field it had supplied. Events now take time_unix_nano, timestamp_unix_nano, time_ns and time as aliases, and attributes defaults, so a named instant no longer needs an empty map to be legal. The failure mode this closes is quiet rather than loud: telemetry clients are conventionally fail-open, so the spans simply never arrived.

• The span search’s token column reads the current OpenTelemetry names. It carried its own third copy of the semantic-convention precedence, which had drifted from src/semconv.rs: it resolved only the deprecated gen_ai.usage.{prompt,completion}_tokens and a llm.usage.prompt_tokens key Traza has never recognized. A span using the current input/output names — resolved correctly by the server and by the trace detail — showed a blank cell. It now uses the shared llmUsage helper, as everything else does.

• Two toolbar actions on the span search worked in name only. A local named window shadowed the global for the whole component, so window.prompt and window.location read undefined through the optional chains guarding them: saving a view never asked for a name and silently numbered every one view N, and “Copy as curl” emitted a hostless URL that curl refuses.

0.22.2 - 2026-08-12

Fixed

• The dashboard’s rail reports the version that is actually running. A hardcoded prop default shipped “0.19” on every install regardless of the build; the rail now takes its version from ui/package.json at build time.

Changed

• The README leads with the install and a picture instead of a wall of sections. A real trace-browser screenshot (an agent swarm’s waterfall, from the seeded corpus) sits above four install paths — release archive, container, crates.io, source — and the HTTP API table, library example, and design prose moved out to the documentation that owns them. A social preview card in the repo’s own design system lives at docs/assets/social-preview.png.

• The release pipeline validates every channel before it publishes to any. v0.22.1 proved the previous shape only failed loudly at the end: the crates.io token was unset, and the GitHub release and container image had already published by the time the pipeline said so. Preflight now runs the full gate on both platforms plus the MSRV check, verifies the crates.io token exists, and runs cargo publish --dry-run --locked — all before a single artifact is built — and the GitHub release moved to the end, behind the crates and container publications it used to precede. Three registries cannot be atomic; prerequisites can be checked before touching any of them.

• The grouped-merge crash test’s deadline is a failsafe, not an oracle. Its 30-second query deadline doubled as the duplicate-version check, which turned shared-runner weather into red durability legs. The deterministic assertions after each query — a hot key surviving other than exactly once, a stale version outranking an acknowledged one, a batch coming back short — were the real oracle all along; the deadline is now 300 seconds and its message says what it is.

• ci.sh audits the shipped dashboard’s dependencies. npm audit --omit=dev --audit-level=high gates the merge bar; development-tool advisories print without blocking. The two open advisories (nanoid, postcss — both development-only) are resolved in the lockfile.

• Workflow actions are pinned to commit SHAs, with the tag each pin was resolved from in a comment, and Dependabot groups cargo, npm, and actions updates into one weekly PR per ecosystem. The changelog’s footer now carries the full compare-link lineage, every release back to v0.1.0.

• Future-facing documentation left the repository. docs/roadmap.md, docs/ha-design.md, and docs/generations-design.md moved to the private planning workspace and were removed from the repository and its git history — the repo documents what Traza does; where it is going is tracked elsewhere. This is the second history rewrite (the first predates going public), and it re-hashed every commit and re-pointed every tag, including the ones the 0.22.0 note below could still call bit-identical when it was written. Stated here for the same reason as last time: a lineage that quietly changed under its tags would be worse than one that says so. Published GitHub release artifacts still contain the files as they were at their release; the crates.io versions that carried them (0.22.0, 0.22.1) were deleted outright within the registry’s deletion window, and this release is the first published from the purged tree.

0.22.1 - 2026-08-07

Fixed

• The release archive’s dashboard now sits where the server actually probes. v0.22.0 packaged the UI at ui/dist inside the archive, but the binary-relative search — the packaging convention the source itself documents — looks for <binary dir>/ui. Launched from inside the extracted directory, the working-directory fallback hid the bug; launched from anywhere else, / was a 404. The archive now places the build at ui/, and every release archive is smoke-tested from an unrelated working directory — dashboard 200, span accepted and read back — before anything publishes, so this class of bug cannot ship twice.
• The README’s container quickstart no longer discards the token it mints. The old one-liner generated TRAZA_TOKENS inline, so the server started and the reader had no way into their own dashboard. The token now lands in a shell variable and is echoed before use.

Added

• --version / -V on traza-server, printing the crate version.
• Third-party notices ship with the artifacts. Release archives and container images now carry THIRD_PARTY_NOTICES.md: the MIT and OFL material for React, Inter and JetBrains Mono in the dashboard build, and the dual-licensed Rust crates linked into the binary.
• The release pipeline fails closed. A preflight job refuses a tag whose version disagrees with Cargo.toml, ui/package.json, or this file, and runs the full ./ci.sh gate on the tagged commit before anything builds; the crates.io job fails loudly when the registry token is absent instead of skipping green. A new CI job checks the whole tree on the advertised MSRV, holding rust-version to being true rather than asserted — and its first runs proved the asserted 1.70 false twice over: the locked dependency tree requires 1.71, and the test suite uses File::set_times, stabilized in 1.75. rust-version now says 1.75, because that is the number the gate can hold.
• SECURITY.md (private vulnerability reporting, supported versions, and the threat model in one paragraph), a code of conduct, issue and PR templates, and Dependabot coverage for Cargo, npm, and GitHub Actions.

Changed

• The container runs as uid 65534, with /data shipped owned by that uid so a named volume inherits it — a bind-mounted data directory must be writable by 65534 — and images carry OCI source, version, and license labels.

0.22.0 - 2026-08-07

Added

• An install, not a build. Tagged releases now ship what the README used to ask you to compile: traza-<version>-{linux-x86_64,linux-aarch64,macos-aarch64}.tar.gz, each carrying the traza-server binary (musl-static on Linux — no libc to match) with the dashboard already built in ui/dist, alongside SHA256SUMS and GitHub build-provenance attestations (gh attestation verify). The same binaries become the container image, ghcr.io/toshish/traza, built FROM scratch — an engine with two dependencies does not need a distribution underneath it. A non-loopback bind still refuses to start without TRAZA_TOKENS, in the container as anywhere else. The tag also publishes to crates.io when the registry token is present.
• The merge bar runs in public. ./ci.sh — the same script, no drift — now runs on Linux and macOS for every push and pull request, and the README carries the badge. Until this release the script had only ever gated one platform: the laptop it was written on.

Fixed

• The dashboard now renders trace media wherever the trace actually put it, instead of only the one lucky shape it recognized before. Found via a production vision-review trace whose image showed as an empty frame: the conversation view understood only data:/http(s): strings in a part’s data/uri/url, which missed most real emitters and — worse — poured every other locator into the transcript as text (a bare-base64 screenshot rendered as eight thousand lines of base64). The messages panel and the trace inspector’s payload modal now recognize, with tests: bare-base64 data lifted into a data: URI via the declared MIME type; Anthropic source.{base64,url,file} parts; Google GenAI typeless inline_data / file_data parts; OpenAI input_audio, image_url-object, and file parts; MCP-style tool-result content lists (screenshots included); width/height metadata; and parts whose bytes were never captured, which now say so with the emitter’s unavailable_reason instead of presenting an empty frame that reads as a rendering bug.

• Offloading no longer demotes media to a JSON dump. A gen_ai.{input,output}.messages attribute past --payload-threshold-bytes becomes a payload reference, and the old conversation view could only show it as preview text — “load full” printed the raw JSON, megabytes of base64 included. The payload body is the original messages JSON, so the conversation view now fetches it back (automatically up to 4 MiB, when the turn nears the viewport; on demand above that, content-addressed and cached across screens) and renders the parsed turns — an offloaded text-to-speech span plays its audio again. The trace inspector’s payload modal gained the same understanding: message payloads render as turns, media payloads as media, anything else as code, with the literal bytes one raw toggle away.

• Every MCP tool now advertises ToolAnnotations. The catalog carried names, descriptions and schemas but no readOnlyHint, destructiveHint, idempotentHint or openWorldHint — and absent hints are not neutral. Each defaults to the pessimistic answer, so nine read-only tools were advertised as potentially destructive and open-world. A host that gates on that asks for approval on every call; one running non-interactively, with nobody to ask, cancels them, which is what a Codex run did. The nine readers now declare readOnlyHint: true with openWorldHint: false, and record_annotation declares an additive, non-idempotent write. Tool nature is a required argument of the builder rather than a step that can be skipped.

• describe_store reports the server version. initialize carried it in serverInfo, but a host reads that once and need not pass it to the model, so an agent asked which Traza it was talking to correctly answered that it could not tell. The overview block — and the traza://store/overview resource that shares it — now names the version.

Changed

• The measurement records moved out of the repository root into docs/benchmarks/, and are named for what they measure rather than shouted: BENCHMARKS.md is now docs/benchmarks/canonical-corpus.md, and INGEST-BENCHMARK.md, STORAGE-BENCHMARK.md, QUERY-BENCHMARK.md and INDEX-MEM-BENCHMARK.{md,json} are ingest.md, storage.md, query.md and index-memory.{md,json}. Six SHOUTING files at the root read as project furniture on the level of README and LICENSE; they are generated documentation, and they belong with the documentation that cites them. Every benchmark binary writes to the new path, and docs/README.md now indexes all five records rather than two. Entries below this one name the files as they were called at the time.

• query-bench publishes its record only for the canonical configuration, the rule bench already held itself to. Now that the record is a published document, an experimental corpus or client count would otherwise have overwritten the committed numbers with an answer to a different question.

• The advertised tool catalog is smaller than before the annotations were added: 14,895 → 14,296 bytes, with 628 bytes of new annotations inside that. The four accepted time formats were restated on ten properties while the initialize instructions already spell them out once per session; that repetition and some restated prose are gone. Every semantic warning stays — word-not-substring, status-is-not-an-attribute, a missing key is kept, and the ranking ceiling that sends a caller to slowest_spans.

• The published history was rewritten once, before the repository went public, to remove an internal live-deployment analysis that had no business in a public tree. Tags v0.1.0 through v0.20.0 are bit-identical to the private originals; commits from the v0.21.0 release onward carry new hashes. Stated here because a lineage that quietly changed under its tags would be worse than one that says so.

0.21.0 - 2026-07-31

Added

• The write buffer now bounds itself in time and self-corrects observed key shadowing, instead of trusting volume thresholds an idle store never reaches. Found by measuring a real deployment: a trickle workload (~150 spans/day against the 10,000-span flush threshold) had parked upserted keys in the write buffer for 36 days, which held ~12 MB of log a restart would replay and — far worse — disqualified every segment holding those keys from answering stats/llm/sessions out of its rollup, for the entire wait. Two new bounds, both mechanism-triggered and both on by default:

  • --max-buffer-age-seconds (default 300, Config::max_buffer_age) — the oldest buffered span’s wait is now a seal trigger, checked on the ingest path and by the new Store::maintain_buffer, which traza-server drives from its maintenance tick. Under real traffic the volume thresholds fire first and this never does; traza_segment_seals_age_total says when it did.
  • A corrective merge on observed segment-key shadowing (--no-shadow-seal to disable, Config::shadow_seal) — the analytics fold already decides, per segment, whether a rollup is safe or the same (trace_id, span_id) exists in a newer segment; that decision now latches a flag instead of being discarded. Maintenance converts the latch into a merge of the shadowed tail run, chosen by a scan that reads only cached or sidecar rollups — never a decode — and bounded by --compaction-max-segment-bytes. The merge rides the existing journaled machinery: same permit pin, same contiguous id claim, same rollup handover. Three deliberate restraints, each the result of adversarial review: buffer-caused shadowing does not latch (a merge cannot retire a key a client is still updating — the age bound handles it); a successful merge cools the pass down for 15 minutes, so a workload that re-poisons after every merge gets a bounded rewrite rate; and a pass that finds nothing mergeable backs off exponentially to hourly instead of re-scanning every interval. Inert when compaction is disabled, which owns all merging. A latency threshold is deliberately NOT the trigger: the latch fires exactly when a query had to decode instead of using a rollup, and stops the moment the duplicates are merged away. Counter: traza_shadow_merges_total.

  Measured end to end on a copy of the deployment that motivated it (32k spans, 46 MB, 3 segments, 1,368 shadowed keys): one seal plus one shadow merge leaves a single deduplicated segment, after which whole-corpus stats/llm answers from rollups in 0.55 ms against ~170 ms before the pass — and against ~20 s on the pre-rollup engine the deployment was running. /v1/stats now reports buffer_age_seconds so a scheduler that stopped calling maintain_buffer is visible.

• A storage-cost benchmark (storage-bench) and an honest comparison against OpenObserve’s published table (docs/storage-comparison.md, STORAGE-BENCHMARK.md): the same eight metrics OpenObserve computes against Elasticsearch, measured for Traza’s segment format, including the configurations where Traza loses the comparison and why.

• An MCP server, embedded in traza-server: POST /v1/mcp. Traza’s read path had two consumers — the dashboard and curl — and the one its own vision implies was missing: the agent that produced the traces. --mcp serves the Model Context Protocol from the same binary, same port, same auth gate, with no new dependency (JSON-RPC is serde_json; the transport is the HTTP server that already exists) and no engine change. Tool handlers call Store directly rather than looping back through the socket, which is why tests/mcp.rs can drive the whole surface with no listener in the process.

  Ten tools, shaped like questions rather than routes: describe_store, search_spans, get_trace, list_sessions, get_session, top_failures, slowest_spans, analyze_cost, get_payload, and record_annotation behind two gates. A mechanical translation of the route index would have been nineteen tools a model cannot tell apart. describe_store exists because an agent that guesses a service name gets an empty result indistinguishable from “nothing is wrong” and reports that everything is fine.

  Results are bounded in tokens, not rows. One LLM span with a prompt and a completion is 20–50 KB, so the REST default of limit=100 would be an unusable call that also costs money to fail: span tools default to 20, stored content is omitted unless asked for, every result is capped by --mcp-max-result-bytes, and every truncation is stated along with the argument that would have narrowed it — a silently shortened answer gets reported by a model as a complete one.

  Stored span text is treated as untrusted. It is confined to a delimited block with a preamble saying so, control characters are escaped so a newline cannot forge a row, the delimiter is neutralized so a value cannot close the block early, and it never reaches a tool name, description, or error message. The load-bearing mitigation is architectural: the server has no fetcher, no shell, no filesystem write and no outbound network path, so an injected instruction has nothing to actuate.

  Also: five resources and three URI templates (traza://trace/{trace_id} and friends, so a trace id in a tool result is something a host can attach), four prompts that each carry the live store overview as an embedded resource, a traza-server mcp --url stdio bridge for clients that launch a subprocess, and a dashboard MCP screen that reads the live surface from the running server rather than describing what the build believes it to be.

• Per-tool authorization on the MCP route. Every other route maps scope to HTTP method, which is right where the method is the operation. MCP tunnels reads and writes through one POST, so the method rule would either lock every ro token out of a read-only surface or hand every caller the write scope. AuthConfig::scope_for authenticates without applying it, and the endpoint authorizes per tool: ro reaches every read tool, and record_annotation needs rw and --mcp-annotations. A tool the token cannot call is never advertised to it — a model shown one calls it, reads the refusal as transient, and retries.

• A mcp route class in /v1/metrics. One POST /v1/mcp can be a lookup, a search or a whole-store rollup depending only on the tool named in the body; filed under other alongside static assets it would describe neither.

Fixed

• The MCP endpoint’s DNS-rebinding defence trusted a header the attack controls. Origin was accepted whenever its authority equalled the request’s Host — which is exactly what a rebinding request supplies, since the attacker owns the name and the browser sends theirs in both headers. A page on any domain could drive a loopback Traza and read the whole store. Origins are now checked against loopback plus an operator-supplied --mcp-allowed-origin allowlist, and nothing else the request carries is consulted; the Host header is no longer read at all, so the comparison cannot be reintroduced by accident.

• list_sessions ranked a page instead of the population. order_by=cost fetched the most recent sessions and re-sorted those, so an expensive session outside the recency window was not lower down — it was absent. Ranking moved into Store::sessions, which already materializes every session in the window, so the comparator change costs nothing and the answer is over the whole population.

• structuredContent escaped --mcp-max-result-bytes. Only the text block was clamped, so one long stored identifier produced an 83-byte text block inside a 100 KB result under a 1 KiB ceiling. Text and structured content are now budgeted and trimmed together; the ceiling bounds the tool result, not one field of it.

• Impossible timestamps became different valid ones. 2026-02-31 resolved to March 3rd, 2026-07-27T99:99:99Z to July 31st, and a +99:99 offset was accepted — each silently substituting a window nobody asked for, over which the answer looks correct. Month lengths, leap years (century rule included), time-of-day and offset ranges are validated before conversion.

• Empty results violated the schema their tool advertises. list_sessions and analyze_cost returned text-only results for an empty store or window, while both declare required outputSchema fields — so a validating client would reject the most routine answer either tool gives. They now return {"sessions": []} and {"group_by": "…", "rows": []}, and the path that trims rows to fit keeps the structured half rather than dropping it.

• Small ceilings were exceeded by the JSON envelope. --mcp-max-result-bytes was applied to the text block, then the result was wrapped — so a 256-byte ceiling shipped 286 bytes. The ceiling is now enforced on the whole serialized result at the single point every tool result passes through, and a ceiling below 1,024 bytes is refused at startup, because beneath that no result can both fit and conform.

• The dashboard generated a stdio configuration that could not work over TLS. It interpolated window.location.origin into traza-server mcp --url, which the bridge refuses for https://. On a secure origin it now asks for the plaintext endpoint instead of emitting a copy-ready snippet that fails.

0.20.0 - 2026-07-27

Added

• GET /v1/tail streams spans as they are admitted, as server-sent events. This is the only route ordered by admission rather than event time, and that is why it exists: /v1/spans?since= answers “what STARTED after T”, which is a different question from “what is arriving”. A span running longer than a client’s polling interval starts before the watermark and arrives after it, so the old polling tail dropped it permanently — not late, never. Admission order comes from a bounded in-memory ring, so it costs no disk, no segment format version, and no branch in the query path. Every /v1/spans predicate works, content search included. An event-time bound is refused with 400 rather than ignored, at the HTTP surface and in Store::tail_after alike.

  Bounded by --tail-ring-spans (count) and --tail-ring-bytes (memory), whichever binds first. The byte bound is the one that matters: the ring owns whole spans once a seal drops them from the write buffer, and an LLM span carrying a prompt is orders of magnitude larger than one carrying a status code. Residency and both bounds are reported at /v1/metrics.json under tail_ring.

  The ring is bounded by bytes as well as count, and the byte estimate counts structure and retained capacity, not only text — a Value slot per element and per map entry, and capacity() rather than len() for every collection. Counting text alone left the ceiling bypassable by shape; counting logical length left it bypassable by a Vec grown large and truncated, which keeps its whole allocation.

  The screen itself is under test now (ui/src/views/TailScreen.test.jsx, jsdom + Testing Library), which found four defects nothing else could reach: resume produced no rows — setRows was given an updater that read buffer.current, and the buffer was cleared before React ran it; every keystroke in the service filter opened a new stream, eight connections for “checkout”, now settled over 250 ms; row keys included the array index, so prepending a batch changed every key and rebuilt all 300 rows per frame, now a client-assigned id; and a full pause buffer discarded its overflow silently, now counted and shown. The rate window is bounded by samples as well as by duration.

  A gap is a visible discontinuity even when the server cannot count it. A cursor from before a restart is not comparable to the new numbering, so missed is null rather than a number, and the client shows the break without inventing a count — tracking counted and uncounted breaks separately, so “unknown, then 5” reads as 5+ rather than as an exact 5. backfill survives a gap unchanged, including backfill=0.

  A streamed span has been acknowledged: entries reach the ring only after the ingest succeeded, past the log’s fsync and past the seal flushed promises. The tail is bounded and may gap, but it never shows a span the store did not accept.

  A subscriber that falls further behind than the ring retains gets a gap frame carrying a count and no position: the dropped entries are exactly the ones no longer addressable, and no query can name an admission range. The stream restarts at the live edge and the client rebuilds from there.

  The dashboard’s Live tail consumes it over one held connection. An idle tail went from roughly forty empty round trips a minute to zero.

Fixed

• Aggregations no longer block ingest. fold_spans held the writer and segment locks for the length of a full-corpus scan, so any series, duration, failure or slowest query stalled writes until it finished — and the Overview screen starts four at once. It now folds against a snapshot: one copy of the bounded write buffer, one reference per segment, locks released before the scan. That also makes an aggregate a reading of one instant rather than of a moving store. Guarded by tests/fold_concurrency.rs, which fails against the previous implementation.

• Three arithmetic overflows reachable from valid requests. A span with end_time_ns = u64::MAX panicked the duration histogram’s top bucket bound; until=u64::MAX overflowed the series bucket width; limit=u64::MAX overflowed the tail’s allocation. Debug builds panicked and closed the connection with no response; release builds would have wrapped, producing wrong series math and an effectively unbounded limit. All three now use checked or saturating arithmetic, with endpoint caps where a bound is also the right policy.

• Failure grouping was unbounded in memory. Every distinct (service, name, status) allocated a duration histogram and limit applied only after collecting them all, so high-cardinality error text — an id in a span name — could reach gigabytes before returning twenty rows. Bounded at 4,096 signatures, with spans_untracked reporting what the bound excluded.

• The dashboard could state a durability guarantee the server never made. Both the Server and Overview screens printed the wal sentence — “survives a kill-9, a panic, or an OS crash” — unconditionally, and Overview read a durability field /v1/metrics.json did not have, defaulting it to wal. A buffered server, which promises the opposite, was described as durable. The field is now served, and the wording is derived from it in one place.

• Drag-to-zoom silently reverted. An absolute window serialized into the hash as t="" and read back as the 1h default, so every brush selection snapped back to the last hour. Absolute ranges now round-trip as t=<since>-<until>, and a malformed one falls back instead of becoming a range of nonsense.

• Live tail could skip spans permanently. It advanced its watermark to max(start_time_ns) + 1 and ignored next_cursor, so when more spans shared the last timestamp of a page than the page held — routine for an SDK-batched flush — the remainder were never returned. It now drains by cursor, and overlapping ticks are prevented rather than deduplicated after the fact.

• AbortSignal did not cancel the request. The shared coalesced request was started with no signal, so aborting rejected the caller’s promise while the fetch, connection and server-side scan continued — the opposite of the intent on a screen that re-queries per keystroke. Subscribers are now reference-counted: one leaving cancels nothing, the last one cancels the work.

• A screen labelled “live” never refreshed. Overview resolved its window once at mount and polled nothing, so every figure was frozen at the moment the tab opened while the live dot pulsed over it.

• Failure shares were computed against truncated data. Both screens summed the groups they had been sent — a page cut to a limit — and used that as the denominator, inflating every signature’s share of “all failures”. The API now returns total, counted before truncation.

• A series window near the top of the u64 range still panicked. The previous round fixed the bucket width and left the bucket starts unchecked three lines below it, so since + width * index overflowed for any window close to u64::MAX. The saturating ceiling was also one nanosecond short per bucket on a full-range window, leaving the last bucket ending before the window did. Starts are saturating and clamped to until; the ceiling is quotient plus non-zero remainder.

• The live tail stranded bursts one page budget further out. Carrying the cursor only within a tick moved the threshold from 200 rows to 1,000 rather than removing it: a burst larger than the budget was still abandoned, and the watermark cannot rescue it because every span in an equal-timestamp burst is >= since. An unfinished cursor chain now survives the tick that could not finish it, and the watermark advances only once the chain is exhausted. The polling state machine moved to ui/src/lib/tail.js so it is testable without a DOM — which is where both of its bugs were actually found.

• Overview compared the wrong periods. One selected window was split down the middle, so “24h” showed the last twelve hours against the twelve before them while the label said “previous 24h” — and the failure and model cards queried the full 24 hours, so three cards on one screen described three different spans of time. Two full periods are resolved now: the series covers both with the midpoint exactly on the boundary, and the other cards are scoped to the current period.

• A settling request could evict its own replacement. The in-flight map was cleared unconditionally in the finalizer, so an aborted request finishing after a fresh one had taken its slot deleted the live entry and made the next caller open a third. Eviction is identity-checked.

• Overview’s refresh was partial. The tick re-ran the window-dependent reads but sessions and metrics kept empty dependency arrays, so “Server, in its own words” stayed frozen under a live indicator.

• A paused tail buffered duplicates. since is inclusive and the paused path skipped the primary-key check, so a quiet page was re-buffered until it filled. Both paths share one dedupe now.

• A drained equal-timestamp burst replayed forever. The dedupe set was evicted by size, but a burst sharing one timestamp cannot advance the inclusive watermark, so its keys are needed on every later poll: 1,250 spans at one timestamp cycled 1000, 250, 1000, 250… indefinitely. The set now retains exactly the keys ON the watermark and prunes only when it moves — bounded by one timestamp’s membership, which is the minimum that is correct. The previous test stopped the moment the drain completed, which is precisely where the replay began.

• A filter change could admit rows from the previous filter. Replacing the tail’s state and clearing the screen does not un-send a request; an in-flight poll still resolved and appended its old-filter rows. Responses are checked against a generation token.

• Overview’s period p95 was the worst bucket’s p95. max(bucket.p95) is not a percentile of a period, and one sparse slow bucket dragged it into seconds while the true figure sat in milliseconds. Each period now reads a duration histogram folded over the whole period.

• The top model’s spend share used a truncated denominator. /v1/stats/llm?limit=6 returns six rows, and their subtotal was the denominator, so the share inflated by whatever the limit left out. The series’ total cost is the honest one.

• Resuming a paused tail reversed its chronology. The buffer is already newest-first, so reversing it handed back an oldest-first block.

• Query cost under-reported content pruning. Both search paths incremented the process-wide counter but not the per-query segments_pruned, so a content-narrowed search understated the work it had avoided.

Changed

• The segment format collapsed to a single version, numbered 6. It had grown by appending header fields behind if version >= N gates, and each gate turned a field into an Option that every reader downstream had to treat as “unknown, therefore assume the worst”: a segment whose timestamp range could not be read had to be scanned by every time-bounded query, and a second attribute-index decoder existed only to read the encoding that predated digests. MIN_READABLE_VERSION, HEADER_LEN_V2, HEADER_LEN_V3 and the legacy decoder are gone; timestamps and content are plain values, so the pruning path no longer carries a case where it cannot prune.

  The failure names one commit that reads every superseded format (traza::LEGACY_SEGMENT_READER), not a per-version release. A store accumulates segments in whichever format was current when each was sealed, so one directory can hold several at once, and the release that reads the oldest cannot read the newest.

  Upgrading makes existing stores unreadable. Versions 2 through 5 were written by tagged 0.x releases — 2 in v0.16/v0.17, 3 in v0.18/v0.19, 5 up to the previous commit — so this is a real on-disk break for anyone running those, not a cleanup of formats that never escaped. Store::open refuses such a segment rather than misreading it, names the file, and points at the migration: back up the directory, open it with the release that wrote it, GET /v1/export, and re-ingest. The README’s pre-1.0 terms permit this — “on-disk formats may change between 0.x versions” — but permitted is not the same as costless, and the work falls on whoever has data.

  The numbering deliberately does not restart. Removing compatibility code and reusing compatibility identifiers are different acts: a header declaring “2” must never be ambiguous between the layout v0.16 wrote and some later one, so 1 through 5 stay spent and the single readable format is 6.

  The version word stays for the same reason it always should have. Two bytes per file, and it is the difference between refusing to open a format this reader does not know and parsing its header at these offsets — which yields section bounds that pass every validation check while addressing the wrong bytes.

  This is a one-time exception, not the policy. From v6 onward a format bump ships with a migrator: the runtime still reads one canonical format, and the conversion from the previous one lives outside the query path rather than in it. See the segment format doc for the rule. Doing it here would have meant resurrecting the decoders this change deletes, for stores that do not exist.

• The logo is the revised mark from the design system. The bars gained a stem, so the mark resolves as a lowercase “t” rather than four unanchored rows. It is a component now (ui/src/components/Logo.jsx) rather than SVG inlined per site, defaulting to currentColor because the design system is explicit that an img-referenced SVG cannot inherit page color and the reversed lockup depends on it. The wordmark’s tracking was a step off the brand card (-0.01em against -0.02em). The favicon now uses the three-bar variant the system specifies for 16px, where the four-bar mark loses its rows.

• Segment format v4: the attribute index is keyed by digest, not by value text. Through v3 the index held every distinct attribute value resident, in full, for the life of the segment. For enum-shaped attributes that costs nothing; for the data Traza exists to store it was fatal — an indexed gen_ai.prompt is kilobytes, every value is distinct, and the resident index therefore grew to roughly the size of the corpus text. A store that reads records from disk on demand specifically so it can outgrow RAM was pulling the largest part of each record back into RAM through its own index.

  v4 interns attribute keys and replaces each value with a 128-bit digest, so an entry costs the same for a status code and for a page of text. Measured before and after on the same machine and command, 256 MiB of all-distinct text: 391 MiB → 21.6 MiB RSS on open at 2 KiB values, 439 MiB → 77.5 MiB at 512 B. The 512 B cell is dearer because it holds four times as many spans — cost tracks cardinality now, which is the point.

  A digest probe returns candidates, so attribute_posting_offsets_ref is now attribute_candidate_offsets and Segment::query_attribute verifies. At 128 bits nothing collides naturally, which means no ordinary test can tell a verifying reader from a trusting one; the acceptance target forges the collision instead.

  v2 and v3 segments still open — their values are hashed and discarded at open time, so an existing store gets the smaller steady state without being rewritten.

Added

• Content search: GET /v1/spans?content=refund. Finds spans by the words in their text — string attributes, nested message arrays, event attributes and event names. Segment format v5 carries a Bloom filter over the words in each 128-record block, stored bit-sliced (one row per bit position spanning all blocks) so a probe reads tens of bytes per segment instead of every block’s whole bitmap. Only a per-segment summary filter is resident, capped at 32 KiB, so the cost scales with segment count and not with text volume.

  Measured against the same corpus with the index off, 200,000 spans holding 145 MiB of text: a word in one span 1.48 ms vs 1,258 ms (849x), two words 0.74 ms vs 1,156 ms (1,554x), a word in no span 0.008 ms (146,000x), and a word in nearly every span 1.0x — with no selectivity to buy there is nothing to win, and that row is published alongside the others. Disk cost +0.1%, resident ~2 KiB per segment.

  It is word matching, not substring matching, and not phrase matching. refund does not match refunds. That is a soundness requirement rather than a limitation of effort: a word index cannot over-approximate a substring query, so driving one with it would skip spans that should have matched — a wrong answer instead of a slow one.

  --no-content-index / Config::content_index turns it off; content search still works, by scanning.

• traza_segments_pruned_by_content_total and traza_records_admitted_by_content_total in /v1/metrics. A content index that stops pruning returns byte-identical results and simply gets slower, so these counters are the only way to see it happen.

• status= and not_status= on span search. Every aggregate in the store counted errors from Span::status, but nothing could select on it: the filter walked span.attributes only. The natural-looking attr.status=error therefore matched an attribute most instrumentation never writes and returned an empty array indistinguishable from “no errors” — and the docs used exactly that expression as their motivating example. “Show me the failures” is now a query the API can answer.

• Cursor pagination on GET /v1/spans. The engine has had query_after and a total order for as long as export has existed; the HTTP surface never exposed it, so clients paged by re-requesting with a larger limit — which re-reads and re-sends every row already in hand. Responses now carry next_cursor, and a short page carries null rather than a token that could only return nothing.

• Per-query cost in the span-search response. cost.elapsed_ns, segments_examined and segments_pruned, counted per query rather than sampled from the process-wide counters, which race under concurrent readers and cannot be attributed. Traza’s argument is that filtered search is cheap; this is the evidence rather than the assertion.

• GET /v1/stats/series, /duration, /failures, /slowest. Aggregations that answer “where should I look”: volume, errors, tokens, cost and duration percentiles bucketed over a window; the duration distribution; errors grouped by (service, name, status) with first/last seen and an example to open; and the slowest matching spans ranked across the whole match set. All four fold in one pass and constant memory through a new Store::fold_spans, so the cost of an aggregate is proportional to the answer rather than to the corpus — a twenty-bucket histogram no longer materializes a million spans to produce it.

• GET /v1/metrics.json, and request latency split by route class (ingest, lookup, search, stats, other). One blended histogram over ingest and search described neither: they differ by orders of magnitude. Also adds traza_uptime_seconds and traza_http_responses_{2xx,4xx,5xx}_total.

• traza_wal_lock_wait_ns_* and traza_wal_write_syscall_ns_* in /v1/metrics, splitting the log append into waiting for Traza’s log lock and the write itself. traza_wal_write covered both, which made the dominant remaining in-lock cost unattributable: it looked equally like lock contention and like slow I/O, and those want opposite fixes.

  The split settled it. Traza’s log lock measures zero wait; the time is the syscall. And the syscall is not slow on its own — the same append costs 0.076 ms at concurrency 1 and 1.778 ms at concurrency 8, a 23x difference in one syscall with no lock involved, because an fsync in flight blocks concurrent write calls to the same file inside the kernel. Widening --wal-commit-window-us from off to 2,000 cut fsyncs 40% and the mean append 57%. Documented in the configuration reference: the lever for a large wal_write is fsync frequency, not the engine’s locking.

• INDEX-MEM-BENCHMARK.md and INDEX-MEM-BENCHMARK.json — the committed measurement record behind the memory figures, with commit SHA, machine, timestamp, load average per row, compacted_away, and the raw per-cell results. The prose numbers were previously traceable only to a binary.

Changed

• Latency percentiles are now publishable: at most 6.25% high, never low. The stage histograms used plain power-of-two buckets, so a reported percentile could be 2x the truth — fine for ranking stages against each other, and this project’s own monitoring guide said in as many words not to publish them as request latencies. Each octave is now split into sixteen even steps. The record path gains one shift; each histogram grows to 8 KiB and moves behind a Box so a Store does not carry 80 KiB inline. _ns_p95 is emitted alongside _ns_p50/_ns_p99, since p95 is the figure the README and the dashboard both quote.

• GET /v1/annotations no longer requires trace_id, and gained source (prefix match), since, until and limit. Scores are produced per trace but are only meaningful as a population; requiring a trace meant an eval run could be read only one trace at a time, which is to say not at all.

• GET /v1/spans returns an envelope, {spans, next_cursor, cost}, rather than a bare array — the carrier for the two items above.

• The dashboard is rebuilt: seventeen screens on a left rail grouped by the question you arrived with, replacing four tabs over five views. New: Overview, Latency, Failures, Scores, Experiments, Datasets, Live tail, Trace compare, Server, Connect. Rebuilt: Traces (predicate builder reaching the whole parameter surface, drag-to-zoom volume brush, sortable columns, cursor paging, query cost) and Trace detail (time ruler, minimap, drag zoom, subtree collapse, critical path, self time, agent mode).

  A query is now a value that serializes into the hash route, so the search that found the bug is a link you can send; ⌘K opens a palette that takes a pasted trace id, which previously had no front door at all. Reads carry an AbortSignal so a superseded query stops occupying a connection, identical in-flight GETs are coalesced, and polling stops in a background tab.

Fixed

• A store under sustained ingest was never compacted. A merge declined outright when a seal held a claimed-but-unpublished segment id, rather than waiting for it. The guard is a correctness requirement — that seal publishes newer data under a lower id, and merged output taking a higher one would outrank it, inverting last-write-wins — but declining made compaction hostage to the write rate. A seal is in flight for much of the time under load, so a tick that checked once and gave up almost never found the store quiet: measured at 25,000 spans/s, one tick in sixteen achieved anything and the segment count grew without bound (14 to 215 in six seconds) while compaction ran on schedule and did nothing. It recovered the instant writes paused, which is why the effect is invisible on an idle store and total on a busy one — the case compaction exists for.

  A merge now takes the seal permit instead of testing a counter, held only long enough to choose a run and claim its output ids. Acquiring it is the slow half: a seal owns the permit from drain through write, fsync, rename, reopen and reconcile, so a merge arriving mid-seal waits out all of that — bounded by one seal, and on the maintenance thread. Expiry already takes the same permit and holds it for a whole deletion.

  Most ingest is unaffected, because a seal that cannot take the permit coalesces into the next one. Two paths wait rather than coalesce: Durability::Flushed, which must seal before it acknowledges, and any mode once the buffer reaches four times flush_spans, where waiting is the backpressure. Both wait only on the microseconds a merge holds the permit, never on the merge itself.

  Under the same load every tick now compacts, and the segment count stays bounded — oscillating between 15 and 80 over six seconds, against 14 climbing to 215 before. The unpublished_seals counter is gone; the permit is what it was approximating.

  Choosing the run moved inside those guards too. Scanning first and then re-checking under the lock left a window for a seal to land and make the answer stale, which retired the whole tick.

  And a tick is now bounded to the backlog it found: segments sealed after it started are left for the next one. Without that, merging kept pace with arrivals and a single call ran until the writes stopped — 2,213 segments merged away and still going.

• The index-memory benchmark’s “peak RSS” was not a peak. It took one RSS sample after compact_segments() returned — after the merge has freed its working set — so it measured the trough and the capacity guide published it as the peak. A comment claimed a peak sampler existed; none did. RSS is now sampled every 20 ms by a background thread for the duration of the merge, and the real figures are far higher: a store serving in 10 MiB peaks at 1,204 MiB to compact itself, not 721. Every completed merge in the record peaks between 1,204 and 1,601 MiB, taken at load average 36-45 and varying up to 8% between runs of the same matrix — the order of magnitude is the finding, not the individual figures.

• A failed probe was published as a zero-memory result. A child that died produced an empty JSON object, and the reporter turned missing fields into 0.0 — so the most important failure this benchmark can have, the child being OOM-killed on the very configuration whose memory is in question, rendered as 0.0 MiB: the best-looking number in the table. Probe failure and unparseable output now abort, naming the configuration and the child’s stderr. Verified by simulating an OOM kill.

• Compaction rows were reported for runs that never compacted. 21 of 26 configurations return 0 from compact_segments(), and their steady-state RSS was appearing under a compaction heading. They now read “did not merge”.

• Per-key index diagnostics moved to their own --by-key invocation: computing them reopens and decodes every segment, and the allocator held those freed blocks across the RSS reading that followed.

• The capacity guide claimed “none is estimated” while carrying an explicitly extrapolated 29 GB projection. It now states the exception and cites the measurement records by name.

• Compaction stopped permanently at the first segment that did not match its neighbours’ size tier. tail_run_to_merge took the tier of the last segment and walked back only while the tier matched, so any discontinuity at the tail cut the run to length 1 — below fanout — and nothing merged. Two ways that happened in practice, both measured. Ingest seals when the write buffer reaches flush_spans and batches do not divide evenly, so a finished load ends on a partial segment; below base_bytes it is a tier of its own, and 10M spans at the defaults compacted 0 of 977 segments. Separately, a merge stopped by max_segment_bytes left a cap-sized segment at the tail, one tier above the segments behind it, which froze those the same way: 20 equal segments went to 17 and then stuck. Neither self-healed. Nothing behind the tail is ever a merge candidate again, so continued ingest compacted only the region after the wall and left the prefix — hundreds or thousands of segments — permanently unmerged. Filtered search costs one index narrowing per segment, so this is the exact cost compaction exists to bound.

  Two changes. Only a larger segment ends a run — the tier that matters is the largest in the run, and a smaller neighbour rides along as a passenger rather than blocking it (passengers never count toward fanout, so four tiny segments still cannot justify merging into a 256 MiB one). And a run now merges into a group of outputs rather than one, splitting at max_segment_bytes instead of truncating the run to fit, so the cap bounds each output without stranding the remainder. Outputs take a contiguous block of ids in group order, which is what keeps last-write-wins intact across them; dedup stays within a group, so a merge still holds only one output’s spans in memory. On the shapes above: 977 segments now compact to the cap floor instead of 0, and a store compacting as it ingests settled at 10 segments where it previously drifted to 30.

  The compaction journal is now one record per merge — .supersede.<first-output>.journal, naming every input and every output — because which way to finish an interrupted merge is a fact about the group. Recovery deletes the inputs once every output is present and parses, and otherwise rolls the merge back — a lone output carries a higher id than every input while holding only its own group’s view of a key, so left beside intact inputs it would shadow a newer version in a group whose output never landed. Rollback requires every input still to be present: any one already gone proves deletion had started, which happens only after every output was durable, so a missing output is one a later merge has since consumed and recovery must roll forward instead. A journal that saw one input at a time could not tell those apart and would delete live segments holding the only copy of an input already removed.

0.19.0 - 2026-07-25

Segment sealing no longer holds the writer lock, which was 74% of everything that lock was held for and 88% of a run’s wall clock. Ingest rises 37-116% depending on --flush-spans, and --flush-spans stops being a throughput setting at all.

Alongside it, four defects from an independent review of v0.17, fixed and tested. They are symptoms of one gap — Traza has several recovery domains and nothing names a state they all agree on — so the class fix is designed too, and scheduled before 1.0 rather than as part of HA: docs/generations-design.md (since moved out of the repository).

Fixed

• TTL-expired spans came back after a restart. Expiry removed them from the write buffer and left the write-ahead log records that carried them intact, so recovery replayed the expired span and record_count went back up. Expiry now rewrites the log to exactly the spans that survived — staged and renamed, because the survivors are still acknowledged and must not be lost to a crash mid-rewrite. The expired bytes leave the disk in that pass rather than being marked dead, which is also what “deleted on request” has to mean.
• Damage in the middle of the write-ahead log silently dropped acknowledged batches. Replay stopped at the first length, CRC or JSON failure and returned everything before it as if that were the whole log: three fsynced frames with a corrupt second one recovered as one, and the third vanished without a word. Recovery now distinguishes the two cases. A frame missing bytes it declared can only be the interrupted final append — it is dropped, and the file is truncated back to the last complete frame so those bytes cannot become interior bytes after the next append. A frame that is complete but fails its checksum or decode fails the open, naming the byte offset and what moving the log aside would cost. See when the log will not open.
• A concurrent export was not one dataset. GET /v1/export ran an independent query per 4,096-row page, so a span re-ingested behind the cursor was emitted twice — 5,001 rows and two versions of one primary key, under X-Traza-Export-Complete: true. Export now pins a SnapshotView and pages that: the write buffer is copied and the segment set is reference-counted, so compaction and expiry may unlink files the export is still reading and the space returns when it finishes. complete: true now means “this is the whole dataset as of the first byte”, each primary key appearing at most once.
• Hot-key updates grew the log without bound. The automatic flush threshold counted unique buffered records, which an update to an existing key never advances: with --flush-spans 2, 500 acknowledged updates to one key left buffered_records: 1, segment_count: 0 and a log of 108 KB that would never seal. --flush-spans now applies to upserts since the last seal as well as to unique records, and a new --flush-wal-bytes (default 64 MiB) bounds the log directly. Recovery also streams frames instead of reading the whole log into memory.
• A failed expiry was not retryable, and resurrected spans. Expiry mutated memory before the durable change it corresponded to: it dropped the span from the write buffer before the log rewrite succeeded, and removed a fully expired segment from the live list before unlinking its file. Either failure left memory ahead of the recovery authority — and left nothing for the retry to find, so the next pass reported Ok(0), never repaired the log or the file, and the restart brought the data back. Both now change durable state first and memory second, with the in-memory step infallible, so a failed expiry leaves the store exactly as retryable as it found it. Wal::rewrite additionally moves every fallible step before its rename, so its failure is never ambiguous about which log is live.
• A deleted segment file was reported gone before the deletion was durable. An unlink is visible immediately but survives a crash only once the directory entry it removed is synced, so expiry could report a segment deleted — and drop it from the live list — over a file a crash would bring back, spans and all. Expiry and compaction now sync the directory after unlinking and before anything downstream depends on it, and the unlink is idempotent so a retry after a partial one can finish instead of failing forever on NotFound.
• Retention rewrote segments under a new id, which moved them to the newest position in an order that is recency order — so after a partial expiry a re-ingested span could revert to an older version held by the rewritten segment. The survivors are renamed onto the same name now, keeping the segment’s place. Found while fixing the above.

Changed

• Segment sealing no longer holds the writer lock. It was the largest thing the engine did while holding the lock every ingesting thread needs: converting spans to records, encoding the segment, writing it, fsyncing it, renaming it, fsyncing the directory and reopening the result — all on a private vector no other thread can reach. Measured at concurrency 8 before the change, the lock was held 88% of a run at the default --flush-spans and 74% of that was the seal; at --flush-spans 5000 it was 97% and 81%. A seal now drains the buffer under a short lock, does every byte of I/O with nothing held, and publishes under a short lock — the shape compaction and retention already had.

  Before and after with the two builds alternated round-robin on a contended host, median of four rounds at concurrency 8: --profile throughput 162,763 → 222,683 spans/s (+37%), balanced 116,612 → 176,004 (+51%), latency 83,400 → 180,331 (+116%). Those levels are depressed by background load; the round-robin is what makes the ratios trustworthy. --flush-spans has stopped being a throughput knob — latency and balanced now land within 3% of each other, where they used to span 2x — so set it for the tail latency and buffer memory you want.

  Two consequences worth knowing:

  • The write buffer can exceed --flush-spans while a seal is in flight, because ingest no longer waits for one. Past four times the threshold an ingesting thread waits for the seal to publish, so it stays bounded — but size memory for that bound.
  • --flush-wal-bytes now governs the log’s real size under sustained ingest. A seal that empties the buffer still discards the whole log, so a quiet store is unchanged; a busy one lets the log run to that bound between reclamations rather than emptying it on every seal, because rewriting the log to the survivors every time would put thousands of re-serialized spans straight back under the writer lock. Restart replay is bounded by the setting, which is what it always documented.

  What made this safe rather than fast-and-wrong: the drain copies the buffer instead of emptying it, so already-acknowledged spans are never in neither the buffer nor a segment — a merge keeps its inputs live until its output is published, and a seal now does the same. The buffer holds Arc<Span> so that copy is pointer-sized and so the post-publish eviction can ask is the value under this key still the one I sealed by handle identity. Comparing values would have destroyed data: a span re-ingested unchanged during a seal is a newer version that happens to look identical. tests/seal_concurrency.rs races reads, ingest and expiry against a seal; tests/durability.rs adds a SIGKILL taken mid-seal.

• traza_segment_seal_locked_ns_* and traza_segment_seals_coalesced_total are new on /v1/metrics. The first is the part of a seal that holds an engine lock; against traza_segment_seal it is the only way to see that the write is off the lock, because query results are identical either way. The second counts seals that found another already in flight and declined to start a second one.

• Compaction and retention no longer stop the server. Both held the segment lock across parsing every input, materializing the result and fsyncing it; queries waited on that lock while holding the writer lock, so ingest queued behind the queries. A merge measured in gigabytes was an outage measured in gigabytes. Both now pin their inputs, do every byte of I/O with no engine lock held, and take the lock back only to publish — after re-checking that what they pinned is still there. A new maintenance lock serializes the two against each other, and only against each other. tests/compaction.rs measures it: the slowest read or ingest during a merge must be a fraction of the merge. What this buys is that reads and ingest no longer wait on maintenance; they still share CPU and disk with it, and that contention remains unmeasured.

• Store::snapshot is public API, returning a SnapshotView that answers from one pinned instant however the store changes afterwards. Any multi-step read should use it; a lock cannot span pages.

• Error::WalCorrupt is a new variant, for the refusal above.

• Config::flush_wal_bytes is a new field (Some(64 MiB) by default). A Config built by struct literal needs it; ..Config::default() does not. Documented in the library Config table alongside the server flag.

• The generations design carries the log inside the boundary. CURRENT and a global wal.log are two recovery authorities that no rename can publish together, so the design now stamps every frame with the generation epoch it belongs to, records folded_through in the manifest, and replays only frames after it. Publishing CURRENT is staged, renamed and directory-fsynced before a single folded frame is reclaimed — a rename is not crash-durable until then, and a durable log reclamation against a CURRENT that rolls back is the one combination that loses acknowledged writes. The crash matrix covers both sides of that fsync, and reclaiming folded frames is described as the roll-over it has to be: they are a prefix, and truncation only removes a suffix.

0.18.0 - 2026-07-25

Search stops answering real questions with silence, and gains the predicates its own analytics already implied.

Search gains the predicates its analytics already implied, and stops answering two classes of question with a silent empty result.

Added

• Range, negation and ordering predicates: min_attr.KEY / max_attr.KEY (numeric, reading stringified numbers too), not_attr.KEY, max_duration_ms / max_duration_ns, and sort=duration|-duration|start|-start. Token and cost analytics could already aggregate what search could not find, so “which calls cost more than a cent” and “the ten slowest” were unanswerable. not_attr.KEY keeps spans that lack the key entirely — “not known to be an error” includes spans that never recorded a status.

• Segment timestamp ranges (format v3), letting a time-filtered query skip a segment’s records. (Every segment is opened and its indexes parsed at store startup; pruning avoids the record reads, not the open.) since/until were pure post-filters, so a “last 15 minutes” search read every segment in the store. v2 segments are still read; they carry no range, are never skipped, and age out through compaction.

  No latency improvement is claimed for this yet. Pruning is verified to skip the right segments by counter, but an attempt to measure the payoff produced a negative result (a windowed query slower than an unwindowed one) on a 40-segment store under load. Forty segments is far too few for per-segment probe cost to dominate — the compaction work needed thousands before the effect was visible — so the benchmark was measuring noise. The mechanism is sound and the work avoided is real; the latency benefit is unmeasured, and is recorded as unmeasured rather than assumed.

• traza_segments_pruned_by_time_total and traza_segments_examined_total in /v1/metrics. Pruning is invisible from results — a skipped segment and a scanned one give the same answer — so these are the only way to see it working.

Changed

• Attribute filters match scalars by value, not by type. attr.code=200 now finds spans that stored 200 and spans that stored "200". Previously only the JSON reading matched, so a store of stringified codes answered every such query with an empty array indistinguishable from no-such-data. Containers still compare structurally.
• The index probe is chosen by selectivity rather than by a fixed order. Only one predicate can drive a scan; the planner took service, then name, then whichever attribute came first. service is usually the least selective term in a trace store, so adding a precise attribute filter to a service query made it slower — it read every span of the service and discarded almost all of them. The smallest posting list now wins.

Fixed

• A sorted query ranks every match rather than the first page. Sorting cannot stream, so past an internal candidate ceiling the query is refused with 400 and guidance to narrow it — a “ten slowest” computed over an arbitrary first page is a wrong answer that looks like a right one.

0.17.0 - 2026-07-25

Ingest throughput roughly doubles at concurrency, and the record of why is corrected in three places where it was wrong. Persistent connections, OTLP decoded straight to spans for both wire formats, --profile for the throughput/latency tradeoff, and a documentation set for users, operators and developers.

Ingest throughput: 108,881 -> 208,973 spans/s at 16 concurrent clients in wal mode, measured through one client against both builds. The roadmap’s 250k target is still 16% away, and the benchmark now says exactly why.

Added

• Persistent HTTP connections. Every response used to carry Connection: close, so a client paid a connect and teardown per batch. Keep-alive is now the default for HTTP/1.1. Worth +11% at batch=20 and nothing at batch=1000 — the honest number, not the hoped-for one.
• GET /v1/metrics in Prometheus text format: per-stage ingest timings (writer-lock wait, WAL encode/write/fsync, buffer upsert, segment seal, decode), request latency, and connection counters. Stage percentiles are power-of-two bucket bounds and are documented as approximate; they exist to rank stages, not to be published as latencies.
• --max-connections (default 1024), replacing --workers. Keep-alive means a connection occupies its handler until the client is done, so a fixed worker pool would serve N clients and leave the rest queued indefinitely. Past the limit clients get 503 rather than silence.
• --wal-commit-window-us (default off): holds an fsync open briefly so more batches join it. A latency-for-amortization trade that does not touch the guarantee — the fsync still precedes the acknowledgement it covers.
• --profile throughput|balanced|latency, setting the write-path knobs (--flush-spans, --wal-commit-window-us) as a coherent group so they can be chosen by intent rather than by reading the internals. An explicit flag always beats the profile, in either argument order. No profile can change --durability — a profile cannot represent one, so none can silently make writes lossy. Measured tradeoffs, including where each profile does not help, are in docs/configuration.md.
• A documentation set for three audiences under docs/: a user guide (getting started, data model, ingest, full HTTP API reference, trace browser), operations (deployment, durability, administration, monitoring, capacity), and internals (architecture, the load-bearing invariants, module map, testing, benchmarking). The README is now an overview that routes onward rather than holding all of it.
• ingest-bench, a benchmark matrix over protocol, keep-alive, concurrency and durability. Reports the median of N runs with its spread, refuses to report a rate from a run that shed a connection or stored fewer spans than it acknowledged, and restarts the server to re-verify every non-buffered result. TRAZA_BENCH_SERVER points it at another build so before/after runs share one client.

Changed

• POST /v1/spans decodes straight to Vec<Span>. It used to parse to a serde_json::Value, deep-clone the array out of it, then re-walk that DOM once per span — three passes and three sets of allocations for one job.
• POST /v1/traces decodes straight to Vec<Span> too, for both encodings. Protobuf lowered to the OTLP/JSON Value shape and OTLP JSON parsed into the same shape, and both then re-walked that DOM. Protobuf additionally hex-encoded every trace and span id through a format! per BYTE. Decode is now 9.2x cheaper for protobuf (4,384 → 479 ns/span) and 1.9x cheaper for OTLP JSON (2,377 → 1,275 ns/span), medians of 5 runs of 1M spans at concurrency 1. Decode is ~2% of ingest cost, so this is a CPU and correctness result, not a throughput one. The mapping rules the two decoders must agree on are shared rather than duplicated, and a differential test pins that agreement across every AnyValue variant.
• ingest-bench measures latency, not just throughput, with an open-loop fixed-arrival-rate mode and coordinated-omission correction. Under a closed-loop generator every saturating configuration reports latency that is just concurrency over throughput, so the tradeoff a latency profile exists to make was not visible at all. Scenarios also run round-robin with their order rotated per round, so background load hits every configuration alike instead of landing on whichever ran during a spike.
• ingest-bench separates wire format from route. It posted JSON to /v1/spans and protobuf to /v1/traces, so every protocol comparison also contained the OTLP mapping; on that basis this project claimed protobuf was slower than JSON. A third protocol, otlp-json, holds the route fixed, and scenario labels now name the route. Measured properly, protobuf decodes 2.3–2.7x faster than OTLP JSON on payloads 2.9x smaller. The benchmark also reports bytes/span and decode ns/span per scenario.
• The WAL encodes a batch before taking the writer lock. Serializing under the lock made every concurrent ingest wait on one thread’s JSON encoding. Only the file write remains inside it.
• Sealing a segment no longer clones the write buffer, and puts the spans back if the write fails.
• Request framing is stricter now that connections persist: transfer-encoded bodies and duplicate Content-Length headers are refused rather than resolved, because either ambiguity lets one request be split in two with the remainder attributed to the client’s next request.

Fixed

• bytesValue attributes are no longer dropped. An OTLP attribute of the bytes variant was stored as null on both ingest paths, with no error and no warning. It is now stored as lowercase hex, the same representation trace and span ids use — protobuf’s raw bytes and OTLP/HTTP JSON’s base64 land on the one value, so what an attribute holds does not depend on how it arrived.
• A connection refused at the limit now reliably receives its 503. Closing a socket while the client’s request bytes sat unread made the kernel send RST, and the RST beat the response — backpressure surfaced as “connection reset by peer”.

0.16.0 - 2026-07-24

Search that scales with the store: size-tiered compaction bounds the segment count, and at a 1 GiB segment cap filtered-search p99 clears the project’s own 50 ms bar at 100M spans for the first time.

Added

• Size-tiered compaction, on by default and configurable with --compaction-fanout (0 disables) and --compaction-max-segment-bytes. Filtered search costs one index probe per segment, so a store that only appends flush-sized segments gets steadily slower to search as it grows. Compaction merges same-size segments to bound that count.
  • Measured over 10M spans, uncompacted vs default compaction: attribute filter p50 14.8 -> 2.4 ms, p95 33.4 -> 4.1 ms, p99 220 -> 14.1 ms (6-15x), and trace lookup p99 4.65 -> 2.28 ms. It costs ingest throughput, which is the trade the flag exists to let you make.
  • Measured at 100M spans (55 GB on disk), uncompacted vs default, both through the same harness: attribute filter p50 155.5 -> 9.8 ms, p95 747.3 -> 27.1 ms, p99 1664.6 -> 72.9 ms (16-28x), trace lookup p99 7.72 -> 1.82 ms, segments ~10,100 -> ~380. It costs about 31% of ingest throughput (59,025 -> 40,894 spans/s) and ~1.5 GB of resident memory for the merge working set.
  • With the 256 MiB default cap the filtered-search target is missed at that size: p99 72.9 ms against the project’s own 50 ms bar (p50 and p95 are inside it). Raising --compaction-max-segment-bytes to 1 GiB clears it — p99 22.2 ms, p95 9.3 ms, p50 2.3 ms, with trace lookup p99 0.99 ms, measured on the same corpus through the same harness. The binding constraint was the cap, not the algorithm: it floors segment count near corpus/cap, and the sampled count fell from ~380 to ~100-125.
  • That cap is a real trade, not a free win. Peak RSS rises 2.0 -> 6.7 GB (a merge materializes its inputs, so the working set tracks the cap) and sustained ingest falls a further 24%, 40,894 -> 31,267 spans/s. Measured at 100M on one machine in a single run; untested above that size.
  • Uncompacted, every segment holds an open file descriptor — ~10,100 at 100M, which would exhaust a default 1024-fd limit. A second reason not to disable compaction on large stores.
  • Only the TAIL of the segment list is merged. Segment path order IS recency order, and a merged segment takes a fresh (newest) id, so merging a run from the middle would promote its spans past segments that legitimately supersede them.
  • Crash safety reuses the existing supersede journal with one marker per input, and the merged segment is renamed into place before any input is deleted.
  • TTL expiry keeps its one-minute cadence; compaction runs on a 5 s tick in the same maintenance thread, which now also starts when only compaction is enabled.

Changed

• The benchmark can vary the segment size cap: TRAZA_BENCH_COMPACTION_MAX_SEGMENT_BYTES passes --compaction-max-segment-bytes to the server it spawns, parallel to the existing TRAZA_BENCH_COMPACTION_FANOUT. It defaults to the real CompactionConfig::default() value rather than a literal, so the two cannot drift, and the configuration is named in the generated BENCHMARKS.md row. Without this the cap could not be measured at all.
• Config gains a compaction field (None disables).
• README performance claims corrected against a measured 10,000,000-span run. The previous text dated from 0.3.1 and claimed search was “effectively scale-independent” at p50 2.9 ms; the measured filter p50 was 14.8 ms across ~1000 segments. The README now states that filtered search scales with segment count, and reports measured RSS (0.25 GB, not 0.71 GB) and disk (~6 GB for the benchmark’s span shape, not 2.4 GB).
• Renamed the segment module segment_v2 → segment (file, module, segment_error helper, acceptance test, format doc): there is only one segment format, so the version in the name was redundant. The suffix constants flipped too — the current format is now the unmarked SEGMENT_SUFFIX (.seg) and the unsupported legacy JSONL suffix is LEGACY_SEGMENT_SUFFIX.
• tests/segment_format_acceptance.rs now asserts against the real encoder’s output instead of a fixture it invented. The old fixture described itself as “deliberately independent” but was only ever checked against itself, never fed to the reader — so it drifted into a layout Traza has never written (a u32 header length at offset 12, three sections instead of four, a .trz2 extension) and passed continuously while asserting nothing. Feeding it to Segment::from_bytes failed with Corrupt("invalid v2 header length"). The tests still parse the header by hand at fixed offsets — that independence is the point — but they parse bytes from segment::encode, round-trip them through Segment::from_bytes and Segment::open, and pin the four-section layout, the record encoding, and what the reader must reject. reopen_persistence no longer writes bytes and reads them back asserting equality (a test of fs, not of Traza).
• docs/segment-format.md describes the layout that shipped. It previously documented only the pre-implementation proposal — a trailing footer, a JSONL payload, and readable v1 segments — none of which match src/segment.rs; v1 JSONL is rejected with a migration pointer. The proposal is retained below the real layout, marked as history.
• Reconciled the on-disk segment magic. The encoder wrote TRAZAV2 while the acceptance test and format doc expected TRAZASEG; they never met, because the acceptance fixture is self-checked and was never fed to the real reader. The magic is now TRAZASEG (the version lives in the VERSION field, not the magic), matching the docs, and the acceptance test pins it to the real segment::MAGIC constant so the two cannot drift again. On-disk format change: existing .seg files written before this are no longer read (acceptable pre-release; no data migration is provided).

0.15.0 - 2026-07-24

Durability: an acknowledged write now means what the deployment says it means.

Added

• Write-ahead log with group commit. Ingest appends a batch to wal.log and fsyncs it BEFORE acknowledging; the log is replayed into the write buffer on open and reclaimed once a flush seals those spans into a segment. The fsync runs outside the writer lock, so concurrent batches coalesce into one sync instead of serializing an fsync per request — measured 13.7k spans/s at concurrency 1 rising to 48.1k at 16, where per-batch fsync would stay flat.
• Explicit durability modes, selected with --durability and reported in every ingest response and in /v1/stats, so a client never has to infer what a 200 promised:
  • buffered — accepted in memory; lossy by design, and no longer the default.
  • wal (new default) — fsynced to the log and recovered on restart.
  • flushed — present in a sealed segment.
• /v1/stats reports durability and wal_bytes (the work a restart would replay).
• tests/durability.rs holds each mode to its own claim under SIGKILL: wal and flushed lose nothing acknowledged, buffered is verified to be lossy rather than accidentally durable, recovery preserves last-write-wins for a re-ingested span, the log is reclaimed after a flush without losing data, and 800 spans acknowledged across 8 concurrent writers all survive.

Changed

• Config::default() is now Durability::Wal. A store that silently loses acknowledged writes is the wrong default even though it is the faster one. Config gains a durability field, so code constructing it as a struct literal must add one (or spread ..Config::default()).
• The benchmark measures the default (wal) and labels the mode, rather than reporting a buffered number no production deployment could rely on.
• tests/auth.rs no longer pins socket teardown to ECONNRESET; the invariant is that a complete HTTP response arrived, and pinning the errno made the test flaky under parallel load.

Notes

• fsync on macOS does not flush the drive’s write cache (F_FULLFSYNC would, and std does not expose it), so a macOS power cut can still lose an acknowledged write. Process death cannot, on any platform. Documented in the README and src/wal.rs rather than left implied.

0.14.0 - 2026-07-24

OpenLLMetry-native tracing, a standalone dashboard served from its build output, and a trace browser that renders what agents actually produce.

Added

• Traza now follows the OpenLLMetry standard (Traceloop’s OpenTelemetry GenAI conventions). Sessions, token analytics, and the dashboard recognize the current OTel GenAI attributes — gen_ai.provider.name, gen_ai.operation.name, gen_ai.usage.input_tokens / output_tokens, gen_ai.request/response.model, gen_ai.conversation.id, and traceloop.* — so an OpenLLMetry-instrumented app populates every derived view over OTLP with no attribute renaming. The OTel-deprecated names (gen_ai.system, gen_ai.usage.prompt_tokens / completion_tokens) and Traza’s native llm.* / session.id shorthand are accepted as aliases; native behavior is unchanged. A new src/semconv.rs normalization layer is the single source of truth for the key precedence.
• GET /v1/stats/llm?group_by=provider rolls tokens up by the resolved provider.
• GET /v1/spans?session=<id> filters spans to a session, unioning every recognized session key so a session whose spans mix conventions (some session.id, some gen_ai.conversation.id) is returned whole. Sessions are grouped by any recognized key, and each reports the session_attribute that grouped it.
• The dashboard’s span detail renders provider/model/token chips and a Messages panel from the current JSON gen_ai.input.messages / gen_ai.output.messages as well as the legacy indexed gen_ai.prompt.* / gen_ai.completion.* attributes and native llm.prompt / llm.completion events.

Changed

• The dashboard is no longer compiled into traza-server. The server now serves the UI’s build output from disk: --ui-dir (default ./ui/dist, produced by cd ui && npm run build) backs GET / and GET /dashboard. Building the server still needs no Node toolchain, a rebuilt UI is picked up without restarting, and a missing build is not fatal — the API runs and the UI routes 404 with build instructions. The shell stays served before the auth gate (it is static build output carrying no data) while every /v1 call it makes remains gated. Path traversal out of the UI directory is refused.

• License is now Apache-2.0 only (previously dual MIT OR Apache-2.0). LICENSE-MIT is removed and LICENSE-APACHE renamed to LICENSE.

• ci.sh is the merge bar for the whole tree: it now builds and tests the dashboard (npm ci, npm test, npm run build, Node per ui/.nvmrc) and rejects source files containing a NUL byte. Rust tooling cannot police the UI, and a broken Vite build must not merge green. TRAZA_SKIP_UI=1 runs the Rust half alone.

• The dashboard has unit tests (ui/, vitest): message parsing, content-type detection, the markdown subset, and the syntax tokenizer — including that highlighting round-trips to the exact source and never linkifies a javascript: URL.

Fixed

• Session resolution now resolves every recognized key under ONE snapshot of the write buffer and segment list. Querying each key separately let a span re-ingested between the queries be seen first in its superseded version, which then locked the newer version out — breaking last-write-wins during ordinary concurrent ingest.
• A numeric session id ("gen_ai.conversation.id": 4711) can now be opened, not just listed. Normalization stringifies numeric attributes, but the lookup matched only JSON strings, so such a session appeared in /v1/sessions while /v1/sessions/4711 returned 404 and /v1/spans?session=4711 returned nothing.
• The server finds a packaged dashboard: with no --ui-dir it searches $TRAZA_UI_DIR, <binary dir>/ui, <binary dir>/../share/traza/ui, then ./ui/dist, and lists every path it tried when none has a build. A CWD-relative default alone meant an installed binary served nothing unless it was launched from a checkout.
• The conversation view pages through long sessions and says when it is showing a prefix. Spans come back oldest-first, so a fixed cap silently dropped the newest turns while presenting the result as complete.
• ui/src/views/ConversationView.jsx no longer contains a literal NUL byte, which made git treat the file as binary and hid it from diff and blame.

Removed

• The checked-in generated src/dashboard.html and the ui/scripts/embed.mjs script that produced it, along with src/dashboard.rs. UI builds no longer regenerate an embedded HTML file, so ui/ changes no longer produce a 368 KB diff in the Rust crate.

Notes

• Cost analytics remain a Traza extension (llm.cost_usd), not part of OpenLLMetry — OpenTelemetry GenAI defines no cost attribute. Cost populates only when the ingest pipeline supplies it.

0.13.0 - 2026-07-23

Wire-contract release: /v1/stats renames its counters to record terminology and /v1/export switches to chunked framing with completion trailers — clients parsing either surface must update.

Fixed

• Export pagination now uses the engine’s exclusive full-key (start_time, end_time, trace_id, span_id) cursor with a fixed 4,096-row page. Equal-timestamp runs no longer trigger exponential prefix re-fetches or corpus-sized pages, and bounded queries borrow resident posting lists instead of cloning them per page.
• Export responses use HTTP chunked framing with explicit X-Traza-Export-Complete and X-Traza-Export-Count trailers. A storage or serialization failure after 200 OK can no longer masquerade as a complete dataset.
• Annotation replay now tolerates only an unterminated final append. A malformed newline-terminated middle record fails startup instead of silently hiding every valid annotation after it; a torn tail is truncated before new appends, a missing final delimiter is restored, and annotation creation/rewrite renames also fsync the parent directory.
• LLM/session integer counters saturate instead of panicking in debug builds or wrapping in release builds. Non-finite cost strings are ignored and floating sums remain finite.
• Payload sweeping holds the touch-registry lock only across each final eligibility check and deletion. The ingest race remains excluded without a large directory walk stalling every oversized-payload write.

Changed

• /v1/stats and Store::stats now name their cheap physical storage counts as record_count / *_records. Immutable historical versions remain physical records until compaction even though last-write-wins queries expose one logical span.
• The server now binds 127.0.0.1 by default. Unauthenticated non-loopback binds are refused unless the operator configures TRAZA_TOKENS or passes --allow-unauthenticated-non-loopback explicitly.
• Current documentation now matches the v2-only file-backed engine, OTLP/HTTP protobuf support, v0.12 crate line, export integrity contract, and safe bind defaults.

0.12.2 - 2026-07-23

Fixed

• Payload TTL race (found in review): compaction snapshotted live references, released the locks, then swept — an ingest committing a new reference to an old deduped file inside that window authorized its deletion. The store is single-process (DirectoryLock), so an in-memory touch registry now records every payload write/dedup BEFORE filesystem work; the sweep spares anything touched within a 10-minute immunity window in addition to the live-reference set.
• Concurrent identical-payload ingest (found in review, reproduced as 9 successes + one ENOENT): all writers shared one <hash>.tmp path, truncating each other’s temp and racing the rename. Temps are now writer-unique; every rename is valid, and identical content makes the last rename byte-identical anyway.
• Export truly streams (found in review): GET /v1/export materialized the complete query result plus a complete NDJSON buffer, defeating the larger-than-RAM design. It now streams close-delimited (no Content-Length) in bounded pages keyed by the query’s total sort order, holding no engine lock across socket writes; equal-timestamp runs wider than a page grow the page until the cursor can cross them.

0.12.1 - 2026-07-23

Fixed

• Payload TTL deleted live data (found in review, reproduced): the content-addressed store dedupes identical payloads to one file WITHOUT refreshing its mtime, while the TTL sweep deleted by mtime alone — a fresh span re-referencing old content kept its span but lost its payload. The sweep now protects every payload referenced by a live span (buffer + all segments, collected via the cached rollups) and deletes only unreferenced-and-old files.
• LLM/session rollups double-counted replaced spans (found in review, reproduced): cached per-segment rollups summed every physical copy of a re-ingested (trace_id, span_id), contradicting the primary key’s last-write-wins semantics — the aggregate said 2 calls / 30 tokens / $0.30 where the visible truth was 1 call / 20 tokens / $0.20. Rollups now walk segments newest-first carrying the seen-key set (FNV-1a prefilter; buffer always wins); a segment containing any possibly-superseded key is re-scanned exactly, dropping stale versions. Collisions can only cost an unnecessary re-scan, never a wrong count.

0.12.0 - 2026-07-23

Added

• Payload offloading: string attribute values above a threshold (server default 256 KiB, --payload-threshold-bytes, 0 disables) are extracted at ingest to a content-addressed store (payloads/<aa>/<sha256>.bin, temp+rename writes) and replaced by {"$payload": "sha256/…", "bytes": N, "preview": "…"}. Identical payloads are stored once; GET /v1/payloads/{ref} serves the bytes (hex-validated — traversal-shaped refs are 404). SHA-256 is implemented in-crate (FIPS 180-4) and verified against the NIST vectors.
• Annotations: post-hoc scores/feedback/eval verdicts attach to spans (or whole traces) without mutating them — an append-only, fsync’d annotations.jsonl with an in-memory index, tolerant of a torn tail. POST /v1/annotations, GET /v1/annotations, and the trace view carries a trace’s annotations alongside its spans.
• Dataset export: GET /v1/export streams any span filter as NDJSON (unbounded by default, unlike interactive search) — the traces-to-eval-dataset path.
• TTL compaction now also drops annotations older than the window and sweeps payload files by mtime (an orphan payload outlives its span by at most one TTL).

0.11.0 - 2026-07-23

Added

• OTLP/HTTP binary protobuf: POST /v1/traces now accepts Content-Type: application/x-protobuf — the encoding OTel SDKs use with OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf — via a dependency-free, bounds-checked wire decoder that lowers protobuf into the OTLP/JSON shape, so both encodings share one mapping. Protobuf clients receive a protobuf-typed empty ExportTraceServiceResponse. Malformed payloads (truncated varints, lying lengths, 32-deep hostile nesting) are 400s, never panics; unknown fields skip per the protobuf contract. gRPC is not served.
• Span links: spans carry a first-class links array (trace_id, span_id, attributes) on the native JSON surface and the OTLP mapping (previously dropped) — the non-tree structure of agentic traces: fan-out/fan-in, retries, cross-agent causality.
• Conformance suite with an independent test-side protobuf encoder (tests/otlp_protobuf.rs).

0.10.0 - 2026-07-23

Added

• Sessions: any span carrying a session.id attribute joins a session — the unit of agentic/LLM work spanning many traces. GET /v1/sessions lists sessions (span/trace counts, token sums, cost, errors, activity window), GET /v1/sessions/{id} adds the per-trace breakdown. The dashboard grew a Sessions panel; clicking a session filters the span table to it.
• LLM aggregation: GET /v1/stats/llm?group_by=model|service|session|day returns exact token/cost/error/latency rollups over an optional since/until window. Numeric attributes are accepted as numbers or numeric strings; explicit llm.total_tokens wins over the prompt+completion sum.
• Aggregation cost model: sealed segments are immutable, so per-segment rollups are computed once and cached; query windows that split a segment decode just that segment for exact edge membership. Rollups for superseded (compacted) segments drop out automatically.
• session.id added to the LLM semantic conventions (docs/llm-semantics.md) with a sessions/aggregation section.

0.9.2 - 2026-07-23

Security / Hardening

• The engine itself now rejects spans with an empty trace_id or span_id (Error::InvalidSpan) — the primary-key invariant no longer depends on each HTTP surface validating correctly, and a batch with any invalid span stores nothing.
• Socket read/write deadline (30s, TRAZA_SOCKET_TIMEOUT_MS to override): a peer that connects and goes silent — or declares a body it never sends — is released instead of parking a worker thread forever.
• Request headers are capped at 64 KiB (bodies keep the 64 MiB cap). Previously a request could spend the full body budget on headers, doubling the per-request memory ceiling.
• With auth enabled, requests are refused from the head alone: an unauthenticated client can no longer make the server buffer a declared 64 MiB body before hearing 401.

Added

• tests/ingest_hardening.rs: adversarial wire probes — lying content-length, oversized headers, silent peers, pre-auth body refusal, malformed OTLP shapes, query-parameter extremes, inverted timestamps, hostile ids (NUL-prefixed / unicode) round-tripping through flush and reopen, and the documented last-write-wins semantics for duplicate keys within one batch.
• Test harnesses kill their spawned server on every exit path (Drop), so a failing test reports instead of hanging cargo test on the leaked child’s pipes.

0.9.1 - 2026-07-23

Fixed

• POST /v1/spans now rejects spans with an empty span_id (400, naming the span index), matching the existing empty-trace_id rejection. Both halves of the (trace_id, span_id) primary key must be non-empty: previously two distinct spans with empty span_id were both counted in {"accepted": N} while the upsert silently collapsed them into one stored span. The OTLP endpoint already rejected empty ids; the native endpoint now agrees. Rejection is atomic — nothing from a rejected batch is stored.

Added

• docs/ha-design.md: the high-availability design document — four compared architectures with a quorum-replicated logical-log recommendation (segment shipping retained for catch-up), grounded in the real engine mechanisms (WriteBuffer acknowledgment boundary, segment_v2 snapshot transfer, replicated supersede-journal transitions, (trace_id, span_id) idempotency, DirectoryLock scope). Design only; no HA behavior is implemented.

0.9.0 - 2026-07-23

Added

• Bundled dashboard: a dependency-free trace browser embedded in the server binary (src/dashboard.html via include_str!), served at GET / and GET /dashboard. Recent-spans view with a filter bar mapped 1:1 onto the /v1/spans query params, a trace waterfall backed by /v1/traces/{id} with error spans highlighted, and a span detail pane (attributes, events, parent, extra fields). Light and dark color schemes follow the browser preference.
• The dashboard consumes only the existing JSON API — no new endpoints. With TRAZA_TOKENS set the shell stays open (it carries no data) while every API call remains gated; the page prompts for a bearer token on the first 401 and stores it in sessionStorage only.
• traza::dashboard: the embedded asset and route helper (route(path) -> Option<DashboardResponse>), consulted by the server before the auth gate for GET requests.
• tests/dashboard.rs: process-level acceptance — the real server serves the embedded page at /, /dashboard, and /dashboard/ (unknown deeper assets 404); a grep oracle proves the page references no external URLs (self-contained, no supply-chain surface); with auth enabled the shell loads open while /v1/* still returns 401/403/200 by scope.

Changed

• README: Features and Roadmap now reflect shipped OTLP ingest, auth, LLM-observability semantics, and the dashboard; remaining roadmap is streaming results, filter throughput at scale, and high availability.

0.8.0 - 2026-07-23

Added

• Roadmap leg 4 — bearer-token auth. TRAZA_TOKENS (comma-separated scope:token, scopes ro|rw) requires Authorization: Bearer on every request: unknown tokens 401 with a WWW-Authenticate: Bearer challenge, insufficient scope 403, token comparison constant-time (all credentials checked even after a match). Unset means open — the development default that keeps every existing test unchanged — while a set-but-invalid value refuses startup rather than silently running open. Zero new dependencies; process-level matrix tests cover open mode, 401/403/200 across ingest, OTLP, and flush endpoints, and the startup refusal.

0.7.0 - 2026-07-23

Added

• Roadmap leg 3 — LLM-observability semantics. Documented gen-AI span conventions (docs/llm-semantics.md): llm.* span names, model/token/temperature/stop-reason/tool/cost attributes (index-served like any attribute), prompt and completion payloads as span events so large text never enters the filter index, and four concrete query recipes over the existing API. Process-level tests prove every recipe through both /v1/spans and OTLP ingest. Purely additive: one doc, one test target, one README section.

0.6.0 - 2026-07-23

Added

• Roadmap leg 2 — OTLP/HTTP JSON ingest. POST /v1/traces accepts an OpenTelemetry ExportTraceServiceRequest in OTLP/HTTP JSON and maps it onto the span model: hex ids lowercased, *TimeUnixNano accepted as string or number, typed AnyValue attributes (string/int/double/bool/ array/kvlist) flattened to plain JSON, resource service.name becoming the span’s service (unknown_service fallback), scope attributes merging beneath span attributes, events mapped, and OTLP status codes becoming ok/error/empty. Structurally invalid requests 400 with a diagnostic; the existing /v1/spans contract is untouched. No new dependencies. Conformance-tested end to end against the real binary, including index-served queries over OTLP-ingested spans.

0.5.0 - 2026-07-23

Changed

• Roadmap leg 1 — larger-than-RAM reads. Segments are file-backed: Segment::open reads only the header and index sections into memory and serves every record access by reading exactly the needed byte range from the file (std Seek+Read; no mmap, no new dependencies). Flushing reopens the new segment file-backed, so no resident payload copy survives the write either. Store::resident_payload_bytes() exposes the invariant (zero after open and after flush). Measured at a 10M-span corpus (2.4 GB on disk): 0.71 GB peak server RSS (was ~2.4 GB bytes-resident, ~5 GB in the pre-v2 engine); trace lookup p50 0.8 ms, attribute filter p50 8.7 ms — RAM is O(indexes) and stores larger than memory serve correctly.
• The lazy limited-query merge caches per-source head timestamps: with file-backed segments every peek is a disk read, and re-peeking all sources per pop had regressed the 10M filter to 125 ms; cached heads restore 9.6 ms.

Known deviation

• The leg’s relative bound (“1M latencies within 2x of 0.4.0”) is missed on filter p95: 3.34 ms vs 1.27 ms (2.6x) — the cost of on-demand file reads at sub-5 ms magnitudes. The absolute gate (< 300 ms) passes with ~90x headroom; recorded here rather than tuned away.

0.4.0 - 2026-07-23

Changed (breaking)

• Span identity is a primary key. (trace_id, span_id) is enforced unique: re-ingesting an existing pair replaces the stored span — in the write buffer, across flushes, and across restart. Last write wins on every read path (trace, filtered, and limited lazy queries), so client retries are idempotent and never produce duplicate copies. This reverses 0.3.1’s at-least-once visible-duplicate semantics.
• v1 JSONL segments are no longer read. The engine is v2-only; opening a directory containing a legacy .jsonl segment fails loudly with a migration pointer (read with 0.3.x first). The dual-format code path is removed.

0.3.1 - 2026-07-23

Fixed

• Data loss across restart: next-segment numbering only recognized .jsonl names, so a reopened v2-only store restarted at id zero and the next flush renamed over an existing segment, destroying persisted spans. Both suffixes count now, and write_segment refuses to replace an existing file outright.
• Acknowledged duplicate cardinality survives restart: content-based duplicate healing is gone. Compaction rewrites are journaled with a supersede marker written before the rewrite begins; recovery finishes an interrupted rewrite from the journal in either direction and never deduplicates by content, so legitimately re-ingested identical spans keep both acknowledged copies.
• A corrupt v2 header with an out-of-range attribute-index offset returns Error::Corrupt instead of panicking through unsigned subtraction.
• User attributes named with a NUL prefix (for example "\u{0}service") can no longer overwrite the reserved service/name index keys and poison those queries; such attributes are stored verbatim but excluded from the index, and filters on them decline index use symmetrically.

Changed

• Limited queries are lazy end to end: per-segment index postings stay undecoded and a k-way merge pops candidates in start-time order, decoding and re-verifying only what the limit returns. Measured: attribute filter p50 18 ms -> 0.53 ms at 1M spans and 209 ms -> 2.9 ms at 10M — the 10M advisory target (<100 ms) is closed with 35x headroom.
• README limitations and roadmap reflect the v2 engine (byte residency, journaled compaction, remaining mmap/streaming work).

0.3.0 - 2026-07-23

Changed

• Segment format v2: new segments are indexed binary files (.seg) — JSON span payloads with an embedded record-offset index, trace index, and attribute index, written with the same temp + fsync + atomic-rename discipline. v1 JSONL segments remain fully readable beside v2 and heal through the same duplicate-recovery path; TTL rewrites produce v2.
• Byte-resident reads: opening a store no longer materializes spans. v2 segments hold raw bytes plus their indexes; spans parse on demand, only for records a query returns. Store::resident_persisted_span_structs exposes the invariant (zero after a v2-only open).
• Index-served queries: get_trace binary-searches the trace index; filters narrow through service/name/attribute indexes or time range and re-verify every predicate on the parsed span. Measured on the bundled benchmark: trace lookup p50 0.185 ms at 1M spans (was 14.2 ms) and 0.536 ms at 10M (was 145.6 ms); attribute filter p50 18 ms at 1M (was 66.5 ms) and 209 ms at 10M (was 4,395 ms).

Known limits

• The 10M advisory filter target (<100 ms) is not yet met: candidate payload parsing dominates large result groups. Posting-list intersection and parse-avoidance are the next optimization.
• Segment bytes are read into memory at open (no mmap yet); resident cost is file bytes + indexes rather than parsed structs.

0.2.3 - 2026-07-22

Fixed

• Segment writes are buffered. serde_json::to_writer against a raw File issued one write() syscall per JSON token, making flush cost ~140 us per span and capping measured end-to-end ingest at 5,450 spans/s. A 256 KiB BufWriter restores flush to ~1.5 us per span; the regenerated benchmark measures 138,180 spans/s and the ingest gate passes again.

0.2.2 - 2026-07-22

Fixed

• ttl_seconds: Some(0) disables expiration as documented instead of expiring every existing span; the library Config TTL default is None as documented, no longer a silent seven days.
• Recovery heals crash-duplicated segments: exact-duplicate spans are dropped at open and fully-duplicate segment files deleted, closing the window where a crash mid-compaction returned two copies of surviving spans on reopen.
• An empty reclamation sentinel left by a reclaimer that died before recording its PID no longer wedges lock recovery: unreadable sentinels older than ten seconds are treated as corpses.
• The README introduction and features no longer claim a manifest, per-segment indexes, or log replay; the configuration tables match the code (server --ttl-seconds drives engine compaction; --host and --flush-spans documented).

Changed

• BENCHMARKS.md regenerated against the engine-backed server. The ingest gate is honestly reported as MISSED (5,450 spans/s against the 50,000 target); read gates pass. Closing the write-path gap is the top of the roadmap.

0.2.1 - 2026-07-22

Fixed

• The documented ingest contract works again: timestamp aliases (start_time_unix_nano, start_timestamp_ns, start_ns, start_time and the matching end_* keys) are accepted, parent_span_id, status, attributes, and events are optional, and unknown span fields are stored and returned verbatim instead of silently discarded. This also un-breaks the bundled benchmark, which emits start_ns/end_ns.
• The documented search filters work again: attr.KEY, min_duration_ms, since/until, and the default limit of 100.
• /v1/stats exposes the documented span_count, segment_count, and bytes_on_disk keys alongside the engine’s finer-grained fields.
• The server binds 0.0.0.0 again by default; --host overrides.
• TTL expiration no longer empties the in-memory segment set when a file operation fails mid-compaction; the store keeps serving its previous view and surfaces the error.
• Stale-lock reclamation is single-winner: a reclamation sentinel closes the window in which a slow reclaimer could delete a fresh lock and defeat the single-writer guarantee.
• The README architecture section describes the engine as built (JSON-lines segments, memory-resident, linear scans, no manifest or per-segment index yet) instead of the roadmap design, and names the compaction crash-atomicity bound in Limitations.

0.2.0 - 2026-07-22

Changed

• traza-server is engine-backed: the segment engine is the server’s only datastore. The HTTP wire contract is unchanged; the server-side append-only log, in-memory indexes, and startup replay are removed, and restart durability and crash recovery are the engine’s.

Added

• POST /v1/flush forces buffered spans into a durable segment on demand.
• --flush-spans server flag to tune the engine’s flush threshold; --port 0 binds an ephemeral port announced on stderr.
• Five process-level server_on_engine integration tests that drive the real server binary end to end, including an engine-authority cross-check and a kill-and-restart persistence test.
• Stale-lock reclamation: Store::open reclaims a lock file whose recorded owner process is verifiably dead, so a crashed server cannot permanently wedge its data directory. A live owner still rejects the open.

Fixed

• Deadlock-capable lock-order inversion between flush() and stats(); all operations now follow a documented writer-before-segments discipline.
• query() and get_trace() now take an atomic combined snapshot of buffered and persisted spans, so a concurrent flush can no longer hide committed spans.
• Crash-orphaned segment temp files no longer wedge subsequent flushes: temp names are unique per process, and Store::open removes orphans during recovery.
• Concurrent writers are rejected: Store::open holds a lock file for the store’s lifetime and a second open fails with Error::AlreadyOpen.

Added

• Concurrency and failure-injection tests: deadlock detection, read-during-flush consistency (exactly-once), stale-temp recovery, and second-open rejection. Direct per-segment ordering assertion in the flush test.

• Renamed the project and crate to Traza (traza).

0.1.0 - 2026-07-20

Added

• The tracing storage engine exposed as a Rust library.
• The traza-server HTTP server for the documented ingestion and query endpoints.
• The bench executable for measuring the existing datastore workloads.
• Four behavioral integration tests: buffer-flush persistence, crash recovery via reopen, randomized filter equivalence against an independent naive reference, and TTL compaction. (Persisted batch ordering is asserted only indirectly; a direct segment-order assertion arrives with the storage-correctness work.)
• Crate documentation, dual MIT/Apache-2.0 licensing, and release automation.

Known Limitations

• This is an initial 0.1 release; consult README.md for the currently documented operational constraints and unsupported use cases.