Counting, done carefully.

1 Runs become paths

A run’s steps — its model, tool and agent calls, in start order — become tokens kind:name: llm:gpt-4o, tool:search_orders. That is a step’s signature without its input, so two calls of one tool are the same behaviour whatever their arguments. Consecutive repeats of one token fold into a single token by bucket, so a loop is one shape rather than a tree that grows a level per iteration:

So live-doomed-0, 5 steps in, is the path llm:gpt-4o → tool:search_orders×3-5. A path holds at most 64 tokens and a run is read from its earliest 192 steps; past either, the path ends in … and the build says so. Names are cut at 160 bytes.

2 Cohorts and outcomes

Runs are compared only with runs of the same agent in the same tenant. A run’s outcome is its latest declared session.outcome, else the status of the span that ended last: success, failure or abandoned. A run with no declaration whose last span ended in the last 15 minutes is still active; it is unknown, excluded from every rate, and counted.

Completed runs

180

Succeeded

98

Failed · abandoned

73 · 9

Unknown (in flight)

6

The base rate is successes over completed runs: 98 ÷ 180 = 54.4%. A map reads the last 30 days by default.

3 Why Wilson intervals

Every node of the tree carries its success rate and a 95% interval. Most branches are small, and the textbook interval p ± 1.96·√(p(1−p)/n) lies about small samples: for 0 of 3 it says 0.0–0.0%, “0%, give or take nothing”. Wilson’s score interval says 0.0–56.2%, which is what three runs actually tell you. Every rate Foresight acts on is judged by its interval, not its point estimate.

On the demo map, the search loop’s 5 of 50 is 4.3–21.4% by Wilson — exactly the interval the engine reports for that node, 4.3–21.4% — where the textbook formula would say 1.7–18.3%. The site’s tests recompute every interval in the demo numbers this way.

4 Fatal forks and hotspots

A child is a fatal fork when three things hold: its success rate is at least 25 points below its siblings’ pooled rate; it and its siblings each have at least 3 runs; and a two-sided Fisher exact test puts the difference at p < 0.05. The last condition is what keeps a handful of unlucky runs from being called a fork.

The support agent’s search loop is the strongest fork in the demo: tool:search_orders×3-5, 10% against 72%, p = 2.2 × 10⁻¹⁴.

A hotspot is a node where runs cannot make up their mind: at least 1 bit of entropy in what they did next, with at least 3 visits. In the demo, llm:gpt-4o is one: after it, 180 runs went on with tool:search_orders, tool:search_orders×3-5, tool:lookup_customer — 1.50 bits.

5 Verdicts for one run

A run’s path is walked down its agent’s map. A completed run is left out of the statistics it is judged by. A live run’s last step is still open, so it is pooled with every run that repeated that step at least as often: that is why live-healthy-0, one refund in, reads 69 of 90 — the runs through tool:issue_refund and tool:issue_refund×2.

• likely_failure live-doomed-0: runs that took this path succeeded 5 of 50 times (10.0%, 95% CI 4.3–21.4%); the agent's overall success rate is 54%. Upper bound 21.4% < 50%.
• on_track live-healthy-0: runs that took this path succeeded 69 of 90 times (76.7%, 95% CI 66.9–84.2%); the agent's overall success rate is 54%. Best next step: llm:gpt-4o, the option with the highest Wilson lower bound among options with at least 30 runs.
• novel live-novel-0: no earlier run of this agent took tool:delete_database at this point; the 180 runs that got here went on with tool:search_orders, tool:search_orders×3-5, tool:lookup_customer.

6 Predictive fuses

A fuse policy may set min_success_odds. At the pre-call check — never on ingest — when a run’s foresight stands on at least 30 earlier runs (foresight_min_support), the run has at least 3 steps (foresight_after_steps), and the interval’s upper bound is below the odds, the policy’s action applies. Comparing the upper bound, not the estimate, means the history must be strong before anything is stopped: 0 of 3 never stops a run.

• stop live-doomed-0 at odds 0.7: foresight: runs that took this path succeeded 5 of 50 times (≤21.4%)
• stop live-healthy-0, naming its next tool: foresight: runs that took this path and then called tool:issue_refund succeeded 15 of 30 times (≤66.8%) (its foresight verdict is still on_track: the verdict compares with the agent, the limit with the policy)

The check reads a run’s path from the run guard’s memory and walks it down a cached map: it never builds one. The background builder takes requested maps every two seconds, two at a time, and a built map is served for 300 seconds by default, so a server’s first checks see no foresight for a moment — and missing evidence is never treated as evidence of failure. Any erasure drops every cached map. A foresight stop is not latched: it is made again at every question.

7 Calibration

A prediction is only worth what it has been worth. Each map predicts every completed run at several depths from the half of the cohort it is not in (2-fold cross-validation, split by a hash of the run id), using only what the run had done by then — or the half’s base rate when fewer than 10 of its runs got there. Predictions are scored by Brier score, and skill = 1 − brier / baseline_brier is how much better than “assume the average” the map does.

8 Limits

• Correlation, not cause. A fatal fork says runs that went that way ended worse. The step may be a symptom rather than the cause; the fork tells you where to look, and its example runs are a success and a failure from the same point to diff.
• Paths ignore inputs. A behaviour that differs only in content — arguments, prompts — is invisible to the map.
• Outcomes are inferred. A run that failed but ended on a successful span reads as a success. Declare session.outcome where you can.
• Many nodes, some luck. No correction is made for how many nodes were tested. The 25-point gap and the support rule make chance forks rarer, and every fork carries its p-value.
• The cohort is the agent, whatever changed. A new prompt, model or tool catalog mid-window mixes two behaviours in one map; narrow the window to after the change.
• Buckets blur an open step. A run on its fourth search is pooled with the ×3-5 runs that stopped at three, as well as those that went on.
• The fuse reads every run as live, so even an ended run has its last step pooled; the run route judges a completed run at its own node.
• Bounded builds. A cohort map holds at most 20,000 completed runs and a build 50,000 nodes; a build that meets a bound says so, and a path into a cut branch gets insufficient_history, never a guess.
• Calibration is noisier than the map. It predicts from half the cohort, so a small cohort’s skill moves with the split.
• One process. Maps are cached per server; replicas build their own.

Every threshold on this page is the engine’s, quoted from its Foresight guide; the site’s tests check each quote is still there.

Demo corpus Real output of traza-server 0.28.0 over the engine’s seeded behaviour corpus (226 runs, 1,342 spans), recorded 2026-10-18 by scripts/demo-numbers.mjs. Synthetic runs with known outcomes, not customer data.