From the open engine’s docs (traza 0.28.0). View source Engine routes reach a Traza Cloud project through the gateway.
TQL: query agent behaviour like a regex
Traza holds every step of every run an agent made. TQL (trajectory query language) asks whether a run’s steps contain a shape: a failing test, then an edit to a test file, then a passing test; the same file edited five times within ten steps; a tool the model was never offered; two agents handing work back and forth. It is a regular expression whose letters are steps.
test[error] ..> edit[file~"(^|/)tests?/"] ..> test[not error]
Read it as: a test run that failed, then (eventually) an edit to a file
under tests/, then (eventually) a test run that passed.
On top of the language sits a detector library — nine named patterns for behaviours worth knowing about (test tampering, give-up loops, hallucinated tools, edit thrash, agent ping-pong, blind retries, context blow-ups, error streaks, unverified finishes) — and a findings report that runs the whole library over a window and compares how the matching runs ended against the rest.
Everything here is deterministic computation over stored spans. Traza never
calls a model to judge a run, and nothing leaves the process. Route shapes are
in the HTTP API reference; the
agent-facing tool is find_behavior.
- See it in thirty seconds
- See it run
- Steps
- Atoms
- Predicates
- Fields
- Composing steps
- Captures
- How a match is chosen
- The regex subset
- Grammar
- Worked examples
- Searching
- The detector library
- From an agent
- Limits, stated plainly
See it in thirty seconds
A coding agent ran three sessions. In tampered it failed the tests, edited
tests/test_login.py, and passed; in honest it failed, edited
src/login.py, and passed; in thrash it edited src/app.py five times,
each edit undoing the one before, and gave up. Which runs passed their tests by changing the tests?
curl -sG localhost:8080/v1/trajectories/search \
--data-urlencode 'q=test[error] ..> edit[file~"(^|/)tests?/"] ..> test[not error]' \
--data-urlencode 'since=1772323199999999999'
{
"matches": [
{
"agent": "coder",
"captures": {},
"captures_truncated": false,
"count": 1,
"count_capped": false,
"first_step": 1,
"inexact": false,
"last_activity_ns": 1772323205000000000,
"last_step": 4,
"outcome": "success",
"run": "tampered",
"run_truncated": false,
"spans": [
{
"kind": "tool",
"name": "Bash",
"span_id": "tampered-1",
"step_index": 1,
"trace_id": "tr-tampered"
},
{
"kind": "tool",
"name": "Edit",
"span_id": "tampered-3",
"step_index": 3,
"trace_id": "tr-tampered"
},
{
"kind": "tool",
"name": "Bash",
"span_id": "tampered-4",
"step_index": 4,
"trace_id": "tr-tampered"
}
],
"spans_truncated": false,
"steps": 6,
"tenant": ""
}
],
"next_cursor": null,
"query": "test[error] ..> edit[file~\"(^|/)tests?/\"] ..> test[not error]",
"runs_in_window": 3,
"runs_inexact": 0,
"runs_scanned": 3,
"runs_skipped": 0,
"since_ns": 1772323199999999999,
"spans_skipped": 0,
"stopped": null,
"truncated": false,
"until_ns": null,
"window_capped": false
}
spans are the steps the pattern’s atoms matched — the failing test (step 1),
the edit (step 3), the passing test (step 4). Step 2, a model call the ..>
skipped, is not listed. Step indices are the run’s
tape indices, so GET /v1/tapes/tampered shows
each one in full.
Always URL-encode the query (curl -G --data-urlencode). An unencoded +
arrives as a space, so tool(bash)+ would be read as tool(bash); the
response’s query field shows the pattern exactly as it was read, so a
surprise is visible.
See it run
examples/coding-agents/run.sh
runs the detector library over imported Claude Code and
Codex sessions: test_tampering finds the session that failed a test,
edited the test’s expectation and passed, and unverified_finish the
sessions that stopped after an edit without running the tests — one of
which stops matching once --watch delivers its test run. A query of its
own, edit[file~"(^|/)migrations/"], finds the migration written after a
compaction.
Steps
A run’s steps are its spans whose step kind is llm, tool or agent
(step kinds), in step order
(start, trace, span) — exactly the steps of its tape, with the same indices.
Workflow wrappers, retrievals and HTTP spans are not steps. A run is a session
id, or trace:<trace_id> for a span with none
(run identity).
A pattern is matched against that sequence. An atom matches one step; the operators combine atoms the way a regex combines characters.
Atoms
| Atom | Matches |
|---|---|
step, step(NAME) |
any step; NAME is its tool, model or agent name |
tool, tool(NAME) |
a tool step; NAME is its tool name |
llm, llm(MODEL) |
a model call; MODEL is its model (response model, else request model) |
agent, agent(NAME) |
an agent invocation; NAME is its agent name |
edit, edit(PATH) |
a tool step that edits a file (see edit); PATH is tested against every file it names |
test |
a tool step whose input runs a test suite (see test); takes no name |
A name is one of:
| Form | Example | Means |
|---|---|---|
| bare word | tool(bash) |
equal, ignoring ASCII case — Bash matches |
| quoted | tool("str_replace_editor") |
the same, for names with other characters |
| regex | tool(/^mcp__/) or tool(~"^mcp__") |
the regex finds a match in the name |
| capture | tool(@t) |
captures the name (see captures) |
| any | tool(*) |
any name — the same as tool |
Alternatives are separated by |: tool(grep|glob|read). When a step has no
tool, model or agent name, its span name stands in.
Predicates
Square brackets after an atom (or inside a gap) test the step:
tool(bash)[args~"pytest"]
step[status=error]
llm[cost>0.5]
step[duration>30s]
tool[result~"(?i)permission denied"]
edit[file~"test_"]
Several brackets must all hold. Inside one bracket, conditions combine with
and (or a comma), or, not, and parentheses; not binds tightest, then
and, then or: [error or status=timeout, cost>0.1] is
[(error or status=timeout) and cost>0.1].
A condition is a field alone (true when present and truthy: non-empty text, a
non-zero number, true, a non-empty list) or field OP value:
| Operator | Means |
|---|---|
=, != |
equal / not equal (== is accepted for =) |
~, !~ |
the regex finds a match / finds none |
>, >=, <, <= |
numeric comparison |
Values are a quoted string ("…" or '…'), a bare word (error,
src/main.rs, gpt-4o), a number (0.5, -3), a duration (500ms, 30s,
2m, 1h, 1d; also ns, us), true / false, a regex (/…/ or
/…/i after ~; a string after ~ is a regex too), a capture (@f), or a
scaled capture (2*@c).
In a quoted string, \\, \", \', \n, \t and \r are escapes and any
other backslash is kept as written, so a regex’s \d or \. reaches the
regex engine as typed.
Missing values. A comparison with a field the step does not have is
false — [cost>0.5] does not match a step that reported no cost — and !=
and !~ are its negation, so [file!="a.py"] matches a step with no file at
all. A list field (file, calls, offered, unknown_calls) matches =
or ~ when any of its values does; != when none does.
Types are checked when the query is parsed. ~ takes text (cost~"1" is
refused), ordering takes numbers (status>3 is refused), duration takes a
unit (duration>30 is refused — write 30s), and booleans are tested bare,
with not, or with =true / =false.
Fields
| Field | Type | Meaning |
|---|---|---|
kind |
text | llm, tool or agent |
name |
text | the tool, model or agent name (else the span name) |
span |
text | the span’s operation name |
service |
text | the emitting service |
agent |
text | the agent the step ran under: its own agent name, else the nearest enclosing span’s that names one |
model |
text | the model, when the span names one |
tool |
text | the tool name, for tool steps |
status |
text | the span status as recorded |
error |
bool | the status is error (any case) |
cost |
number | USD, metered by the span or priced from the server’s rates |
duration |
duration | end minus start |
tokens |
number | total tokens reported |
input_tokens, output_tokens |
number | prompt / completion tokens reported |
cache_read, cache_write |
number | prompt tokens read from / written to a provider cache |
context |
number | the context the call carried, where that can be told (why it is not just input tokens) |
index |
number | the step’s position in the run, from 0 |
args (alias input) |
text | the step’s input as text |
result (alias output) |
text | the step’s output as text |
args.<path>, result.<path> |
dynamic | a value inside the input / output, by key and array index: args.command, args.edits.0.new_string |
attr.<key> |
dynamic | any span attribute, verbatim: attr.http.status_code, attr."key with spaces" |
file |
list | paths named by a tool step’s arguments |
digest |
text | SHA-256 of the step’s stored input — equal digests, identical input |
fingerprint |
text | kind:name#hash, the repeat identity the guard uses |
signature |
text | kind:name |
calls |
list | tool names an llm step’s response asked for |
offered |
list | the tool catalog an llm step offered; on a tool step, its requesting llm step’s |
in_catalog |
bool | a tool step’s name is in its requesting llm step’s catalog, ignoring ASCII case — absent when that catalog is unknown, or was cut at 512 names without listing it |
unknown_calls |
list | calls an llm step made to tools outside its own catalog (ignoring ASCII case) — absent when the catalog is unknown or was cut at 512 names |
edit |
bool | a tool step that edits a file |
test |
bool | a tool step whose input runs a test suite |
conversation |
text | the conversation the step names (gen_ai.conversation.id) — the recorders give a session’s main thread one and each sub-agent its own |
added |
text | an edit step’s new text (see added, removed) |
removed |
text | the text an edit step replaced — absent for a whole-file write |
Text fields. args and result are the stored value as text: a string as
itself, a scalar as its JSON text, a structure as its string leaves in
document order (object keys sorted), one per line. ^ and $ in a regex
anchor at line boundaries, so args~"^pytest" matches a command leaf that
starts with pytest. A dynamic field yields text, a number, a boolean, or —
for an array of scalars — a list.
Every text is capped. Whatever a predicate reads — args, result, a
value at args.<path> or result.<path> (an object there reads as its JSON
text), an attr.<key>, a name, each entry of a list — is cut to its first
64 KiB. A list holds at most 512 entries.
Derived fields
These are heuristics, documented so their edges are known:
file— the string values of the keysfile_path,filePath,path,file,filename,target_file,notebook_pathandabsolute_pathanywhere in a tool step’s arguments, plus the file headers of a patch (*** Update File:,*** Add File:,*** Delete File:,*** Move to:,+++ b/). At most 512 per step.edit— a tool step that names at least one file and either is one of the editing toolsEdit,MultiEdit,Write,NotebookEdit,apply_patch,str_replace_editor,str_replace_based_edit_tool,edit_file,write_file,create_file,replace_in_file,write_to_file,apply_diff,search_replace,insert_edit_into_file(any case), or carries new content undernew_string,new_str,content,contents,file_text,edits,patch,diff,code_editornew_source, or contains a patch — and whosecommandargument, if any, is notvieworread.test— a tool step that is not aneditand whose command runs a test runner at a command position. The command is the value of the argumentcommand,cmd,commands,scriptorshell_command(an array of strings, as Codex records argv, is joined with spaces), or the arguments themselves when they are one string; file contents, descriptions and other arguments are never read. A command position is the start of a line or the point after&&,||,;,|,(, a backtick or$(, optionally followed by environment assignments (CI=1), wrappers (sudo,env,time,nice,exec,command,npx,bunx,uvx,yarn,pnpm,uv run,poetry run,pipenv run,pdm run,hatch run,rye run,npm exec,pnpm exec,yarn dlx,pnpm dlx,xvfb-run,timeout N,sh|bash|zsh|dash -flags, then an optional quote) and a directory (.venv/bin/,./). The runners:pytest,py.test,tox,nox,jest,vitest,mocha,rspec,phpunit,ctest,go test,cargo test,cargo nextest,npm|yarn|pnpm [run] test,bun test,deno test,python -m pytest|unittest,mvn … test|verify,gradle[w] … test|check(also:module:test),make … test|check,dotnet test,swift test,mix test,rake test,bundle exec rspec|rake. Socd repo && pytest,uv run pytestandCI=1 npm testare test runs;pip install pytest,cat pytest.ini,grep pytestand anEditwhose new text saysimport pytestare not. The command is matched as text, not parsed as shell: a runner after a;inside a quoted string still counts. A test run “fails” when its span status iserror— which is what a shell tool’s span records for a non-zero exit only when its instrumentation maps the exit code to the status.calls— entries of atool_callslist (nameorfunction.name), content blocks typedtool_call,tool_useorfunction_call, and a legacyfunction_callobject, anywhere in an llm step’s response.offered—gen_ai.tool.definitions(a list, or its JSON text, of{name}or{function: {name}}), OpenLLMetry’sllm.request.functions.{i}.name, or OpenInference’sllm.tools.{i}.tool.json_schema, at most 512 names. A tool step’s requesting llm step is its parent span when that is a model call, else the latest model call before it under the same agent (seeagentbelow); another agent’s call, whose tools may differ, is never used, and then the catalog is unknown.agent— the step’s own agent name (precedence), else the nearest ancestor span (up to 64 levels, within the run) that names one.conversation— the step’s owngen_ai.conversation.id, verbatim; absent when it has none (or an empty one). Not inherited from a parent. Imported Claude Code and Codex model calls name the main thread’s conversation (the session) or a sub-agent’s own (<session>/<agent>for Claude Code, the sub-agent’s thread for Codex), soconversationtells a sub-agent’s calls from the main thread’s even when two sub-agents share an agent name.added,removed— aneditstep’s edits as Agent Blame reads them: the lines it wrote and the lines it replaced —new_string/old_string(and aMultiEdit’sedits, in order),new_str/old_strof the text-editor tool, a whole-file write’scontent(orfile_text,new_source), a Codex patch’s+/-lines and a Codex file change’s diff. Each line is whitespace-normalised (runs of whitespace fold to one space, ends trimmed), blank lines are dropped, and the lines are joined by newlines — so an undo that re-indents still equals what it undoes, andadded=@xagainst a laterremoved=@xfinds text written and then taken out. A patch touching several files yields all of their lines together.removedis absent for a whole-file write (what it replaced was not recorded) andaddedfor a pure deletion; both are absent on a step that is no edit.
Composing steps
| Syntax | Means |
|---|---|
A -> B |
B is the very next step after A |
A ..> B |
B comes later: any steps in between (as few as possible) |
A ..[pred]> B |
later, and every step in between satisfies pred |
A ..{n,m}> B |
later, with n to m steps in between ({,m}, {n,}, {n} too) |
A ..[pred]{n,m}> B |
both |
A (..> B){4,} |
a group that starts with a link continues the step before it — here, A then four or more later Bs |
X* X+ X? X{n} X{n,} X{n,m} |
repetition of adjacent matches; greedy |
X*? X+? X?? X{n,m}? |
the same, lazy |
A | B |
either; the left one is preferred |
( … ) |
grouping |
^A, B$ |
A is the run’s first step; B is its last |
… within 10 / within 10 steps |
the whole match covers at most 10 steps, gap steps included |
… within 5m |
the whole match’s last step starts at most 5 minutes after its first |
| has the lowest precedence, then the links, then quantifiers:
tool(a) -> tool(b) | llm is (tool(a) -> tool(b)) | llm.
/v1/trajectories/explain renders any query with its grouping
made explicit.
Gaps
The steps a ..> skips are its gap. A gap predicate restricts them:
tool(a) ..[kind=llm]> tool(b) says only model calls may come between. Gap
predicates test and never capture. Gap steps are part of the
match’s range (first_step..last_step, and within) but are not listed in
its spans.
What cannot be written
A pattern that can match zero steps — tool?, step*, ^ — would match
every run and is refused. So is a repetition of something that can match zero
steps ((tool?)+): it is ambiguous and never useful.
Captures
@name captures a value the first time it is met and requires the same value
wherever it appears again:
edit[file=@f] ..> edit[file=@f] the same file edited twice
tool(@t)[error] -> tool(@t) a failing tool called again immediately
agent(@a) ..> agent(@b)[name!=@a] ..> agent(@a) A, then another agent, then A again
A capture is bound by field=@name in a bracket’s top-level and chain,
or by a lone @name in an atom’s names (tool(@t)). Everywhere else — under
or or not, in a gap, with !=, ~ or an ordering, or as one alternative
among several names — it is only compared, and a comparison with a capture
not yet bound is false (so its != is true). A query that references a
capture nothing can bind is refused, naming the column.
A capture on a list field binds each value in turn: edit[file=@f] on a patch
naming two files tries both. Several such bindings on one step multiply
(llm[offered=@a, offered=@b] over 64 tools is 4,096 pairs); one step forks
at most as many paths as the matcher can still hold (see the path
cap), keeping the first in priority order and marking
the run inexact. A scaled capture compares numbers:
llm[context=@c] ..> llm[context>=2*@c].
Captures are reported with each match ("captures": {"f": "src/app.py"}).
They are producer data. A reported value is cut to its first 1 KiB, with
captures_truncated: true on the match; the comparison itself used the whole
value (up to the 64 KiB text cap).
How a match is chosen
TQL uses leftmost-first semantics, the rule Perl, PCRE and RE2 share:
- The match that starts at the earliest step wins.
- Among matches from that step, the one a backtracking matcher would find first: alternatives left to right, quantifiers taking as many repetitions as they can (as few, when lazy), and gaps skipping as few steps as they can.
So tool(read)+ -> edit over read read read edit reports all three reads,
while tool(read)+? ..> edit reports one read and the edit.
After a match, the search resumes at the step after it, so the occurrences
counted in a run (count) never overlap.
Under the hood the pattern compiles to an NFA run by a Pike VM — every live path through the pattern advances in lock step, one step at a time, in priority order, and two paths that reach the same point with the same captures keep only the higher-priority one. That is what keeps the cost at steps × live paths rather than exponential, and why the answer equals the backtracking one exactly. The test suite checks that: thousands of random patterns over random runs, matched both ways, must agree on every match, every listed step and every capture.
The regex subset
~ uses a small, std-only regex engine that runs in linear time (a Thompson
NFA simulated breadth-first), so no pattern can make it backtrack
exponentially.
| Syntax | Meaning |
|---|---|
abc |
literal characters |
. |
any character except a newline |
[abc] [a-z] [^…] |
character class, ranges, negation |
\d \w \s / \D \W \S |
digit, word character, whitespace / complements (also inside classes) |
* + ? {n} {n,} {n,m} |
repetition (counts up to 256); a trailing ? is accepted |
a|b |
alternation |
( … ) (?: … ) |
grouping (no capture groups) |
^ $ |
start / end of a line |
\A \z |
start / end of the whole text |
\b \B |
word boundary / not a word boundary |
(?i) |
at the very start: case-insensitive (ASCII and simple Unicode folding); /…/i is the same |
\n \t \r \f \v \xHH \u{H…} |
control characters and code points |
\ + punctuation |
that character literally |
Refused, with the column: backreferences, lookaround, (?…) groups other
than (?:, inline flags other than a leading (?i), POSIX classes
([:alpha:]), and Unicode classes (\p{…}). A regex is at most 1024
characters and 4096 compiled instructions, and one query’s regexes at most
8,192 instructions in all — the regex that crosses it is refused at its
column. Compiling is bounded too: parts that can only match nothing
(a{0}, or a group of them repeated) are dropped before repetitions are
expanded, and the compiler visits at most 16,384 pattern nodes, so neither a
search nor explain can be made to spin on a pattern like
((((a{0}){256}){256}){256}){256} — which compiles to “matches anywhere”,
as it should. Matching costs up to text length × instructions; that work is
charged to the query deadline as it happens, so a heavy regex over a long
text stops the scan at the deadline ("stopped": "deadline") rather than
running past it.
Grammar
query = pattern [ "within" bound ] ;
bound = INTEGER [ "steps" | "step" ] | DURATION ;
pattern = sequence { "|" sequence } ;
sequence = [ "^" ] term { link term | continuation } [ "$" ] ;
link = "->" | gap ;
gap = ".." [ "[" predicate "]" ] [ range ] ">" ;
continuation= "(" cont_seq { "|" cont_seq } ")" [ quantifier ] ;
cont_seq = link term { link term } [ "$" ] ;
term = primary [ quantifier ] ;
quantifier = ( "*" | "+" | "?" | range ) [ "?" ] ;
range = "{" INTEGER "}" | "{" INTEGER "," [ INTEGER ] "}"
| "{" "," INTEGER "}" ; (* ",m" form: gaps only *)
primary = atom { "[" predicate "]" } | "(" pattern ")" ;
atom = kind [ "(" name { "|" name } ")" ] ;
kind = "step" | "tool" | "llm" | "agent" | "edit" | "test" ;
name = WORD | STRING | REGEX | "~" STRING | CAPTURE | "*" ;
predicate = conjunction { "or" conjunction } ;
conjunction = negation { ( "and" | "," ) negation } ;
negation = "not" negation | "(" predicate ")" | condition ;
condition = field [ operator value ] ;
field = WORD [ STRING ] ; (* attr.KEY, args.a.b, attr."x y" *)
operator = "=" | "==" | "!=" | "~" | "!~" | ">" | ">=" | "<" | "<=" ;
value = STRING | REGEX | WORD | NUMBER | DURATION | "true" | "false"
| CAPTURE | NUMBER "*" CAPTURE | CAPTURE "*" NUMBER ;
Tokens:
| Token | Form |
|---|---|
WORD |
a letter or _, then letters, digits, _, :, /, - (not before >) and . (not before .) |
STRING |
"…" or '…' with the escapes above |
REGEX |
/…/ (a \/ inside is a /), optionally followed by i |
NUMBER |
digits with an optional fraction and leading - |
DURATION |
a non-negative number immediately followed by ns, us, ms, s, m, h or d |
CAPTURE |
@ then a name ([A-Za-z_][A-Za-z0-9_]*) |
Keywords (and, or, not, within, true, false) and step kinds are
case-insensitive; fields are lower-case. Whitespace is free. Counts in
quantifiers and gap bounds are at most 100; a query is at most 4096
characters, 256 atoms, 16 captures, 64 levels of nesting, 20,000 compiled
instructions, and 8,192 regex instructions.
Every refusal names its column (1-based, in characters) and, for an unknown step kind or field, the likely intended one:
curl -sG localhost:8080/v1/trajectories/search \
--data-urlencode 'q=tool(bash)[stauts=error]'
{"column":12,"error":"column 12: unknown field `stauts`; did you mean `status`? Fields: kind, name, span, service, agent, model, tool, status, error, cost, duration, tokens, input_tokens, output_tokens, cache_read, cache_write, context, index, args, result, file, digest, fingerprint, signature, calls, offered, in_catalog, unknown_calls, edit, test, conversation, added, removed, plus args.<path>, result.<path> and attr.<key>"}
Worked examples
1. Every failing tool call.
tool[error] — one occurrence per failing tool step.
2. One tool failing. tool(bash)[error] — names ignore ASCII case, so this
matches Claude Code’s Bash.
3. Output containing a phrase. tool[result~"(?i)permission denied"].
4. A slow step. step[duration>30s].
5. An expensive model call. llm[cost>0.5] — metered cost, or priced from
the server’s rates.
6. A model family over a budget. llm(/^gpt-4o/)[tokens>20000].
7. A dangerous command, by argument. tool(bash)[args.command~"^rm -rf"].
8. Any raw attribute. step[attr.http.status_code>=500].
9. Tests failed, later passed. test[error] ..> test[not error].
10. Test tampering, your own way.
test[error] ..> edit[file~"(^|/)tests?/"] ..> test[not error] — the library’s
test_tampering adds a guard: no passing test between the failure and the
edit.
11. The same file edited twice. edit[file=@f] ..> edit[file=@f] —
captures.f names the file.
12. Many edits of one file. edit[file=@f] (..> edit[file=@f]){4,} within 10 —
five edits of one file inside ten steps. Often just a feature being written;
the library’s edit_thrash asks for edits that undo each other instead.
12a. An edit undone, then redone.
edit[file=@f, added=@x] ..> edit[file=@f, removed=@x] ..> edit[file=@f, added=@x]
— text written, taken out, written again (captures.x is the text).
13. A retry storm, back to back. tool(@t)[error] (-> tool(@t)[error]){2,}
— the same tool failing three or more times in a row.
14. A blind retry. step[error, digest=@d] (..[kind=llm]> step[digest=@d]){1,}
— a failed request sent again byte-for-byte, with only model calls between.
15. A tool that was never offered. tool[in_catalog=false] — written with
=false on purpose: [not in_catalog] would also match steps whose catalog
is unknown.
16. The model asked for a tool outside its catalog. llm[unknown_calls].
17. Read, then immediately edit, the same file.
tool(read)[file=@f] -> edit[file=@f].
18. A handoff and back. agent(planner) ..> agent(coder) ..> agent(planner).
19. Ping-pong between any two agents.
agent(@a) (..[kind!=agent]> agent(@b)[name!=@a] ..[kind!=agent]> agent(@a)){3,}
— A→B→A three times, no third agent in between.
20. The run ended on an error. step[error]$.
21. The run started with a huge context. ^llm[context>50000].
22. Context doubling. llm[context>=1000, context=@c] ..> llm[context>=2*@c] within 6
— the second call at most five steps after the first. In a run with
sub-agents, add conversation=@v to both atoms to compare a conversation’s
calls only with each other (the library’s context_blowup does, falling back
to the agent).
23. Only model calls between two searches.
tool(search) ..[kind=llm]> tool(search).
24. A rollback soon after a deploy. tool(deploy) ..> tool(rollback) within 10m.
25. An exploration spree. tool(grep|glob|read){5,} — five or more
searches and reads in a row.
26. Gave up. tool(@t)[error, fingerprint=@c] (..[not (name=@t and not error)]> tool[error, fingerprint=@c]){2,} ..[not (name=@t and not error)]> llm$
— the same call (same tool, same input: its fingerprint) failing three
times, never a success of that tool, and the run ends on a model answer.
(This is the library’s give_up_loop.) Bind name instead of fingerprint
to count any three failures of one tool.
27. Finished without testing. edit ..[not test]> llm$.
28. Greedy against lazy. Over read read read edit,
tool(read)+ -> edit lists all three reads; tool(read)+? ..> edit lists the
first read and the edit (the other two are gap steps).
Searching
GET /v1/trajectories/search?q=&since=&until=&agent=&outcome=&limit=&run=&cursor=&tenant=
| Parameter | Default | Means |
|---|---|---|
q |
required | the TQL query |
since, until |
the last 30 days | the window, in Unix nanoseconds (since_ns / until_ns also accepted): runs with a span starting in it |
agent |
any | only runs of this agent: the run’s first step’s agent name, else its earliest span’s service |
outcome |
any |
success, failure, abandoned, unknown — how the run ended, by the session outcome rule |
limit |
50 | matching runs returned, at most 500 |
run |
— | search this one run and list every occurrence in it; the window and agent do not apply. In the operator view a run id two tenants share is read as their union, as a tape is — name the tenant |
cursor |
— | continue an earlier answer: its next_cursor, as given. It carries that answer’s window, so leave since and until out (or repeat them unchanged — a different one is a 400); keep q and the filters. Not with run |
tenant |
— | for an unbound credential: one tenant (empty is the default tenant); omitted is the operator view across tenants, each run read within its own tenant |
The scan reads the window’s most recent runs first: it enumerates the runs
with activity in the window (keeping the 5,000 most recent), then reads each
run whole — every step, not just those inside the window — and matches it,
until limit runs have matched. The response:
| Field | Means |
|---|---|
query |
the query as read (canonical form) |
matches[] |
matching runs, newest first: run, tenant, agent, outcome, the first occurrence’s spans (trace_id, span_id, step_index, kind, name, at most 100), captures (each cut to 1 KiB) and captures_truncated, first_step, last_step, count (non-overlapping occurrences, at most 1,000), steps, last_activity_ns, run_truncated, inexact; with run=, also occurrences[] (at most 100) |
runs_scanned |
candidate runs examined, newest first — including those the agent filter left out |
runs_in_window |
candidate runs (at most 5,000); with a cursor, those after it |
runs_inexact |
scanned runs not listed whose matching hit a matcher cap: each may hold an occurrence the caps dropped, so their absence is not proof. A listed run that hit a cap says so itself (inexact) |
stopped |
"limit" or "deadline" when the scan stopped before the last candidate, else null |
window_capped |
the window held more than 5,000 runs; only the most recent were considered |
truncated |
there may be matches this answer does not list (any of the above, or runs_inexact above 0) |
next_cursor |
when runs went unchecked (the scan stopped, or the window was capped): pass it back as cursor to continue with exactly those runs. It holds the window — an open end fixed at the latest span the first answer saw, so later activity cannot reorder the runs — and the place in the scan order of the last run decided. Runs active at the same instant are ordered by a digest of their tenant and id, so a page boundary falls between two of them rather than on their shared timestamp: following the cursors lists every run of the window once. It is digits, hex and dots — no run id or other producer text. A run that gains activity inside the window between two pages can be missed by the later one |
spans_skipped |
spans in the window left out because their run id does not read back to them: a session id of the trace:<id> form naming another trace (it would be read as that trace’s run), or naming none |
runs_skipped |
candidate runs left out because their id was refused when read |
One compute budget (--query-deadline-ms) covers the request.
Enumerating the window’s runs refuses with 400 when it runs out — narrow the
window. Scanning stops where the budget ran out and answers what it found with
"stopped": "deadline", "truncated": true and a next_cursor to resume
from: the runs it did not reach come after every run it did. The budget
holds inside a run too: the matcher charges its work — paths advanced,
captures forked, regex instructions stepped over text — to a meter that
checks the deadline every fraction of a millisecond of work, so neither a
heavy predicate nor a capture-heavy pattern can run past it. Reading a run is
held to it as well — its step facts, agent ancestry and tool catalogs, a
huge catalog included — and the deadline is consulted once more before a run
counts as checked, so a run read past the budget is reported as not reached
rather than as done. A long match’s steps are copied out once, when it is
final, not each time a greedy pattern meets a longer provisional match.
Explain
GET /v1/trajectories/explain?q= parses and compiles a query without touching
the store — for a query box that validates as you type:
curl -sG localhost:8080/v1/trajectories/explain \
--data-urlencode 'q=edit[file=@f] (..> edit[file=@f]){4,} within 10'
{
"atoms": 2,
"captures": [
"f"
],
"fields": [
"file"
],
"instructions": 24,
"max_steps": null,
"min_steps": 5,
"query": "edit[file=@f] (..> edit[file=@f]){4,} within 10",
"within": {
"steps": 10
}
}
The detector library
GET /v1/detectors lists them. Each is an ordinary TQL query — paste it into
a search to see the same matches, or adapt its threshold.
| Detector | Severity | Finds | Query |
|---|---|---|---|
test_tampering |
high | A test run fails, a test file is edited before any test passes, and a later test run passes | test[error] ..[not (test and not error)]> edit[file~"<test file>"] ..> test[not error] |
hallucinated_tool |
high | A tool step outside its requesting model call’s catalog, or a model call asking for a tool outside its own — only where the catalog was recorded | tool[in_catalog=false] | llm[unknown_calls] |
give_up_loop |
medium | The same tool call (same tool and input) fails ≥ 3 times, that tool never succeeds in between or after, and the run ends on a model answer | see example 26 |
edit_thrash |
medium; low when the run succeeded | Within 30 steps, one file’s edits undo and redo each other (text written, removed, written again; or the file rewritten whole back to an earlier version), or the file is edited three times with a failing test after each and no passing one between | see below |
ping_pong |
medium | A→B→A agent handoffs ≥ 3 times, no other agent in between | see example 19 |
context_blowup |
medium | A model call’s context (≥ 1000 tokens) at least doubled by a call of the same conversation ≤ 5 steps later | see below |
retry_without_change |
low | A failed request repeated byte-for-byte, only model calls between | see example 14 |
tool_error_streak |
low | ≥ 5 failing tool calls with only model calls between | tool[error] (..[kind=llm]> tool[error]){4,} |
unverified_finish |
low | No test run after the last edit, and the run ends on a model answer | edit ..[not test]> llm$ |
A test file is a path under a test, tests, spec or __tests__
directory, or named test_*, *_test.*, *.test.* or *.spec.*.
edit_thrash
edit[file=@f, added=@x] ..> edit[file=@f, removed=@x] ..> edit[file=@f, added=@x]
| edit[file=@f, added=@x, not removed] ..> edit[file=@f, added!=@x] ..> edit[file=@f, added=@x, not removed]
| edit[file=@f] ..[not (test and not error)]> test[error]
(..[not (test and not error)]> edit[file=@f] ..[not (test and not error)]> test[error]){2,}
within 30
Thrash is edits that go nowhere, not many edits: a coding agent writing a feature edits one file dozens of times and succeeds. So the detector asks for one of three shapes in one file, within 30 steps:
- Undo and redo — text an edit wrote (
added) is taken out by a later edit (removed) and written again. That covers A→B, B→A, A→B, and two attempts both reverted to the same original (A→B, B→A, A→C, C→A). A single revert — a change of mind, a debug print taken out — is not thrash. - Back to an earlier version — the file is written whole with some content, changed, then written whole with that same content again.
- Edits that keep failing — the file is edited, a test run fails, three times over, with no passing test run anywhere between.
It is medium, and low in a run that succeeded: the findings report gives
each example its own severity, and a detector’s row is low only when every
run it matched succeeded.
context_blowup
llm[context>=1000, context=@c, conversation=@v] ..> llm[context>=2*@c, conversation=@v]
| llm[context>=1000, context=@c, not conversation, agent=@a] ..> llm[context>=2*@c, not conversation, agent=@a]
| llm[context>=1000, context=@c, not conversation, not agent] ..> llm[context>=2*@c, not conversation, not agent]
within 6
Two calls are compared only when they carried one context: both name the
same conversation; or neither names one and both run
under the same agent; or neither names either. A coding
agent’s sub-agent starts a small context of its own, and the main thread’s
next call after it — or the sub-agent’s first call after a large main-thread
call — is not a blow-up. Steps between the two calls count toward the six
whatever conversation they belong to, so interleaved sub-agents make the
window shorter, never longer.
Findings
GET /v1/detectors/findings?since=&until=&agent=&detector=&cursor=&tenant=
runs the library (or the detectors named — detector=a,b or repeated) over
the window’s 2,000 most recent runs, and reports per detector the runs and
occurrences it matched, how those runs ended, their success_rate, its
severity over those runs, up to five example runs (newest first, each with
its own severity given how it ended), and when it last fired — next to the baseline:
how every checked run ended. runs_checked counts those runs (the
baseline’s total, and the denominator for a detector’s runs_matched);
runs_scanned counts every candidate examined, including runs the agent
filter left out. A detector whose matched runs succeed far less often than
the baseline is a behaviour costing you outcomes. Detectors with findings
come first, by severity, then by runs matched. A detector’s runs_inexact
counts checked runs it did not match where its matcher hit a cap — its
runs_matched is then a lower bound, and the report’s truncated is set.
cursor and next_cursor page through a window as for a search.
curl -sG localhost:8080/v1/detectors/findings \
--data-urlencode 'since=1772323199999999999' \
--data-urlencode 'detector=test_tampering,edit_thrash'
Abridged — each example run is cut to run, outcome, captures and severity, and each detector’s query and summary (the same as in /v1/detectors) are left out:
{
"baseline": {
"abandoned": 0,
"failure": 1,
"success": 2,
"unknown": 0
},
"baseline_success_rate": 0.6666666666666666,
"detectors": [
{
"examples": [
{
"run": "tampered",
"outcome": "success",
"captures": {},
"severity": "high"
}
],
"last_seen_ns": 1772323205000000000,
"name": "test_tampering",
"occurrences": 1,
"outcomes": {
"abandoned": 0,
"failure": 0,
"success": 1,
"unknown": 0
},
"runs_inexact": 0,
"runs_matched": 1,
"severity": "high",
"success_rate": 1.0,
"title": "Edited the tests to make them pass"
},
{
"examples": [
{
"run": "thrash",
"outcome": "failure",
"captures": {
"f": "src/app.py",
"x": "b"
},
"severity": "medium"
}
],
"last_seen_ns": 1772323410000000000,
"name": "edit_thrash",
"occurrences": 1,
"outcomes": {
"abandoned": 0,
"failure": 1,
"success": 0,
"unknown": 0
},
"runs_inexact": 0,
"runs_matched": 1,
"severity": "medium",
"success_rate": 0.0,
"title": "Edits of one file going back and forth"
}
],
"next_cursor": null,
"runs_checked": 3,
"runs_in_window": 3,
"runs_scanned": 3,
"runs_skipped": 0,
"since_ns": 1772323199999999999,
"spans_skipped": 0,
"stopped": null,
"truncated": false,
"until_ns": null,
"window_capped": false
}
success_rate counts runs whose outcome is known: successes over successes,
failures and abandonments. A run still active, or longer than the span budget,
is unknown and is in neither side of the rate.
From an agent
The MCP tool find_behavior takes a query or a
detector (or neither, to list the library), plus since, until, agent,
outcome, limit and cursor, and returns each match’s run, outcome and matched
steps — run ids, names and captured values inside the untrusted-telemetry
block, a parse error with a caret under the column.
Limits, stated plainly
- Heuristics are heuristics.
edit,test,fileand the test-file pattern read tool names, argument keys and command text. A test runner invoked through a wrapper script (./ci.sh) is not atest; a tool that writes files under an unknown name and unknown keys is not anedit. The lists are above; match the raw fields (tool(...),args.<path>) when yours differ. - Failure means span status. A test run counts as failed only when its
span says
error. A runner that exits 0 on failure, or instrumentation that never marks errors, makestest[error]match nothing. - Catalogs are optional.
in_catalog,unknown_callsandhallucinated_toolsay nothing when the model call did not record its tool definitions — most instrumentation does not by default — when its catalog was cut at 512 names, or when the tool step’s requesting call is neither its parent nor an earlier call under the same agent. - Text is capped. Every text a predicate reads is matched on its first 64 KiB. A value ingest offloaded to the payload store is its reference and inline preview (256 characters), not its full text — the same rule as content search. Reported captures are cut to 1 KiB.
- Runs are read whole, windows pick them. The window selects runs by span
start time; a selected run is then matched over all its steps, up to 20,000
spans (past that, its beginning is matched and
run_truncatedis set; its outcome is thenunknown). - The agent filter is applied after recency. With
agent=, the scan still considers the window’s 5,000 most recent runs of any agent and skips the others; in a busy multi-agent store, narrow the window to reach older runs of one agent. - Partial answers say so. A scan stopped by the deadline or the limit sets
truncatedandstopped, and a scanned run the matcher caps cut without a match is counted inruns_inexact(and setstruncated), so an empty answer is never passed off as proof. Enumerating the window is all-or-nothing. - Path cap. The matcher keeps at most 4,096 live paths per step, lets one
step fork no more paths than that list still has room for, and remembers at
most 65,536 states per step for deduplication. A capture-heavy pattern over a
long run with thousands of distinct values (one path per file waiting for
its pair), or several captures over one long list (
offered=@a, offered=@b, …), can exceed them; the lowest-priority alternatives are dropped and the run is markedinexact— or, when nothing matched, counted in the answer’sruns_inexact(a detector’s, in findings). Memory and work stay bounded either way. - Run ids that do not read back are left out. A run whose session id
has the
trace:<id>form is read as trace<id>; a span carrying such a session id in a different trace (ortrace:alone) is not part of the run its id names, so it is skipped and counted inspans_skippedrather than attributed to another run or failing the request. - No index. A search reads every span in its window once, to find the
window’s runs, and then each candidate run in full — work proportional to
the window’s spans plus the scanned runs’ spans, under the query deadline.
Narrow windows and a
limitkeep it fast; it is not built to scan millions of runs in one request. - Names ignore ASCII case;
=does not.tool(bash)matchesBash,[tool=bash]does not. - Not expressible: “never” over the whole run (negative lookbehind, “a file edited without ever being read before”), arithmetic beyond a scaled capture, and comparisons between two different fields of one step.