Skip to content

Add Trajectory.called_before?/4 for relative tool-call ordering assertions#565

Merged
brainlid merged 1 commit into
mainfrom
issue-564-called-before
Jun 9, 2026
Merged

Add Trajectory.called_before?/4 for relative tool-call ordering assertions#565
brainlid merged 1 commit into
mainfrom
issue-564-called-before

Conversation

@brainlid

@brainlid brainlid commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Summary

Adds LangChain.Trajectory.called_before?/4 and the companion
assert_called_before/4 / refute_called_before/4 macros for asserting
relative tool-call ordering — "tool A was called before tool B" —
independent of how many other calls happen in between.

This fills the gap between the existing modes:

  • :superset enforces containment but is order-independent.
  • :strict requires the whole sequence with exact count.

Relative ordering ("search before answer", "validate before write") is the
common middle ground for agent trajectory evals, and previously required
hand-rolling index math over tool_calls in every eval.

API

Trajectory.called_before?(trajectory, "search", "answer")
#=> true

assert_called_before trajectory, "search", "answer"
refute_called_before trajectory, "write_file", "read_file"

Accepts a Trajectory, an LLMChain, or a bare list of tool-call maps.

Semantics decisions

These resolve the open questions from the issue:

  1. Which occurrences — uses the recommended "any A precedes any B"
    reading: min(index of A) < max(index of B). This tolerates interleaving
    (A…B…A…B) while still catching a pure B-before-A ordering.
  2. Missing tools — by default a missing tool returns false (so
    refute_called_before passes vacuously). Passing require_both: true
    raises an ArgumentError naming the absent tool, so a missing tool is
    detected rather than silently collapsing to false. This mirrors the
    module's existing convention of raising ArgumentError on misuse in
    matches?/3, and applies symmetrically to both assert_ and refute_.
  3. Same turn — ordered by flat-list position in tool_calls, consistent
    with the existing helpers.
  4. Args matching — out of scope for v1; name-only, as suggested.

Tests

25 new tests covering the predicate (ordering, missing tools, interleaving,
repeated calls, LLMChain/list inputs, :require_both raising) and both
assertion macros. Full mix precommit is green (1799 tests + 31 doctests, 0
failures).

Closes #564

…tions

Adds a predicate for asserting relative tool-call ordering — "tool A was
called before tool B" — independent of how many other calls happen in
between. This fills the gap between :superset (containment, order-independent)
and :strict (whole sequence, exact count), a pattern that shows up constantly
in agent trajectory evals.

Semantics:
- Returns true when any A precedes any B, i.e. min(index of A) < max(index of B).
- Operates over the flat, ordered tool_calls list; same-turn calls are ordered
  by their position in that list.
- Accepts a Trajectory, an LLMChain, or a bare list of tool-call maps.
- By default a missing tool returns false; :require_both raises ArgumentError so
  a missing tool is detected rather than silently collapsing to false.

Also adds the companion assert_called_before/4 and refute_called_before/4
macros to LangChain.Trajectory.Assertions.

Closes #564

<noreply@anthropic.com>
@brainlid brainlid merged commit ca3f99e into main Jun 9, 2026
2 checks passed
@brainlid brainlid deleted the issue-564-called-before branch June 9, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Trajectory.called_before?/3 for relative tool-call ordering assertions

1 participant