[pull] main from danny-avila:main - #154
Merged
Merged
Conversation
* 🧪 chore: SDK-Side Activity-Label Prose Eval Harness
Port of LibreChat's activity-label eval harness (LibreChat #14527) into
the SDK, rendering the user prompt with the REAL buildActivityLabelPrompt
from src/prompts/activityLabel.ts instead of a hand-port, so builder
changes are measured as they will ship. corpus/checks/report are
byte-identical to the LibreChat originals for cross-repo comparability
(verified: 57 renders, 0 byte mismatches against the hand-port).
- run.ts: runner on the repo's tsconfig-paths loader convention; verbatim
captured steps get their continuity section rendered through the real
builder (empty-batch render) so production sanitation and the 3-label
cap apply exactly; framing hypotheses applied as marker-exact
substitutions that degrade to the control when markers don't match
- variants.ts: sdk-default (ACTIVITY_LABEL_PROMPT), host-shipped (pinned
fixture of LibreChat's ACTIVITY_INSTRUCTION), guard-heading/guard-full
(the host fallback's "do not restate these" framing, measured before
any port into the builder)
- npm scripts label:eval / label:rescore; results/ gitignored
First sweep (claude-haiku-4-5, 3 samples x 28 steps): the guard framing
replicates its LibreChat-side result (template 6->3, tool-echo 2->0,
len 15->9/10 vs host-shipped); sdk-default is format-tight but
attempt-oriented ("Tested network access" with the answer never becoming
the line), quantifying the quality cliff for hosts that pass no prompt.
* fix: address Codex review on the label eval harness
- requestLabel: catch transport rejections (DNS, connection reset) and
retry them like 429/500/529 instead of letting one flake reject the
pool and discard every completed record; latencyMs now spans the full
attempt sequence including backoff
- apply the production extractor's single-line normalization (collapse
whitespace) so the harness grades and chains the same label users
receive, and newlines cannot break the per-case Markdown tables
- replace the Record<string, unknown> record bag with an explicit
DryRunRecord | ErrorRecord | LabelRecord union and type the CJS
module boundaries (AGENTS.md type-safety rules)
- derive the continuity-section terminal from an empty builder render
instead of assuming 'Label:', so the harness keeps working across the
very terminal rename guard-full evaluates
- reorder imports to the documented package / type / local convention
- validate --samples and --concurrency as positive integers
- checks.cjs: normalize namespaced MCP tool names in echo matching
(double underscores no longer defeat the match; post-namespace tail
matched as a phrase, single middle segments deliberately excluded to
avoid flagging legitimate subjects like "GitHub") — one deliberate
divergence from the LibreChat copy, backport candidate
Declined: variant-specific word bounds (one shared len rule keeps
columns comparable across variants and repos; the avg-words column
already surfaces per-variant drift).
* fix: address Codex round-2 review on the label eval harness
- rescore.cjs: the chain key's delimiters were literal NUL bytes
(inherited from the LibreChat original), making Git classify the
source as binary; write them as \0 escapes — identical runtime key,
reviewable text diffs
- applyFraming: rewrite the entries heading when the prompt STARTS with
the section — a bare batch with no previous labels, intent, or
excerpts puts 'Tool calls:' first, so six synthetic steps were
silently rendering the control framing under guard variants
- requestLabel: move response.json()/response.text() inside the attempt
try so a connection reset after headers arrive retries instead of
rejecting the worker pool
- resolveSelection: a typo in --variants/--cases now fails up front with
the available names instead of silently dropping the requested column
from a billed run
* fix: address Codex round-3 review on the label eval harness
- parseArgs: reject unknown options — a mistyped --dryy must not fall
through to a fully billed default sweep
- task queue: variants innermost so adjacent queue slots cycle arms;
a variant-major queue correlates variant identity with elapsed run
time and rate-limit pressure, biasing latency/error comparisons
- results JSON: persist a sha256 corpus fingerprint; rescore.cjs refuses
a stored run whose corpus has since drifted (warns on pre-fingerprint
files instead of failing them)
- report.cjs: numeric sample ordering (lexicographic sort scrambles
sweeps of ten-plus samples) and a keyed record index instead of a
records.find per table cell (quadratic on large sweeps; AGENTS.md
iteration guidance) — both backport candidates
* fix: address Codex round-4 review on the label eval harness
- resolveSelection: reject duplicate --variants/--cases names — a repeat
schedules the same paid arm twice and merges both runs under one
aggregate row while the keyed report shows only the last
- checks.cjs COUNT_ECHO: also catch spelled-out counts ("three tool
calls"), which the instruction forbids the same as digits
- rescore.cjs: create results/ before writing latest.md so rescoring an
archived JSON by path works in a fresh checkout
- pool: shared index instead of queue.shift(), which reindexes the
remaining array per task (quadratic over a large sweep)
* fix: address Codex round-5 review on the label eval harness
- requestLabel: 401/403 throws FatalRunError through the pool — an
invalid key previously produced hundreds of identical doomed requests
and an all-error report instead of failing on the first response
- report.cjs: case-fold opener tallies; counting 'Found' and 'found'
separately made a sentence-case violation read as MORE register
diversity, the opposite of what the tally detects
* fix: address Codex round-6 review on the label eval harness
Fatal classification happens before (and independent of) the error-body
read: a dropped 401/403 body stream previously threw in response.text()
and demoted the definitive failure to a retryable transport error. 404
(unknown --model) joins the run-fatal set — like bad credentials, every
remaining task would send the same doomed request. Both verified live:
invalid key and bogus model each die on the first response.
* fix: address Codex round-7 review on the label eval harness
- AggregateRow: replace the false Record<string, string | number>
assertion with the row shape aggregate() actually returns (nested
flagCounts, pre-formatted avgWords/costUsd strings) so the reporting
boundary is genuinely type-checked
- parseArgs: value-taking options reject a missing value or a following
flag — `--model --dry` previously consumed `--dry` as the model and
silently disabled dry-run mode; a trailing `--model` omitted the field
and 400'd the whole sweep per task
* chore: align harness with the shipped #363 framing after rebase
The guard-heading/guard-full hypothesis variants shipped into the
builder in #363 and are the baseline now — pruned, per the follow-up
noted in that PR. applyFraming's current-builder markers move to the
new heading and Header: terminal so future framing hypotheses
substitute against what the builder actually emits; captured verbatim
prompts predate the change and degrade to control by design.
* fix: address Codex round-8 review on the label eval harness
- requestLabel: an empty normalized label becomes an error record —
production's extractLabel returns {} and commits no header, so
grading it as a len:0 step would count an outcome users never see
- report.cjs: opener tallies use a null-prototype object; a label
opening with 'Constructor' hit Object.prototype.constructor and
rendered a garbage top-opener ('__proto__' vanished entirely)
- dry runs seed the continuity chain (captured production labels, else
a marked placeholder) so multi-step prompts render the
Previous-headers section without a live spend
* fix: address Codex round-9 review on the label eval harness
- error diagnostics collapse to a single line before storage — a
multiline gateway body previously split its Markdown table row (the
report escapes only pipes)
- dry runs persist every rendered prompt to results/dry-latest.md; the
console preview showed only the first three, hiding the later
prompts that demonstrate the three-label continuity cap
- an empty-label 200 keeps its billed usage on the error record, and
aggregate() counts it — an all-empty variant no longer reports $0
* fix: address Codex round-10 review on the label eval harness
- per-attempt AbortSignal.timeout(60s) on the API request — a
connection an intermediary leaves open would otherwise pin its
worker forever and keep the run from ever writing its report; the
timeout surfaces as a retryable transport failure
- error records keep their latencyMs and aggregate() includes it, so
an all-empty or all-retry variant shows real wall time, not 0
- the aggregate topOpener cell escapes pipes like the per-case
renderer — a label opening with 'a|b' shifted every later column
* fix: address Codex round-11 review on the label eval harness
Retry completed transient gateway statuses (408/502/503/504) alongside
429/500/529 — a brief proxy outage must not bias an arm as a recorded
experiment error.
* 🔒 fix: Stamp Langfuse Runtime Scopes With Run Identity
Cross-tenant span-routing leak: LangChain executes non-awaited
callbacks on a process-wide background queue (`consumeCallback`, a
module-singleton p-queue with concurrency 1), so a handler callback can
run inside a DIFFERENT concurrent run's async context. The handler's
ambient-scope-first config resolution then adopted the other run's
tenant config, and `RoutingLangfuseSpanProcessor` exported the span
through the wrong tenant's processor — with two tenants running
concurrently, one tenant's spans landed in the other's Langfuse
project.
Runtime scopes and handlers now carry the identity of the run they
belong to (`runId` on `LangfuseRuntimeContext` and handler creation):
a handler only adopts an ambient scope stamped with its own run, so
same-run agent overlays keep winning while a foreign concurrent run's
scope falls back to the handler's own configuration and seed. Unstamped
scopes on unstamped handlers keep the existing scope-first semantics
for host-managed handlers.
Stamped call sites: stream, title, and activity-label paths in `Run`
(each with its distinct per-path identity), and the model-invoke
handler, agent overlay, and provider-fallback scopes in `Graph`.
Detached subagent graphs stamp their own child run id, so their spans
keep inheriting the parent scope only through the child's own
re-stamped scope.
Both previously-failing routing integration tests now pass (CI never
runs them — validate.yml ignores `integration.test.ts` — so the leak
was invisible); unit regressions pinning foreign-scope rejection and
same-run overlay adoption live in the CI-covered callbacks spec.
* 🛡️ fix: Address Codex Review — Sub-Scoped Agent Stamps, Policy Reset, Graph Fallback
P1 (unstamped graphs): directly-constructed graphs without a public
runId now stamp scopes/handlers with a generated `graph-<nanoid>`
identity, so concurrent host-managed graphs keep foreign-scope
protection.
P1 (tool-output policy): a foreign scope's redaction policy no longer
leaks through — the replacement scope carries the handler's own
resolved `toolOutputTracing`, so a permissive foreign run can't cause
this run's redacted tool output to export.
P1 (parallel agent overlays): agent overlay scopes are stamped as
`<run>#<agentId>` sub-scopes, and the handler verifies the callback's
own agent (via inherited `langgraph_node` metadata) against the
sub-scope before adopting it — a sibling fan-out agent's overlay is
rejected in favor of the run-level config, while the callback's own
agent overlay is still adopted. `#` is reserved in run ids.
Regressions: sibling-vs-own agent overlay adoption and foreign-scope
tool-output policy in the CI-covered callbacks spec; generated graph
stamps in the routing spec.
* 🧹 fix: Address Codex Round 2 — Replace-Mode Foreign Rejection, Field-Based Agent Scopes
P1 (merge inheritance on rejection): rejecting a foreign scope now uses
replace semantics — `replaceLangfuseRuntimeContext` plus OTEL key
deletion via `withLangfuseRuntimeScope(..., { replace: true })` — so a
run relying on env credentials or without a deterministic seed no
longer re-adopts the foreign run's explicit destination or trace seed
through `undefined`-field merge inheritance.
P2 (reserved separator): the per-agent overlay identity is now a
separate `agentId` scope field instead of a `#`-encoded stamp — no
parsing, no reserved characters in host-supplied run ids.
Regression: env-credential run inside a foreign explicit scope exports
through env-credential params with no foreign trace id.
* 🕶️ fix: Address Codex Round 3 — Tool Scope Agent Stamps, Propagation Clearing
P1 (tool scopes): `CustomToolNode.invoke` now stamps its runtime scope
with `executingAgentId` (run identity inherits from the ambient stream
scope), so a concurrent sibling agent's queued tool callback can no
longer adopt this agent's overlay during parallel tool supersteps.
P1 (propagated identity): rejecting a foreign scope now also clears
`@langfuse/tracing`'s propagated trace attributes (userId, sessionId,
tags, metadata, …) by deleting `LangfuseOtelContextKeys` from the
context and re-propagating this handler's own identity — and drops the
foreign active span first, both so detached runs root their own trace
and so `propagateAttributes` cannot stamp this run's identity onto the
foreign run's still-recording span. Adds `@langfuse/core` (already the
shared core of the installed @langfuse packages) as a direct dependency
for the context keys.
Regressions: sibling-agent tool callback rejection, and foreign
propagated identity replaced with the handler's own (session cleared
where the handler has none).
* 🔤 fix: Address Codex Round 4 — Verbatim Agent-Id Match Before Prefix Decode
An agent id that itself begins with an internal node prefix (an agent
literally named `agent=research`) is carried VERBATIM by its outer
workflow node; decoding the prefix first made the handler reject its
own agent scope and fall back to run-level routing. The callback-node
comparison now checks an exact agent-id match before decoding the
inner `agent=` / `tools=` / `summarize=` subgraph prefixes.
* 🎛️ fix: Address Codex Round 5 — Aggregate Policy, Explicit Agent Identity, Opaque Derived Ids
P1: handlers now carry their run's RESOLVED tool-output policy — for
multi-agent streams the conservative aggregate across agents — and
restore it on foreign-scope rejection instead of re-deriving from the
primary agent's config, which could be more permissive.
P2s:
- Explicit `agentId` invoke metadata (already stamped by the graph's
model path; ToolNode now stamps it too) takes precedence over
node-name parsing, disambiguating agents whose ids literally embed
node prefixes from siblings' inner nodes.
- Foreign-scope rejection re-propagates the handler's configured trace
name along with its identity, so rejected root callbacks keep their
intended trace names.
- Derived scope ids (title, activity-label) carry an opaque
per-execution nanoid component so a public run id can never collide
into another run's scope identity.
* 🆔 fix: Address Codex Round 6 — Per-Execution Stream Stamps, Summarizer Agent Key
P1: public run ids are unrestricted and may repeat across concurrently
executing runs (retries, duplicate submissions, tenant-local message
ids), so equal stamps let those runs adopt each other's scopes. The
graph's scope stamp now always carries an opaque per-instance nanoid
component (`<runId>:<nanoid>`), and the stream handler/scope share the
graph's stamp — one graph instance, one execution identity. Public ids
still drive deterministic trace seeds unchanged.
P2: the explicit agent-identity lookup also accepts the summarization
node's `agent_id` metadata key, so summarizer callbacks disambiguate
via identity rather than the ambiguous `summarize=<id>` node name.
* 📇 fix: Address Codex Round 7 — Prefer Summarizer-Owned Agent Identity
`agent_id` is stamped by the summarization node at the invoke closest
to its own work; `agentId` can be inherited from an enclosing scope and
name the wrong agent on a summarization callback carrying both keys —
check `agent_id` first.
* 🪪 fix: Address Codex Round 8 — Canonical Agent-Identity Key
Key priority alone cannot arbitrate between `agentId` and `agent_id`:
each is only authoritative for the component that stamped it closest,
and an inherited key of either casing can name the wrong agent. The
summarization node now overwrites the canonical `agentId` (keeping
`agent_id` for compatibility) like the graph model path and ToolNode
already do, so spread order guarantees the closest stamper wins;
`agent_id` remains a fallback for third-party graphs.
* 📏 fix: Address Codex Round 9 — Stamp Agent Identity Outside Metadata Filtering
`createLangfuseTraceMetadata` caps values at 200 chars for trace-
metadata hygiene, silently dropping longer agent ids from the model
invoke metadata and losing the explicit identity stamp scope trust
relies on. The canonical `agentId` is now stamped verbatim after the
filtered spread, matching ToolNode and the summarizer which already
stamp raw ids.
* 📦 chore: Bump @langchain/langgraph to 1.4.8 Picks up the pregel sync-durability barrier fix (langchain-ai/langgraphjs#2604): checkpoint put()/putWrites() from a completed superstep now settle before the next superstep dispatches when durability is 'sync'. Our 'exit' default is unchanged; consumers passing durability: 'sync' through the run config now get true per-superstep persistence barriers for HITL interrupt/resume. Also inherits 1.4.7's removal of the vestigial zod-to-json-schema peer (last source of Zod v4 install-time peer conflicts). Transitive bumps: @langchain/langgraph-sdk 1.9.25 -> 1.9.28, p-queue 9.3.0 -> 9.3.3. Lockfile-only: the existing ^1.4.6 range already covers 1.4.8. * 📌 chore: Pin @langchain/langgraph to exact 1.4.8 Switch the range ^1.4.6 to an exact pin, matching the exact-pin convention used for the other @langchain provider deps (openai, anthropic, google-*).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )