Skip to content

observability: let the liquidity board's log panels read the bot's own lines [RAI-3064] - #1697

Open
JuaniRios wants to merge 1 commit into
rai-2998-event-linesfrom
rai-2998-board-logs
Open

JuaniRios wants to merge 1 commit into
rai-2998-event-linesfrom
rai-2998-board-logs

Conversation

@JuaniRios

@JuaniRios JuaniRios commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Part of RAI-3064.

The board's log panels now follow Source too. On bot they read the bot's own liq_trade, liq_transfer and liq_event lines and its own log lines, instead of the exporter's logs. Item 6b of the RAI-2998 stack, on top of #1696.

Live effect: none until Source is bot and the bot logs JSON to liquidity-bot. The default (exporter) shows the same rows as today. After a failed refresh, the detail dialog can show a new out-of-date notice on either source · Risk: low to medium (read-only board, but every refresh on the default source also runs the hidden bot queries, see Decisions) · Ships: on merge, through the observability workflow

Why

6a left Trades, Rebalances, the detail dialog and the Logs tab on the exporter's logs, and #1654's USD columns and event timeline were waiting for real bot lines. #1696 writes those lines, so the board can read them.

Decisions

  • Two queries per panel, one shown. A Cloud Logging query can't pick its log by a variable. So each log panel has exporter-<name> and bot-<name>, and its first transformation is filterByRefId on ${source:text}-<name>. Grafana fills variables into transformation options, and the refId match is exact. check_log_sources in the generator fails on any log panel that doesn't follow this. Allowed exceptions: the dialog's bot-only bot-events, and Orders, which stays on the exporter's log because the bot writes no order lines.
  • Cost of the hidden query. It still runs. On the default source each refresh also scans the bot's log for Trades, Rebalances, the dialog's events, and the Logs tab: about 10 s each over 30 days while the log is empty (measured with gcloud logging read on prod), plus one more list call per panel against the project's read quota. The plugin shows a throttled call as missing rows, not as an error. I found no clean way to skip a query by $source alone: Grafana's Mixed datasource skips a query whose datasource uid resolves to empty, but no format of $source gives cloudlog for one source and empty for the other. A second variable would do it, at the price of two switches. It's in the docs. lmk if you'd rather have that, or a generator flag flipped at Stage 3.
  • Field mapping. The bot's lines use the exporter's field names, so the columns are the same. The bot source adds a USD column (from usd) to Trades and Rebalances. A bot line's body is its message, not its JSON, so the Rebalances Type column reads the bot's labels as JSON text (a step limited to the bot frame).
  • Trades "Last updated" on bot is when the bot logged the terminal status. For an onchain fill it caught up on, that's later than the fill. The dialog shows occurred_at. Documented in the panel and the docs.
  • Detail dialog (log-lines.js, new, prepended like the other helpers):
    • Reads the picked source's frames. Drops a line whose event_id it already has.
    • Drops a line from another project (resource.labels.project_id, which both sources carry). This closes the 6a gap where a cancelled query after an Environment switch released the hold with the other project's rows. The hold itself stays.
    • Bot status history is ordered by the sequence in event_id, not by time.
    • Event timeline from liq_event, filtered by parent and transfer kind, not by venue, since a venue correction changes it. Payload is parsed from its JSON string.
    • Flags a timeline that doesn't run 1..n, or ends before the row's latest status line. That covers the newest-500 window, a line not loaded yet, and a line the bot never wrote.
    • Only an error of a query the shown source reads blocks the dialog. A failed hidden query, or a failed bot-events, doesn't. The timeline says when the event query failed. The event check reads the normalized error list, a single error too.
    • Grafana's Mixed datasource keeps only the last failing query's errors, so a failed table query can hide behind a failed bot-events. When only errors that do not block are listed, a red notice says the row and its CLI commands can be out of date. Refresh before you run a command. log-lines.js
  • Logs tab on bot uses the same level filter as the exporter (ERROR, WARN, INFO, default all), not WARN and up. The exporter shipped INFO, and a WARN-only bot query would make the INFO option empty. DEBUG and TRACE stay out either way.

Proof

  • gen-t0-liquidity.py --check passes. The new check fails as expected when I drop a pick_source, rename bot-events, or point a bot query at an exporter log.
  • vitest: 408 passed before the latest fix push. New tests cover queryState, lineEntries (labels vs JSON body, project filter, event_id dedupe), eventTimeline, timelineIncomplete, blockingErrors, and the sequence ordering in latest(). eslint and svelte-check clean, pre-commit clean.
  • I ran the generated afterRender in happy-dom for both sources: bot trade with timeline, other-project rows, exporter ignoring bot frames, bot transfer with status history, a failed hidden query, a timeline missing sequence 1, and a failed bot-events with an old frame.
  • The bot queries parse in Cloud Logging (gcloud logging read on prod, read-only).
  • Grafana and plugin behavior checked from source, not run: transformation variable interpolation (Scenes interpolates the whole config), the byRefId matcher, frame-scoped filter, convertFieldType to string (JSON.stringify for objects) and to number ("" becomes null), the plugin's frame shape and labels.
  • Review: multi-model loop (sol, Cursor Grok, composer, flash, opus), 4 passes. Fixed: the failed hidden query blocking the dialog, timeline gap detection, the generator check missing inherited datasources, infra detail in the docs, and doc gaps. Pass 4 still raised a medium (a failed bot-events refresh with an old frame was silent) and two lows, after the cap. I fixed those and an opus fix-verifier plus the tests checked them, but no fifth panel pass ran.
  • Not verified: any of this on a live Grafana with real bot lines. Staging needs log_format = "json" and T0Trade/t0.devops#792 applied first. Also not checked live: the extra latency and quota on the default source with several viewers open.

Merge conditions

Same as the stack below. This deploys on merge to master through the observability workflow, not with a bot release. That's safe before the bot logs JSON, because Source defaults to exporter. Before Stage 4 flips the default, open the board with Source = bot on staging and check every log panel has rows.

@linear-code

linear-code Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

RAI-2998

RAI-3064

@JuaniRios JuaniRios self-assigned this Oct 9, 2026
@JuaniRios
JuaniRios requested a review from findolor October 9, 2026 04:08

JuaniRios commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more


How to use the Graphite Merge Queue

Add the label add-to-gt-merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR lets the liquidity board's log panels read the bot's own liq_trade, liq_transfer and liq_event lines behind a Source switch, with exporter still the default. It adds the per-row event timeline and the query-state notice in the detail dialog.

Since the last review (0d461c6): the head was restacked onto the parent PR again. The PR's own commit is unchanged in every hand-written file; git range-diff shows only the generated board JSON differs, because it embeds the parent's new recovery-commands.js text for the Alpaca to Base converting resume (it now asks the operator to confirm at Alpaca that no USD to USDC order is still open before the resume, and limits the hand settlement to a ConversionFailed end). That text and its transfer.ts twin belong to the lower PR and are reviewed there. On acb3654, gen-t0-liquidity.py --check passes (5 boards match the generator) and the transfer-board parity and transfer Vitest suites pass (90 tests). I found no new defects.

Earlier findings: my last review had no open findings, and nothing has regressed.

Threads resolved by hand:

  • The narrow Stage 4 precondition (docs/observability.md:446, resolved by JuaniRios): addressed. The missing items are on the RAI-3000 checklist, which gates the flip.
  • The preconditions not being written down (docs/observability.md:446, resolved by JuaniRios): addressed in the same RAI-3000 comment.
  • CodeRabbit's point that an uncertain refresh releases the env-switch hold (t0-liquidity.json:2741, resolved by JuaniRios): dismissed for now, and the reason still holds. That code is unchanged on this head. It needs Grafana to drop a failed table query from its error list, the worst result is a wrong "not in the newest 500" note with no commands shown, and it goes with the masked table-query errors follow-up in RAI-3000. CodeRabbit agreed to defer it.

The other resolved threads were withdrawn by CodeRabbit or resolved by me with a reply that still holds. Nothing blocks this PR.

claude-opus-5-5 · high · 1 min

@JuaniRios
JuaniRios force-pushed the rai-2998-event-lines branch from 6421f57 to e0ddd8c Compare October 10, 2026 04:11
@JuaniRios
JuaniRios force-pushed the rai-2998-board-logs branch from acb3654 to c5f3d9c Compare October 10, 2026 04:11
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@rain-marvin approve

@coderabbitai

coderabbitai Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@rain-marvin

rain-marvin Bot commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

🔎 Verifying that my findings from acb3654 are addressed at c5f3d9c, started by @JuaniRios. The review will appear here when it's done.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR lets the liquidity board's log panels read the bot's own liq_trade, liq_transfer and liq_event lines behind a Source switch, with exporter still the default. It adds the per-row event timeline and the query-state notice in the detail dialog.

Since the last review (acb3654): this head is only a restack onto the parent PR. git range-diff shows that the PR's own commit is the same in every hand-written file. Only the generated board JSON changed, because it embeds the parent's new recovery-commands.js text: after the nonce-closing step of the Base to Alpaca fail, an operator whose withdrawal mined is now told to start the bot, run resume-usdc and not fail the transfer. The guide's resume-usdc entry also asks for an Alpaca order check before a resume at converting. That text and its transfer.ts and recovery-guide.json twins belong to the lower PR and get their review there. On c5f3d9c, gen-t0-liquidity.py --check passes (all 5 boards match the generator), and the transfer and transfer-board Vitest suites pass (91 tests). I found no new defects.

Earlier findings: my last review had no open findings, and nothing has regressed.

Threads resolved by hand:

  • The narrow Stage 4 precondition (docs/observability.md:446, resolved by JuaniRios): addressed. The missing items are on the RAI-3000 checklist, which gates the flip.
  • The Stage 4 preconditions not being written down (docs/observability.md:446, resolved by JuaniRios): addressed in the same RAI-3000 comment.
  • CodeRabbit's point that an uncertain refresh releases the env-switch hold (t0-liquidity.json, resolved by JuaniRios): dismissed for now, and the reason still holds. That code has not changed on this head. The case needs Grafana to leave a failed table query out of its error list. The worst result is a wrong "not in the newest 500" note with no commands shown, and the fix goes with the masked table-query errors follow-up in RAI-3000. CodeRabbit agreed to defer it.

All other resolved threads were withdrawn by CodeRabbit or resolved by me with a reply that still holds. Nothing blocks this PR.

claude-opus-5-5 · high · 13 min

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 10, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

This was referenced Oct 10, 2026
@rain-marvin

rain-marvin Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

@JuaniRios 🔔 Follow-up: a day after my review

@JuaniRios, this PR has been quiet since the approvals on c5f3d9c. CodeRabbit and I both approved that head, the PR can merge without conflicts, and the CI run on it passed. The only red check is human/reviewed, because no engineer has approved yet. The stray failing backend test ${{ matrix.partition }} of 4 comes from an earlier run on the same SHA that was cancelled; it is not a real failure.

Next step: get an engineer to approve. #1696 (Graphite) needs one too, since this PR sits on top of it and is still waiting for its own human approval.

claude-opus-5-5 · high · 1 h 54 min

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant