Skip to content

observability: pin liq_ board queries to the exporter job and fix process start [RAI-3083] - #1673

Open
JuaniRios wants to merge 1 commit into
masterfrom
rai-2998-board-pin-process-start
Open

JuaniRios wants to merge 1 commit into
masterfrom
rai-2998-board-pin-process-start

Conversation

@JuaniRios

@JuaniRios JuaniRios commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Part of RAI-3083.

Pins every liq_ query on the liquidity boards to the exporter's scrape job, fixes the scan-skipped native row, and makes /health uptime count from process start. First PR of the RAI-2998 stack (bot publishes the liq_* metrics itself).

Live effect: board queries read only job="t0-liquidity-exporter" (same data as today), the scan-skipped breakdown starts showing reasons, and uptime is right after a restart · Risk: low (no money path, no config, no stored data; a revert undoes it) · Ships: boards on merge (observability workflow); uptime on the next bot release

Why

The bot is about to publish the same liq_* names the exporter sidecar publishes (#1665). The boards sum() those series with no job filter, so both copies would be counted. The pin has to be in before the bot publishes anything.

Decisions

  • gen-t0-liquidity.py adds the job matcher in promql() and fails generation if any liq_ selector is left unpinned, so a new panel can't silently double count. CI runs it with --check.
  • The scan-skipped row grouped hedge_scan_skipped_total by cause, but the bot labels it reason (src/position_check.rs), so the breakdown was always empty.
  • STARTED_AT was a lazy static that started on the first /health call. PROCESS_START is forced first thing in main.

Proof

Merge conditions

  • T0Trade/grafana duplicates: t0.grafana#26 merged (its liquidity boards read liq_* unpinned).

@linear-code

linear-code Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

RAI-2998

RAI-3058

RAI-3083

JuaniRios commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

How to use the Graphite Merge Queue

Add the label add-to-gt-merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@JuaniRios JuaniRios changed the title [RAI-2998] Pin liq_ board queries to the exporter job and fix process start observability: pin liq_ board queries to the exporter job, fix process start Oct 8, 2026
@JuaniRios JuaniRios self-assigned this Oct 8, 2026
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@rain-marvin review

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@rain-marvin

rain-marvin Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

🔎 Reviewing b3c8de6, started by @JuaniRios. The review will appear here when it's done.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: 04073abf-5166-4129-ae57-ee10dfcb9e0d

📥 Commits

Reviewing files that changed from the base of the PR and between 76a7a16 and b3c8de6.


📒 Files selected for processing (15)
  • .github/workflows/ci.yaml
  • docs/ci.md
  • docs/observability.md
  • flake.nix
  • observability/dashboards/liquidity/t0-liquidity.json
  • observability/dashboards/liquidity/tabs/t0-liquidity-logs.json
  • observability/dashboards/liquidity/tabs/t0-liquidity-orders.json
  • observability/dashboards/liquidity/tabs/t0-liquidity-performance.json
  • observability/dashboards/liquidity/tabs/t0-liquidity-pnl.json
  • observability/gen-t0-liquidity.py
  • observability/t0-liquidity-native-rows.json
  • src/api.rs
  • src/bin/server.rs
  • src/lib.rs
  • src/startup.rs

Included review availability: This review used your included allowance. 7 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 8 reviews per hour.



Walkthrough

The generator pins liquidity PromQL selectors to job="t0-liquidity-exporter" and checks that generated board JSON matches committed output. The liquidity dashboards now filter their queries to that job. CI runs the generator check, and documentation describes the pinning and check. The server initializes a shared process-start timestamp, which the health endpoint uses to calculate uptime.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to b3c8d

No actionable merge-blocking issue was found in the current changes; normal checks and the stated dashboard dependency still apply.

Pre-merge checks | Passed 4 | Inconclusive 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage Inconclusive Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files. (11 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly summarizes the primary observability changes: pinning liquidity board queries to the exporter job and fixing process-start tracking.
Description check Passed The description directly explains the board query pinning, scan-skipped label correction, process-start uptime fix, validation results, and merge conditions.

Full details: Docstring Coverage

Explanation

Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files. (11 skipped: 10 unsupported, 1 too large.)



✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR

🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR


  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR pins every liq_ query on the five liquidity Grafana boards to job="t0-liquidity-exporter", so the boards do not count two copies of each series once the bot itself publishes liq_* (#1665). The generator now adds the matcher in promql(), refuses to emit a board with an unpinned liq_ selector, and CI runs it with --check. It also fixes the native "Hedge scans skipped" row to group by reason, the label the bot really emits, and makes /health uptime count from process start instead of from the first /health call.

Overall read: correct and low risk. I ran gen-t0-liquidity.py --check at this head: all five boards match the generator, and the JSON diff only adds the job matcher and the reason fix. The exporter's scrape job_name is t0-liquidity-exporter in both terraform/staging-liquidity/main.tf and terraform/production-liquidity/main.tf in t0.devops, so the pinned queries still return today's data in both environments. PROCESS_START is forced before config load, and /health is only served by the server binary. No blockers. The two minor comments are about the pinning helper accepting broken PromQL, and other liq_* readers outside this repo that will still sum both jobs once #1665 deploys. Also outside the diff: the hedge_scan_skipped_total description in src/metrics.rs still says "by symbol and cause".

claude-opus-5-5 · high · 10 min

Comment thread observability/gen-t0-liquidity.py
Comment thread docs/observability.md
Comment thread observability/t0-liquidity-native-rows.json
Comment thread src/api.rs
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@rain-marvin approve

@rain-marvin

rain-marvin Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

🔎 Verifying that my findings from b3c8de6 are addressed at 3551837, started by @JuaniRios. The review will appear here when it's done.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR pins every liq_ selector on the liquidity Grafana boards to job="t0-liquidity-exporter" through observability/gen-t0-liquidity.py, adds a --check step to CI that fails on drift or on an unpinned selector, fixes the native-row aggregation to sum by (reason, symbol), and moves the process start time to one shared PROCESS_START used by /health.

The new head is a plain rebase onto master. git range-diff shows the PR's own commit unchanged; only the surrounding context in docs/observability.md and flake.nix moved. The generator check line still sits in the backend CI block ahead of the master ci-audit step, and python3 observability/gen-t0-liquidity.py --check passes at this head (5 boards match). No new defects.

Earlier findings:

  • src/api.rs:123 (nit, GIT_COMMIT visibility and comments that describe the next PR): still in the code, left on purpose. The next PR up the stack, #1665, reads api::GIT_COMMIT and adds liq_bot_start_timestamp_seconds, so the code and comments become accurate when it lands. The reason holds for a nit.

Threads (all resolved after agreement, none by hand by someone else):

  • observability/gen-t0-liquidity.py:90 (minor, pin_liq rewrites inside string literals and by (...) lists): dismissed as latent and tracked as a follow-up. That holds: no current board hits it and the change is not needed for this PR to be correct.
  • docs/observability.md:108 (minor, t0.devops readers sum liq_* without a job matcher): addressed outside this repo by T0Trade/t0.devops#787, a merge condition on #1665, which is where double counting would start.
  • observability/t0-liquidity-native-rows.json:295 (nit, nested legendFormat is ignored): dismissed as a follow-up. The aggregation fix stands alone and the nested field does no harm.
  • src/api.rs:123 (nit): as above.

Nothing blocks the merge.

claude-opus-5-5 · high · 43 s

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

… start

The bot is about to publish liq_* names that the exporter sidecar already
publishes. Every liq_ selector on the liquidity boards now reads
job="t0-liquidity-exporter" so the two copies are never summed; the
board generator adds the matcher and fails on any unpinned selector, and
CI runs it with --check.

The scan-skipped native row grouped hedge_scan_skipped_total by cause,
but the bot labels it reason, so the breakdown was always empty.

/health uptime counted from the first /health request because
STARTED_AT was a lazy static. The server now captures PROCESS_START
before loading the config.
@JuaniRios
JuaniRios force-pushed the rai-2998-board-pin-process-start branch from 3551837 to 5d21383 Compare October 9, 2026 22:27
@JuaniRios JuaniRios changed the title observability: pin liq_ board queries to the exporter job, fix process start observability: pin liq_ board queries to the exporter job and fix process start [RAI-3083] Oct 9, 2026
This was referenced Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants