Skip to content

feat: publish the liq_* foundation families from the bot [RAI-3085] - #1665

Open
JuaniRios wants to merge 1 commit into
rai-2998-liq-parity-comparefrom
rai-2998-liq-metrics-foundation
Open

JuaniRios wants to merge 1 commit into
rai-2998-liq-parity-comparefrom
rai-2998-liq-metrics-foundation

Conversation

@JuaniRios

@JuaniRios JuaniRios commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Part of RAI-3085.

Adds a liq_* family store to the bot's /metrics and publishes the first two families, health and settings, with the same names, labels and values as the exporter sidecar. Part of the RAI-2998 stack; items 2 to 4 port inventory, performance and PnL on top.

Live effect: the bot's /metrics gains liq_bot_*, liq_settings_* and liq_asset_* series (scraped as job t0-liquidity); nothing reads them yet because every consumer is pinned to the exporter job · Risk: low (no money path or stored data; boot only builds a few gauges) · Ships: on the next bot release

Why

The exporter re-derives numbers the bot already has, from outside the app (see the Zulip thread and the plan). Moving the metrics into the bot starts here.

Decisions

  • Own store, not the metrics recorder. The recorder never forgets a label set, so a removed symbol would keep its last value forever. The store replaces a whole family on each publish, like the exporter does. ADR 0026 (status Proposed).
  • Typed gauges. Every liq_* block has # HELP plus # TYPE <name> gauge. Why: Managed Prometheus ingests untyped samples twice (as gauge and as counter), so untyped roughly doubles the liq_* sample cost. Typing now also saves a second migration later, and gives one descriptor per name, so tools treat them right. Every name really is a gauge: snapshots, _24h windows that go down, *_total names that mean "count right now", and precomputed quantiles that aren't real summaries.
  • Cost we accept: the exporter's untyped history and the bot's gauge history are separate descriptors, so a query by name with no job selector has a gap at cutover. While both run, the same name exists as untyped (exporter job) and gauge (bot job). Nothing counts twice: the board selects by job=~"$source", and the capital probe and TVL report are pinned to the exporter job (t0.devops#787).
  • promtool's default lint warns on the gauges named *_total (liq_pending_orders_total, liq_raindex_orders_total, liq_pending_orders_uncapped_total, and the inventory totals) and on the _ms names. Known exceptions, the names keep the exporter's meaning. Use promtool check metrics --lint=none --extended.
  • A test fails if any liq_ name goes through the metrics macros.
  • Values match the exporter's quirks on purpose (trading_mode="", absent vs zero). The rule and its one exception are in SPEC.md under "Prometheus metrics".
  • Golden tests compare the Rust builders with the exporter's own output for the same fixtures, samples only (the exporter goldens have no # TYPE). golden.py regenerates them and refuses any exporter file that isn't t0.devops 226c029 (sha256 pinned).

Proof

  • clippy, fmt, doc and pre-commit clean; metrics tests 31 passed; golden.py --check matches.
  • After the switch to typed gauges, on the stack top: full cargo nextest run -p st0x-hedge -p st0x-config 5354 passed, golden.py --check 12 match, test_compare.py and gen-t0-liquidity.py --check pass.
  • Multi-model review loop: 3 passes, converged (sol, grok, composer, opus clean in the last pass; flash clean in pass 2, errored in pass 3). The typed-gauge change got its own 3-pass review (sol, grok, composer, flash, opus), clean in the last pass.
  • Not verified: staging /metrics and the Cloud Monitoring descriptor check. Both need a release.

Merge conditions

  • t0.devops#787 merged and applied in prod (pins the capital probe and TVL report to the exporter job). A prod release cut from master would ship this.
  • t0.grafana#26 merged.

@linear-code

linear-code Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

RAI-2998

RAI-3058

RAI-3085

JuaniRios commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more


How to use the Graphite Merge Queue

Add the label add-to-gt-merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@JuaniRios JuaniRios self-assigned this Oct 7, 2026
@JuaniRios JuaniRios changed the title [RAI-2998] Document the bot metrics scrape jobs feat: publish liq_* foundation metrics from the bot Oct 7, 2026
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-foundation branch from 87ac2d9 to 7864332 Compare October 8, 2026 11:47
Comment thread scripts/liq-parity/compare.py Outdated
@JuaniRios
JuaniRios changed the base branch from master to graphite-base/1665 October 8, 2026 12:39
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-foundation branch from 7864332 to 5bfdb50 Compare October 8, 2026 12:39
@JuaniRios
JuaniRios changed the base branch from graphite-base/1665 to rai-2998-liq-parity-compare October 8, 2026 12:39
@JuaniRios JuaniRios changed the title feat: publish liq_* foundation metrics from the bot metrics: publish liq_* foundation families from the bot Oct 8, 2026
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@rain-marvin review

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@rain-marvin

rain-marvin Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

🔎 Reviewing 5bfdb50, started by @JuaniRios. The review will appear here when it's done.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: 5ae6edd1-f84c-499a-9a81-45595cafd5b6

📥 Commits

Reviewing files that changed from the base of the PR and between 5bfdb50 and 44039e0.


📒 Files selected for processing (7)
  • SPEC.md
  • adrs/0026-liq-metrics-family-store-and-untyped-exposition.md
  • adrs/README.md
  • docs/observability.md
  • src/metrics.rs
  • src/metrics/liquidity.rs
  • src/metrics/liquidity/settings.rs

Included review availability: This review used your included allowance. 6 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 8 reviews per hour.



Walkthrough

The change adds a typed liq_* family store that renders gauge metrics after recorder output. It adds boot-time health and settings metrics, sample validation, whole-family replacement, and exposition tests. A parity script renders or checks golden files against a pinned exporter revision. Documentation describes the metric contract and the process for adding metric families.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to 44039

No additional merge-blocking issue is established. Complete the stated deployment checks and merge conditions before release.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 67.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 73 functions across 5 files. (4 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description clearly explains the family store, health and settings metric publication, typed gauges, compatibility behavior, tests, and deployment conditions. It is directly related to the changes…
Title check Passed The title clearly identifies the main change: publishing the liq_* foundation families from the bot. It is concise and specific.

Full details: Docstring Coverage

Explanation

Docstring coverage is 67.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 73 functions across 5 files. (4 skipped: 4 unsupported.)



  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR

🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR


  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a liq_* family store to the bot's /metrics. Each family is replaced as one unit, so a removed symbol drops out on the next scrape. It publishes the health and settings families with the exporter sidecar's names, labels and untyped exposition. Golden tests compare the Rust builders with the exporter's own output, and SPEC.md, ADR 0026 and the observability docs describe the contract.

Overall read: the store is small and correct. Replace takes the lock only to swap an Arc, and render formats outside the lock. Samples are checked at construction (label keys, finite values). Wrong-family and duplicate samples are dropped and logged, and a liq_ name the recorder already rendered is skipped. Nothing reads the new series yet, so the live risk is low.

Several lanes raised that the new docs disclose monitoring topology in a public repo, and that /metrics serves the settings without auth. I checked both and dropped them. The job names, t0.devops, the exporter path, the capital probe and Cloud Monitoring were already public in this repo before the PR. The same settings DTO is already served on the same listener under /api, and the exporter already republishes it. Both endpoints listen only on the VM (docs/observability.md).

Two minor test-quality notes remain. Neither blocks.

claude-opus-5-5 · high · 11 min

Comment thread src/metrics.rs
Comment thread src/metrics/liquidity.rs
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-parity-compare branch from adf06ec to 9b093f3 Compare October 8, 2026 12:52
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-foundation branch from 5bfdb50 to 70c7eef Compare October 8, 2026 12:52
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-parity-compare branch from 9b093f3 to cca682c Compare October 8, 2026 12:54
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-foundation branch from 70c7eef to c766b5f Compare October 8, 2026 12:54
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-parity-compare branch from cca682c to 8cf45d2 Compare October 8, 2026 13:07
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-foundation branch from c766b5f to 85754af Compare October 8, 2026 13:07
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@JuaniRios
JuaniRios force-pushed the rai-2998-liq-parity-compare branch from c6ff16c to bbf7fcf Compare October 9, 2026 17:59
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-foundation branch from 44039e0 to 815310e Compare October 9, 2026 17:59
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@rain-marvin review

@rain-marvin

rain-marvin Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

🔎 Reviewing 815310e, started by @JuaniRios. The review will appear here when it's done.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a liq_* family store to the bot's /metrics endpoint and publishes the first two families, health and settings, with the same names, labels and values as the exporter sidecar. The store replaces each family as a whole on publish, so removed symbols leave /metrics instead of keeping their last value, and every liq_* block is typed as a gauge. Boot publishes both families once, after the config loads. Nothing reads the new series yet, and the change touches no money path or stored data.

Overall read: sound and ready to merge. The store holds its mutex only to swap or clone a family, and formatting runs after the lock is released, so a scrape cannot stall the bot. Rendering follows the Prometheus text format and matches the exporter goldens. No schema, migration or config-format change can break an upgrade. Three small comments:

  • SettingsInput::from_ctx repeats the dashboard's settings-selection rules, and the parity test only covers the branches where the two copies cannot differ.
  • The freshness series for health and settings stays at the boot time, which a later staleness alert can misread as a stalled collector.
  • The ADR file name still says "untyped", which contradicts the decision.

One more concern was checked and dismissed: liq_settings_info carries the wallet address and the Turnkey organization id as labels. The public repo only holds placeholder values, the exporter already publishes the same labels to the same private Prometheus, and neither value is a secret.

Earlier threads (the macro scan, the concurrency test name, DROP_LIST) are resolved, and nothing here repeats them.

claude-opus-5-5 · high · 25 min

Comment thread src/metrics/liquidity/settings.rs
Comment thread SPEC.md
Comment thread adrs/README.md
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

Add a liq_* family store to the bot's /metrics endpoint, rendered after
the recorder's output and untyped like the exporter sidecar, so Cloud
Monitoring keeps one series history across the cutover. Each refresh
replaces a whole family, so a removed symbol, day or chain disappears
instead of keeping its last value.

This PR publishes the health and settings families with the exporter's
names, labels and values. Golden tests compare the builders with the
exporter's output for the same fixtures; scripts/liq-parity/golden.py
regenerates those files from a pinned exporter revision.

Adds ADR 0026 (proposed), the SPEC metrics contract and the metrics
catalog in docs/observability.md.
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-parity-compare branch from bbf7fcf to ddc1e42 Compare October 9, 2026 22:27
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-foundation branch from 815310e to d1e622c Compare October 9, 2026 22:27
@JuaniRios JuaniRios changed the title metrics: publish liq_* foundation families from the bot feat: publish the liq_* foundation families from the bot [RAI-3085] Oct 9, 2026
This was referenced Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants