Skip to content

feat: count error and warning logs natively for liq_ reliability [RAI-3091] - #1683

Open
JuaniRios wants to merge 1 commit into
rai-2998-liq-metrics-performancefrom
rai-2998-liq-metrics-log-counts
Open

JuaniRios wants to merge 1 commit into
rai-2998-liq-metrics-performancefrom
rai-2998-liq-metrics-log-counts

Conversation

@JuaniRios

@JuaniRios JuaniRios commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Part of RAI-3091.

Counts error and warning logs as the bot writes them, for liq_reliability_log_count_24h and liq_log_target_count_24h. Item 3 of the RAI-2998 stack, part 3 of 3.

Live effect: the reliability family gains the two log-count names, and /metrics gains log_events_total{level,target}. After each start a background task reads the last 24 h of log files once · Risk: low to medium (a new layer runs on every file log event, and the /logs scanner was refactored) · Ships: on the next bot release

Why

The exporter got these counts by rescanning up to 50,000 log entries through /performance/reliability every minute. Counting as events happen is cheaper and has no cap.

Decisions

  • st0x_config gets a LogEventSink trait. The counting layer is attached only on the file-logging branch, with the file layer's own filter. So it counts what the file log writes, and no dependency TRACE callsite is enabled for it.
  • Counts live in one-minute buckets per level and target. Server main activates them before the subscriber exists.
  • Seed (decision J7): once per process, in the background, the counter adds the entries the earlier process wrote in the 24 h before activation. It takes only entries strictly before activation, so nothing is counted twice. Until the seed finishes the two names are absent, not low. A scan with a read error adds nothing and is retried, and the 3rd attempt keeps what it read.
  • Without file logging both level rows are 0 and there are no target rows, same as the endpoint.
  • Known differences against the endpoint, all in compare.py with evidence: no 50k cap (decision J11), the first minute of the window counts whole, lines an operator's st0x-cli run appends to the same files are counted only after the next restart, and a line the lossy non-blocking writer drops is still counted. liq_reliability_log_count_24h compares keys only, and liq_log_target_count_24h is ignored. A target can exist on one side only.
  • The /logs file scanner is split into visitor helpers that the seed shares. /logs and /performance/reliability behave the same, except one bug fix. A persistent read error, for example a directory named st0x-hedge.log.*, used to loop forever on that file. Now it stops reading that file.

Proof

  • clippy, fmt, and pre-commit clean. golden.py --check, test_compare.py, and gen-t0-liquidity.py --check pass. The reliability golden now includes the log names.
  • Tests: the layer filter (dependency TRACE not dispatched), no counter on the console-only fallback, the 24 h window and the boundary minute, a target leaving, the seed taking only pre-activation entries, a one-time background seed, a read error retried then accepted, the refresh publishing log names only after the seed, and log_events_total.
  • Full st0x-hedge and st0x-config suites on this head: 5283 passed. One failure was api::tests::a_fee_bumped_copy_of_the_burn_is_adopted_as_its_replacement (anvil "max fee per gas less than block base fee"). It's unrelated and passed 3 of 3 times on rerun.
  • Multi-model review loop over the 3 PRs (sol, grok, composer, flash, and opus) converged after 4 passes. It found and fixed a seed on the collectors' critical path, read errors cached as a done seed, the endless read loop, unbounded retries, and an O(n) prune per event.
  • Not verified: counts on staging against /performance/reliability, and the seed time on production-size log files. Those need a release.

Merge conditions

Same as the stack below.

@linear-code

linear-code Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

RAI-2998

RAI-3060

RAI-3091

JuaniRios commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more


How to use the Graphite Merge Queue

Add the label add-to-gt-merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@JuaniRios
JuaniRios requested a review from agryaznov October 8, 2026 20:31
@JuaniRios JuaniRios self-assigned this Oct 8, 2026
@JuaniRios JuaniRios changed the title feat(metrics): count error and warning logs natively for liq_ reliability [RAI-2998] Count error and warning logs natively for liq_ reliability Oct 8, 2026
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: 052b5286-a83d-491f-8e64-e2bfbaee2034

📥 Commits

Reviewing files that changed from the base of the PR and between 11fd31c and 8d9608f.


📒 Files selected for processing (3)
  • docs/observability.md
  • src/lib.rs
  • src/metrics.rs

Limit details: You’ve used all 4 included reviews currently available. Your 59 included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.



Walkthrough

The change adds file-log event counting and two 24-hour reliability metrics, labeled by level and by level and target. The counter records events accepted by the configured file-log filter, seeds prior entries asynchronously, and maintains one-minute buckets. Reliability metrics are omitted until seeding completes. The change also updates tracing setup, log-file scanning, metric documentation, and parity comparison rules.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 8d960

The new reliability metrics count file-log errors and warnings, becoming available after seeding. Persistent read failures may leave partial historical counts, as documented and logged; no actionable merge-blocking risk is identified.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 67.65% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 68 functions across 11 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly summarizes the main change: native counting of error and warning logs for liq reliability metrics. The issue reference and minor spacing issue do not prevent understanding.
Description check Passed The description directly explains the log-counting implementation, metrics, seeding behavior, known differences, testing, and remaining verification work.

Full details: Docstring Coverage

Explanation

Docstring coverage is 67.65% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 68 functions across 11 files. (1 skipped: 1 unsupported.)



  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR

🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR


  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-log-counts branch from a8677c5 to 123d4a8 Compare October 9, 2026 14:41
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-performance branch from eaf0749 to 410233c Compare October 9, 2026 17:59
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-log-counts branch from 123d4a8 to 0eb74db Compare October 9, 2026 17:59
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@rain-marvin review

@rain-marvin

rain-marvin Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

🔎 Reviewing 0eb74db, started by @JuaniRios. The review will appear here when it's done.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes the bot count its own error and warning log lines as it writes them. A tracing layer sits behind the file log's filter and feeds one-minute buckets per level and target. The bot then publishes liq_reliability_log_count_24h, liq_log_target_count_24h and log_events_total itself, so the exporter no longer rescans up to 50,000 log entries every minute. After each start, a one-time background seed adds the previous process's last 24 h from the files, and it takes only entries strictly before activation. The /logs scanner is split into visitor helpers that the seed shares, which also fixes an endless loop on a persistent read error.

Overall this is sound. The activation boundary prevents double counting. The seed state machine is bounded (two retries, then it keeps a partial read) and runs off the collectors' path. Nothing logs while a lock is held. The /logs and /performance/reliability behaviour is preserved apart from the documented fix. A concern that the bucket mutex could stall logging threads during window() or merge() was checked independently and dismissed: the maps are bounded at about 1,440 minutes per target, and the lock is held for microseconds to low milliseconds. I found no upgrade or shared-stall risk. What remains is two minor points (log counts freeze while the SQLite loaders fail, and no positive test that the production setup attaches the counting layer) and two documentation nits.

claude-opus-5-5 · high · 1 h 2 min

Comment thread src/metrics/liquidity/refresh.rs
Comment thread crates/config/src/telemetry.rs
Comment thread crates/config/src/telemetry.rs
Comment thread SPEC.md
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-performance branch from 410233c to 2d275dc Compare October 9, 2026 22:27
@JuaniRios
JuaniRios force-pushed the rai-2998-liq-metrics-log-counts branch from 0eb74db to 045776e Compare October 9, 2026 22:27
@JuaniRios JuaniRios changed the title [RAI-2998] Count error and warning logs natively for liq_ reliability feat: count error and warning logs natively for liq_ reliability [RAI-3091] Oct 9, 2026
This was referenced Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant