feat: log a typed kind on every operational alert [RAI-3103]

[JuaniRios]

Part of RAI-3103.

Every operational alert now logs a typed kind field. Item 5 of the RAI-2998 stack, part 3 of 3.

Live effect: each operational_alert line gets kind="...". The message text is unchanged · Risk: medium (the alert pager reads these lines. A text-mode label changes only for an alert whose embedded error text named a different phrase, see below) · Ships: with the next bot release

Why

Today the kind label on operational_alerts comes from a regex over the message text in t0.devops. Every new or reworded alert needs a t0.devops PR, or it pages as unclassified. With JSON logs the bot can emit the kind itself (RAI-2998 comment, scope addition for item 5).

Decisions

• AlertKind in src/alerts/mod.rs is a closed enum, declared by one macro in two groups:
  • extracted: 68 kinds. Their strings are exactly the extractor's phrases, in its order. Tests pin the list against the unclassified rule in observability/alerting/liquidity.rules.yml, and a proptest checks the bot classifier against the real regex.
  • not_extracted: 26 new kinds for alerts the regex cannot classify today (hedge stall, capital CCTP burns, recovery jobs, PnL ledger startup, and others). They still page as unclassified until the extractor and the rules learn them.
• Notifier::notify(kind, message). All 31 direct error!(target: "operational_alert", ...) sites log kind = AlertKind::X.as_str(), and a test fails on a site without one.
• CapturingNotifier checks every alert a test drives: the kind must be what the regex reads from that message.
• Wrapper alerts that embed a rendered error (job dead letters, worker terminal failures, and the Alpaca to Base burn latch) take the most specific kind in their text, with the same rightmost-phrase rule as the regex. This stays until those errors carry a typed kind.
• In text mode the line renders kind="..." after the message, and the regex takes the rightmost phrase. So a text line now classifies by the field. This only changes a label when an embedded runtime error named a different phrase.

Before the kind comparison on staging

• T0Trade/t0.devops#752 is still OPEN. The deployed extractor has 35 of the 68 phrases. Apply feat(offchain-order): record limit_price + client_order_id on the Placed event for audit (M3) #752 first, or kind and alert_kind on operational_alert_kinds will show false mismatches.
• T0Trade/t0.devops#792 must be applied, and the staging VM replaced, before log_format = "json" is set. No config flips it here.

Proof

• Full cargo nextest run -p st0x-hedge -p st0x-config passes. Clippy, fmt, pre-commit, and gen-t0-liquidity.py --check are clean.
• I scripted every direct site's message through the extractor rule. All agree with their kind.
• Multi-model review (sol, Cursor Grok, composer, flash, opus): 3 passes, the last clean on every lane.
• Not verified: a test alert on staging through the new metric. That is the rollout check.

Not in this PR

The liq_trade, liq_transfer, and liq_event structured event lines. A stable event_id needs the committed event sequence, and the event-sorcery Reactor drops it before the reactor sees it. That needs a sequence-aware hook in event-sorcery first, so it comes as a follow-up.

Merge conditions

Same as the stack below.

[linear-code]

RAI-2998

RAI-3063

RAI-3103

[JuaniRios]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• test: harden the liq_ store tests and the parity tool [RAI-3146] #1723
• fix: seed the log counts by file length and give them a family [RAI-3145] #1722
• fix: make the collector metrics say how fresh they are [RAI-3144] #1721
• perf: replay the P&L ledger once for all six windows [RAI-3143] #1720
• fix: bound each P&L window refresh without cancelling its report [RAI-3142] #1719
• test: pin each alert's kind to its message [RAI-3141] #1718
• fix: keep the board's dialogs open and add Copy buttons [RAI-3148] #1717
• fix: pin, legend and sort the board's native rows correctly [RAI-3147] #1716
• fix: write opentelemetry's internal logs with one message key [RAI-3140] #1715
• fix: say what was filled on the liq_trade line [RAI-3150] #1714
• feat: restore missing transfer status lines with a sweep [RAI-3139] #1713
• feat: publish the allocation planner's band verdict per chain [RAI-3137] #1712
• feat: grey the USDC ratio while the corridor is off, and tighten the settings family [RAI-3136] #1711
• feat: publish per-chain equity in flight and take the hedged chains from config [RAI-3135] #1710
• fix: wait for the auxiliary supervisor when the bot shuts down [RAI-3134] #1709
• fix: hold the liq_ inventory series when a snapshot event fails to apply [RAI-3133] #1708
• fix: complete the recovery steps and parse-check the shown commands [RAI-3132] #1707
• docs: close the burn nonce before failing a pre-burn USDC transfer [RAI-3128] #1706
• observability: let the liquidity board's log panels read the bot's own lines [RAI-3064] #1697
• feat: log liq_trade, liq_transfer and liq_event lines from committed events [RAI-3104] #1696
• feat: log a typed kind on every operational alert [RAI-3103] #1695 👈 (View in Graphite)
• feat: flatten console JSON so the alert text stays at message [RAI-3102] #1694
• feat: carry the fill price on the dashboard trade [RAI-3101] #1693
• observability: show each liquidity row's recovery commands in its detail dialog [RAI-3100] #1692
• observability: open a detail dialog from each liquidity trade and rebalance row [RAI-3099] #1691
• observability: show the liquidity inventory as per-chain Grafana tables [RAI-3098] #1690
• observability: give every liquidity board tab the SPA header and recovery guide [RAI-3097] #1689
• feat: show the SPA's recovery commands for the operations client [RAI-3096] #1688
• observability: let the liquidity boards read liq_ from the exporter or the bot [RAI-3095] #1687
• feat: publish the liq_ pending-order and Raindex series from the bot [RAI-3094] #1686
• feat: publish the liq_pnl_* windows from the bot every 5 minutes [RAI-3093] #1685
• refactor: share the /pnl report path so other callers build the same report [RAI-3092] #1684
• feat: count error and warning logs natively for liq_ reliability [RAI-3091] #1683
• feat: publish the liq_ performance families from the bot [RAI-3090] #1682
• feat: count dependency calls and order-fill polls natively [RAI-3089] #1681
• feat: publish liq_ prices and exposure and refresh the inventory every 60 s [RAI-3088] #1680
• feat: publish per-chain inventory and USDC corridor liq_ series [RAI-3087] #1679
• feat: publish the liq_ inventory series on every inventory write [RAI-3086] #1678
• feat: publish the liq_* foundation families from the bot [RAI-3085] #1665
• feat: add the liq_* parity comparison tool [RAI-3084] #1674
• observability: pin liq_ board queries to the exporter job and fix process start [RAI-3083] #1673
• master

---

How to use the Graphite Merge Queue

Add the label add-to-gt-merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[JuaniRios]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.