Skip to content

observability: pin liq_ board queries to the exporter job and fix process start [RAI-3083] #1673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JuaniRios wants to merge 1 commit into
base: master
Choose a base branch
from
+280 −113
Open

observability: pin liq_ board queries to the exporter job and fix process start [RAI-3083] #1673

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,15 @@ jobs:
workspaces: ". -> target/test"
save-if: ${{ env.UNTRUSTED_PR != 'true' }}

# The board JSON must match its generator, with every liq_ selector
# pinned to the exporter job.
- name: Board generator check
run: |
nix develop .#ci-backend -c bash -c "
set -euxo pipefail
python3 observability/gen-t0-liquidity.py --check
"

# Build the test binaries once into a nextest archive shared by all four
# partitions, and run clippy and rustdoc here since this is the single
# full compile.
Expand Down
8 changes: 8 additions & 0 deletions docs/ci.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,11 @@ cannot satisfy pull request branch protection.
Use a pull request event for merge-blocking validation. Reserve
`workflow_dispatch` for manual runs whose result does not need to become a pull
request status check.

## Board check

The `backend build` job also runs
`python3 observability/gen-t0-liquidity.py --check` in the `ci-backend` shell,
and `nix run .#ci` runs it too. It fails when the committed liquidity board JSON
differs from the generator output, or when a `liq_` selector is not pinned to
the exporter job.
10 changes: 10 additions & 0 deletions docs/observability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,3 +104,13 @@ changes land. A refused token file does not alert: whoever publishes it checks
also covers startup failure, alerts through
`liquidity-token-file-fallback-production` in
`observability/alerting/liquidity.rules.yml`.

## Metrics

### Board pins

Until consumers move to the bot, every `liq_` selector on the liquidity boards
Comment thread
rain-marvin[bot] marked this conversation as resolved.
reads `job="t0-liquidity-exporter"`. `observability/gen-t0-liquidity.py` adds
the matcher and fails if any `liq_` selector is left unpinned.
`python3 observability/gen-t0-liquidity.py --check` (run in CI) fails when the
committed board JSON differs from the generator output.
5 changes: 4 additions & 1 deletion flake.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,7 @@
cargo nextest run --workspace --all-features
cargo clippy --workspace --all-targets --all-features
cargo fmt -- --check
python3 observability/gen-t0-liquidity.py --check
'

nix develop .#ci-audit -c bash -c './scripts/test-rust-audit.sh && ./scripts/audit-rust.sh'
Expand Down Expand Up @@ -848,14 +849,16 @@
# CI: cargo check/nextest/clippy + sqlx db reset. No terraform, no
# deploy scripts. abiEnv is needed because build.rs / sol! macros
# consume ST0X_*_ABI at compile time. foundry is required because
# tests spawn anvil for local EVM simulation.
# tests spawn anvil for local EVM simulation. python3 runs the
# board generator check.
ci-backend = pkgs.mkShell (
{
buildInputs =
backendInputs
++ [
pkgs.sqlx-cli
pkgs.cargo-nextest
pkgs.python3
foundryBin
]
++ moldInputs;
Expand Down
22 changes: 11 additions & 11 deletions observability/dashboards/liquidity/t0-liquidity.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_up)",
"expr": "max(liq_up{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -244,7 +244,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info{job=\"t0-liquidity-exporter\"}), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -331,7 +331,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "time() - max(liq_bot_start_timestamp_seconds)",
"expr": "time() - max(liq_bot_start_timestamp_seconds{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -419,7 +419,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (broker) (liq_settings_info)",
"expr": "sum by (broker) (liq_settings_info{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -506,7 +506,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_equity_target)",
"expr": "max(liq_settings_equity_target{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -594,7 +594,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_usdc_target)",
"expr": "max(liq_settings_usdc_target{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -682,7 +682,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_execution_threshold_usd)",
"expr": "max(liq_settings_execution_threshold_usd{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -770,7 +770,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_onchain_available, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Raindex\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_total, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Inflight\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Alpaca Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_total, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Ratio\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio_deviation and liq_usdc_total > 0, \"row\", \"USDC\", \"\", \"\"), \"col\", \"\u0394\", \"\", \"\"))",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_onchain_available{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Raindex\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Inflight\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Alpaca Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Ratio\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio_deviation{job=\"t0-liquidity-exporter\"} and liq_usdc_total{job=\"t0-liquidity-exporter\"} > 0, \"row\", \"USDC\", \"\", \"\"), \"col\", \"\u0394\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1084,7 +1084,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_usdc, \"row\", \"USDC\", \"\", \"\"), \"col\", \"USDC\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_rebalanceable, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Rebalanceable\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Counter-tradeable\", \"\", \"\"))",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_usdc{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"USDC\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_rebalanceable{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Rebalanceable\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Counter-tradeable\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1267,7 +1267,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_ethereum_wallet, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Eth\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_base_wallet, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Base\", \"\", \"\"))",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_ethereum_wallet{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Eth\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_base_wallet{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Base\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1429,7 +1429,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (symbol, col) (label_replace(liq_asset_counter_trading, \"col\", \"CT\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_rebalancing, \"col\", \"Rebal\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_extended_hours, \"col\", \"Ext\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_onchain_available, \"col\", \"Raindex\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_inflight_total, \"col\", \"Inflight\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_offchain_available, \"col\", \"Alpaca\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_total, \"col\", \"Total\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio, \"col\", \"Ratio\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio_deviation and liq_equity_total > 0, \"col\", \"\u0394\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_exposure_usd or (liq_equity_total * NaN), \"col\", \"Exposure\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_unwrapped, \"col\", \"Unwrapped\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_wrapped, \"col\", \"Wrapped\", \"\", \"\"))",
"expr": "sum by (symbol, col) (label_replace(liq_asset_counter_trading{job=\"t0-liquidity-exporter\"}, \"col\", \"CT\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_rebalancing{job=\"t0-liquidity-exporter\"}, \"col\", \"Rebal\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_extended_hours{job=\"t0-liquidity-exporter\"}, \"col\", \"Ext\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_onchain_available{job=\"t0-liquidity-exporter\"}, \"col\", \"Raindex\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_inflight_total{job=\"t0-liquidity-exporter\"}, \"col\", \"Inflight\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_offchain_available{job=\"t0-liquidity-exporter\"}, \"col\", \"Alpaca\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_total{job=\"t0-liquidity-exporter\"}, \"col\", \"Total\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio{job=\"t0-liquidity-exporter\"}, \"col\", \"Ratio\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio_deviation{job=\"t0-liquidity-exporter\"} and liq_equity_total{job=\"t0-liquidity-exporter\"} > 0, \"col\", \"\u0394\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_exposure_usd{job=\"t0-liquidity-exporter\"} or (liq_equity_total{job=\"t0-liquidity-exporter\"} * NaN), \"col\", \"Exposure\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_unwrapped{job=\"t0-liquidity-exporter\"}, \"col\", \"Unwrapped\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_wrapped{job=\"t0-liquidity-exporter\"}, \"col\", \"Wrapped\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down
14 changes: 7 additions & 7 deletions observability/dashboards/liquidity/tabs/t0-liquidity-logs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_up)",
"expr": "max(liq_up{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -406,7 +406,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info{job=\"t0-liquidity-exporter\"}), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -493,7 +493,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "time() - max(liq_bot_start_timestamp_seconds)",
"expr": "time() - max(liq_bot_start_timestamp_seconds{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -581,7 +581,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (broker) (liq_settings_info)",
"expr": "sum by (broker) (liq_settings_info{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -668,7 +668,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_equity_target)",
"expr": "max(liq_settings_equity_target{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -756,7 +756,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_usdc_target)",
"expr": "max(liq_settings_usdc_target{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -844,7 +844,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_execution_threshold_usd)",
"expr": "max(liq_settings_execution_threshold_usd{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down
20 changes: 10 additions & 10 deletions observability/dashboards/liquidity/tabs/t0-liquidity-orders.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_up)",
"expr": "max(liq_up{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -244,7 +244,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info{job=\"t0-liquidity-exporter\"}), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -331,7 +331,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "time() - max(liq_bot_start_timestamp_seconds)",
"expr": "time() - max(liq_bot_start_timestamp_seconds{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -419,7 +419,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (broker) (liq_settings_info)",
"expr": "sum by (broker) (liq_settings_info{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -506,7 +506,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_equity_target)",
"expr": "max(liq_settings_equity_target{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -594,7 +594,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_usdc_target)",
"expr": "max(liq_settings_usdc_target{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -682,7 +682,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_execution_threshold_usd)",
"expr": "max(liq_settings_execution_threshold_usd{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1055,7 +1055,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_pending_orders_total) or vector(0)",
"expr": "max(liq_pending_orders_total{job=\"t0-liquidity-exporter\"}) or vector(0)",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1137,7 +1137,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_raindex_orders_total)",
"expr": "max(liq_raindex_orders_total{job=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1219,7 +1219,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_raindex_orders_unavailable) or vector(0)",
"expr": "max(liq_raindex_orders_unavailable{job=\"t0-liquidity-exporter\"}) or vector(0)",
"step": "60s"
},
"timeSeriesList": {
Expand Down
Loading
Loading