Skip to content

observability: let the liquidity boards read liq_ from the exporter or the bot [RAI-3095] #1687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JuaniRios wants to merge 1 commit into
base: rai-2998-liq-metrics-orders
Choose a base branch
from
+314 −118
Open

observability: let the liquidity boards read liq_ from the exporter or the bot [RAI-3095] #1687

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions docs/observability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,11 +277,34 @@ its `pnl_<window>` collector time stops advancing.
compares samples only. Extend `PORTED` and, where needed, `KNOWN_DIFFS` in
`scripts/liq-parity/compare.py`.

### Board pins

Until consumers move to the bot, every `liq_` selector on the liquidity boards
reads `job="t0-liquidity-exporter"`. `observability/gen-t0-liquidity.py` adds
the matcher and fails if any `liq_` selector is left unpinned.
### Board source

Both the exporter sidecar and the bot serve `liq_*`, so a selector without a job
would add the two together. The liquidity boards have a `Source` variable,
`$source`, and every `liq_` selector on them reads `job=~"$source"`:

| Source | Job matcher |
| -------------------- | ------------------------------------ |
| `exporter` (default) | `t0-liquidity-exporter` |
| `bot` | `t0-liquidity\|t0-liquidity-staging` |

The bot's job is `t0-liquidity` in production and `t0-liquidity-staging` in
staging. `$env` already picks the project, so one regex covers both. PromQL
anchors a regex matcher, so `bot` never matches the exporter job.

The bot never emits `liq_up`, so the header's Bot pill reads the exporter's
`liq_up` on the exporter source and the bot scrape's own `up` on the bot source:
`max(liq_up{job=~"$source"}) or max(up{job=~"$source",job!="t0-liquidity-exporter"})`.
A stopped bot shows red on either source. A stopped exporter shows no data, not
red, because the exporter's own `up` says nothing about the bot.

Pick `bot` to compare the two side by side. Stage 4 of the migration flips the
default to `bot` in `SOURCE_VAR`, a one-line change; a later stage removes the
Comment thread
rain-marvin[bot] marked this conversation as resolved.
variable with the exporter. The log tables (Trades, Rebalances, Logs) read the
exporter's log names on either source until the bot writes its own.

`observability/gen-t0-liquidity.py` adds the matcher and fails if any `liq_`
Comment thread
rain-marvin[bot] marked this conversation as resolved.
selector is left without exactly `job=~"$source"`.
`python3 observability/gen-t0-liquidity.py --check` (run in CI) fails when the
committed board JSON differs from the generator output.

Expand Down
8 changes: 8 additions & 0 deletions observability/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@ The Liquidity bot board and its four tab boards are generated: edit
Do not edit those files by hand or replace them with a Grafana export; the next
run of the generator overwrites them.

The liquidity boards read `liq_*` from one source at a time, picked by the
`Source` variable: the exporter sidecar (`job="t0-liquidity-exporter"`, the
default) or the bot's own `/metrics` (`t0-liquidity` in production,
`t0-liquidity-staging` in staging). The generator writes `job=~"$source"` into
every `liq_` selector and fails on one without it, because an unpinned selector
adds both sources together. Stage 4 of the migration makes `bot` the default
(`SOURCE_VAR` in the generator). See `docs/observability.md`, "Board source".

`alerting/liquidity.rules.yml` is the alert rules, in the `Alerts` folder. Each
rule's `uid` is permanent and must be unique across every repo's file. Removing
a rule from the file does not remove it from Grafana: add its `uid` to
Expand Down
51 changes: 39 additions & 12 deletions observability/dashboards/liquidity/t0-liquidity.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,33 @@
"value": "t0-liquidity-staging"
}
]
},
{
"name": "source",
"type": "custom",
"label": "Source",
"description": "Which process serves the liq_* metrics: the exporter sidecar (the default for now) or the bot's own /metrics. Pick bot to compare the two.",
"query": "exporter : t0-liquidity-exporter, bot : t0-liquidity|t0-liquidity-staging",
"includeAll": false,
"multi": false,
"hide": 0,
"current": {
"selected": true,
"text": "exporter",
"value": "t0-liquidity-exporter"
},
"options": [
{
"selected": true,
"text": "exporter",
"value": "t0-liquidity-exporter"
},
{
"selected": false,
"text": "bot",
"value": "t0-liquidity|t0-liquidity-staging"
}
]
}
]
},
Expand Down Expand Up @@ -127,7 +154,7 @@
"id": 1,
"type": "stat",
"title": "",
"description": "The SPA header's connection light: is the bot process up and answering /health? '\u2014' means the exporter itself is not reporting.",
"description": "The SPA header's connection light: is the bot process up and answering? The exporter source reads liq_up, which is 0 when the exporter cannot reach the bot's /health. The bot never emits liq_up, so the bot source reads the scrape's own up. '\u2014' means the source is not reporting.",
"datasource": {
"type": "stackdriver",
"uid": "cloudmon"
Expand All @@ -142,7 +169,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_up{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_up{job=~\"$source\"}) or max(up{job=~\"$source\",job!=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -244,7 +271,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info{job=\"t0-liquidity-exporter\"}), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info{job=~\"$source\"}), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -331,7 +358,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "time() - max(liq_bot_start_timestamp_seconds{job=\"t0-liquidity-exporter\"})",
"expr": "time() - max(liq_bot_start_timestamp_seconds{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -419,7 +446,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (broker) (liq_settings_info{job=\"t0-liquidity-exporter\"})",
"expr": "sum by (broker) (liq_settings_info{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -506,7 +533,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_equity_target{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_settings_equity_target{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -594,7 +621,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_usdc_target{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_settings_usdc_target{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -682,7 +709,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_execution_threshold_usd{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_settings_execution_threshold_usd{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -770,7 +797,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_onchain_available{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Raindex\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Inflight\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Alpaca Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Ratio\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio_deviation{job=\"t0-liquidity-exporter\"} and liq_usdc_total{job=\"t0-liquidity-exporter\"} > 0, \"row\", \"USDC\", \"\", \"\"), \"col\", \"\u0394\", \"\", \"\"))",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_onchain_available{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Raindex\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_total{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Inflight\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Alpaca Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_total{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Total\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Ratio\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_ratio_deviation{job=~\"$source\"} and liq_usdc_total{job=~\"$source\"} > 0, \"row\", \"USDC\", \"\", \"\"), \"col\", \"\u0394\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1084,7 +1111,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_usdc{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"USDC\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_rebalanceable{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Rebalanceable\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Counter-tradeable\", \"\", \"\"))",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_usdc{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"USDC\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_rebalanceable{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Rebalanceable\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_alpaca_total{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Counter-tradeable\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1267,7 +1294,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_ethereum_wallet{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Eth\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_base_wallet{job=\"t0-liquidity-exporter\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Base\", \"\", \"\"))",
"expr": "sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_ethereum_wallet{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Eth\", \"\", \"\")) or sum by (row, col) (label_replace(label_replace(liq_usdc_inflight_base_wallet{job=~\"$source\"}, \"row\", \"USDC\", \"\", \"\"), \"col\", \"Base\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -1429,7 +1456,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (symbol, col) (label_replace(liq_asset_counter_trading{job=\"t0-liquidity-exporter\"}, \"col\", \"CT\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_rebalancing{job=\"t0-liquidity-exporter\"}, \"col\", \"Rebal\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_extended_hours{job=\"t0-liquidity-exporter\"}, \"col\", \"Ext\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_onchain_available{job=\"t0-liquidity-exporter\"}, \"col\", \"Raindex\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_inflight_total{job=\"t0-liquidity-exporter\"}, \"col\", \"Inflight\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_offchain_available{job=\"t0-liquidity-exporter\"}, \"col\", \"Alpaca\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_total{job=\"t0-liquidity-exporter\"}, \"col\", \"Total\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio{job=\"t0-liquidity-exporter\"}, \"col\", \"Ratio\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio_deviation{job=\"t0-liquidity-exporter\"} and liq_equity_total{job=\"t0-liquidity-exporter\"} > 0, \"col\", \"\u0394\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_exposure_usd{job=\"t0-liquidity-exporter\"} or (liq_equity_total{job=\"t0-liquidity-exporter\"} * NaN), \"col\", \"Exposure\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_unwrapped{job=\"t0-liquidity-exporter\"}, \"col\", \"Unwrapped\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_wrapped{job=\"t0-liquidity-exporter\"}, \"col\", \"Wrapped\", \"\", \"\"))",
"expr": "sum by (symbol, col) (label_replace(liq_asset_counter_trading{job=~\"$source\"}, \"col\", \"CT\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_rebalancing{job=~\"$source\"}, \"col\", \"Rebal\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_asset_extended_hours{job=~\"$source\"}, \"col\", \"Ext\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_onchain_available{job=~\"$source\"}, \"col\", \"Raindex\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_inflight_total{job=~\"$source\"}, \"col\", \"Inflight\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_offchain_available{job=~\"$source\"}, \"col\", \"Alpaca\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_total{job=~\"$source\"}, \"col\", \"Total\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio{job=~\"$source\"}, \"col\", \"Ratio\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_ratio_deviation{job=~\"$source\"} and liq_equity_total{job=~\"$source\"} > 0, \"col\", \"\u0394\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_exposure_usd{job=~\"$source\"} or (liq_equity_total{job=~\"$source\"} * NaN), \"col\", \"Exposure\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_unwrapped{job=~\"$source\"}, \"col\", \"Unwrapped\", \"\", \"\")) or sum by (symbol, col) (label_replace(liq_equity_wrapped{job=~\"$source\"}, \"col\", \"Wrapped\", \"\", \"\"))",
"step": "60s"
},
"timeSeriesList": {
Expand Down
43 changes: 35 additions & 8 deletions observability/dashboards/liquidity/tabs/t0-liquidity-logs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,33 @@
}
]
},
{
"name": "source",
"type": "custom",
"label": "Source",
"description": "Which process serves the liq_* metrics: the exporter sidecar (the default for now) or the bot's own /metrics. Pick bot to compare the two.",
"query": "exporter : t0-liquidity-exporter, bot : t0-liquidity|t0-liquidity-staging",
"includeAll": false,
"multi": false,
"hide": 0,
"current": {
"selected": true,
"text": "exporter",
"value": "t0-liquidity-exporter"
},
"options": [
{
"selected": true,
"text": "exporter",
"value": "t0-liquidity-exporter"
},
{
"selected": false,
"text": "bot",
"value": "t0-liquidity|t0-liquidity-staging"
}
]
},
{
"name": "level",
"type": "custom",
Expand Down Expand Up @@ -289,7 +316,7 @@
"id": 108,
"type": "stat",
"title": "",
"description": "The SPA header's connection light: is the bot process up and answering /health? '\u2014' means the exporter itself is not reporting.",
"description": "The SPA header's connection light: is the bot process up and answering? The exporter source reads liq_up, which is 0 when the exporter cannot reach the bot's /health. The bot never emits liq_up, so the bot source reads the scrape's own up. '\u2014' means the source is not reporting.",
"datasource": {
"type": "stackdriver",
"uid": "cloudmon"
Expand All @@ -304,7 +331,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_up{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_up{job=~\"$source\"}) or max(up{job=~\"$source\",job!=\"t0-liquidity-exporter\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -406,7 +433,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info{job=\"t0-liquidity-exporter\"}), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"expr": "max by (git_commit) (topk(1, label_replace(timestamp(liq_bot_info{job=~\"$source\"}), \"git_commit\", \"$1\", \"git_commit\", \"(.{7}).*\")))",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -493,7 +520,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "time() - max(liq_bot_start_timestamp_seconds{job=\"t0-liquidity-exporter\"})",
"expr": "time() - max(liq_bot_start_timestamp_seconds{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -581,7 +608,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "sum by (broker) (liq_settings_info{job=\"t0-liquidity-exporter\"})",
"expr": "sum by (broker) (liq_settings_info{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -668,7 +695,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_equity_target{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_settings_equity_target{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -756,7 +783,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_usdc_target{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_settings_usdc_target{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down Expand Up @@ -844,7 +871,7 @@
"queryType": "promQL",
"promQLQuery": {
"projectName": "$env",
"expr": "max(liq_settings_execution_threshold_usd{job=\"t0-liquidity-exporter\"})",
"expr": "max(liq_settings_execution_threshold_usd{job=~\"$source\"})",
"step": "60s"
},
"timeSeriesList": {
Expand Down
Loading
Loading