Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -8736,7 +8736,9 @@ field unchanged; the bot pins that list against the rules. A kind the extractor
does not know yet pages as unclassified until the extractor and the rules learn
it. A wrapper alert that embeds a rendered error (a dead-lettered job, a
supervised worker's terminal failure) takes the most specific known kind named
in its text, as the extractor does, and falls back to its own kind. The message
in its text, as the extractor does. Each job's terminal failure message carries
its own kind's phrase, so that kind applies when the error names nothing more
specific; the bot pins that every such message holds a known phrase. The message
Comment thread
JuaniRios marked this conversation as resolved.
text stays the same. A text log line also renders `kind="..."` after the
message, and the extractor takes the rightmost phrase on the line, so a text
line now classifies by the field unless a later field names another phrase. This
Expand Down
23 changes: 19 additions & 4 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,10 @@ stays lost, and so does the `liq_trade` line of a hedge release.
Every `operational_alert` line carries a `kind` field, an `AlertKind` from
`src/alerts/mod.rs`. Send alerts through `Notifier::notify(kind, message)`. A
direct `error!(target: "operational_alert", ...)` line must also log
`kind = AlertKind::...as_str()`; a test fails on one that does not.
`kind = AlertKind::Variant.as_str()` over a plain string literal message. A test
scans every direct line: it fails on a line without a kind, and it checks the
kind against the message literal with the same rules `CapturingNotifier` uses
(below).

- An existing kind's string is exactly the label the `operational_alerts`
extractor (t0.devops `observability-consumer`) gives the line once that
Expand All @@ -242,9 +245,21 @@ direct `error!(target: "operational_alert", ...)` line must also log
- A new alert gets a new kind in the `not_extracted` group, whose string is a
fixed phrase of its message. Teach the extractor that phrase and add a rule;
then move the kind into the `extracted` group in the extractor's order.
- `CapturingNotifier` checks every alert a test drives: an extracted kind must
be what the extractor reads from the message, and a not-extracted kind must
match no extracted phrase.
- Every kind's string must appear in its message. `CapturingNotifier` checks
every alert a test drives: an extracted kind must be what the extractor reads
from the message, and the message of a not-extracted kind must hold no
extracted phrase.
- A job's terminal failure page (a dead letter or a supervised worker's terminal
failure) takes its kind from its text: the most specific extracted phrase in
the rendered alert, else `JobFailedAfterRetries`. The job's
`TERMINAL_FAILURE_MSG` decides the kind when the embedded error names nothing
Comment thread
JuaniRios marked this conversation as resolved.
more specific, so an override must carry the phrase of its own extracted kind.
`build_supervised_worker!` and `build_best_effort_worker!` fail the build on a
message with no extracted phrase, and a test checks every
`TERMINAL_FAILURE_MSG` in the source. So a job's terminal page cannot use a
`not_extracted` kind: the rule above for a new alert does not apply to it
until the extractor knows its phrase and the kind moves to the `extracted`
group.

The phrase extractor is transitional. Once the `operational_alerts` pager reads
`jsonPayload.kind`, a new alert needs only a new `AlertKind` and a rule in
Expand Down
Loading
Loading