Skip to content

fix: per-dimension alert descriptions when multiple dimensions fail #971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joostboon wants to merge 3 commits into
base: master
Choose a base branch
from
Open

fix: per-dimension alert descriptions when multiple dimensions fail #971

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
57ccbb0
fix: show per-dimension alert descriptions when multiple dimensions fail
joostboon Mar 13, 2026
0eecf7d
fix: black formatting and Vertica Decimal special-value handling
joostboon Mar 15, 2026
892db60
fix: apply black 22.12.0 formatting to test assertion
joostboon Mar 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions integration_tests/tests/test_dimension_anomalies.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,3 +315,76 @@ def test_anomaly_in_detection_period(
)

assert test_result["status"] == expected_status


def test_dimension_anomalies_alert_description_few_failures(
test_id: str, dbt_project: DbtProject
):
"""When ≤5 dimension values fail, description shows each one's anomaly details."""
utc_today = datetime.utcnow().date()
test_date, *training_dates = generate_dates(base_date=utc_today - timedelta(1))

# 3 dimension values all spike on test_date (training: 1/day, test: 10/day)
anomalous_dimensions = ["Batman", "Superman", "Spiderman"]

data: List[Dict[str, Any]] = [
{TIMESTAMP_COLUMN: test_date.strftime(DATE_FORMAT), "superhero": hero}
for hero in anomalous_dimensions
for _ in range(10)
]
data += [
{TIMESTAMP_COLUMN: cur_date.strftime(DATE_FORMAT), "superhero": hero}
for cur_date in training_dates
for hero in anomalous_dimensions
]

test_args = {
"timestamp_column": TIMESTAMP_COLUMN,
"dimensions": ["superhero"],
"sensitivity": 2,
}
test_result = dbt_project.test(test_id, DBT_TEST_NAME, test_args, data=data)
assert test_result["status"] == "fail"

description = test_result["test_results_description"]
# Each failing dimension value should appear in the description
for hero in anomalous_dimensions:
assert hero in description, f"Expected '{hero}' in description: {description}"
# Should NOT show the high-volume summary message
assert "dimension values are anomalous" not in description


def test_dimension_anomalies_alert_description_many_failures(
test_id: str, dbt_project: DbtProject
):
"""When >5 dimension values fail, description shows a count summary."""
utc_today = datetime.utcnow().date()
test_date, *training_dates = generate_dates(base_date=utc_today - timedelta(1))

# 6 dimension values all spike on test_date (>5 threshold)
anomalous_dimensions = ["Batman", "Superman", "Spiderman", "IronMan", "Thor", "Hulk"]

data: List[Dict[str, Any]] = [
{TIMESTAMP_COLUMN: test_date.strftime(DATE_FORMAT), "superhero": hero}
for hero in anomalous_dimensions
for _ in range(10)
]
data += [
{TIMESTAMP_COLUMN: cur_date.strftime(DATE_FORMAT), "superhero": hero}
for cur_date in training_dates
for hero in anomalous_dimensions
]

test_args = {
"timestamp_column": TIMESTAMP_COLUMN,
"dimensions": ["superhero"],
"sensitivity": 2,
}
test_result = dbt_project.test(test_id, DBT_TEST_NAME, test_args, data=data)
assert test_result["status"] == "fail"

description = test_result["test_results_description"]
# Should show the count summary for many failures
assert "dimension values are anomalous" in description, (
f"Expected summary message in description: {description}"
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if the macro threshold logic is correct
rg -n "max_dimension_alerts" --type sql

# Verify the condition uses > not >=
rg -n "anomalous_rows.*length.*max_dimension_alerts" --type sql -A 2 -B 2

# Check if there are any existing tests that verify >5 threshold behavior
rg -n "dimension values are anomalous" integration_tests/

Repository: elementary-data/dbt-data-reliability

Length of output: 2518

🏁 Script executed:

#!/bin/bash

# Get more context of the test function
sed -n '340,395p' integration_tests/tests/test_dimension_anomalies.py

# Look at the anomaly detection macro logic in detail
cat -n macros/edr/data_monitoring/anomaly_detection/store_anomaly_test_results.sql | sed -n '80,110p'

# Check if there's a specific dimension anomaly detection macro
fd -t f "dimension" macros/ | head -20

Repository: elementary-data/dbt-data-reliability

Length of output: 4172

🏁 Script executed:

#!/bin/bash

# Find the dimension anomalies test macro
cat -n macros/edr/tests/test_dimension_anomalies.sql | head -100

# Check what DBT_TEST_NAME is in the test file
rg -n "DBT_TEST_NAME" integration_tests/tests/test_dimension_anomalies.py -B 5 | head -20

# Check if the test data generation uses helpers
rg -n "generate_dates\|DATE_FORMAT\|TIMESTAMP_COLUMN" integration_tests/tests/test_dimension_anomalies.py | head -20

Repository: elementary-data/dbt-data-reliability

Length of output: 5313

Fix Black formatting and investigate anomaly detection failure.

Two issues found:

  1. Black formatting violation (line 362): The anomalous_dimensions list must be reformatted to comply with Black's line length rules.

  2. Test assertion failure (lines 385-389): The test expects "dimension values are anomalous" in the description but it's not present. The macro logic is correct (threshold is 5, condition uses >), so the issue is that the anomaly detection algorithm is not detecting all 6 dimensions as anomalous despite the clear spike pattern. With sensitivity=2, investigate whether:

    • The sensitivity value is too strict for the data pattern
    • The anomaly scoring calculation is not generating high enough scores for all dimensions to exceed the anomalous threshold

Add a debug print before the assertion to see how many dimensions are actually detected:

print(f"DEBUG: description={description}")

Then inspect the anomaly scores being generated for each of the 6 dimensions during test execution to determine why some are not being flagged as anomalous.

Fix Black formatting 
     # 6 dimension values all spike on test_date (>5 threshold)
-    anomalous_dimensions = ["Batman", "Superman", "Spiderman", "IronMan", "Thor", "Hulk"]
+    anomalous_dimensions = [
+        "Batman",
+        "Superman",
+        "Spiderman",
+        "IronMan",
+        "Thor",
+        "Hulk",
+    ]
🧰 Tools
🪛 GitHub Actions: Run pre-commit hooks

[error] 362-366: Black formatting: file reformatted by hook and changes applied. Run 'pre-commit run --all-files' to reproduce locally.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@integration_tests/tests/test_dimension_anomalies.py` around lines 357 - 389,
Reformat the anomalous_dimensions list in
test_dimension_anomalies_alert_description_many_failures to satisfy Black
line-length rules (split items across multiple lines) and add runtime
diagnostics to debug why not all six heroes are flagged: before the final
assertion log the test_result["test_results_description"] and inspect the
anomaly scoring output used by the anomaly detection routine (check where
sensitivity is applied and the score->anomalous threshold is computed) for the
heroes in anomalous_dimensions; verify whether sensitivity=2 is appropriate or
whether the scoring calculation is suppressing scores below the anomalous
threshold and adjust the scoring or threshold logic accordingly so the macro
condition (">5") correctly triggers for six failing dimensions.

)
28 changes: 21 additions & 7 deletions macros/edr/data_monitoring/anomaly_detection/store_anomaly_test_results.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,23 +74,37 @@
and upper(column_name) = upper({{ elementary.const_as_string(column_name) }})
{%- endif %}
{%- endset -%}
{% set test_results_description %}
{% if rows_with_score %}
{{ elementary.insensitive_get_dict_value(rows_with_score[-1], 'anomaly_description') }}
{% else %}
Not enough data to calculate anomaly score.
{% endif %}
{% endset %}
{% set failures = namespace(data=0) %}
{% set filtered_anomaly_scores_rows = [] %}
{% set anomalous_rows = [] %}
{% for row in anomaly_scores_rows %}
{% if row.anomaly_score is not none %}
{% do filtered_anomaly_scores_rows.append(row) %}
{% if row.is_anomalous %}
{% set failures.data = failures.data + 1 %}
{% do anomalous_rows.append(row) %}
{% endif %}
{% endif %}
{% endfor %}
{%- set max_dimension_alerts = 5 -%}
{% set test_results_description %}
{%- if rows_with_score -%}
{%- set sample_row = rows_with_score[-1] -%}
{%- set row_dimension = elementary.insensitive_get_dict_value(sample_row, "dimension") -%}
{%- if row_dimension is not none and anomalous_rows | length > 0 -%}
{%- if anomalous_rows | length > max_dimension_alerts -%}
{%- set remaining = (anomalous_rows | length) - max_dimension_alerts -%}
{{ anomalous_rows | length }} dimension values are anomalous. Showing first {{ max_dimension_alerts }}: {% for row in anomalous_rows[:max_dimension_alerts] %}{{ elementary.insensitive_get_dict_value(row, "dimension_value") }}{% if not loop.last %}, {% endif %}{% endfor %}, and {{ remaining }} more.
{%- else -%}
{% for row in anomalous_rows %}{{ elementary.insensitive_get_dict_value(row, "anomaly_description") }}{% if not loop.last %} | {% endif %}{% endfor %}
{%- endif -%}
{%- else -%}
{{ elementary.insensitive_get_dict_value(rows_with_score[-1], "anomaly_description") }}
{%- endif -%}
{%- else -%}
Not enough data to calculate anomaly score.
{%- endif -%}
{% endset %}
{% set test_result_dict = {
"id": elementary.insensitive_get_dict_value(latest_row, "id"),
"data_issue_id": elementary.insensitive_get_dict_value(
Expand Down
Loading