perf: skip unnecessary metadata fetch calls for tags when not configured by tejassp-db · Pull Request #1387 · databricks/dbt-databricks

tejassp-db · 2026-04-10T12:45:36Z

Summary

Skips fetch_tags and fetch_column_tags queries to information_schema during incremental and view materializations when the model has no tags configured
Falls back to fetching metadata when model config is unavailable (safe default)
No behavior change for models with tags configured — diffs still computed correctly

What changed

Added requires_server_metadata_for_diff() to TagsConfig and ColumnTagsConfig to signal when server metadata is needed
IncrementalTableAPI._describe_relation and ViewAPI._describe_relation now check model config before making tag fetch calls
get_relation_config accepts the model config to pass through to _describe_relation
Fixed duplicate TagsProcessor in StreamingTableConfig.config_components

Test plan

Unit tests: empty tags (skip), non-empty tags (fetch), null config fallback (fetch), both tags + column tags present, hive_metastore (skip)
Functional tests: override fetch_tags/fetch_column_tags macros to raise errors, confirming calls are actually skipped when expected

PECOBLR-2497

Skip fetch_tags and fetch_column_tags information_schema queries during incremental and view materializations when the model has no tags configured. This avoids unnecessary server roundtrips on every run for models that don't use tags, while preserving full fetch behavior when tags are present or when the model config is unavailable. PECOBLR-2497

github-actions · 2026-04-10T13:00:03Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
impl.py					970, 978-989, 1070, 1081, 1089, 1101, 1130, 1169, 1196, 1256, 1294, 1303-1315
dbt/adapters/databricks/relation_configs
base.py					41
column_tags.py
tags.py
Project Total

_{This report was generated by python-coverage-comment-action}

Add unit and functional coverage for skipping tag metadata queries when model config does not require them. This protects the new fetch-planning logic across incremental, view, streaming table, and materialized view test paths without changing unrelated unstaged work.

partial_parse.msgpack does not persist compiled_code, causing get_config_from_model to raise "Cannot compile model ... with no SQL query" for materialized view and streaming table change tests. Co-authored-by: Isaac

Co-authored-by: Isaac

The previous `dict and len(...) > 0` short-circuited to a dict literal when empty, breaking the declared `-> bool` return type. Co-authored-by: Isaac

sd-db

Overall intentation of the change looks good, there are a few things specifically where we need changes

Function signature - make new param optional + naming + other related changes
Coverage - Can you confirm on coverage across all supported materialisation types (MV/ST missing; Views don't have support for column_tags)
Functional test changes

sd-db · 2026-05-07T06:40:18Z

+            else:
+                results["information_schema.tags"] = None
+
+            column_tag_config = (


Similar comments on naming + checks on desried_column_tag_config

Suffix _config and surrounding code makes it obvious that its coming from the user configuration.

Variable naming should still be intentional and descriptive, we should update it to model_column_tag_config

- Add functional tests for optional tag fetching in Materialized view. - Fix tests to check exceptions, and not log lines. - Make parameter optional to be backward compatible with jinja calls.

sd-db

Added some more comments, overall much better. I still feel we should add a comment in all the materialisation types where we are doing this optimisation that this only works because we don't support deleting/removing tags through dbt-databricks, as that will break the optimisations done and better to encode as a comment

sd-db · 2026-05-07T10:36:38Z

-        results["information_schema.tags"] = adapter.execute_macro("fetch_tags", kwargs=kwargs)
        results["show_tblproperties"] = adapter.execute_macro("fetch_tbl_properties", kwargs=kwargs)
+
+        table_tag_config = model_config.config.get(TagsProcessor.name) if model_config else None


rename to model_table_tag_config we should differentiate between desired(model) configs and existing (relation) configs better

Config only comes from user and its obvious here.

sd-db

Reviewing tests 1/2

sd-db

Reviewing Tests 2/2

sd-db

Changes look good, thanks !

tejassp-db · 2026-05-12T07:03:23Z

/integration-test

github-actions · 2026-05-12T07:03:37Z

Integration tests dispatched for PR #1387 by @tejassp-db. Track progress in the Actions tab.

github-actions · 2026-05-12T07:55:19Z

Integration results for PR #1387 — UC cluster ❌ failure · SQL warehouse ❌ failure · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

tejassp-db · 2026-05-13T05:51:43Z

/integration-test

github-actions · 2026-05-13T05:53:06Z

Integration tests dispatched for PR #1387 by @tejassp-db. Track progress in the Actions tab.

github-actions · 2026-05-13T06:45:37Z

Integration results for PR #1387 — UC cluster ✅ success · SQL warehouse ✅ success · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

tejassp-db self-assigned this Apr 10, 2026

tejassp-db requested review from benc-db, jprakash-db and sd-db as code owners April 10, 2026 12:45

tejassp-db had a problem deploying to azure-prod April 10, 2026 12:45 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 10, 2026 12:45 — with GitHub Actions Inactive

tejassp-db temporarily deployed to azure-prod April 13, 2026 10:22 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 13, 2026 10:22 — with GitHub Actions Failure

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

815ff55

tejassp-db temporarily deployed to azure-prod April 15, 2026 05:25 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 15, 2026 05:25 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 15, 2026 05:25 — with GitHub Actions Inactive

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

7839417

tejassp-db temporarily deployed to azure-prod April 17, 2026 04:10 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 17, 2026 04:10 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 17, 2026 04:10 — with GitHub Actions Inactive

tejassp-db and others added 9 commits April 27, 2026 10:28

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

91e0dea

fix: backfill compiled_code from raw_code in MV/ST test helper

a84e0f0

partial_parse.msgpack does not persist compiled_code, causing get_config_from_model to raise "Cannot compile model ... with no SQL query" for materialized view and streaming table change tests. Co-authored-by: Isaac

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

51a495e

chore: ruff format pass

0ed403b

Co-authored-by: Isaac

fix: return bool from requires_server_metadata_for_diff

f4399cd

The previous `dict and len(...) > 0` short-circuited to a dict literal when empty, breaking the declared `-> bool` return type. Co-authored-by: Isaac

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

e6adab7

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

ba26eb1

Fix failing tests.

f985dc8

Fix name shadowing

616239f

sd-db requested changes May 7, 2026

View reviewed changes

fix review comments.

12e008a

- Add functional tests for optional tag fetching in Materialized view. - Fix tests to check exceptions, and not log lines. - Make parameter optional to be backward compatible with jinja calls.

sd-db reviewed May 7, 2026

View reviewed changes

Comment thread dbt/adapters/databricks/impl.py

Rename variable

adee055

sd-db reviewed May 7, 2026

View reviewed changes

sd-db self-requested a review May 8, 2026 08:31

tejassp-db added 2 commits May 8, 2026 15:49

Add test for cascading table tags config

c727e76

Clarify backward compatible behavior with comments

8b57e75

sd-db reviewed May 8, 2026

View reviewed changes

Comment thread tests/functional/adapter/views/test_view_metadata_fetch_skips.py Outdated

Comment thread tests/functional/adapter/helpers.py Outdated

Comment thread tests/unit/relation_configs/test_materialized_view_config.py Outdated

sd-db reviewed May 8, 2026

View reviewed changes

Comment thread tests/unit/test_adapter.py

Reformat code

637d361

tejassp-db changed the base branch from main to 1.12.latest May 11, 2026 06:48

tejassp-db added 6 commits May 11, 2026 12:37

Merge branch '1.12.latest' into PECOBLR-2497/skip-unused-metadata-calls

f29ddeb

Add conditional tag fetch in MetricView

202249b

Refactor tests

4399e92

Refactor tests

c352241

MV fetch tags tests

637e84a

Refactor tests

aef43ea

sd-db approved these changes May 12, 2026

View reviewed changes

Update CHANGELOG

63542cc

Fix arguments to macro calls

c2d4999

tejassp-db merged commit 578375a into 1.12.latest May 13, 2026
7 checks passed

tejassp-db deleted the PECOBLR-2497/skip-unused-metadata-calls branch May 13, 2026 07:01

Conversation

tejassp-db commented Apr 10, 2026

Uh oh!

github-actions Bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sd-db May 7, 2026

Choose a reason for hiding this comment

Uh oh!

tejassp-db May 7, 2026

Choose a reason for hiding this comment

Uh oh!

sd-db May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db May 7, 2026

Choose a reason for hiding this comment

Uh oh!

tejassp-db May 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

tejassp-db commented May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

tejassp-db commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Apr 10, 2026 •

edited

Loading