feat: Replace information schema with describe calls by tejassp-db · Pull Request #1432 · databricks/dbt-databricks

tejassp-db · 2026-04-29T06:37:12Z

Description

Replace expensive information_schema queries with a single DESCRIBE TABLE EXTENDED ... AS JSON call for metadata fetching in _describe_relation methods. Gated behind DBRCapability.DESCRIBE_TABLE_EXTENDED_AS_JSON (DBR 17.3+), falls back to existing info_schema queries on older runtimes.

Changes:

IncrementalTableAPI._describe_relation: replaces 4 info_schema queries (PK, FK, non-null constraints, column masks) with AS JSON parsing
MaterializedViewAPI._describe_relation: replaces get_view_description (info_schema.views) with AS JSON parsing
ViewAPI._describe_relation: same as MV
New DatabricksDescribeJsonMetadata parser class with composite PK/FK support
New is_describe_as_json_supported() gating method (checks HMS, foreign table, capability)
New is_foreign_table property on DatabricksRelation
New describe_table_extended_as_json Jinja macro

Testing:

56 unit tests: parser, processor roundtrip, diff validation, edge cases
Capability boundary tests (17.2 vs 17.3)
is_describe_as_json_supported unit tests (UC, HMS, foreign table, no capability)
Jinja macro SQL generation test
Functional tests validated on both code paths:
- SQL warehouse (DBR 17.3+): all 14 constraint/mask tests passed via DESCRIBE AS JSON path
- UC cluster (DBR 16.4 LTS): all 14 tests passed via information_schema fallback path

Checklist

I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

PECOBLR-2546

Add alternate code path for metadata fetching in _describe_relation methods using DESCRIBE TABLE EXTENDED AS JSON (DBR 17.3+). Replaces 4 information_schema queries in IncrementalTableAPI and get_view_description in MaterializedViewAPI/ViewAPI. Falls back to legacy queries when capability is unavailable. PECOBLR-2546

Wrap long docstrings and expressions to satisfy E501, and add the missing return annotation on DatabricksRelation.is_foreign_table for mypy.

github-actions · 2026-04-29T06:52:07Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
dbr_capabilities.py
impl.py					444-445, 1226-1236, 1280-1286, 1336-1337, 1384, 1414, 1492, 1535, 1563, 1603-1606, 1628, 1756, 1800
relation.py					151
Project Total

_{This report was generated by python-coverage-comment-action}

Fix primary key and foreign key constraint parser to handle non-delimited identifiers within backticks.

sd-db · 2026-05-09T07:52:23Z

/integration-test

github-actions · 2026-05-09T07:52:36Z

Integration tests dispatched for PR #1432 by @sd-db. Track progress in the Actions tab.

github-actions · 2026-05-09T08:39:44Z

Integration results for PR #1432 — UC cluster ✅ success · SQL warehouse ❌ failure · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

sd-db · 2026-05-09T11:46:31Z

On the failure,

FAILED tests/functional/adapter/python_model/test_python_model.py::TestSpecifyingHttpPath::test_singular_tests - AssertionError: dbt exit state did not match expected

this is known flaky test so can be ignored (I will look to solve for this separately)

sd-db

PR Review 1/x. Added a few comments on structure

sd-db · 2026-05-12T06:46:45Z

                f"Current connection does not meet this requirement."
            )

+    def is_describe_as_json_supported(self, relation: DatabricksRelation) -> bool:


Again I feel we should rename this, this check is specifically for using DECRIBE TABLE EXTENDED AS JSON for getting metadata we should mark it as such as there can be other places where we are using DESCRIBE... AS JSON that don't go through this check. Some suggestions is_describe_as_json_relation_metadata_supported() or maybe can_use_describe_json_for_relation_metadata()

This method specifically is to check whether "DESCRIBE TABLE EXTENDED AS JSON" is supported. The method should not take responsibility of how the call response is consumed.

The reason for raising this was actually for the behaviour flag check --> bool(self.behavior.use_describe_as_json_for_relation_metadata.no_warn); since this is dependent on the flag it is a more specific version/flavour, we can look to either move this out or rename

sd-db · 2026-05-12T08:59:58Z

+        )
+
+    @classmethod
+    def parse_column_masks(cls, json_metadata: dict[str, Any]) -> "Table":


For all the parse methods we are heavily relying on the structure of the reponse from the DESCRIBE...JSON call. While this works. I feel it might have been much better to define our out class/data-model and load the results from the DESCRIBE...JSON call into the data-model. Basically a DAO layer. Here we are at the mercy of the server and whatever validations we might/might not add on correctness and there can still be edge-cases.

…data

…ibe-calls

sd-db

Overall looks good, made some minor comments

…ibe-calls

tejassp-db · 2026-05-13T16:08:24Z

/integration-test

github-actions · 2026-05-13T16:09:37Z

Integration tests dispatched for PR #1432 by @tejassp-db. Track progress in the Actions tab.

github-actions · 2026-05-13T16:57:56Z

Integration results for PR #1432 — UC cluster ✅ success · SQL warehouse ✅ success · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

tejassp-db · 2026-05-14T06:30:06Z

/integration-test

github-actions · 2026-05-14T06:30:20Z

Integration tests dispatched for PR #1432 by @tejassp-db. Track progress in the Actions tab.

tejassp-db · 2026-05-14T07:21:33Z

/integration-test

github-actions · 2026-05-14T07:21:46Z

Integration tests dispatched for PR #1432 by @tejassp-db. Track progress in the Actions tab.

github-actions · 2026-05-14T07:23:09Z

Integration results for PR #1432 — UC cluster ❌ cancelled · SQL warehouse ❌ failure · All-purpose cluster ✅ success · Shard coverage ❌ failure

Run details.

github-actions · 2026-05-14T08:20:19Z

Integration results for PR #1432 — UC cluster ✅ success · SQL warehouse ❌ failure · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

tejassp-db · 2026-05-14T08:22:19Z

Integration results for PR #1432 — UC cluster ✅ success · SQL warehouse ❌ failure · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

Flaky integration test, this code path does not touch those tests or their code path.

tejassp-db added 4 commits April 29, 2026 11:57

Parse json output

a5ac268

Refactor code and add docs.

9664cd5

chore: fix ruff line-length and mypy lint errors

1fe7103

Wrap long docstrings and expressions to satisfy E501, and add the missing return annotation on DatabricksRelation.is_foreign_table for mypy.

tejassp-db self-assigned this Apr 29, 2026

tejassp-db requested review from benc-db, jprakash-db and sd-db as code owners April 29, 2026 06:37

tejassp-db had a problem deploying to azure-prod April 29, 2026 06:37 — with GitHub Actions Error

tejassp-db had a problem deploying to azure-prod April 29, 2026 06:37 — with GitHub Actions Failure

tejassp-db had a problem deploying to azure-prod April 29, 2026 06:37 — with GitHub Actions Error

Parse row filter from describe json output

ce1e4c9

tejassp-db had a problem deploying to azure-prod April 29, 2026 08:04 — with GitHub Actions Error

Format code

9eb181b

tejassp-db had a problem deploying to azure-prod April 29, 2026 08:09 — with GitHub Actions Failure

tejassp-db had a problem deploying to azure-prod April 29, 2026 08:09 — with GitHub Actions Error

Fix mypy errors

53770b4

tejassp-db temporarily deployed to azure-prod April 29, 2026 09:26 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 29, 2026 09:26 — with GitHub Actions Failure

tejassp-db mentioned this pull request Apr 29, 2026

feat: Replace information schema with describe calls #1415

Closed

3 tasks

tejassp-db added 2 commits May 4, 2026 10:59

Lazy import agate to improve startup time

14a6fc0

Fix constraint parser.

fc2b972

Fix primary key and foreign key constraint parser to handle non-delimited identifiers within backticks.

tejassp-db had a problem deploying to azure-prod May 4, 2026 09:23 — with GitHub Actions Failure

sd-db reviewed May 12, 2026

View reviewed changes

Comment thread dbt/adapters/databricks/impl.py

tejassp-db added 5 commits May 13, 2026 11:48

More informative error messages on parse failures

d33d94d

More useful error message when parsing json output

a47fe80

Rename use_describe_as_json -> use_describe_as_json_for_relation_meta…

fb46528

…data

Update CHANGELOG.md

eefacb0

Merge branch '1.12.latest' into replace-information-schema-with-descr…

2a60835

…ibe-calls

sd-db reviewed May 13, 2026

View reviewed changes

Merge branch '1.12.latest' into replace-information-schema-with-descr…

7456ae2

…ibe-calls

tejassp-db added 2 commits May 14, 2026 09:11

Format code

f99208e

Reword changelog

bfb55fa

sd-db approved these changes May 14, 2026

View reviewed changes

Remove unused tests

7382b85

tejassp-db mentioned this pull request May 14, 2026

docs(databricks): document use_describe_as_json_for_relation_metadata flag dbt-labs/docs.getdbt.com#9176

Open

3 tasks

tejassp-db merged commit 3caad33 into 1.12.latest May 14, 2026
7 checks passed

tejassp-db deleted the replace-information-schema-with-describe-calls branch May 14, 2026 08:22

Conversation

tejassp-db commented Apr 29, 2026

Description

Checklist

Uh oh!

github-actions Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

sd-db commented May 9, 2026

Uh oh!

github-actions Bot commented May 9, 2026

Uh oh!

github-actions Bot commented May 9, 2026

Uh oh!

sd-db commented May 9, 2026

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db May 12, 2026

Choose a reason for hiding this comment

Uh oh!

tejassp-db May 13, 2026

Choose a reason for hiding this comment

Uh oh!

sd-db May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sd-db May 12, 2026

Choose a reason for hiding this comment

Uh oh!

tejassp-db May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

tejassp-db commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

tejassp-db commented May 14, 2026

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

tejassp-db commented May 14, 2026

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

tejassp-db commented May 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Apr 29, 2026 •

edited

Loading