Skip to content

feat(db-databricks): add a timeoutMs setting that cancels long-running queries - #3024

Merged
mtoy-googly-moogly merged 3 commits into
malloydata:mainfrom
girishjeswani:feat/db-databricks-timeout
Jul 31, 2026
Merged

feat(db-databricks): add a timeoutMs setting that cancels long-running queries#3024
mtoy-googly-moogly merged 3 commits into
malloydata:mainfrom
girishjeswani:feat/db-databricks-timeout

Conversation

@girishjeswani

@girishjeswani girishjeswani commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

The Databricks connector had no query timeout: every statement was run and its
results fetched with no client-side bound and nothing to cancel a runaway query.
This adds a timeoutMs connection setting that bounds the wait and cancels the
operation when it is exceeded, bringing Databricks in line with the BigQuery and
Snowflake connectors. The same mechanism also honors Malloy's abortSignal, so
a cancelled query cancels the Databricks operation (it was previously ignored).

Approach

All statements run through a single executeRaw chokepoint
(session.executeStatement(...) then operation.fetchAll()), so the wait is
bounded there:

  • Race fetchAll() against a timeoutMs deadline and the caller's
    abortSignal. On either, best-effort operation.cancel() (which cancels the
    statement server-side too) and reject with an actionable error. The operation
    is closed without being awaited on the failure path, so a close() that hangs
    on the same unresponsive connection cannot swallow the timeout.
  • Client-side cancel rather than the SDK's server-side queryTimeout: per the
    @databricks/sql docs, queryTimeout is "effective only with Compute
    clusters" and is ignored on SQL Warehouses, which is what this connector
    targets (the path is a SQL warehouse HTTP path). operation.cancel() works
    for both and bounds the actual client wait.
  • timeoutMs resolves to a positive number or the default (10 min): unset,
    non-numeric, zero, or negative fall back, matching the BigQuery and Snowflake
    connectors.
  • abortSignal is threaded runSQL -> runRawSQL -> executeRaw; an
    already-aborted signal fails fast before a statement is dispatched.

Scope

Databricks-only. This is the third of the timeout follow-ups from
#3010 (BigQuery) and #3022 (Snowflake). The
server-side STATEMENT_TIMEOUT (the SQL Warehouse equivalent of a server-side
cap) is intentionally not set here; the client-side cancel is portable across
warehouses and clusters and already cancels the server statement.

Tests

  • databricks_connection.unit.spec.ts stubs the session/operation and uses fake
    timers to exercise behavior that reading the code cannot verify: a query past
    timeoutMs is cancelled with the actionable error; the timeout still fires
    even if closing the operation hangs; a query that finishes in time is not
    cancelled; an abort mid-flight cancels the operation; and an already-aborted
    signal throws before executing.
  • The existing live integration spec (databricks_connection.spec.ts,
    env-gated) is unaffected.

tsc, eslint, and the unit spec pass.

Checklist

@mtoy-googly-moogly

Copy link
Copy Markdown
Collaborator

Please delete the it.each block that resolves zero / negative / unset timeoutMs to 600000. It asserts that a ternary is a ternary — it can only go red if someone edits the test and the code together — and it pays for that by advancing fake time 600 seconds. Same note as on #3022.

The rest of the file earns its keep. The timeout, the abort, the already-aborted case, and especially "times out even if closing the operation hangs" are all real behavior you can't check by reading the code.

The rule I'd like applied going forward: if a test can't go red for a reason that isn't "someone edited both sides", don't write it. Either write one that could actually fail, or leave the thing untested — a two-line guard with a comment explaining why it's > 0 is sufficient documentation of intent.

For what it's worth, I checked the two @databricks/sql claims against 1.15.0 and both hold: runAsync is hardcoded to true in executeStatement and marked deprecated, and queryTimeout's own type doc says it's Compute-clusters-only. I also confirmed live that operation.cancel() stops an in-flight fetchAll and that the abandoned promise rejects rather than hanging, so the void fetchAll.catch() is doing real work.

…g queries

The Databricks connector had no query timeout. Add a timeoutMs connection setting: executeRaw races operation.fetchAll() against a timeoutMs deadline and the caller's abortSignal, best-effort cancels the operation (which cancels the statement server-side) on either, and rejects with an actionable error. timeoutMs resolves via Number(...) > 0 to the 10-minute default, matching the BigQuery and Snowflake connectors.

Client-side cancel is used rather than the SDK's server-side queryTimeout, which is ignored on SQL Warehouses (the connector's target) and effective only on Compute clusters. abortSignal is threaded runSQL -> runRawSQL -> executeRaw, so a cancelled Malloy query now cancels the Databricks operation; an already-aborted signal fails fast.

Adds a hermetic databricks_connection.unit.spec.ts (stubbed session/operation + fake timers) covering timeout-cancel, non-positive fallback, no-cancel-on-fast-query, abort, and already-aborted.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
Awaiting operation.close() in the cleanup path could hang on the same unresponsive connection that caused the timeout, swallowing the timeout/abort error. Close on the success path (the operation is complete, so it's a quick round-trip) but fire-and-forget on the failure path so the caller settles promptly.

Also drop the deprecated no-op runAsync execute option (the SDK ignores it and always polls in fetchAll). Adds a test that a never-resolving close() still lets the timeout reject.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
Signed-off-by: Girish Jeswani <girish@credibledata.com>
@mtoy-googly-moogly
mtoy-googly-moogly force-pushed the feat/db-databricks-timeout branch from 6fbc323 to 185a3c2 Compare July 31, 2026 13:00
@mtoy-googly-moogly
mtoy-googly-moogly merged commit c14bade into malloydata:main Jul 31, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants