Enable deletion vector DML for Delta 4.2 catalog-managed tables

[nartal1]

Contributes to #15956.

Description

Enable GPU DELETE, UPDATE, and MERGE for Delta 4.2 OSS Unity Catalog managed tables with persistent deletion vectors, building on #16022 and #15869.

This change:

• Reuses the generic Delta 4.x deletion-vector DML implementation.
• Adds CPU/GPU parity and GPU-plan coverage for rewrite and persistent-DV modes, including tables with existing deletion vectors.
• Updates the catalog-managed table documentation.

Persistent-DV dynamic partition overwrite remains on CPU, consistent with the existing non-managed Delta limitation.

AI assistance disclosure: Codex assisted with issue implementation and validation. It was reviewed by the author before publication.

Validation

• Unity Catalog 0.6.0 managed-table suite: 29/29 tests passed on both Spark versions.

Performance

Single RTX A5000, local fake-S3, 10M rows: dynamic partition overwrite reached 1.23x; persistent-DV DELETE, UPDATE, and MERGE measured 0.68x, 0.43x, and 0.53x. These local runs are dominated by catalog and commit overhead and do not replace cluster-scale qualification. For context, #15869 reported GPU speedups for the underlying generic DML path on an 8-node A100 cluster. Will do another set of runs on 8-node A100 cluster.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[nartal1]

build

[greptile-apps]

Confidence Score: 4/5

[High risk] Enables GPU acceleration for deletion-vector mutations on catalog-managed tables.

The PR appears safe to merge; the remaining findings concern how conclusively its tests prove the intended execution paths.

Summary

The PR enables Delta 4.2 catalog-managed persistent-DV DELETE, UPDATE, and MERGE by removing the catalog-specific GPU vetoes, and expands parity, failure-retry, and repeated-DV integration coverage.

• Updates user and test documentation to describe the newly accelerated DML modes.
• Three test assertions could more precisely prove the GPU failure path, target scan, and split-file scenario they are intended to cover.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  D[Delta 4.2 DML command] --> M[Common GPU eligibility metadata]
  M --> G[GPU DELETE / UPDATE / MERGE]
  G --> V[Delta DV actions and transaction commit]
  V --> C[Catalog-managed table state]

Loading

_{Reviews (1) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}