Skip to content

Commit b45a6f2

Browse files
spicyfalafelclaude
andcommitted
docs: strip "Phase 0/N" jargon + document S3 staging cleanup in $vd-export
Two parts: 1. **Strip "Phase 0/N" jargon** from the data-lakehouse tutorial. Phase numbers are an internal codebase organisation detail (managed- initial-export!'s body is structured as Phase 0..5) and mean nothing to a reader. Rephrase three places: - EXTERNAL_USE_LOCATION privilege rows (managed-zerobus + managed- sql tabs) — describe what the grant lets the module do, not which phase calls it. - Troubleshooting #10 (DELTA_CREATE_TABLE_SCHEME_MISMATCH) — refer to "auto-cleanup" instead of "Phase 0 auto-cleanup". - Troubleshooting #11 (staging-s3-cleanup-skipped log) — same. 2. **Document the same S3-staging-cleanup behaviour in $viewdefinition-export reference.** The $vd-export code path uses the identical `ensure-staging-table!` machinery (S3 files outlive UC `DROP TABLE`; cleanup runs at the start of the next call), but the reference said nothing about it. Add a "Staging cleanup on S3" subsection inside "How it works (kind=data-lakehouse)" covering the two practical consequences for typical cron-driven use: prefix rotation defeats auto-cleanup, and the `EXTERNAL_USE_LOCATION` grant is what enables it. Link out to the data-lakehouse tutorial's setup for the grant table itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent cf7125d commit b45a6f2

2 files changed

Lines changed: 14 additions & 5 deletions

File tree

‎docs/modules/sql-on-fhir/operation-viewdefinition-export.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,15 @@ With `chunkCount = N` (default `N=1`), for each chunk the module creates a per-c
179179
The `MERGE` is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must select an `id` column.
180180
{% endhint %}
181181

182+
### Staging cleanup on S3
183+
184+
After the `MERGE` succeeds the module drops the per-chunk staging tables (Unity Catalog metadata) but does **not** delete the underlying Parquet and `_delta_log/` files on S3. They are recursive-deleted at the **start** of the next call that reuses the same `stagingTablePath`, before the new staging tables are created.
185+
186+
Two consequences worth knowing:
187+
188+
- **If you rotate `stagingTablePath` between runs** (for example, a date-stamped prefix), each prior prefix is left behind in your bucket and the auto-cleanup never fires for it. Either reuse one prefix across runs, or `aws s3 rm --recursive` the old ones yourself.
189+
- **The auto-cleanup uses Unity Catalog `temporary-path-credentials`**, so the principal needs `EXTERNAL_USE_LOCATION` on the External Location that covers `stagingTablePath`. Without that grant the cleanup is skipped and a `staging-s3-cleanup-skipped` event is logged; the export itself still runs. The same grant table is documented in the [Data Lakehouse tutorial setup](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#setup).
190+
182191
## _since (incremental export)
183192

184193
`_since` turns the export into an incremental one: instead of materializing the entire view, the source query is filtered by the view's timestamp column.

‎docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -301,7 +301,7 @@ aws s3api create-bucket --bucket "$STAGING_BUCKET" --region "$AWS_REGION"
301301

302302
`PLACEHOLDER` is patched in by the Storage Credential step below.
303303

304-
Create the role with a temporary trust policy (the `PLACEHOLDER` external-id is replaced once Unity Catalog gives us a real one):
304+
Create the role with a temporary trust policy:
305305

306306
```sh
307307
aws iam create-role --role-name "$IAM_ROLE_NAME" \
@@ -484,7 +484,7 @@ Grant only the set matching your `writeMode`. The SP runs the module at request
484484
| `SELECT`, `MODIFY` | the target table | `DESCRIBE` + initial-bulk `MERGE INTO` |
485485
| `USE_SCHEMA`, `EXTERNAL_USE_SCHEMA`, `CREATE_TABLE` | the staging schema | resolve the sibling schema, vend STS for the staging table, and let the sender register it (initial-export only) |
486486
| `READ_FILES`, `WRITE_FILES`, `CREATE_EXTERNAL_TABLE` | the External Location | write bulk Parquet via vended STS (initial-export only) |
487-
| `EXTERNAL_USE_LOCATION` | the External Location | vend path-credentials for Phase 0 staging cleanup (recursive-deletes orphan Parquet/`_delta_log` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works) |
487+
| `EXTERNAL_USE_LOCATION` | the External Location | vend path-credentials so the module can recursive-delete orphan Parquet / `_delta_log/` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works |
488488
| `CAN_USE` | the SQL warehouse | bootstrap schema-sync statements + initial-bulk `MERGE` (no warehouse traffic during live writes) — already granted in the SP/warehouse step |
489489

490490
Catalog-level navigate grant:
@@ -536,7 +536,7 @@ Identical privilege set to `managed-zerobus` — the SQL warehouse is hit on eve
536536
| `SELECT`, `MODIFY` | the target table | `DESCRIBE` + every-batch `INSERT` + initial-bulk `MERGE INTO` |
537537
| `USE_SCHEMA`, `EXTERNAL_USE_SCHEMA`, `CREATE_TABLE` | the staging schema | resolve the sibling schema, vend STS for the staging table, and let the sender register it (initial-export only) |
538538
| `READ_FILES`, `WRITE_FILES`, `CREATE_EXTERNAL_TABLE` | the External Location | write bulk Parquet via vended STS (initial-export only) |
539-
| `EXTERNAL_USE_LOCATION` | the External Location | vend path-credentials for Phase 0 staging cleanup (recursive-deletes orphan Parquet/`_delta_log` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works) |
539+
| `EXTERNAL_USE_LOCATION` | the External Location | vend path-credentials so the module can recursive-delete orphan Parquet / `_delta_log/` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works |
540540
| `CAN_USE` | the SQL warehouse | every-batch INSERT + bootstrap + initial-bulk — already granted in the SP/warehouse step |
541541

542542
Catalog-level navigate grant:
@@ -1083,8 +1083,8 @@ The module only ADDS columns automatically. Column drops, renames, or narrowing
10831083
7. **Duplicate rows after recreating destination** — deleting and recreating a destination triggers initial export again. Set `skipInitialExport: true` when recreating a destination that already has its data exported.
10841084
8. **`LOCATION_OVERLAP` during initial export** — `stagingTablePath` either equals the staging schema's `storage_root` (which UC treats as the schema's own managed location) or doesn't sit under your External Location. Set it to a sub-prefix of the External Location, e.g. `s3://<bucket>/staging/patient_flat/`, not the External Location root itself.
10851085
9. **`Unsupported table kind. Tables created in default storage are not supported` (Zerobus error 4024)** — the catalog backing your target table was created without `--storage-root`, so Unity Catalog placed it in the workspace's default-storage prefix. `managed-zerobus` refuses to write into default storage. Recreate the catalog with `databricks catalogs create <name> --storage-root s3://<bucket>/managed/` pointing inside a registered External Location (see [Create the catalog and target schema](#create-the-catalog-and-target-schema) in the usage example).
1086-
10. **`DELTA_CREATE_TABLE_SCHEME_MISMATCH` on initial export retry** — your `stagingTablePath` contains a `_delta_log/` from a previous initial-export run, and the new run has a different ViewDefinition schema (e.g. you added `ts`/`cts` columns). The module drops the UC staging table metadata on cleanup but does NOT delete S3 files, so the old `_delta_log/` survives and conflicts. Fix: either point `stagingTablePath` at a fresh sub-prefix (e.g. append a nonce: `s3://<bucket>/staging/<table>-v2/`), or manually `aws s3 rm --recursive` the old prefix. Granting `EXTERNAL_USE_LOCATION` on the External Location (see item 11) enables the module's Phase 0 auto-cleanup so this stops biting on column-add re-runs.
1087-
11. **`staging-s3-cleanup-skipped` log with `:reason :no-external-use-location-grant`** — the SP lacks `EXTERNAL_USE_LOCATION` on the External Location covering the staging path. Phase 0 cleanup (auto-recursive-delete of stale staging Parquet/`_delta_log` between runs) is disabled and files accumulate. Init-export itself keeps working. Fix by granting the privilege:
1086+
10. **`DELTA_CREATE_TABLE_SCHEME_MISMATCH` on initial export retry** — your `stagingTablePath` contains a `_delta_log/` from a previous initial-export run, and the new run has a different ViewDefinition schema (e.g. you added `ts`/`cts` columns). The module drops the UC staging table metadata on cleanup but does NOT delete S3 files, so the old `_delta_log/` survives and conflicts. Fix: either point `stagingTablePath` at a fresh sub-prefix (e.g. append a nonce: `s3://<bucket>/staging/<table>-v2/`), or manually `aws s3 rm --recursive` the old prefix. Granting `EXTERNAL_USE_LOCATION` on the External Location (see item 11) enables the module's auto-cleanup so this stops biting on column-add re-runs.
1087+
11. **`staging-s3-cleanup-skipped` log with `:reason :no-external-use-location-grant`** — the SP lacks `EXTERNAL_USE_LOCATION` on the External Location covering the staging path. Auto-cleanup of stale staging Parquet / `_delta_log/` between runs is disabled and files accumulate. Init-export itself keeps working. Fix by granting the privilege:
10881088
```sh
10891089
databricks grants update external-location "$EXTERNAL_LOCATION_NAME" --json '{
10901090
"changes":[{"principal":"'"$BOX_DATABRICKS_DATA_LAKEHOUSE_CLIENT_ID"'","add":["EXTERNAL_USE_LOCATION"]}]}'

0 commit comments

Comments
 (0)