You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b45a6f2
Browse filesBrowse the repository at this point in the historyBrowse files
Two parts:
1. **Strip "Phase 0/N" jargon** from the data-lakehouse tutorial. Phase
numbers are an internal codebase organisation detail (managed-
initial-export!'s body is structured as Phase 0..5) and mean
nothing to a reader. Rephrase three places:
- EXTERNAL_USE_LOCATION privilege rows (managed-zerobus + managed-
sql tabs) — describe what the grant lets the module do, not which
phase calls it.
- Troubleshooting #10 (DELTA_CREATE_TABLE_SCHEME_MISMATCH) — refer
to "auto-cleanup" instead of "Phase 0 auto-cleanup".
- Troubleshooting #11 (staging-s3-cleanup-skipped log) — same.
2. **Document the same S3-staging-cleanup behaviour in
$viewdefinition-export reference.** The $vd-export code path uses
the identical `ensure-staging-table!` machinery (S3 files outlive
UC `DROP TABLE`; cleanup runs at the start of the next call), but
the reference said nothing about it. Add a "Staging cleanup on
S3" subsection inside "How it works (kind=data-lakehouse)"
covering the two practical consequences for typical cron-driven
use: prefix rotation defeats auto-cleanup, and the
`EXTERNAL_USE_LOCATION` grant is what enables it. Link out to
the data-lakehouse tutorial's setup for the grant table itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/modules/sql-on-fhir/operation-viewdefinition-export.md
+9Lines changed: 9 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -179,6 +179,15 @@ With `chunkCount = N` (default `N=1`), for each chunk the module creates a per-c
179
179
The `MERGE` is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must select an `id` column.
180
180
{% endhint %}
181
181
182
+
### Staging cleanup on S3
183
+
184
+
After the `MERGE` succeeds the module drops the per-chunk staging tables (Unity Catalog metadata) but does **not** delete the underlying Parquet and `_delta_log/` files on S3. They are recursive-deleted at the **start** of the next call that reuses the same `stagingTablePath`, before the new staging tables are created.
185
+
186
+
Two consequences worth knowing:
187
+
188
+
-**If you rotate `stagingTablePath` between runs** (for example, a date-stamped prefix), each prior prefix is left behind in your bucket and the auto-cleanup never fires for it. Either reuse one prefix across runs, or `aws s3 rm --recursive` the old ones yourself.
189
+
-**The auto-cleanup uses Unity Catalog `temporary-path-credentials`**, so the principal needs `EXTERNAL_USE_LOCATION` on the External Location that covers `stagingTablePath`. Without that grant the cleanup is skipped and a `staging-s3-cleanup-skipped` event is logged; the export itself still runs. The same grant table is documented in the [Data Lakehouse tutorial setup](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#setup).
190
+
182
191
## _since (incremental export)
183
192
184
193
`_since` turns the export into an incremental one: instead of materializing the entire view, the source query is filtered by the view's timestamp column.
`PLACEHOLDER` is patched in by the Storage Credential step below.
303
303
304
-
Create the role with a temporary trust policy (the `PLACEHOLDER` external-id is replaced once Unity Catalog gives us a real one):
304
+
Create the role with a temporary trust policy:
305
305
306
306
```sh
307
307
aws iam create-role --role-name "$IAM_ROLE_NAME" \
@@ -484,7 +484,7 @@ Grant only the set matching your `writeMode`. The SP runs the module at request
484
484
|`SELECT`, `MODIFY`| the target table |`DESCRIBE` + initial-bulk `MERGE INTO`|
485
485
|`USE_SCHEMA`, `EXTERNAL_USE_SCHEMA`, `CREATE_TABLE`| the staging schema | resolve the sibling schema, vend STS for the staging table, and let the sender register it (initial-export only) |
486
486
|`READ_FILES`, `WRITE_FILES`, `CREATE_EXTERNAL_TABLE`| the External Location | write bulk Parquet via vended STS (initial-export only) |
487
-
|`EXTERNAL_USE_LOCATION`| the External Location | vend path-credentials for Phase 0 staging cleanup (recursive-deletes orphan Parquet/`_delta_log` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works)|
487
+
|`EXTERNAL_USE_LOCATION`| the External Location | vend path-credentials so the module can recursive-delete orphan Parquet / `_delta_log/` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works |
488
488
|`CAN_USE`| the SQL warehouse | bootstrap schema-sync statements + initial-bulk `MERGE` (no warehouse traffic during live writes) — already granted in the SP/warehouse step |
489
489
490
490
Catalog-level navigate grant:
@@ -536,7 +536,7 @@ Identical privilege set to `managed-zerobus` — the SQL warehouse is hit on eve
|`USE_SCHEMA`, `EXTERNAL_USE_SCHEMA`, `CREATE_TABLE`| the staging schema | resolve the sibling schema, vend STS for the staging table, and let the sender register it (initial-export only) |
538
538
|`READ_FILES`, `WRITE_FILES`, `CREATE_EXTERNAL_TABLE`| the External Location | write bulk Parquet via vended STS (initial-export only) |
539
-
|`EXTERNAL_USE_LOCATION`| the External Location | vend path-credentials for Phase 0 staging cleanup (recursive-deletes orphan Parquet/`_delta_log` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works)|
539
+
|`EXTERNAL_USE_LOCATION`| the External Location | vend path-credentials so the module can recursive-delete orphan Parquet / `_delta_log/` left from prior runs before the next `CREATE TABLE` — without this grant cleanup is skipped and files accumulate, but init-export still works |
540
540
|`CAN_USE`| the SQL warehouse | every-batch INSERT + bootstrap + initial-bulk — already granted in the SP/warehouse step |
541
541
542
542
Catalog-level navigate grant:
@@ -1083,8 +1083,8 @@ The module only ADDS columns automatically. Column drops, renames, or narrowing
1083
1083
7.**Duplicate rows after recreating destination** — deleting and recreating a destination triggers initial export again. Set `skipInitialExport: true` when recreating a destination that already has its data exported.
1084
1084
8.**`LOCATION_OVERLAP` during initial export** — `stagingTablePath` either equals the staging schema's `storage_root` (which UC treats as the schema's own managed location) or doesn't sit under your External Location. Set it to a sub-prefix of the External Location, e.g. `s3://<bucket>/staging/patient_flat/`, not the External Location root itself.
1085
1085
9.**`Unsupported table kind. Tables created in default storage are not supported` (Zerobus error 4024)** — the catalog backing your target table was created without `--storage-root`, so Unity Catalog placed it in the workspace's default-storage prefix. `managed-zerobus` refuses to write into default storage. Recreate the catalog with `databricks catalogs create <name> --storage-root s3://<bucket>/managed/` pointing inside a registered External Location (see [Create the catalog and target schema](#create-the-catalog-and-target-schema) in the usage example).
1086
-
10.**`DELTA_CREATE_TABLE_SCHEME_MISMATCH` on initial export retry** — your `stagingTablePath` contains a `_delta_log/` from a previous initial-export run, and the new run has a different ViewDefinition schema (e.g. you added `ts`/`cts` columns). The module drops the UC staging table metadata on cleanup but does NOT delete S3 files, so the old `_delta_log/` survives and conflicts. Fix: either point `stagingTablePath` at a fresh sub-prefix (e.g. append a nonce: `s3://<bucket>/staging/<table>-v2/`), or manually `aws s3 rm --recursive` the old prefix. Granting `EXTERNAL_USE_LOCATION` on the External Location (see item 11) enables the module's Phase 0 auto-cleanup so this stops biting on column-add re-runs.
1087
-
11.**`staging-s3-cleanup-skipped` log with `:reason :no-external-use-location-grant`** — the SP lacks `EXTERNAL_USE_LOCATION` on the External Location covering the staging path. Phase 0 cleanup (auto-recursive-delete of stale staging Parquet/`_delta_log` between runs) is disabled and files accumulate. Init-export itself keeps working. Fix by granting the privilege:
1086
+
10.**`DELTA_CREATE_TABLE_SCHEME_MISMATCH` on initial export retry** — your `stagingTablePath` contains a `_delta_log/` from a previous initial-export run, and the new run has a different ViewDefinition schema (e.g. you added `ts`/`cts` columns). The module drops the UC staging table metadata on cleanup but does NOT delete S3 files, so the old `_delta_log/` survives and conflicts. Fix: either point `stagingTablePath` at a fresh sub-prefix (e.g. append a nonce: `s3://<bucket>/staging/<table>-v2/`), or manually `aws s3 rm --recursive` the old prefix. Granting `EXTERNAL_USE_LOCATION` on the External Location (see item 11) enables the module's auto-cleanup so this stops biting on column-add re-runs.
1087
+
11.**`staging-s3-cleanup-skipped` log with `:reason :no-external-use-location-grant`** — the SP lacks `EXTERNAL_USE_LOCATION` on the External Location covering the staging path. Auto-cleanup of stale staging Parquet / `_delta_log/` between runs is disabled and files accumulate. Init-export itself keeps working. Fix by granting the privilege:
0 commit comments