Repository navigation
WIP: Data Lakehouse AidboxTopicDestination tutorial - #15
Merged
Merged
Conversation
Covers both writeModes (managed → Databricks SQL warehouse INSERT; external-direct → direct Delta commit on customer's bucket). Includes a concept primer (Databricks / lakehouse / Delta / managed vs external), an auth-flow sequence diagram, side-by-side privilege grants in tabs, and a stepped Patient export walkthrough. Companion to the module PR HealthSamurai/topic-destination-deltalake#1 which ships in Aidbox 2605. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…etails - Overview / Auth flow / Compaction / Cost notes — break dense paragraphs into per-mode bullet lists so the differences read at a glance. - Strip user-irrelevant implementation details (Kernel, Hadoop S3A, TransactionBuilder, library URLs). Replace with user-facing concepts (the module writes Parquet + Delta commits, stable transaction id, etc.). Keep only the Delta protocol spec link in Related Documentation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop the version-compat warning block (table + "older versions don't support this module"). Keep just the simple "available from 2605" hint. - Drop the "four-minute primer" sentence — let the section header speak. - Drop the "Common confusion: External Location vs external table" hint block — over-explained for the audience. - Background: don't mention Spark or paste a REST URL into the SQL warehouse explanation; describe it functionally. Reword the "two-things bundled" closing line so it doesn't reference `managed` mode before it's introduced. - Overview: split the single mode-dispatch mermaid into two diagrams, one per writeMode. Easier to read; matches the user's mental model. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Overview: drop the "Mode summary" heading; put the explanation bullets
immediately under each writeMode diagram with a one-line lead-in.
- Replace the "Hot path" jargon — describe what actually happens per batch.
- Promote the Delivery-guarantee {% hint %} block into a real top-level
section. Remove the duplicate `## Idempotency` section that said the
same thing in different words. Soft Deletes now references the single
dedup query example instead of repeating it. Old "Delivery Guarantees
and Retry" section becomes plain "Retry behavior" (the guarantees half
is hoisted up).
- Required Databricks privileges: split into "Common to both modes" +
per-mode tabs so the shared grants live in one place. external-direct
tab now spells out the EXTERNAL LOCATION grants too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GitHub's mermaid renderer rejects `<` (treated as arrow syntax start) and
unescaped HTML like `<br/>` inside sequenceDiagram lines. Replace with
plain prose ("under 5 min remain", "about 1h TTL") and drop the inline
endpoint paths from arrow labels — they were implementation detail anyway.
Also drop the PAT-not-supported note: user-facing docs shouldn't argue
against a feature we don't ship.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The staging table is what makes managed-mode initial bulk export work at all — Databricks-managed tables refuse direct writes, so the module writes Parquet to a temporary external Delta table, asks the SQL warehouse to copy from it into the managed target, and drops it. The previous version mentioned this in prose but never drew it. - Add a numbered-flow mermaid in "How it works — managed mode" showing the staging relay (sof.view → staging → INSERT SELECT → DROP). - Add a one-step diagram in "How it works — external-direct mode" so both paths are visually symmetric. - Cross-link from the Overview managed bullets to the staging section so the staging concept lands before the configuration parameters. Avoids GitHub mermaid issues by using `<br/>` only inside node labels (supported in flowchart, unlike sequenceDiagram). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rst language - Add a top-level architecture diagram in Overview showing how Aidbox / PostgreSQL / the module / Databricks / cloud storage fit together — readers want this before per-mode flow diagrams. - Hoist "append-only output" (CREATE / UPDATE / DELETE → new rows with is_deleted flag) to Overview. Used to be buried in the Data Transformation section near the bottom. The Soft Deletes subsection there now just cross-links. - Collapse the artificial split between "Data lakehouse" and "Delta Lake" subsections — Delta is what gives the lake ACID, so introducing them as if they were two separate concepts was misleading. - Replace customer / we / us terminology with Aidbox / User / the module. The doc is for the User; talking about them in third person is the wrong voice. - Choosing-modes table: drop the confusing "Target" row (both modes target Delta tables); add "Table type" + "Storage backends" rows that actually tell the reader something different per mode. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round of correctness + UX polish: - Costs: we don't actually know a 2X-Small Serverless warehouse costs $2k/month, and "Free via Predictive Optimization" conflated "you don't think about it" with "you don't pay". Replace specific dollar numbers with "see Databricks pricing"; rephrase Predictive Optimization rows to talk about who runs OPTIMIZE/VACUUM, not money. - Drop the "Best for" row from Choosing — the recommendation was a guess. Drop the "Choosing batch parameters" hint for the same reason. - Architecture diagram: User is a rectangle, not a circle; module label trimmed; remove duplicate trailing paragraph. - Background → SQL warehouse: don't tell readers they "send statements over REST" — they don't; describe it functionally and note the module drives it programmatically. - External vs managed tables table: add explicit row "Can Aidbox write directly?" so the consequence for the module is visible alongside the conceptual difference. - Initial export row in Choosing now spells out staging → INSERT SELECT → drop, with link to the staging diagram. - Configuration → tabs (per-mode), with common parameters duplicated inside each tab so readers see one self-contained config table. - Before you begin: drop the service-principal prerequisite — it gets created in the Authentication setup section, not before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Larger pass driven by review: Structure: - Move Authentication + Required Privileges above Configuration — Authentication is conceptual; Configuration is the User filling in values, which only makes sense after the concepts are in place. - Add a "In a hurry? Jump to Usage Example" link at the top so readers who don't need the background can jump straight to setup. - Drop the Key Features bullet list — pure filler; each bullet was already explained in its own section. - Add numbered annotations under the high-level architecture diagram explaining what each arrow means + a forward link to Initial Export. Accuracy: - Costs: drop dollar figures (we don't know them); frame the section as "where the costs come from" with links to vendor pricing. - Initial Export Retry: drop the misleading "capped at 30s" — with 3 attempts the actual max delay is 4s. - Storage backends listed for external-direct: drop MinIO + Garage from user-facing copy. They're test-only S3-compatible emulators; doc readers shouldn't think we ship a product against them. - "writes Parquet + commits" → "writes the Delta files (Parquet + a transaction-log commit)" — Delta is the umbrella; Parquet alone isn't accurate. - Bucket scheme example shows s3:// / gs:// / abfss:// instead of just s3://. Other: - Move the dedup SQL out of Delivery guarantees (was dense code in a conceptual section) and into Soft Deletes and Updates where it has always belonged. Delivery guarantees now describes the dedup pattern in words + cross-links. Verified against module source: retry counts, backoff math, error codes in Troubleshooting all match the implementation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… sections
Larger restructure addressing four review items:
1) Drop the standalone "Required Databricks privileges" section + the
"Setting up a service principal" / "Storing the secret in Vault"
subsections inside Authentication. All the actual setup steps now
live in one place — Usage Example Step 4 (a-g):
a-c: catalog / schema / table / warehouse
d: create the service principal
e: GRANT SQL (common + per-mode)
f: (optional) External Location for managed-mode initial export
g: (optional, prod) wire up the SP secret through vault
Authentication is now purely conceptual; readers don't have to
triangulate between three sections to figure out the setup order.
2) Merge Append-only output / Delivery guarantees / Soft Deletes-and-dedup
into a single "Output semantics" section right after Overview:
- Append-only (Create/Update/Delete → new row)
- At-least-once delivery + per-mode dedup story
- One window-function dedup query (was duplicated across two spots)
The "Soft Deletes and Updates" subsection in Data Transformation is
now a single cross-link.
3) Slim the Choosing-modes table from 7 rows to 3 essential ones:
Table type / Who runs maintenance / Databricks compute cost surface.
The rest (schema drift handling, initial-export path, storage
backends) moves to bullets below the table.
4) Configuration tables: split per-mode tab into "Required" + collapsible
"Advanced parameters" details block (and "Authentication parameters"
for external-direct, since auth there is conditional). Required-first
makes the surface area scannable.
Also drop the "Local Testing with OSS Unity Catalog" section — it
referenced the (private) module repo's docker-compose.yaml from this
public doc, which couldn't help readers. Local-test setup is module-
developer info, not user-facing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Those two bullets repeated what the Overview section's per-mode diagrams + bullets already say, and what the "Can Aidbox write directly?" row in the table above them encodes. Replace with a one-line bridge to the Overview. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reflects the module change: initial bulk materialization is now an
idempotent MERGE keyed on the resource `id` natural key, not a plain
INSERT SELECT. Sequence diagrams updated; the staging-flow narrative
gains a {% hint %} block explaining the idempotency rationale and the
implicit ViewDefinition contract (must have `id`).
Output semantics also updated: managed mode is now half-idempotent —
initial bulk safe on replay, hot path still at-least-once and still
dedupped on read.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…abricks docs Adds the new default writeMode `managed-zerobus` (Databricks Zerobus gRPC streaming-ingest) alongside the renamed `managed-sql` and unchanged `external-direct`. Three new params (`databricksWorkspaceId`, `databricksRegion`, plus `writeMode`) get configuration entries + worked examples; the Step 5 example switches from managed-sql to managed-zerobus; `managed-sql` becomes the first "Alternative" section, then external-direct. Fact-checked against the official Databricks docs and corrected: - Liquid Clustering on managed tables: opt-in per table (previously "Default"; the official docs say automatic liquid clustering is opt-in and depends on Predictive Optimization). - Predictive Optimization: managed tables only, billed against Jobs Serverless SKU (was "Automatic — Databricks runs ..."; the docs confirm it's not free and the external-tables row no longer claims "Not applicable" without the explicit-restriction citation). - Zerobus billing: pay-per-batch against Jobs Serverless SKU (was "no compute charge", which over-promised — Zerobus has its own bill). - Zerobus endpoint URL: includes `https://` scheme per the SDK README, not a bare host. - Managed-table write restriction: reframed as "sanctioned write paths from outside Databricks compute" (Zerobus + warehouse INSERT) vs "STS credential vending refused for managed targets" — the original "Refused. UC won't hand out write creds" line was correct for direct Hadoop FS writes but missed that Zerobus is also a managed-table write path. - Managed-mode auto-heal: applies to both managed-sql AND managed-zerobus, not just "managed".
- Background → Databricks: list Zerobus as a third Databricks component
in line with the other two; describe it as a push-based ingest API
per the official Databricks docs ("high-throughput, low-overhead row
ingestion", "per-stream ordering"), without name-dropping our internal
writeMode values or the cost framing — that's downstream concern.
- Background gets a one-line clarification that "Data Lakehouse" is the
destination category name (generic, not a Databricks-bound product).
- `writeMode: managed-zerobus` card: lead with what actually makes
Zerobus useful — purpose-built ingest pipe, no SQL parsing/planning
per batch, long-lived stream, server-side offset tracking. Cost
framing moved to a one-line link to the existing Cost section.
- Drop the loud "$2k/month 2X-Small" comparison everywhere (was an
overclaim).
- At-least-once bullets reordered: managed-zerobus, managed-sql,
external-direct (matches mode-card order elsewhere).
- "Recovering the latest state on read" renamed to make the read-side
intent obvious ("Reading current state out of the append-only
history") and the opening explains why the projection is needed.
- Drop the "Default is managed-zerobus" info hint (redundant with the
mode cards above).
- Drop the "This is a partial Deployment manifest" / "Updating the
module" sections — module deployment docs aren't this tutorial's job.
- Drop the Zerobus endpoint how-to-find-your-workspace-id info hint;
the parameter description in the config tab is enough.
Overview / Background:
- "using ViewDefinitions and SQL-on-FHIR technology" → "(SQL-on-FHIR)"
- drop the "High-level architecture" subheading; the diagram speaks for itself
- diagram now labels all three modes' arrows (managed-zerobus gRPC, managed-sql SQL INSERT, external-direct Delta write) instead of one generic "bearer-authenticated calls" line
- drop the redundant "ships as a JAR / no separate process / no state of its own" paragraph
- "What the arrows in the diagram mean, in order:" → "The flow:"
- "the module also performs an initial export" softened to "may also perform" with a link to the section that covers when it runs vs is skipped
- write-modes intro now links forward to the Configuration section
Output semantics:
- Append-only example replaced with a worked four-row table (create / update / update / delete on the same id)
- soften "warehouse must stay warm — you trade compute hours for governance" framing on the managed-sql card; the warm-warehouse implication is already in the Cost section
- expand the external-direct cleanup bullet ("In return, the User runs OPTIMIZE / VACUUM on the table") to explain why and forward-link to Compaction
- staging-table phrasing now disambiguates from Databricks' "temporary tables" feature with a link
Authentication:
- moved out from between Installation and Configuration to right after "Choosing between the three modes" — it's a concept (how the module reaches Databricks), conceptually it belongs before deployment, not after
Schema naming:
- rename example UC schema `aidbox_export.healthcare` → `aidbox_export.fhir`; "healthcare" was an awkward placeholder
Vault:
- compress 4g vault step from inline-tutorial to a one-line pointer at our External Secrets docs ("the module fully supports vault integration; see External Secrets")
- drop the duplicate "With Vault secret reference" payload example in Step 5 (kept the inline-secret example)
- 4f staging heading was "managed mode only" — now explicitly names both managed-zerobus and managed-sql; added a one-liner clarifying that external-direct does not go through staging. - Authentication sequence diagram trimmed hard: was trying to depict token exchange AND three per-mode write paths in one alt-block, which buried the actual point (OAuth M2M token exchange + caching). Now the diagram shows only the token exchange + cache rule, with a one-line bridge into the per-mode bullets that already explain "what the bearer authorizes" below. - Dropped the Cost considerations section. Its content was already covered (and more concisely) in the Choosing-between-modes table and in each mode's card; the standalone section was just restating the same trade-offs.
- Reordered the step-by-step so Databricks setup is Step 1, then
Subscription Topic / ViewDefinition / Materialize / Configure
destination / Verify. Without the Databricks-side values (workspace
URL/ID/region, SP id+secret, warehouse id, table name) in hand,
there's nothing to plug into the destination payload.
- Step 1e (grants) now opens with a privilege table — privilege /
granted on / needed by / purpose — covering all three modes' grants
including the External-Location grants for staging vs target. The
raw SQL is still below but framed as concrete examples that fill in
the table.
- Delta-log paragraph in Background → "Delta Lake" tightened from a
three-sentence explanation to one ("readers replay it; writers
append a new commit; concurrent writers race on the next log
filename — that's where Delta's ACID comes from").
The combined privilege table tried to disambiguate three modes' grants via "Needed by" rows, which made the customer hunt for "just the rows I need". Split it into three GitBook tab panes (managed-zerobus, managed-sql, external-direct) — each tab has a per-mode privilege table plus a runnable GRANT SQL block scoped to that mode only. 1f (staging-location setup) trimmed accordingly: the GRANT block at the bottom was a duplicate of what now lives in the managed-* tabs of 1e, replaced with a reference back.
- Authentication: dropped the three-paragraph "How the same token gets
used differently in each mode" walls — the side-by-side table that
was already at the bottom of the section conveys the same in a
fraction of the room. Removed Databricks API endpoint paths
(`/api/2.0/sql/statements`, `/api/2.1/unity-catalog/...`); they were
debug-level detail in a customer tutorial.
- Before-you-begin: collapsed three per-mode prerequisite bullets into
a single shorter list ("a SQL warehouse — skip only for
external-direct"; "for managed-zerobus: Zerobus enabled on your
SKU"; "for initial-export in the managed-* modes: a UC External
Location for staging").
- Choosing-between-modes: opens with explicit "default to
managed-zerobus; switch only when ..." recommendations, then the
comparison table. Pulled the former trailing bullets ("Schema
drift", "Initial export path", "Storage backends") into rows of the
same table to keep the section single-screen.
- Vault: 1g now ships one concrete FHIR primitive-extension JSON
snippet so customers can copy-paste the secret-reference pattern
without bouncing into the External Secrets docs. The actual
configuration steps remain in the linked External Secrets doc.
spicyfalafel
marked this pull request as ready for review
May 18, 2026 12:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Draft tutorial for the new `Data Lakehouse AidboxTopicDestination` (kind `data-lakehouse-at-least-once`), shipping with the module PR https://github.com/HealthSamurai/topic-destination-deltalake/pull/1 in Aidbox 2605.
Covers:
Status
Draft — opens for early review while:
Test plan
🤖 Generated with Claude Code