From ddf7e75847481a31853802419888b88fcee69af9 Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Tue, 28 Apr 2026 12:01:22 -0400 Subject: [PATCH 1/6] docs: add v2 PRD for template connectors restart Captures scope, splittable changes (EnvVarName/headers/icons), and sequenced PR plan derived from #8981. Co-Authored-By: Claude Opus 4.7 --- PRD-template-connectors-v2.md | 238 ++++++++++++++++++++++++++++++++++ 1 file changed, 238 insertions(+) create mode 100644 PRD-template-connectors-v2.md diff --git a/PRD-template-connectors-v2.md b/PRD-template-connectors-v2.md new file mode 100644 index 000000000000..30f64384a1a2 --- /dev/null +++ b/PRD-template-connectors-v2.md @@ -0,0 +1,238 @@ +# Template Connectors — v2 (restart of PR #8981) + +Reference: https://github.com/rilldata/rill/pull/8981 — `feat-template-connector` branch (108 files, +14,556 / −2,282). + +## Goal + +Move connector schemas, SQL generation, and YAML templating out of hand-written TypeScript and into declarative JSON template definitions served by the runtime. The frontend stops owning per-connector form schemas; instead it fetches them from a new `ListTemplates` RPC and renders YAML through a `GenerateFile` RPC. + +End state, per connector: one JSON file describing the JSON Schema (form), output file templates, and OLAP-specific SQL. No TypeScript schema, no Go-side template code. + +## Why restart + +The original PR grew across ~5 weeks of code review and contains three intertwined efforts: + +1. The template-connectors refactor (the actual goal). +2. Standalone backend metadata changes (`PropertySpec.EnvVarName` on driver specs, HTTPS `headers` config) that don't depend on the templates package. +3. A ClickHouse data-path improvement (`url()` table function with auto-detected format and `headers()` syntax) bundled in because it overlaps with HTTPS. + +The add-data flow code in particular went through several iterations (server-side preview, sync vs. async, `.env` conflict detection moved client-to-server). Restarting against current `main` lets us land it cleanly in stages instead of carrying all the churn. + +## What ships separately (not in this restart) + +Cut these out of #8981 and ship as their own PRs first. They're useful on their own and unblock the v2 work. + +### PR-A: `PropertySpec.EnvVarName` field + driver annotations + +- `runtime/drivers/connectors.go` — adds `EnvVarName string` to `PropertySpec`. +- 13 driver files — populates `EnvVarName` on secret properties: + - `athena`, `azure`, `bigquery`, `clickhouse`, `druid`, `duckdb` (motherduck), `gcs`, `mysql`, `pinot`, `postgres`, `redshift`, `s3`, `salesforce`, `snowflake`. +- ~50 LOC total. No consumers required — the field is just metadata. + +### PR-B: HTTPS `headers` ConfigProperty + +- `runtime/drivers/https/https.go` — adds `ConfigProperties` with `headers` field (8 LOC). +- Optional follow-up: corresponding driver/runtime changes that *use* the header value at request time (check whether the existing PR also wires this through; if not, this stays metadata-only until templates land). + +### PR-C: ClickHouse HTTPS `url()` + `headers()` (separate from templating) + +- ClickHouse's `url()` table function with auto-detected format parameter and authenticated `headers()` syntax. +- This is a real data-path change, not a templating change. It belongs to the ClickHouse driver, not `runtime/templates/`. +- Verify scope with Aditya — the original PR notes "Aditya will work on proper changes here" so the actual data-path may already be elsewhere. + +### PR-D: New connector icons + +- `web-common/src/components/icons/connectors/`: + - `ApacheHudi.svelte`, `ApacheHudiIcon.svelte` + - `ApacheKafka.svelte`, `ApacheKafkaIcon.svelte` + - `LocalFileIcon.svelte` + - `MongoDB.svelte`, `MongoDBIcon.svelte` + - `GoogleCloudStorageIcon.svelte` (rework of existing icon — diff this carefully against current `main` before adopting). +- Wire-up to `connectorIconMapping` happens in this PR; the icons aren't *referenced* until the templates land, so consider keeping the mapping change small. + +## In scope: template-connector restart + +### Backend — `runtime/templates/` package + +New Go package, ~1,100 LOC across 7 files (excluding tests): + +- `template.go` — `Template` struct: `Name`, `DisplayName`, `Driver`, `OLAP`, `Tags`, `JSONSchema` (raw), `Files` (output specs), `Icon`, `SmallIcon`, `Description`, `DocsURL`. +- `registry.go` — registry built from `embed.FS` over `definitions/`. Provides `List(tags ...)`, `Get(name)`. Loads at process start. +- `render.go` — renders a template with property values. Produces one or more `GeneratedFile{Path, Blob}`. Handles env-var extraction for `x-secret` properties. +- `funcmap.go` — `text/template` helper functions used by `code_template` strings: `duckdbSQL`, `propVal`, `default`, `azureContainer`, `azureBlobPath`, `clickhouseURLSuffix`, `renderProps`. Delimiters are `[[ ]]` (not `{{ }}`) so YAML's `{{ }}` doesn't collide with templating. +- `env.go` — `ResolveEnvVarNameForKey(driver, key, explicit, existingEnv)` and `ReadEnvKeys(repo)`. Implements the `EnvVarName` resolution + suffix-on-conflict policy (`FOO`, `FOO_1`, `FOO_2`, ...). +- `headers.go` — parses HTTPS-style `headers` strings. + +Tests next to each module + `render_test.go` covering golden YAML for each template. + +### Backend — JSON template definitions + +`runtime/templates/definitions//.json`. Three groups in #8981: + +- `olap/` — 6 files: `clickhouse`, `druid`, `duckdb`, `motherduck`, `pinot`, `starrocks`. +- `duckdb-models/` — 18 files: source connectors that target DuckDB (`s3-duckdb`, `gcs-duckdb`, `azure-duckdb`, `postgres-duckdb`, `snowflake-duckdb`, `bigquery-duckdb`, `athena-duckdb`, `redshift-duckdb`, `mysql-duckdb`, `salesforce-duckdb`, `delta-duckdb`, `iceberg-duckdb`, `https-duckdb`, `local-file-duckdb`, `sqlite-duckdb`, `clickhouse-duckdb`, `duckdb-duckdb`, `supabase-duckdb`). +- `clickhouse-models/` — 12 files: source connectors that target ClickHouse OLAP (`s3-clickhouse`, `gcs-clickhouse`, `azure-clickhouse`, `postgres-clickhouse`, `mysql-clickhouse`, `delta-clickhouse`, `iceberg-clickhouse`, `hudi-clickhouse`, `kafka-clickhouse`, `mongodb-clickhouse`, `https-clickhouse`, `supabase-clickhouse`). + +Note: ClickHouse `local-file-clickhouse` and `sqlite-clickhouse` are intentionally absent — those drivers require ClickHouse server-side `user_files` and aren't usable in Rill Cloud. If we want them to appear at all, they need stub templates with `_reason` so the UI can hide/disable them gracefully (decide in §Open decisions). + +JSON shape (see `s3-duckdb.json` as the reference): + +```json +{ + "name": "s3-duckdb", + "display_name": "Amazon S3", + "driver": "s3", + "olap": "duckdb", + "icon": "AmazonS3", + "small_icon": "AmazonS3Icon", + "tags": ["source", "duckdb", "s3", ...], + "json_schema": { /* JSON Schema with x-* extensions */ }, + "files": [ + { "name": "connector", "path_template": "connectors/[[ .connector_name ]].yaml", "code_template": "..." }, + { "name": "model", "path_template": "models/[[ .model_name ]].yaml", "code_template": "..." } + ] +} +``` + +Custom JSON Schema extensions (must be documented somewhere — likely a README inside `definitions/`): + +- `x-step` — `connector` | `source` | `explorer` (for multi-step flows). +- `x-display` — `radio`, etc. +- `x-grouped-fields` — fields shown when a radio enum value is selected. +- `x-visible-if` — conditional visibility based on other field values. +- `x-secret` — value is a credential, gets routed through `.env`. +- `x-env-var` — explicit env var name override (matches `PropertySpec.EnvVarName`). +- `x-placeholder`, `x-enum-labels`, `x-enum-descriptions`, `x-ui-only`. +- `x-category` — `objectStore` | `warehouse` | `sqlStore` | `olap` | `sourceOnly` | … +- `x-form-height` — `tall` (used by snowflake, salesforce). +- `x-omit-if-default` — generic replacement for the per-driver `key == "managed"` skip used in v1; suppresses property output when value matches schema default. + +### Backend — proto API (`proto/rill/runtime/v1/api.proto`) + +#8981 adds three RPCs. **Decide before starting v2** which we keep: + +- `ListTemplates(ListTemplatesRequest{tags})` → `ListTemplatesResponse{templates: [Template]}`. **Keep.** Drives the `AddDataModal` connector list and form schemas. +- `GenerateFile(GenerateFileRequest{template_name, output, properties, connector_name, preview})` → `GenerateFileResponse{files: [GeneratedFile{path, blob}], env_vars}`. **Keep.** Handles both preview (`preview=true`) and write paths. +- `GenerateTemplate(GenerateTemplateRequest{resource_type, driver, properties, connector_name})` → `GenerateTemplateResponse{blob, env_vars, resource_type, driver}`. **Reconsider.** In #8981 the frontend's `generate-template.ts` *only* calls `GenerateFile`; `GenerateTemplate` appears unused by the new flow. Recommend: drop `GenerateTemplate` entirely in v2, do everything through `GenerateFile`. If there's a CLI/SDK consumer that needs the older shape, document it before keeping. + +Also adds `Template` and `TemplateFile` proto messages — keep both. + +### Backend — runtime server + +- `runtime/server/templates.go` — `ListTemplates` handler. Queries the registry, optionally filters by tags. +- `runtime/server/generate_template.go` — `GenerateFile` (and possibly `GenerateTemplate`) handler. Renders, optionally writes files via `repo.Put` and merges `.env`. +- `runtime/server/server.go` — wires the new RPCs into the gRPC server. + +Hardening items called out in the v1 review (must be in v2): + +- `appendEnvVar` strips newlines and quotes values containing spaces / special characters (env var injection defense). +- Repo access errors in `GenerateTemplate`, `GenerateFile`, `writeRenderedFiles` log warnings, do not silently swallow. +- `defaultVal` doc comment warns against pipeline syntax — only positional `[[ default (expr) "fallback" ]]` is safe. + +### Frontend — `web-common/src/features/sources/modal/` + +This is the heart of the restart. Touch list: + +- **`AddDataModal.svelte`** — replace `connectors` import (static) with `createRuntimeServiceListTemplates` queries: + - `sourceTemplatesQuery` — derived from `instanceQuery.olapConnector`, requests `{tags: ["source", olap]}`. + - `olapTemplatesQuery` — `{tags: ["olap"]}`, no instance dependency. + - Map templates → `ConnectorInfo`. Source vs OLAP split lives in template tags, not frontend constants. + +- **`AddDataForm.svelte`**: + - Remove `onMount` `.env` blob fetch + `existingEnvBlob` state. + - Replace sync `formManager.computeYamlPreview(...)` with debounced (150 ms) async call. Use `onDestroy` to clear the timer. Keep last-valid preview on error so the YAML pane doesn't blank during typing. + - Wrap `paramsError` in a max-h-32 scroll container (already in v1; pull through). + +- **`AddDataFormManager.ts`** — `computeYamlPreview` becomes async. Four old branches collapse into two: + - Multi-step connector step → `GenerateFile{template_name, output: "connector", properties}`. + - Multi-step source/explorer step → `GenerateFile{template_name, output: "model", properties: combinedValues, connector_name}`. + - Single-step connector and single-step source forms route through the same two paths. + - Drop imports of `compileConnectorYAML`, `compileSourceYAML`, `prepareSourceFormData`, `getSchemaSecretKeys`, `getSchemaStringKeys`. These all move server-side. + +- **`connector-schemas.ts`**: + - Add `populateSchemaCache(map)` (test seam) and `registerTemplateSchema(driver, templateName, schema, displayName)` (runtime registration when `ListTemplates` resolves). + - Replace static `multiStepFormSchemas` table with cache populated from API responses. Static fallback may still be useful for SSR / tests — decide in §Open decisions. + - Remove unused exports `getBackendConnectorName`. + +- **`generate-template.ts` (new)**: + - `generateTemplate(client, {resourceType, driver, properties, connectorName})` → wraps `runtimeServiceGenerateFile` with `preview: true`. Handles `driver → templateName` resolution via OLAP cache. + - `mergeEnvVars(client, queryClient, envVars)` — invalidates `.env` query, fetches fresh blob, merges via `replaceOrAddEnvVariable`, returns `{newBlob, originalBlob}` for rollback. + - OLAP cache (`olapCache: Map`) populated by `createConnectorSchemas()` when schemas load. Avoids one extra `GetInstance` round-trip. + +- **Deletes**: + - `web-common/src/features/templates/schemas/*.ts` (16 hand-written schema files: `athena`, `azure`, `bigquery`, `clickhouse`, `delta`, `druid`, `duckdb`, `gcs`, `https`, `iceberg`, `local_file`, `motherduck`, `mysql`, `pinot`, `postgres`, `redshift`, `s3`, `salesforce`, `snowflake`, `sqlite`, `starrocks`, `supabase`). Keep AI schemas (`claude`, `gemini`, `openai`) and DuckLake (`ducklake`, `ducklake-utils`) — they're not in the template path. + - `web-common/src/features/sources/sourceUtils.ts` — `compileSourceYAML`, `prepareSourceFormData`. All YAML generation moves server-side. + - `web-common/src/features/sources/modal/submitAddDataForm.ts` — replaced by direct `GenerateFile` calls + `mergeEnvVars`. + - `web-common/src/features/templates/JSONSchemaFormRenderer.svelte` — verify no other callers before delete. + +- **`web-common/src/features/add-data/manager/selectors.ts`** — same templates-query pattern as `AddDataModal.svelte` but for the new add-data manager. + +- **Tests**: + - `connector-schemas.spec.ts` — gain ~70 LOC of test schema fixtures (because static schemas go away, tests need to seed the cache via `populateSchemaCache`). + - `AddDataFormManager.spec.ts`, `add-source-visibility.spec.ts`, `FormValidation.test.ts` — same pattern. + - `generate-template.spec.ts` (new, ~180 LOC) — covers `mergeEnvVars` edge cases (existing keys, empty file, 404, suffix conflicts). + - `FormValidation.test.ts` imports the actual `s3-duckdb.json` to ensure frontend validation stays in sync with the backend schema. **This is a meaningful pattern — keep it.** + +## Behavior changes (not pure refactor) + +These are real product behavior changes that need explicit sign-off and QA, not just transparent refactor: + +1. **Async, debounced YAML preview.** Was: synchronous on every keystroke against in-memory schema. Now: 150 ms debounce → server round-trip. Risks: race conditions on rapid edits (last RPC may not be the latest input), perceptible lag on slow connections, blank preview during transient errors. Mitigation: keep last-valid preview on error; consider AbortController to cancel in-flight requests on new input. + +2. **Server-side `.env` conflict detection.** Was: client pre-fetched `.env` on form mount, computed conflicts client-side. Now: server resolves names + suffixes inside `GenerateFile`. Cleaner, but means the frontend doesn't know about conflicts until it gets the response. Acceptable, but worth verifying the UX doesn't regress (e.g. same env var name being used twice across two add-source flows). + +3. **ClickHouse `local_file` / `sqlite` become unselectable.** Decide whether to (a) hide them from the UI entirely when OLAP is ClickHouse, (b) show with a disabled state + tooltip explaining why, (c) ship stubs that surface a server-side error message. + +4. **Connector list filtered by OLAP.** `ListTemplates(tags=["source", olap])` means non-supported source/OLAP combos disappear from the picker. Previously the frontend hard-coded which source-OLAP combos were valid. Less code, but make sure the filtering matches the matrix users currently see. + +5. **OLAP detection fix** (`normalizeOlapForTemplate` checks `projectConnectors` too) — this resolves arbitrarily-named OLAP connectors like `clickhouse_1`. Pure bugfix, but worth its own commit so it can be cherry-picked. + +## Sequencing + +Recommended order: + +1. **PR-A** — `EnvVarName` field + driver annotations. Unblocks template env-var resolution. +2. **PR-B** — HTTPS `headers` ConfigProperty. Unblocks HTTPS templates. +3. **PR-D** — New connector icons + mapping wiring. Pure additive, can land anytime. +4. **PR 1: backend templates package** — `runtime/templates/` + `definitions/` JSON + tests. Lands without consumers; new RPCs not yet exposed. +5. **PR 2: proto + server handlers** — `ListTemplates`, `GenerateFile` (drop `GenerateTemplate` if §Open decisions confirms). Generated bindings + Orval client. +6. **PR 3: frontend rewire (add-data modal)** — Switch `AddDataModal`, `AddDataFormManager`, `selectors.ts`, add `generate-template.ts`, update tests. +7. **PR 4: delete TS schemas** — Remove `templates/schemas/*.ts`, `sourceUtils.ts`, `submitAddDataForm.ts` once nothing imports them. Keep this isolated so any forgotten import surfaces clearly. +8. **PR-C** — ClickHouse `url()` / `headers()` data-path improvements. Independent; can interleave anywhere. + +PRs 1–4 can be authored concurrently but should land in order. PR 1 + PR 2 could be combined if the diff stays manageable. + +## Open decisions + +- **Drop `GenerateTemplate` RPC?** v1 has both. Frontend goes through `GenerateFile`. Recommend dropping unless a CLI/SDK consumer depends on the simpler shape. **Action: search for callers before final proto land.** +- **Static schema fallback in `connector-schemas.ts`?** v1 keeps the cache populated dynamically. Static fallback could simplify SSR / tests but reintroduces the dual-source-of-truth problem. Recommend: no fallback; tests use `populateSchemaCache` (already the v1 pattern). +- **ClickHouse `local_file` / `sqlite` UX** — see Behavior change #3. +- **Where does the JSON Schema extension dictionary live?** A README in `runtime/templates/definitions/` makes it discoverable for future connector authors. Recommend writing this when PR 1 lands, not retrofitted later. +- **`compileConnectorYAML` (in `web-common/src/features/connectors/code-utils.ts`)** — used by `AddDataFormManager` v1. After the RPC migration, is it still used elsewhere? If not, delete in PR 4. + +## Out of scope + +- AI connectors (`claude`, `gemini`, `openai`) — different add-data flow, leave as TS schemas. +- DuckLake (`ducklake.ts`, `ducklake-utils.ts`) — separate feature, not part of templating. +- Error message quality — flagged in v1 as needing a follow-up PR. Confusing "manually setting columns" errors will need their own work after the templates land. +- Embedded dashboard surface — not affected by add-data changes. + +## Test strategy + +- **Backend unit**: each `runtime/templates/` module has its own `_test.go`. `render_test.go` runs golden tests for representative templates. +- **Backend integration**: extend `runtime/server/generate_template_test.go` to cover the preview/write split, env-var merge, and `connector_name` overrides. +- **Frontend unit**: test-only `populateSchemaCache` in `connector-schemas.ts` — schema fixtures live in the test file. `FormValidation.test.ts` imports real JSON to keep validation in sync. +- **Frontend integration**: `generate-template.spec.ts` covers `mergeEnvVars`. The `AddDataFormManager` spec covers preview path branching. +- **Manual QA matrix** (carry over from v1): + - Public GCS, HMAC GCS, Azure connection string, Azure storage key, S3 access keys, Postgres — all on ClickHouse OLAP. + - Same set on DuckDB OLAP. + - Multi-step flow: enter connector creds → save → enter source path → save. + - Single-step flow: ClickHouse OLAP form, DuckDB OLAP form. + - Existing `.env` with conflicting key — verify suffix `_1` rendering. + +## Risks + +- **Server round-trip on every keystroke (debounced)** — worst case a 150 ms input delay + RPC cost. If this lags noticeably on cloud, fall back to client-side preview for the `connector` step (which is small) and keep server preview only for the `source/model` step (which has SQL generation). +- **Schema-divergence between frontend `x-*` extensions and backend renderer** — the JSON files are the source of truth, but the frontend still has to *interpret* `x-step`, `x-visible-if`, `x-grouped-fields`, etc. Drift is possible. Mitigation: a documented list of supported `x-*` keys in the `definitions/README.md` and a typed schema interface on the frontend that fails loudly on unknown keys. +- **Embed of `definitions/`** — `embed.FS` includes everything matching the glob. Make sure no editor swap files / JSON comments slip in (use a strict `//go:embed definitions/*/*.json`). +- **Cherry-pick to release branch** — v1 checks "Intend to cherry-pick into the release branch." Decide early whether v2 is targeting a release branch; if so, keep PR-A through PR 4 individually small enough to cherry-pick. From 592623b6aeba7f9b326b2c6be97f14209468b641 Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Tue, 28 Apr 2026 12:04:28 -0400 Subject: [PATCH 2/6] docs: drop PR-B/PR-C, fold into PR 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit HTTPS headers ConfigProperty is form metadata only — should ship with the HTTPS template that uses it. ClickHouse url()/headers() SQL is inside runtime/templates/funcmap.go and headers.go, not a separate driver change. Co-Authored-By: Claude Opus 4.7 --- PRD-template-connectors-v2.md | 40 ++++++++++++----------------------- 1 file changed, 13 insertions(+), 27 deletions(-) diff --git a/PRD-template-connectors-v2.md b/PRD-template-connectors-v2.md index 30f64384a1a2..af544a058e9e 100644 --- a/PRD-template-connectors-v2.md +++ b/PRD-template-connectors-v2.md @@ -10,17 +10,11 @@ End state, per connector: one JSON file describing the JSON Schema (form), outpu ## Why restart -The original PR grew across ~5 weeks of code review and contains three intertwined efforts: - -1. The template-connectors refactor (the actual goal). -2. Standalone backend metadata changes (`PropertySpec.EnvVarName` on driver specs, HTTPS `headers` config) that don't depend on the templates package. -3. A ClickHouse data-path improvement (`url()` table function with auto-detected format and `headers()` syntax) bundled in because it overlaps with HTTPS. - -The add-data flow code in particular went through several iterations (server-side preview, sync vs. async, `.env` conflict detection moved client-to-server). Restarting against current `main` lets us land it cleanly in stages instead of carrying all the churn. +The original PR grew across ~5 weeks of code review. The add-data flow code in particular went through several iterations: server-side preview moved client-to-server, sync preview became async + debounced, `.env` conflict detection moved client-to-server. Restarting against current `main` lets us land it cleanly in stages instead of carrying all the churn. ## What ships separately (not in this restart) -Cut these out of #8981 and ship as their own PRs first. They're useful on their own and unblock the v2 work. +Two genuinely splittable PRs. ### PR-A: `PropertySpec.EnvVarName` field + driver annotations @@ -29,17 +23,6 @@ Cut these out of #8981 and ship as their own PRs first. They're useful on their - `athena`, `azure`, `bigquery`, `clickhouse`, `druid`, `duckdb` (motherduck), `gcs`, `mysql`, `pinot`, `postgres`, `redshift`, `s3`, `salesforce`, `snowflake`. - ~50 LOC total. No consumers required — the field is just metadata. -### PR-B: HTTPS `headers` ConfigProperty - -- `runtime/drivers/https/https.go` — adds `ConfigProperties` with `headers` field (8 LOC). -- Optional follow-up: corresponding driver/runtime changes that *use* the header value at request time (check whether the existing PR also wires this through; if not, this stays metadata-only until templates land). - -### PR-C: ClickHouse HTTPS `url()` + `headers()` (separate from templating) - -- ClickHouse's `url()` table function with auto-detected format parameter and authenticated `headers()` syntax. -- This is a real data-path change, not a templating change. It belongs to the ClickHouse driver, not `runtime/templates/`. -- Verify scope with Aditya — the original PR notes "Aditya will work on proper changes here" so the actual data-path may already be elsewhere. - ### PR-D: New connector icons - `web-common/src/components/icons/connectors/`: @@ -50,6 +33,11 @@ Cut these out of #8981 and ship as their own PRs first. They're useful on their - `GoogleCloudStorageIcon.svelte` (rework of existing icon — diff this carefully against current `main` before adopting). - Wire-up to `connectorIconMapping` happens in this PR; the icons aren't *referenced* until the templates land, so consider keeping the mapping change small. +### Folded into PR 1 (the templates package), not separate PRs + +- **HTTPS `headers` ConfigProperty.** The HTTPS driver already consumes `headers` (`runtime/drivers/https/https.go` reads `Headers` at request time). The PR's 8-line addition is purely the *spec metadata* that tells the form to render a headers field. It's only useful once an HTTPS template asks the user for headers — so it should land alongside the HTTPS template, not before. +- **ClickHouse `url()` + `headers()` SQL.** Not a separable driver change. The `headers()` SQL fragment is generated by `runtime/templates/funcmap.go:clickhouseURLSuffix` and `runtime/templates/headers.go` — both are part of the templates package itself. ClickHouse server already supports `url()`/`headers()` natively. The "Aditya will work on proper changes here" note in #8981 was about ClickHouse cloud verification workflow (a different effort). + ## In scope: template-connector restart ### Backend — `runtime/templates/` package @@ -191,14 +179,12 @@ These are real product behavior changes that need explicit sign-off and QA, not Recommended order: -1. **PR-A** — `EnvVarName` field + driver annotations. Unblocks template env-var resolution. -2. **PR-B** — HTTPS `headers` ConfigProperty. Unblocks HTTPS templates. -3. **PR-D** — New connector icons + mapping wiring. Pure additive, can land anytime. -4. **PR 1: backend templates package** — `runtime/templates/` + `definitions/` JSON + tests. Lands without consumers; new RPCs not yet exposed. -5. **PR 2: proto + server handlers** — `ListTemplates`, `GenerateFile` (drop `GenerateTemplate` if §Open decisions confirms). Generated bindings + Orval client. -6. **PR 3: frontend rewire (add-data modal)** — Switch `AddDataModal`, `AddDataFormManager`, `selectors.ts`, add `generate-template.ts`, update tests. -7. **PR 4: delete TS schemas** — Remove `templates/schemas/*.ts`, `sourceUtils.ts`, `submitAddDataForm.ts` once nothing imports them. Keep this isolated so any forgotten import surfaces clearly. -8. **PR-C** — ClickHouse `url()` / `headers()` data-path improvements. Independent; can interleave anywhere. +1. **PR-A** — `EnvVarName` field + driver annotations. Unblocks template env-var resolution. Land before PR 1. +2. **PR-D** — New connector icons + mapping wiring. Pure additive, can land anytime. +3. **PR 1: backend templates package** — `runtime/templates/` + `definitions/` JSON + tests. Includes the HTTPS `headers` ConfigProperty addition and the `clickhouseURLSuffix`/`headers.go` SQL helpers. Lands without consumers; new RPCs not yet exposed. +4. **PR 2: proto + server handlers** — `ListTemplates`, `GenerateFile` (drop `GenerateTemplate` if §Open decisions confirms). Generated bindings + Orval client. +5. **PR 3: frontend rewire (add-data modal)** — Switch `AddDataModal`, `AddDataFormManager`, `selectors.ts`, add `generate-template.ts`, update tests. +6. **PR 4: delete TS schemas** — Remove `templates/schemas/*.ts`, `sourceUtils.ts`, `submitAddDataForm.ts` once nothing imports them. Keep this isolated so any forgotten import surfaces clearly. PRs 1–4 can be authored concurrently but should land in order. PR 1 + PR 2 could be combined if the diff stays manageable. From 193092114a3af186afd35fb3e6cb2ee995b81c18 Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Tue, 28 Apr 2026 13:51:38 -0400 Subject: [PATCH 3/6] docs: simplify PRD sequencing for template connectors v2 Drop the PR-A and PR-D split notes; land the HTTPS `headers` ConfigProperty and ClickHouse `url()`/`headers()` SQL helpers as part of PR 1 instead. Remove the DuckLake out-of-scope bullet since DuckLake never used template schemas. Co-Authored-By: Claude Opus 4.7 --- PRD-template-connectors-v2.md | 36 ++++------------------------------- 1 file changed, 4 insertions(+), 32 deletions(-) diff --git a/PRD-template-connectors-v2.md b/PRD-template-connectors-v2.md index af544a058e9e..03fd4b2fdae1 100644 --- a/PRD-template-connectors-v2.md +++ b/PRD-template-connectors-v2.md @@ -12,31 +12,6 @@ End state, per connector: one JSON file describing the JSON Schema (form), outpu The original PR grew across ~5 weeks of code review. The add-data flow code in particular went through several iterations: server-side preview moved client-to-server, sync preview became async + debounced, `.env` conflict detection moved client-to-server. Restarting against current `main` lets us land it cleanly in stages instead of carrying all the churn. -## What ships separately (not in this restart) - -Two genuinely splittable PRs. - -### PR-A: `PropertySpec.EnvVarName` field + driver annotations - -- `runtime/drivers/connectors.go` — adds `EnvVarName string` to `PropertySpec`. -- 13 driver files — populates `EnvVarName` on secret properties: - - `athena`, `azure`, `bigquery`, `clickhouse`, `druid`, `duckdb` (motherduck), `gcs`, `mysql`, `pinot`, `postgres`, `redshift`, `s3`, `salesforce`, `snowflake`. -- ~50 LOC total. No consumers required — the field is just metadata. - -### PR-D: New connector icons - -- `web-common/src/components/icons/connectors/`: - - `ApacheHudi.svelte`, `ApacheHudiIcon.svelte` - - `ApacheKafka.svelte`, `ApacheKafkaIcon.svelte` - - `LocalFileIcon.svelte` - - `MongoDB.svelte`, `MongoDBIcon.svelte` - - `GoogleCloudStorageIcon.svelte` (rework of existing icon — diff this carefully against current `main` before adopting). -- Wire-up to `connectorIconMapping` happens in this PR; the icons aren't *referenced* until the templates land, so consider keeping the mapping change small. - -### Folded into PR 1 (the templates package), not separate PRs - -- **HTTPS `headers` ConfigProperty.** The HTTPS driver already consumes `headers` (`runtime/drivers/https/https.go` reads `Headers` at request time). The PR's 8-line addition is purely the *spec metadata* that tells the form to render a headers field. It's only useful once an HTTPS template asks the user for headers — so it should land alongside the HTTPS template, not before. -- **ClickHouse `url()` + `headers()` SQL.** Not a separable driver change. The `headers()` SQL fragment is generated by `runtime/templates/funcmap.go:clickhouseURLSuffix` and `runtime/templates/headers.go` — both are part of the templates package itself. ClickHouse server already supports `url()`/`headers()` natively. The "Aditya will work on proper changes here" note in #8981 was about ClickHouse cloud verification workflow (a different effort). ## In scope: template-connector restart @@ -179,12 +154,10 @@ These are real product behavior changes that need explicit sign-off and QA, not Recommended order: -1. **PR-A** — `EnvVarName` field + driver annotations. Unblocks template env-var resolution. Land before PR 1. -2. **PR-D** — New connector icons + mapping wiring. Pure additive, can land anytime. -3. **PR 1: backend templates package** — `runtime/templates/` + `definitions/` JSON + tests. Includes the HTTPS `headers` ConfigProperty addition and the `clickhouseURLSuffix`/`headers.go` SQL helpers. Lands without consumers; new RPCs not yet exposed. -4. **PR 2: proto + server handlers** — `ListTemplates`, `GenerateFile` (drop `GenerateTemplate` if §Open decisions confirms). Generated bindings + Orval client. -5. **PR 3: frontend rewire (add-data modal)** — Switch `AddDataModal`, `AddDataFormManager`, `selectors.ts`, add `generate-template.ts`, update tests. -6. **PR 4: delete TS schemas** — Remove `templates/schemas/*.ts`, `sourceUtils.ts`, `submitAddDataForm.ts` once nothing imports them. Keep this isolated so any forgotten import surfaces clearly. +1. **PR 1: backend templates package** — `runtime/templates/` + `definitions/` JSON + tests. Includes the HTTPS `headers` ConfigProperty addition and the `clickhouseURLSuffix`/`headers.go` SQL helpers. Lands without consumers; new RPCs not yet exposed. +2. **PR 2: proto + server handlers** — `ListTemplates`, `GenerateFile` (drop `GenerateTemplate` if §Open decisions confirms). Generated bindings + Orval client. +3. **PR 3: frontend rewire (add-data modal)** — Switch `AddDataModal`, `AddDataFormManager`, `selectors.ts`, add `generate-template.ts`, update tests. +4. **PR 4: delete TS schemas** — Remove `templates/schemas/*.ts`, `sourceUtils.ts`, `submitAddDataForm.ts` once nothing imports them. Keep this isolated so any forgotten import surfaces clearly. PRs 1–4 can be authored concurrently but should land in order. PR 1 + PR 2 could be combined if the diff stays manageable. @@ -199,7 +172,6 @@ PRs 1–4 can be authored concurrently but should land in order. PR 1 + PR 2 cou ## Out of scope - AI connectors (`claude`, `gemini`, `openai`) — different add-data flow, leave as TS schemas. -- DuckLake (`ducklake.ts`, `ducklake-utils.ts`) — separate feature, not part of templating. - Error message quality — flagged in v1 as needing a follow-up PR. Confusing "manually setting columns" errors will need their own work after the templates land. - Embedded dashboard surface — not affected by add-data changes. From 934bb61bc894ed094c3d0ea909b9475849577165 Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Tue, 28 Apr 2026 13:52:33 -0400 Subject: [PATCH 4/6] feat: add `runtime/templates` package for declarative connector templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces a new package that loads connector and OLAP templates from embedded JSON definitions, validates them against a JSON Schema, and renders YAML output via Go `text/template` with `[[ ]]` delimiters. This replaces the hand-written TypeScript schemas and the Go-side template code used by the legacy add-data flow; the runtime becomes the single source of truth for connector form metadata, secret extraction, and YAML generation. Includes: - `registry.go` — loads `definitions/*/*.json` at process start (strict embed glob), preserves JSON property order for stable form rendering. - `render.go` — schema-driven property processing: `x-secret` extraction to `.env`, `x-omit-if-default` suppression, `x-step` splitting between connector and source outputs, header map handling. - `funcmap.go` — template helpers: `renderProps`, `duckdbSQL`, Azure URI decomposition, `s3ToHTTPS` / `gcsToHTTPS`, `clickhouseFormat`, `clickhouseURLSuffix`. `default` only supports positional syntax. - `env.go` — `ResolveEnvVarNameForKey` with `_1`/`_2` suffix-on-conflict policy and `.env` key reader. - `headers.go` — sensitive header detection, auth-scheme prefix split, env-var name derivation for HTTPS `headers` properties. - 36 JSON definitions: 6 OLAPs, 18 DuckDB-targeted source connectors, 12 ClickHouse-targeted source connectors. - `definitions/README.md` — documents the JSON Schema `x-*` extension dictionary and template helpers. No consumers yet; the new `ListTemplates` and `GenerateFile` RPCs land in a follow-up PR. Co-Authored-By: Claude Opus 4.7 --- runtime/templates/definitions/README.md | 129 ++++ .../clickhouse-models/azure-clickhouse.json | 147 ++++ .../clickhouse-models/delta-clickhouse.json | 69 ++ .../clickhouse-models/gcs-clickhouse.json | 118 ++++ .../clickhouse-models/https-clickhouse.json | 62 ++ .../clickhouse-models/hudi-clickhouse.json | 69 ++ .../clickhouse-models/iceberg-clickhouse.json | 69 ++ .../clickhouse-models/kafka-clickhouse.json | 76 ++ .../clickhouse-models/mongodb-clickhouse.json | 87 +++ .../clickhouse-models/mysql-clickhouse.json | 87 +++ .../postgres-clickhouse.json | 88 +++ .../clickhouse-models/s3-clickhouse.json | 118 ++++ .../supabase-clickhouse.json | 88 +++ .../duckdb-models/athena-duckdb.json | 88 +++ .../duckdb-models/azure-duckdb.json | 196 ++++++ .../duckdb-models/bigquery-duckdb.json | 81 +++ .../duckdb-models/clickhouse-duckdb.json | 50 ++ .../duckdb-models/delta-duckdb.json | 193 ++++++ .../duckdb-models/duckdb-duckdb.json | 47 ++ .../definitions/duckdb-models/gcs-duckdb.json | 164 +++++ .../duckdb-models/https-duckdb.json | 80 +++ .../duckdb-models/iceberg-duckdb.json | 78 +++ .../duckdb-models/local-file-duckdb.json | 49 ++ .../duckdb-models/mysql-duckdb.json | 171 +++++ .../duckdb-models/postgres-duckdb.json | 173 +++++ .../duckdb-models/redshift-duckdb.json | 105 +++ .../definitions/duckdb-models/s3-duckdb.json | 164 +++++ .../duckdb-models/salesforce-duckdb.json | 142 ++++ .../duckdb-models/snowflake-duckdb.json | 232 +++++++ .../duckdb-models/sqlite-duckdb.json | 53 ++ .../duckdb-models/supabase-duckdb.json | 147 ++++ .../definitions/olap/clickhouse.json | 385 +++++++++++ runtime/templates/definitions/olap/druid.json | 123 ++++ .../templates/definitions/olap/duckdb.json | 125 ++++ .../definitions/olap/motherduck.json | 57 ++ runtime/templates/definitions/olap/pinot.json | 144 ++++ .../templates/definitions/olap/starrocks.json | 148 ++++ runtime/templates/duckdb_test.go | 81 +++ runtime/templates/env.go | 48 ++ runtime/templates/env_test.go | 40 ++ runtime/templates/funcmap.go | 288 ++++++++ runtime/templates/headers.go | 52 ++ runtime/templates/headers_test.go | 56 ++ runtime/templates/registry.go | 225 ++++++ runtime/templates/registry_test.go | 157 +++++ runtime/templates/render.go | 389 +++++++++++ runtime/templates/render_test.go | 649 ++++++++++++++++++ runtime/templates/template.go | 43 ++ 48 files changed, 6430 insertions(+) create mode 100644 runtime/templates/definitions/README.md create mode 100644 runtime/templates/definitions/clickhouse-models/azure-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/delta-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/gcs-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/https-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/hudi-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/iceberg-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/kafka-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/mongodb-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/mysql-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/postgres-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/s3-clickhouse.json create mode 100644 runtime/templates/definitions/clickhouse-models/supabase-clickhouse.json create mode 100644 runtime/templates/definitions/duckdb-models/athena-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/azure-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/bigquery-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/delta-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/duckdb-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/gcs-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/https-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/iceberg-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/local-file-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/mysql-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/postgres-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/redshift-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/s3-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/salesforce-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/snowflake-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/sqlite-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/supabase-duckdb.json create mode 100644 runtime/templates/definitions/olap/clickhouse.json create mode 100644 runtime/templates/definitions/olap/druid.json create mode 100644 runtime/templates/definitions/olap/duckdb.json create mode 100644 runtime/templates/definitions/olap/motherduck.json create mode 100644 runtime/templates/definitions/olap/pinot.json create mode 100644 runtime/templates/definitions/olap/starrocks.json create mode 100644 runtime/templates/duckdb_test.go create mode 100644 runtime/templates/env.go create mode 100644 runtime/templates/env_test.go create mode 100644 runtime/templates/funcmap.go create mode 100644 runtime/templates/headers.go create mode 100644 runtime/templates/headers_test.go create mode 100644 runtime/templates/registry.go create mode 100644 runtime/templates/registry_test.go create mode 100644 runtime/templates/render.go create mode 100644 runtime/templates/render_test.go create mode 100644 runtime/templates/template.go diff --git a/runtime/templates/definitions/README.md b/runtime/templates/definitions/README.md new file mode 100644 index 000000000000..f56f25c5a977 --- /dev/null +++ b/runtime/templates/definitions/README.md @@ -0,0 +1,129 @@ +# Template definitions + +JSON files in this tree are the source of truth for the connector and OLAP forms shown in the Rill add-data flow. Each file describes one template: a JSON Schema (which drives the form on the frontend) plus one or more output files (rendered on the backend through `text/template`). + +The runtime loads every `definitions/*/*.json` file at process start via `embed.FS` (see `registry.go`). Stub files — empty or containing only a `_reason` field — are skipped, so an empty JSON file can be used as a placeholder while a connector is being designed. + +## Layout + +``` +definitions/ + olap/ connectors that act as the project's OLAP engine + duckdb-models/ source connectors targeting DuckDB OLAP + clickhouse-models/ source connectors targeting ClickHouse OLAP +``` + +Templates are keyed by their `name` field, not by file path. Filenames follow the convention `-.json` (e.g. `s3-duckdb.json`, `s3-clickhouse.json`) or just `.json` for OLAP connectors. + +## Template shape + +```jsonc +{ + "name": "s3-duckdb", + "display_name": "Amazon S3", + "driver": "s3", + "olap": "duckdb", + "icon": "AmazonS3", + "small_icon": "AmazonS3Icon", + "tags": ["source", "duckdb", "s3", "objectStore"], + "description": "...", + "docs_url": "https://docs.rilldata.com/...", + "json_schema": { /* JSON Schema with x-* extensions, see below */ }, + "files": [ + { "name": "connector", "path_template": "connectors/[[ .connector_name ]].yaml", "code_template": "..." }, + { "name": "model", "path_template": "models/[[ .model_name ]].yaml", "code_template": "..." } + ] +} +``` + +### File outputs + +Each entry in `files` produces one rendered file: + +- `name` — `"connector"` or `"model"`. The `GenerateFile` RPC accepts an `output` filter to render a single entry (used for previewing one step of a multi-step flow). +- `path_template` — Go `text/template` for the output path, relative to the project root. +- `code_template` — inline Go `text/template` for the file contents. +- `code_template_file` — alternative to `code_template`: a path (relative to the JSON definition) to a separate `.tmpl` file. Useful for long templates. If both are set, the file wins. + +Templates use **`[[ ]]` delimiters** (not `{{ }}`) so they don't collide with Rill's runtime templating syntax (`{{ .env.FOO }}`) inside the rendered YAML. + +### Property order + +JSON object key order is preserved at load time (`extractPropertyOrder` in `registry.go`) and re-exposed as `x-property-order` for the frontend. This means the order of fields in `json_schema.properties` is the order they appear in the form and the order their values are emitted by `renderProps`. + +## JSON Schema extensions + +Standard JSON Schema fields (`type`, `properties`, `required`, `enum`, `default`, `description`) work as expected. The following `x-*` extensions are also recognised. Most are interpreted by both the backend renderer and the frontend form — keep them in sync. + +### Form structure + +| Key | Type | Meaning | +|---|---|---| +| `x-step` | `"connector"` \| `"source"` \| `"explorer"` | Which form step a property belongs to. Connector-step props are rendered into the connector YAML; source-step props into the model YAML. Explorer-step props are accessed directly as template variables (e.g. `[[ .sql ]]`) and are not emitted by `renderProps`. Props without `x-step` go into both connector and source outputs. | +| `x-grouped-fields` | `{ enumValue: [...] }` | When the parent property is a radio/select with enum values, lists the fields shown when each value is selected. Used by the frontend to scope visibility to the active branch. | +| `x-visible-if` | `{ otherKey: [allowedValue, ...] }` | Conditional visibility: show this field only when the named field has one of the listed values. | +| `x-tab-group` | `{ tabName: [fieldKey, ...] }` | Groups fields under named tabs in the form. | +| `x-form-height` | `"tall"` | Frontend hint to use a taller form layout (currently used by Snowflake and Salesforce). | +| `x-form-width` | `"wide"` | Frontend hint to use a wider form layout. | +| `x-category` | `"objectStore"` \| `"warehouse"` \| `"sqlStore"` \| `"olap"` \| `"sourceOnly"` | Connector category, used to organise the picker. | + +### Field display + +| Key | Type | Meaning | +|---|---|---| +| `x-display` | `"radio"` \| `"select"` \| `"file"` \| ... | Override the default input control. | +| `x-select-style` | `"rich"` | Render a select with icons / descriptions instead of plain options. | +| `x-placeholder` | string | Placeholder text for text inputs. | +| `x-hint` | string | Helper text shown beneath the input. | +| `x-enum-labels` | array | Display labels for `enum` values, in the same order. | +| `x-enum-descriptions` | array | Sub-labels / descriptions for `enum` values. | +| `x-button-labels` | nested object | Custom labels for nested radio groups (see `clickhouse.json` for an example). | +| `x-informational` | bool | Marks a read-only / explanatory field that is not part of the rendered output. | + +### File uploads + +| Key | Type | Meaning | +|---|---|---| +| `x-file-accept` | string | Comma-separated `accept` filter for file inputs (e.g. `".json"`, `".pem,.p8"`). | +| `x-file-encoding` | `"base64"` \| `"json"` | How the uploaded file should be encoded before submission. | +| `x-file-extract` | `{ schemaKey: jsonPath }` | When the upload is a JSON file, populate other form fields from values inside it (used by BigQuery's service-account flow). | + +### Backend / rendering + +| Key | Type | Meaning | +|---|---|---| +| `x-secret` | bool | Value is a credential. The renderer extracts it to a `.env` variable and emits `{{ .env. }}` in the YAML. | +| `x-env-var` | string | Override the default env var name (`_`). When the chosen name already exists in `.env`, the renderer appends a numeric suffix (`FOO`, `FOO_1`, `FOO_2`, ...). | +| `x-omit-if-default` | bool | Skip rendering the property when its value equals the schema `default`. Replaces the per-driver "skip if `key == "managed"`" hack used in v1. | +| `x-ui-only` | bool | Property exists only to drive the form (e.g. an `auth_method` radio that selects between branches). It is never written to the rendered YAML. | + +## Template helpers + +`text/template` data passed to each file includes: + +- `.driver`, `.connector_name`, `.docs_url`, `.model_name` — basics. +- `.` — every property declared in `json_schema.properties` is exposed as a top-level key. Empty values fall back to the schema `default` if defined, otherwise to `""`. This means conditional template branches can rely on the key existing. +- `.props`, `.config_props`, `.source_props` — pre-processed `[]ProcessedProp` slices for use with `renderProps`. `config_props` and `source_props` correspond to `x-step: connector` and `x-step: source`; `.props` is an alias for `config_props`. + +Functions registered in `funcmap.go`: + +| Name | Purpose | +|---|---| +| `renderProps` | Renders a `[]ProcessedProp` as YAML key/value lines, quoting strings/secrets and leaving numbers and booleans bare. | +| `propVal` | Looks up a single property value by key (used inside conditional branches). | +| `default` | Positional `[[ default (expr) "fallback" ]]` — returns the fallback when the value is empty. **Do not use pipeline syntax** (`[[ expr \| default "fallback" ]]`); `text/template` pipes into the last argument and would swap the values. | +| `indent`, `quote` | String helpers. | +| `duckdbSQL` | Maps a file path to the appropriate DuckDB `read_*` call based on the extension. | +| `s3ToHTTPS`, `gcsToHTTPS` | Convert `s3://` / `gs://` URIs to HTTPS for ClickHouse's `s3()` / `gcs()` functions. | +| `azureContainer`, `azureBlobPath`, `azureEndpoint` | Decompose Azure URIs into the parts ClickHouse's `azureBlobStorage()` function expects. | +| `clickhouseFormat`, `clickhouseURLSuffix` | Map URL extensions to ClickHouse input formats and emit the optional `, Format, headers(...)` suffix when custom HTTPS headers are present. | + +See `funcmap.go` for the exact behaviour of each function and `render_test.go` for golden-output examples. + +## Authoring a new template + +1. Create a JSON file under the appropriate group (`olap/`, `duckdb-models/`, `clickhouse-models/`). +2. Write the JSON Schema for the form. Use `x-step` to split fields between the connector and model steps when the form is multi-step. +3. Add the `files` entries with `path_template` and `code_template` (or `code_template_file`). +4. Add a golden test case in `../render_test.go`. +5. Run `go test ./runtime/templates/...` to verify. diff --git a/runtime/templates/definitions/clickhouse-models/azure-clickhouse.json b/runtime/templates/definitions/clickhouse-models/azure-clickhouse.json new file mode 100644 index 000000000000..011a6cffa9be --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/azure-clickhouse.json @@ -0,0 +1,147 @@ +{ + "name": "azure-clickhouse", + "display_name": "Azure Blob Storage", + "description": "Read Azure Blob Storage files into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/azure", + "driver": "azure", + "olap": "clickhouse", + "icon": "MicrosoftAzureBlobStorage", + "small_icon": "MicrosoftAzureBlobStorageIcon", + "tags": [ + "azure", + "microsoft", + "object-storage", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to Azure Blob Storage", + "enum": [ + "connection_string", + "account_key", + "public" + ], + "default": "connection_string", + "x-display": "radio", + "x-enum-labels": [ + "Connection String", + "Storage Account Key", + "Public" + ], + "x-enum-descriptions": [ + "Provide a full Azure Storage connection string.", + "Provide the storage account name and access key.", + "No credentials needed; the data is publicly accessible." + ], + "x-ui-only": true, + "x-grouped-fields": { + "connection_string": [ + "azure_storage_connection_string" + ], + "account_key": [ + "azure_storage_account", + "azure_storage_key" + ], + "public": [] + } + }, + "azure_storage_connection_string": { + "type": "string", + "title": "Connection string", + "description": "Paste an Azure Storage connection string", + "x-placeholder": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_CONNECTION_STRING", + "x-visible-if": { + "auth_method": "connection_string" + } + }, + "azure_storage_account": { + "type": "string", + "title": "Storage account", + "description": "Azure storage account name", + "x-placeholder": "mystorageaccount", + "x-visible-if": { + "auth_method": "account_key" + } + }, + "azure_storage_key": { + "type": "string", + "title": "Storage key", + "description": "Azure storage account key", + "x-placeholder": "Enter storage key", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_KEY", + "x-visible-if": { + "auth_method": "account_key" + } + }, + "path": { + "type": "string", + "title": "Azure URI", + "description": "Path to your Azure container and blob", + "pattern": "^(azure://|https?://).+", + "errorMessage": { + "pattern": "Must be an Azure URI (e.g. azure://container/path or https://account.blob.core.windows.net/container/path)" + }, + "x-placeholder": "azure://container/path" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "connection_string" + } + } + }, + "then": { + "required": [ + "azure_storage_connection_string" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "account_key" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_key" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |[[ if eq .auth_method \"connection_string\" ]]\n SELECT * FROM azureBlobStorage(\n '[[ propVal .props \"azure_storage_connection_string\" ]]',\n '[[ azureContainer .path ]]',\n '[[ azureBlobPath .path ]]'\n )[[ else if eq .auth_method \"account_key\" ]]\n SELECT * FROM azureBlobStorage(\n '[[ azureEndpoint .path .azure_storage_account ]]',\n '[[ azureContainer .path ]]',\n '[[ azureBlobPath .path ]]',\n '[[ propVal .props \"azure_storage_account\" ]]',\n '[[ propVal .props \"azure_storage_key\" ]]'\n )[[ else ]]\n SELECT * FROM azureBlobStorage(\n '[[ azureEndpoint .path \"\" ]]',\n '[[ azureContainer .path ]]',\n '[[ azureBlobPath .path ]]'\n )[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/delta-clickhouse.json b/runtime/templates/definitions/clickhouse-models/delta-clickhouse.json new file mode 100644 index 000000000000..9032cb4c9427 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/delta-clickhouse.json @@ -0,0 +1,69 @@ +{ + "name": "delta-clickhouse", + "display_name": "Delta Lake", + "description": "Query Delta Lake tables in ClickHouse using deltaLake table function", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/delta", + "driver": "delta", + "olap": "clickhouse", + "icon": "DeltaLake", + "small_icon": "DeltaLakeIcon", + "tags": [ + "delta", + "table-format", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY" + }, + "path": { + "type": "string", + "title": "Delta Table Path", + "description": "S3 path to your Delta table", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/delta_table)" + }, + "x-placeholder": "s3://bucket/delta_table" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_delta_table" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM deltaLake(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/gcs-clickhouse.json b/runtime/templates/definitions/clickhouse-models/gcs-clickhouse.json new file mode 100644 index 000000000000..444d0b5de74a --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/gcs-clickhouse.json @@ -0,0 +1,118 @@ +{ + "name": "gcs-clickhouse", + "display_name": "Google Cloud Storage", + "description": "Read GCS files into ClickHouse using table functions (HMAC keys)", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/gcs", + "driver": "gcs", + "olap": "clickhouse", + "icon": "GoogleCloudStorage", + "small_icon": "GoogleCloudStorageIcon", + "tags": [ + "gcs", + "google", + "object-storage", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to GCS", + "enum": [ + "hmac_key", + "public" + ], + "default": "hmac_key", + "x-display": "radio", + "x-enum-labels": [ + "HMAC Key", + "Public" + ], + "x-enum-descriptions": [ + "Provide GCS HMAC key credentials.", + "No credentials needed; the data is publicly accessible." + ], + "x-ui-only": true, + "x-grouped-fields": { + "hmac_key": [ + "key_id", + "secret" + ], + "public": [] + } + }, + "key_id": { + "type": "string", + "title": "HMAC Access Key", + "description": "GCS HMAC access key ID", + "x-placeholder": "Enter HMAC access key", + "x-secret": true, + "x-env-var": "GCP_ACCESS_KEY_ID", + "x-visible-if": { + "auth_method": "hmac_key" + } + }, + "secret": { + "type": "string", + "title": "HMAC Secret", + "description": "GCS HMAC secret key", + "x-placeholder": "Enter HMAC secret", + "x-secret": true, + "x-env-var": "GCP_SECRET_ACCESS_KEY", + "x-visible-if": { + "auth_method": "hmac_key" + } + }, + "path": { + "type": "string", + "title": "GCS URI", + "description": "Path to your GCS bucket or prefix", + "pattern": "^(gs://|https?://).+", + "errorMessage": { + "pattern": "Must be a GCS URI (e.g. gs://bucket/path or https://storage.googleapis.com/bucket/path)" + }, + "x-placeholder": "gs://bucket/path" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "hmac_key" + } + } + }, + "then": { + "required": [ + "key_id", + "secret" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |[[ if eq .auth_method \"hmac_key\" ]]\n SELECT * FROM gcs(\n '[[ gcsToHTTPS .path ]]',\n '[[ propVal .props \"key_id\" ]]',\n '[[ propVal .props \"secret\" ]]'\n )[[ else ]]\n SELECT * FROM gcs('[[ gcsToHTTPS .path ]]')[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/https-clickhouse.json b/runtime/templates/definitions/clickhouse-models/https-clickhouse.json new file mode 100644 index 000000000000..694832e169ae --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/https-clickhouse.json @@ -0,0 +1,62 @@ +{ + "name": "https-clickhouse", + "display_name": "HTTP(S) Endpoint", + "description": "Read files from HTTP/HTTPS URLs into ClickHouse", + "docs_url": "https://docs.rilldata.com/developers/build/connect/#adding-a-remote-source", + "driver": "https", + "olap": "clickhouse", + "icon": "HTTPS", + "small_icon": "HTTPSIcon", + "tags": [ + "https", + "http", + "url", + "clickhouse", + "source" + ], + + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + + "properties": { + "headers": { + "title": "Headers", + "description": "HTTP headers to include in the request", + "x-display": "key-value", + "x-placeholder": "Header name", + "x-hint": "e.g. Authorization: Bearer " + }, + + "path": { + "type": "string", + "title": "URL", + "description": "HTTP(S) URL to the file", + "pattern": "^https?://.+", + "errorMessage": { + "pattern": "Must be a valid HTTP(S) URL" + }, + "x-placeholder": "https://example.com/data.csv" + }, + + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + + "required": ["path", "name"] + }, + + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\n\ntype: model\nmaterialize: true\n\nconnector: clickhouse\n\nsql: |\n SELECT * FROM url('[[ .path ]]'[[ clickhouseURLSuffix .path .props ]])\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/hudi-clickhouse.json b/runtime/templates/definitions/clickhouse-models/hudi-clickhouse.json new file mode 100644 index 000000000000..9662ba663bc8 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/hudi-clickhouse.json @@ -0,0 +1,69 @@ +{ + "name": "hudi-clickhouse", + "display_name": "Hudi", + "description": "Query Apache Hudi tables in ClickHouse using hudi table function", + "docs_url": "https://clickhouse.com/docs/en/sql-reference/table-functions/hudi", + "driver": "hudi", + "olap": "clickhouse", + "icon": "ApacheHudi", + "small_icon": "ApacheHudiIcon", + "tags": [ + "hudi", + "table-format", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Hudi table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Hudi table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY" + }, + "path": { + "type": "string", + "title": "Hudi Table Path", + "description": "S3 path to your Hudi table", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/hudi_table)" + }, + "x-placeholder": "s3://bucket/hudi_table" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_hudi_table" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM hudi(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/iceberg-clickhouse.json b/runtime/templates/definitions/clickhouse-models/iceberg-clickhouse.json new file mode 100644 index 000000000000..6dc5010c6237 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/iceberg-clickhouse.json @@ -0,0 +1,69 @@ +{ + "name": "iceberg-clickhouse", + "display_name": "Iceberg", + "description": "Query Apache Iceberg tables in ClickHouse using icebergS3 table function", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/iceberg", + "driver": "iceberg", + "olap": "clickhouse", + "icon": "ApacheIceberg", + "small_icon": "ApacheIcebergIcon", + "tags": [ + "iceberg", + "table-format", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY" + }, + "path": { + "type": "string", + "title": "Iceberg Table Path", + "description": "S3 path to your Iceberg table metadata", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/warehouse/my_table)" + }, + "x-placeholder": "s3://bucket/warehouse/my_table" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_iceberg_table" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM icebergS3(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/kafka-clickhouse.json b/runtime/templates/definitions/clickhouse-models/kafka-clickhouse.json new file mode 100644 index 000000000000..c3a36ae77419 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/kafka-clickhouse.json @@ -0,0 +1,76 @@ +{ + "name": "kafka-clickhouse", + "display_name": "Kafka", + "description": "Read Kafka topics into ClickHouse using Kafka table engine", + "docs_url": "https://clickhouse.com/docs/en/engines/table-engines/integrations/kafka", + "driver": "kafka", + "olap": "clickhouse", + "icon": "ApacheKafka", + "small_icon": "ApacheKafkaIcon", + "tags": [ + "kafka", + "streaming", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "broker_list": { + "type": "string", + "title": "Broker List", + "description": "Comma-separated list of Kafka brokers", + "x-placeholder": "broker1:9092,broker2:9092" + }, + "topic": { + "type": "string", + "title": "Topic", + "description": "Kafka topic to consume from", + "x-placeholder": "my_topic" + }, + "group_name": { + "type": "string", + "title": "Consumer Group", + "description": "Kafka consumer group name", + "x-placeholder": "rill_consumer_group" + }, + "format": { + "type": "string", + "title": "Message Format", + "description": "Format of messages in the Kafka topic", + "enum": [ + "JSONEachRow", + "CSV", + "TSV", + "Avro", + "Parquet" + ], + "default": "JSONEachRow", + "x-display": "select" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "broker_list", + "topic", + "group_name", + "format", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM kafka(\n '[[ .broker_list ]]',\n '[[ .topic ]]',\n '[[ .group_name ]]',\n '[[ .format ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/mongodb-clickhouse.json b/runtime/templates/definitions/clickhouse-models/mongodb-clickhouse.json new file mode 100644 index 000000000000..0f405faa94e9 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/mongodb-clickhouse.json @@ -0,0 +1,87 @@ +{ + "name": "mongodb-clickhouse", + "display_name": "MongoDB", + "description": "Query MongoDB collections in ClickHouse using mongodb table function", + "docs_url": "https://clickhouse.com/docs/en/sql-reference/table-functions/mongodb", + "driver": "mongodb", + "olap": "clickhouse", + "icon": "MongoDB", + "small_icon": "MongoDBIcon", + "tags": [ + "mongodb", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "MongoDB server hostname or IP", + "x-placeholder": "localhost" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MongoDB server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "27017", + "x-placeholder": "27017" + }, + "database": { + "type": "string", + "title": "Database", + "description": "MongoDB database name", + "x-placeholder": "my_database" + }, + "collection": { + "type": "string", + "title": "Collection", + "description": "MongoDB collection name", + "x-placeholder": "my_collection" + }, + "user": { + "type": "string", + "title": "Username", + "description": "MongoDB user", + "x-placeholder": "mongo_user" + }, + "password": { + "type": "string", + "title": "Password", + "description": "MongoDB password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "MONGODB_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "database", + "collection", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM mongodb(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"27017\" ]]',\n '[[ .database ]]',\n '[[ .collection ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/mysql-clickhouse.json b/runtime/templates/definitions/clickhouse-models/mysql-clickhouse.json new file mode 100644 index 000000000000..f7438e521415 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/mysql-clickhouse.json @@ -0,0 +1,87 @@ +{ + "name": "mysql-clickhouse", + "display_name": "MySQL", + "description": "Read MySQL tables into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/mysql", + "driver": "mysql", + "olap": "clickhouse", + "icon": "MySQL", + "small_icon": "MySqlIcon", + "tags": [ + "mysql", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "MySQL server hostname or IP", + "x-placeholder": "localhost" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MySQL server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "3306", + "x-placeholder": "3306" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "my_database" + }, + "table": { + "type": "string", + "title": "Table", + "description": "Table name to query", + "x-placeholder": "my_table" + }, + "user": { + "type": "string", + "title": "Username", + "description": "MySQL user", + "x-placeholder": "mysql" + }, + "password": { + "type": "string", + "title": "Password", + "description": "MySQL password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "MYSQL_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "database", + "table", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM mysql(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"3306\" ]]',\n '[[ .database ]]',\n '[[ .table ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/postgres-clickhouse.json b/runtime/templates/definitions/clickhouse-models/postgres-clickhouse.json new file mode 100644 index 000000000000..97eac39a41ff --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/postgres-clickhouse.json @@ -0,0 +1,88 @@ +{ + "name": "postgres-clickhouse", + "display_name": "PostgreSQL", + "description": "Read PostgreSQL tables into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "postgres", + "olap": "clickhouse", + "icon": "Postgres", + "small_icon": "PostgresIcon", + "tags": [ + "postgres", + "postgresql", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "Postgres server hostname or IP", + "x-placeholder": "localhost" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Postgres server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "5432", + "x-placeholder": "5432" + }, + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres" + }, + "table": { + "type": "string", + "title": "Table", + "description": "Table name to query", + "x-placeholder": "my_table" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Postgres user", + "x-placeholder": "postgres" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Postgres password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "POSTGRES_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "dbname", + "table", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM postgresql(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"5432\" ]]',\n '[[ .dbname ]]',\n '[[ .table ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/s3-clickhouse.json b/runtime/templates/definitions/clickhouse-models/s3-clickhouse.json new file mode 100644 index 000000000000..5f92243b79b8 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/s3-clickhouse.json @@ -0,0 +1,118 @@ +{ + "name": "s3-clickhouse", + "display_name": "Amazon S3", + "description": "Read S3 files into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/s3", + "driver": "s3", + "olap": "clickhouse", + "icon": "AmazonS3", + "small_icon": "AmazonS3Icon", + "tags": [ + "s3", + "aws", + "object-storage", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to S3", + "enum": [ + "access_key", + "public" + ], + "default": "access_key", + "x-display": "radio", + "x-enum-labels": [ + "Access Key", + "Public" + ], + "x-enum-descriptions": [ + "Provide AWS access key credentials.", + "No credentials needed; the data is publicly accessible." + ], + "x-ui-only": true, + "x-grouped-fields": { + "access_key": [ + "aws_access_key_id", + "aws_secret_access_key" + ], + "public": [] + } + }, + "aws_access_key_id": { + "type": "string", + "title": "Access Key ID", + "description": "AWS access key ID for the bucket", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-visible-if": { + "auth_method": "access_key" + } + }, + "aws_secret_access_key": { + "type": "string", + "title": "Secret Access Key", + "description": "AWS secret access key for the bucket", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-visible-if": { + "auth_method": "access_key" + } + }, + "path": { + "type": "string", + "title": "S3 URI", + "description": "Path to your S3 bucket or prefix", + "pattern": "^(s3://|https?://).+", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/path or https://bucket.s3.amazonaws.com/path)" + }, + "x-placeholder": "s3://bucket/path" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "access_key" + } + } + }, + "then": { + "required": [ + "aws_access_key_id", + "aws_secret_access_key" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |[[ if eq .auth_method \"access_key\" ]]\n SELECT * FROM s3(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )[[ else ]]\n SELECT * FROM s3('[[ s3ToHTTPS .path ]]')[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/supabase-clickhouse.json b/runtime/templates/definitions/clickhouse-models/supabase-clickhouse.json new file mode 100644 index 000000000000..9ede0b9a27dc --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/supabase-clickhouse.json @@ -0,0 +1,88 @@ +{ + "name": "supabase-clickhouse", + "display_name": "Supabase", + "description": "Read Supabase tables into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "supabase", + "olap": "clickhouse", + "icon": "Supabase", + "small_icon": "SupabaseIcon", + "tags": [ + "supabase", + "postgres", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "Supabase database host", + "x-placeholder": "aws-0-[region].pooler.supabase.com" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Supabase database port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "5432", + "x-placeholder": "5432" + }, + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres" + }, + "table": { + "type": "string", + "title": "Table", + "description": "Table name to query", + "x-placeholder": "my_table" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Supabase database user", + "x-placeholder": "postgres.[ref]" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Supabase database password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SUPABASE_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "dbname", + "table", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM postgresql(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"5432\" ]]',\n '[[ .dbname ]]',\n '[[ .table ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/athena-duckdb.json b/runtime/templates/definitions/duckdb-models/athena-duckdb.json new file mode 100644 index 000000000000..372babbe7faf --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/athena-duckdb.json @@ -0,0 +1,88 @@ +{ + "name": "athena-duckdb", + "display_name": "Amazon Athena", + "description": "Query Amazon Athena and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/athena", + "driver": "athena", + "olap": "duckdb", + "icon": "AmazonAthena", + "small_icon": "AthenaIcon", + "tags": [ + "athena", + "aws", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS access key ID", + "description": "AWS access key ID used to authenticate to Athena", + "x-placeholder": "your_access_key_id", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS secret access key", + "description": "AWS secret access key paired with the access key ID", + "x-placeholder": "your_secret_access_key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector" + }, + "output_location": { + "type": "string", + "title": "S3 output location", + "description": "S3 URI where Athena should write query results (e.g., s3://bucket/path/)", + "pattern": "^s3://.+", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g., s3://bucket/path/)" + }, + "x-placeholder": "s3://bucket-name/path/", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "output_location", + "sql", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: athena\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/athena\n\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/azure-duckdb.json b/runtime/templates/definitions/duckdb-models/azure-duckdb.json new file mode 100644 index 000000000000..49040dd0a36b --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/azure-duckdb.json @@ -0,0 +1,196 @@ +{ + "name": "azure-duckdb", + "display_name": "Azure Blob Storage", + "description": "Read files from Azure Blob Storage into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/azure", + "driver": "azure", + "olap": "duckdb", + "icon": "MicrosoftAzureBlobStorage", + "small_icon": "MicrosoftAzureBlobStorageIcon", + "tags": [ + "azure", + "microsoft", + "object-storage", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to Azure Blob Storage", + "enum": [ + "connection_string", + "account_key", + "sas_token", + "public" + ], + "default": "connection_string", + "x-display": "radio", + "x-enum-labels": [ + "Connection String", + "Storage Account Key", + "SAS Token", + "Public" + ], + "x-enum-descriptions": [ + "Provide a full Azure Storage connection string.", + "Provide the storage account name and access key.", + "Provide the storage account name and SAS token.", + "Access publicly readable blobs without credentials." + ], + "x-ui-only": true, + "x-grouped-fields": { + "connection_string": [ + "azure_storage_connection_string" + ], + "account_key": [ + "azure_storage_account", + "azure_storage_key" + ], + "sas_token": [ + "azure_storage_account", + "azure_storage_sas_token" + ], + "public": [] + }, + "x-step": "connector" + }, + "azure_storage_connection_string": { + "type": "string", + "title": "Connection string", + "description": "Paste an Azure Storage connection string", + "x-placeholder": "Enter Azure storage connection string", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_CONNECTION_STRING", + "x-step": "connector", + "x-visible-if": { + "auth_method": "connection_string" + } + }, + "azure_storage_account": { + "type": "string", + "title": "Storage account", + "description": "The name of the Azure storage account", + "x-placeholder": "Enter Azure storage account", + "x-step": "connector", + "x-visible-if": { + "auth_method": [ + "account_key", + "sas_token" + ] + } + }, + "azure_storage_key": { + "type": "string", + "title": "Access key", + "description": "Primary or secondary access key for the storage account", + "x-placeholder": "Enter Azure storage access key", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_KEY", + "x-step": "connector", + "x-visible-if": { + "auth_method": "account_key" + } + }, + "azure_storage_sas_token": { + "type": "string", + "title": "SAS token", + "description": "Shared Access Signature token for the storage account (starting with ?sv=)", + "x-placeholder": "Enter Azure SAS token", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_SAS_TOKEN", + "x-step": "connector", + "x-visible-if": { + "auth_method": "sas_token" + } + }, + "path": { + "type": "string", + "title": "Blob URI", + "description": "URI to the Azure blob container or directory", + "pattern": "^(azure://|https?://).+", + "errorMessage": { + "pattern": "Must be an Azure URI (e.g. azure://container/path or https://account.blob.core.windows.net/container/path)" + }, + "x-placeholder": "azure://container/path", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "connection_string" + } + } + }, + "then": { + "required": [ + "azure_storage_connection_string" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "account_key" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_key" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "sas_token" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_sas_token" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: azure\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path false ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/bigquery-duckdb.json b/runtime/templates/definitions/duckdb-models/bigquery-duckdb.json new file mode 100644 index 000000000000..44f2085c387f --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/bigquery-duckdb.json @@ -0,0 +1,81 @@ +{ + "name": "bigquery-duckdb", + "display_name": "BigQuery", + "description": "Query Google BigQuery and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/bigquery", + "driver": "bigquery", + "olap": "duckdb", + "icon": "GoogleBigQuery", + "small_icon": "GoogleBigQueryIcon", + "tags": [ + "bigquery", + "google", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "google_application_credentials": { + "type": "string", + "title": "GCP credentials", + "description": "Service account JSON (uploaded or pasted)", + "format": "file", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-file-extract": { + "project_id": "project_id" + }, + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-step": "connector" + }, + "project_id": { + "type": "string", + "title": "Project ID", + "description": "Google Cloud project ID to use for queries", + "x-placeholder": "my-project", + "x-hint": "If empty, Rill will use the project ID from your credentials when available.", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "google_application_credentials", + "project_id", + "sql", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: bigquery\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/bigquery\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json b/runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json new file mode 100644 index 000000000000..b4e018ce9421 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json @@ -0,0 +1,50 @@ +{ + "name": "clickhouse-duckdb", + "display_name": "ClickHouse", + "description": "Query ClickHouse tables and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/clickhouse", + "driver": "clickhouse", + "olap": "duckdb", + "icon": "ClickHouse", + "small_icon": "ClickHouseIcon", + "tags": [ + "clickhouse", + "olap", + "duckdb", + "source", + "model" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + "properties": { + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against ClickHouse", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/clickhouse\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/delta-duckdb.json b/runtime/templates/definitions/duckdb-models/delta-duckdb.json new file mode 100644 index 000000000000..f0d2adf9b300 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/delta-duckdb.json @@ -0,0 +1,193 @@ +{ + "name": "delta-duckdb", + "display_name": "Delta Lake", + "description": "Query Delta Lake tables using DuckDB's delta extension", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/delta", + "driver": "delta", + "olap": "duckdb", + "icon": "DeltaLake", + "small_icon": "DeltaLakeIcon", + "tags": [ + "delta", + "table-format", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "storage_type": { + "type": "string", + "title": "Storage type", + "description": "Where your Delta table is stored", + "enum": [ + "s3", + "gcs", + "azure", + "local" + ], + "default": "s3", + "x-display": "radio", + "x-enum-labels": [ + "Amazon S3", + "Google Cloud Storage", + "Azure Blob Storage", + "Local / Public" + ], + "x-enum-descriptions": [ + "Delta table stored in an S3 bucket.", + "Delta table stored in a GCS bucket.", + "Delta table stored in Azure Blob Storage.", + "Local path or publicly accessible URL." + ], + "x-ui-only": true, + "x-grouped-fields": { + "s3": [ + "aws_access_key_id", + "aws_secret_access_key", + "aws_region" + ], + "gcs": [ + "google_application_credentials" + ], + "azure": [ + "azure_storage_account", + "azure_storage_key" + ], + "local": [] + }, + "x-step": "connector" + }, + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-visible-if": { + "storage_type": "s3" + }, + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-visible-if": { + "storage_type": "s3" + }, + "x-step": "connector" + }, + "aws_region": { + "type": "string", + "title": "AWS Region", + "description": "AWS region of the S3 bucket", + "x-placeholder": "us-east-1", + "x-visible-if": { + "storage_type": "s3" + }, + "x-step": "connector" + }, + "google_application_credentials": { + "type": "string", + "title": "Service Account Key (JSON)", + "description": "Upload a JSON key file for a service account with GCS access", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-visible-if": { + "storage_type": "gcs" + }, + "x-step": "connector" + }, + "azure_storage_account": { + "type": "string", + "title": "Storage Account Name", + "description": "Azure storage account name", + "x-placeholder": "mystorageaccount", + "x-visible-if": { + "storage_type": "azure" + }, + "x-step": "connector" + }, + "azure_storage_key": { + "type": "string", + "title": "Storage Account Key", + "description": "Azure storage account access key", + "x-placeholder": "Enter storage account key", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_KEY", + "x-visible-if": { + "storage_type": "azure" + }, + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "Delta Table Path", + "description": "Path to your Delta table (e.g. s3://bucket/delta_table, gs://bucket/delta_table)", + "x-placeholder": "s3://bucket/delta_table", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_delta_table", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "storage_type": { + "const": "s3" + } + } + }, + "then": { + "required": [ + "aws_access_key_id", + "aws_secret_access_key" + ] + } + }, + { + "if": { + "properties": { + "storage_type": { + "const": "azure" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_key" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\n\nsql: |\n SELECT * FROM delta_scan('[[ .path ]]');\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/duckdb-duckdb.json b/runtime/templates/definitions/duckdb-models/duckdb-duckdb.json new file mode 100644 index 000000000000..f5612f7948e9 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/duckdb-duckdb.json @@ -0,0 +1,47 @@ +{ + "name": "duckdb-duckdb", + "display_name": "DuckDB", + "description": "Write SQL models against DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/duckdb", + "driver": "duckdb", + "olap": "duckdb", + "icon": "DuckDB", + "small_icon": "DuckDBIcon", + "tags": [ + "duckdb", + "model" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + "properties": { + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/duckdb\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/gcs-duckdb.json b/runtime/templates/definitions/duckdb-models/gcs-duckdb.json new file mode 100644 index 000000000000..7a47a9d0dd8f --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/gcs-duckdb.json @@ -0,0 +1,164 @@ +{ + "name": "gcs-duckdb", + "display_name": "Google Cloud Storage", + "description": "Read files from Google Cloud Storage into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/gcs", + "driver": "gcs", + "olap": "duckdb", + "icon": "GoogleCloudStorage", + "small_icon": "GoogleCloudStorageIcon", + "tags": [ + "gcs", + "google", + "object-storage", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to GCS", + "enum": [ + "credentials", + "hmac", + "public" + ], + "default": "credentials", + "x-display": "radio", + "x-enum-labels": [ + "GCP credentials", + "HMAC keys", + "Public" + ], + "x-enum-descriptions": [ + "Upload a JSON key file for a service account with GCS access.", + "Use HMAC access key and secret for S3-compatible authentication.", + "Access publicly readable buckets without credentials." + ], + "x-ui-only": true, + "x-grouped-fields": { + "credentials": [ + "google_application_credentials" + ], + "hmac": [ + "key_id", + "secret" + ], + "public": [] + }, + "x-step": "connector" + }, + "google_application_credentials": { + "type": "string", + "title": "Service account key", + "description": "Upload a JSON key file for a service account with GCS access.", + "format": "file", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-step": "connector", + "x-visible-if": { + "auth_method": "credentials" + } + }, + "key_id": { + "type": "string", + "title": "Access Key ID", + "description": "HMAC access key ID for S3-compatible authentication", + "x-placeholder": "Enter your HMAC access key ID", + "x-secret": true, + "x-env-var": "GCP_ACCESS_KEY_ID", + "x-step": "connector", + "x-visible-if": { + "auth_method": "hmac" + } + }, + "secret": { + "type": "string", + "title": "Secret Access Key", + "description": "HMAC secret access key for S3-compatible authentication", + "x-placeholder": "Enter your HMAC secret access key", + "x-secret": true, + "x-env-var": "GCP_SECRET_ACCESS_KEY", + "x-step": "connector", + "x-visible-if": { + "auth_method": "hmac" + } + }, + "path": { + "type": "string", + "title": "GCS URI", + "description": "Path to your GCS bucket or prefix", + "pattern": "^gs://[^/]+(/.*)?$", + "errorMessage": { + "pattern": "Must be a GS URI (e.g. gs://bucket/path)" + }, + "x-placeholder": "gs://bucket/path", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "credentials" + } + } + }, + "then": { + "required": [ + "google_application_credentials" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "hmac" + } + } + }, + "then": { + "required": [ + "key_id", + "secret" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: gcs\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path false ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/https-duckdb.json b/runtime/templates/definitions/duckdb-models/https-duckdb.json new file mode 100644 index 000000000000..477ce5a97cd0 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/https-duckdb.json @@ -0,0 +1,80 @@ +{ + "name": "https-duckdb", + "display_name": "HTTP(S) Endpoint", + "description": "Read files from an HTTP/HTTPS URL into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connect/#adding-a-remote-source", + "driver": "https", + "olap": "duckdb", + "icon": "HTTPS", + "small_icon": "HTTPSIcon", + "tags": [ + "https", + "http", + "url", + "file-store", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "fileStore", + "x-button-labels": { + "*": { + "*": { + "idle": "Continue", + "loading": "Continuing..." + } + } + }, + "properties": { + "headers": { + "title": "Headers", + "description": "HTTP headers to include in the request", + "x-display": "key-value", + "x-placeholder": "Header name", + "x-hint": "e.g. Authorization: Bearer ", + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "URI", + "description": "HTTP/HTTPS URL to the remote file", + "pattern": "^https?://.+", + "errorMessage": { + "pattern": "URI must start with http:// or https://" + }, + "x-placeholder": "https://example.com/file.csv", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "errorMessage": { + "pattern": "Model name can only contain letters, numbers, and underscores" + }, + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: https\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path true ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/iceberg-duckdb.json b/runtime/templates/definitions/duckdb-models/iceberg-duckdb.json new file mode 100644 index 000000000000..563b15657433 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/iceberg-duckdb.json @@ -0,0 +1,78 @@ +{ + "name": "iceberg-duckdb", + "display_name": "Iceberg", + "description": "Query Apache Iceberg tables using DuckDB's iceberg extension", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/iceberg", + "driver": "iceberg", + "olap": "duckdb", + "icon": "ApacheIceberg", + "small_icon": "ApacheIcebergIcon", + "tags": [ + "iceberg", + "table-format", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector" + }, + "aws_region": { + "type": "string", + "title": "AWS Region", + "description": "AWS region of the S3 bucket", + "x-placeholder": "us-east-1", + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "Iceberg Table Path", + "description": "S3 path to your Iceberg table metadata", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/warehouse/my_table)" + }, + "x-placeholder": "s3://bucket/warehouse/my_table", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_iceberg_table", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\n\nsql: |\n SELECT * FROM iceberg_scan('[[ .path ]]');\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/local-file-duckdb.json b/runtime/templates/definitions/duckdb-models/local-file-duckdb.json new file mode 100644 index 000000000000..a32c76ae7d51 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/local-file-duckdb.json @@ -0,0 +1,49 @@ +{ + "name": "local_file-duckdb", + "display_name": "Local File", + "description": "Read local files into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/duckdb", + "driver": "local_file", + "olap": "duckdb", + "icon": "LocalFile", + "small_icon": "LocalFileIcon", + "tags": [ + "local-file", + "file", + "file-store", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "fileStore", + "properties": { + "path": { + "type": "string", + "title": "Path", + "description": "Local file path or glob (relative to project root)", + "x-placeholder": "data/*.parquet", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Source name", + "description": "Name for the source", + "x-placeholder": "my_new_source", + "x-step": "explorer" + } + }, + "required": [ + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/duckdb\ntype: model\nconnector: duckdb\nsql: |\n [[ duckdbSQL .path true ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/mysql-duckdb.json b/runtime/templates/definitions/duckdb-models/mysql-duckdb.json new file mode 100644 index 000000000000..adbcd40d26dc --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/mysql-duckdb.json @@ -0,0 +1,171 @@ +{ + "name": "mysql-duckdb", + "display_name": "MySQL", + "description": "Query MySQL databases and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/mysql", + "driver": "mysql", + "olap": "duckdb", + "icon": "MySQL", + "small_icon": "MySqlIcon", + "tags": [ + "mysql", + "database", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "database", + "user", + "password", + "ssl-mode" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "MySQL connection string", + "description": "Full DSN, e.g. mysql://user:password@host:3306/database?ssl-mode=REQUIRED", + "x-placeholder": "mysql://user:password@host:3306/database", + "x-secret": true, + "x-env-var": "MYSQL_DSN", + "x-hint": "Use DSN or fill host/user/password/database below (not both at once).", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "MySQL server hostname or IP", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MySQL server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "3306", + "x-placeholder": "3306", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "my_database", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "MySQL user", + "x-placeholder": "mysql", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "MySQL password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "MYSQL_PASSWORD", + "x-step": "connector" + }, + "ssl-mode": { + "type": "string", + "title": "SSL mode", + "description": "Use DISABLED, PREFERRED, or REQUIRED", + "enum": [ + "DISABLED", + "PREFERRED", + "REQUIRED" + ], + "x-placeholder": "PREFERRED", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your database", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + }, + "else": { + "required": [ + "host", + "database", + "user" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: mysql\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/mysql\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/postgres-duckdb.json b/runtime/templates/definitions/duckdb-models/postgres-duckdb.json new file mode 100644 index 000000000000..2a32814ad2fb --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/postgres-duckdb.json @@ -0,0 +1,173 @@ +{ + "name": "postgres-duckdb", + "display_name": "PostgreSQL", + "description": "Query PostgreSQL databases and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "postgres", + "olap": "duckdb", + "icon": "Postgres", + "small_icon": "PostgresIcon", + "tags": [ + "postgres", + "postgresql", + "database", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "user", + "password", + "dbname", + "sslmode" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Postgres connection string", + "description": "e.g. postgresql://user:password@host:5432/dbname?sslmode=require", + "x-placeholder": "postgresql://postgres:postgres@localhost:5432/postgres", + "x-secret": true, + "x-env-var": "POSTGRES_DSN", + "x-hint": "Use a DSN or provide host/user/password/dbname below (but not both).", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Postgres server hostname or IP", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Postgres server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "5432", + "x-placeholder": "5432", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Postgres user", + "x-placeholder": "postgres", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Postgres password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "POSTGRES_PASSWORD", + "x-step": "connector" + }, + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres", + "x-step": "connector" + }, + "sslmode": { + "type": "string", + "title": "SSL mode", + "description": "Use disable, allow, prefer, require", + "enum": [ + "disable", + "allow", + "prefer", + "require" + ], + "x-placeholder": "require", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your database", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + }, + "else": { + "required": [ + "host", + "user", + "dbname" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: postgres\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/postgres\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/redshift-duckdb.json b/runtime/templates/definitions/duckdb-models/redshift-duckdb.json new file mode 100644 index 000000000000..dc355e6b0c74 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/redshift-duckdb.json @@ -0,0 +1,105 @@ +{ + "name": "redshift-duckdb", + "display_name": "Amazon Redshift", + "description": "Query Amazon Redshift and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/redshift", + "driver": "redshift", + "olap": "duckdb", + "icon": "AmazonRedshift", + "small_icon": "RedshiftIcon", + "tags": [ + "redshift", + "aws", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS access key ID", + "description": "AWS access key ID", + "x-placeholder": "your_access_key_id", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS secret access key", + "description": "AWS secret access key", + "x-placeholder": "your_secret_access_key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector" + }, + "region": { + "type": "string", + "title": "AWS region", + "description": "AWS region (e.g. us-east-1)", + "x-placeholder": "us-east-1", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Redshift database name", + "x-placeholder": "dev", + "x-step": "connector" + }, + "workgroup": { + "type": "string", + "title": "Workgroup", + "description": "Redshift Serverless workgroup name", + "x-placeholder": "default", + "x-step": "connector" + }, + "cluster_identifier": { + "type": "string", + "title": "Cluster identifier", + "description": "Redshift cluster identifier (use when not using serverless)", + "x-placeholder": "redshift-cluster-1", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "database", + "sql", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: redshift\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/redshift\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/s3-duckdb.json b/runtime/templates/definitions/duckdb-models/s3-duckdb.json new file mode 100644 index 000000000000..3b6827fac7a9 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/s3-duckdb.json @@ -0,0 +1,164 @@ +{ + "name": "s3-duckdb", + "display_name": "Amazon S3", + "description": "Read files from Amazon S3 into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/s3", + "driver": "s3", + "olap": "duckdb", + "icon": "AmazonS3", + "small_icon": "AmazonS3Icon", + "tags": [ + "s3", + "aws", + "object-storage", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to S3", + "enum": [ + "access_keys", + "public" + ], + "default": "access_keys", + "x-display": "radio", + "x-enum-labels": [ + "Access keys", + "Public" + ], + "x-enum-descriptions": [ + "Use AWS access key ID and secret access key.", + "Access publicly readable buckets without credentials." + ], + "x-ui-only": true, + "x-grouped-fields": { + "access_keys": [ + "aws_access_key_id", + "aws_secret_access_key", + "region", + "endpoint", + "aws_role_arn" + ], + "public": [] + }, + "x-step": "connector" + }, + "aws_access_key_id": { + "type": "string", + "title": "Access Key ID", + "description": "AWS access key ID for the bucket", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "aws_secret_access_key": { + "type": "string", + "title": "Secret Access Key", + "description": "AWS secret access key for the bucket", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "region": { + "type": "string", + "title": "Region", + "description": "Rill uses your default AWS region unless you set it explicitly.", + "x-placeholder": "us-east-1", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "endpoint": { + "type": "string", + "title": "Endpoint", + "description": "Override the S3 endpoint (for S3-compatible services like R2/MinIO).", + "x-placeholder": "https://s3.example.com", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "aws_role_arn": { + "type": "string", + "title": "AWS Role ARN", + "description": "AWS Role ARN to assume", + "x-placeholder": "arn:aws:iam::123456789012:role/MyRole", + "x-secret": true, + "x-env-var": "AWS_ROLE_ARN", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "path": { + "type": "string", + "title": "S3 URI", + "description": "Path to your S3 bucket or prefix", + "pattern": "^s3://[^/]+(/.*)?$", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/path)" + }, + "x-placeholder": "s3://bucket/path", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "access_keys" + } + } + }, + "then": { + "required": [ + "aws_access_key_id", + "aws_secret_access_key" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: s3\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path false ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/salesforce-duckdb.json b/runtime/templates/definitions/duckdb-models/salesforce-duckdb.json new file mode 100644 index 000000000000..b088e7bee64d --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/salesforce-duckdb.json @@ -0,0 +1,142 @@ +{ + "name": "salesforce-duckdb", + "display_name": "Salesforce", + "description": "Query Salesforce objects and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/salesforce", + "driver": "salesforce", + "olap": "duckdb", + "icon": "Salesforce", + "small_icon": "SalesforceIcon", + "tags": [ + "salesforce", + "crm", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "x-form-height": "tall", + "properties": { + "soql": { + "type": "string", + "title": "SOQL", + "description": "SOQL query to extract data", + "x-placeholder": "SELECT Id, Name FROM Opportunity", + "x-step": "source" + }, + "sobject": { + "type": "string", + "title": "SObject", + "description": "Salesforce object to query", + "x-placeholder": "Opportunity", + "x-step": "source" + }, + "queryAll": { + "type": "boolean", + "title": "Query all", + "description": "Include deleted and archived records", + "default": false, + "x-step": "source" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Salesforce username (usually an email)", + "x-placeholder": "user@example.com" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Salesforce password, optionally followed by security token if required", + "x-placeholder": "your_password_or_password+token", + "x-secret": true, + "x-env-var": "SALESFORCE_PASSWORD" + }, + "key": { + "type": "string", + "title": "JWT private key", + "description": "PEM-formatted private key for JWT auth", + "x-display": "textarea", + "x-placeholder": "your_private_key", + "x-secret": true, + "x-env-var": "SALESFORCE_KEY" + }, + "client_id": { + "type": "string", + "title": "Connected App Client ID", + "description": "Client ID (consumer key) for JWT auth", + "x-placeholder": "Connected App client ID" + }, + "endpoint": { + "type": "string", + "title": "Login endpoint", + "description": "Salesforce login URL (e.g., login.salesforce.com or test.salesforce.com)", + "x-placeholder": "login.salesforce.com" + }, + "name": { + "type": "string", + "title": "Source name", + "description": "Name for the source", + "x-placeholder": "my_new_source", + "x-step": "source" + } + }, + "required": [ + "soql", + "sobject", + "name" + ], + "allOf": [ + { + "if": { + "not": { + "required": [ + "key" + ], + "properties": { + "key": { + "minLength": 1 + } + } + } + }, + "then": { + "required": [ + "username", + "password", + "endpoint" + ] + } + }, + { + "if": { + "required": [ + "key" + ], + "properties": { + "key": { + "minLength": 1 + } + } + }, + "then": { + "required": [ + "client_id", + "username" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/salesforce\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ renderProps .source_props -]]\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/snowflake-duckdb.json b/runtime/templates/definitions/duckdb-models/snowflake-duckdb.json new file mode 100644 index 000000000000..c1f970a23d2d --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/snowflake-duckdb.json @@ -0,0 +1,232 @@ +{ + "name": "snowflake-duckdb", + "display_name": "Snowflake", + "description": "Query Snowflake data warehouses and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/snowflake", + "driver": "snowflake", + "olap": "duckdb", + "icon": "Snowflake", + "small_icon": "SnowflakeIcon", + "tags": [ + "snowflake", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "x-form-height": "tall", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "password", + "private_key", + "dsn" + ], + "default": "password", + "x-display": "tabs", + "x-enum-labels": [ + "User/Password", + "Private Key", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "password": [ + "account", + "user", + "password", + "warehouse", + "database", + "schema", + "role" + ], + "private_key": [ + "account", + "user", + "privateKey", + "warehouse", + "database", + "schema", + "role" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "account": { + "type": "string", + "title": "Account identifier", + "description": "Snowflake account identifier (from your Snowflake URL, before .snowflakecomputing.com)", + "x-placeholder": "abc12345.us-east-1", + "x-hint": "e.g. abc12345 or abc12345.us-east-1 \u2014 don't include https://", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Snowflake username", + "x-placeholder": "your_username", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Snowflake password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PASSWORD", + "x-visible-if": { + "auth_method": "password" + }, + "x-step": "connector" + }, + "privateKey": { + "type": "string", + "title": "Private key", + "description": "Upload your Snowflake private key file (.pem or .p8)", + "format": "file", + "x-display": "file", + "x-file-accept": ".pem,.p8", + "x-file-encoding": "base64", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PRIVATEKEY", + "x-visible-if": { + "auth_method": "private_key" + }, + "x-step": "connector" + }, + "warehouse": { + "type": "string", + "title": "Warehouse", + "description": "Compute warehouse", + "x-placeholder": "your_warehouse", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Snowflake database", + "x-placeholder": "your_database", + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Default schema", + "x-placeholder": "public", + "x-step": "connector" + }, + "role": { + "type": "string", + "title": "Role", + "description": "Snowflake role", + "x-placeholder": "your_role", + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Snowflake DSN, e.g. @//?warehouse=&role=", + "x-placeholder": "@//?warehouse=&role=", + "x-secret": true, + "x-env-var": "SNOWFLAKE_DSN", + "x-hint": "Include authenticator and privateKey query params for JWT if needed.", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "password" + } + } + }, + "then": { + "required": [ + "account", + "user", + "password", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "private_key" + } + } + }, + "then": { + "required": [ + "account", + "user", + "privateKey", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: snowflake\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/snowflake\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/sqlite-duckdb.json b/runtime/templates/definitions/duckdb-models/sqlite-duckdb.json new file mode 100644 index 000000000000..9c3d71458ae6 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/sqlite-duckdb.json @@ -0,0 +1,53 @@ +{ + "name": "sqlite-duckdb", + "display_name": "SQLite", + "description": "Read SQLite tables into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/sqlite", + "driver": "sqlite", + "olap": "duckdb", + "icon": "SQLite", + "small_icon": "SQLiteIcon", + "tags": [ + "sqlite", + "database", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "db": { + "type": "string", + "title": "Database file", + "description": "Path to SQLite db file", + "x-placeholder": "/path/to/sqlite.db" + }, + "table": { + "type": "string", + "title": "Table", + "description": "SQLite table name", + "x-placeholder": "table" + }, + "name": { + "type": "string", + "title": "Source name", + "description": "Name of the source", + "x-placeholder": "my_new_source" + } + }, + "required": [ + "db", + "table", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/sqlite\ntype: model\nconnector: duckdb\nsql: |\n SELECT * FROM sqlite_scan('[[ .db ]]', '[[ .table ]]');\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/supabase-duckdb.json b/runtime/templates/definitions/duckdb-models/supabase-duckdb.json new file mode 100644 index 000000000000..28e5401ed439 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/supabase-duckdb.json @@ -0,0 +1,147 @@ +{ + "name": "supabase-duckdb", + "display_name": "Supabase", + "description": "Query Supabase databases and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "supabase", + "olap": "duckdb", + "icon": "Supabase", + "small_icon": "SupabaseIcon", + "tags": [ + "supabase", + "postgres", + "database", + "duckdb", + "source", + "connector" + ], + + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": ["parameters", "dsn"], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": ["Enter parameters", "Enter connection string"], + "x-ui-only": true, + "x-tab-group": { + "parameters": ["host", "port", "user", "password", "dbname", "sslmode"], + "dsn": ["dsn"] + }, + "x-step": "connector" + }, + + "dsn": { + "type": "string", + "title": "Supabase connection string", + "description": "e.g. postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres", + "x-placeholder": "postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres", + "x-secret": true, + "x-env-var": "SUPABASE_DSN", + "x-hint": "Use a DSN or provide host/user/password/dbname below (but not both).", + "x-step": "connector" + }, + + "host": { + "type": "string", + "title": "Host", + "description": "Supabase database host", + "x-placeholder": "aws-0-[region].pooler.supabase.com", + "x-step": "connector" + }, + + "port": { + "type": "string", + "title": "Port", + "description": "Supabase database port", + "pattern": "^\\d+$", + "errorMessage": { "pattern": "Port must be a number" }, + "default": "5432", + "x-placeholder": "5432", + "x-step": "connector" + }, + + "user": { + "type": "string", + "title": "Username", + "description": "Supabase database user", + "x-placeholder": "postgres.[ref]", + "x-step": "connector" + }, + + "password": { + "type": "string", + "title": "Password", + "description": "Supabase database password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SUPABASE_PASSWORD", + "x-step": "connector" + }, + + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres", + "x-step": "connector" + }, + + "sslmode": { + "type": "string", + "title": "SSL mode", + "description": "Use disable, allow, prefer, require", + "enum": ["disable", "allow", "prefer", "require"], + "default": "require", + "x-placeholder": "require", + "x-step": "connector" + }, + + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your database", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + + "required": ["sql", "name"], + + "allOf": [ + { + "if": { "properties": { "connection_mode": { "const": "dsn" } } }, + "then": { "required": ["dsn"] }, + "else": { "required": ["host", "user", "dbname"] } + } + ] + }, + + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: supabase\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/postgres\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/clickhouse.json b/runtime/templates/definitions/olap/clickhouse.json new file mode 100644 index 000000000000..f0161026bbaa --- /dev/null +++ b/runtime/templates/definitions/olap/clickhouse.json @@ -0,0 +1,385 @@ +{ + "name": "clickhouse", + "display_name": "ClickHouse", + "description": "Connect to a ClickHouse OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/clickhouse", + "driver": "clickhouse", + "icon": "ClickHouse", + "small_icon": "ClickHouseIcon", + "tags": [ + "clickhouse", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-height": "tall", + "x-form-width": "wide", + "x-button-labels": { + "deployment_type": { + "playground": { + "idle": "Connect", + "loading": "Connecting..." + }, + "rill-managed": { + "idle": "Connect", + "loading": "Connecting..." + } + } + }, + "properties": { + "deployment_type": { + "type": "string", + "title": "Connection type", + "enum": [ + "cloud", + "playground", + "self-managed", + "rill-managed" + ], + "default": "cloud", + "x-display": "select", + "x-select-style": "rich", + "x-enum-labels": [ + "ClickHouse Cloud", + "ClickHouse Playground", + "Self Managed", + "Rill-Managed" + ], + "x-enum-descriptions": [ + "Connect to your ClickHouse Cloud instance", + "Free public instance for testing and demos", + "Connect to your own self-hosted server", + "Rill-managed ClickHouse infrastructure" + ], + "x-ui-only": true, + "x-grouped-fields": { + "cloud": [ + "connection_mode" + ], + "playground": [ + "playground_info" + ], + "self-managed": [ + "connection_mode" + ], + "rill-managed": [ + "managed" + ] + }, + "x-step": "connector" + }, + "connection_mode": { + "type": "string", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "username", + "password", + "database", + "cluster", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "DSN connection string (use instead of individual host/port/user settings)", + "x-placeholder": "clickhouse://localhost:9000?username=default&password=password", + "x-secret": true, + "x-env-var": "CLICKHOUSE_DSN", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "managed": { + "type": "boolean", + "title": "Managed", + "description": "This option uses ClickHouse as an OLAP engine with Rill-managed infrastructure. No additional configuration is required.", + "default": false, + "x-informational": true, + "x-omit-if-default": true, + "x-visible-if": { + "deployment_type": "rill-managed" + }, + "x-step": "connector" + }, + "playground_info": { + "type": "boolean", + "title": "Playground", + "description": "Connect to ClickHouse's free public playground instance. This is a read-only demo environment with sample datasets, perfect for testing.", + "default": true, + "x-informational": true, + "x-ui-only": true, + "x-visible-if": { + "deployment_type": "playground" + }, + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Hostname or IP address of the ClickHouse server", + "x-placeholder": "your.clickhouse.server.com", + "x-hint": "Your ClickHouse hostname (e.g., your-server.com)", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Port number of the ClickHouse server (8443 HTTPS, 9440 Native TLS, 8123 HTTP, 9000 Native TCP)", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "8443", + "x-placeholder": "8443", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Username to connect to the ClickHouse server", + "default": "default", + "x-placeholder": "default", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Password to connect to the ClickHouse server", + "x-placeholder": "Database password", + "x-secret": true, + "x-env-var": "CLICKHOUSE_PASSWORD", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Name of the ClickHouse database to connect to", + "default": "default", + "x-placeholder": "default", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "cluster": { + "type": "string", + "title": "Cluster", + "description": "Cluster name. If set, models are created as distributed tables.", + "x-placeholder": "Cluster name", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Use SSL to connect to the ClickHouse server", + "default": true, + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + } + }, + "required": [ + "deployment_type" + ], + "allOf": [ + { + "if": { + "properties": { + "deployment_type": { + "const": "rill-managed" + } + } + }, + "then": { + "required": [ + "managed" + ], + "properties": { + "managed": { + "const": true + } + } + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "playground" + } + } + }, + "then": { + "properties": { + "host": { + "const": "play.clickhouse.com" + }, + "port": { + "const": "9440" + }, + "username": { + "const": "play" + }, + "password": { + "const": "" + }, + "database": { + "const": "default" + }, + "ssl": { + "const": true + } + } + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "cloud" + }, + "connection_mode": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "username", + "port" + ] + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "cloud" + }, + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "self-managed" + }, + "connection_mode": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "username" + ] + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "self-managed" + }, + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/clickhouse\ntype: connector\ndriver: clickhouse\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/druid.json b/runtime/templates/definitions/olap/druid.json new file mode 100644 index 000000000000..9d46c2d7023b --- /dev/null +++ b/runtime/templates/definitions/olap/druid.json @@ -0,0 +1,123 @@ +{ + "name": "druid", + "display_name": "Druid", + "description": "Connect to an Apache Druid OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/druid", + "driver": "druid", + "icon": "ApacheDruid", + "small_icon": "ApacheDruidIcon", + "tags": [ + "druid", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "username", + "password", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Druid SQL/Avatica endpoint", + "x-placeholder": "https://example.com/druid/v2/sql/avatica-protobuf?authentication=BASIC&avaticaUser=user&avaticaPassword=pass", + "x-secret": true, + "x-env-var": "DRUID_DSN", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Druid host or IP", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Druid port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "x-placeholder": "8888", + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Druid username", + "x-placeholder": "default", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Druid password", + "x-placeholder": "password", + "x-secret": true, + "x-env-var": "DRUID_PASSWORD", + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Use SSL for the connection", + "default": true, + "x-step": "connector" + } + }, + "required": [], + "oneOf": [ + { + "title": "Use connection string", + "required": [ + "dsn" + ] + }, + { + "title": "Use individual parameters", + "required": [ + "host", + "ssl" + ] + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/druid\ntype: connector\ndriver: druid\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/duckdb.json b/runtime/templates/definitions/olap/duckdb.json new file mode 100644 index 000000000000..93d09dae3b48 --- /dev/null +++ b/runtime/templates/definitions/olap/duckdb.json @@ -0,0 +1,125 @@ +{ + "name": "duckdb", + "display_name": "DuckDB", + "description": "Connect to a DuckDB OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/duckdb", + "driver": "duckdb", + "icon": "DuckDB", + "small_icon": "DuckDBIcon", + "tags": [ + "duckdb", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-button-labels": { + "connector_type": { + "rill-managed": { + "idle": "Connect", + "loading": "Connecting..." + } + } + }, + "properties": { + "connector_type": { + "type": "string", + "title": "Connection type", + "enum": [ + "rill-managed", + "self-hosted" + ], + "default": "rill-managed", + "x-display": "select", + "x-select-style": "rich", + "x-enum-labels": [ + "Rill Managed", + "Local File" + ], + "x-enum-descriptions": [ + "Rill manages your DuckDB infrastructure", + "Connect to your own DuckDB database file" + ], + "x-ui-only": true, + "x-grouped-fields": { + "rill-managed": [ + "managed" + ], + "self-hosted": [ + "path" + ] + }, + "x-step": "connector" + }, + "managed": { + "type": "boolean", + "title": "Managed", + "description": "This option uses DuckDB as an OLAP engine with Rill-managed infrastructure. No additional configuration is required.", + "default": true, + "x-informational": true, + "x-ui-only": true, + "x-visible-if": { + "connector_type": "rill-managed" + }, + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "Path", + "description": "Path to external DuckDB database", + "x-placeholder": "/path/to/main.db", + "x-visible-if": { + "connector_type": "self-hosted" + }, + "x-step": "connector" + } + }, + "required": [ + "connector_type" + ], + "allOf": [ + { + "if": { + "properties": { + "connector_type": { + "const": "rill-managed" + } + } + }, + "then": { + "required": [ + "managed" + ], + "properties": { + "managed": { + "const": true + } + } + } + }, + { + "if": { + "properties": { + "connector_type": { + "const": "self-hosted" + } + } + }, + "then": { + "required": [ + "path" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/duckdb\ntype: connector\ndriver: duckdb\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/motherduck.json b/runtime/templates/definitions/olap/motherduck.json new file mode 100644 index 000000000000..f4466f131921 --- /dev/null +++ b/runtime/templates/definitions/olap/motherduck.json @@ -0,0 +1,57 @@ +{ + "name": "motherduck", + "display_name": "MotherDuck", + "description": "Connect to a MotherDuck OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/motherduck", + "driver": "motherduck", + "icon": "MotherDuck", + "small_icon": "MotherDuckIcon", + "tags": [ + "motherduck", + "duckdb", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "path": { + "type": "string", + "title": "Path", + "description": "MotherDuck database path (prefix with md:)", + "x-placeholder": "md:my_db", + "x-step": "connector" + }, + "token": { + "type": "string", + "title": "Token", + "description": "MotherDuck token", + "x-placeholder": "your_motherduck_token", + "x-secret": true, + "x-env-var": "MOTHERDUCK_TOKEN", + "x-step": "connector" + }, + "schema_name": { + "type": "string", + "title": "Schema name", + "description": "Default schema to use", + "x-placeholder": "main", + "x-step": "connector" + } + }, + "required": [ + "path", + "token", + "schema_name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/motherduck\ntype: connector\ndriver: motherduck\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/pinot.json b/runtime/templates/definitions/olap/pinot.json new file mode 100644 index 000000000000..d5560ba1f49a --- /dev/null +++ b/runtime/templates/definitions/olap/pinot.json @@ -0,0 +1,144 @@ +{ + "name": "pinot", + "display_name": "Pinot", + "description": "Connect to an Apache Pinot OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/pinot", + "driver": "pinot", + "icon": "ApachePinot", + "small_icon": "ApachePinotIcon", + "tags": [ + "pinot", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "broker_host", + "broker_port", + "controller_host", + "controller_port", + "username", + "password", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Pinot connection string", + "x-placeholder": "https://username:password@localhost:8000?controller=localhost:9000", + "x-secret": true, + "x-env-var": "PINOT_DSN", + "x-step": "connector" + }, + "broker_host": { + "type": "string", + "title": "Broker host", + "description": "Pinot broker host", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "broker_port": { + "type": "string", + "title": "Broker port", + "description": "Pinot broker port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "x-placeholder": "8000", + "x-step": "connector" + }, + "controller_host": { + "type": "string", + "title": "Controller host", + "description": "Pinot controller host", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "controller_port": { + "type": "string", + "title": "Controller port", + "description": "Pinot controller port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "x-placeholder": "9000", + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Pinot username", + "x-placeholder": "default", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Pinot password", + "x-placeholder": "password", + "x-secret": true, + "x-env-var": "PINOT_PASSWORD", + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Use SSL", + "default": true, + "x-step": "connector" + } + }, + "required": [], + "oneOf": [ + { + "title": "Use connection string", + "required": [ + "dsn" + ] + }, + { + "title": "Use individual parameters", + "required": [ + "broker_host", + "controller_host", + "ssl" + ] + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/pinot\ntype: connector\ndriver: pinot\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/starrocks.json b/runtime/templates/definitions/olap/starrocks.json new file mode 100644 index 000000000000..2442024771da --- /dev/null +++ b/runtime/templates/definitions/olap/starrocks.json @@ -0,0 +1,148 @@ +{ + "name": "starrocks", + "display_name": "StarRocks", + "description": "Connect to a StarRocks OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/starrocks", + "driver": "starrocks", + "icon": "StarRocks", + "small_icon": "StarRocksIcon", + "tags": [ + "starrocks", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-height": "tall", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "username", + "password", + "catalog", + "database", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "MySQL DSN format. If provided, do not set host/port/username/password.", + "x-placeholder": "user:password@tcp(host:9030)/?timeout=30s&readTimeout=300s&parseTime=true", + "x-secret": true, + "x-env-var": "STARROCKS_DSN", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Hostname or IP address of the StarRocks FE node", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MySQL protocol port of the StarRocks FE node", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "9030", + "x-placeholder": "9030", + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Username to connect to StarRocks", + "default": "root", + "x-placeholder": "root", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Password to connect to StarRocks", + "x-placeholder": "password", + "x-secret": true, + "x-env-var": "STARROCKS_PASSWORD", + "x-step": "connector" + }, + "catalog": { + "type": "string", + "title": "Catalog", + "description": "StarRocks catalog name. Use default_catalog for internal tables, or specify an external catalog.", + "default": "default_catalog", + "x-placeholder": "default_catalog", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Name of the StarRocks database to connect to", + "x-placeholder": "default", + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Enable SSL/TLS encryption for the connection", + "x-step": "connector" + } + }, + "required": [], + "allOf": [ + { + "if": { + "properties": { + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + }, + "else": { + "required": [ + "host" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/starrocks\ntype: connector\ndriver: starrocks\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/duckdb_test.go b/runtime/templates/duckdb_test.go new file mode 100644 index 000000000000..99ec50137cb1 --- /dev/null +++ b/runtime/templates/duckdb_test.go @@ -0,0 +1,81 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestDuckdbSQL(t *testing.T) { + tests := []struct { + name string + path string + defaultToJSON bool + wantContains string + }{ + {"parquet", "s3://bucket/data.parquet", false, "read_parquet"}, + {"csv", "s3://bucket/data.csv", false, "read_csv"}, + {"tsv", "s3://bucket/data.tsv", false, "read_csv"}, + {"json", "s3://bucket/data.json", false, "read_json"}, + {"ndjson", "s3://bucket/data.ndjson", false, "read_json"}, + {"compound parquet.gz", "s3://bucket/data.v1.parquet.gz", false, "read_parquet"}, + {"unknown default generic", "s3://bucket/data.xyz", false, "select * from 's3://bucket/data.xyz'"}, + {"unknown default json", "https://example.com/api", true, "read_json"}, + {"txt", "s3://bucket/data.txt", false, "read_csv"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + query := duckdbSQL(tt.path, tt.defaultToJSON) + require.Contains(t, query, tt.wantContains) + require.Contains(t, query, tt.path) + }) + } +} + +func TestMatchesExtFixedFalsePositive(t *testing.T) { + // This was a bug: "parquet-archive/readme.txt" should NOT match ".parquet" + require.False(t, matchesExt("s3://bucket/parquet-archive/readme.txt", ".parquet"), + "path with 'parquet' in directory name should not match .parquet extension") + + // But actual parquet files should match + require.True(t, matchesExt("s3://bucket/data.parquet", ".parquet")) + require.True(t, matchesExt("s3://bucket/data.v1.parquet.gz", ".parquet")) +} + +func TestMatchesExtCompound(t *testing.T) { + require.True(t, matchesExt("data.csv.gz", ".csv")) + require.True(t, matchesExt("data.ndjson.gz", ".ndjson")) + require.False(t, matchesExt("data.xyz", ".csv", ".json")) +} + +func TestClickhouseURLSuffix(t *testing.T) { + props := []ProcessedProp{ + {Key: "headers", Value: "\n Authorization: \"Bearer {{ .env.connector.https.authorization }}\"\n X-API-Key: \"{{ .env.connector.https.x_api_key }}\"", Quoted: false}, + } + + // CSV URL with headers: includes format and headers() + result := clickhouseURLSuffix("https://example.com/data.csv", props) + require.Contains(t, result, "CSVWithNames") + require.Contains(t, result, "headers(") + require.Contains(t, result, "'Authorization'='Bearer {{ .env.connector.https.authorization }}'") + require.Contains(t, result, "'X-API-Key'='{{ .env.connector.https.x_api_key }}'") + + // Parquet URL with headers + result = clickhouseURLSuffix("https://example.com/data.parquet", props) + require.Contains(t, result, "Parquet") + require.Contains(t, result, "headers(") + + // JSON URL with headers + result = clickhouseURLSuffix("https://example.com/data.ndjson", props) + require.Contains(t, result, "JSONEachRow") + + // No headers: returns empty (ClickHouse auto-detects) + noHeaders := []ProcessedProp{ + {Key: "path", Value: "https://example.com", Quoted: true}, + } + require.Equal(t, "", clickhouseURLSuffix("https://example.com/data.csv", noHeaders)) + + // Nil/wrong type: returns empty + require.Equal(t, "", clickhouseURLSuffix("https://example.com/data.csv", nil)) +} diff --git a/runtime/templates/env.go b/runtime/templates/env.go new file mode 100644 index 000000000000..8fc8edee09da --- /dev/null +++ b/runtime/templates/env.go @@ -0,0 +1,48 @@ +package templates + +import ( + "bufio" + "context" + "fmt" + "strings" + + "github.com/rilldata/rill/runtime/drivers" +) + +// ResolveEnvVarNameForKey determines the env var name for a secret property key, +// using an explicit env var name when provided, or falling back to DRIVER_KEY format. +// This variant does not require a *drivers.PropertySpec; used by schema-based templates. +func ResolveEnvVarNameForKey(driverName, key, explicitEnvVarName string, existingEnv map[string]bool) string { + var base string + if explicitEnvVarName != "" { + base = explicitEnvVarName + } else { + base = strings.ToUpper(driverName) + "_" + strings.ToUpper(key) + } + + candidate := base + for i := 1; existingEnv[candidate]; i++ { + candidate = fmt.Sprintf("%s_%d", base, i) + } + return candidate +} + +// ReadEnvKeys parses a .env file into a set of key names. +func ReadEnvKeys(ctx context.Context, repo drivers.RepoStore) map[string]bool { + keys := make(map[string]bool) + content, err := repo.Get(ctx, ".env") + if err != nil { + return keys + } + scanner := bufio.NewScanner(strings.NewReader(content)) + for scanner.Scan() { + line := strings.TrimSpace(scanner.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + if idx := strings.IndexByte(line, '='); idx > 0 { + keys[line[:idx]] = true + } + } + return keys +} diff --git a/runtime/templates/env_test.go b/runtime/templates/env_test.go new file mode 100644 index 000000000000..6bcbb160c6a5 --- /dev/null +++ b/runtime/templates/env_test.go @@ -0,0 +1,40 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestResolveEnvVarNameWithExplicitName(t *testing.T) { + existing := make(map[string]bool) + + name := ResolveEnvVarNameForKey("s3", "aws_access_key_id", "AWS_ACCESS_KEY_ID", existing) + require.Equal(t, "AWS_ACCESS_KEY_ID", name) +} + +func TestResolveEnvVarNameFallback(t *testing.T) { + existing := make(map[string]bool) + + name := ResolveEnvVarNameForKey("starrocks", "password", "", existing) + require.Equal(t, "STARROCKS_PASSWORD", name) +} + +func TestResolveEnvVarNameConflict(t *testing.T) { + existing := map[string]bool{ + "AWS_ACCESS_KEY_ID": true, + } + + name := ResolveEnvVarNameForKey("s3", "aws_access_key_id", "AWS_ACCESS_KEY_ID", existing) + require.Equal(t, "AWS_ACCESS_KEY_ID_1", name) +} + +func TestResolveEnvVarNameDoubleConflict(t *testing.T) { + existing := map[string]bool{ + "AWS_ACCESS_KEY_ID": true, + "AWS_ACCESS_KEY_ID_1": true, + } + + name := ResolveEnvVarNameForKey("s3", "aws_access_key_id", "AWS_ACCESS_KEY_ID", existing) + require.Equal(t, "AWS_ACCESS_KEY_ID_2", name) +} diff --git a/runtime/templates/funcmap.go b/runtime/templates/funcmap.go new file mode 100644 index 000000000000..6b54199bc9c2 --- /dev/null +++ b/runtime/templates/funcmap.go @@ -0,0 +1,288 @@ +package templates + +import ( + "fmt" + "path/filepath" + "strings" + "text/template" +) + +// sharedFuncMap is the template function map available to all template definitions. +// Allocated once since all entries are stateless functions. +var sharedFuncMap = template.FuncMap{ + "renderProps": renderProps, + "indent": indent, + "quote": quote, + "propVal": propVal, + "default": defaultVal, + "duckdbSQL": duckdbSQL, + "s3ToHTTPS": s3ToHTTPS, + "gcsToHTTPS": gcsToHTTPS, + "azureContainer": azureContainer, + "azureBlobPath": azureBlobPath, + "azureEndpoint": azureEndpoint, + "clickhouseFormat": clickhouseFormat, + "clickhouseURLSuffix": clickhouseURLSuffix, +} + +// renderProps renders a slice of ProcessedProp as YAML key-value lines. +// Each property is rendered on its own line with appropriate formatting: +// quoted values get double quotes, unquoted values are rendered as-is. +func renderProps(props []ProcessedProp) string { + if len(props) == 0 { + return "" + } + var b strings.Builder + for i, p := range props { + if i > 0 { + b.WriteByte('\n') + } + if p.Quoted { + fmt.Fprintf(&b, "%s: %q", p.Key, p.Value) + } else { + fmt.Fprintf(&b, "%s: %s", p.Key, p.Value) + } + } + return b.String() +} + +// indent prepends each line of text with n spaces. +func indent(n int, text string) string { + pad := strings.Repeat(" ", n) + lines := strings.Split(text, "\n") + for i, line := range lines { + if line != "" { + lines[i] = pad + line + } + } + return strings.Join(lines, "\n") +} + +// quote wraps a string in double quotes. +func quote(s string) string { + return fmt.Sprintf("%q", s) +} + +// propVal extracts a value from a []ProcessedProp by key. +// Returns "" if the key is not found or props is not the expected type. +func propVal(props any, key string) string { + ps, ok := props.([]ProcessedProp) + if !ok { + return "" + } + for _, p := range ps { + if p.Key == key { + return p.Value + } + } + return "" +} + +// defaultVal returns val if non-empty, otherwise fallback. +// Registered as "default" in the template function map. +// NOTE: Use positional syntax only: [[ default (expr) "fallback" ]]. +// Pipeline syntax ([[ expr | default "fallback" ]]) would swap arguments +// because text/template pipes into the last parameter. +func defaultVal(val, fallback string) string { + if val == "" { + return fallback + } + return val +} + +// duckdbSQL maps a file path to a DuckDB read function call based on file extension. +// When defaultToJSON is true, unknown extensions default to read_json; otherwise select * from 'path'. +func duckdbSQL(path string, defaultToJSON bool) string { + switch { + case matchesExt(path, ".csv", ".tsv", ".txt"): + return fmt.Sprintf("select * from read_csv('%s', auto_detect=true, ignore_errors=1, header=true)", path) + case matchesExt(path, ".parquet"): + return fmt.Sprintf("select * from read_parquet('%s')", path) + case matchesExt(path, ".json", ".ndjson"): + return fmt.Sprintf("select * from read_json('%s', auto_detect=true, format='auto')", path) + default: + if defaultToJSON { + return fmt.Sprintf("select * from read_json('%s', auto_detect=true, format='auto')", path) + } + return fmt.Sprintf("select * from '%s'", path) + } +} + +// matchesExt checks if the file path has any of the target extensions. +// Handles compound extensions like .v1.parquet.gz by checking the basename +// for extension substrings, using suffix matching on path segments to avoid +// false positives (e.g. "parquet-archive/readme.txt" should NOT match ".parquet"). +func matchesExt(path string, targets ...string) bool { + ext := strings.ToLower(filepath.Ext(path)) + base := strings.ToLower(filepath.Base(path)) + for _, t := range targets { + if ext == t { + return true + } + // Check for compound extensions in the basename only (not the full path). + // Look for the target extension followed by a dot or end-of-string. + idx := strings.Index(base, t) + if idx > 0 { + after := idx + len(t) + if after == len(base) || base[after] == '.' { + return true + } + } + } + return false +} + +// s3ToHTTPS converts an s3:// URI to an HTTPS URL for ClickHouse's s3() function. +// "s3://bucket/key" becomes "https://bucket.s3.amazonaws.com/key". +// If the path is already HTTPS, it is returned as-is. +func s3ToHTTPS(path string) string { + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + return path + } + trimmed := strings.TrimPrefix(path, "s3://") + idx := strings.IndexByte(trimmed, '/') + if idx < 0 { + return fmt.Sprintf("https://%s.s3.amazonaws.com", trimmed) + } + bucket := trimmed[:idx] + key := trimmed[idx:] + return fmt.Sprintf("https://%s.s3.amazonaws.com%s", bucket, key) +} + +// gcsToHTTPS converts a gs:// URI to an HTTPS URL for ClickHouse's gcs() function. +// "gs://bucket/key" becomes "https://storage.googleapis.com/bucket/key". +// If the path is already HTTPS, it is returned as-is. +func gcsToHTTPS(path string) string { + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + return path + } + trimmed := strings.TrimPrefix(path, "gs://") + return fmt.Sprintf("https://storage.googleapis.com/%s", trimmed) +} + +// clickhouseURLSuffix generates additional arguments for ClickHouse's url() table function. +// When headers are present, returns ", Format, headers('K1'='V1', 'K2'='V2')" where +// Format is auto-detected from the URL extension. Returns empty string when no headers +// are present (ClickHouse auto-detects everything from the URL). +func clickhouseURLSuffix(path string, props any) string { + hdrs := extractClickhouseHeaders(props) + if hdrs == "" { + return "" + } + format := clickhouseFormat(path) + return fmt.Sprintf(",\n %s,\n %s\n ", format, hdrs) +} + +// clickhouseFormat maps a URL path to a ClickHouse input format name. +func clickhouseFormat(path string) string { + switch { + case matchesExt(path, ".csv", ".txt"): + return "CSVWithNames" + case matchesExt(path, ".tsv"): + return "TabSeparatedWithNames" + case matchesExt(path, ".json", ".ndjson", ".jsonl"): + return "JSONEachRow" + case matchesExt(path, ".parquet"): + return "Parquet" + default: + return "JSONEachRow" + } +} + +// extractClickhouseHeaders extracts header ProcessedProps and formats them +// for ClickHouse's headers() syntax: headers('Key1'='value1', 'Key2'='value2'). +// Returns empty string if no headers are present. +func extractClickhouseHeaders(props any) string { + ps, ok := props.([]ProcessedProp) + if !ok { + return "" + } + var headerProp *ProcessedProp + for i := range ps { + if ps[i].Key == "headers" { + headerProp = &ps[i] + break + } + } + if headerProp == nil { + return "" + } + + // Parse the YAML-style header lines (e.g. "Authorization: \"Bearer {{ .env.X }}\"") + var pairs []string + for _, line := range strings.Split(headerProp.Value, "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + idx := strings.IndexByte(line, ':') + if idx < 0 { + continue + } + key := strings.TrimSpace(line[:idx]) + val := strings.TrimSpace(line[idx+1:]) + val = strings.Trim(val, "\"") + pairs = append(pairs, fmt.Sprintf("'%s'='%s'", key, val)) + } + if len(pairs) == 0 { + return "" + } + return fmt.Sprintf("headers(%s)", strings.Join(pairs, ", ")) +} + +// azureContainer extracts the container name from an Azure URI. +// Supports both "azure://container/blob/path" and "https://account.blob.core.windows.net/container/blob/path". +func azureContainer(path string) string { + path = stripAzurePrefix(path) + idx := strings.IndexByte(path, '/') + if idx < 0 { + return path + } + return path[:idx] +} + +// azureBlobPath extracts the blob path from an Azure URI. +// Supports both "azure://container/blob/path" and "https://account.blob.core.windows.net/container/blob/path". +func azureBlobPath(path string) string { + path = stripAzurePrefix(path) + idx := strings.IndexByte(path, '/') + if idx < 0 { + return "" + } + return path[idx+1:] +} + +// azureEndpoint returns the blob service endpoint for a given Azure URI. +// For "https://account.blob.core.windows.net/..." it returns "https://account.blob.core.windows.net". +// For "azure://..." it builds the endpoint from the account property. +func azureEndpoint(path, account string) string { + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + u := strings.TrimPrefix(path, "https://") + u = strings.TrimPrefix(u, "http://") + idx := strings.IndexByte(u, '/') + if idx > 0 { + return "https://" + u[:idx] + } + return "https://" + u + } + return fmt.Sprintf("https://%s.blob.core.windows.net", account) +} + +// stripAzurePrefix strips the scheme and host from an Azure URI, returning "container/blob/path". +// Handles "azure://container/path" and "https://account.blob.core.windows.net/container/path". +func stripAzurePrefix(path string) string { + if strings.HasPrefix(path, "azure://") { + return strings.TrimPrefix(path, "azure://") + } + // HTTPS: strip scheme + host, leaving "/container/path" + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + noScheme := strings.TrimPrefix(path, "https://") + noScheme = strings.TrimPrefix(noScheme, "http://") + idx := strings.IndexByte(noScheme, '/') + if idx >= 0 { + return noScheme[idx+1:] + } + return "" + } + return path +} diff --git a/runtime/templates/headers.go b/runtime/templates/headers.go new file mode 100644 index 000000000000..3764b13fa7af --- /dev/null +++ b/runtime/templates/headers.go @@ -0,0 +1,52 @@ +package templates + +import ( + "fmt" + "regexp" + "strings" +) + +// sensitiveHeaderPattern matches header keys that carry secret values. +var sensitiveHeaderPattern = regexp.MustCompile( + `(?i)^(authorization|x-api-key|api-key|token|x-token|x-auth|x-secret|proxy-authorization)$`, +) + +// headerKeyCleanupPattern sanitizes a header key into a valid .env variable segment. +// Compiled at package level to avoid per-call regex compilation. +var headerKeyCleanupPattern = regexp.MustCompile(`[^a-z0-9]+`) + +// IsSensitiveHeaderKey returns true when a header key likely carries a secret value. +func IsSensitiveHeaderKey(key string) bool { + return sensitiveHeaderPattern.MatchString(strings.TrimSpace(key)) +} + +// AuthSchemePrefixes are common HTTP authentication scheme prefixes. +// When a sensitive header value starts with one of these (case-insensitive), +// only the token portion is stored in .env. +var AuthSchemePrefixes = []string{"Bearer ", "Basic ", "Token ", "Bot "} + +// SplitAuthSchemePrefix splits a value into scheme prefix and secret if it starts +// with a known auth scheme. Returns ("", "", false) when no prefix matches. +func SplitAuthSchemePrefix(value string) (scheme, secret string, ok bool) { + for _, prefix := range AuthSchemePrefixes { + if len(value) > len(prefix) && strings.EqualFold(value[:len(prefix)], prefix) { + return value[:len(prefix)], value[len(prefix):], true + } + } + return "", "", false +} + +// HeaderKeyToEnvSegment sanitizes a header key into a valid .env variable segment. +func HeaderKeyToEnvSegment(key string) string { + return headerKeyCleanupPattern.ReplaceAllString(strings.ToLower(key), "_") +} + +// ResolveHeaderEnvVarName determines the env var name for a header, resolving conflicts. +func ResolveHeaderEnvVarName(connectorName, segment string, existingEnv map[string]bool) string { + base := fmt.Sprintf("connector.%s.%s", connectorName, segment) + candidate := base + for i := 1; existingEnv[candidate]; i++ { + candidate = fmt.Sprintf("%s_%d", base, i) + } + return candidate +} diff --git a/runtime/templates/headers_test.go b/runtime/templates/headers_test.go new file mode 100644 index 000000000000..be0dc08d31b4 --- /dev/null +++ b/runtime/templates/headers_test.go @@ -0,0 +1,56 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestIsSensitiveHeaderKey(t *testing.T) { + sensitive := []string{"Authorization", "X-API-Key", "API-Key", "Token", "X-Token", "X-Auth", "X-Secret", "Proxy-Authorization"} + for _, key := range sensitive { + require.True(t, IsSensitiveHeaderKey(key), "expected %q to be sensitive", key) + } + + notSensitive := []string{"Content-Type", "Accept", "X-Request-ID", "User-Agent"} + for _, key := range notSensitive { + require.False(t, IsSensitiveHeaderKey(key), "expected %q to not be sensitive", key) + } +} + +func TestSplitAuthSchemePrefix(t *testing.T) { + scheme, secret, ok := SplitAuthSchemePrefix("Bearer my-token-123") + require.True(t, ok) + require.Equal(t, "Bearer ", scheme) + require.Equal(t, "my-token-123", secret) + + scheme, secret, ok = SplitAuthSchemePrefix("Basic dXNlcjpwYXNz") + require.True(t, ok) + require.Equal(t, "Basic ", scheme) + require.Equal(t, "dXNlcjpwYXNz", secret) + + // No prefix + _, _, ok = SplitAuthSchemePrefix("plain-token") + require.False(t, ok) + + // Prefix only (no token part) + _, _, ok = SplitAuthSchemePrefix("Bearer") + require.False(t, ok) +} + +func TestHeaderKeyToEnvSegment(t *testing.T) { + require.Equal(t, "x_api_key", HeaderKeyToEnvSegment("X-API-Key")) + require.Equal(t, "authorization", HeaderKeyToEnvSegment("Authorization")) + require.Equal(t, "content_type", HeaderKeyToEnvSegment("Content-Type")) +} + +func TestResolveHeaderEnvVarName(t *testing.T) { + existing := make(map[string]bool) + name := ResolveHeaderEnvVarName("my_conn", "authorization", existing) + require.Equal(t, "connector.my_conn.authorization", name) + + // Conflict + existing[name] = true + name2 := ResolveHeaderEnvVarName("my_conn", "authorization", existing) + require.Equal(t, "connector.my_conn.authorization_1", name2) +} diff --git a/runtime/templates/registry.go b/runtime/templates/registry.go new file mode 100644 index 000000000000..0dbfc2b8270e --- /dev/null +++ b/runtime/templates/registry.go @@ -0,0 +1,225 @@ +package templates + +import ( + "bytes" + "embed" + "encoding/json" + "fmt" + "io/fs" + "path" + "sort" + "strings" +) + +//go:embed definitions/*/*.json +var definitionsFS embed.FS + +// Registry holds all loaded template definitions. +type Registry struct { + templates map[string]*Template + sorted []*Template // sorted by name for stable List() output +} + +// NewRegistry loads all embedded template definitions from the definitions/ directory tree. +func NewRegistry() (*Registry, error) { + r := &Registry{ + templates: make(map[string]*Template), + } + + err := fs.WalkDir(definitionsFS, "definitions", func(fpath string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !strings.HasSuffix(d.Name(), ".json") { + return nil + } + + data, err := definitionsFS.ReadFile(fpath) + if err != nil { + return fmt.Errorf("reading template %s: %w", fpath, err) + } + + // Skip stub files (placeholders for future implementation); + // these may be empty or contain only a "_reason" field. + if len(bytes.TrimSpace(data)) == 0 { + return nil + } + + var t Template + if err := json.Unmarshal(data, &t); err != nil { + return fmt.Errorf("parsing template %s: %w", fpath, err) + } + + if t.Name == "" { + // Stub file with metadata (e.g. _reason) but no template definition; skip it + return nil + } + + if _, exists := r.templates[t.Name]; exists { + return fmt.Errorf("duplicate template name %q in %s", t.Name, fpath) + } + + // Resolve code_template_file references + dir := path.Dir(fpath) + for i := range t.Files { + if t.Files[i].CodeTemplateFile == "" { + continue + } + tmplPath := dir + "/" + t.Files[i].CodeTemplateFile + content, err := definitionsFS.ReadFile(tmplPath) + if err != nil { + return fmt.Errorf("reading code template file %s for %s: %w", tmplPath, fpath, err) + } + t.Files[i].CodeTemplate = string(content) + } + + // Preserve JSON-defined property order (Go maps lose insertion order). + // Store on struct for backend use, and in the schema as []any for + // protobuf Struct conversion ([]string is not protobuf-compatible). + if t.JSONSchema != nil { + t.PropertyOrder = extractPropertyOrder(data) + if len(t.PropertyOrder) > 0 { + orderAny := make([]any, len(t.PropertyOrder)) + for i, k := range t.PropertyOrder { + orderAny[i] = k + } + t.JSONSchema["x-property-order"] = orderAny + } + } + + r.templates[t.Name] = &t + return nil + }) + if err != nil { + return nil, fmt.Errorf("loading template definitions: %w", err) + } + + // Build sorted list + r.sorted = make([]*Template, 0, len(r.templates)) + for _, t := range r.templates { + r.sorted = append(r.sorted, t) + } + sort.Slice(r.sorted, func(i, j int) bool { + return r.sorted[i].Name < r.sorted[j].Name + }) + + return r, nil +} + +// List returns all templates sorted by name. +func (r *Registry) List() []*Template { + return r.sorted +} + +// ListByTags returns templates that match ALL of the given tags. +// If tags is empty, returns all templates. +func (r *Registry) ListByTags(tags []string) []*Template { + if len(tags) == 0 { + return r.sorted + } + + var result []*Template + for _, t := range r.sorted { + if matchesAllTags(t.Tags, tags) { + result = append(result, t) + } + } + return result +} + +// Get returns a template by name, or nil and false if not found. +func (r *Registry) Get(name string) (*Template, bool) { + t, ok := r.templates[name] + return t, ok +} + +// LookupByDriver finds the template for a given driver and output type. +// This is used by the backward-compatible GenerateTemplate RPC to map +// (driver, resource_type) pairs to template names. +func (r *Registry) LookupByDriver(driver, resourceType string) (*Template, bool) { + switch resourceType { + case "connector": + // Combined templates (e.g. s3-duckdb) contain both connector and model files. + // Check for a combined template first; fall back to standalone connector template. + if t, ok := r.Get(driver + "-duckdb"); ok && hasFileNamed(t, "connector") { + return t, true + } + return r.Get(driver) + case "model": + // Model templates use the pattern driver-{olap} (e.g. s3-duckdb, s3-clickhouse). + // LookupByDriver defaults to DuckDB; for other OLAPs use Get() directly. + return r.Get(driver + "-duckdb") + } + return nil, false +} + +// hasFileNamed returns true if the template has a file entry with the given name. +func hasFileNamed(t *Template, name string) bool { + for _, f := range t.Files { + if f.Name == name { + return true + } + } + return false +} + +// extractPropertyOrder parses raw JSON bytes to extract the key ordering of +// json_schema.properties. Go's map[string]any loses insertion order on unmarshal, +// but json.Decoder preserves it. +func extractPropertyOrder(raw []byte) []string { + var outer struct { + JSONSchema json.RawMessage `json:"json_schema"` + } + if err := json.Unmarshal(raw, &outer); err != nil || outer.JSONSchema == nil { + return nil + } + + var schema struct { + Properties json.RawMessage `json:"properties"` + } + if err := json.Unmarshal(outer.JSONSchema, &schema); err != nil || schema.Properties == nil { + return nil + } + + dec := json.NewDecoder(bytes.NewReader(schema.Properties)) + tok, err := dec.Token() // opening { + if err != nil { + return nil + } + if delim, ok := tok.(json.Delim); !ok || delim != '{' { + return nil + } + + var keys []string + for dec.More() { + tok, err := dec.Token() + if err != nil { + break + } + key, ok := tok.(string) + if !ok { + break + } + keys = append(keys, key) + // Skip the property value object + var discard json.RawMessage + if err := dec.Decode(&discard); err != nil { + break + } + } + return keys +} + +// matchesAllTags returns true if the template's tags contain all of the required tags. +func matchesAllTags(templateTags, requiredTags []string) bool { + tagSet := make(map[string]bool, len(templateTags)) + for _, t := range templateTags { + tagSet[t] = true + } + for _, req := range requiredTags { + if !tagSet[req] { + return false + } + } + return true +} diff --git a/runtime/templates/registry_test.go b/runtime/templates/registry_test.go new file mode 100644 index 000000000000..8723474c0dd2 --- /dev/null +++ b/runtime/templates/registry_test.go @@ -0,0 +1,157 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" + "google.golang.org/protobuf/types/known/structpb" +) + +func TestNewRegistry(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + require.NotNil(t, r) + + // Verify all definitions loaded + all := r.List() + require.Greater(t, len(all), 25, "expected at least 25 template definitions") +} + +func TestRegistryGet(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + // Known templates exist + for _, name := range []string{"clickhouse", "s3-duckdb", "gcs-duckdb", "iceberg-duckdb", "snowflake-duckdb"} { + t.Run(name, func(t *testing.T) { + tmpl, ok := r.Get(name) + require.True(t, ok, "template %q should exist", name) + require.Equal(t, name, tmpl.Name) + require.NotEmpty(t, tmpl.DisplayName) + require.NotEmpty(t, tmpl.Files) + }) + } + + // Unknown template doesn't exist + _, ok := r.Get("nonexistent") + require.False(t, ok) +} + +func TestRegistryListByTags(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + // Filter by "duckdb" should return DuckDB-related templates + duckdbTemplates := r.ListByTags([]string{"duckdb"}) + require.Greater(t, len(duckdbTemplates), 5, "expected several duckdb-tagged templates") + for _, tmpl := range duckdbTemplates { + require.Contains(t, tmpl.Tags, "duckdb") + } + + // Filter by "olap" + "connector" should return OLAP connector templates + olapConnectors := r.ListByTags([]string{"olap", "connector"}) + require.Greater(t, len(olapConnectors), 0) + for _, tmpl := range olapConnectors { + require.Contains(t, tmpl.Tags, "olap") + require.Contains(t, tmpl.Tags, "connector") + } + + // Empty tags returns all templates + allTemplates := r.ListByTags(nil) + require.Equal(t, len(r.List()), len(allTemplates)) +} + +func TestRegistryLookupByDriver(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + // Connector lookup: combined templates (s3-duckdb has connector file) + tmpl, ok := r.LookupByDriver("s3", "connector") + require.True(t, ok) + require.Equal(t, "s3-duckdb", tmpl.Name) + + // Model lookup for object stores: driver-duckdb + tmpl, ok = r.LookupByDriver("s3", "model") + require.True(t, ok) + require.Equal(t, "s3-duckdb", tmpl.Name) + + // Model lookup for warehouses: driver-duckdb + tmpl, ok = r.LookupByDriver("snowflake", "model") + require.True(t, ok) + require.Equal(t, "snowflake-duckdb", tmpl.Name) +} + +func TestRegistryTemplatesSorted(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + all := r.List() + for i := 1; i < len(all); i++ { + require.Less(t, all[i-1].Name, all[i].Name, "templates should be sorted by name") + } +} + +func TestRegistryAllDefinitionsValid(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + for _, tmpl := range r.List() { + t.Run(tmpl.Name, func(t *testing.T) { + require.NotEmpty(t, tmpl.Name) + require.NotEmpty(t, tmpl.DisplayName) + require.NotEmpty(t, tmpl.Tags) + require.NotEmpty(t, tmpl.Files) + + for _, f := range tmpl.Files { + require.NotEmpty(t, f.Name, "file name required for template %s", tmpl.Name) + require.NotEmpty(t, f.PathTemplate, "path template required for template %s file %s", tmpl.Name, f.Name) + require.NotEmpty(t, f.CodeTemplate, "code template required for template %s file %s", tmpl.Name, f.Name) + require.Contains(t, []string{"connector", "model"}, f.Name, "file name must be connector or model for template %s", tmpl.Name) + } + }) + } +} + +func TestRegistryTemplatesWithJSONSchemaValid(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + var found int + for _, tmpl := range r.List() { + if tmpl.JSONSchema == nil { + continue + } + found++ + t.Run(tmpl.Name, func(t *testing.T) { + // Schema must be an object type + typ, ok := tmpl.JSONSchema["type"] + require.True(t, ok, "json_schema must have a type field") + require.Equal(t, "object", typ) + + // Schema must have properties + propsRaw, ok := tmpl.JSONSchema["properties"] + require.True(t, ok, "json_schema must have properties") + propsMap, ok := propsRaw.(map[string]any) + require.True(t, ok, "properties must be a map") + require.Greater(t, len(propsMap), 0, "properties must not be empty") + + // Each property with x-secret must have x-env-var + for key, propRaw := range propsMap { + prop, ok := propRaw.(map[string]any) + if !ok { + continue + } + if secret, _ := prop["x-secret"].(bool); secret { + envVar, hasEnvVar := prop["x-env-var"] + require.True(t, hasEnvVar, "property %q has x-secret but no x-env-var", key) + require.NotEmpty(t, envVar, "x-env-var for %q must not be empty", key) + } + } + + // Verify structpb conversion works (catches int vs float64 issues) + _, err := structpb.NewStruct(tmpl.JSONSchema) + require.NoError(t, err, "json_schema for %q must be convertible to protobuf Struct", tmpl.Name) + }) + } + require.Greater(t, found, 0, "expected at least one template with json_schema") +} diff --git a/runtime/templates/render.go b/runtime/templates/render.go new file mode 100644 index 000000000000..dfc15ad06cab --- /dev/null +++ b/runtime/templates/render.go @@ -0,0 +1,389 @@ +package templates + +import ( + "bytes" + "fmt" + "sort" + "strings" + "text/template" +) + +// RenderInput contains all parameters for rendering a template. +type RenderInput struct { + Template *Template + Output string // "connector", "model", or "" for all files + Properties map[string]any // raw form values + ConnectorName string // for model outputs: the connector to reference + ExistingEnv map[string]bool +} + +// RenderOutput contains the result of rendering. +type RenderOutput struct { + Files []RenderedFile + EnvVars map[string]string +} + +// RenderedFile is a single rendered output file. +type RenderedFile struct { + Path string + Blob string +} + +// Render executes a template with the given input, producing rendered files and env vars. +// The rendering pipeline: +// 1. Pre-processes properties (secret extraction, empty filtering, derived fields) +// 2. Builds a template data map +// 3. Renders each matching file's path and code templates +func Render(input *RenderInput) (*RenderOutput, error) { + if input.Template == nil { + return nil, fmt.Errorf("template is nil") + } + + envVars := make(map[string]string) + existingEnv := cloneEnvMap(input.ExistingEnv) + + // Build the template data context + data := buildTemplateData(input, existingEnv, envVars) + + // Render each matching file + var files []RenderedFile + for _, f := range input.Template.Files { + if input.Output != "" && f.Name != input.Output { + continue + } + + path, err := renderString(f.Name+"_path", f.PathTemplate, data) + if err != nil { + return nil, fmt.Errorf("rendering path template for %q: %w", f.Name, err) + } + + blob, err := renderString(f.Name+"_code", f.CodeTemplate, data) + if err != nil { + return nil, fmt.Errorf("rendering code template for %q: %w", f.Name, err) + } + + files = append(files, RenderedFile{ + Path: strings.TrimSpace(path), + Blob: blob, + }) + } + + return &RenderOutput{Files: files, EnvVars: envVars}, nil +} + +// buildTemplateData creates the data map passed to Go templates. +// It pre-processes properties: extracts secrets, filters empties, adds derived fields. +func buildTemplateData(input *RenderInput, existingEnv map[string]bool, envVars map[string]string) map[string]any { + data := make(map[string]any) + + // Basic fields + data["driver"] = input.Template.Driver + data["connector_name"] = input.ConnectorName + data["docs_url"] = input.Template.DocsURL + + // Derive model_name from the "name" property if present + if name, ok := input.Properties["name"]; ok && !isEmpty(name) { + data["model_name"] = fmt.Sprintf("%v", name) + } + + // Pre-populate schema properties: use "default" if defined, otherwise empty string. + // This ensures templates render cleanly and conditional branches (e.g. auth_method) + // work correctly even when the frontend omits optional UI-only fields. + if input.Template.JSONSchema != nil { + for k, prop := range schemaProperties(input.Template.JSONSchema) { + if def := schemaFieldString(prop, "default"); def != "" { + data[k] = def + } else { + data[k] = "" + } + } + } + + // Copy all raw properties into data (overwrites empty defaults above) + for k, v := range input.Properties { + if !isEmpty(v) { + data[k] = fmt.Sprintf("%v", v) + } + } + + // Schema-based path: extract property metadata from JSON Schema + if input.Template.JSONSchema != nil { + allProps := processPropertiesFromSchema( + input.Template.JSONSchema, + input.Template.PropertyOrder, + input.Template.Driver, + input.Properties, + existingEnv, + envVars, + ) + + // Split by x-step so connector and model files get the right properties + configProps, sourceProps := splitPropsByStep(allProps, input.Template.JSONSchema) + data["props"] = configProps + data["config_props"] = configProps + data["source_props"] = sourceProps + + return data + } + + // Template without schema: pass properties as-is + return data +} + +// processPropertiesFromSchema pre-processes properties using JSON Schema metadata. +// Fields with "x-secret": true are extracted to env vars; "x-ui-only": true fields are skipped. +// Quoting is determined by the schema "type" field (number/boolean unquoted; everything else quoted). +func processPropertiesFromSchema( + schema map[string]any, + propertyOrder []string, + driverName string, + rawProps map[string]any, + existingEnv map[string]bool, + envVars map[string]string, +) []ProcessedProp { + propsMap := schemaProperties(schema) + if len(propsMap) == 0 { + return nil + } + + // Use schema-defined property order if available (preserves JSON key ordering); + // fall back to alphabetical for deterministic output. + keys := propertyOrder + if len(keys) == 0 { + keys = make([]string, 0, len(propsMap)) + for k := range propsMap { + keys = append(keys, k) + } + sort.Strings(keys) + } + + var result []ProcessedProp + for _, key := range keys { + prop := propsMap[key] + + // Skip UI-only fields (e.g. auth_method radio buttons) + if schemaFieldBool(prop, "x-ui-only") { + continue + } + + val, ok := rawProps[key] + if !ok || isEmpty(val) { + continue + } + + // Skip properties whose value matches the schema default when x-omit-if-default is set. + // For example, ClickHouse's "managed" defaults to false; rendering "managed: false" + // is unnecessary noise, but "managed: true" (rill-managed) must appear. + if schemaFieldBool(prop, "x-omit-if-default") { + if defVal, hasDefault := prop["default"]; hasDefault && fmt.Sprint(val) == fmt.Sprint(defVal) { + continue + } + } + + // Handle map-typed properties (e.g. headers). + // The frontend key-value editor sends [{key, value}, ...]; convert to a flat map. + mapVal, isMap := val.(map[string]any) + if !isMap { + if arrVal, isArr := val.([]any); isArr { + mapVal = kvArrayToMap(arrVal) + if mapVal == nil { + // Empty or malformed array (e.g. []); skip to avoid rendering "[]" + continue + } + isMap = true + } + } + if isMap { + headerIdent := driverName + headerProps := processHeaders(mapVal, headerIdent, existingEnv, envVars) + result = append(result, headerProps...) + continue + } + + strVal := fmt.Sprintf("%v", val) + + if schemaFieldBool(prop, "x-secret") { + envName := ResolveEnvVarNameForKey(driverName, key, schemaFieldString(prop, "x-env-var"), existingEnv) + existingEnv[envName] = true + envVars[envName] = strVal + result = append(result, ProcessedProp{ + Key: key, + Value: fmt.Sprintf("{{ .env.%s }}", envName), + Quoted: true, + }) + } else { + propType := schemaFieldString(prop, "type") + quoted := propType != "number" && propType != "boolean" + result = append(result, ProcessedProp{ + Key: key, + Value: strVal, + Quoted: quoted, + }) + } + } + return result +} + +// splitPropsByStep separates processed props into connector-step and source-step slices +// based on the x-step field in the JSON Schema. Props without x-step go into both. +func splitPropsByStep(props []ProcessedProp, schema map[string]any) (configProps, sourceProps []ProcessedProp) { + propsMap := schemaProperties(schema) + for _, p := range props { + step := "" + if propSchema, ok := propsMap[p.Key]; ok { + step = schemaFieldString(propSchema, "x-step") + } + switch step { + case "connector": + configProps = append(configProps, p) + case "source": + sourceProps = append(sourceProps, p) + case "explorer": + // Explorer-step props are accessed directly as template variables (e.g. .sql, .name); + // they are excluded from renderProps output to avoid duplication. + default: + configProps = append(configProps, p) + sourceProps = append(sourceProps, p) + } + } + return +} + +// kvArrayToMap converts [{key: "k", value: "v"}, ...] (from the frontend key-value editor) +// to map[string]any{"k": "v", ...}. Returns nil if the array is empty or not in the expected format. +func kvArrayToMap(arr []any) map[string]any { + result := make(map[string]any, len(arr)) + for _, item := range arr { + obj, ok := item.(map[string]any) + if !ok { + return nil + } + k, _ := obj["key"].(string) + v, _ := obj["value"].(string) + if k != "" { + result[k] = v + } + } + if len(result) == 0 { + return nil + } + return result +} + +// schemaProperties extracts the "properties" map from a JSON Schema object. +func schemaProperties(schema map[string]any) map[string]map[string]any { + raw, ok := schema["properties"] + if !ok { + return nil + } + outer, ok := raw.(map[string]any) + if !ok { + return nil + } + result := make(map[string]map[string]any, len(outer)) + for k, v := range outer { + if prop, ok := v.(map[string]any); ok { + result[k] = prop + } + } + return result +} + +// schemaFieldBool returns the bool value of a field in a schema property map. +func schemaFieldBool(prop map[string]any, key string) bool { + v, _ := prop[key].(bool) + return v +} + +// schemaFieldString returns the string value of a field in a schema property map. +func schemaFieldString(prop map[string]any, key string) string { + v, _ := prop[key].(string) + return v +} + +// processHeaders processes a map of header key-value pairs, extracting sensitive values. +func processHeaders(headers map[string]any, connectorName string, existingEnv map[string]bool, envVars map[string]string) []ProcessedProp { + if len(headers) == 0 { + return nil + } + + // Sort keys for deterministic output + keys := make([]string, 0, len(headers)) + for k := range headers { + keys = append(keys, k) + } + sort.Strings(keys) + + // Build a single "headers" prop whose value is a YAML mapping + var headerLines []string + for _, key := range keys { + strVal := fmt.Sprintf("%v", headers[key]) + if IsSensitiveHeaderKey(key) { + envSegment := HeaderKeyToEnvSegment(key) + envName := ResolveHeaderEnvVarName(connectorName, envSegment, existingEnv) + existingEnv[envName] = true + if scheme, secret, ok := SplitAuthSchemePrefix(strVal); ok { + envVars[envName] = secret + headerLines = append(headerLines, fmt.Sprintf(" %s: \"%s{{ .env.%s }}\"", key, scheme, envName)) + } else { + envVars[envName] = strVal + headerLines = append(headerLines, fmt.Sprintf(" %s: \"{{ .env.%s }}\"", key, envName)) + } + } else { + headerLines = append(headerLines, fmt.Sprintf(" %s: %q", key, strVal)) + } + } + + // Return as a single rendered block (the template will output it as-is) + return []ProcessedProp{{ + Key: "headers", + Value: "\n" + strings.Join(headerLines, "\n"), + Quoted: false, // The value is a nested YAML mapping, not a scalar + }} +} + +// renderString renders a Go template string using [[ ]] delimiters. +func renderString(name, tmplText string, data map[string]any) (string, error) { + t, err := template.New(name). + Delims("[[", "]]"). + Funcs(sharedFuncMap). + Parse(tmplText) + if err != nil { + return "", fmt.Errorf("parsing template: %w", err) + } + + var buf bytes.Buffer + if err := t.Execute(&buf, data); err != nil { + return "", fmt.Errorf("executing template: %w", err) + } + return buf.String(), nil +} + +// isEmpty checks if a value is empty (nil, empty string, or empty map). +func isEmpty(v any) bool { + if v == nil { + return true + } + switch val := v.(type) { + case string: + return val == "" + case bool: + return false // bools are never "empty" + case map[string]any: + return len(val) == 0 + default: + return fmt.Sprintf("%v", v) == "" + } +} + +// cloneEnvMap creates a shallow copy of an env key map. +func cloneEnvMap(m map[string]bool) map[string]bool { + if m == nil { + return make(map[string]bool) + } + clone := make(map[string]bool, len(m)) + for k, v := range m { + clone[k] = v + } + return clone +} diff --git a/runtime/templates/render_test.go b/runtime/templates/render_test.go new file mode 100644 index 000000000000..dbb54faa00a5 --- /dev/null +++ b/runtime/templates/render_test.go @@ -0,0 +1,649 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestRenderConnectorTemplate(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + // s3-duckdb is a combined template with both connector and model files + tmpl, ok := registry.Get("s3-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{ + "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE", + "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", + "region": "us-east-1", + }, + ConnectorName: "my_s3", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: connector") + require.Contains(t, blob, "driver: s3") + require.Contains(t, blob, `aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}"`) + require.Contains(t, blob, `aws_secret_access_key: "{{ .env.AWS_SECRET_ACCESS_KEY }}"`) + require.Contains(t, blob, `region: "us-east-1"`) + require.NotContains(t, blob, "AKIAIOSFODNN7EXAMPLE") + + // Connector output should NOT contain source-step properties + require.NotContains(t, blob, "path") + require.NotContains(t, blob, "name") + + // Verify env vars extracted + require.Equal(t, "AKIAIOSFODNN7EXAMPLE", result.EnvVars["AWS_ACCESS_KEY_ID"]) + require.Equal(t, "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", result.EnvVars["AWS_SECRET_ACCESS_KEY"]) + + // Verify path + require.Equal(t, "connectors/my_s3.yaml", result.Files[0].Path) +} + +func TestRenderDuckDBModelTemplate(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("s3-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "s3://my-bucket/data/*.parquet", + "name": "my_model", + }, + ConnectorName: "my_s3_connector", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: duckdb") + require.Contains(t, blob, `create_secrets_from_connectors: "my_s3_connector"`) + require.Contains(t, blob, "read_parquet") + require.Contains(t, blob, "s3://my-bucket/data/*.parquet") + + // Verify path uses model_name from "name" property + require.Equal(t, "models/my_model.yaml", result.Files[0].Path) +} + +func TestRenderWarehouseModelTemplate(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("snowflake-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "sql": "SELECT * FROM my_table", + "name": "snowflake_data", + }, + ConnectorName: "my_snowflake", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, `connector: "my_snowflake"`) + require.Contains(t, blob, "materialize: true") + require.Contains(t, blob, "SELECT * FROM my_table") + require.Contains(t, blob, "SELECT * FROM my_table limit 10000") + + require.Equal(t, "models/snowflake_data.yaml", result.Files[0].Path) +} + +func TestRenderRedshiftModelNoDevSection(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("redshift-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "sql": "SELECT * FROM my_table", + "name": "rs_data", + }, + ConnectorName: "my_redshift", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, `connector: "my_redshift"`) + // Redshift template should NOT have a dev section + require.NotContains(t, blob, "dev:") + require.NotContains(t, blob, `ref "self"`) +} + +func TestRenderIcebergDuckDB(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("iceberg-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "s3://my-iceberg-bucket/warehouse/my_table", + "name": "iceberg_data", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "iceberg_scan") + require.Contains(t, blob, "s3://my-iceberg-bucket/warehouse/my_table") + + require.Equal(t, "models/iceberg_data.yaml", result.Files[0].Path) +} + +func TestRenderS3ClickHouseModel(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("s3-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE", + "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", + "path": "s3://my-bucket/data/events.parquet", + "name": "s3_events", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: clickhouse") + require.Contains(t, blob, "materialize: true") + // SQL should show the s3() function with env var refs + require.Contains(t, blob, "FROM s3(") + require.Contains(t, blob, "https://my-bucket.s3.amazonaws.com/data/events.parquet") + require.Contains(t, blob, "{{ .env.AWS_ACCESS_KEY_ID }}") + require.Contains(t, blob, "{{ .env.AWS_SECRET_ACCESS_KEY }}") + // Raw secrets should NOT appear in the blob + require.NotContains(t, blob, "AKIAIOSFODNN7EXAMPLE") + + // Env vars should be extracted + require.Equal(t, "AKIAIOSFODNN7EXAMPLE", result.EnvVars["AWS_ACCESS_KEY_ID"]) + + require.Equal(t, "models/s3_events.yaml", result.Files[0].Path) +} + +func TestRenderMySQLClickHouseModel(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("mysql-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "host": "db.example.com", + "port": "3306", + "user": "myuser", + "password": "secret123", + "database": "mydb", + "table": "events", + "name": "mysql_events", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "connector: clickhouse") + require.Contains(t, blob, "FROM mysql(") + require.Contains(t, blob, "db.example.com:3306") + require.Contains(t, blob, "mydb") + require.Contains(t, blob, "events") + require.Contains(t, blob, "myuser") + require.Contains(t, blob, "{{ .env.MYSQL_PASSWORD }}") + require.NotContains(t, blob, "secret123") +} + +func TestRenderHTTPSClickHouseWithHeaders(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("https-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + // Frontend key-value editor sends [{key, value}, ...] format + "headers": []any{ + map[string]any{"key": "Authorization", "value": "Bearer my-secret-token"}, + map[string]any{"key": "X-API-Key", "value": "key123"}, + }, + "path": "https://example.com/data.csv", + "name": "api_data", + }, + ConnectorName: "my_https", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: clickhouse") + require.Contains(t, blob, "url(") + require.Contains(t, blob, "https://example.com/data.csv") + require.Contains(t, blob, "CSVWithNames") + require.Contains(t, blob, "headers(") + require.Contains(t, blob, "'Authorization'=") + require.Contains(t, blob, "'X-API-Key'=") + // Raw secrets should NOT appear in the blob + require.NotContains(t, blob, "my-secret-token") + require.NotContains(t, blob, "key123") + + // Env vars should be extracted for sensitive headers + require.Contains(t, result.EnvVars, "connector.https.authorization") + require.Contains(t, result.EnvVars, "connector.https.x_api_key") +} + +func TestRenderHTTPSClickHouseNoHeaders(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("https-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "https://example.com/data.csv", + "name": "simple_data", + }, + ConnectorName: "my_https", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "url('https://example.com/data.csv')") + // No headers() clause when no headers provided + require.NotContains(t, blob, "headers(") +} + +func TestRenderEnvVarConflict(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + // s3-duckdb is a combined template with json_schema + tmpl, ok := registry.Get("s3-duckdb") + require.True(t, ok) + + // Pre-populate existing env to force conflict + existingEnv := map[string]bool{ + "AWS_ACCESS_KEY_ID": true, + "AWS_SECRET_ACCESS_KEY": true, + } + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{ + "aws_access_key_id": "AKIA_NEW", + "aws_secret_access_key": "SECRET_NEW", + }, + ConnectorName: "s3_2", + ExistingEnv: existingEnv, + }) + require.NoError(t, err) + + // Env vars should have _1 suffix due to conflict + require.Contains(t, result.EnvVars, "AWS_ACCESS_KEY_ID_1") + require.Contains(t, result.EnvVars, "AWS_SECRET_ACCESS_KEY_1") + require.Contains(t, result.Files[0].Blob, "AWS_ACCESS_KEY_ID_1") +} + +func TestRenderEmptyPropertiesFiltered(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{ + "host": "localhost", + "port": 9000, + "password": "secret", + "database": "", // empty; should be filtered + }, + ConnectorName: "my_ch", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + + blob := result.Files[0].Blob + require.Contains(t, blob, "host") + require.Contains(t, blob, "port") + require.Contains(t, blob, "CLICKHOUSE_PASSWORD") + require.NotContains(t, blob, "database") +} + +func TestRenderOutputFilter(t *testing.T) { + // Create a two-file template inline for testing + tmpl := &Template{ + Name: "test-multi", + DisplayName: "Test Multi-file", + Tags: []string{"test"}, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/[[ .connector_name ]].yaml", + CodeTemplate: "type: connector\ndriver: test\n", + }, + { + Name: "model", + PathTemplate: "models/[[ .model_name ]].yaml", + CodeTemplate: "type: model\nconnector: duckdb\n", + }, + }, + } + + // All files + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "", + Properties: map[string]any{"name": "test"}, + ConnectorName: "test_conn", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 2) + + // Connector only + result, err = Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{"name": "test"}, + ConnectorName: "test_conn", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + require.Contains(t, result.Files[0].Blob, "type: connector") + + // Model only + result, err = Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{"name": "test"}, + ConnectorName: "test_conn", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + require.Contains(t, result.Files[0].Blob, "type: model") +} + +func TestRenderLocalFileDuckDB(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("local_file-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "data/sales.csv", + "name": "sales", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: duckdb") + require.Contains(t, blob, "read_csv") + require.Contains(t, blob, "data/sales.csv") +} + +func TestRenderSQLiteDuckDB(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("sqlite-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "db": "/data/app.db", + "table": "users", + "name": "sqlite_users", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: duckdb") + require.Contains(t, blob, "sqlite_scan") + require.Contains(t, blob, "/data/app.db") + require.Contains(t, blob, "users") +} + +// --- Schema-based template tests --- + +func TestRenderIcebergDuckDBWithSchema(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("iceberg-duckdb") + require.True(t, ok) + require.NotNil(t, tmpl.JSONSchema, "iceberg-duckdb should have json_schema") + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "aws_access_key_id": "AKIAEXAMPLE", + "aws_secret_access_key": "wJalrXUtnFEMI/EXAMPLEKEY", + "aws_region": "us-west-2", + "path": "s3://my-iceberg-bucket/warehouse/my_table", + "name": "iceberg_test", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "iceberg_scan") + require.Contains(t, blob, "s3://my-iceberg-bucket/warehouse/my_table") + + // Verify path uses model_name from "name" property + require.Equal(t, "models/iceberg_test.yaml", result.Files[0].Path) + + // Verify secret extraction via JSON Schema + require.Equal(t, "AKIAEXAMPLE", result.EnvVars["AWS_ACCESS_KEY_ID"]) + require.Equal(t, "wJalrXUtnFEMI/EXAMPLEKEY", result.EnvVars["AWS_SECRET_ACCESS_KEY"]) + + // Raw secrets should NOT appear in the blob + require.NotContains(t, blob, "AKIAEXAMPLE") + require.NotContains(t, blob, "wJalrXUtnFEMI/EXAMPLEKEY") +} + +func TestRenderSchemaEnvVarConflict(t *testing.T) { + tmpl := &Template{ + Name: "test-schema", + DisplayName: "Test Schema", + Tags: []string{"test"}, + JSONSchema: map[string]any{ + "type": "object", + "properties": map[string]any{ + "api_key": map[string]any{ + "type": "string", + "x-secret": true, + "x-env-var": "MY_API_KEY", + }, + }, + }, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/test.yaml", + CodeTemplate: "type: connector\n[[ renderProps .props ]]", + }, + }, + } + + // Pre-populate existing env to force conflict + existingEnv := map[string]bool{"MY_API_KEY": true} + + result, err := Render(&RenderInput{ + Template: tmpl, + Properties: map[string]any{"api_key": "secret123"}, + ExistingEnv: existingEnv, + }) + require.NoError(t, err) + + // Should use MY_API_KEY_1 due to conflict + require.Contains(t, result.EnvVars, "MY_API_KEY_1") + require.Equal(t, "secret123", result.EnvVars["MY_API_KEY_1"]) + require.Contains(t, result.Files[0].Blob, "MY_API_KEY_1") +} + +func TestRenderSchemaUIOnlyFieldsSkipped(t *testing.T) { + tmpl := &Template{ + Name: "test-ui-only", + DisplayName: "Test UI Only", + Tags: []string{"test"}, + JSONSchema: map[string]any{ + "type": "object", + "properties": map[string]any{ + "auth_method": map[string]any{ + "type": "string", + "x-ui-only": true, + }, + "host": map[string]any{ + "type": "string", + }, + }, + }, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/test.yaml", + CodeTemplate: "type: connector\n[[ renderProps .props ]]", + }, + }, + } + + result, err := Render(&RenderInput{ + Template: tmpl, + Properties: map[string]any{ + "auth_method": "access_keys", + "host": "example.com", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + + blob := result.Files[0].Blob + require.Contains(t, blob, "host") + require.NotContains(t, blob, "auth_method") +} + +func TestRenderSchemaPropertyTypes(t *testing.T) { + tmpl := &Template{ + Name: "test-types", + DisplayName: "Test Types", + Tags: []string{"test"}, + JSONSchema: map[string]any{ + "type": "object", + "properties": map[string]any{ + "host": map[string]any{ + "type": "string", + }, + "port": map[string]any{ + "type": "number", + }, + "ssl": map[string]any{ + "type": "boolean", + }, + }, + }, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/test.yaml", + CodeTemplate: "type: connector\n[[ renderProps .props ]]", + }, + }, + } + + result, err := Render(&RenderInput{ + Template: tmpl, + Properties: map[string]any{ + "host": "example.com", + "port": "9440", + "ssl": "true", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + + blob := result.Files[0].Blob + // Strings should be quoted + require.Contains(t, blob, `host: "example.com"`) + // Numbers should not be quoted + require.Contains(t, blob, "port: 9440") + // Booleans should not be quoted + require.Contains(t, blob, "ssl: true") +} diff --git a/runtime/templates/template.go b/runtime/templates/template.go new file mode 100644 index 000000000000..9113b58c9b72 --- /dev/null +++ b/runtime/templates/template.go @@ -0,0 +1,43 @@ +package templates + +// Template is a declarative definition for generating project files (connectors, models, etc.) +// from structured form data. Each template knows how to produce one or more output files. +// +// Templates with JSONSchema are self-contained: the schema drives form rendering (frontend) +// and property metadata like secret detection (backend). Templates without JSONSchema fall +// back to drivers.Spec for property metadata. +type Template struct { + Name string `json:"name"` + DisplayName string `json:"display_name"` + Description string `json:"description,omitempty"` // short description for UI display + DocsURL string `json:"docs_url,omitempty"` // link to documentation + Driver string `json:"driver"` // primary driver (e.g. "s3"); empty for driverless templates like iceberg + OLAP string `json:"olap"` // target OLAP engine (e.g. "duckdb"); empty for OLAP connector templates + Icon string `json:"icon,omitempty"` // icon component name for full-size display (e.g. add-data grid) + SmallIcon string `json:"small_icon,omitempty"` // icon component name for small display (e.g. nav, cards) + Tags []string `json:"tags"` + JSONSchema map[string]any `json:"json_schema,omitempty"` // JSON Schema for form generation and property metadata + PropertyOrder []string `json:"-"` // JSON-defined property key order; computed at load time + Files []File `json:"files"` +} + +// File describes a single output file within a template. +// PathTemplate and CodeTemplate use Go text/template syntax with [[ ]] delimiters +// to avoid collision with Rill's {{ .env.VAR }} runtime syntax. +// +// CodeTemplate can be specified inline (code_template) or loaded from a separate file +// (code_template_file) for readability. If both are set, code_template_file wins. +type File struct { + Name string `json:"name"` // output name: "connector" or "model" + PathTemplate string `json:"path_template"` // Go template for the file path + CodeTemplate string `json:"code_template,omitempty"` // Go template for the file content (inline) + CodeTemplateFile string `json:"code_template_file,omitempty"` // path to .tmpl file (relative to the JSON definition) +} + +// ProcessedProp is a property that has been pre-processed for template rendering. +// Secret values are replaced with {{ .env.VAR }} references; empty values are filtered. +type ProcessedProp struct { + Key string + Value string + Quoted bool // true for strings and secrets; false for numbers and booleans +} From d8e47ddffc36dbab6cdbd41c168c1f01317754d8 Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Tue, 28 Apr 2026 13:55:01 -0400 Subject: [PATCH 5/6] chore: remove v2 PRD scratch file Working notes; not intended to ship. Co-Authored-By: Claude Opus 4.7 --- PRD-template-connectors-v2.md | 196 ---------------------------------- 1 file changed, 196 deletions(-) delete mode 100644 PRD-template-connectors-v2.md diff --git a/PRD-template-connectors-v2.md b/PRD-template-connectors-v2.md deleted file mode 100644 index 03fd4b2fdae1..000000000000 --- a/PRD-template-connectors-v2.md +++ /dev/null @@ -1,196 +0,0 @@ -# Template Connectors — v2 (restart of PR #8981) - -Reference: https://github.com/rilldata/rill/pull/8981 — `feat-template-connector` branch (108 files, +14,556 / −2,282). - -## Goal - -Move connector schemas, SQL generation, and YAML templating out of hand-written TypeScript and into declarative JSON template definitions served by the runtime. The frontend stops owning per-connector form schemas; instead it fetches them from a new `ListTemplates` RPC and renders YAML through a `GenerateFile` RPC. - -End state, per connector: one JSON file describing the JSON Schema (form), output file templates, and OLAP-specific SQL. No TypeScript schema, no Go-side template code. - -## Why restart - -The original PR grew across ~5 weeks of code review. The add-data flow code in particular went through several iterations: server-side preview moved client-to-server, sync preview became async + debounced, `.env` conflict detection moved client-to-server. Restarting against current `main` lets us land it cleanly in stages instead of carrying all the churn. - - -## In scope: template-connector restart - -### Backend — `runtime/templates/` package - -New Go package, ~1,100 LOC across 7 files (excluding tests): - -- `template.go` — `Template` struct: `Name`, `DisplayName`, `Driver`, `OLAP`, `Tags`, `JSONSchema` (raw), `Files` (output specs), `Icon`, `SmallIcon`, `Description`, `DocsURL`. -- `registry.go` — registry built from `embed.FS` over `definitions/`. Provides `List(tags ...)`, `Get(name)`. Loads at process start. -- `render.go` — renders a template with property values. Produces one or more `GeneratedFile{Path, Blob}`. Handles env-var extraction for `x-secret` properties. -- `funcmap.go` — `text/template` helper functions used by `code_template` strings: `duckdbSQL`, `propVal`, `default`, `azureContainer`, `azureBlobPath`, `clickhouseURLSuffix`, `renderProps`. Delimiters are `[[ ]]` (not `{{ }}`) so YAML's `{{ }}` doesn't collide with templating. -- `env.go` — `ResolveEnvVarNameForKey(driver, key, explicit, existingEnv)` and `ReadEnvKeys(repo)`. Implements the `EnvVarName` resolution + suffix-on-conflict policy (`FOO`, `FOO_1`, `FOO_2`, ...). -- `headers.go` — parses HTTPS-style `headers` strings. - -Tests next to each module + `render_test.go` covering golden YAML for each template. - -### Backend — JSON template definitions - -`runtime/templates/definitions//.json`. Three groups in #8981: - -- `olap/` — 6 files: `clickhouse`, `druid`, `duckdb`, `motherduck`, `pinot`, `starrocks`. -- `duckdb-models/` — 18 files: source connectors that target DuckDB (`s3-duckdb`, `gcs-duckdb`, `azure-duckdb`, `postgres-duckdb`, `snowflake-duckdb`, `bigquery-duckdb`, `athena-duckdb`, `redshift-duckdb`, `mysql-duckdb`, `salesforce-duckdb`, `delta-duckdb`, `iceberg-duckdb`, `https-duckdb`, `local-file-duckdb`, `sqlite-duckdb`, `clickhouse-duckdb`, `duckdb-duckdb`, `supabase-duckdb`). -- `clickhouse-models/` — 12 files: source connectors that target ClickHouse OLAP (`s3-clickhouse`, `gcs-clickhouse`, `azure-clickhouse`, `postgres-clickhouse`, `mysql-clickhouse`, `delta-clickhouse`, `iceberg-clickhouse`, `hudi-clickhouse`, `kafka-clickhouse`, `mongodb-clickhouse`, `https-clickhouse`, `supabase-clickhouse`). - -Note: ClickHouse `local-file-clickhouse` and `sqlite-clickhouse` are intentionally absent — those drivers require ClickHouse server-side `user_files` and aren't usable in Rill Cloud. If we want them to appear at all, they need stub templates with `_reason` so the UI can hide/disable them gracefully (decide in §Open decisions). - -JSON shape (see `s3-duckdb.json` as the reference): - -```json -{ - "name": "s3-duckdb", - "display_name": "Amazon S3", - "driver": "s3", - "olap": "duckdb", - "icon": "AmazonS3", - "small_icon": "AmazonS3Icon", - "tags": ["source", "duckdb", "s3", ...], - "json_schema": { /* JSON Schema with x-* extensions */ }, - "files": [ - { "name": "connector", "path_template": "connectors/[[ .connector_name ]].yaml", "code_template": "..." }, - { "name": "model", "path_template": "models/[[ .model_name ]].yaml", "code_template": "..." } - ] -} -``` - -Custom JSON Schema extensions (must be documented somewhere — likely a README inside `definitions/`): - -- `x-step` — `connector` | `source` | `explorer` (for multi-step flows). -- `x-display` — `radio`, etc. -- `x-grouped-fields` — fields shown when a radio enum value is selected. -- `x-visible-if` — conditional visibility based on other field values. -- `x-secret` — value is a credential, gets routed through `.env`. -- `x-env-var` — explicit env var name override (matches `PropertySpec.EnvVarName`). -- `x-placeholder`, `x-enum-labels`, `x-enum-descriptions`, `x-ui-only`. -- `x-category` — `objectStore` | `warehouse` | `sqlStore` | `olap` | `sourceOnly` | … -- `x-form-height` — `tall` (used by snowflake, salesforce). -- `x-omit-if-default` — generic replacement for the per-driver `key == "managed"` skip used in v1; suppresses property output when value matches schema default. - -### Backend — proto API (`proto/rill/runtime/v1/api.proto`) - -#8981 adds three RPCs. **Decide before starting v2** which we keep: - -- `ListTemplates(ListTemplatesRequest{tags})` → `ListTemplatesResponse{templates: [Template]}`. **Keep.** Drives the `AddDataModal` connector list and form schemas. -- `GenerateFile(GenerateFileRequest{template_name, output, properties, connector_name, preview})` → `GenerateFileResponse{files: [GeneratedFile{path, blob}], env_vars}`. **Keep.** Handles both preview (`preview=true`) and write paths. -- `GenerateTemplate(GenerateTemplateRequest{resource_type, driver, properties, connector_name})` → `GenerateTemplateResponse{blob, env_vars, resource_type, driver}`. **Reconsider.** In #8981 the frontend's `generate-template.ts` *only* calls `GenerateFile`; `GenerateTemplate` appears unused by the new flow. Recommend: drop `GenerateTemplate` entirely in v2, do everything through `GenerateFile`. If there's a CLI/SDK consumer that needs the older shape, document it before keeping. - -Also adds `Template` and `TemplateFile` proto messages — keep both. - -### Backend — runtime server - -- `runtime/server/templates.go` — `ListTemplates` handler. Queries the registry, optionally filters by tags. -- `runtime/server/generate_template.go` — `GenerateFile` (and possibly `GenerateTemplate`) handler. Renders, optionally writes files via `repo.Put` and merges `.env`. -- `runtime/server/server.go` — wires the new RPCs into the gRPC server. - -Hardening items called out in the v1 review (must be in v2): - -- `appendEnvVar` strips newlines and quotes values containing spaces / special characters (env var injection defense). -- Repo access errors in `GenerateTemplate`, `GenerateFile`, `writeRenderedFiles` log warnings, do not silently swallow. -- `defaultVal` doc comment warns against pipeline syntax — only positional `[[ default (expr) "fallback" ]]` is safe. - -### Frontend — `web-common/src/features/sources/modal/` - -This is the heart of the restart. Touch list: - -- **`AddDataModal.svelte`** — replace `connectors` import (static) with `createRuntimeServiceListTemplates` queries: - - `sourceTemplatesQuery` — derived from `instanceQuery.olapConnector`, requests `{tags: ["source", olap]}`. - - `olapTemplatesQuery` — `{tags: ["olap"]}`, no instance dependency. - - Map templates → `ConnectorInfo`. Source vs OLAP split lives in template tags, not frontend constants. - -- **`AddDataForm.svelte`**: - - Remove `onMount` `.env` blob fetch + `existingEnvBlob` state. - - Replace sync `formManager.computeYamlPreview(...)` with debounced (150 ms) async call. Use `onDestroy` to clear the timer. Keep last-valid preview on error so the YAML pane doesn't blank during typing. - - Wrap `paramsError` in a max-h-32 scroll container (already in v1; pull through). - -- **`AddDataFormManager.ts`** — `computeYamlPreview` becomes async. Four old branches collapse into two: - - Multi-step connector step → `GenerateFile{template_name, output: "connector", properties}`. - - Multi-step source/explorer step → `GenerateFile{template_name, output: "model", properties: combinedValues, connector_name}`. - - Single-step connector and single-step source forms route through the same two paths. - - Drop imports of `compileConnectorYAML`, `compileSourceYAML`, `prepareSourceFormData`, `getSchemaSecretKeys`, `getSchemaStringKeys`. These all move server-side. - -- **`connector-schemas.ts`**: - - Add `populateSchemaCache(map)` (test seam) and `registerTemplateSchema(driver, templateName, schema, displayName)` (runtime registration when `ListTemplates` resolves). - - Replace static `multiStepFormSchemas` table with cache populated from API responses. Static fallback may still be useful for SSR / tests — decide in §Open decisions. - - Remove unused exports `getBackendConnectorName`. - -- **`generate-template.ts` (new)**: - - `generateTemplate(client, {resourceType, driver, properties, connectorName})` → wraps `runtimeServiceGenerateFile` with `preview: true`. Handles `driver → templateName` resolution via OLAP cache. - - `mergeEnvVars(client, queryClient, envVars)` — invalidates `.env` query, fetches fresh blob, merges via `replaceOrAddEnvVariable`, returns `{newBlob, originalBlob}` for rollback. - - OLAP cache (`olapCache: Map`) populated by `createConnectorSchemas()` when schemas load. Avoids one extra `GetInstance` round-trip. - -- **Deletes**: - - `web-common/src/features/templates/schemas/*.ts` (16 hand-written schema files: `athena`, `azure`, `bigquery`, `clickhouse`, `delta`, `druid`, `duckdb`, `gcs`, `https`, `iceberg`, `local_file`, `motherduck`, `mysql`, `pinot`, `postgres`, `redshift`, `s3`, `salesforce`, `snowflake`, `sqlite`, `starrocks`, `supabase`). Keep AI schemas (`claude`, `gemini`, `openai`) and DuckLake (`ducklake`, `ducklake-utils`) — they're not in the template path. - - `web-common/src/features/sources/sourceUtils.ts` — `compileSourceYAML`, `prepareSourceFormData`. All YAML generation moves server-side. - - `web-common/src/features/sources/modal/submitAddDataForm.ts` — replaced by direct `GenerateFile` calls + `mergeEnvVars`. - - `web-common/src/features/templates/JSONSchemaFormRenderer.svelte` — verify no other callers before delete. - -- **`web-common/src/features/add-data/manager/selectors.ts`** — same templates-query pattern as `AddDataModal.svelte` but for the new add-data manager. - -- **Tests**: - - `connector-schemas.spec.ts` — gain ~70 LOC of test schema fixtures (because static schemas go away, tests need to seed the cache via `populateSchemaCache`). - - `AddDataFormManager.spec.ts`, `add-source-visibility.spec.ts`, `FormValidation.test.ts` — same pattern. - - `generate-template.spec.ts` (new, ~180 LOC) — covers `mergeEnvVars` edge cases (existing keys, empty file, 404, suffix conflicts). - - `FormValidation.test.ts` imports the actual `s3-duckdb.json` to ensure frontend validation stays in sync with the backend schema. **This is a meaningful pattern — keep it.** - -## Behavior changes (not pure refactor) - -These are real product behavior changes that need explicit sign-off and QA, not just transparent refactor: - -1. **Async, debounced YAML preview.** Was: synchronous on every keystroke against in-memory schema. Now: 150 ms debounce → server round-trip. Risks: race conditions on rapid edits (last RPC may not be the latest input), perceptible lag on slow connections, blank preview during transient errors. Mitigation: keep last-valid preview on error; consider AbortController to cancel in-flight requests on new input. - -2. **Server-side `.env` conflict detection.** Was: client pre-fetched `.env` on form mount, computed conflicts client-side. Now: server resolves names + suffixes inside `GenerateFile`. Cleaner, but means the frontend doesn't know about conflicts until it gets the response. Acceptable, but worth verifying the UX doesn't regress (e.g. same env var name being used twice across two add-source flows). - -3. **ClickHouse `local_file` / `sqlite` become unselectable.** Decide whether to (a) hide them from the UI entirely when OLAP is ClickHouse, (b) show with a disabled state + tooltip explaining why, (c) ship stubs that surface a server-side error message. - -4. **Connector list filtered by OLAP.** `ListTemplates(tags=["source", olap])` means non-supported source/OLAP combos disappear from the picker. Previously the frontend hard-coded which source-OLAP combos were valid. Less code, but make sure the filtering matches the matrix users currently see. - -5. **OLAP detection fix** (`normalizeOlapForTemplate` checks `projectConnectors` too) — this resolves arbitrarily-named OLAP connectors like `clickhouse_1`. Pure bugfix, but worth its own commit so it can be cherry-picked. - -## Sequencing - -Recommended order: - -1. **PR 1: backend templates package** — `runtime/templates/` + `definitions/` JSON + tests. Includes the HTTPS `headers` ConfigProperty addition and the `clickhouseURLSuffix`/`headers.go` SQL helpers. Lands without consumers; new RPCs not yet exposed. -2. **PR 2: proto + server handlers** — `ListTemplates`, `GenerateFile` (drop `GenerateTemplate` if §Open decisions confirms). Generated bindings + Orval client. -3. **PR 3: frontend rewire (add-data modal)** — Switch `AddDataModal`, `AddDataFormManager`, `selectors.ts`, add `generate-template.ts`, update tests. -4. **PR 4: delete TS schemas** — Remove `templates/schemas/*.ts`, `sourceUtils.ts`, `submitAddDataForm.ts` once nothing imports them. Keep this isolated so any forgotten import surfaces clearly. - -PRs 1–4 can be authored concurrently but should land in order. PR 1 + PR 2 could be combined if the diff stays manageable. - -## Open decisions - -- **Drop `GenerateTemplate` RPC?** v1 has both. Frontend goes through `GenerateFile`. Recommend dropping unless a CLI/SDK consumer depends on the simpler shape. **Action: search for callers before final proto land.** -- **Static schema fallback in `connector-schemas.ts`?** v1 keeps the cache populated dynamically. Static fallback could simplify SSR / tests but reintroduces the dual-source-of-truth problem. Recommend: no fallback; tests use `populateSchemaCache` (already the v1 pattern). -- **ClickHouse `local_file` / `sqlite` UX** — see Behavior change #3. -- **Where does the JSON Schema extension dictionary live?** A README in `runtime/templates/definitions/` makes it discoverable for future connector authors. Recommend writing this when PR 1 lands, not retrofitted later. -- **`compileConnectorYAML` (in `web-common/src/features/connectors/code-utils.ts`)** — used by `AddDataFormManager` v1. After the RPC migration, is it still used elsewhere? If not, delete in PR 4. - -## Out of scope - -- AI connectors (`claude`, `gemini`, `openai`) — different add-data flow, leave as TS schemas. -- Error message quality — flagged in v1 as needing a follow-up PR. Confusing "manually setting columns" errors will need their own work after the templates land. -- Embedded dashboard surface — not affected by add-data changes. - -## Test strategy - -- **Backend unit**: each `runtime/templates/` module has its own `_test.go`. `render_test.go` runs golden tests for representative templates. -- **Backend integration**: extend `runtime/server/generate_template_test.go` to cover the preview/write split, env-var merge, and `connector_name` overrides. -- **Frontend unit**: test-only `populateSchemaCache` in `connector-schemas.ts` — schema fixtures live in the test file. `FormValidation.test.ts` imports real JSON to keep validation in sync. -- **Frontend integration**: `generate-template.spec.ts` covers `mergeEnvVars`. The `AddDataFormManager` spec covers preview path branching. -- **Manual QA matrix** (carry over from v1): - - Public GCS, HMAC GCS, Azure connection string, Azure storage key, S3 access keys, Postgres — all on ClickHouse OLAP. - - Same set on DuckDB OLAP. - - Multi-step flow: enter connector creds → save → enter source path → save. - - Single-step flow: ClickHouse OLAP form, DuckDB OLAP form. - - Existing `.env` with conflicting key — verify suffix `_1` rendering. - -## Risks - -- **Server round-trip on every keystroke (debounced)** — worst case a 150 ms input delay + RPC cost. If this lags noticeably on cloud, fall back to client-side preview for the `connector` step (which is small) and keep server preview only for the `source/model` step (which has SQL generation). -- **Schema-divergence between frontend `x-*` extensions and backend renderer** — the JSON files are the source of truth, but the frontend still has to *interpret* `x-step`, `x-visible-if`, `x-grouped-fields`, etc. Drift is possible. Mitigation: a documented list of supported `x-*` keys in the `definitions/README.md` and a typed schema interface on the frontend that fails loudly on unknown keys. -- **Embed of `definitions/`** — `embed.FS` includes everything matching the glob. Make sure no editor swap files / JSON comments slip in (use a strict `//go:embed definitions/*/*.json`). -- **Cherry-pick to release branch** — v1 checks "Intend to cherry-pick into the release branch." Decide early whether v2 is targeting a release branch; if so, keep PR-A through PR 4 individually small enough to cherry-pick. From a310d1ff2bd94cd2e2ce072878ea1008c0b375e2 Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Tue, 28 Apr 2026 14:13:41 -0400 Subject: [PATCH 6/6] feat: add `databricks`, `bigquery`, `snowflake`, `ducklake` OLAPs and databricks source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Catches the templates package up to current `main`, which now ships OLAP support for these drivers and a databricks-as-source path that postdate the original v1 PR. - `olap/databricks.json`, `olap/bigquery.json`, `olap/snowflake.json` — OLAP connector forms backed by `runtime/drivers/{driver}/olap.go`. - `olap/ducklake.json` — driver `duckdb` with the phase-1 single-field ATTACH-clause form from `web-common/.../schemas/ducklake.ts`. - `duckdb-models/databricks-duckdb.json` — Databricks as a DuckDB source (per `feat: databricks as duckdb src` #9288). - Removes `duckdb-models/clickhouse-duckdb.json`; ClickHouse is not a source-into-DuckDB target. Co-Authored-By: Claude Opus 4.7 --- .../duckdb-models/clickhouse-duckdb.json | 50 ----- .../duckdb-models/databricks-duckdb.json | 180 +++++++++++++++ .../templates/definitions/olap/bigquery.json | 56 +++++ .../definitions/olap/databricks.json | 154 +++++++++++++ .../templates/definitions/olap/ducklake.json | 60 +++++ .../templates/definitions/olap/snowflake.json | 206 ++++++++++++++++++ 6 files changed, 656 insertions(+), 50 deletions(-) delete mode 100644 runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json create mode 100644 runtime/templates/definitions/duckdb-models/databricks-duckdb.json create mode 100644 runtime/templates/definitions/olap/bigquery.json create mode 100644 runtime/templates/definitions/olap/databricks.json create mode 100644 runtime/templates/definitions/olap/ducklake.json create mode 100644 runtime/templates/definitions/olap/snowflake.json diff --git a/runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json b/runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json deleted file mode 100644 index b4e018ce9421..000000000000 --- a/runtime/templates/definitions/duckdb-models/clickhouse-duckdb.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "name": "clickhouse-duckdb", - "display_name": "ClickHouse", - "description": "Query ClickHouse tables and ingest into DuckDB", - "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/clickhouse", - "driver": "clickhouse", - "olap": "duckdb", - "icon": "ClickHouse", - "small_icon": "ClickHouseIcon", - "tags": [ - "clickhouse", - "olap", - "duckdb", - "source", - "model" - ], - "json_schema": { - "$schema": "http://json-schema.org/draft-07/schema#", - "type": "object", - "x-category": "sqlStore", - "properties": { - "sql": { - "type": "string", - "title": "SQL", - "description": "SQL query to run against ClickHouse", - "x-placeholder": "SELECT * FROM my_table", - "x-step": "explorer" - }, - "name": { - "type": "string", - "title": "Model name", - "description": "Name for the source model", - "pattern": "^[a-zA-Z0-9_]+$", - "x-placeholder": "my_model", - "x-step": "explorer" - } - }, - "required": [ - "sql", - "name" - ] - }, - "files": [ - { - "name": "model", - "path_template": "models/[[ .model_name ]].yaml", - "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/clickhouse\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" - } - ] -} diff --git a/runtime/templates/definitions/duckdb-models/databricks-duckdb.json b/runtime/templates/definitions/duckdb-models/databricks-duckdb.json new file mode 100644 index 000000000000..c7c112c05e5e --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/databricks-duckdb.json @@ -0,0 +1,180 @@ +{ + "name": "databricks-duckdb", + "display_name": "Databricks", + "description": "Query a Databricks SQL warehouse and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/databricks", + "driver": "databricks", + "olap": "duckdb", + "icon": "Databricks", + "small_icon": "DatabricksIcon", + "tags": [ + "databricks", + "warehouse", + "lakehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Workspace details", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "http_path", + "token", + "catalog", + "schema" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Databricks SQL warehouse hostname", + "x-placeholder": "dbc-xxxxxxxx-xxxx.cloud.databricks.com", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "http_path": { + "type": "string", + "title": "HTTP path", + "description": "HTTP path for the SQL warehouse", + "x-placeholder": "/sql/1.0/warehouses/xxxxxxxxxxxxxxxx", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "token": { + "type": "string", + "title": "Access token", + "description": "Databricks personal access token", + "x-placeholder": "dapi...", + "x-secret": true, + "x-env-var": "DATABRICKS_TOKEN", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "catalog": { + "type": "string", + "title": "Catalog", + "description": "Unity Catalog name (optional)", + "x-placeholder": "main", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Schema within the catalog (optional)", + "x-placeholder": "default", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Databricks DSN.", + "x-placeholder": "token:@:443/?catalog=&schema=", + "x-secret": true, + "x-env-var": "DATABRICKS_DSN", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "http_path", + "token" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: databricks\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/databricks\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/bigquery.json b/runtime/templates/definitions/olap/bigquery.json new file mode 100644 index 000000000000..6073b8e43088 --- /dev/null +++ b/runtime/templates/definitions/olap/bigquery.json @@ -0,0 +1,56 @@ +{ + "name": "bigquery", + "display_name": "BigQuery", + "description": "Connect to Google BigQuery as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/bigquery", + "driver": "bigquery", + "icon": "GoogleBigQuery", + "small_icon": "GoogleBigQueryIcon", + "tags": [ + "bigquery", + "google", + "warehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "google_application_credentials": { + "type": "string", + "title": "GCP credentials", + "description": "Service account JSON (uploaded or pasted)", + "format": "file", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-file-extract": { + "project_id": "project_id" + }, + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-step": "connector" + }, + "project_id": { + "type": "string", + "title": "Project ID", + "description": "Google Cloud project ID to use for queries", + "x-placeholder": "my-project", + "x-hint": "If empty, Rill will use the project ID from your credentials when available.", + "x-step": "connector" + } + }, + "required": [ + "google_application_credentials" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: bigquery\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/databricks.json b/runtime/templates/definitions/olap/databricks.json new file mode 100644 index 000000000000..182bf96b3f11 --- /dev/null +++ b/runtime/templates/definitions/olap/databricks.json @@ -0,0 +1,154 @@ +{ + "name": "databricks", + "display_name": "Databricks", + "description": "Connect to a Databricks SQL warehouse as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/databricks", + "driver": "databricks", + "icon": "Databricks", + "small_icon": "DatabricksIcon", + "tags": [ + "databricks", + "warehouse", + "lakehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Workspace details", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "http_path", + "token", + "catalog", + "schema" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Databricks SQL warehouse hostname", + "x-placeholder": "dbc-xxxxxxxx-xxxx.cloud.databricks.com", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "http_path": { + "type": "string", + "title": "HTTP path", + "description": "HTTP path for the SQL warehouse", + "x-placeholder": "/sql/1.0/warehouses/xxxxxxxxxxxxxxxx", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "token": { + "type": "string", + "title": "Access token", + "description": "Databricks personal access token", + "x-placeholder": "dapi...", + "x-secret": true, + "x-env-var": "DATABRICKS_TOKEN", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "catalog": { + "type": "string", + "title": "Catalog", + "description": "Unity Catalog name (optional; defaults to the workspace default)", + "x-placeholder": "main", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Schema within the catalog (optional; defaults to the workspace default)", + "x-placeholder": "default", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Databricks DSN.", + "x-placeholder": "token:@:443/?catalog=&schema=", + "x-secret": true, + "x-env-var": "DATABRICKS_DSN", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + } + }, + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "http_path", + "token" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: databricks\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/ducklake.json b/runtime/templates/definitions/olap/ducklake.json new file mode 100644 index 000000000000..1df03d09d36c --- /dev/null +++ b/runtime/templates/definitions/olap/ducklake.json @@ -0,0 +1,60 @@ +{ + "name": "ducklake", + "display_name": "DuckLake", + "description": "Connect to a DuckLake catalog as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/ducklake", + "driver": "duckdb", + "icon": "DuckLake", + "small_icon": "DuckLakeIcon", + "tags": [ + "ducklake", + "duckdb", + "lakehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-width": "wide", + "properties": { + "attach": { + "type": "string", + "title": "Attach clause", + "description": "DuckDB ATTACH clause that points at your DuckLake catalog. Include the metadata backend and DATA_PATH.", + "x-placeholder": "'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'data/')", + "x-monospace": true, + "x-hint": "Supported metadata backends: DuckDB file, SQLite, Postgres, MySQL. Data path can be local or object storage (s3://, gs://, azure://).", + "x-step": "connector" + }, + "mode": { + "type": "string", + "title": "Mode", + "description": "Set to 'readwrite' to allow Rill to create and modify tables. Defaults to 'read' (read-only).", + "enum": [ + "read", + "readwrite" + ], + "default": "read", + "x-display": "select", + "x-enum-labels": [ + "Read only", + "Read / write" + ], + "x-omit-if-default": true, + "x-step": "connector" + } + }, + "required": [ + "attach" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: duckdb\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/snowflake.json b/runtime/templates/definitions/olap/snowflake.json new file mode 100644 index 000000000000..84daba37fffc --- /dev/null +++ b/runtime/templates/definitions/olap/snowflake.json @@ -0,0 +1,206 @@ +{ + "name": "snowflake", + "display_name": "Snowflake", + "description": "Connect to a Snowflake data warehouse as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/snowflake", + "driver": "snowflake", + "icon": "Snowflake", + "small_icon": "SnowflakeIcon", + "tags": [ + "snowflake", + "warehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-height": "tall", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "password", + "private_key", + "dsn" + ], + "default": "password", + "x-display": "tabs", + "x-enum-labels": [ + "User/Password", + "Private Key", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "password": [ + "account", + "user", + "password", + "warehouse", + "database", + "schema", + "role" + ], + "private_key": [ + "account", + "user", + "privateKey", + "warehouse", + "database", + "schema", + "role" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "account": { + "type": "string", + "title": "Account identifier", + "description": "Snowflake account identifier (from your Snowflake URL, before .snowflakecomputing.com)", + "x-placeholder": "abc12345.us-east-1", + "x-hint": "e.g. abc12345 or abc12345.us-east-1 — don't include https://", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Snowflake username", + "x-placeholder": "your_username", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Snowflake password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PASSWORD", + "x-visible-if": { + "auth_method": "password" + }, + "x-step": "connector" + }, + "privateKey": { + "type": "string", + "title": "Private key", + "description": "Upload your Snowflake private key file (.pem or .p8)", + "format": "file", + "x-display": "file", + "x-file-accept": ".pem,.p8", + "x-file-encoding": "base64", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PRIVATEKEY", + "x-visible-if": { + "auth_method": "private_key" + }, + "x-step": "connector" + }, + "warehouse": { + "type": "string", + "title": "Warehouse", + "description": "Compute warehouse", + "x-placeholder": "your_warehouse", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Snowflake database", + "x-placeholder": "your_database", + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Default schema", + "x-placeholder": "public", + "x-step": "connector" + }, + "role": { + "type": "string", + "title": "Role", + "description": "Snowflake role", + "x-placeholder": "your_role", + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Snowflake DSN.", + "x-placeholder": "@//?warehouse=&role=", + "x-secret": true, + "x-env-var": "SNOWFLAKE_DSN", + "x-hint": "Include authenticator and privateKey query params for JWT if needed.", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + } + }, + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "password" + } + } + }, + "then": { + "required": [ + "account", + "user", + "password", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "private_key" + } + } + }, + "then": { + "required": [ + "account", + "user", + "privateKey", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: snowflake\n[[ renderProps .props ]]\n" + } + ] +}