diff --git a/runtime/templates/definitions/README.md b/runtime/templates/definitions/README.md new file mode 100644 index 000000000000..f56f25c5a977 --- /dev/null +++ b/runtime/templates/definitions/README.md @@ -0,0 +1,129 @@ +# Template definitions + +JSON files in this tree are the source of truth for the connector and OLAP forms shown in the Rill add-data flow. Each file describes one template: a JSON Schema (which drives the form on the frontend) plus one or more output files (rendered on the backend through `text/template`). + +The runtime loads every `definitions/*/*.json` file at process start via `embed.FS` (see `registry.go`). Stub files — empty or containing only a `_reason` field — are skipped, so an empty JSON file can be used as a placeholder while a connector is being designed. + +## Layout + +``` +definitions/ + olap/ connectors that act as the project's OLAP engine + duckdb-models/ source connectors targeting DuckDB OLAP + clickhouse-models/ source connectors targeting ClickHouse OLAP +``` + +Templates are keyed by their `name` field, not by file path. Filenames follow the convention `-.json` (e.g. `s3-duckdb.json`, `s3-clickhouse.json`) or just `.json` for OLAP connectors. + +## Template shape + +```jsonc +{ + "name": "s3-duckdb", + "display_name": "Amazon S3", + "driver": "s3", + "olap": "duckdb", + "icon": "AmazonS3", + "small_icon": "AmazonS3Icon", + "tags": ["source", "duckdb", "s3", "objectStore"], + "description": "...", + "docs_url": "https://docs.rilldata.com/...", + "json_schema": { /* JSON Schema with x-* extensions, see below */ }, + "files": [ + { "name": "connector", "path_template": "connectors/[[ .connector_name ]].yaml", "code_template": "..." }, + { "name": "model", "path_template": "models/[[ .model_name ]].yaml", "code_template": "..." } + ] +} +``` + +### File outputs + +Each entry in `files` produces one rendered file: + +- `name` — `"connector"` or `"model"`. The `GenerateFile` RPC accepts an `output` filter to render a single entry (used for previewing one step of a multi-step flow). +- `path_template` — Go `text/template` for the output path, relative to the project root. +- `code_template` — inline Go `text/template` for the file contents. +- `code_template_file` — alternative to `code_template`: a path (relative to the JSON definition) to a separate `.tmpl` file. Useful for long templates. If both are set, the file wins. + +Templates use **`[[ ]]` delimiters** (not `{{ }}`) so they don't collide with Rill's runtime templating syntax (`{{ .env.FOO }}`) inside the rendered YAML. + +### Property order + +JSON object key order is preserved at load time (`extractPropertyOrder` in `registry.go`) and re-exposed as `x-property-order` for the frontend. This means the order of fields in `json_schema.properties` is the order they appear in the form and the order their values are emitted by `renderProps`. + +## JSON Schema extensions + +Standard JSON Schema fields (`type`, `properties`, `required`, `enum`, `default`, `description`) work as expected. The following `x-*` extensions are also recognised. Most are interpreted by both the backend renderer and the frontend form — keep them in sync. + +### Form structure + +| Key | Type | Meaning | +|---|---|---| +| `x-step` | `"connector"` \| `"source"` \| `"explorer"` | Which form step a property belongs to. Connector-step props are rendered into the connector YAML; source-step props into the model YAML. Explorer-step props are accessed directly as template variables (e.g. `[[ .sql ]]`) and are not emitted by `renderProps`. Props without `x-step` go into both connector and source outputs. | +| `x-grouped-fields` | `{ enumValue: [...] }` | When the parent property is a radio/select with enum values, lists the fields shown when each value is selected. Used by the frontend to scope visibility to the active branch. | +| `x-visible-if` | `{ otherKey: [allowedValue, ...] }` | Conditional visibility: show this field only when the named field has one of the listed values. | +| `x-tab-group` | `{ tabName: [fieldKey, ...] }` | Groups fields under named tabs in the form. | +| `x-form-height` | `"tall"` | Frontend hint to use a taller form layout (currently used by Snowflake and Salesforce). | +| `x-form-width` | `"wide"` | Frontend hint to use a wider form layout. | +| `x-category` | `"objectStore"` \| `"warehouse"` \| `"sqlStore"` \| `"olap"` \| `"sourceOnly"` | Connector category, used to organise the picker. | + +### Field display + +| Key | Type | Meaning | +|---|---|---| +| `x-display` | `"radio"` \| `"select"` \| `"file"` \| ... | Override the default input control. | +| `x-select-style` | `"rich"` | Render a select with icons / descriptions instead of plain options. | +| `x-placeholder` | string | Placeholder text for text inputs. | +| `x-hint` | string | Helper text shown beneath the input. | +| `x-enum-labels` | array | Display labels for `enum` values, in the same order. | +| `x-enum-descriptions` | array | Sub-labels / descriptions for `enum` values. | +| `x-button-labels` | nested object | Custom labels for nested radio groups (see `clickhouse.json` for an example). | +| `x-informational` | bool | Marks a read-only / explanatory field that is not part of the rendered output. | + +### File uploads + +| Key | Type | Meaning | +|---|---|---| +| `x-file-accept` | string | Comma-separated `accept` filter for file inputs (e.g. `".json"`, `".pem,.p8"`). | +| `x-file-encoding` | `"base64"` \| `"json"` | How the uploaded file should be encoded before submission. | +| `x-file-extract` | `{ schemaKey: jsonPath }` | When the upload is a JSON file, populate other form fields from values inside it (used by BigQuery's service-account flow). | + +### Backend / rendering + +| Key | Type | Meaning | +|---|---|---| +| `x-secret` | bool | Value is a credential. The renderer extracts it to a `.env` variable and emits `{{ .env. }}` in the YAML. | +| `x-env-var` | string | Override the default env var name (`_`). When the chosen name already exists in `.env`, the renderer appends a numeric suffix (`FOO`, `FOO_1`, `FOO_2`, ...). | +| `x-omit-if-default` | bool | Skip rendering the property when its value equals the schema `default`. Replaces the per-driver "skip if `key == "managed"`" hack used in v1. | +| `x-ui-only` | bool | Property exists only to drive the form (e.g. an `auth_method` radio that selects between branches). It is never written to the rendered YAML. | + +## Template helpers + +`text/template` data passed to each file includes: + +- `.driver`, `.connector_name`, `.docs_url`, `.model_name` — basics. +- `.` — every property declared in `json_schema.properties` is exposed as a top-level key. Empty values fall back to the schema `default` if defined, otherwise to `""`. This means conditional template branches can rely on the key existing. +- `.props`, `.config_props`, `.source_props` — pre-processed `[]ProcessedProp` slices for use with `renderProps`. `config_props` and `source_props` correspond to `x-step: connector` and `x-step: source`; `.props` is an alias for `config_props`. + +Functions registered in `funcmap.go`: + +| Name | Purpose | +|---|---| +| `renderProps` | Renders a `[]ProcessedProp` as YAML key/value lines, quoting strings/secrets and leaving numbers and booleans bare. | +| `propVal` | Looks up a single property value by key (used inside conditional branches). | +| `default` | Positional `[[ default (expr) "fallback" ]]` — returns the fallback when the value is empty. **Do not use pipeline syntax** (`[[ expr \| default "fallback" ]]`); `text/template` pipes into the last argument and would swap the values. | +| `indent`, `quote` | String helpers. | +| `duckdbSQL` | Maps a file path to the appropriate DuckDB `read_*` call based on the extension. | +| `s3ToHTTPS`, `gcsToHTTPS` | Convert `s3://` / `gs://` URIs to HTTPS for ClickHouse's `s3()` / `gcs()` functions. | +| `azureContainer`, `azureBlobPath`, `azureEndpoint` | Decompose Azure URIs into the parts ClickHouse's `azureBlobStorage()` function expects. | +| `clickhouseFormat`, `clickhouseURLSuffix` | Map URL extensions to ClickHouse input formats and emit the optional `, Format, headers(...)` suffix when custom HTTPS headers are present. | + +See `funcmap.go` for the exact behaviour of each function and `render_test.go` for golden-output examples. + +## Authoring a new template + +1. Create a JSON file under the appropriate group (`olap/`, `duckdb-models/`, `clickhouse-models/`). +2. Write the JSON Schema for the form. Use `x-step` to split fields between the connector and model steps when the form is multi-step. +3. Add the `files` entries with `path_template` and `code_template` (or `code_template_file`). +4. Add a golden test case in `../render_test.go`. +5. Run `go test ./runtime/templates/...` to verify. diff --git a/runtime/templates/definitions/clickhouse-models/azure-clickhouse.json b/runtime/templates/definitions/clickhouse-models/azure-clickhouse.json new file mode 100644 index 000000000000..011a6cffa9be --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/azure-clickhouse.json @@ -0,0 +1,147 @@ +{ + "name": "azure-clickhouse", + "display_name": "Azure Blob Storage", + "description": "Read Azure Blob Storage files into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/azure", + "driver": "azure", + "olap": "clickhouse", + "icon": "MicrosoftAzureBlobStorage", + "small_icon": "MicrosoftAzureBlobStorageIcon", + "tags": [ + "azure", + "microsoft", + "object-storage", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to Azure Blob Storage", + "enum": [ + "connection_string", + "account_key", + "public" + ], + "default": "connection_string", + "x-display": "radio", + "x-enum-labels": [ + "Connection String", + "Storage Account Key", + "Public" + ], + "x-enum-descriptions": [ + "Provide a full Azure Storage connection string.", + "Provide the storage account name and access key.", + "No credentials needed; the data is publicly accessible." + ], + "x-ui-only": true, + "x-grouped-fields": { + "connection_string": [ + "azure_storage_connection_string" + ], + "account_key": [ + "azure_storage_account", + "azure_storage_key" + ], + "public": [] + } + }, + "azure_storage_connection_string": { + "type": "string", + "title": "Connection string", + "description": "Paste an Azure Storage connection string", + "x-placeholder": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_CONNECTION_STRING", + "x-visible-if": { + "auth_method": "connection_string" + } + }, + "azure_storage_account": { + "type": "string", + "title": "Storage account", + "description": "Azure storage account name", + "x-placeholder": "mystorageaccount", + "x-visible-if": { + "auth_method": "account_key" + } + }, + "azure_storage_key": { + "type": "string", + "title": "Storage key", + "description": "Azure storage account key", + "x-placeholder": "Enter storage key", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_KEY", + "x-visible-if": { + "auth_method": "account_key" + } + }, + "path": { + "type": "string", + "title": "Azure URI", + "description": "Path to your Azure container and blob", + "pattern": "^(azure://|https?://).+", + "errorMessage": { + "pattern": "Must be an Azure URI (e.g. azure://container/path or https://account.blob.core.windows.net/container/path)" + }, + "x-placeholder": "azure://container/path" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "connection_string" + } + } + }, + "then": { + "required": [ + "azure_storage_connection_string" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "account_key" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_key" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |[[ if eq .auth_method \"connection_string\" ]]\n SELECT * FROM azureBlobStorage(\n '[[ propVal .props \"azure_storage_connection_string\" ]]',\n '[[ azureContainer .path ]]',\n '[[ azureBlobPath .path ]]'\n )[[ else if eq .auth_method \"account_key\" ]]\n SELECT * FROM azureBlobStorage(\n '[[ azureEndpoint .path .azure_storage_account ]]',\n '[[ azureContainer .path ]]',\n '[[ azureBlobPath .path ]]',\n '[[ propVal .props \"azure_storage_account\" ]]',\n '[[ propVal .props \"azure_storage_key\" ]]'\n )[[ else ]]\n SELECT * FROM azureBlobStorage(\n '[[ azureEndpoint .path \"\" ]]',\n '[[ azureContainer .path ]]',\n '[[ azureBlobPath .path ]]'\n )[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/delta-clickhouse.json b/runtime/templates/definitions/clickhouse-models/delta-clickhouse.json new file mode 100644 index 000000000000..9032cb4c9427 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/delta-clickhouse.json @@ -0,0 +1,69 @@ +{ + "name": "delta-clickhouse", + "display_name": "Delta Lake", + "description": "Query Delta Lake tables in ClickHouse using deltaLake table function", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/delta", + "driver": "delta", + "olap": "clickhouse", + "icon": "DeltaLake", + "small_icon": "DeltaLakeIcon", + "tags": [ + "delta", + "table-format", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY" + }, + "path": { + "type": "string", + "title": "Delta Table Path", + "description": "S3 path to your Delta table", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/delta_table)" + }, + "x-placeholder": "s3://bucket/delta_table" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_delta_table" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM deltaLake(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/gcs-clickhouse.json b/runtime/templates/definitions/clickhouse-models/gcs-clickhouse.json new file mode 100644 index 000000000000..444d0b5de74a --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/gcs-clickhouse.json @@ -0,0 +1,118 @@ +{ + "name": "gcs-clickhouse", + "display_name": "Google Cloud Storage", + "description": "Read GCS files into ClickHouse using table functions (HMAC keys)", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/gcs", + "driver": "gcs", + "olap": "clickhouse", + "icon": "GoogleCloudStorage", + "small_icon": "GoogleCloudStorageIcon", + "tags": [ + "gcs", + "google", + "object-storage", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to GCS", + "enum": [ + "hmac_key", + "public" + ], + "default": "hmac_key", + "x-display": "radio", + "x-enum-labels": [ + "HMAC Key", + "Public" + ], + "x-enum-descriptions": [ + "Provide GCS HMAC key credentials.", + "No credentials needed; the data is publicly accessible." + ], + "x-ui-only": true, + "x-grouped-fields": { + "hmac_key": [ + "key_id", + "secret" + ], + "public": [] + } + }, + "key_id": { + "type": "string", + "title": "HMAC Access Key", + "description": "GCS HMAC access key ID", + "x-placeholder": "Enter HMAC access key", + "x-secret": true, + "x-env-var": "GCP_ACCESS_KEY_ID", + "x-visible-if": { + "auth_method": "hmac_key" + } + }, + "secret": { + "type": "string", + "title": "HMAC Secret", + "description": "GCS HMAC secret key", + "x-placeholder": "Enter HMAC secret", + "x-secret": true, + "x-env-var": "GCP_SECRET_ACCESS_KEY", + "x-visible-if": { + "auth_method": "hmac_key" + } + }, + "path": { + "type": "string", + "title": "GCS URI", + "description": "Path to your GCS bucket or prefix", + "pattern": "^(gs://|https?://).+", + "errorMessage": { + "pattern": "Must be a GCS URI (e.g. gs://bucket/path or https://storage.googleapis.com/bucket/path)" + }, + "x-placeholder": "gs://bucket/path" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "hmac_key" + } + } + }, + "then": { + "required": [ + "key_id", + "secret" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |[[ if eq .auth_method \"hmac_key\" ]]\n SELECT * FROM gcs(\n '[[ gcsToHTTPS .path ]]',\n '[[ propVal .props \"key_id\" ]]',\n '[[ propVal .props \"secret\" ]]'\n )[[ else ]]\n SELECT * FROM gcs('[[ gcsToHTTPS .path ]]')[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/https-clickhouse.json b/runtime/templates/definitions/clickhouse-models/https-clickhouse.json new file mode 100644 index 000000000000..694832e169ae --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/https-clickhouse.json @@ -0,0 +1,62 @@ +{ + "name": "https-clickhouse", + "display_name": "HTTP(S) Endpoint", + "description": "Read files from HTTP/HTTPS URLs into ClickHouse", + "docs_url": "https://docs.rilldata.com/developers/build/connect/#adding-a-remote-source", + "driver": "https", + "olap": "clickhouse", + "icon": "HTTPS", + "small_icon": "HTTPSIcon", + "tags": [ + "https", + "http", + "url", + "clickhouse", + "source" + ], + + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + + "properties": { + "headers": { + "title": "Headers", + "description": "HTTP headers to include in the request", + "x-display": "key-value", + "x-placeholder": "Header name", + "x-hint": "e.g. Authorization: Bearer " + }, + + "path": { + "type": "string", + "title": "URL", + "description": "HTTP(S) URL to the file", + "pattern": "^https?://.+", + "errorMessage": { + "pattern": "Must be a valid HTTP(S) URL" + }, + "x-placeholder": "https://example.com/data.csv" + }, + + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + + "required": ["path", "name"] + }, + + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\n\ntype: model\nmaterialize: true\n\nconnector: clickhouse\n\nsql: |\n SELECT * FROM url('[[ .path ]]'[[ clickhouseURLSuffix .path .props ]])\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/hudi-clickhouse.json b/runtime/templates/definitions/clickhouse-models/hudi-clickhouse.json new file mode 100644 index 000000000000..9662ba663bc8 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/hudi-clickhouse.json @@ -0,0 +1,69 @@ +{ + "name": "hudi-clickhouse", + "display_name": "Hudi", + "description": "Query Apache Hudi tables in ClickHouse using hudi table function", + "docs_url": "https://clickhouse.com/docs/en/sql-reference/table-functions/hudi", + "driver": "hudi", + "olap": "clickhouse", + "icon": "ApacheHudi", + "small_icon": "ApacheHudiIcon", + "tags": [ + "hudi", + "table-format", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Hudi table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Hudi table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY" + }, + "path": { + "type": "string", + "title": "Hudi Table Path", + "description": "S3 path to your Hudi table", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/hudi_table)" + }, + "x-placeholder": "s3://bucket/hudi_table" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_hudi_table" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM hudi(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/iceberg-clickhouse.json b/runtime/templates/definitions/clickhouse-models/iceberg-clickhouse.json new file mode 100644 index 000000000000..6dc5010c6237 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/iceberg-clickhouse.json @@ -0,0 +1,69 @@ +{ + "name": "iceberg-clickhouse", + "display_name": "Iceberg", + "description": "Query Apache Iceberg tables in ClickHouse using icebergS3 table function", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/iceberg", + "driver": "iceberg", + "olap": "clickhouse", + "icon": "ApacheIceberg", + "small_icon": "ApacheIcebergIcon", + "tags": [ + "iceberg", + "table-format", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY" + }, + "path": { + "type": "string", + "title": "Iceberg Table Path", + "description": "S3 path to your Iceberg table metadata", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/warehouse/my_table)" + }, + "x-placeholder": "s3://bucket/warehouse/my_table" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_iceberg_table" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM icebergS3(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/kafka-clickhouse.json b/runtime/templates/definitions/clickhouse-models/kafka-clickhouse.json new file mode 100644 index 000000000000..c3a36ae77419 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/kafka-clickhouse.json @@ -0,0 +1,76 @@ +{ + "name": "kafka-clickhouse", + "display_name": "Kafka", + "description": "Read Kafka topics into ClickHouse using Kafka table engine", + "docs_url": "https://clickhouse.com/docs/en/engines/table-engines/integrations/kafka", + "driver": "kafka", + "olap": "clickhouse", + "icon": "ApacheKafka", + "small_icon": "ApacheKafkaIcon", + "tags": [ + "kafka", + "streaming", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "broker_list": { + "type": "string", + "title": "Broker List", + "description": "Comma-separated list of Kafka brokers", + "x-placeholder": "broker1:9092,broker2:9092" + }, + "topic": { + "type": "string", + "title": "Topic", + "description": "Kafka topic to consume from", + "x-placeholder": "my_topic" + }, + "group_name": { + "type": "string", + "title": "Consumer Group", + "description": "Kafka consumer group name", + "x-placeholder": "rill_consumer_group" + }, + "format": { + "type": "string", + "title": "Message Format", + "description": "Format of messages in the Kafka topic", + "enum": [ + "JSONEachRow", + "CSV", + "TSV", + "Avro", + "Parquet" + ], + "default": "JSONEachRow", + "x-display": "select" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "broker_list", + "topic", + "group_name", + "format", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM kafka(\n '[[ .broker_list ]]',\n '[[ .topic ]]',\n '[[ .group_name ]]',\n '[[ .format ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/mongodb-clickhouse.json b/runtime/templates/definitions/clickhouse-models/mongodb-clickhouse.json new file mode 100644 index 000000000000..0f405faa94e9 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/mongodb-clickhouse.json @@ -0,0 +1,87 @@ +{ + "name": "mongodb-clickhouse", + "display_name": "MongoDB", + "description": "Query MongoDB collections in ClickHouse using mongodb table function", + "docs_url": "https://clickhouse.com/docs/en/sql-reference/table-functions/mongodb", + "driver": "mongodb", + "olap": "clickhouse", + "icon": "MongoDB", + "small_icon": "MongoDBIcon", + "tags": [ + "mongodb", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "MongoDB server hostname or IP", + "x-placeholder": "localhost" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MongoDB server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "27017", + "x-placeholder": "27017" + }, + "database": { + "type": "string", + "title": "Database", + "description": "MongoDB database name", + "x-placeholder": "my_database" + }, + "collection": { + "type": "string", + "title": "Collection", + "description": "MongoDB collection name", + "x-placeholder": "my_collection" + }, + "user": { + "type": "string", + "title": "Username", + "description": "MongoDB user", + "x-placeholder": "mongo_user" + }, + "password": { + "type": "string", + "title": "Password", + "description": "MongoDB password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "MONGODB_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "database", + "collection", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM mongodb(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"27017\" ]]',\n '[[ .database ]]',\n '[[ .collection ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/mysql-clickhouse.json b/runtime/templates/definitions/clickhouse-models/mysql-clickhouse.json new file mode 100644 index 000000000000..f7438e521415 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/mysql-clickhouse.json @@ -0,0 +1,87 @@ +{ + "name": "mysql-clickhouse", + "display_name": "MySQL", + "description": "Read MySQL tables into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/mysql", + "driver": "mysql", + "olap": "clickhouse", + "icon": "MySQL", + "small_icon": "MySqlIcon", + "tags": [ + "mysql", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "MySQL server hostname or IP", + "x-placeholder": "localhost" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MySQL server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "3306", + "x-placeholder": "3306" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "my_database" + }, + "table": { + "type": "string", + "title": "Table", + "description": "Table name to query", + "x-placeholder": "my_table" + }, + "user": { + "type": "string", + "title": "Username", + "description": "MySQL user", + "x-placeholder": "mysql" + }, + "password": { + "type": "string", + "title": "Password", + "description": "MySQL password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "MYSQL_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "database", + "table", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM mysql(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"3306\" ]]',\n '[[ .database ]]',\n '[[ .table ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/postgres-clickhouse.json b/runtime/templates/definitions/clickhouse-models/postgres-clickhouse.json new file mode 100644 index 000000000000..97eac39a41ff --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/postgres-clickhouse.json @@ -0,0 +1,88 @@ +{ + "name": "postgres-clickhouse", + "display_name": "PostgreSQL", + "description": "Read PostgreSQL tables into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "postgres", + "olap": "clickhouse", + "icon": "Postgres", + "small_icon": "PostgresIcon", + "tags": [ + "postgres", + "postgresql", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "Postgres server hostname or IP", + "x-placeholder": "localhost" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Postgres server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "5432", + "x-placeholder": "5432" + }, + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres" + }, + "table": { + "type": "string", + "title": "Table", + "description": "Table name to query", + "x-placeholder": "my_table" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Postgres user", + "x-placeholder": "postgres" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Postgres password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "POSTGRES_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "dbname", + "table", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM postgresql(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"5432\" ]]',\n '[[ .dbname ]]',\n '[[ .table ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/s3-clickhouse.json b/runtime/templates/definitions/clickhouse-models/s3-clickhouse.json new file mode 100644 index 000000000000..5f92243b79b8 --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/s3-clickhouse.json @@ -0,0 +1,118 @@ +{ + "name": "s3-clickhouse", + "display_name": "Amazon S3", + "description": "Read S3 files into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/s3", + "driver": "s3", + "olap": "clickhouse", + "icon": "AmazonS3", + "small_icon": "AmazonS3Icon", + "tags": [ + "s3", + "aws", + "object-storage", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to S3", + "enum": [ + "access_key", + "public" + ], + "default": "access_key", + "x-display": "radio", + "x-enum-labels": [ + "Access Key", + "Public" + ], + "x-enum-descriptions": [ + "Provide AWS access key credentials.", + "No credentials needed; the data is publicly accessible." + ], + "x-ui-only": true, + "x-grouped-fields": { + "access_key": [ + "aws_access_key_id", + "aws_secret_access_key" + ], + "public": [] + } + }, + "aws_access_key_id": { + "type": "string", + "title": "Access Key ID", + "description": "AWS access key ID for the bucket", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-visible-if": { + "auth_method": "access_key" + } + }, + "aws_secret_access_key": { + "type": "string", + "title": "Secret Access Key", + "description": "AWS secret access key for the bucket", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-visible-if": { + "auth_method": "access_key" + } + }, + "path": { + "type": "string", + "title": "S3 URI", + "description": "Path to your S3 bucket or prefix", + "pattern": "^(s3://|https?://).+", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/path or https://bucket.s3.amazonaws.com/path)" + }, + "x-placeholder": "s3://bucket/path" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "access_key" + } + } + }, + "then": { + "required": [ + "aws_access_key_id", + "aws_secret_access_key" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |[[ if eq .auth_method \"access_key\" ]]\n SELECT * FROM s3(\n '[[ s3ToHTTPS .path ]]',\n '[[ propVal .props \"aws_access_key_id\" ]]',\n '[[ propVal .props \"aws_secret_access_key\" ]]'\n )[[ else ]]\n SELECT * FROM s3('[[ s3ToHTTPS .path ]]')[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/clickhouse-models/supabase-clickhouse.json b/runtime/templates/definitions/clickhouse-models/supabase-clickhouse.json new file mode 100644 index 000000000000..9ede0b9a27dc --- /dev/null +++ b/runtime/templates/definitions/clickhouse-models/supabase-clickhouse.json @@ -0,0 +1,88 @@ +{ + "name": "supabase-clickhouse", + "display_name": "Supabase", + "description": "Read Supabase tables into ClickHouse using table functions", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "supabase", + "olap": "clickhouse", + "icon": "Supabase", + "small_icon": "SupabaseIcon", + "tags": [ + "supabase", + "postgres", + "database", + "clickhouse", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "host": { + "type": "string", + "title": "Host", + "description": "Supabase database host", + "x-placeholder": "aws-0-[region].pooler.supabase.com" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Supabase database port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "5432", + "x-placeholder": "5432" + }, + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres" + }, + "table": { + "type": "string", + "title": "Table", + "description": "Table name to query", + "x-placeholder": "my_table" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Supabase database user", + "x-placeholder": "postgres.[ref]" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Supabase database password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SUPABASE_PASSWORD" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model" + } + }, + "required": [ + "host", + "dbname", + "table", + "user", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/reference/project-files/models\ntype: model\nmaterialize: true\nconnector: clickhouse\nsql: |\n SELECT * FROM postgresql(\n '[[ .host ]]:[[ default (propVal .props \"port\") \"5432\" ]]',\n '[[ .dbname ]]',\n '[[ .table ]]',\n '[[ propVal .props \"user\" ]]',\n '[[ propVal .props \"password\" ]]'\n )\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/athena-duckdb.json b/runtime/templates/definitions/duckdb-models/athena-duckdb.json new file mode 100644 index 000000000000..372babbe7faf --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/athena-duckdb.json @@ -0,0 +1,88 @@ +{ + "name": "athena-duckdb", + "display_name": "Amazon Athena", + "description": "Query Amazon Athena and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/athena", + "driver": "athena", + "olap": "duckdb", + "icon": "AmazonAthena", + "small_icon": "AthenaIcon", + "tags": [ + "athena", + "aws", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS access key ID", + "description": "AWS access key ID used to authenticate to Athena", + "x-placeholder": "your_access_key_id", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS secret access key", + "description": "AWS secret access key paired with the access key ID", + "x-placeholder": "your_secret_access_key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector" + }, + "output_location": { + "type": "string", + "title": "S3 output location", + "description": "S3 URI where Athena should write query results (e.g., s3://bucket/path/)", + "pattern": "^s3://.+", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g., s3://bucket/path/)" + }, + "x-placeholder": "s3://bucket-name/path/", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "output_location", + "sql", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: athena\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/athena\n\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/azure-duckdb.json b/runtime/templates/definitions/duckdb-models/azure-duckdb.json new file mode 100644 index 000000000000..49040dd0a36b --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/azure-duckdb.json @@ -0,0 +1,196 @@ +{ + "name": "azure-duckdb", + "display_name": "Azure Blob Storage", + "description": "Read files from Azure Blob Storage into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/azure", + "driver": "azure", + "olap": "duckdb", + "icon": "MicrosoftAzureBlobStorage", + "small_icon": "MicrosoftAzureBlobStorageIcon", + "tags": [ + "azure", + "microsoft", + "object-storage", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to Azure Blob Storage", + "enum": [ + "connection_string", + "account_key", + "sas_token", + "public" + ], + "default": "connection_string", + "x-display": "radio", + "x-enum-labels": [ + "Connection String", + "Storage Account Key", + "SAS Token", + "Public" + ], + "x-enum-descriptions": [ + "Provide a full Azure Storage connection string.", + "Provide the storage account name and access key.", + "Provide the storage account name and SAS token.", + "Access publicly readable blobs without credentials." + ], + "x-ui-only": true, + "x-grouped-fields": { + "connection_string": [ + "azure_storage_connection_string" + ], + "account_key": [ + "azure_storage_account", + "azure_storage_key" + ], + "sas_token": [ + "azure_storage_account", + "azure_storage_sas_token" + ], + "public": [] + }, + "x-step": "connector" + }, + "azure_storage_connection_string": { + "type": "string", + "title": "Connection string", + "description": "Paste an Azure Storage connection string", + "x-placeholder": "Enter Azure storage connection string", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_CONNECTION_STRING", + "x-step": "connector", + "x-visible-if": { + "auth_method": "connection_string" + } + }, + "azure_storage_account": { + "type": "string", + "title": "Storage account", + "description": "The name of the Azure storage account", + "x-placeholder": "Enter Azure storage account", + "x-step": "connector", + "x-visible-if": { + "auth_method": [ + "account_key", + "sas_token" + ] + } + }, + "azure_storage_key": { + "type": "string", + "title": "Access key", + "description": "Primary or secondary access key for the storage account", + "x-placeholder": "Enter Azure storage access key", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_KEY", + "x-step": "connector", + "x-visible-if": { + "auth_method": "account_key" + } + }, + "azure_storage_sas_token": { + "type": "string", + "title": "SAS token", + "description": "Shared Access Signature token for the storage account (starting with ?sv=)", + "x-placeholder": "Enter Azure SAS token", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_SAS_TOKEN", + "x-step": "connector", + "x-visible-if": { + "auth_method": "sas_token" + } + }, + "path": { + "type": "string", + "title": "Blob URI", + "description": "URI to the Azure blob container or directory", + "pattern": "^(azure://|https?://).+", + "errorMessage": { + "pattern": "Must be an Azure URI (e.g. azure://container/path or https://account.blob.core.windows.net/container/path)" + }, + "x-placeholder": "azure://container/path", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "connection_string" + } + } + }, + "then": { + "required": [ + "azure_storage_connection_string" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "account_key" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_key" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "sas_token" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_sas_token" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: azure\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path false ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/bigquery-duckdb.json b/runtime/templates/definitions/duckdb-models/bigquery-duckdb.json new file mode 100644 index 000000000000..44f2085c387f --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/bigquery-duckdb.json @@ -0,0 +1,81 @@ +{ + "name": "bigquery-duckdb", + "display_name": "BigQuery", + "description": "Query Google BigQuery and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/bigquery", + "driver": "bigquery", + "olap": "duckdb", + "icon": "GoogleBigQuery", + "small_icon": "GoogleBigQueryIcon", + "tags": [ + "bigquery", + "google", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "google_application_credentials": { + "type": "string", + "title": "GCP credentials", + "description": "Service account JSON (uploaded or pasted)", + "format": "file", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-file-extract": { + "project_id": "project_id" + }, + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-step": "connector" + }, + "project_id": { + "type": "string", + "title": "Project ID", + "description": "Google Cloud project ID to use for queries", + "x-placeholder": "my-project", + "x-hint": "If empty, Rill will use the project ID from your credentials when available.", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "google_application_credentials", + "project_id", + "sql", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: bigquery\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/bigquery\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/databricks-duckdb.json b/runtime/templates/definitions/duckdb-models/databricks-duckdb.json new file mode 100644 index 000000000000..c7c112c05e5e --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/databricks-duckdb.json @@ -0,0 +1,180 @@ +{ + "name": "databricks-duckdb", + "display_name": "Databricks", + "description": "Query a Databricks SQL warehouse and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/databricks", + "driver": "databricks", + "olap": "duckdb", + "icon": "Databricks", + "small_icon": "DatabricksIcon", + "tags": [ + "databricks", + "warehouse", + "lakehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Workspace details", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "http_path", + "token", + "catalog", + "schema" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Databricks SQL warehouse hostname", + "x-placeholder": "dbc-xxxxxxxx-xxxx.cloud.databricks.com", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "http_path": { + "type": "string", + "title": "HTTP path", + "description": "HTTP path for the SQL warehouse", + "x-placeholder": "/sql/1.0/warehouses/xxxxxxxxxxxxxxxx", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "token": { + "type": "string", + "title": "Access token", + "description": "Databricks personal access token", + "x-placeholder": "dapi...", + "x-secret": true, + "x-env-var": "DATABRICKS_TOKEN", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "catalog": { + "type": "string", + "title": "Catalog", + "description": "Unity Catalog name (optional)", + "x-placeholder": "main", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Schema within the catalog (optional)", + "x-placeholder": "default", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Databricks DSN.", + "x-placeholder": "token:@:443/?catalog=&schema=", + "x-secret": true, + "x-env-var": "DATABRICKS_DSN", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "http_path", + "token" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: databricks\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/databricks\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/delta-duckdb.json b/runtime/templates/definitions/duckdb-models/delta-duckdb.json new file mode 100644 index 000000000000..f0d2adf9b300 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/delta-duckdb.json @@ -0,0 +1,193 @@ +{ + "name": "delta-duckdb", + "display_name": "Delta Lake", + "description": "Query Delta Lake tables using DuckDB's delta extension", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/delta", + "driver": "delta", + "olap": "duckdb", + "icon": "DeltaLake", + "small_icon": "DeltaLakeIcon", + "tags": [ + "delta", + "table-format", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "storage_type": { + "type": "string", + "title": "Storage type", + "description": "Where your Delta table is stored", + "enum": [ + "s3", + "gcs", + "azure", + "local" + ], + "default": "s3", + "x-display": "radio", + "x-enum-labels": [ + "Amazon S3", + "Google Cloud Storage", + "Azure Blob Storage", + "Local / Public" + ], + "x-enum-descriptions": [ + "Delta table stored in an S3 bucket.", + "Delta table stored in a GCS bucket.", + "Delta table stored in Azure Blob Storage.", + "Local path or publicly accessible URL." + ], + "x-ui-only": true, + "x-grouped-fields": { + "s3": [ + "aws_access_key_id", + "aws_secret_access_key", + "aws_region" + ], + "gcs": [ + "google_application_credentials" + ], + "azure": [ + "azure_storage_account", + "azure_storage_key" + ], + "local": [] + }, + "x-step": "connector" + }, + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-visible-if": { + "storage_type": "s3" + }, + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Delta table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-visible-if": { + "storage_type": "s3" + }, + "x-step": "connector" + }, + "aws_region": { + "type": "string", + "title": "AWS Region", + "description": "AWS region of the S3 bucket", + "x-placeholder": "us-east-1", + "x-visible-if": { + "storage_type": "s3" + }, + "x-step": "connector" + }, + "google_application_credentials": { + "type": "string", + "title": "Service Account Key (JSON)", + "description": "Upload a JSON key file for a service account with GCS access", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-visible-if": { + "storage_type": "gcs" + }, + "x-step": "connector" + }, + "azure_storage_account": { + "type": "string", + "title": "Storage Account Name", + "description": "Azure storage account name", + "x-placeholder": "mystorageaccount", + "x-visible-if": { + "storage_type": "azure" + }, + "x-step": "connector" + }, + "azure_storage_key": { + "type": "string", + "title": "Storage Account Key", + "description": "Azure storage account access key", + "x-placeholder": "Enter storage account key", + "x-secret": true, + "x-env-var": "AZURE_STORAGE_KEY", + "x-visible-if": { + "storage_type": "azure" + }, + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "Delta Table Path", + "description": "Path to your Delta table (e.g. s3://bucket/delta_table, gs://bucket/delta_table)", + "x-placeholder": "s3://bucket/delta_table", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_delta_table", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "storage_type": { + "const": "s3" + } + } + }, + "then": { + "required": [ + "aws_access_key_id", + "aws_secret_access_key" + ] + } + }, + { + "if": { + "properties": { + "storage_type": { + "const": "azure" + } + } + }, + "then": { + "required": [ + "azure_storage_account", + "azure_storage_key" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\n\nsql: |\n SELECT * FROM delta_scan('[[ .path ]]');\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/duckdb-duckdb.json b/runtime/templates/definitions/duckdb-models/duckdb-duckdb.json new file mode 100644 index 000000000000..f5612f7948e9 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/duckdb-duckdb.json @@ -0,0 +1,47 @@ +{ + "name": "duckdb-duckdb", + "display_name": "DuckDB", + "description": "Write SQL models against DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/duckdb", + "driver": "duckdb", + "olap": "duckdb", + "icon": "DuckDB", + "small_icon": "DuckDBIcon", + "tags": [ + "duckdb", + "model" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + "properties": { + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/duckdb\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/gcs-duckdb.json b/runtime/templates/definitions/duckdb-models/gcs-duckdb.json new file mode 100644 index 000000000000..7a47a9d0dd8f --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/gcs-duckdb.json @@ -0,0 +1,164 @@ +{ + "name": "gcs-duckdb", + "display_name": "Google Cloud Storage", + "description": "Read files from Google Cloud Storage into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/gcs", + "driver": "gcs", + "olap": "duckdb", + "icon": "GoogleCloudStorage", + "small_icon": "GoogleCloudStorageIcon", + "tags": [ + "gcs", + "google", + "object-storage", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to GCS", + "enum": [ + "credentials", + "hmac", + "public" + ], + "default": "credentials", + "x-display": "radio", + "x-enum-labels": [ + "GCP credentials", + "HMAC keys", + "Public" + ], + "x-enum-descriptions": [ + "Upload a JSON key file for a service account with GCS access.", + "Use HMAC access key and secret for S3-compatible authentication.", + "Access publicly readable buckets without credentials." + ], + "x-ui-only": true, + "x-grouped-fields": { + "credentials": [ + "google_application_credentials" + ], + "hmac": [ + "key_id", + "secret" + ], + "public": [] + }, + "x-step": "connector" + }, + "google_application_credentials": { + "type": "string", + "title": "Service account key", + "description": "Upload a JSON key file for a service account with GCS access.", + "format": "file", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-step": "connector", + "x-visible-if": { + "auth_method": "credentials" + } + }, + "key_id": { + "type": "string", + "title": "Access Key ID", + "description": "HMAC access key ID for S3-compatible authentication", + "x-placeholder": "Enter your HMAC access key ID", + "x-secret": true, + "x-env-var": "GCP_ACCESS_KEY_ID", + "x-step": "connector", + "x-visible-if": { + "auth_method": "hmac" + } + }, + "secret": { + "type": "string", + "title": "Secret Access Key", + "description": "HMAC secret access key for S3-compatible authentication", + "x-placeholder": "Enter your HMAC secret access key", + "x-secret": true, + "x-env-var": "GCP_SECRET_ACCESS_KEY", + "x-step": "connector", + "x-visible-if": { + "auth_method": "hmac" + } + }, + "path": { + "type": "string", + "title": "GCS URI", + "description": "Path to your GCS bucket or prefix", + "pattern": "^gs://[^/]+(/.*)?$", + "errorMessage": { + "pattern": "Must be a GS URI (e.g. gs://bucket/path)" + }, + "x-placeholder": "gs://bucket/path", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "credentials" + } + } + }, + "then": { + "required": [ + "google_application_credentials" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "hmac" + } + } + }, + "then": { + "required": [ + "key_id", + "secret" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: gcs\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path false ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/https-duckdb.json b/runtime/templates/definitions/duckdb-models/https-duckdb.json new file mode 100644 index 000000000000..477ce5a97cd0 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/https-duckdb.json @@ -0,0 +1,80 @@ +{ + "name": "https-duckdb", + "display_name": "HTTP(S) Endpoint", + "description": "Read files from an HTTP/HTTPS URL into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connect/#adding-a-remote-source", + "driver": "https", + "olap": "duckdb", + "icon": "HTTPS", + "small_icon": "HTTPSIcon", + "tags": [ + "https", + "http", + "url", + "file-store", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "fileStore", + "x-button-labels": { + "*": { + "*": { + "idle": "Continue", + "loading": "Continuing..." + } + } + }, + "properties": { + "headers": { + "title": "Headers", + "description": "HTTP headers to include in the request", + "x-display": "key-value", + "x-placeholder": "Header name", + "x-hint": "e.g. Authorization: Bearer ", + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "URI", + "description": "HTTP/HTTPS URL to the remote file", + "pattern": "^https?://.+", + "errorMessage": { + "pattern": "URI must start with http:// or https://" + }, + "x-placeholder": "https://example.com/file.csv", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "errorMessage": { + "pattern": "Model name can only contain letters, numbers, and underscores" + }, + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: https\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path true ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/iceberg-duckdb.json b/runtime/templates/definitions/duckdb-models/iceberg-duckdb.json new file mode 100644 index 000000000000..563b15657433 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/iceberg-duckdb.json @@ -0,0 +1,78 @@ +{ + "name": "iceberg-duckdb", + "display_name": "Iceberg", + "description": "Query Apache Iceberg tables using DuckDB's iceberg extension", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/iceberg", + "driver": "iceberg", + "olap": "duckdb", + "icon": "ApacheIceberg", + "small_icon": "ApacheIcebergIcon", + "tags": [ + "iceberg", + "table-format", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS Access Key ID", + "description": "Access key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS Secret Access Key", + "description": "Secret key for the S3 bucket containing your Iceberg table", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector" + }, + "aws_region": { + "type": "string", + "title": "AWS Region", + "description": "AWS region of the S3 bucket", + "x-placeholder": "us-east-1", + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "Iceberg Table Path", + "description": "S3 path to your Iceberg table metadata", + "pattern": "^s3://.*", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/warehouse/my_table)" + }, + "x-placeholder": "s3://bucket/warehouse/my_table", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_iceberg_table", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\n\nsql: |\n SELECT * FROM iceberg_scan('[[ .path ]]');\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/local-file-duckdb.json b/runtime/templates/definitions/duckdb-models/local-file-duckdb.json new file mode 100644 index 000000000000..a32c76ae7d51 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/local-file-duckdb.json @@ -0,0 +1,49 @@ +{ + "name": "local_file-duckdb", + "display_name": "Local File", + "description": "Read local files into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/duckdb", + "driver": "local_file", + "olap": "duckdb", + "icon": "LocalFile", + "small_icon": "LocalFileIcon", + "tags": [ + "local-file", + "file", + "file-store", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "fileStore", + "properties": { + "path": { + "type": "string", + "title": "Path", + "description": "Local file path or glob (relative to project root)", + "x-placeholder": "data/*.parquet", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Source name", + "description": "Name for the source", + "x-placeholder": "my_new_source", + "x-step": "explorer" + } + }, + "required": [ + "path", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/duckdb\ntype: model\nconnector: duckdb\nsql: |\n [[ duckdbSQL .path true ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/mysql-duckdb.json b/runtime/templates/definitions/duckdb-models/mysql-duckdb.json new file mode 100644 index 000000000000..adbcd40d26dc --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/mysql-duckdb.json @@ -0,0 +1,171 @@ +{ + "name": "mysql-duckdb", + "display_name": "MySQL", + "description": "Query MySQL databases and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/mysql", + "driver": "mysql", + "olap": "duckdb", + "icon": "MySQL", + "small_icon": "MySqlIcon", + "tags": [ + "mysql", + "database", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "database", + "user", + "password", + "ssl-mode" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "MySQL connection string", + "description": "Full DSN, e.g. mysql://user:password@host:3306/database?ssl-mode=REQUIRED", + "x-placeholder": "mysql://user:password@host:3306/database", + "x-secret": true, + "x-env-var": "MYSQL_DSN", + "x-hint": "Use DSN or fill host/user/password/database below (not both at once).", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "MySQL server hostname or IP", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MySQL server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "3306", + "x-placeholder": "3306", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "my_database", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "MySQL user", + "x-placeholder": "mysql", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "MySQL password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "MYSQL_PASSWORD", + "x-step": "connector" + }, + "ssl-mode": { + "type": "string", + "title": "SSL mode", + "description": "Use DISABLED, PREFERRED, or REQUIRED", + "enum": [ + "DISABLED", + "PREFERRED", + "REQUIRED" + ], + "x-placeholder": "PREFERRED", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your database", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + }, + "else": { + "required": [ + "host", + "database", + "user" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: mysql\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/mysql\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/postgres-duckdb.json b/runtime/templates/definitions/duckdb-models/postgres-duckdb.json new file mode 100644 index 000000000000..2a32814ad2fb --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/postgres-duckdb.json @@ -0,0 +1,173 @@ +{ + "name": "postgres-duckdb", + "display_name": "PostgreSQL", + "description": "Query PostgreSQL databases and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "postgres", + "olap": "duckdb", + "icon": "Postgres", + "small_icon": "PostgresIcon", + "tags": [ + "postgres", + "postgresql", + "database", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "user", + "password", + "dbname", + "sslmode" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Postgres connection string", + "description": "e.g. postgresql://user:password@host:5432/dbname?sslmode=require", + "x-placeholder": "postgresql://postgres:postgres@localhost:5432/postgres", + "x-secret": true, + "x-env-var": "POSTGRES_DSN", + "x-hint": "Use a DSN or provide host/user/password/dbname below (but not both).", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Postgres server hostname or IP", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Postgres server port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "5432", + "x-placeholder": "5432", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Postgres user", + "x-placeholder": "postgres", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Postgres password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "POSTGRES_PASSWORD", + "x-step": "connector" + }, + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres", + "x-step": "connector" + }, + "sslmode": { + "type": "string", + "title": "SSL mode", + "description": "Use disable, allow, prefer, require", + "enum": [ + "disable", + "allow", + "prefer", + "require" + ], + "x-placeholder": "require", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your database", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + }, + "else": { + "required": [ + "host", + "user", + "dbname" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: postgres\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/postgres\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/redshift-duckdb.json b/runtime/templates/definitions/duckdb-models/redshift-duckdb.json new file mode 100644 index 000000000000..dc355e6b0c74 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/redshift-duckdb.json @@ -0,0 +1,105 @@ +{ + "name": "redshift-duckdb", + "display_name": "Amazon Redshift", + "description": "Query Amazon Redshift and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/redshift", + "driver": "redshift", + "olap": "duckdb", + "icon": "AmazonRedshift", + "small_icon": "RedshiftIcon", + "tags": [ + "redshift", + "aws", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "properties": { + "aws_access_key_id": { + "type": "string", + "title": "AWS access key ID", + "description": "AWS access key ID", + "x-placeholder": "your_access_key_id", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector" + }, + "aws_secret_access_key": { + "type": "string", + "title": "AWS secret access key", + "description": "AWS secret access key", + "x-placeholder": "your_secret_access_key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector" + }, + "region": { + "type": "string", + "title": "AWS region", + "description": "AWS region (e.g. us-east-1)", + "x-placeholder": "us-east-1", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Redshift database name", + "x-placeholder": "dev", + "x-step": "connector" + }, + "workgroup": { + "type": "string", + "title": "Workgroup", + "description": "Redshift Serverless workgroup name", + "x-placeholder": "default", + "x-step": "connector" + }, + "cluster_identifier": { + "type": "string", + "title": "Cluster identifier", + "description": "Redshift cluster identifier (use when not using serverless)", + "x-placeholder": "redshift-cluster-1", + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "aws_access_key_id", + "aws_secret_access_key", + "database", + "sql", + "name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: redshift\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/redshift\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n[[ end ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/s3-duckdb.json b/runtime/templates/definitions/duckdb-models/s3-duckdb.json new file mode 100644 index 000000000000..3b6827fac7a9 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/s3-duckdb.json @@ -0,0 +1,164 @@ +{ + "name": "s3-duckdb", + "display_name": "Amazon S3", + "description": "Read files from Amazon S3 into DuckDB using the appropriate file reader", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/s3", + "driver": "s3", + "olap": "duckdb", + "icon": "AmazonS3", + "small_icon": "AmazonS3Icon", + "tags": [ + "s3", + "aws", + "object-storage", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "objectStore", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "description": "Choose how to authenticate to S3", + "enum": [ + "access_keys", + "public" + ], + "default": "access_keys", + "x-display": "radio", + "x-enum-labels": [ + "Access keys", + "Public" + ], + "x-enum-descriptions": [ + "Use AWS access key ID and secret access key.", + "Access publicly readable buckets without credentials." + ], + "x-ui-only": true, + "x-grouped-fields": { + "access_keys": [ + "aws_access_key_id", + "aws_secret_access_key", + "region", + "endpoint", + "aws_role_arn" + ], + "public": [] + }, + "x-step": "connector" + }, + "aws_access_key_id": { + "type": "string", + "title": "Access Key ID", + "description": "AWS access key ID for the bucket", + "x-placeholder": "Enter AWS access key ID", + "x-secret": true, + "x-env-var": "AWS_ACCESS_KEY_ID", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "aws_secret_access_key": { + "type": "string", + "title": "Secret Access Key", + "description": "AWS secret access key for the bucket", + "x-placeholder": "Enter AWS secret access key", + "x-secret": true, + "x-env-var": "AWS_SECRET_ACCESS_KEY", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "region": { + "type": "string", + "title": "Region", + "description": "Rill uses your default AWS region unless you set it explicitly.", + "x-placeholder": "us-east-1", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "endpoint": { + "type": "string", + "title": "Endpoint", + "description": "Override the S3 endpoint (for S3-compatible services like R2/MinIO).", + "x-placeholder": "https://s3.example.com", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "aws_role_arn": { + "type": "string", + "title": "AWS Role ARN", + "description": "AWS Role ARN to assume", + "x-placeholder": "arn:aws:iam::123456789012:role/MyRole", + "x-secret": true, + "x-env-var": "AWS_ROLE_ARN", + "x-step": "connector", + "x-visible-if": { + "auth_method": "access_keys" + } + }, + "path": { + "type": "string", + "title": "S3 URI", + "description": "Path to your S3 bucket or prefix", + "pattern": "^s3://[^/]+(/.*)?$", + "errorMessage": { + "pattern": "Must be an S3 URI (e.g. s3://bucket/path)" + }, + "x-placeholder": "s3://bucket/path", + "x-step": "source" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "source" + } + }, + "required": [ + "path", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "access_keys" + } + } + }, + "then": { + "required": [ + "aws_access_key_id", + "aws_secret_access_key" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: s3\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "type: model\nconnector: duckdb\n[[ if .connector_name -]]\ncreate_secrets_from_connectors: \"[[ .connector_name ]]\"\n[[ end -]]\nsql: |\n [[ duckdbSQL .path false ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/salesforce-duckdb.json b/runtime/templates/definitions/duckdb-models/salesforce-duckdb.json new file mode 100644 index 000000000000..b088e7bee64d --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/salesforce-duckdb.json @@ -0,0 +1,142 @@ +{ + "name": "salesforce-duckdb", + "display_name": "Salesforce", + "description": "Query Salesforce objects and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/salesforce", + "driver": "salesforce", + "olap": "duckdb", + "icon": "Salesforce", + "small_icon": "SalesforceIcon", + "tags": [ + "salesforce", + "crm", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "x-form-height": "tall", + "properties": { + "soql": { + "type": "string", + "title": "SOQL", + "description": "SOQL query to extract data", + "x-placeholder": "SELECT Id, Name FROM Opportunity", + "x-step": "source" + }, + "sobject": { + "type": "string", + "title": "SObject", + "description": "Salesforce object to query", + "x-placeholder": "Opportunity", + "x-step": "source" + }, + "queryAll": { + "type": "boolean", + "title": "Query all", + "description": "Include deleted and archived records", + "default": false, + "x-step": "source" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Salesforce username (usually an email)", + "x-placeholder": "user@example.com" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Salesforce password, optionally followed by security token if required", + "x-placeholder": "your_password_or_password+token", + "x-secret": true, + "x-env-var": "SALESFORCE_PASSWORD" + }, + "key": { + "type": "string", + "title": "JWT private key", + "description": "PEM-formatted private key for JWT auth", + "x-display": "textarea", + "x-placeholder": "your_private_key", + "x-secret": true, + "x-env-var": "SALESFORCE_KEY" + }, + "client_id": { + "type": "string", + "title": "Connected App Client ID", + "description": "Client ID (consumer key) for JWT auth", + "x-placeholder": "Connected App client ID" + }, + "endpoint": { + "type": "string", + "title": "Login endpoint", + "description": "Salesforce login URL (e.g., login.salesforce.com or test.salesforce.com)", + "x-placeholder": "login.salesforce.com" + }, + "name": { + "type": "string", + "title": "Source name", + "description": "Name for the source", + "x-placeholder": "my_new_source", + "x-step": "source" + } + }, + "required": [ + "soql", + "sobject", + "name" + ], + "allOf": [ + { + "if": { + "not": { + "required": [ + "key" + ], + "properties": { + "key": { + "minLength": 1 + } + } + } + }, + "then": { + "required": [ + "username", + "password", + "endpoint" + ] + } + }, + { + "if": { + "required": [ + "key" + ], + "properties": { + "key": { + "minLength": 1 + } + } + }, + "then": { + "required": [ + "client_id", + "username" + ] + } + } + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/salesforce\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ renderProps .source_props -]]\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/snowflake-duckdb.json b/runtime/templates/definitions/duckdb-models/snowflake-duckdb.json new file mode 100644 index 000000000000..c1f970a23d2d --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/snowflake-duckdb.json @@ -0,0 +1,232 @@ +{ + "name": "snowflake-duckdb", + "display_name": "Snowflake", + "description": "Query Snowflake data warehouses and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/snowflake", + "driver": "snowflake", + "olap": "duckdb", + "icon": "Snowflake", + "small_icon": "SnowflakeIcon", + "tags": [ + "snowflake", + "warehouse", + "duckdb", + "source", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "warehouse", + "x-form-height": "tall", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "password", + "private_key", + "dsn" + ], + "default": "password", + "x-display": "tabs", + "x-enum-labels": [ + "User/Password", + "Private Key", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "password": [ + "account", + "user", + "password", + "warehouse", + "database", + "schema", + "role" + ], + "private_key": [ + "account", + "user", + "privateKey", + "warehouse", + "database", + "schema", + "role" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "account": { + "type": "string", + "title": "Account identifier", + "description": "Snowflake account identifier (from your Snowflake URL, before .snowflakecomputing.com)", + "x-placeholder": "abc12345.us-east-1", + "x-hint": "e.g. abc12345 or abc12345.us-east-1 \u2014 don't include https://", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Snowflake username", + "x-placeholder": "your_username", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Snowflake password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PASSWORD", + "x-visible-if": { + "auth_method": "password" + }, + "x-step": "connector" + }, + "privateKey": { + "type": "string", + "title": "Private key", + "description": "Upload your Snowflake private key file (.pem or .p8)", + "format": "file", + "x-display": "file", + "x-file-accept": ".pem,.p8", + "x-file-encoding": "base64", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PRIVATEKEY", + "x-visible-if": { + "auth_method": "private_key" + }, + "x-step": "connector" + }, + "warehouse": { + "type": "string", + "title": "Warehouse", + "description": "Compute warehouse", + "x-placeholder": "your_warehouse", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Snowflake database", + "x-placeholder": "your_database", + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Default schema", + "x-placeholder": "public", + "x-step": "connector" + }, + "role": { + "type": "string", + "title": "Role", + "description": "Snowflake role", + "x-placeholder": "your_role", + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Snowflake DSN, e.g. @//?warehouse=&role=", + "x-placeholder": "@//?warehouse=&role=", + "x-secret": true, + "x-env-var": "SNOWFLAKE_DSN", + "x-hint": "Include authenticator and privateKey query params for JWT if needed.", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + }, + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your warehouse", + "x-placeholder": "Input SQL", + "x-step": "explorer" + }, + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + "required": [ + "sql", + "name" + ], + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "password" + } + } + }, + "then": { + "required": [ + "account", + "user", + "password", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "private_key" + } + } + }, + "then": { + "required": [ + "account", + "user", + "privateKey", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: snowflake\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/snowflake\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/sqlite-duckdb.json b/runtime/templates/definitions/duckdb-models/sqlite-duckdb.json new file mode 100644 index 000000000000..9c3d71458ae6 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/sqlite-duckdb.json @@ -0,0 +1,53 @@ +{ + "name": "sqlite-duckdb", + "display_name": "SQLite", + "description": "Read SQLite tables into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/sqlite", + "driver": "sqlite", + "olap": "duckdb", + "icon": "SQLite", + "small_icon": "SQLiteIcon", + "tags": [ + "sqlite", + "database", + "duckdb", + "source" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sourceOnly", + "properties": { + "db": { + "type": "string", + "title": "Database file", + "description": "Path to SQLite db file", + "x-placeholder": "/path/to/sqlite.db" + }, + "table": { + "type": "string", + "title": "Table", + "description": "SQLite table name", + "x-placeholder": "table" + }, + "name": { + "type": "string", + "title": "Source name", + "description": "Name of the source", + "x-placeholder": "my_new_source" + } + }, + "required": [ + "db", + "table", + "name" + ] + }, + "files": [ + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/sqlite\ntype: model\nconnector: duckdb\nsql: |\n SELECT * FROM sqlite_scan('[[ .db ]]', '[[ .table ]]');\n" + } + ] +} diff --git a/runtime/templates/definitions/duckdb-models/supabase-duckdb.json b/runtime/templates/definitions/duckdb-models/supabase-duckdb.json new file mode 100644 index 000000000000..28e5401ed439 --- /dev/null +++ b/runtime/templates/definitions/duckdb-models/supabase-duckdb.json @@ -0,0 +1,147 @@ +{ + "name": "supabase-duckdb", + "display_name": "Supabase", + "description": "Query Supabase databases and ingest into DuckDB", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/data-source/postgres", + "driver": "supabase", + "olap": "duckdb", + "icon": "Supabase", + "small_icon": "SupabaseIcon", + "tags": [ + "supabase", + "postgres", + "database", + "duckdb", + "source", + "connector" + ], + + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "sqlStore", + + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": ["parameters", "dsn"], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": ["Enter parameters", "Enter connection string"], + "x-ui-only": true, + "x-tab-group": { + "parameters": ["host", "port", "user", "password", "dbname", "sslmode"], + "dsn": ["dsn"] + }, + "x-step": "connector" + }, + + "dsn": { + "type": "string", + "title": "Supabase connection string", + "description": "e.g. postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres", + "x-placeholder": "postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres", + "x-secret": true, + "x-env-var": "SUPABASE_DSN", + "x-hint": "Use a DSN or provide host/user/password/dbname below (but not both).", + "x-step": "connector" + }, + + "host": { + "type": "string", + "title": "Host", + "description": "Supabase database host", + "x-placeholder": "aws-0-[region].pooler.supabase.com", + "x-step": "connector" + }, + + "port": { + "type": "string", + "title": "Port", + "description": "Supabase database port", + "pattern": "^\\d+$", + "errorMessage": { "pattern": "Port must be a number" }, + "default": "5432", + "x-placeholder": "5432", + "x-step": "connector" + }, + + "user": { + "type": "string", + "title": "Username", + "description": "Supabase database user", + "x-placeholder": "postgres.[ref]", + "x-step": "connector" + }, + + "password": { + "type": "string", + "title": "Password", + "description": "Supabase database password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SUPABASE_PASSWORD", + "x-step": "connector" + }, + + "dbname": { + "type": "string", + "title": "Database", + "description": "Database name", + "x-placeholder": "postgres", + "x-step": "connector" + }, + + "sslmode": { + "type": "string", + "title": "SSL mode", + "description": "Use disable, allow, prefer, require", + "enum": ["disable", "allow", "prefer", "require"], + "default": "require", + "x-placeholder": "require", + "x-step": "connector" + }, + + "sql": { + "type": "string", + "title": "SQL", + "description": "SQL query to run against your database", + "x-placeholder": "SELECT * FROM my_table", + "x-step": "explorer" + }, + + "name": { + "type": "string", + "title": "Model name", + "description": "Name for the source model", + "pattern": "^[a-zA-Z0-9_]+$", + "x-placeholder": "my_model", + "x-step": "explorer" + } + }, + + "required": ["sql", "name"], + + "allOf": [ + { + "if": { "properties": { "connection_mode": { "const": "dsn" } } }, + "then": { "required": ["dsn"] }, + "else": { "required": ["host", "user", "dbname"] } + } + ] + }, + + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: supabase\n[[ renderProps .config_props ]]\n" + }, + { + "name": "model", + "path_template": "models/[[ .model_name ]].yaml", + "code_template": "# Model YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/postgres\ntype: model\n[[ if .connector_name -]]\nconnector: \"[[ .connector_name ]]\"\n[[ end -]]\nmaterialize: true\n[[ if .sql -]]\nsql: |\n [[ .sql | indent 2 ]]\n\ndev:\n sql: |\n [[ .sql ]] limit 10000\n[[ end -]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/bigquery.json b/runtime/templates/definitions/olap/bigquery.json new file mode 100644 index 000000000000..6073b8e43088 --- /dev/null +++ b/runtime/templates/definitions/olap/bigquery.json @@ -0,0 +1,56 @@ +{ + "name": "bigquery", + "display_name": "BigQuery", + "description": "Connect to Google BigQuery as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/bigquery", + "driver": "bigquery", + "icon": "GoogleBigQuery", + "small_icon": "GoogleBigQueryIcon", + "tags": [ + "bigquery", + "google", + "warehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "google_application_credentials": { + "type": "string", + "title": "GCP credentials", + "description": "Service account JSON (uploaded or pasted)", + "format": "file", + "x-display": "file", + "x-file-accept": ".json", + "x-file-encoding": "json", + "x-file-extract": { + "project_id": "project_id" + }, + "x-secret": true, + "x-env-var": "GOOGLE_APPLICATION_CREDENTIALS", + "x-step": "connector" + }, + "project_id": { + "type": "string", + "title": "Project ID", + "description": "Google Cloud project ID to use for queries", + "x-placeholder": "my-project", + "x-hint": "If empty, Rill will use the project ID from your credentials when available.", + "x-step": "connector" + } + }, + "required": [ + "google_application_credentials" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: bigquery\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/clickhouse.json b/runtime/templates/definitions/olap/clickhouse.json new file mode 100644 index 000000000000..f0161026bbaa --- /dev/null +++ b/runtime/templates/definitions/olap/clickhouse.json @@ -0,0 +1,385 @@ +{ + "name": "clickhouse", + "display_name": "ClickHouse", + "description": "Connect to a ClickHouse OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/clickhouse", + "driver": "clickhouse", + "icon": "ClickHouse", + "small_icon": "ClickHouseIcon", + "tags": [ + "clickhouse", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-height": "tall", + "x-form-width": "wide", + "x-button-labels": { + "deployment_type": { + "playground": { + "idle": "Connect", + "loading": "Connecting..." + }, + "rill-managed": { + "idle": "Connect", + "loading": "Connecting..." + } + } + }, + "properties": { + "deployment_type": { + "type": "string", + "title": "Connection type", + "enum": [ + "cloud", + "playground", + "self-managed", + "rill-managed" + ], + "default": "cloud", + "x-display": "select", + "x-select-style": "rich", + "x-enum-labels": [ + "ClickHouse Cloud", + "ClickHouse Playground", + "Self Managed", + "Rill-Managed" + ], + "x-enum-descriptions": [ + "Connect to your ClickHouse Cloud instance", + "Free public instance for testing and demos", + "Connect to your own self-hosted server", + "Rill-managed ClickHouse infrastructure" + ], + "x-ui-only": true, + "x-grouped-fields": { + "cloud": [ + "connection_mode" + ], + "playground": [ + "playground_info" + ], + "self-managed": [ + "connection_mode" + ], + "rill-managed": [ + "managed" + ] + }, + "x-step": "connector" + }, + "connection_mode": { + "type": "string", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "username", + "password", + "database", + "cluster", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "DSN connection string (use instead of individual host/port/user settings)", + "x-placeholder": "clickhouse://localhost:9000?username=default&password=password", + "x-secret": true, + "x-env-var": "CLICKHOUSE_DSN", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "managed": { + "type": "boolean", + "title": "Managed", + "description": "This option uses ClickHouse as an OLAP engine with Rill-managed infrastructure. No additional configuration is required.", + "default": false, + "x-informational": true, + "x-omit-if-default": true, + "x-visible-if": { + "deployment_type": "rill-managed" + }, + "x-step": "connector" + }, + "playground_info": { + "type": "boolean", + "title": "Playground", + "description": "Connect to ClickHouse's free public playground instance. This is a read-only demo environment with sample datasets, perfect for testing.", + "default": true, + "x-informational": true, + "x-ui-only": true, + "x-visible-if": { + "deployment_type": "playground" + }, + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Hostname or IP address of the ClickHouse server", + "x-placeholder": "your.clickhouse.server.com", + "x-hint": "Your ClickHouse hostname (e.g., your-server.com)", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Port number of the ClickHouse server (8443 HTTPS, 9440 Native TLS, 8123 HTTP, 9000 Native TCP)", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "8443", + "x-placeholder": "8443", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Username to connect to the ClickHouse server", + "default": "default", + "x-placeholder": "default", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Password to connect to the ClickHouse server", + "x-placeholder": "Database password", + "x-secret": true, + "x-env-var": "CLICKHOUSE_PASSWORD", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Name of the ClickHouse database to connect to", + "default": "default", + "x-placeholder": "default", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "cluster": { + "type": "string", + "title": "Cluster", + "description": "Cluster name. If set, models are created as distributed tables.", + "x-placeholder": "Cluster name", + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Use SSL to connect to the ClickHouse server", + "default": true, + "x-visible-if": { + "deployment_type": [ + "cloud", + "self-managed" + ] + }, + "x-step": "connector" + } + }, + "required": [ + "deployment_type" + ], + "allOf": [ + { + "if": { + "properties": { + "deployment_type": { + "const": "rill-managed" + } + } + }, + "then": { + "required": [ + "managed" + ], + "properties": { + "managed": { + "const": true + } + } + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "playground" + } + } + }, + "then": { + "properties": { + "host": { + "const": "play.clickhouse.com" + }, + "port": { + "const": "9440" + }, + "username": { + "const": "play" + }, + "password": { + "const": "" + }, + "database": { + "const": "default" + }, + "ssl": { + "const": true + } + } + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "cloud" + }, + "connection_mode": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "username", + "port" + ] + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "cloud" + }, + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "self-managed" + }, + "connection_mode": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "username" + ] + } + }, + { + "if": { + "properties": { + "deployment_type": { + "const": "self-managed" + }, + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/clickhouse\ntype: connector\ndriver: clickhouse\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/databricks.json b/runtime/templates/definitions/olap/databricks.json new file mode 100644 index 000000000000..182bf96b3f11 --- /dev/null +++ b/runtime/templates/definitions/olap/databricks.json @@ -0,0 +1,154 @@ +{ + "name": "databricks", + "display_name": "Databricks", + "description": "Connect to a Databricks SQL warehouse as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/databricks", + "driver": "databricks", + "icon": "Databricks", + "small_icon": "DatabricksIcon", + "tags": [ + "databricks", + "warehouse", + "lakehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Workspace details", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "http_path", + "token", + "catalog", + "schema" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Databricks SQL warehouse hostname", + "x-placeholder": "dbc-xxxxxxxx-xxxx.cloud.databricks.com", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "http_path": { + "type": "string", + "title": "HTTP path", + "description": "HTTP path for the SQL warehouse", + "x-placeholder": "/sql/1.0/warehouses/xxxxxxxxxxxxxxxx", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "token": { + "type": "string", + "title": "Access token", + "description": "Databricks personal access token", + "x-placeholder": "dapi...", + "x-secret": true, + "x-env-var": "DATABRICKS_TOKEN", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "catalog": { + "type": "string", + "title": "Catalog", + "description": "Unity Catalog name (optional; defaults to the workspace default)", + "x-placeholder": "main", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Schema within the catalog (optional; defaults to the workspace default)", + "x-placeholder": "default", + "x-visible-if": { + "auth_method": "parameters" + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Databricks DSN.", + "x-placeholder": "token:@:443/?catalog=&schema=", + "x-secret": true, + "x-env-var": "DATABRICKS_DSN", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + } + }, + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "parameters" + } + } + }, + "then": { + "required": [ + "host", + "http_path", + "token" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: databricks\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/druid.json b/runtime/templates/definitions/olap/druid.json new file mode 100644 index 000000000000..9d46c2d7023b --- /dev/null +++ b/runtime/templates/definitions/olap/druid.json @@ -0,0 +1,123 @@ +{ + "name": "druid", + "display_name": "Druid", + "description": "Connect to an Apache Druid OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/druid", + "driver": "druid", + "icon": "ApacheDruid", + "small_icon": "ApacheDruidIcon", + "tags": [ + "druid", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "username", + "password", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Druid SQL/Avatica endpoint", + "x-placeholder": "https://example.com/druid/v2/sql/avatica-protobuf?authentication=BASIC&avaticaUser=user&avaticaPassword=pass", + "x-secret": true, + "x-env-var": "DRUID_DSN", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Druid host or IP", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "Druid port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "x-placeholder": "8888", + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Druid username", + "x-placeholder": "default", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Druid password", + "x-placeholder": "password", + "x-secret": true, + "x-env-var": "DRUID_PASSWORD", + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Use SSL for the connection", + "default": true, + "x-step": "connector" + } + }, + "required": [], + "oneOf": [ + { + "title": "Use connection string", + "required": [ + "dsn" + ] + }, + { + "title": "Use individual parameters", + "required": [ + "host", + "ssl" + ] + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/druid\ntype: connector\ndriver: druid\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/duckdb.json b/runtime/templates/definitions/olap/duckdb.json new file mode 100644 index 000000000000..93d09dae3b48 --- /dev/null +++ b/runtime/templates/definitions/olap/duckdb.json @@ -0,0 +1,125 @@ +{ + "name": "duckdb", + "display_name": "DuckDB", + "description": "Connect to a DuckDB OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/duckdb", + "driver": "duckdb", + "icon": "DuckDB", + "small_icon": "DuckDBIcon", + "tags": [ + "duckdb", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-button-labels": { + "connector_type": { + "rill-managed": { + "idle": "Connect", + "loading": "Connecting..." + } + } + }, + "properties": { + "connector_type": { + "type": "string", + "title": "Connection type", + "enum": [ + "rill-managed", + "self-hosted" + ], + "default": "rill-managed", + "x-display": "select", + "x-select-style": "rich", + "x-enum-labels": [ + "Rill Managed", + "Local File" + ], + "x-enum-descriptions": [ + "Rill manages your DuckDB infrastructure", + "Connect to your own DuckDB database file" + ], + "x-ui-only": true, + "x-grouped-fields": { + "rill-managed": [ + "managed" + ], + "self-hosted": [ + "path" + ] + }, + "x-step": "connector" + }, + "managed": { + "type": "boolean", + "title": "Managed", + "description": "This option uses DuckDB as an OLAP engine with Rill-managed infrastructure. No additional configuration is required.", + "default": true, + "x-informational": true, + "x-ui-only": true, + "x-visible-if": { + "connector_type": "rill-managed" + }, + "x-step": "connector" + }, + "path": { + "type": "string", + "title": "Path", + "description": "Path to external DuckDB database", + "x-placeholder": "/path/to/main.db", + "x-visible-if": { + "connector_type": "self-hosted" + }, + "x-step": "connector" + } + }, + "required": [ + "connector_type" + ], + "allOf": [ + { + "if": { + "properties": { + "connector_type": { + "const": "rill-managed" + } + } + }, + "then": { + "required": [ + "managed" + ], + "properties": { + "managed": { + "const": true + } + } + } + }, + { + "if": { + "properties": { + "connector_type": { + "const": "self-hosted" + } + } + }, + "then": { + "required": [ + "path" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/duckdb\ntype: connector\ndriver: duckdb\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/ducklake.json b/runtime/templates/definitions/olap/ducklake.json new file mode 100644 index 000000000000..1df03d09d36c --- /dev/null +++ b/runtime/templates/definitions/olap/ducklake.json @@ -0,0 +1,60 @@ +{ + "name": "ducklake", + "display_name": "DuckLake", + "description": "Connect to a DuckLake catalog as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/ducklake", + "driver": "duckdb", + "icon": "DuckLake", + "small_icon": "DuckLakeIcon", + "tags": [ + "ducklake", + "duckdb", + "lakehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-width": "wide", + "properties": { + "attach": { + "type": "string", + "title": "Attach clause", + "description": "DuckDB ATTACH clause that points at your DuckLake catalog. Include the metadata backend and DATA_PATH.", + "x-placeholder": "'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'data/')", + "x-monospace": true, + "x-hint": "Supported metadata backends: DuckDB file, SQLite, Postgres, MySQL. Data path can be local or object storage (s3://, gs://, azure://).", + "x-step": "connector" + }, + "mode": { + "type": "string", + "title": "Mode", + "description": "Set to 'readwrite' to allow Rill to create and modify tables. Defaults to 'read' (read-only).", + "enum": [ + "read", + "readwrite" + ], + "default": "read", + "x-display": "select", + "x-enum-labels": [ + "Read only", + "Read / write" + ], + "x-omit-if-default": true, + "x-step": "connector" + } + }, + "required": [ + "attach" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: duckdb\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/motherduck.json b/runtime/templates/definitions/olap/motherduck.json new file mode 100644 index 000000000000..f4466f131921 --- /dev/null +++ b/runtime/templates/definitions/olap/motherduck.json @@ -0,0 +1,57 @@ +{ + "name": "motherduck", + "display_name": "MotherDuck", + "description": "Connect to a MotherDuck OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/motherduck", + "driver": "motherduck", + "icon": "MotherDuck", + "small_icon": "MotherDuckIcon", + "tags": [ + "motherduck", + "duckdb", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "path": { + "type": "string", + "title": "Path", + "description": "MotherDuck database path (prefix with md:)", + "x-placeholder": "md:my_db", + "x-step": "connector" + }, + "token": { + "type": "string", + "title": "Token", + "description": "MotherDuck token", + "x-placeholder": "your_motherduck_token", + "x-secret": true, + "x-env-var": "MOTHERDUCK_TOKEN", + "x-step": "connector" + }, + "schema_name": { + "type": "string", + "title": "Schema name", + "description": "Default schema to use", + "x-placeholder": "main", + "x-step": "connector" + } + }, + "required": [ + "path", + "token", + "schema_name" + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/motherduck\ntype: connector\ndriver: motherduck\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/pinot.json b/runtime/templates/definitions/olap/pinot.json new file mode 100644 index 000000000000..d5560ba1f49a --- /dev/null +++ b/runtime/templates/definitions/olap/pinot.json @@ -0,0 +1,144 @@ +{ + "name": "pinot", + "display_name": "Pinot", + "description": "Connect to an Apache Pinot OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/pinot", + "driver": "pinot", + "icon": "ApachePinot", + "small_icon": "ApachePinotIcon", + "tags": [ + "pinot", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "broker_host", + "broker_port", + "controller_host", + "controller_port", + "username", + "password", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Pinot connection string", + "x-placeholder": "https://username:password@localhost:8000?controller=localhost:9000", + "x-secret": true, + "x-env-var": "PINOT_DSN", + "x-step": "connector" + }, + "broker_host": { + "type": "string", + "title": "Broker host", + "description": "Pinot broker host", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "broker_port": { + "type": "string", + "title": "Broker port", + "description": "Pinot broker port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "x-placeholder": "8000", + "x-step": "connector" + }, + "controller_host": { + "type": "string", + "title": "Controller host", + "description": "Pinot controller host", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "controller_port": { + "type": "string", + "title": "Controller port", + "description": "Pinot controller port", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "x-placeholder": "9000", + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Pinot username", + "x-placeholder": "default", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Pinot password", + "x-placeholder": "password", + "x-secret": true, + "x-env-var": "PINOT_PASSWORD", + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Use SSL", + "default": true, + "x-step": "connector" + } + }, + "required": [], + "oneOf": [ + { + "title": "Use connection string", + "required": [ + "dsn" + ] + }, + { + "title": "Use individual parameters", + "required": [ + "broker_host", + "controller_host", + "ssl" + ] + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/pinot\ntype: connector\ndriver: pinot\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/snowflake.json b/runtime/templates/definitions/olap/snowflake.json new file mode 100644 index 000000000000..84daba37fffc --- /dev/null +++ b/runtime/templates/definitions/olap/snowflake.json @@ -0,0 +1,206 @@ +{ + "name": "snowflake", + "display_name": "Snowflake", + "description": "Connect to a Snowflake data warehouse as your OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/snowflake", + "driver": "snowflake", + "icon": "Snowflake", + "small_icon": "SnowflakeIcon", + "tags": [ + "snowflake", + "warehouse", + "olap", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-height": "tall", + "properties": { + "auth_method": { + "type": "string", + "title": "Authentication method", + "enum": [ + "password", + "private_key", + "dsn" + ], + "default": "password", + "x-display": "tabs", + "x-enum-labels": [ + "User/Password", + "Private Key", + "Connection String" + ], + "x-ui-only": true, + "x-tab-group": { + "password": [ + "account", + "user", + "password", + "warehouse", + "database", + "schema", + "role" + ], + "private_key": [ + "account", + "user", + "privateKey", + "warehouse", + "database", + "schema", + "role" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "account": { + "type": "string", + "title": "Account identifier", + "description": "Snowflake account identifier (from your Snowflake URL, before .snowflakecomputing.com)", + "x-placeholder": "abc12345.us-east-1", + "x-hint": "e.g. abc12345 or abc12345.us-east-1 — don't include https://", + "x-step": "connector" + }, + "user": { + "type": "string", + "title": "Username", + "description": "Snowflake username", + "x-placeholder": "your_username", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Snowflake password", + "x-placeholder": "your_password", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PASSWORD", + "x-visible-if": { + "auth_method": "password" + }, + "x-step": "connector" + }, + "privateKey": { + "type": "string", + "title": "Private key", + "description": "Upload your Snowflake private key file (.pem or .p8)", + "format": "file", + "x-display": "file", + "x-file-accept": ".pem,.p8", + "x-file-encoding": "base64", + "x-secret": true, + "x-env-var": "SNOWFLAKE_PRIVATEKEY", + "x-visible-if": { + "auth_method": "private_key" + }, + "x-step": "connector" + }, + "warehouse": { + "type": "string", + "title": "Warehouse", + "description": "Compute warehouse", + "x-placeholder": "your_warehouse", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Snowflake database", + "x-placeholder": "your_database", + "x-step": "connector" + }, + "schema": { + "type": "string", + "title": "Schema", + "description": "Default schema", + "x-placeholder": "public", + "x-step": "connector" + }, + "role": { + "type": "string", + "title": "Role", + "description": "Snowflake role", + "x-placeholder": "your_role", + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "Full Snowflake DSN.", + "x-placeholder": "@//?warehouse=&role=", + "x-secret": true, + "x-env-var": "SNOWFLAKE_DSN", + "x-hint": "Include authenticator and privateKey query params for JWT if needed.", + "x-visible-if": { + "auth_method": "dsn" + }, + "x-step": "connector" + } + }, + "allOf": [ + { + "if": { + "properties": { + "auth_method": { + "const": "password" + } + } + }, + "then": { + "required": [ + "account", + "user", + "password", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "private_key" + } + } + }, + "then": { + "required": [ + "account", + "user", + "privateKey", + "database", + "warehouse" + ] + } + }, + { + "if": { + "properties": { + "auth_method": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: [[ .docs_url ]]\ntype: connector\ndriver: snowflake\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/definitions/olap/starrocks.json b/runtime/templates/definitions/olap/starrocks.json new file mode 100644 index 000000000000..2442024771da --- /dev/null +++ b/runtime/templates/definitions/olap/starrocks.json @@ -0,0 +1,148 @@ +{ + "name": "starrocks", + "display_name": "StarRocks", + "description": "Connect to a StarRocks OLAP engine", + "docs_url": "https://docs.rilldata.com/developers/build/connectors/olap/starrocks", + "driver": "starrocks", + "icon": "StarRocks", + "small_icon": "StarRocksIcon", + "tags": [ + "starrocks", + "olap", + "analytics", + "connector" + ], + "json_schema": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "x-category": "olap", + "x-form-height": "tall", + "properties": { + "connection_mode": { + "type": "string", + "title": "Connection method", + "enum": [ + "parameters", + "dsn" + ], + "default": "parameters", + "x-display": "tabs", + "x-enum-labels": [ + "Enter parameters", + "Enter connection string" + ], + "x-ui-only": true, + "x-tab-group": { + "parameters": [ + "host", + "port", + "username", + "password", + "catalog", + "database", + "ssl" + ], + "dsn": [ + "dsn" + ] + }, + "x-step": "connector" + }, + "dsn": { + "type": "string", + "title": "Connection string", + "description": "MySQL DSN format. If provided, do not set host/port/username/password.", + "x-placeholder": "user:password@tcp(host:9030)/?timeout=30s&readTimeout=300s&parseTime=true", + "x-secret": true, + "x-env-var": "STARROCKS_DSN", + "x-step": "connector" + }, + "host": { + "type": "string", + "title": "Host", + "description": "Hostname or IP address of the StarRocks FE node", + "x-placeholder": "localhost", + "x-step": "connector" + }, + "port": { + "type": "string", + "title": "Port", + "description": "MySQL protocol port of the StarRocks FE node", + "pattern": "^\\d+$", + "errorMessage": { + "pattern": "Port must be a number" + }, + "default": "9030", + "x-placeholder": "9030", + "x-step": "connector" + }, + "username": { + "type": "string", + "title": "Username", + "description": "Username to connect to StarRocks", + "default": "root", + "x-placeholder": "root", + "x-step": "connector" + }, + "password": { + "type": "string", + "title": "Password", + "description": "Password to connect to StarRocks", + "x-placeholder": "password", + "x-secret": true, + "x-env-var": "STARROCKS_PASSWORD", + "x-step": "connector" + }, + "catalog": { + "type": "string", + "title": "Catalog", + "description": "StarRocks catalog name. Use default_catalog for internal tables, or specify an external catalog.", + "default": "default_catalog", + "x-placeholder": "default_catalog", + "x-step": "connector" + }, + "database": { + "type": "string", + "title": "Database", + "description": "Name of the StarRocks database to connect to", + "x-placeholder": "default", + "x-step": "connector" + }, + "ssl": { + "type": "boolean", + "title": "SSL", + "description": "Enable SSL/TLS encryption for the connection", + "x-step": "connector" + } + }, + "required": [], + "allOf": [ + { + "if": { + "properties": { + "connection_mode": { + "const": "dsn" + } + } + }, + "then": { + "required": [ + "dsn" + ] + }, + "else": { + "required": [ + "host" + ] + } + } + ] + }, + "files": [ + { + "name": "connector", + "path_template": "connectors/[[ .connector_name ]].yaml", + "code_template": "# Connector YAML\n# Reference documentation: https://docs.rilldata.com/developers/build/connectors/olap/starrocks\ntype: connector\ndriver: starrocks\n[[ renderProps .props ]]\n" + } + ] +} diff --git a/runtime/templates/duckdb_test.go b/runtime/templates/duckdb_test.go new file mode 100644 index 000000000000..99ec50137cb1 --- /dev/null +++ b/runtime/templates/duckdb_test.go @@ -0,0 +1,81 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestDuckdbSQL(t *testing.T) { + tests := []struct { + name string + path string + defaultToJSON bool + wantContains string + }{ + {"parquet", "s3://bucket/data.parquet", false, "read_parquet"}, + {"csv", "s3://bucket/data.csv", false, "read_csv"}, + {"tsv", "s3://bucket/data.tsv", false, "read_csv"}, + {"json", "s3://bucket/data.json", false, "read_json"}, + {"ndjson", "s3://bucket/data.ndjson", false, "read_json"}, + {"compound parquet.gz", "s3://bucket/data.v1.parquet.gz", false, "read_parquet"}, + {"unknown default generic", "s3://bucket/data.xyz", false, "select * from 's3://bucket/data.xyz'"}, + {"unknown default json", "https://example.com/api", true, "read_json"}, + {"txt", "s3://bucket/data.txt", false, "read_csv"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + query := duckdbSQL(tt.path, tt.defaultToJSON) + require.Contains(t, query, tt.wantContains) + require.Contains(t, query, tt.path) + }) + } +} + +func TestMatchesExtFixedFalsePositive(t *testing.T) { + // This was a bug: "parquet-archive/readme.txt" should NOT match ".parquet" + require.False(t, matchesExt("s3://bucket/parquet-archive/readme.txt", ".parquet"), + "path with 'parquet' in directory name should not match .parquet extension") + + // But actual parquet files should match + require.True(t, matchesExt("s3://bucket/data.parquet", ".parquet")) + require.True(t, matchesExt("s3://bucket/data.v1.parquet.gz", ".parquet")) +} + +func TestMatchesExtCompound(t *testing.T) { + require.True(t, matchesExt("data.csv.gz", ".csv")) + require.True(t, matchesExt("data.ndjson.gz", ".ndjson")) + require.False(t, matchesExt("data.xyz", ".csv", ".json")) +} + +func TestClickhouseURLSuffix(t *testing.T) { + props := []ProcessedProp{ + {Key: "headers", Value: "\n Authorization: \"Bearer {{ .env.connector.https.authorization }}\"\n X-API-Key: \"{{ .env.connector.https.x_api_key }}\"", Quoted: false}, + } + + // CSV URL with headers: includes format and headers() + result := clickhouseURLSuffix("https://example.com/data.csv", props) + require.Contains(t, result, "CSVWithNames") + require.Contains(t, result, "headers(") + require.Contains(t, result, "'Authorization'='Bearer {{ .env.connector.https.authorization }}'") + require.Contains(t, result, "'X-API-Key'='{{ .env.connector.https.x_api_key }}'") + + // Parquet URL with headers + result = clickhouseURLSuffix("https://example.com/data.parquet", props) + require.Contains(t, result, "Parquet") + require.Contains(t, result, "headers(") + + // JSON URL with headers + result = clickhouseURLSuffix("https://example.com/data.ndjson", props) + require.Contains(t, result, "JSONEachRow") + + // No headers: returns empty (ClickHouse auto-detects) + noHeaders := []ProcessedProp{ + {Key: "path", Value: "https://example.com", Quoted: true}, + } + require.Equal(t, "", clickhouseURLSuffix("https://example.com/data.csv", noHeaders)) + + // Nil/wrong type: returns empty + require.Equal(t, "", clickhouseURLSuffix("https://example.com/data.csv", nil)) +} diff --git a/runtime/templates/env.go b/runtime/templates/env.go new file mode 100644 index 000000000000..8fc8edee09da --- /dev/null +++ b/runtime/templates/env.go @@ -0,0 +1,48 @@ +package templates + +import ( + "bufio" + "context" + "fmt" + "strings" + + "github.com/rilldata/rill/runtime/drivers" +) + +// ResolveEnvVarNameForKey determines the env var name for a secret property key, +// using an explicit env var name when provided, or falling back to DRIVER_KEY format. +// This variant does not require a *drivers.PropertySpec; used by schema-based templates. +func ResolveEnvVarNameForKey(driverName, key, explicitEnvVarName string, existingEnv map[string]bool) string { + var base string + if explicitEnvVarName != "" { + base = explicitEnvVarName + } else { + base = strings.ToUpper(driverName) + "_" + strings.ToUpper(key) + } + + candidate := base + for i := 1; existingEnv[candidate]; i++ { + candidate = fmt.Sprintf("%s_%d", base, i) + } + return candidate +} + +// ReadEnvKeys parses a .env file into a set of key names. +func ReadEnvKeys(ctx context.Context, repo drivers.RepoStore) map[string]bool { + keys := make(map[string]bool) + content, err := repo.Get(ctx, ".env") + if err != nil { + return keys + } + scanner := bufio.NewScanner(strings.NewReader(content)) + for scanner.Scan() { + line := strings.TrimSpace(scanner.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + if idx := strings.IndexByte(line, '='); idx > 0 { + keys[line[:idx]] = true + } + } + return keys +} diff --git a/runtime/templates/env_test.go b/runtime/templates/env_test.go new file mode 100644 index 000000000000..6bcbb160c6a5 --- /dev/null +++ b/runtime/templates/env_test.go @@ -0,0 +1,40 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestResolveEnvVarNameWithExplicitName(t *testing.T) { + existing := make(map[string]bool) + + name := ResolveEnvVarNameForKey("s3", "aws_access_key_id", "AWS_ACCESS_KEY_ID", existing) + require.Equal(t, "AWS_ACCESS_KEY_ID", name) +} + +func TestResolveEnvVarNameFallback(t *testing.T) { + existing := make(map[string]bool) + + name := ResolveEnvVarNameForKey("starrocks", "password", "", existing) + require.Equal(t, "STARROCKS_PASSWORD", name) +} + +func TestResolveEnvVarNameConflict(t *testing.T) { + existing := map[string]bool{ + "AWS_ACCESS_KEY_ID": true, + } + + name := ResolveEnvVarNameForKey("s3", "aws_access_key_id", "AWS_ACCESS_KEY_ID", existing) + require.Equal(t, "AWS_ACCESS_KEY_ID_1", name) +} + +func TestResolveEnvVarNameDoubleConflict(t *testing.T) { + existing := map[string]bool{ + "AWS_ACCESS_KEY_ID": true, + "AWS_ACCESS_KEY_ID_1": true, + } + + name := ResolveEnvVarNameForKey("s3", "aws_access_key_id", "AWS_ACCESS_KEY_ID", existing) + require.Equal(t, "AWS_ACCESS_KEY_ID_2", name) +} diff --git a/runtime/templates/funcmap.go b/runtime/templates/funcmap.go new file mode 100644 index 000000000000..6b54199bc9c2 --- /dev/null +++ b/runtime/templates/funcmap.go @@ -0,0 +1,288 @@ +package templates + +import ( + "fmt" + "path/filepath" + "strings" + "text/template" +) + +// sharedFuncMap is the template function map available to all template definitions. +// Allocated once since all entries are stateless functions. +var sharedFuncMap = template.FuncMap{ + "renderProps": renderProps, + "indent": indent, + "quote": quote, + "propVal": propVal, + "default": defaultVal, + "duckdbSQL": duckdbSQL, + "s3ToHTTPS": s3ToHTTPS, + "gcsToHTTPS": gcsToHTTPS, + "azureContainer": azureContainer, + "azureBlobPath": azureBlobPath, + "azureEndpoint": azureEndpoint, + "clickhouseFormat": clickhouseFormat, + "clickhouseURLSuffix": clickhouseURLSuffix, +} + +// renderProps renders a slice of ProcessedProp as YAML key-value lines. +// Each property is rendered on its own line with appropriate formatting: +// quoted values get double quotes, unquoted values are rendered as-is. +func renderProps(props []ProcessedProp) string { + if len(props) == 0 { + return "" + } + var b strings.Builder + for i, p := range props { + if i > 0 { + b.WriteByte('\n') + } + if p.Quoted { + fmt.Fprintf(&b, "%s: %q", p.Key, p.Value) + } else { + fmt.Fprintf(&b, "%s: %s", p.Key, p.Value) + } + } + return b.String() +} + +// indent prepends each line of text with n spaces. +func indent(n int, text string) string { + pad := strings.Repeat(" ", n) + lines := strings.Split(text, "\n") + for i, line := range lines { + if line != "" { + lines[i] = pad + line + } + } + return strings.Join(lines, "\n") +} + +// quote wraps a string in double quotes. +func quote(s string) string { + return fmt.Sprintf("%q", s) +} + +// propVal extracts a value from a []ProcessedProp by key. +// Returns "" if the key is not found or props is not the expected type. +func propVal(props any, key string) string { + ps, ok := props.([]ProcessedProp) + if !ok { + return "" + } + for _, p := range ps { + if p.Key == key { + return p.Value + } + } + return "" +} + +// defaultVal returns val if non-empty, otherwise fallback. +// Registered as "default" in the template function map. +// NOTE: Use positional syntax only: [[ default (expr) "fallback" ]]. +// Pipeline syntax ([[ expr | default "fallback" ]]) would swap arguments +// because text/template pipes into the last parameter. +func defaultVal(val, fallback string) string { + if val == "" { + return fallback + } + return val +} + +// duckdbSQL maps a file path to a DuckDB read function call based on file extension. +// When defaultToJSON is true, unknown extensions default to read_json; otherwise select * from 'path'. +func duckdbSQL(path string, defaultToJSON bool) string { + switch { + case matchesExt(path, ".csv", ".tsv", ".txt"): + return fmt.Sprintf("select * from read_csv('%s', auto_detect=true, ignore_errors=1, header=true)", path) + case matchesExt(path, ".parquet"): + return fmt.Sprintf("select * from read_parquet('%s')", path) + case matchesExt(path, ".json", ".ndjson"): + return fmt.Sprintf("select * from read_json('%s', auto_detect=true, format='auto')", path) + default: + if defaultToJSON { + return fmt.Sprintf("select * from read_json('%s', auto_detect=true, format='auto')", path) + } + return fmt.Sprintf("select * from '%s'", path) + } +} + +// matchesExt checks if the file path has any of the target extensions. +// Handles compound extensions like .v1.parquet.gz by checking the basename +// for extension substrings, using suffix matching on path segments to avoid +// false positives (e.g. "parquet-archive/readme.txt" should NOT match ".parquet"). +func matchesExt(path string, targets ...string) bool { + ext := strings.ToLower(filepath.Ext(path)) + base := strings.ToLower(filepath.Base(path)) + for _, t := range targets { + if ext == t { + return true + } + // Check for compound extensions in the basename only (not the full path). + // Look for the target extension followed by a dot or end-of-string. + idx := strings.Index(base, t) + if idx > 0 { + after := idx + len(t) + if after == len(base) || base[after] == '.' { + return true + } + } + } + return false +} + +// s3ToHTTPS converts an s3:// URI to an HTTPS URL for ClickHouse's s3() function. +// "s3://bucket/key" becomes "https://bucket.s3.amazonaws.com/key". +// If the path is already HTTPS, it is returned as-is. +func s3ToHTTPS(path string) string { + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + return path + } + trimmed := strings.TrimPrefix(path, "s3://") + idx := strings.IndexByte(trimmed, '/') + if idx < 0 { + return fmt.Sprintf("https://%s.s3.amazonaws.com", trimmed) + } + bucket := trimmed[:idx] + key := trimmed[idx:] + return fmt.Sprintf("https://%s.s3.amazonaws.com%s", bucket, key) +} + +// gcsToHTTPS converts a gs:// URI to an HTTPS URL for ClickHouse's gcs() function. +// "gs://bucket/key" becomes "https://storage.googleapis.com/bucket/key". +// If the path is already HTTPS, it is returned as-is. +func gcsToHTTPS(path string) string { + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + return path + } + trimmed := strings.TrimPrefix(path, "gs://") + return fmt.Sprintf("https://storage.googleapis.com/%s", trimmed) +} + +// clickhouseURLSuffix generates additional arguments for ClickHouse's url() table function. +// When headers are present, returns ", Format, headers('K1'='V1', 'K2'='V2')" where +// Format is auto-detected from the URL extension. Returns empty string when no headers +// are present (ClickHouse auto-detects everything from the URL). +func clickhouseURLSuffix(path string, props any) string { + hdrs := extractClickhouseHeaders(props) + if hdrs == "" { + return "" + } + format := clickhouseFormat(path) + return fmt.Sprintf(",\n %s,\n %s\n ", format, hdrs) +} + +// clickhouseFormat maps a URL path to a ClickHouse input format name. +func clickhouseFormat(path string) string { + switch { + case matchesExt(path, ".csv", ".txt"): + return "CSVWithNames" + case matchesExt(path, ".tsv"): + return "TabSeparatedWithNames" + case matchesExt(path, ".json", ".ndjson", ".jsonl"): + return "JSONEachRow" + case matchesExt(path, ".parquet"): + return "Parquet" + default: + return "JSONEachRow" + } +} + +// extractClickhouseHeaders extracts header ProcessedProps and formats them +// for ClickHouse's headers() syntax: headers('Key1'='value1', 'Key2'='value2'). +// Returns empty string if no headers are present. +func extractClickhouseHeaders(props any) string { + ps, ok := props.([]ProcessedProp) + if !ok { + return "" + } + var headerProp *ProcessedProp + for i := range ps { + if ps[i].Key == "headers" { + headerProp = &ps[i] + break + } + } + if headerProp == nil { + return "" + } + + // Parse the YAML-style header lines (e.g. "Authorization: \"Bearer {{ .env.X }}\"") + var pairs []string + for _, line := range strings.Split(headerProp.Value, "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + idx := strings.IndexByte(line, ':') + if idx < 0 { + continue + } + key := strings.TrimSpace(line[:idx]) + val := strings.TrimSpace(line[idx+1:]) + val = strings.Trim(val, "\"") + pairs = append(pairs, fmt.Sprintf("'%s'='%s'", key, val)) + } + if len(pairs) == 0 { + return "" + } + return fmt.Sprintf("headers(%s)", strings.Join(pairs, ", ")) +} + +// azureContainer extracts the container name from an Azure URI. +// Supports both "azure://container/blob/path" and "https://account.blob.core.windows.net/container/blob/path". +func azureContainer(path string) string { + path = stripAzurePrefix(path) + idx := strings.IndexByte(path, '/') + if idx < 0 { + return path + } + return path[:idx] +} + +// azureBlobPath extracts the blob path from an Azure URI. +// Supports both "azure://container/blob/path" and "https://account.blob.core.windows.net/container/blob/path". +func azureBlobPath(path string) string { + path = stripAzurePrefix(path) + idx := strings.IndexByte(path, '/') + if idx < 0 { + return "" + } + return path[idx+1:] +} + +// azureEndpoint returns the blob service endpoint for a given Azure URI. +// For "https://account.blob.core.windows.net/..." it returns "https://account.blob.core.windows.net". +// For "azure://..." it builds the endpoint from the account property. +func azureEndpoint(path, account string) string { + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + u := strings.TrimPrefix(path, "https://") + u = strings.TrimPrefix(u, "http://") + idx := strings.IndexByte(u, '/') + if idx > 0 { + return "https://" + u[:idx] + } + return "https://" + u + } + return fmt.Sprintf("https://%s.blob.core.windows.net", account) +} + +// stripAzurePrefix strips the scheme and host from an Azure URI, returning "container/blob/path". +// Handles "azure://container/path" and "https://account.blob.core.windows.net/container/path". +func stripAzurePrefix(path string) string { + if strings.HasPrefix(path, "azure://") { + return strings.TrimPrefix(path, "azure://") + } + // HTTPS: strip scheme + host, leaving "/container/path" + if strings.HasPrefix(path, "https://") || strings.HasPrefix(path, "http://") { + noScheme := strings.TrimPrefix(path, "https://") + noScheme = strings.TrimPrefix(noScheme, "http://") + idx := strings.IndexByte(noScheme, '/') + if idx >= 0 { + return noScheme[idx+1:] + } + return "" + } + return path +} diff --git a/runtime/templates/headers.go b/runtime/templates/headers.go new file mode 100644 index 000000000000..3764b13fa7af --- /dev/null +++ b/runtime/templates/headers.go @@ -0,0 +1,52 @@ +package templates + +import ( + "fmt" + "regexp" + "strings" +) + +// sensitiveHeaderPattern matches header keys that carry secret values. +var sensitiveHeaderPattern = regexp.MustCompile( + `(?i)^(authorization|x-api-key|api-key|token|x-token|x-auth|x-secret|proxy-authorization)$`, +) + +// headerKeyCleanupPattern sanitizes a header key into a valid .env variable segment. +// Compiled at package level to avoid per-call regex compilation. +var headerKeyCleanupPattern = regexp.MustCompile(`[^a-z0-9]+`) + +// IsSensitiveHeaderKey returns true when a header key likely carries a secret value. +func IsSensitiveHeaderKey(key string) bool { + return sensitiveHeaderPattern.MatchString(strings.TrimSpace(key)) +} + +// AuthSchemePrefixes are common HTTP authentication scheme prefixes. +// When a sensitive header value starts with one of these (case-insensitive), +// only the token portion is stored in .env. +var AuthSchemePrefixes = []string{"Bearer ", "Basic ", "Token ", "Bot "} + +// SplitAuthSchemePrefix splits a value into scheme prefix and secret if it starts +// with a known auth scheme. Returns ("", "", false) when no prefix matches. +func SplitAuthSchemePrefix(value string) (scheme, secret string, ok bool) { + for _, prefix := range AuthSchemePrefixes { + if len(value) > len(prefix) && strings.EqualFold(value[:len(prefix)], prefix) { + return value[:len(prefix)], value[len(prefix):], true + } + } + return "", "", false +} + +// HeaderKeyToEnvSegment sanitizes a header key into a valid .env variable segment. +func HeaderKeyToEnvSegment(key string) string { + return headerKeyCleanupPattern.ReplaceAllString(strings.ToLower(key), "_") +} + +// ResolveHeaderEnvVarName determines the env var name for a header, resolving conflicts. +func ResolveHeaderEnvVarName(connectorName, segment string, existingEnv map[string]bool) string { + base := fmt.Sprintf("connector.%s.%s", connectorName, segment) + candidate := base + for i := 1; existingEnv[candidate]; i++ { + candidate = fmt.Sprintf("%s_%d", base, i) + } + return candidate +} diff --git a/runtime/templates/headers_test.go b/runtime/templates/headers_test.go new file mode 100644 index 000000000000..be0dc08d31b4 --- /dev/null +++ b/runtime/templates/headers_test.go @@ -0,0 +1,56 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestIsSensitiveHeaderKey(t *testing.T) { + sensitive := []string{"Authorization", "X-API-Key", "API-Key", "Token", "X-Token", "X-Auth", "X-Secret", "Proxy-Authorization"} + for _, key := range sensitive { + require.True(t, IsSensitiveHeaderKey(key), "expected %q to be sensitive", key) + } + + notSensitive := []string{"Content-Type", "Accept", "X-Request-ID", "User-Agent"} + for _, key := range notSensitive { + require.False(t, IsSensitiveHeaderKey(key), "expected %q to not be sensitive", key) + } +} + +func TestSplitAuthSchemePrefix(t *testing.T) { + scheme, secret, ok := SplitAuthSchemePrefix("Bearer my-token-123") + require.True(t, ok) + require.Equal(t, "Bearer ", scheme) + require.Equal(t, "my-token-123", secret) + + scheme, secret, ok = SplitAuthSchemePrefix("Basic dXNlcjpwYXNz") + require.True(t, ok) + require.Equal(t, "Basic ", scheme) + require.Equal(t, "dXNlcjpwYXNz", secret) + + // No prefix + _, _, ok = SplitAuthSchemePrefix("plain-token") + require.False(t, ok) + + // Prefix only (no token part) + _, _, ok = SplitAuthSchemePrefix("Bearer") + require.False(t, ok) +} + +func TestHeaderKeyToEnvSegment(t *testing.T) { + require.Equal(t, "x_api_key", HeaderKeyToEnvSegment("X-API-Key")) + require.Equal(t, "authorization", HeaderKeyToEnvSegment("Authorization")) + require.Equal(t, "content_type", HeaderKeyToEnvSegment("Content-Type")) +} + +func TestResolveHeaderEnvVarName(t *testing.T) { + existing := make(map[string]bool) + name := ResolveHeaderEnvVarName("my_conn", "authorization", existing) + require.Equal(t, "connector.my_conn.authorization", name) + + // Conflict + existing[name] = true + name2 := ResolveHeaderEnvVarName("my_conn", "authorization", existing) + require.Equal(t, "connector.my_conn.authorization_1", name2) +} diff --git a/runtime/templates/registry.go b/runtime/templates/registry.go new file mode 100644 index 000000000000..0dbfc2b8270e --- /dev/null +++ b/runtime/templates/registry.go @@ -0,0 +1,225 @@ +package templates + +import ( + "bytes" + "embed" + "encoding/json" + "fmt" + "io/fs" + "path" + "sort" + "strings" +) + +//go:embed definitions/*/*.json +var definitionsFS embed.FS + +// Registry holds all loaded template definitions. +type Registry struct { + templates map[string]*Template + sorted []*Template // sorted by name for stable List() output +} + +// NewRegistry loads all embedded template definitions from the definitions/ directory tree. +func NewRegistry() (*Registry, error) { + r := &Registry{ + templates: make(map[string]*Template), + } + + err := fs.WalkDir(definitionsFS, "definitions", func(fpath string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !strings.HasSuffix(d.Name(), ".json") { + return nil + } + + data, err := definitionsFS.ReadFile(fpath) + if err != nil { + return fmt.Errorf("reading template %s: %w", fpath, err) + } + + // Skip stub files (placeholders for future implementation); + // these may be empty or contain only a "_reason" field. + if len(bytes.TrimSpace(data)) == 0 { + return nil + } + + var t Template + if err := json.Unmarshal(data, &t); err != nil { + return fmt.Errorf("parsing template %s: %w", fpath, err) + } + + if t.Name == "" { + // Stub file with metadata (e.g. _reason) but no template definition; skip it + return nil + } + + if _, exists := r.templates[t.Name]; exists { + return fmt.Errorf("duplicate template name %q in %s", t.Name, fpath) + } + + // Resolve code_template_file references + dir := path.Dir(fpath) + for i := range t.Files { + if t.Files[i].CodeTemplateFile == "" { + continue + } + tmplPath := dir + "/" + t.Files[i].CodeTemplateFile + content, err := definitionsFS.ReadFile(tmplPath) + if err != nil { + return fmt.Errorf("reading code template file %s for %s: %w", tmplPath, fpath, err) + } + t.Files[i].CodeTemplate = string(content) + } + + // Preserve JSON-defined property order (Go maps lose insertion order). + // Store on struct for backend use, and in the schema as []any for + // protobuf Struct conversion ([]string is not protobuf-compatible). + if t.JSONSchema != nil { + t.PropertyOrder = extractPropertyOrder(data) + if len(t.PropertyOrder) > 0 { + orderAny := make([]any, len(t.PropertyOrder)) + for i, k := range t.PropertyOrder { + orderAny[i] = k + } + t.JSONSchema["x-property-order"] = orderAny + } + } + + r.templates[t.Name] = &t + return nil + }) + if err != nil { + return nil, fmt.Errorf("loading template definitions: %w", err) + } + + // Build sorted list + r.sorted = make([]*Template, 0, len(r.templates)) + for _, t := range r.templates { + r.sorted = append(r.sorted, t) + } + sort.Slice(r.sorted, func(i, j int) bool { + return r.sorted[i].Name < r.sorted[j].Name + }) + + return r, nil +} + +// List returns all templates sorted by name. +func (r *Registry) List() []*Template { + return r.sorted +} + +// ListByTags returns templates that match ALL of the given tags. +// If tags is empty, returns all templates. +func (r *Registry) ListByTags(tags []string) []*Template { + if len(tags) == 0 { + return r.sorted + } + + var result []*Template + for _, t := range r.sorted { + if matchesAllTags(t.Tags, tags) { + result = append(result, t) + } + } + return result +} + +// Get returns a template by name, or nil and false if not found. +func (r *Registry) Get(name string) (*Template, bool) { + t, ok := r.templates[name] + return t, ok +} + +// LookupByDriver finds the template for a given driver and output type. +// This is used by the backward-compatible GenerateTemplate RPC to map +// (driver, resource_type) pairs to template names. +func (r *Registry) LookupByDriver(driver, resourceType string) (*Template, bool) { + switch resourceType { + case "connector": + // Combined templates (e.g. s3-duckdb) contain both connector and model files. + // Check for a combined template first; fall back to standalone connector template. + if t, ok := r.Get(driver + "-duckdb"); ok && hasFileNamed(t, "connector") { + return t, true + } + return r.Get(driver) + case "model": + // Model templates use the pattern driver-{olap} (e.g. s3-duckdb, s3-clickhouse). + // LookupByDriver defaults to DuckDB; for other OLAPs use Get() directly. + return r.Get(driver + "-duckdb") + } + return nil, false +} + +// hasFileNamed returns true if the template has a file entry with the given name. +func hasFileNamed(t *Template, name string) bool { + for _, f := range t.Files { + if f.Name == name { + return true + } + } + return false +} + +// extractPropertyOrder parses raw JSON bytes to extract the key ordering of +// json_schema.properties. Go's map[string]any loses insertion order on unmarshal, +// but json.Decoder preserves it. +func extractPropertyOrder(raw []byte) []string { + var outer struct { + JSONSchema json.RawMessage `json:"json_schema"` + } + if err := json.Unmarshal(raw, &outer); err != nil || outer.JSONSchema == nil { + return nil + } + + var schema struct { + Properties json.RawMessage `json:"properties"` + } + if err := json.Unmarshal(outer.JSONSchema, &schema); err != nil || schema.Properties == nil { + return nil + } + + dec := json.NewDecoder(bytes.NewReader(schema.Properties)) + tok, err := dec.Token() // opening { + if err != nil { + return nil + } + if delim, ok := tok.(json.Delim); !ok || delim != '{' { + return nil + } + + var keys []string + for dec.More() { + tok, err := dec.Token() + if err != nil { + break + } + key, ok := tok.(string) + if !ok { + break + } + keys = append(keys, key) + // Skip the property value object + var discard json.RawMessage + if err := dec.Decode(&discard); err != nil { + break + } + } + return keys +} + +// matchesAllTags returns true if the template's tags contain all of the required tags. +func matchesAllTags(templateTags, requiredTags []string) bool { + tagSet := make(map[string]bool, len(templateTags)) + for _, t := range templateTags { + tagSet[t] = true + } + for _, req := range requiredTags { + if !tagSet[req] { + return false + } + } + return true +} diff --git a/runtime/templates/registry_test.go b/runtime/templates/registry_test.go new file mode 100644 index 000000000000..8723474c0dd2 --- /dev/null +++ b/runtime/templates/registry_test.go @@ -0,0 +1,157 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" + "google.golang.org/protobuf/types/known/structpb" +) + +func TestNewRegistry(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + require.NotNil(t, r) + + // Verify all definitions loaded + all := r.List() + require.Greater(t, len(all), 25, "expected at least 25 template definitions") +} + +func TestRegistryGet(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + // Known templates exist + for _, name := range []string{"clickhouse", "s3-duckdb", "gcs-duckdb", "iceberg-duckdb", "snowflake-duckdb"} { + t.Run(name, func(t *testing.T) { + tmpl, ok := r.Get(name) + require.True(t, ok, "template %q should exist", name) + require.Equal(t, name, tmpl.Name) + require.NotEmpty(t, tmpl.DisplayName) + require.NotEmpty(t, tmpl.Files) + }) + } + + // Unknown template doesn't exist + _, ok := r.Get("nonexistent") + require.False(t, ok) +} + +func TestRegistryListByTags(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + // Filter by "duckdb" should return DuckDB-related templates + duckdbTemplates := r.ListByTags([]string{"duckdb"}) + require.Greater(t, len(duckdbTemplates), 5, "expected several duckdb-tagged templates") + for _, tmpl := range duckdbTemplates { + require.Contains(t, tmpl.Tags, "duckdb") + } + + // Filter by "olap" + "connector" should return OLAP connector templates + olapConnectors := r.ListByTags([]string{"olap", "connector"}) + require.Greater(t, len(olapConnectors), 0) + for _, tmpl := range olapConnectors { + require.Contains(t, tmpl.Tags, "olap") + require.Contains(t, tmpl.Tags, "connector") + } + + // Empty tags returns all templates + allTemplates := r.ListByTags(nil) + require.Equal(t, len(r.List()), len(allTemplates)) +} + +func TestRegistryLookupByDriver(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + // Connector lookup: combined templates (s3-duckdb has connector file) + tmpl, ok := r.LookupByDriver("s3", "connector") + require.True(t, ok) + require.Equal(t, "s3-duckdb", tmpl.Name) + + // Model lookup for object stores: driver-duckdb + tmpl, ok = r.LookupByDriver("s3", "model") + require.True(t, ok) + require.Equal(t, "s3-duckdb", tmpl.Name) + + // Model lookup for warehouses: driver-duckdb + tmpl, ok = r.LookupByDriver("snowflake", "model") + require.True(t, ok) + require.Equal(t, "snowflake-duckdb", tmpl.Name) +} + +func TestRegistryTemplatesSorted(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + all := r.List() + for i := 1; i < len(all); i++ { + require.Less(t, all[i-1].Name, all[i].Name, "templates should be sorted by name") + } +} + +func TestRegistryAllDefinitionsValid(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + for _, tmpl := range r.List() { + t.Run(tmpl.Name, func(t *testing.T) { + require.NotEmpty(t, tmpl.Name) + require.NotEmpty(t, tmpl.DisplayName) + require.NotEmpty(t, tmpl.Tags) + require.NotEmpty(t, tmpl.Files) + + for _, f := range tmpl.Files { + require.NotEmpty(t, f.Name, "file name required for template %s", tmpl.Name) + require.NotEmpty(t, f.PathTemplate, "path template required for template %s file %s", tmpl.Name, f.Name) + require.NotEmpty(t, f.CodeTemplate, "code template required for template %s file %s", tmpl.Name, f.Name) + require.Contains(t, []string{"connector", "model"}, f.Name, "file name must be connector or model for template %s", tmpl.Name) + } + }) + } +} + +func TestRegistryTemplatesWithJSONSchemaValid(t *testing.T) { + r, err := NewRegistry() + require.NoError(t, err) + + var found int + for _, tmpl := range r.List() { + if tmpl.JSONSchema == nil { + continue + } + found++ + t.Run(tmpl.Name, func(t *testing.T) { + // Schema must be an object type + typ, ok := tmpl.JSONSchema["type"] + require.True(t, ok, "json_schema must have a type field") + require.Equal(t, "object", typ) + + // Schema must have properties + propsRaw, ok := tmpl.JSONSchema["properties"] + require.True(t, ok, "json_schema must have properties") + propsMap, ok := propsRaw.(map[string]any) + require.True(t, ok, "properties must be a map") + require.Greater(t, len(propsMap), 0, "properties must not be empty") + + // Each property with x-secret must have x-env-var + for key, propRaw := range propsMap { + prop, ok := propRaw.(map[string]any) + if !ok { + continue + } + if secret, _ := prop["x-secret"].(bool); secret { + envVar, hasEnvVar := prop["x-env-var"] + require.True(t, hasEnvVar, "property %q has x-secret but no x-env-var", key) + require.NotEmpty(t, envVar, "x-env-var for %q must not be empty", key) + } + } + + // Verify structpb conversion works (catches int vs float64 issues) + _, err := structpb.NewStruct(tmpl.JSONSchema) + require.NoError(t, err, "json_schema for %q must be convertible to protobuf Struct", tmpl.Name) + }) + } + require.Greater(t, found, 0, "expected at least one template with json_schema") +} diff --git a/runtime/templates/render.go b/runtime/templates/render.go new file mode 100644 index 000000000000..dfc15ad06cab --- /dev/null +++ b/runtime/templates/render.go @@ -0,0 +1,389 @@ +package templates + +import ( + "bytes" + "fmt" + "sort" + "strings" + "text/template" +) + +// RenderInput contains all parameters for rendering a template. +type RenderInput struct { + Template *Template + Output string // "connector", "model", or "" for all files + Properties map[string]any // raw form values + ConnectorName string // for model outputs: the connector to reference + ExistingEnv map[string]bool +} + +// RenderOutput contains the result of rendering. +type RenderOutput struct { + Files []RenderedFile + EnvVars map[string]string +} + +// RenderedFile is a single rendered output file. +type RenderedFile struct { + Path string + Blob string +} + +// Render executes a template with the given input, producing rendered files and env vars. +// The rendering pipeline: +// 1. Pre-processes properties (secret extraction, empty filtering, derived fields) +// 2. Builds a template data map +// 3. Renders each matching file's path and code templates +func Render(input *RenderInput) (*RenderOutput, error) { + if input.Template == nil { + return nil, fmt.Errorf("template is nil") + } + + envVars := make(map[string]string) + existingEnv := cloneEnvMap(input.ExistingEnv) + + // Build the template data context + data := buildTemplateData(input, existingEnv, envVars) + + // Render each matching file + var files []RenderedFile + for _, f := range input.Template.Files { + if input.Output != "" && f.Name != input.Output { + continue + } + + path, err := renderString(f.Name+"_path", f.PathTemplate, data) + if err != nil { + return nil, fmt.Errorf("rendering path template for %q: %w", f.Name, err) + } + + blob, err := renderString(f.Name+"_code", f.CodeTemplate, data) + if err != nil { + return nil, fmt.Errorf("rendering code template for %q: %w", f.Name, err) + } + + files = append(files, RenderedFile{ + Path: strings.TrimSpace(path), + Blob: blob, + }) + } + + return &RenderOutput{Files: files, EnvVars: envVars}, nil +} + +// buildTemplateData creates the data map passed to Go templates. +// It pre-processes properties: extracts secrets, filters empties, adds derived fields. +func buildTemplateData(input *RenderInput, existingEnv map[string]bool, envVars map[string]string) map[string]any { + data := make(map[string]any) + + // Basic fields + data["driver"] = input.Template.Driver + data["connector_name"] = input.ConnectorName + data["docs_url"] = input.Template.DocsURL + + // Derive model_name from the "name" property if present + if name, ok := input.Properties["name"]; ok && !isEmpty(name) { + data["model_name"] = fmt.Sprintf("%v", name) + } + + // Pre-populate schema properties: use "default" if defined, otherwise empty string. + // This ensures templates render cleanly and conditional branches (e.g. auth_method) + // work correctly even when the frontend omits optional UI-only fields. + if input.Template.JSONSchema != nil { + for k, prop := range schemaProperties(input.Template.JSONSchema) { + if def := schemaFieldString(prop, "default"); def != "" { + data[k] = def + } else { + data[k] = "" + } + } + } + + // Copy all raw properties into data (overwrites empty defaults above) + for k, v := range input.Properties { + if !isEmpty(v) { + data[k] = fmt.Sprintf("%v", v) + } + } + + // Schema-based path: extract property metadata from JSON Schema + if input.Template.JSONSchema != nil { + allProps := processPropertiesFromSchema( + input.Template.JSONSchema, + input.Template.PropertyOrder, + input.Template.Driver, + input.Properties, + existingEnv, + envVars, + ) + + // Split by x-step so connector and model files get the right properties + configProps, sourceProps := splitPropsByStep(allProps, input.Template.JSONSchema) + data["props"] = configProps + data["config_props"] = configProps + data["source_props"] = sourceProps + + return data + } + + // Template without schema: pass properties as-is + return data +} + +// processPropertiesFromSchema pre-processes properties using JSON Schema metadata. +// Fields with "x-secret": true are extracted to env vars; "x-ui-only": true fields are skipped. +// Quoting is determined by the schema "type" field (number/boolean unquoted; everything else quoted). +func processPropertiesFromSchema( + schema map[string]any, + propertyOrder []string, + driverName string, + rawProps map[string]any, + existingEnv map[string]bool, + envVars map[string]string, +) []ProcessedProp { + propsMap := schemaProperties(schema) + if len(propsMap) == 0 { + return nil + } + + // Use schema-defined property order if available (preserves JSON key ordering); + // fall back to alphabetical for deterministic output. + keys := propertyOrder + if len(keys) == 0 { + keys = make([]string, 0, len(propsMap)) + for k := range propsMap { + keys = append(keys, k) + } + sort.Strings(keys) + } + + var result []ProcessedProp + for _, key := range keys { + prop := propsMap[key] + + // Skip UI-only fields (e.g. auth_method radio buttons) + if schemaFieldBool(prop, "x-ui-only") { + continue + } + + val, ok := rawProps[key] + if !ok || isEmpty(val) { + continue + } + + // Skip properties whose value matches the schema default when x-omit-if-default is set. + // For example, ClickHouse's "managed" defaults to false; rendering "managed: false" + // is unnecessary noise, but "managed: true" (rill-managed) must appear. + if schemaFieldBool(prop, "x-omit-if-default") { + if defVal, hasDefault := prop["default"]; hasDefault && fmt.Sprint(val) == fmt.Sprint(defVal) { + continue + } + } + + // Handle map-typed properties (e.g. headers). + // The frontend key-value editor sends [{key, value}, ...]; convert to a flat map. + mapVal, isMap := val.(map[string]any) + if !isMap { + if arrVal, isArr := val.([]any); isArr { + mapVal = kvArrayToMap(arrVal) + if mapVal == nil { + // Empty or malformed array (e.g. []); skip to avoid rendering "[]" + continue + } + isMap = true + } + } + if isMap { + headerIdent := driverName + headerProps := processHeaders(mapVal, headerIdent, existingEnv, envVars) + result = append(result, headerProps...) + continue + } + + strVal := fmt.Sprintf("%v", val) + + if schemaFieldBool(prop, "x-secret") { + envName := ResolveEnvVarNameForKey(driverName, key, schemaFieldString(prop, "x-env-var"), existingEnv) + existingEnv[envName] = true + envVars[envName] = strVal + result = append(result, ProcessedProp{ + Key: key, + Value: fmt.Sprintf("{{ .env.%s }}", envName), + Quoted: true, + }) + } else { + propType := schemaFieldString(prop, "type") + quoted := propType != "number" && propType != "boolean" + result = append(result, ProcessedProp{ + Key: key, + Value: strVal, + Quoted: quoted, + }) + } + } + return result +} + +// splitPropsByStep separates processed props into connector-step and source-step slices +// based on the x-step field in the JSON Schema. Props without x-step go into both. +func splitPropsByStep(props []ProcessedProp, schema map[string]any) (configProps, sourceProps []ProcessedProp) { + propsMap := schemaProperties(schema) + for _, p := range props { + step := "" + if propSchema, ok := propsMap[p.Key]; ok { + step = schemaFieldString(propSchema, "x-step") + } + switch step { + case "connector": + configProps = append(configProps, p) + case "source": + sourceProps = append(sourceProps, p) + case "explorer": + // Explorer-step props are accessed directly as template variables (e.g. .sql, .name); + // they are excluded from renderProps output to avoid duplication. + default: + configProps = append(configProps, p) + sourceProps = append(sourceProps, p) + } + } + return +} + +// kvArrayToMap converts [{key: "k", value: "v"}, ...] (from the frontend key-value editor) +// to map[string]any{"k": "v", ...}. Returns nil if the array is empty or not in the expected format. +func kvArrayToMap(arr []any) map[string]any { + result := make(map[string]any, len(arr)) + for _, item := range arr { + obj, ok := item.(map[string]any) + if !ok { + return nil + } + k, _ := obj["key"].(string) + v, _ := obj["value"].(string) + if k != "" { + result[k] = v + } + } + if len(result) == 0 { + return nil + } + return result +} + +// schemaProperties extracts the "properties" map from a JSON Schema object. +func schemaProperties(schema map[string]any) map[string]map[string]any { + raw, ok := schema["properties"] + if !ok { + return nil + } + outer, ok := raw.(map[string]any) + if !ok { + return nil + } + result := make(map[string]map[string]any, len(outer)) + for k, v := range outer { + if prop, ok := v.(map[string]any); ok { + result[k] = prop + } + } + return result +} + +// schemaFieldBool returns the bool value of a field in a schema property map. +func schemaFieldBool(prop map[string]any, key string) bool { + v, _ := prop[key].(bool) + return v +} + +// schemaFieldString returns the string value of a field in a schema property map. +func schemaFieldString(prop map[string]any, key string) string { + v, _ := prop[key].(string) + return v +} + +// processHeaders processes a map of header key-value pairs, extracting sensitive values. +func processHeaders(headers map[string]any, connectorName string, existingEnv map[string]bool, envVars map[string]string) []ProcessedProp { + if len(headers) == 0 { + return nil + } + + // Sort keys for deterministic output + keys := make([]string, 0, len(headers)) + for k := range headers { + keys = append(keys, k) + } + sort.Strings(keys) + + // Build a single "headers" prop whose value is a YAML mapping + var headerLines []string + for _, key := range keys { + strVal := fmt.Sprintf("%v", headers[key]) + if IsSensitiveHeaderKey(key) { + envSegment := HeaderKeyToEnvSegment(key) + envName := ResolveHeaderEnvVarName(connectorName, envSegment, existingEnv) + existingEnv[envName] = true + if scheme, secret, ok := SplitAuthSchemePrefix(strVal); ok { + envVars[envName] = secret + headerLines = append(headerLines, fmt.Sprintf(" %s: \"%s{{ .env.%s }}\"", key, scheme, envName)) + } else { + envVars[envName] = strVal + headerLines = append(headerLines, fmt.Sprintf(" %s: \"{{ .env.%s }}\"", key, envName)) + } + } else { + headerLines = append(headerLines, fmt.Sprintf(" %s: %q", key, strVal)) + } + } + + // Return as a single rendered block (the template will output it as-is) + return []ProcessedProp{{ + Key: "headers", + Value: "\n" + strings.Join(headerLines, "\n"), + Quoted: false, // The value is a nested YAML mapping, not a scalar + }} +} + +// renderString renders a Go template string using [[ ]] delimiters. +func renderString(name, tmplText string, data map[string]any) (string, error) { + t, err := template.New(name). + Delims("[[", "]]"). + Funcs(sharedFuncMap). + Parse(tmplText) + if err != nil { + return "", fmt.Errorf("parsing template: %w", err) + } + + var buf bytes.Buffer + if err := t.Execute(&buf, data); err != nil { + return "", fmt.Errorf("executing template: %w", err) + } + return buf.String(), nil +} + +// isEmpty checks if a value is empty (nil, empty string, or empty map). +func isEmpty(v any) bool { + if v == nil { + return true + } + switch val := v.(type) { + case string: + return val == "" + case bool: + return false // bools are never "empty" + case map[string]any: + return len(val) == 0 + default: + return fmt.Sprintf("%v", v) == "" + } +} + +// cloneEnvMap creates a shallow copy of an env key map. +func cloneEnvMap(m map[string]bool) map[string]bool { + if m == nil { + return make(map[string]bool) + } + clone := make(map[string]bool, len(m)) + for k, v := range m { + clone[k] = v + } + return clone +} diff --git a/runtime/templates/render_test.go b/runtime/templates/render_test.go new file mode 100644 index 000000000000..dbb54faa00a5 --- /dev/null +++ b/runtime/templates/render_test.go @@ -0,0 +1,649 @@ +package templates + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestRenderConnectorTemplate(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + // s3-duckdb is a combined template with both connector and model files + tmpl, ok := registry.Get("s3-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{ + "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE", + "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", + "region": "us-east-1", + }, + ConnectorName: "my_s3", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: connector") + require.Contains(t, blob, "driver: s3") + require.Contains(t, blob, `aws_access_key_id: "{{ .env.AWS_ACCESS_KEY_ID }}"`) + require.Contains(t, blob, `aws_secret_access_key: "{{ .env.AWS_SECRET_ACCESS_KEY }}"`) + require.Contains(t, blob, `region: "us-east-1"`) + require.NotContains(t, blob, "AKIAIOSFODNN7EXAMPLE") + + // Connector output should NOT contain source-step properties + require.NotContains(t, blob, "path") + require.NotContains(t, blob, "name") + + // Verify env vars extracted + require.Equal(t, "AKIAIOSFODNN7EXAMPLE", result.EnvVars["AWS_ACCESS_KEY_ID"]) + require.Equal(t, "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", result.EnvVars["AWS_SECRET_ACCESS_KEY"]) + + // Verify path + require.Equal(t, "connectors/my_s3.yaml", result.Files[0].Path) +} + +func TestRenderDuckDBModelTemplate(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("s3-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "s3://my-bucket/data/*.parquet", + "name": "my_model", + }, + ConnectorName: "my_s3_connector", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: duckdb") + require.Contains(t, blob, `create_secrets_from_connectors: "my_s3_connector"`) + require.Contains(t, blob, "read_parquet") + require.Contains(t, blob, "s3://my-bucket/data/*.parquet") + + // Verify path uses model_name from "name" property + require.Equal(t, "models/my_model.yaml", result.Files[0].Path) +} + +func TestRenderWarehouseModelTemplate(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("snowflake-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "sql": "SELECT * FROM my_table", + "name": "snowflake_data", + }, + ConnectorName: "my_snowflake", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, `connector: "my_snowflake"`) + require.Contains(t, blob, "materialize: true") + require.Contains(t, blob, "SELECT * FROM my_table") + require.Contains(t, blob, "SELECT * FROM my_table limit 10000") + + require.Equal(t, "models/snowflake_data.yaml", result.Files[0].Path) +} + +func TestRenderRedshiftModelNoDevSection(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("redshift-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "sql": "SELECT * FROM my_table", + "name": "rs_data", + }, + ConnectorName: "my_redshift", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, `connector: "my_redshift"`) + // Redshift template should NOT have a dev section + require.NotContains(t, blob, "dev:") + require.NotContains(t, blob, `ref "self"`) +} + +func TestRenderIcebergDuckDB(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("iceberg-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "s3://my-iceberg-bucket/warehouse/my_table", + "name": "iceberg_data", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "iceberg_scan") + require.Contains(t, blob, "s3://my-iceberg-bucket/warehouse/my_table") + + require.Equal(t, "models/iceberg_data.yaml", result.Files[0].Path) +} + +func TestRenderS3ClickHouseModel(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("s3-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE", + "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", + "path": "s3://my-bucket/data/events.parquet", + "name": "s3_events", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: clickhouse") + require.Contains(t, blob, "materialize: true") + // SQL should show the s3() function with env var refs + require.Contains(t, blob, "FROM s3(") + require.Contains(t, blob, "https://my-bucket.s3.amazonaws.com/data/events.parquet") + require.Contains(t, blob, "{{ .env.AWS_ACCESS_KEY_ID }}") + require.Contains(t, blob, "{{ .env.AWS_SECRET_ACCESS_KEY }}") + // Raw secrets should NOT appear in the blob + require.NotContains(t, blob, "AKIAIOSFODNN7EXAMPLE") + + // Env vars should be extracted + require.Equal(t, "AKIAIOSFODNN7EXAMPLE", result.EnvVars["AWS_ACCESS_KEY_ID"]) + + require.Equal(t, "models/s3_events.yaml", result.Files[0].Path) +} + +func TestRenderMySQLClickHouseModel(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("mysql-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "host": "db.example.com", + "port": "3306", + "user": "myuser", + "password": "secret123", + "database": "mydb", + "table": "events", + "name": "mysql_events", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "connector: clickhouse") + require.Contains(t, blob, "FROM mysql(") + require.Contains(t, blob, "db.example.com:3306") + require.Contains(t, blob, "mydb") + require.Contains(t, blob, "events") + require.Contains(t, blob, "myuser") + require.Contains(t, blob, "{{ .env.MYSQL_PASSWORD }}") + require.NotContains(t, blob, "secret123") +} + +func TestRenderHTTPSClickHouseWithHeaders(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("https-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + // Frontend key-value editor sends [{key, value}, ...] format + "headers": []any{ + map[string]any{"key": "Authorization", "value": "Bearer my-secret-token"}, + map[string]any{"key": "X-API-Key", "value": "key123"}, + }, + "path": "https://example.com/data.csv", + "name": "api_data", + }, + ConnectorName: "my_https", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: clickhouse") + require.Contains(t, blob, "url(") + require.Contains(t, blob, "https://example.com/data.csv") + require.Contains(t, blob, "CSVWithNames") + require.Contains(t, blob, "headers(") + require.Contains(t, blob, "'Authorization'=") + require.Contains(t, blob, "'X-API-Key'=") + // Raw secrets should NOT appear in the blob + require.NotContains(t, blob, "my-secret-token") + require.NotContains(t, blob, "key123") + + // Env vars should be extracted for sensitive headers + require.Contains(t, result.EnvVars, "connector.https.authorization") + require.Contains(t, result.EnvVars, "connector.https.x_api_key") +} + +func TestRenderHTTPSClickHouseNoHeaders(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("https-clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "https://example.com/data.csv", + "name": "simple_data", + }, + ConnectorName: "my_https", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "url('https://example.com/data.csv')") + // No headers() clause when no headers provided + require.NotContains(t, blob, "headers(") +} + +func TestRenderEnvVarConflict(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + // s3-duckdb is a combined template with json_schema + tmpl, ok := registry.Get("s3-duckdb") + require.True(t, ok) + + // Pre-populate existing env to force conflict + existingEnv := map[string]bool{ + "AWS_ACCESS_KEY_ID": true, + "AWS_SECRET_ACCESS_KEY": true, + } + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{ + "aws_access_key_id": "AKIA_NEW", + "aws_secret_access_key": "SECRET_NEW", + }, + ConnectorName: "s3_2", + ExistingEnv: existingEnv, + }) + require.NoError(t, err) + + // Env vars should have _1 suffix due to conflict + require.Contains(t, result.EnvVars, "AWS_ACCESS_KEY_ID_1") + require.Contains(t, result.EnvVars, "AWS_SECRET_ACCESS_KEY_1") + require.Contains(t, result.Files[0].Blob, "AWS_ACCESS_KEY_ID_1") +} + +func TestRenderEmptyPropertiesFiltered(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("clickhouse") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{ + "host": "localhost", + "port": 9000, + "password": "secret", + "database": "", // empty; should be filtered + }, + ConnectorName: "my_ch", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + + blob := result.Files[0].Blob + require.Contains(t, blob, "host") + require.Contains(t, blob, "port") + require.Contains(t, blob, "CLICKHOUSE_PASSWORD") + require.NotContains(t, blob, "database") +} + +func TestRenderOutputFilter(t *testing.T) { + // Create a two-file template inline for testing + tmpl := &Template{ + Name: "test-multi", + DisplayName: "Test Multi-file", + Tags: []string{"test"}, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/[[ .connector_name ]].yaml", + CodeTemplate: "type: connector\ndriver: test\n", + }, + { + Name: "model", + PathTemplate: "models/[[ .model_name ]].yaml", + CodeTemplate: "type: model\nconnector: duckdb\n", + }, + }, + } + + // All files + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "", + Properties: map[string]any{"name": "test"}, + ConnectorName: "test_conn", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 2) + + // Connector only + result, err = Render(&RenderInput{ + Template: tmpl, + Output: "connector", + Properties: map[string]any{"name": "test"}, + ConnectorName: "test_conn", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + require.Contains(t, result.Files[0].Blob, "type: connector") + + // Model only + result, err = Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{"name": "test"}, + ConnectorName: "test_conn", + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + require.Contains(t, result.Files[0].Blob, "type: model") +} + +func TestRenderLocalFileDuckDB(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("local_file-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "path": "data/sales.csv", + "name": "sales", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: duckdb") + require.Contains(t, blob, "read_csv") + require.Contains(t, blob, "data/sales.csv") +} + +func TestRenderSQLiteDuckDB(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("sqlite-duckdb") + require.True(t, ok) + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "db": "/data/app.db", + "table": "users", + "name": "sqlite_users", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "connector: duckdb") + require.Contains(t, blob, "sqlite_scan") + require.Contains(t, blob, "/data/app.db") + require.Contains(t, blob, "users") +} + +// --- Schema-based template tests --- + +func TestRenderIcebergDuckDBWithSchema(t *testing.T) { + registry, err := NewRegistry() + require.NoError(t, err) + + tmpl, ok := registry.Get("iceberg-duckdb") + require.True(t, ok) + require.NotNil(t, tmpl.JSONSchema, "iceberg-duckdb should have json_schema") + + result, err := Render(&RenderInput{ + Template: tmpl, + Output: "model", + Properties: map[string]any{ + "aws_access_key_id": "AKIAEXAMPLE", + "aws_secret_access_key": "wJalrXUtnFEMI/EXAMPLEKEY", + "aws_region": "us-west-2", + "path": "s3://my-iceberg-bucket/warehouse/my_table", + "name": "iceberg_test", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + require.Len(t, result.Files, 1) + + blob := result.Files[0].Blob + require.Contains(t, blob, "type: model") + require.Contains(t, blob, "iceberg_scan") + require.Contains(t, blob, "s3://my-iceberg-bucket/warehouse/my_table") + + // Verify path uses model_name from "name" property + require.Equal(t, "models/iceberg_test.yaml", result.Files[0].Path) + + // Verify secret extraction via JSON Schema + require.Equal(t, "AKIAEXAMPLE", result.EnvVars["AWS_ACCESS_KEY_ID"]) + require.Equal(t, "wJalrXUtnFEMI/EXAMPLEKEY", result.EnvVars["AWS_SECRET_ACCESS_KEY"]) + + // Raw secrets should NOT appear in the blob + require.NotContains(t, blob, "AKIAEXAMPLE") + require.NotContains(t, blob, "wJalrXUtnFEMI/EXAMPLEKEY") +} + +func TestRenderSchemaEnvVarConflict(t *testing.T) { + tmpl := &Template{ + Name: "test-schema", + DisplayName: "Test Schema", + Tags: []string{"test"}, + JSONSchema: map[string]any{ + "type": "object", + "properties": map[string]any{ + "api_key": map[string]any{ + "type": "string", + "x-secret": true, + "x-env-var": "MY_API_KEY", + }, + }, + }, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/test.yaml", + CodeTemplate: "type: connector\n[[ renderProps .props ]]", + }, + }, + } + + // Pre-populate existing env to force conflict + existingEnv := map[string]bool{"MY_API_KEY": true} + + result, err := Render(&RenderInput{ + Template: tmpl, + Properties: map[string]any{"api_key": "secret123"}, + ExistingEnv: existingEnv, + }) + require.NoError(t, err) + + // Should use MY_API_KEY_1 due to conflict + require.Contains(t, result.EnvVars, "MY_API_KEY_1") + require.Equal(t, "secret123", result.EnvVars["MY_API_KEY_1"]) + require.Contains(t, result.Files[0].Blob, "MY_API_KEY_1") +} + +func TestRenderSchemaUIOnlyFieldsSkipped(t *testing.T) { + tmpl := &Template{ + Name: "test-ui-only", + DisplayName: "Test UI Only", + Tags: []string{"test"}, + JSONSchema: map[string]any{ + "type": "object", + "properties": map[string]any{ + "auth_method": map[string]any{ + "type": "string", + "x-ui-only": true, + }, + "host": map[string]any{ + "type": "string", + }, + }, + }, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/test.yaml", + CodeTemplate: "type: connector\n[[ renderProps .props ]]", + }, + }, + } + + result, err := Render(&RenderInput{ + Template: tmpl, + Properties: map[string]any{ + "auth_method": "access_keys", + "host": "example.com", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + + blob := result.Files[0].Blob + require.Contains(t, blob, "host") + require.NotContains(t, blob, "auth_method") +} + +func TestRenderSchemaPropertyTypes(t *testing.T) { + tmpl := &Template{ + Name: "test-types", + DisplayName: "Test Types", + Tags: []string{"test"}, + JSONSchema: map[string]any{ + "type": "object", + "properties": map[string]any{ + "host": map[string]any{ + "type": "string", + }, + "port": map[string]any{ + "type": "number", + }, + "ssl": map[string]any{ + "type": "boolean", + }, + }, + }, + Files: []File{ + { + Name: "connector", + PathTemplate: "connectors/test.yaml", + CodeTemplate: "type: connector\n[[ renderProps .props ]]", + }, + }, + } + + result, err := Render(&RenderInput{ + Template: tmpl, + Properties: map[string]any{ + "host": "example.com", + "port": "9440", + "ssl": "true", + }, + ExistingEnv: make(map[string]bool), + }) + require.NoError(t, err) + + blob := result.Files[0].Blob + // Strings should be quoted + require.Contains(t, blob, `host: "example.com"`) + // Numbers should not be quoted + require.Contains(t, blob, "port: 9440") + // Booleans should not be quoted + require.Contains(t, blob, "ssl: true") +} diff --git a/runtime/templates/template.go b/runtime/templates/template.go new file mode 100644 index 000000000000..9113b58c9b72 --- /dev/null +++ b/runtime/templates/template.go @@ -0,0 +1,43 @@ +package templates + +// Template is a declarative definition for generating project files (connectors, models, etc.) +// from structured form data. Each template knows how to produce one or more output files. +// +// Templates with JSONSchema are self-contained: the schema drives form rendering (frontend) +// and property metadata like secret detection (backend). Templates without JSONSchema fall +// back to drivers.Spec for property metadata. +type Template struct { + Name string `json:"name"` + DisplayName string `json:"display_name"` + Description string `json:"description,omitempty"` // short description for UI display + DocsURL string `json:"docs_url,omitempty"` // link to documentation + Driver string `json:"driver"` // primary driver (e.g. "s3"); empty for driverless templates like iceberg + OLAP string `json:"olap"` // target OLAP engine (e.g. "duckdb"); empty for OLAP connector templates + Icon string `json:"icon,omitempty"` // icon component name for full-size display (e.g. add-data grid) + SmallIcon string `json:"small_icon,omitempty"` // icon component name for small display (e.g. nav, cards) + Tags []string `json:"tags"` + JSONSchema map[string]any `json:"json_schema,omitempty"` // JSON Schema for form generation and property metadata + PropertyOrder []string `json:"-"` // JSON-defined property key order; computed at load time + Files []File `json:"files"` +} + +// File describes a single output file within a template. +// PathTemplate and CodeTemplate use Go text/template syntax with [[ ]] delimiters +// to avoid collision with Rill's {{ .env.VAR }} runtime syntax. +// +// CodeTemplate can be specified inline (code_template) or loaded from a separate file +// (code_template_file) for readability. If both are set, code_template_file wins. +type File struct { + Name string `json:"name"` // output name: "connector" or "model" + PathTemplate string `json:"path_template"` // Go template for the file path + CodeTemplate string `json:"code_template,omitempty"` // Go template for the file content (inline) + CodeTemplateFile string `json:"code_template_file,omitempty"` // path to .tmpl file (relative to the JSON definition) +} + +// ProcessedProp is a property that has been pre-processed for template rendering. +// Secret values are replaced with {{ .env.VAR }} references; empty values are filtered. +type ProcessedProp struct { + Key string + Value string + Quoted bool // true for strings and secrets; false for numbers and booleans +}