From 130837f086cf017996adb76d5e6f949590690159 Mon Sep 17 00:00:00 2001 From: Data Contract Enthusiast <80169154+DatSciMeMa@users.noreply.github.com> Date: Tue, 5 May 2026 16:08:17 +0200 Subject: [PATCH 01/29] Update custom-other-properties.md As discussed in the working group added an avis for an updated description for version 3.2.0. --- docs/custom-other-properties.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/custom-other-properties.md b/docs/custom-other-properties.md index 7ec49cc..a3ea323 100644 --- a/docs/custom-other-properties.md +++ b/docs/custom-other-properties.md @@ -39,6 +39,7 @@ customProperties: | customProperties.property | string | Property | No | The name of the key. Names should be in camel case–the same as if they were permanent properties in the contract. | | customProperties.value | any | Value | No | The value of the key. It can be an array. | | customProperties.description | string | Description | No | Description for humans. | +Avis: With version 3.2.0 the Description of 'customProperties.property' will be updated to "The name of the key. Could be in any notation. If this field is used for referencing it should be in camel case–the same as if they were permanent properties in the contract. Note that since ODCS version 3.1 the field 'customProperties.id' should be used for referencing purposes. In this case the name of the key could be human-readable and self-explanatory to the greatest extent." ## Authoritative Definitions From e1fcce86114494609ee009b32a8550e2eb24ed0e Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 5 May 2026 07:34:22 -0700 Subject: [PATCH 02/29] fix(schema): mark dataProduct as deprecated in v3.1.0 The v3.1.0 standard deprecates `dataProduct`, but neither the v3.1.0 JSON schema, the `latest` JSON schema, nor `docs/fundamentals.md` reflected the deprecation. Adds `"deprecated": true` to both schemas and strikes through the property in the docs table. Closes #226. Supersedes #238 (which mishandled the dated-snapshot naming convention; that policy lives on `dev` and is not in scope here). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/fundamentals.md | 2 +- schema/odcs-json-schema-latest.json | 3 ++- schema/odcs-json-schema-v3.1.0.json | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/fundamentals.md b/docs/fundamentals.md index a7b604f..06656ac 100644 --- a/docs/fundamentals.md +++ b/docs/fundamentals.md @@ -44,7 +44,7 @@ tags: ['finance'] | tenant | string | Tenant | No | Indicates the property the data is primarily associated with. Value is case insensitive. | | tags | array | Tags | No | A list of tags that may be assigned to the elements (object or property); the tags keyword may appear at any level. Tags may be used to better categorize an element. For example, `finance`, `sensitive`, `employee_record`. | | domain | string | Domain | No | Name of the logical data domain. | -| dataProduct | string | Data Product | No | Name of the data product. | +| ~~dataProduct~~ | string | Data Product | No | Name of the data product. **DEPRECATED since v3.1.0.** | | authoritativeDefinitions | array | Authoritative Definitions | No | List of links to sources that provide more details on the data contract. | | description | object | Description | No | Object containing the descriptions. | | description.purpose | string | Purpose | No | Intended purpose for the provided data. | diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 3596e68..fb74095 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -51,7 +51,8 @@ }, "dataProduct": { "type": "string", - "description": "The name of the data product." + "description": "The name of the data product. DEPRECATED since v3.1.0.", + "deprecated": true }, "description": { "type": "object", diff --git a/schema/odcs-json-schema-v3.1.0.json b/schema/odcs-json-schema-v3.1.0.json index 3596e68..fb74095 100644 --- a/schema/odcs-json-schema-v3.1.0.json +++ b/schema/odcs-json-schema-v3.1.0.json @@ -51,7 +51,8 @@ }, "dataProduct": { "type": "string", - "description": "The name of the data product." + "description": "The name of the data product. DEPRECATED since v3.1.0.", + "deprecated": true }, "description": { "type": "object", From 946dbdbbb026aa6417e07e50fbeff5606c623115 Mon Sep 17 00:00:00 2001 From: nightmoose Date: Tue, 5 May 2026 10:53:55 -0400 Subject: [PATCH 03/29] Update vendors.md Add ContractGate to official ODCS vendors list --- vendors.md | 1 + 1 file changed, 1 insertion(+) diff --git a/vendors.md b/vendors.md index 7a1d87b..23a2ea5 100644 --- a/vendors.md +++ b/vendors.md @@ -7,6 +7,7 @@ Vendors who natively support ODCS (Open Data Contract Standard). A non-exhaustive, alphabetical list of organizations offering solutions natively compatible with ODCS, such as data catalogs, data quality platforms, security tools, and more. * [Actian Data Intelligence Platform](https://www.actian.com/data-intelligence/platform/) - A cloud-native, AI-ready solution that helps organizations discover, understand, trust, and activate their enterprise data assets at scale through intelligent metadata, governance, cataloging, and data marketplace capabilities. +* [ContractGate](https://github.com/nightmoose/contractgate) - Real-time semantic contract enforcement gateway. Enforces rich ODCS-aligned contracts (ontology, glossary, patterns, enums, computed metrics) at ingestion (<50µs p99). Rust engine + hosted SaaS platform. Patent pending. [Platform](https://app.datacontractgate.com/) * [Databricks Ontos](https://marketplace.databricks.com/details/8c582cfa-4c6b-4fdc-93a2-f63b35d93906/Databricks_Ontos) - Governance and metadata management application using ODCS and ODPS as the core standards for data contracts and data products * [Data Caterer](https://data.catering/latest/docs/guide/data-source/metadata/open-data-contract-standard/) - Test data management tool using data contracts as a metadata source * [Data Contract CLI](https://cli.datacontract.com) - Open Source tooling around data contracts. From 244357ed284dd32c16fb27a89195569335b98766 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 5 May 2026 09:08:34 -0700 Subject: [PATCH 04/29] schema: add v3.1.0-20260505 dated snapshot for the deprecation silent fix v3.1.0 is a released line, so the silent-update rule applies: any modification to `odcs-json-schema-v3.1.0.json` should be accompanied by a frozen dated snapshot so downstream consumers who pin a specific day's validator can recover the exact bytes. Snapshot file equals the patched rolling file (`deprecated: true` on `dataProduct`) at this commit. UTC committer date is 2026-05-05. Refs #226, #274. Co-Authored-By: Claude Opus 4.7 (1M context) --- schema/odcs-json-schema-v3.1.0-20260505.json | 2929 ++++++++++++++++++ 1 file changed, 2929 insertions(+) create mode 100644 schema/odcs-json-schema-v3.1.0-20260505.json diff --git a/schema/odcs-json-schema-v3.1.0-20260505.json b/schema/odcs-json-schema-v3.1.0-20260505.json new file mode 100644 index 0000000..fb74095 --- /dev/null +++ b/schema/odcs-json-schema-v3.1.0-20260505.json @@ -0,0 +1,2929 @@ +{ + "$schema": "https://json-schema.org/draft/2019-09/schema", + "title": "Open Data Contract Standard (ODCS)", + "description": "An open data contract specification to establish agreement between data producers and consumers.", + "type": "object", + "properties": { + "version": { + "type": "string", + "description": "Current version of the data contract." + }, + "kind": { + "type": "string", + "default": "DataContract", + "description": "The kind of file this is. Valid value is `DataContract`.", + "enum": ["DataContract"] + }, + "apiVersion": { + "type": "string", + "default": "v3.1.0", + "description": "Version of the standard used to build data contract. Default value is v3.1.0.", + "enum": ["v3.1.0", "v3.0.2", "v3.0.1", "v3.0.0", "v2.2.2", "v2.2.1", "v2.2.0"] + }, + "id": { + "type": "string", + "description": "A unique identifier used to reduce the risk of dataset name collisions, such as a UUID." + }, + "name": { + "type": "string", + "description": "Name of the data contract." + }, + "tenant": { + "type": "string", + "description": "Indicates the property the data is primarily associated with. Value is case insensitive." + }, + "tags": { + "$ref": "#/$defs/Tags" + }, + "status": { + "type": "string", + "description": "Current status of the dataset.", + "examples": [ + "proposed", "draft", "active", "deprecated", "retired" + ] + }, + "servers": { + "type": "array", + "description": "List of servers where the datasets reside.", + "items": { + "$ref": "#/$defs/Server" + } + }, + "dataProduct": { + "type": "string", + "description": "The name of the data product. DEPRECATED since v3.1.0.", + "deprecated": true + }, + "description": { + "type": "object", + "description": "High level description of the dataset.", + "properties": { + "usage": { + "type": "string", + "description": "Intended usage of the dataset." + }, + "purpose": { + "type": "string", + "description": "Purpose of the dataset." + }, + "limitations": { + "type": "string", + "description": "Limitations of the dataset." + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + } + } + }, + "domain": { + "type": "string", + "description": "Name of the logical data domain.", + "examples": ["imdb_ds_aggregate", "receiver_profile_out", "transaction_profile_out"] + }, + "schema": { + "type": "array", + "description": "A list of elements within the schema to be cataloged.", + "items": { + "$ref": "#/$defs/SchemaObject" + } + }, + "support": { + "$ref": "#/$defs/Support" + }, + "price": { + "$ref": "#/$defs/Pricing" + }, + "team": { + "oneOf": [ + { + "$ref": "#/$defs/Team", + "description": "Team information object with members array (v3.1.0+)." + }, + { + "type": "array", + "description": "DEPRECATED: Array of team members. Use the Team object structure instead. This array format is maintained for backward compatibility with v3.0.2 and earlier versions and will be removed in ODCS 4.0.", + "deprecated": true, + "items": { + "$ref": "#/$defs/TeamMember" + } + } + ] + }, + "roles": { + "type": "array", + "description": "A list of roles that will provide user access to the dataset.", + "items": { + "$ref": "#/$defs/Role" + } + }, + "slaDefaultElement": { + "type": "string", + "description": "DEPRECATED SINCE 3.1. WILL BE REMOVED IN ODCS 4.0. Element (using the element path notation) to do the checks on.", + "deprecated": true + }, + "slaProperties": { + "type": "array", + "description": "A list of key/value pairs for SLA specific properties. There is no limit on the type of properties (more details to come).", + "items": { + "$ref": "#/$defs/ServiceLevelAgreementProperty" + } + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + }, + "contractCreatedTs": { + "type": "string", + "format": "date-time", + "description": "Timestamp in UTC of when the data contract was created." + } + }, + "required": ["version", "apiVersion", "kind", "id", "status"], + "additionalProperties": false, + "unevaluatedProperties": false, + "$defs": { + "ShorthandReference": { + "type": "string", + "description": "Shorthand notation using name fields (table_name.column_name)", + "pattern": "^[A-Za-z_][A-Za-z0-9_]*\\.[A-Za-z_][A-Za-z0-9_]*$" + }, + "FullyQualifiedReference": { + "type": "string", + "description": "Fully qualified notation using id fields (section/id/properties/id), optionally prefixed with external file reference", + "pattern": "^(?:(?:https?:\\/\\/)?[A-Za-z0-9._\\-\\/]+\\.yaml#)?\\/?[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+(?:\\/[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+)*$" + }, + "StableId": { + "type": "string", + "description": "Stable technical identifier for references. Must be unique within its containing array. Cannot contain special characters ('-', '_' allowed).", + "pattern": "^[A-Za-z0-9_-]+$" + }, + "Server": { + "type": "object", + "description": "Data source details of where data is physically stored.", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "server": { + "type": "string", + "description": "Identifier of the server." + }, + "type": { + "type": "string", + "description": "Type of the server.", + "enum": [ + "api", "athena", "azure", "bigquery", "clickhouse", "databricks", "denodo", "dremio", + "duckdb", "glue", "cloudsql", "db2", "hive", "impala", "informix", "kafka", "kinesis", "local", + "mysql", "oracle", "postgresql", "postgres", "presto", "pubsub", + "redshift", "s3", "sftp", "snowflake", "sqlserver", "synapse", "trino", "vertica", "zen", "custom" + ] + }, + "description": { + "type": "string", + "description": "Description of the server." + }, + "environment": { + "type": "string", + "description": "Environment of the server.", + "examples": ["prod", "preprod", "dev", "uat"] + }, + "roles": { + "type": "array", + "description": "List of roles that have access to the server.", + "items": { + "$ref": "#/$defs/Role" + } + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + } + }, + "allOf": [ + { + "if": { + "properties": { + "type": { + "const": "api" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/ApiServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "athena" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/AthenaServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "azure" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/AzureServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "bigquery" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/BigQueryServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "clickhouse" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/ClickHouseServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "databricks" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/DatabricksServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "denodo" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/DenodoServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "dremio" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/DremioServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "duckdb" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/DuckdbServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "glue" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/GlueServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "cloudsql" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/GoogleCloudSqlServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "db2" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/IBMDB2Server" + } + }, + { + "if": { + "properties": { + "type": { + "const": "hive" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/HiveServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "impala" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/ImpalaServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "informix" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/InformixServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "zen" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/ZenServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "custom" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/CustomServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "kafka" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/KafkaServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "kinesis" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/KinesisServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "local" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/LocalServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "mysql" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/MySqlServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "oracle" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/OracleServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "postgresql" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/PostgresServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "postgres" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/PostgresServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "presto" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/PrestoServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "pubsub" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/PubSubServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "redshift" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/RedshiftServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "s3" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/S3Server" + } + }, + { + "if": { + "properties": { + "type": { + "const": "sftp" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/SftpServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "snowflake" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/SnowflakeServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "sqlserver" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/SqlserverServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "synapse" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/SynapseServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "trino" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/TrinoServer" + } + }, + { + "if": { + "properties": { + "type": { + "const": "vertica" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/VerticaServer" + } + } + ], + "required": ["server", "type"], + "unevaluatedProperties": false + }, + "ServerSource": { + "ApiServer": { + "type": "object", + "title": "AthenaServer", + "properties": { + "location": { + "type": "string", + "format": "uri", + "description": "The url to the API.", + "examples": [ + "https://api.example.com/v1" + ] + } + }, + "required": [ + "location" + ] + }, + "AthenaServer": { + "type": "object", + "title": "AthenaServer", + "properties": { + "stagingDir": { + "type": "string", + "format": "uri", + "description": "Amazon Athena automatically stores query results and metadata information for each query that runs in a query result location that you can specify in Amazon S3.", + "examples": [ + "s3://my_storage_account_name/my_container/path" + ] + }, + "schema": { + "type": "string", + "description": "Identify the schema in the data source in which your tables exist." + }, + "catalog": { + "type": "string", + "description": "Identify the name of the Data Source, also referred to as a Catalog.", + "default": "awsdatacatalog" + }, + "regionName": { + "type": "string", + "description": "The region your AWS account uses.", + "examples": ["eu-west-1"] + } + }, + "required": [ + "stagingDir", + "schema" + ] + }, + "AzureServer": { + "type": "object", + "title": "AzureServer", + "properties": { + "location": { + "type": "string", + "format": "uri", + "description": "Fully qualified path to Azure Blob Storage or Azure Data Lake Storage (ADLS), supports globs.", + "examples": [ + "az://my_storage_account_name.blob.core.windows.net/my_container/path/*.parquet", + "abfss://my_storage_account_name.dfs.core.windows.net/my_container_name/path/*.parquet" + ] + }, + "format": { + "type": "string", + "examples": [ + "parquet", + "delta", + "json", + "csv" + ], + "description": "File format." + }, + "delimiter": { + "type": "string", + "examples": [ + "new_line", + "array" + ], + "description": "Only for format = json. How multiple json documents are delimited within one file" + } + }, + "required": [ + "location", + "format" + ] + }, + "BigQueryServer": { + "type": "object", + "title": "BigQueryServer", + "properties": { + "project": { + "type": "string", + "description": "The GCP project name." + }, + "dataset": { + "type": "string", + "description": "The GCP dataset name." + } + }, + "required": [ + "project", + "dataset" + ] + }, + "ClickHouseServer": { + "type": "object", + "title": "ClickHouseServer", + "properties": { + "host": { + "type": "string", + "description": "The host of the ClickHouse server." + }, + "port": { + "type": "integer", + "description": "The port to the ClickHouse server." + }, + "database": { + "type": "string", + "description": "The name of the database." + } + }, + "required": [ + "host", + "port", + "database" + ] + }, + "DatabricksServer": { + "type": "object", + "title": "DatabricksServer", + "properties": { + "host": { + "type": "string", + "description": "The Databricks host", + "examples": [ + "dbc-abcdefgh-1234.cloud.databricks.com" + ] + }, + "catalog": { + "type": "string", + "description": "The name of the Hive or Unity catalog" + }, + "schema": { + "type": "string", + "description": "The schema name in the catalog" + } + }, + "required": [ + "catalog", + "schema" + ] + }, + "DenodoServer": { + "type": "object", + "title": "DenodoServer", + "properties": { + "host": { + "type": "string", + "description": "The host of the Denodo server." + }, + "port": { + "type": "integer", + "description": "The port of the Denodo server." + }, + "database": { + "type": "string", + "description": "The name of the database." + } + }, + "required": [ + "host", + "port" + ] + }, + "DremioServer": { + "type": "object", + "title": "DremioServer", + "properties": { + "host": { + "type": "string", + "description": "The host of the Dremio server." + }, + "port": { + "type": "integer", + "description": "The port of the Dremio server." + }, + "schema": { + "type": "string", + "description": "The name of the schema." + } + }, + "required": [ + "host", + "port" + ] + }, + "DuckdbServer": { + "type": "object", + "title": "DuckdbServer", + "properties": { + "database": { + "type": "string", + "description": "Path to duckdb database file." + }, + "schema": { + "type": "string", + "description": "The name of the schema." + } + }, + "required": [ + "database" + ] + }, + "GlueServer": { + "type": "object", + "title": "GlueServer", + "properties": { + "account": { + "type": "string", + "description": "The AWS Glue account", + "examples": [ + "1234-5678-9012" + ] + }, + "database": { + "type": "string", + "description": "The AWS Glue database name", + "examples": [ + "my_database" + ] + }, + "location": { + "type": "string", + "format": "uri", + "description": "The AWS S3 path. Must be in the form of a URL.", + "examples": [ + "s3://datacontract-example-orders-latest/data/{model}" + ] + }, + "format": { + "type": "string", + "description": "The format of the files", + "examples": [ + "parquet", + "csv", + "json", + "delta" + ] + } + }, + "required": [ + "account", + "database" + ] + }, + "GoogleCloudSqlServer": { + "type": "object", + "title": "GoogleCloudSqlServer", + "properties": { + "host": { + "type": "string", + "description": "The host of the Google Cloud Sql server." + }, + "port": { + "type": "integer", + "description": "The port of the Google Cloud Sql server." + }, + "database": { + "type": "string", + "description": "The name of the database." + }, + "schema": { + "type": "string", + "description": "The name of the schema." + } + }, + "required": [ + "host", + "port", + "database", + "schema" + ] + }, + "IBMDB2Server": { + "type": "object", + "title": "IBMDB2Server", + "properties": { + "host": { + "type": "string", + "description": "The host of the IBM DB2 server." + }, + "port": { + "type": "integer", + "description": "The port of the IBM DB2 server." + }, + "database": { + "type": "string", + "description": "The name of the database." + }, + "schema": { + "type": "string", + "description": "The name of the schema." + } + }, + "required": [ + "host", + "port", + "database" + ] + }, + "HiveServer": { + "type": "object", + "title": "HiveServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the Hive server. " + }, + "port": { + "type": "integer", + "description": "The port to the Hive server. Defaults to 10000." + }, + "database": { + "type": "string", + "description": "The name of the Hive database." + } + }, + "required": [ + "host", + "database" + ] + }, + "ImpalaServer": { + "type": "object", + "title": "ImpalaServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the Impala server." + }, + "port": { + "type": "integer", + "description": "The port to the Impala server. Defaults to 21050." + }, + "database": { + "type": "string", + "description": "The name of the Impala database." + } + }, + "required": [ + "host", + "database" + ] + }, + "InformixServer": { + "type": "object", + "title": "InformixServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the Informix server. " + }, + "port": { + "type": "integer", + "description": "The port to the Informix server. Defaults to 9088." + }, + "database": { + "type": "string", + "description": "The name of the database." + } + }, + "required": [ + "host", + "database" + ] + }, + "ZenServer": { + "type": "object", + "title": "ZenServer", + "properties": { + "host": { + "type": "string", + "description": "Hostname or IP address of the Zen server." + }, + "port": { + "type": "integer", + "description": "Zen server SQL connections port. Defaults to 1583." + }, + "database": { + "type": "string", + "description": "Database name to connect to on the Zen server." + } + }, + "required": ["host", "database"] + }, + "CustomServer": { + "type": "object", + "title": "CustomServer", + "properties": { + "account": { + "type": "string", + "description": "Account used by the server." + }, + "catalog": { + "type": "string", + "description": "Name of the catalog." + }, + "database": { + "type": "string", + "description": "Name of the database." + }, + "dataset": { + "type": "string", + "description": "Name of the dataset." + }, + "delimiter": { + "type": "string", + "description": "Delimiter." + }, + "endpointUrl": { + "type": "string", + "description": "Server endpoint.", + "format": "uri" + }, + "format": { + "type": "string", + "description": "File format." + }, + "host": { + "type": "string", + "description": "Host name or IP address." + }, + "location": { + "type": "string", + "description": "A URL to a location.", + "format": "uri" + }, + "path": { + "type": "string", + "description": "Relative or absolute path to the data file(s)." + }, + "port": { + "type": "integer", + "description": "Port to the server. No default value is assumed for custom servers." + }, + "project": { + "type": "string", + "description": "Project name." + }, + "region": { + "type": "string", + "description": "Cloud region." + }, + "regionName": { + "type": "string", + "description": "Region name." + }, + "schema": { + "type": "string", + "description": "Name of the schema." + }, + "serviceName": { + "type": "string", + "description": "Name of the service." + }, + "stagingDir": { + "type": "string", + "description": "Staging directory." + }, + "warehouse": { + "type": "string", + "description": "Name of the cluster or warehouse." + }, + "stream": { + "type": "string", + "description": "Name of the data stream." + } + } + }, + "KafkaServer": { + "type": "object", + "title": "KafkaServer", + "description": "Kafka Server", + "properties": { + "host": { + "type": "string", + "description": "The bootstrap server of the kafka cluster." + }, + "format": { + "type": "string", + "description": "The format of the messages.", + "examples": ["json", "avro", "protobuf", "xml"], + "default": "json" + } + }, + "required": [ + "host" + ] + }, + "KinesisServer": { + "type": "object", + "title": "KinesisDataStreamsServer", + "description": "Kinesis Data Streams Server", + "properties": { + "region": { + "type": "string", + "description": "AWS region.", + "examples": [ + "eu-west-1" + ] + }, + "format": { + "type": "string", + "description": "The format of the record", + "examples": [ + "json", + "avro", + "protobuf" + ] + } + } + }, + "LocalServer": { + "type": "object", + "title": "LocalServer", + "properties": { + "path": { + "type": "string", + "description": "The relative or absolute path to the data file(s).", + "examples": [ + "./folder/data.parquet", + "./folder/*.parquet" + ] + }, + "format": { + "type": "string", + "description": "The format of the file(s)", + "examples": [ + "json", + "parquet", + "delta", + "csv" + ] + } + }, + "required": [ + "path", + "format" + ] + }, + "MySqlServer": { + "type": "object", + "title": "MySqlServer", + "properties": { + "host": { + "type": "string", + "description": "The host of the MySql server." + }, + "port": { + "type": "integer", + "description": "The port of the MySql server." + }, + "database": { + "type": "string", + "description": "The name of the database." + } + }, + "required": [ + "host", + "port", + "database" + ] + }, + "OracleServer": { + "type": "object", + "title": "OracleServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the oracle server", + "examples": [ + "localhost" + ] + }, + "port": { + "type": "integer", + "description": "The port to the oracle server.", + "examples": [ + 1523 + ] + }, + "serviceName": { + "type": "string", + "description": "The name of the service.", + "examples": [ + "service" + ] + } + }, + "required": [ + "host", + "port", + "serviceName" + ] + }, + "PostgresServer": { + "type": "object", + "title": "PostgresServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the Postgres server" + }, + "port": { + "type": "integer", + "description": "The port to the Postgres server." + }, + "database": { + "type": "string", + "description": "The name of the database." + }, + "schema": { + "type": "string", + "description": "The name of the schema in the database." + } + }, + "required": [ + "host", + "port", + "database", + "schema" + ] + }, + "PrestoServer": { + "type": "object", + "title": "PrestoServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the Presto server", + "examples": [ + "localhost:8080" + ] + }, + "catalog": { + "type": "string", + "description": "The name of the catalog.", + "examples": [ + "postgres" + ] + }, + "schema": { + "type": "string", + "description": "The name of the schema.", + "examples": [ + "public" + ] + } + }, + "required": [ + "host" + ] + }, + "PubSubServer": { + "type": "object", + "title": "PubSubServer", + "properties": { + "project": { + "type": "string", + "description": "The GCP project name." + } + }, + "required": [ + "project" + ] + }, + "RedshiftServer": { + "type": "object", + "title": "RedshiftServer", + "properties": { + "host": { + "type": "string", + "description": "An optional string describing the server." + }, + "database": { + "type": "string", + "description": "The name of the database." + }, + "schema": { + "type": "string", + "description": "The name of the schema." + }, + "region": { + "type": "string", + "description": "AWS region of Redshift server.", + "examples": ["us-east-1"] + }, + "account": { + "type": "string", + "description": "The account used by the server." + } + }, + "required": [ + "database", + "schema" + ] + }, + "S3Server": { + "type": "object", + "title": "S3Server", + "properties": { + "location": { + "type": "string", + "format": "uri", + "description": "S3 URL, starting with `s3://`", + "examples": [ + "s3://datacontract-example-orders-latest/data/{model}/*.json" + ] + }, + "endpointUrl": { + "type": "string", + "format": "uri", + "description": "The server endpoint for S3-compatible servers.", + "examples": ["https://minio.example.com"] + }, + "format": { + "type": "string", + "examples": [ + "parquet", + "delta", + "json", + "csv" + ], + "description": "File format." + }, + "delimiter": { + "type": "string", + "examples": [ + "new_line", + "array" + ], + "description": "Only for format = json. How multiple json documents are delimited within one file" + } + }, + "required": [ + "location" + ] + }, + "SftpServer": { + "type": "object", + "title": "SftpServer", + "properties": { + "location": { + "type": "string", + "format": "uri", + "pattern": "^sftp://.*", + "description": "SFTP URL, starting with `sftp://`", + "examples": [ + "sftp://123.123.12.123/{model}/*.json" + ] + }, + "format": { + "type": "string", + "examples": [ + "parquet", + "delta", + "json", + "csv" + ], + "description": "File format." + }, + "delimiter": { + "type": "string", + "examples": [ + "new_line", + "array" + ], + "description": "Only for format = json. How multiple json documents are delimited within one file" + } + }, + "required": [ + "location" + ] + }, + "SnowflakeServer": { + "type": "object", + "title": "SnowflakeServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the Snowflake server" + }, + "port": { + "type": "integer", + "description": "The port to the Snowflake server." + }, + "account": { + "type": "string", + "description": "The Snowflake account used by the server." + }, + "database": { + "type": "string", + "description": "The name of the database." + }, + "schema": { + "type": "string", + "description": "The name of the schema." + }, + "warehouse": { + "type": "string", + "description": "The name of the cluster of resources that is a Snowflake virtual warehouse." + } + }, + "required": [ + "account", + "database", + "schema" + ] + }, + "SqlserverServer": { + "type": "object", + "title": "SqlserverServer", + "properties": { + "host": { + "type": "string", + "description": "The host to the database server", + "examples": [ + "localhost" + ] + }, + "port": { + "type": "integer", + "description": "The port to the database server.", + "default": 1433, + "examples": [ + 1433 + ] + }, + "database": { + "type": "string", + "description": "The name of the database.", + "examples": [ + "database" + ] + }, + "schema": { + "type": "string", + "description": "The name of the schema in the database.", + "examples": [ + "dbo" + ] + } + }, + "required": [ + "host", + "database", + "schema" + ] + }, + "SynapseServer": { + "type": "object", + "title": "SynapseServer", + "properties": { + "host": { + "type": "string", + "description": "The host of the Synapse server." + }, + "port": { + "type": "integer", + "description": "The port of the Synapse server." + }, + "database": { + "type": "string", + "description": "The name of the database." + } + }, + "required": [ + "host", + "port", + "database" + ] + }, + "TrinoServer": { + "type": "object", + "title": "TrinoServer", + "properties": { + "host": { + "type": "string", + "description": "The Trino host URL.", + "examples": [ + "localhost" + ] + }, + "port": { + "type": "integer", + "description": "The Trino port." + }, + "catalog": { + "type": "string", + "description": "The name of the catalog.", + "examples": [ + "hive" + ] + }, + "schema": { + "type": "string", + "description": "The name of the schema in the database.", + "examples": [ + "my_schema" + ] + } + }, + "required": [ + "host", + "port", + "catalog", + "schema" + ] + }, + "VerticaServer": { + "type": "object", + "title": "VerticaServer", + "properties": { + "host": { + "type": "string", + "description": "The host of the Vertica server." + }, + "port": { + "type": "integer", + "description": "The port of the Vertica server." + }, + "database": { + "type": "string", + "description": "The name of the database." + }, + "schema": { + "type": "string", + "description": "The name of the schema." + } + }, + "required": [ + "host", + "port", + "database", + "schema" + ] + } + }, + "SchemaElement": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "name": { + "type": "string", + "description": "Name of the element." + }, + "physicalType": { + "type": "string", + "description": "The physical element data type in the data source.", + "examples": ["table", "view", "topic", "file"] + }, + "description": { + "type": "string", + "description": "Description of the element." + }, + "businessName": { + "type": "string", + "description": "The business name of the element." + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" + }, + "tags": { + "$ref": "#/$defs/Tags" + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + } + } + }, + "SchemaObject": { + "type": "object", + "properties": { + "logicalType": { + "type": "string", + "description": "The logical element data type.", + "enum": ["object"] + }, + "physicalName": { + "type": "string", + "description": "Physical name.", + "examples": ["table_1_2_0"] + }, + "dataGranularityDescription": { + "type": "string", + "description": "Granular level of the data in the object.", + "examples": ["Aggregation by country"] + }, + "properties": { + "type": "array", + "description": "A list of properties for the object.", + "items": { + "$ref": "#/$defs/SchemaProperty" + } + }, + "relationships": { + "type": "array", + "description": "A list of relationships to other properties. Each relationship must have 'from', 'to' and optionally 'type' field.", + "items": { + "$ref": "#/$defs/RelationshipSchemaLevel" + } + }, + "quality": { + "$ref": "#/$defs/DataQualityChecks" + } + }, + "allOf": [ + { + "$ref": "#/$defs/SchemaElement" + } + ], + "required": ["name"], + "unevaluatedProperties": false + }, + "SchemaBaseProperty": { + "type": "object", + "properties": { + "primaryKey": { + "type": "boolean", + "description": "Boolean value specifying whether the element is primary or not. Default is false." + }, + "primaryKeyPosition": { + "type": "integer", + "default": -1, + "description": "If element is a primary key, the position of the primary key element. Starts from 1. Example of `account_id, name` being primary key columns, `account_id` has primaryKeyPosition 1 and `name` primaryKeyPosition 2. Default to -1." + }, + "logicalType": { + "type": "string", + "description": "The logical element data type.", + "enum": ["string", "date", "timestamp", "time", "number", "integer", "object", "array", "boolean"] + }, + "logicalTypeOptions": { + "type": "object", + "description": "Additional optional metadata to describe the logical type." + }, + "physicalType": { + "type": "string", + "description": "The physical element data type in the data source. For example, VARCHAR(2), DOUBLE, INT." + }, + "physicalName": { + "type": "string", + "description": "Physical name.", + "examples": ["col_str_a"] + }, + "required": { + "type": "boolean", + "default": false, + "description": "Indicates if the element may contain Null values; possible values are true and false. Default is false." + }, + "unique": { + "type": "boolean", + "default": false, + "description": "Indicates if the element contains unique values; possible values are true and false. Default is false." + }, + "partitioned": { + "type": "boolean", + "default": false, + "description": "Indicates if the element is partitioned; possible values are true and false." + }, + "partitionKeyPosition": { + "type": "integer", + "default": -1, + "description": "If element is used for partitioning, the position of the partition element. Starts from 1. Example of `country, year` being partition columns, `country` has partitionKeyPosition 1 and `year` partitionKeyPosition 2. Default to -1." + }, + "classification": { + "type": "string", + "description": "Can be anything, like confidential, restricted, and public to more advanced categorization. Some companies like PayPal, use data classification indicating the class of data in the element; expected values are 1, 2, 3, 4, or 5.", + "examples": ["confidential", "restricted", "public"] + }, + "encryptedName": { + "type": "string", + "description": "The element name within the dataset that contains the encrypted element value. For example, unencrypted element `email_address` might have an encryptedName of `email_address_encrypt`." + }, + "transformSourceObjects": { + "type": "array", + "description": "List of objects in the data source used in the transformation.", + "items": { + "type": "string" + } + }, + "transformLogic": { + "type": "string", + "description": "Logic used in the element transformation." + }, + "transformDescription": { + "type": "string", + "description": "Describes the transform logic in very simple terms." + }, + "examples": { + "type": "array", + "description": "List of sample element values.", + "items": { + "$ref": "#/$defs/AnyType" + } + }, + "criticalDataElement": { + "type": "boolean", + "default": false, + "description": "True or false indicator; If element is considered a critical data element (CDE) then true else false." + }, + "relationships": { + "type": "array", + "description": "A list of relationships to other properties. When defined at property level, the 'from' field is implicit and should not be specified.", + "items": { + "$ref": "#/$defs/RelationshipPropertyLevel" + } + }, + "quality": { + "$ref": "#/$defs/DataQualityChecks" + } + }, + "allOf": [ + { + "$ref": "#/$defs/SchemaElement" + }, + { + "if": { + "properties": { + "logicalType": { + "const": "string" + } + } + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "minLength": { + "type": "integer", + "minimum": 0, + "description": "Minimum length of the string." + }, + "maxLength": { + "type": "integer", + "minimum": 0, + "description": "Maximum length of the string." + }, + "pattern": { + "type": "string", + "description": "Regular expression pattern to define valid value. Follows regular expression syntax from ECMA-262 (https://262.ecma-international.org/5.1/#sec-15.10.1)." + }, + "format": { + "type": "string", + "examples": ["password", "byte", "binary", "email", "uuid", "uri", "hostname", "ipv4", "ipv6"], + "description": "Provides extra context about what format the string follows." + } + } + } + } + } + }, + { + "if": { + "properties": { + "logicalType": { + "const": "date" + } + } + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "format": { + "type": "string", + "examples": ["yyyy-MM-dd", "yyyy-MM-dd HH:mm:ss", "HH:mm:ss"], + "description": "Format of the date. Follows the format as prescribed by [JDK DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html). For example, format 'yyyy-MM-dd'." + }, + "exclusiveMaximum": { + "type": "string", + "description": "All values must be strictly less than this value (values < exclusiveMaximum)." + }, + "maximum": { + "type": "string", + "description": "All date values are less than or equal to this value (values <= maximum)." + }, + "exclusiveMinimum": { + "type": "string", + "description": "All values must be strictly greater than this value (values > exclusiveMinimum)." + }, + "minimum": { + "type": "string", + "description": "All date values are greater than or equal to this value (values >= minimum)." + } + } + } + } + } + }, + { + "if": { + "anyOf": [ + { + "properties": { + "logicalType": { + "const": "timestamp" + } + } + }, + { + "properties": { + "logicalType": { + "const": "time" + } + } + } + ] + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "format": { + "type": "string", + "examples": ["yyyy-MM-dd", "yyyy-MM-dd HH:mm:ss", "HH:mm:ss"], + "description": "Format of the date. Follows the format as prescribed by [JDK DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html). For example, format 'yyyy-MM-dd'." + }, + "exclusiveMaximum": { + "type": "string", + "description": "All values must be strictly less than this value (values < exclusiveMaximum)." + }, + "maximum": { + "type": "string", + "description": "All date values are less than or equal to this value (values <= maximum)." + }, + "exclusiveMinimum": { + "type": "string", + "description": "All values must be strictly greater than this value (values > exclusiveMinimum)." + }, + "minimum": { + "type": "string", + "description": "All date values are greater than or equal to this value (values >= minimum)." + }, + "timezone": { + "type": "boolean", + "description": "Whether the timestamp defines the timezone or not. If true, timezone information is included in the timestamp." + }, + "defaultTimezone": { + "type": "string", + "description": "The default timezone of the timestamp. If timezone is not defined, the default timezone UTC is used.", + "default": "Etc/UTC" + } + } + } + } + } + }, + { + "if": { + "anyOf": [ + { + "properties": { + "logicalType": { + "const": "integer" + } + } + } + ] + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "multipleOf": { + "type": "number", + "exclusiveMinimum": 0, + "description": "Values must be multiples of this number. For example, multiple of 5 has valid values 0, 5, 10, -5." + }, + "maximum": { + "type": "number", + "description": "All values are less than or equal to this value (values <= maximum)." + }, + "exclusiveMaximum": { + "type": "number", + "description": "All values must be strictly less than this value (values < exclusiveMaximum)." + }, + "minimum": { + "type": "number", + "description": "All values are greater than or equal to this value (values >= minimum)." + }, + "exclusiveMinimum": { + "type": "number", + "description": "All values must be strictly greater than this value (values > exclusiveMinimum)." + }, + "format": { + "type": "string", + "default": "i32", + "description": "Format of the value in terms of how many bits of space it can use and whether it is signed or unsigned (follows the Rust integer types).", + "enum": ["i8", "i16", "i32", "i64", "i128", "u8", "u16", "u32", "u64", "u128"] + } + } + } + } + } + }, + { + "if": { + "anyOf": [ + { + "properties": { + "logicalType": { + "const": "number" + } + } + } + ] + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "multipleOf": { + "type": "number", + "exclusiveMinimum": 0, + "description": "Values must be multiples of this number. For example, multiple of 5 has valid values 0, 5, 10, -5." + }, + "maximum": { + "type": "number", + "description": "All values are less than or equal to this value (values <= maximum)." + }, + "exclusiveMaximum": { + "type": "number", + "description": "All values must be strictly less than this value (values < exclusiveMaximum)." + }, + "minimum": { + "type": "number", + "description": "All values are greater than or equal to this value (values >= minimum)." + }, + "exclusiveMinimum": { + "type": "number", + "description": "All values must be strictly greater than this value (values > exclusiveMinimum)." + }, + "format": { + "type": "string", + "default": "i32", + "description": "Format of the value in terms of how many bits of space it can use (follows the Rust float types).", + "enum": ["f32", "f64"] + } + } + } + } + } + }, + { + "if": { + "properties": { + "logicalType": { + "const": "object" + } + } + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "maxProperties": { + "type": "integer", + "minimum": 0, + "description": "Maximum number of properties." + }, + "minProperties": { + "type": "integer", + "minimum": 0, + "default": 0, + "description": "Minimum number of properties." + }, + "required": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1, + "uniqueItems": true, + "description": "Property names that are required to exist in the object." + } + } + }, + "properties": { + "type": "array", + "description": "A list of properties for the object.", + "items": { + "$ref": "#/$defs/SchemaProperty" + } + } + } + } + }, + { + "if": { + "properties": { + "logicalType": { + "const": "array" + } + } + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "maxItems": { + "type": "integer", + "minimum": 0, + "description": "Maximum number of items." + }, + "minItems": { + "type": "integer", + "minimum": 0, + "default": 0, + "description": "Minimum number of items" + }, + "uniqueItems": { + "type": "boolean", + "default": false, + "description": "If set to true, all items in the array are unique." + } + } + }, + "items": { + "$ref": "#/$defs/SchemaItemProperty", + "description": "List of items in an array (only applicable when `logicalType: array`)." + } + } + } + } + ], + "unevaluatedProperties": false + }, + "SchemaProperty": { + "type": "object", + "$ref": "#/$defs/SchemaBaseProperty", + "required": ["name"], + "unevaluatedProperties": false + }, + "SchemaItemProperty": { + "type": "object", + "$ref": "#/$defs/SchemaBaseProperty", + "properties": { + "properties": { + "type": "array", + "description": "A list of properties for the object.", + "items": { + "$ref": "#/$defs/SchemaProperty" + } + } + }, + "unevaluatedProperties": false + }, + "Tags": { + "type": "array", + "description": "A list of tags that may be assigned to the elements (object or property); the tags keyword may appear at any level. Tags may be used to better categorize an element. For example, `finance`, `sensitive`, `employee_record`.", + "examples": ["finance", "sensitive", "employee_record"], + "items": { + "type": "string" + } + }, + "DataQuality": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" + }, + "businessImpact": { + "type": "string", + "description": "Consequences of the rule failure.", + "examples": ["operational", "regulatory"] + }, + "customProperties": { + "type": "array", + "description": "Additional properties required for rule execution.", + "items": { + "$ref": "#/$defs/CustomProperty" + } + }, + "description": { + "type": "string", + "description": "Describe the quality check to be completed." + }, + "dimension": { + "type": "string", + "description": "The key performance indicator (KPI) or dimension for data quality.", + "enum": ["accuracy", "completeness", "conformity", "consistency", "coverage", "timeliness", "uniqueness"] + }, + "method": { + "type": "string", + "examples": ["reconciliation"] + }, + "name": { + "type": "string", + "description": "Name of the data quality check." + }, + "schedule": { + "type": "string", + "description": "Rule execution schedule details.", + "examples": ["0 20 * * *"] + }, + "scheduler": { + "type": "string", + "description": "The name or type of scheduler used to start the data quality check.", + "examples": ["cron"] + }, + "severity": { + "type": "string", + "description": "The severance of the quality rule.", + "examples": ["info", "warning", "error"] + }, + "tags": { + "$ref": "#/$defs/Tags" + }, + "type": { + "type": "string", + "description": "The type of quality check. 'text' is human-readable text that describes the quality of the data. 'library' is a set of maintained predefined quality attributes such as row count or unique. 'sql' is an individual SQL query that returns a value that can be compared. 'custom' is quality attributes that are vendor-specific, such as Soda or Great Expectations.", + "enum": ["text", "library", "sql", "custom"], + "default": "library" + }, + "unit": { + "type": "string", + "description": "Unit the rule is using, popular values are `rows` or `percent`, but any value is allowed.", + "examples": ["rows", "percent"] + } + }, + "allOf": [ + { + "if": { + "anyOf": [ + { + "properties": { + "type": { + "const": "library" + } + }, + "required": ["type"] + }, + { + "properties": { + "metric": { + "type": "string" + } + }, + "required": ["metric"] + } + ] + }, + "then": { + "$ref": "#/$defs/DataQualityLibrary" + } + }, + { + "if": { + "properties": { + "type": { + "const": "sql" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/DataQualitySql" + } + }, + { + "if": { + "properties": { + "type": { + "const": "custom" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/DataQualityCustom" + } + } + ], + "unevaluatedProperties": false + }, + "DataQualityChecks": { + "type": "array", + "description": "Data quality rules with all the relevant information for rule setup and execution.", + "items": { + "$ref": "#/$defs/DataQuality" + } + }, + "DataQualityOperators": { + "type": "object", + "description": "Common comparison operators for data quality checks.", + "oneOf": [ + { + "properties": { + "mustBe": { + "description": "Must be equal to the value to be valid. When using numbers, it is equivalent to '='." + } + }, + "required": ["mustBe"] + }, + { + "properties": { + "mustNotBe": { + "description": "Must not be equal to the value to be valid. When using numbers, it is equivalent to '!='." + } + }, + "required": ["mustNotBe"] + }, + { + "properties": { + "mustBeGreaterThan": { + "type": "number", + "description": "Must be greater than the value to be valid. It is equivalent to '>'." + } + }, + "required": ["mustBeGreaterThan"] + }, + { + "properties": { + "mustBeGreaterOrEqualTo": { + "type": "number", + "description": "Must be greater than or equal to the value to be valid. It is equivalent to '>='." + } + }, + "required": ["mustBeGreaterOrEqualTo"] + }, + { + "properties": { + "mustBeLessThan": { + "type": "number", + "description": "Must be less than the value to be valid. It is equivalent to '<'." + } + }, + "required": ["mustBeLessThan"] + }, + { + "properties": { + "mustBeLessOrEqualTo": { + "type": "number", + "description": "Must be less than or equal to the value to be valid. It is equivalent to '<='." + } + }, + "required": ["mustBeLessOrEqualTo"] + }, + { + "properties": { + "mustBeBetween": { + "type": "array", + "description": "Must be between the two numbers to be valid. Smallest number first in the array.", + "minItems": 2, + "maxItems": 2, + "uniqueItems": true, + "items": { + "type": "number" + } + } + }, + "required": ["mustBeBetween"] + }, + { + "properties": { + "mustNotBeBetween": { + "type": "array", + "description": "Must not be between the two numbers to be valid. Smallest number first in the array.", + "minItems": 2, + "maxItems": 2, + "uniqueItems": true, + "items": { + "type": "number" + } + } + }, + "required": ["mustNotBeBetween"] + } + ] + }, + "DataQualityLibrary": { + "type": "object", + "allOf": [ + { + "$ref": "#/$defs/DataQualityOperators" + } + ], + "properties": { + "metric": { + "type": "string", + "description": "Define a data quality check based on the predefined metrics as per ODCS.", + "enum": ["nullValues", "missingValues", "invalidValues", "duplicateValues", "rowCount"] + }, + "rule": { + "type": "string", + "deprecated": true, + "description": "Use metric instead" + }, + "arguments": { + "type": "object", + "description": "Additional arguments for the metric, if needed." + } + }, + "required": ["metric"] + }, + "DataQualitySql": { + "type": "object", + "allOf": [ + { + "$ref": "#/$defs/DataQualityOperators" + } + ], + "properties": { + "query": { + "type": "string", + "description": "Query string that adheres to the dialect of the provided server.", + "examples": ["SELECT COUNT(*) FROM ${table} WHERE ${column} IS NOT NULL"] + } + }, + "required": ["query"] + }, + "DataQualityCustom": { + "type": "object", + "properties": { + "engine": { + "type": "string", + "description": "Name of the engine which executes the data quality checks.", + "examples": ["soda", "great-expectations", "monte-carlo", "dbt"] + }, + "implementation": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "object" + } + ] + } + }, + "required": ["engine", "implementation"] + }, + "AuthoritativeDefinitions": { + "type": "array", + "description": "List of links to sources that provide more details on the dataset; examples would be a link to an external definition, a training video, a git repo, data catalog, or another tool. Authoritative definitions follow the same structure in the standard.", + "items": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "url": { + "type": "string", + "description": "URL to the authority." + }, + "type": { + "type": "string", + "description": "Type of definition for authority: v2.3 adds standard values: `businessDefinition`, `transformationImplementation`, `videoTutorial`, `tutorial`, and `implementation`.", + "examples": ["businessDefinition", "transformationImplementation", "videoTutorial", "tutorial", "implementation"] + }, + "description": { + "type": "string", + "description": "Description of the authoritative definition for humans." + } + }, + "required": ["url", "type"], + "additionalProperties": false + } + }, + "Support": { + "type": "array", + "description": "Top level for support channels.", + "items": { + "$ref": "#/$defs/SupportItem" + } + }, + "SupportItem": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "channel": { + "type": "string", + "description": "Channel name or identifier." + }, + "url": { + "type": "string", + "description": "Access URL using normal [URL scheme](https://en.wikipedia.org/wiki/URL#Syntax) (https, mailto, etc.)." + }, + "description": { + "type": "string", + "description": "Description of the channel, free text." + }, + "tool": { + "type": "string", + "description": "Name of the tool, value can be `email`, `slack`, `teams`, `discord`, `ticket`, `googlechat`, or `other`.", + "examples": ["email", "slack", "teams", "discord", "ticket", "googlechat", "other"] + }, + "scope": { + "type": "string", + "description": "Scope can be: `interactive`, `announcements`, `issues`, `notifications`.", + "examples": ["interactive", "announcements", "issues", "notifications"] + }, + "invitationUrl": { + "type": "string", + "description": "Some tools uses invitation URL for requesting or subscribing. Follows the [URL scheme](https://en.wikipedia.org/wiki/URL#Syntax)." + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + } + }, + "required": ["channel"], + "additionalProperties": false + }, + "Pricing": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "priceAmount": { + "type": "number", + "description": "Subscription price per unit of measure in `priceUnit`." + }, + "priceCurrency": { + "type": "string", + "description": "Currency of the subscription price in `price.priceAmount`." + }, + "priceUnit": { + "type": "string", + "description": "The unit of measure for calculating cost. Examples megabyte, gigabyte." + } + }, + "additionalProperties": false + }, + "TeamMember": { + "type": "object", + "description": "Team member information.", + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "username": { + "type": "string", + "description": "The user's username or email." + }, + "name": { + "type": "string", + "description": "The user's name." + }, + "description": { + "type": "string", + "description": "The user's description." + }, + "role": { + "type": "string", + "description": "The user's job role; Examples might be owner, data steward. There is no limit on the role." + }, + "dateIn": { + "type": "string", + "format": "date", + "description": "The date when the user joined the team." + }, + "dateOut": { + "type": "string", + "format": "date", + "description": "The date when the user ceased to be part of the team." + }, + "replacedByUsername": { + "type": "string", + "description": "The username of the user who replaced the previous user." + }, + "tags": { + "$ref": "#/$defs/Tags" + }, + "customProperties": { + "type": "array", + "description": "Custom properties block.", + "items": { + "$ref": "#/$defs/CustomProperty" + } + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" + } + }, + "required": ["username"] + }, + "Team": { + "type": "object", + "description": "Team information.", + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "name": { + "type": "string", + "description": "Team name." + }, + "description": { + "type": "string", + "description": "Team description." + }, + "members": { + "type": "array", + "description": "List of members.", + "items": { + "$ref": "#/$defs/TeamMember" + } + }, + "tags": { + "$ref": "#/$defs/Tags" + }, + "customProperties": { + "type": "array", + "description": "Custom properties block.", + "items": { + "$ref": "#/$defs/CustomProperty" + } + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" + } + } + }, + "Role": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "role": { + "type": "string", + "description": "Name of the IAM role that provides access to the dataset." + }, + "description": { + "type": "string", + "description": "Description of the IAM role and its permissions." + }, + "access": { + "type": "string", + "description": "The type of access provided by the IAM role." + }, + "firstLevelApprovers": { + "type": "string", + "description": "The name(s) of the first-level approver(s) of the role." + }, + "secondLevelApprovers": { + "type": "string", + "description": "The name(s) of the second-level approver(s) of the role." + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + } + }, + "required": ["role"], + "additionalProperties": false + }, + "ServiceLevelAgreementProperty": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "property": { + "type": "string", + "description": "Specific property in SLA, check the periodic table. May requires units (more details to come)." + }, + "value": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "number" + }, + { + "type": "integer" + }, + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "description": "Agreement value. The label will change based on the property itself." + }, + "valueExt": { + "$ref": "#/$defs/AnyNonCollectionType", + "description": "Extended agreement value. The label will change based on the property itself." + }, + "unit": { + "type": "string", + "description": "**d**, day, days for days; **y**, yr, years for years, etc. Units use the ISO standard." + }, + "element": { + "type": "string", + "description": "Element(s) to check on. Multiple elements should be extremely rare and, if so, separated by commas." + }, + "driver": { + "type": "string", + "description": "Describes the importance of the SLA from the list of: `regulatory`, `analytics`, or `operational`.", + "examples": ["regulatory", "analytics", "operational"] + }, + "description": { + "type": "string", + "description": "Description of the SLA for humans.", + "examples": ["99.9% of the time, data is available by 6 AM UTC"] + }, + "scheduler": { + "type": "string", + "description": "Name of the scheduler, can be cron or any tool your organization support.", + "examples": ["cron"] + }, + "schedule": { + "type": "string", + "description": "Configuration information for the scheduling tool, for cron a possible value is 0 20 * * *.", + "examples": ["0 20 * * *"] + } + }, + "required": ["property", "value"], + "additionalProperties": false + }, + "CustomProperties": { + "type": "array", + "description": "A list of key/value pairs for custom properties.", + "items": { + "$ref": "#/$defs/CustomProperty" + } + }, + "CustomProperty": { + "type": "object", + "properties": { + "id": { + "$ref": "#/$defs/StableId" + }, + "property": { + "type": "string", + "description": "The name of the key. Names should be in camel case–the same as if they were permanent properties in the contract." + }, + "value": { + "$ref": "#/$defs/AnyType", + "description": "The value of the key." + }, + "description": { + "type": "string", + "description": "Description of the custom property." + } + }, + "required": ["property", "value"], + "additionalProperties": false + }, + "AnyType": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "number" + }, + { + "type": "integer" + }, + { + "type": "boolean" + }, + { + "type": "null" + }, + { + "type": "array" + }, + { + "type": "object" + } + ] + }, + "AnyNonCollectionType": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "number" + }, + { + "type": "integer" + }, + { + "type": "boolean" + }, + { + "type": "null" + } + ] + }, + "RelationshipBase": { + "type": "object", + "description": "Base definition for relationships between properties, typically for foreign key constraints.", + "properties": { + "type": { + "type": "string", + "description": "The type of relationship. Defaults to 'foreignKey'.", + "default": "foreignKey", + "enum": ["foreignKey"] + }, + "from": { + "oneOf": [ + { + "anyOf": [ + { + "$ref": "#/$defs/ShorthandReference" + }, + { + "$ref": "#/$defs/FullyQualifiedReference" + } + ], + "description": "Source property reference using fully qualified or shorthand notation." + }, + { + "type": "array", + "description": "Array of source properties for composite keys.", + "items": { + "anyOf": [ + { + "$ref": "#/$defs/ShorthandReference" + }, + { + "$ref": "#/$defs/FullyQualifiedReference" + } + ] + }, + "minItems": 1 + } + ], + "description": "Source property or properties." + }, + "to": { + "oneOf": [ + { + "anyOf": [ + { + "$ref": "#/$defs/ShorthandReference" + }, + { + "$ref": "#/$defs/FullyQualifiedReference" + } + ], + "description": "Target property reference using fully qualified or shorthand notation." + }, + { + "type": "array", + "description": "Array of target properties for composite keys.", + "items": { + "anyOf": [ + { + "$ref": "#/$defs/ShorthandReference" + }, + { + "$ref": "#/$defs/FullyQualifiedReference" + } + ] + }, + "minItems": 1 + } + ], + "description": "Target property or properties to reference." + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + } + } + }, + "RelationshipSchemaLevel": { + "type": "object", + "description": "Relationship definition at schema level, requiring both 'from' and 'to' fields with matching types.", + "allOf": [ + { + "$ref": "#/$defs/RelationshipBase" + }, + { + "required": ["from", "to"] + }, + { + "oneOf": [ + { + "description": "Single-column relationship - both fields must be strings", + "properties": { + "from": { + "type": "string" + }, + "to": { + "type": "string" + } + } + }, + { + "description": "Composite key relationship - both fields must be arrays with matching lengths", + "properties": { + "from": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1 + }, + "to": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1 + } + } + } + ] + } + ], + "unevaluatedProperties": false + }, + "RelationshipPropertyLevel": { + "type": "object", + "description": "Relationship definition at property level, where 'from' is implicitly the current property.", + "allOf": [ + { + "$ref": "#/$defs/RelationshipBase" + }, + { + "type": "object", + "required": ["to"] + }, + { + "not": { + "required": ["from"] + }, + "description": "The 'from' field must not be specified at property level as it is implicitly derived from the property context" + } + ], + "unevaluatedProperties": false + }, + "Relationship": { + "description": "Compatibility wrapper for relationship definitions.", + "oneOf": [ + { + "$ref": "#/$defs/RelationshipSchemaLevel" + }, + { + "$ref": "#/$defs/RelationshipPropertyLevel" + } + ] + } + } +} From d4c2fb733abf2a631ef0c9fa48b94062de06570e Mon Sep 17 00:00:00 2001 From: Data Contract Enthusiast <80169154+DatSciMeMa@users.noreply.github.com> Date: Tue, 5 May 2026 18:27:00 +0200 Subject: [PATCH 05/29] Update custom-other-properties.md Avis was supposed to be under the table. --- docs/custom-other-properties.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/custom-other-properties.md b/docs/custom-other-properties.md index a3ea323..4a312dd 100644 --- a/docs/custom-other-properties.md +++ b/docs/custom-other-properties.md @@ -39,6 +39,7 @@ customProperties: | customProperties.property | string | Property | No | The name of the key. Names should be in camel case–the same as if they were permanent properties in the contract. | | customProperties.value | any | Value | No | The value of the key. It can be an array. | | customProperties.description | string | Description | No | Description for humans. | + Avis: With version 3.2.0 the Description of 'customProperties.property' will be updated to "The name of the key. Could be in any notation. If this field is used for referencing it should be in camel case–the same as if they were permanent properties in the contract. Note that since ODCS version 3.1 the field 'customProperties.id' should be used for referencing purposes. In this case the name of the key could be human-readable and self-explanatory to the greatest extent." ## Authoritative Definitions From 7539af48e853c7ab1e0a170432a4a06a4d72c80f Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Sun, 24 May 2026 17:02:12 +0200 Subject: [PATCH 06/29] docs(vendors): add FLUID Forge CLI Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com> --- vendors.md | 1 + 1 file changed, 1 insertion(+) diff --git a/vendors.md b/vendors.md index 23a2ea5..7c0519a 100644 --- a/vendors.md +++ b/vendors.md @@ -15,6 +15,7 @@ A non-exhaustive, alphabetical list of organizations offering solutions natively * [Data Contract Playground](https://data-catering.github.io/data-contract-playground/) - Playground site for creating, exporting, and validating data contracts. * [DataVow](https://github.com/ludovicschmetz-stack/datavow) - Open-source CLI for ODCS v3.1 data contract enforcement with DuckDB validation, dbt test sync, GitHub Action, and Vow Score reporting * [DQC.ai | DQ PLATFORM](https://www.dqc.ai/dqc-platform) - [Enhancing Data Quality with ODCS: A Standard Ensured by the DQ Platform](https://www.dqc.ai/post/enhancing-data-quality-with-odcs-a-standard-ensured-by-the-dq-platform). +* [FLUID Forge](https://github.com/Agenticstiger/forge-cli) - Open-source Python CLI for authoring, planning, and applying data contracts. Bidirectional ODCS v3.1.0 import/export with lossless round-trip, Bitol ODPS v1.0.0 product wrapper with per-port contractId linking, native cloud apply (AWS Glue, BigQuery, Snowflake), and Data Mesh Manager publish. * [IBM](https://www.ibm.com/) - Supports [data contracts](https://www.ibm.com/docs/en/announcements/watsonxdata-intelligence-v23-introduces-support-openlineage-natural-language-sql-functionality-open-data-contracts), aligned with the Open Data Contract Standard, via its data & AI platform, watsonx * [jgp.ai Cloud Services](https://medium.com/data-mesh-learning/so-you-want-to-work-with-data-contracts-and-data-products-03e86f099710) - A set of services to create, archive, validate, and operationalize data contracts. * [Meta Analysis](https://www.meta-analysis.fr/en/home/) - Governance repository & Data Catalog, supports ODCS via a [flexible metamodel and open API](https://www.meta-analysis.fr/en/article-en/open-data-contract-standard-adoption/). From c4fd50f64b9ccd97d3124c41e44d63127e9c5b2e Mon Sep 17 00:00:00 2001 From: "Dr. Simon Harrer" Date: Tue, 2 Jun 2026 22:16:57 +0200 Subject: [PATCH 07/29] Update vendors.md --- vendors.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/vendors.md b/vendors.md index 7c0519a..9fa97ce 100644 --- a/vendors.md +++ b/vendors.md @@ -10,11 +10,12 @@ A non-exhaustive, alphabetical list of organizations offering solutions natively * [ContractGate](https://github.com/nightmoose/contractgate) - Real-time semantic contract enforcement gateway. Enforces rich ODCS-aligned contracts (ontology, glossary, patterns, enums, computed metrics) at ingestion (<50µs p99). Rust engine + hosted SaaS platform. Patent pending. [Platform](https://app.datacontractgate.com/) * [Databricks Ontos](https://marketplace.databricks.com/details/8c582cfa-4c6b-4fdc-93a2-f63b35d93906/Databricks_Ontos) - Governance and metadata management application using ODCS and ODPS as the core standards for data contracts and data products * [Data Caterer](https://data.catering/latest/docs/guide/data-source/metadata/open-data-contract-standard/) - Test data management tool using data contracts as a metadata source -* [Data Contract CLI](https://cli.datacontract.com) - Open Source tooling around data contracts. -* [Data Contract Manager](https://datacontract-manager.com) - Professional data contract management tool with Data Marketplace, Access Management, and Data Governance AI. +* [Data Contract CLI](https://cli.datacontract.com) - Open Source tooling for ODCS data contracts. Supports data contract linting, testing against all popular databases, and has imports and exports for all popular schema formats and html. +* [Data Contract Editor](https://editor.datacontract.com) - Open Source editor for ODCS data contracts. Integrates with Data Contract CLI. * [Data Contract Playground](https://data-catering.github.io/data-contract-playground/) - Playground site for creating, exporting, and validating data contracts. * [DataVow](https://github.com/ludovicschmetz-stack/datavow) - Open-source CLI for ODCS v3.1 data contract enforcement with DuckDB validation, dbt test sync, GitHub Action, and Vow Score reporting * [DQC.ai | DQ PLATFORM](https://www.dqc.ai/dqc-platform) - [Enhancing Data Quality with ODCS: A Standard Ensured by the DQ Platform](https://www.dqc.ai/post/enhancing-data-quality-with-odcs-a-standard-ensured-by-the-dq-platform). +* [Entropy Data](https://www.entropy-data.com) - Data Product Marketplace built on Data Contracts. Allows contract-first development of data products. * [FLUID Forge](https://github.com/Agenticstiger/forge-cli) - Open-source Python CLI for authoring, planning, and applying data contracts. Bidirectional ODCS v3.1.0 import/export with lossless round-trip, Bitol ODPS v1.0.0 product wrapper with per-port contractId linking, native cloud apply (AWS Glue, BigQuery, Snowflake), and Data Mesh Manager publish. * [IBM](https://www.ibm.com/) - Supports [data contracts](https://www.ibm.com/docs/en/announcements/watsonxdata-intelligence-v23-introduces-support-openlineage-natural-language-sql-functionality-open-data-contracts), aligned with the Open Data Contract Standard, via its data & AI platform, watsonx * [jgp.ai Cloud Services](https://medium.com/data-mesh-learning/so-you-want-to-work-with-data-contracts-and-data-products-03e86f099710) - A set of services to create, archive, validate, and operationalize data contracts. From e62cebe64a1d43cfedf76faaaafefe2d58d8a573 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Wed, 24 Jun 2026 09:18:19 +1000 Subject: [PATCH 08/29] fix: use #/ in external shorthand reference example (#287) Aligns the external shorthand reference example with the fully-qualified example above it (both use '#/'). Same fix as #221 (merged to dev), applied here to the dev-v3.2.0 line. Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: FriedrichBu Co-authored-by: Claude Opus 4.8 (1M context) --- docs/references.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/references.md b/docs/references.md index 28c2080..67adcc7 100644 --- a/docs/references.md +++ b/docs/references.md @@ -344,7 +344,7 @@ schema: value: "Externally referenced contract (fully qualified)" # To external contract (shorthand) - - to: https://example.com/data-contract-v1.yaml#profiles.user_id + - to: https://example.com/data-contract-v1.yaml#/profiles.user_id customProperties: - property: description value: "Externally referenced contract (shorthand)" From 8ac22e998a671210881d62e851db73a78e804c70 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Wed, 24 Jun 2026 09:22:52 +1000 Subject: [PATCH 09/29] docs: add Collate (OpenMetadata) to vendors list (#286) Adds a single vendor entry for Collate, the managed cloud service built on the open-source OpenMetadata platform, with direct ODCS documentation links for both. Reconstructed from #232 against dev-v3.2.0. Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: PubChimps Co-authored-by: Claude Opus 4.8 (1M context) --- vendors.md | 1 + 1 file changed, 1 insertion(+) diff --git a/vendors.md b/vendors.md index 185fae4..b110b07 100644 --- a/vendors.md +++ b/vendors.md @@ -12,6 +12,7 @@ Vendors who natively support ODCS (Open Data Contract Standard). A non-exhaustive, alphabetical list of organizations offering solutions natively compatible with ODCS, such as data catalogs, data quality platforms, security tools, and more. * [Actian Data Intelligence Platform](https://www.actian.com/data-intelligence/platform/) - A cloud-native, AI-ready solution that helps organizations discover, understand, trust, and activate their enterprise data assets at scale through intelligent metadata, governance, cataloging, and data marketplace capabilities. +* [Collate](https://docs.getcollate.io/api-reference/data-contracts/odcs#import-from-odcs) - Managed cloud service built on the open-source [OpenMetadata](https://docs.open-metadata.org/latest/api-reference/data-contracts/odcs#import-from-odcs) platform for unified metadata management, data discovery, observability, and governance. * [ContractGate](https://github.com/nightmoose/contractgate) - Real-time semantic contract enforcement gateway. Enforces rich ODCS-aligned contracts (ontology, glossary, patterns, enums, computed metrics) at ingestion (<50µs p99). Rust engine + hosted SaaS platform. Patent pending. [Platform](https://app.datacontractgate.com/) * [Databricks Ontos](https://marketplace.databricks.com/details/8c582cfa-4c6b-4fdc-93a2-f63b35d93906/Databricks_Ontos) - Governance and metadata management application using ODCS and ODPS as the core standards for data contracts and data products * [Data Caterer](https://data.catering/latest/docs/guide/data-source/metadata/open-data-contract-standard/) - Test data management tool using data contracts as a metadata source From e5589680507a6a1c5308ff39b19774d646e3bbd6 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Wed, 24 Jun 2026 09:23:12 +1000 Subject: [PATCH 10/29] fix: accept .yml extension in FullyQualifiedReference pattern (#285) Allow both .yaml and .yml file extensions in external references (\.yaml# -> \.ya?ml#) in odcs-json-schema-latest.json. Clean reapplication of #220, which also injected a stray leading 'r' into the pattern, edited the frozen v3.1.0 schema, and reformatted the whole file. This applies only the intended change to the latest schema. Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: FriedrichBu Co-authored-by: Claude Opus 4.8 (1M context) --- schema/odcs-json-schema-latest.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 3644b89..5a2dba0 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -209,7 +209,7 @@ "FullyQualifiedReference": { "type": "string", "description": "Fully qualified notation using id fields (section/id/properties/id), optionally prefixed with external file reference", - "pattern": "^(?:(?:https?:\\/\\/)?[A-Za-z0-9._\\-\\/]+\\.yaml#)?\\/?[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+(?:\\/[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+)*$" + "pattern": "^(?:(?:https?:\\/\\/)?[A-Za-z0-9._\\-\\/]+\\.ya?ml#)?\\/?[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+(?:\\/[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+)*$" }, "StableId": { "type": "string", From d0cb739e019dd25821fe61cf226dfcfabd50975c Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Wed, 24 Jun 2026 10:06:10 +1000 Subject: [PATCH 11/29] fix: sync .yml extension into v3.2.0 rolling schema + changelog (#288) PR #285 merged a stale single-file revision (latest.json only) due to a force-push/merge race, dropping the rolling-minor sync. Per schema/README.md, the .yaml->.ya?ml relaxation must also live in the rolling v3.2.0 file so latest and v3.2.0 stay consistent. - schema/odcs-json-schema-v3.2.0.json: apply \.yaml# -> \.ya?ml# - schema/README.md: v3.2.0 snapshot changelog entry (2026-06-23) Frozen v3.1.0 schema intentionally untouched. Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- schema/README.md | 1 + schema/odcs-json-schema-v3.2.0.json | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/schema/README.md b/schema/README.md index f0a02cf..8cb3c0f 100644 --- a/schema/README.md +++ b/schema/README.md @@ -96,6 +96,7 @@ Concise summary of what changed between each retained dated snapshot. | Date | Notes | |------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 2026-04-29 | First v3.2.0 snapshot. `apiVersion` default raised to `v3.2.0`. Adds RFC 0033 (Enum) — new `enum` array on schema properties and new `EnumValue` `$def`. Adds RFC 0030 (Maps) — new `map` value for `logicalType` and companion `map` block. | +| 2026-06-23 | `FullyQualifiedReference` external-file pattern relaxed to accept the `.yml` extension in addition to `.yaml`. | ## SchemaStore registration diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index 3644b89..5a2dba0 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -209,7 +209,7 @@ "FullyQualifiedReference": { "type": "string", "description": "Fully qualified notation using id fields (section/id/properties/id), optionally prefixed with external file reference", - "pattern": "^(?:(?:https?:\\/\\/)?[A-Za-z0-9._\\-\\/]+\\.yaml#)?\\/?[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+(?:\\/[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+)*$" + "pattern": "^(?:(?:https?:\\/\\/)?[A-Za-z0-9._\\-\\/]+\\.ya?ml#)?\\/?[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+(?:\\/[A-Za-z_][A-Za-z0-9_]*\\/[A-Za-z0-9_-]+)*$" }, "StableId": { "type": "string", From e2cd39ff19a54b4729a84c0bfd3c4483bb2ef8b0 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Wed, 24 Jun 2026 11:27:44 +1000 Subject: [PATCH 12/29] docs: document logicalType at the object level (#289) logicalType is valid on schema objects (the JSON schema allows it on SchemaObject, and the examples use logicalType: object), but the schema docs only listed it under 'Applicable to Properties'. Move it to the shared 'Applicable to Elements (either Objects or Properties)' table and note its object-level usage. logicalTypeOptions, items, and map remain property-specific. Closes #228. Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- docs/schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/schema.md b/docs/schema.md index de32344..bcaea4b 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -161,6 +161,7 @@ schema: | name | string | Name | Yes | Name of the element. | | physicalName | string | Physical Name | No | Physical name. | | physicalType | string | Physical Type | No | The physical element data type in the data source. For objects: `table`, `view`, `topic`, `file`. For properties: `VARCHAR(2)`, `DOUBLE`, `INT`, etc. | +| logicalType | string | Logical Type | No | The logical data type of the element. One of `string`, `date`, `timestamp`, `time`, `number`, `integer`, `object`, `array`, `boolean`, or `map`. At the object level this is typically `object` (or `array` for an array of objects); the data-type keywords above apply to properties. | | quality | array | Quality | No | List of data quality attributes. | | synonyms | array | Synonyms | No | A list of alternative names for the element (object or property), helping catalogs, AI/LLM tools, and natural language interfaces resolve business vocabulary to the underlying object. See [Synonyms](#synonyms). | | authoritativeDefinitions | array | Authoritative Definitions | No | List of links to sources that provide more details on the element; examples would be a link to privacy statement, terms and conditions, license agreements, data catalog, or another tool. | @@ -186,7 +187,6 @@ Some keys are more applicable when the described property is a column. | enum | array | Enum | No | Enumeration of allowed values for this property. See [Enumerations](#enumerations). | | examples | array | Example Values | No | List of sample element values. | | items | object | Items | No | List of items in an array (only applicable when `logicalType: array`). | -| logicalType | string | Logical Type | No | The logical field datatype. One of `string`, `date`, `timestamp`, `time`, `number`, `integer`, `object`, `array`, `boolean`, or `map`. | | logicalTypeOptions | object | Logical Type Options | No | Additional optional metadata to describe the logical type. See [Logical Type Options](#logical-type-options) for more details about supported options for each `logicalType`. | | map | object | Map | No | Key/value definition (required when `logicalType: map`). See [Maps](#maps). | | partitioned | boolean | Partitioned | No | Indicates if the element is partitioned; possible values are true and false. | From 1a417fcf79678f43e2abce3fab3c32031f813459 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 14:16:31 -0400 Subject: [PATCH 13/29] feat(rfc-0043): add optional `encoding` field to server definitions (#293) Implements approved RFC-0043 (Physical Data Encoding) for ODCS v3.2.0. Adds an optional free-form `encoding` string to the server types that expose serialized payloads (Azure, Glue, Custom, Kafka, Kinesis, Local, S3, SFTP), alongside `format`/`delimiter`. Default UTF-8, non-breaking. - schema: encoding on 8 server defs in both latest + v3.2.0 (lockstep) - docs: encoding row in infrastructure-servers.md for those servers - example: docs/examples/server/s3-server-encoding.odcs.yaml (validated) - negative test: server-encoding-wrong-type.odcs.yaml (schema rejects) - CHANGELOG entry Approved RFC: https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0043-physical-data-encoding.md Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 3 + .../server/s3-server-encoding.odcs.yaml | 32 ++++++++ docs/infrastructure-servers.md | 8 ++ schema/odcs-json-schema-latest.json | 80 +++++++++++++++++++ schema/odcs-json-schema-v3.2.0.json | 80 +++++++++++++++++++ .../server-encoding-wrong-type.odcs.yaml | 23 ++++++ 6 files changed, 226 insertions(+) create mode 100644 docs/examples/server/s3-server-encoding.odcs.yaml create mode 100644 src/script/negative-tests/server-encoding-wrong-type.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d0fad2..730b390 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,9 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * New optional `synonyms` array on schema objects and properties, recording alternative names for catalogs, AI/LLM tools, and natural language interfaces. * Each `synonyms` entry is an object with a required `synonym` plus optional `id`, `description`, `locale` (BCP 47), `source`, `status`, and `customProperties`. * Allowed only on schema objects and properties; non-breaking, as `synonyms` is optional. +* **Adds** Physical data encoding ([RFC 0043](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0043-physical-data-encoding.md)): + * New optional `encoding` string field on server definitions that expose serialized payloads (Azure, Glue, Custom, Kafka, Kinesis, Local, S3, SFTP), declaring the expected character encoding of the data, e.g. `UTF-8`, `ISO-8859-1`, `ASCII`, `UTF-16`. + * Free-form string (no enum), default `UTF-8`; documents physical-payload encoding separately from the ODCS document encoding. Non-breaking, as `encoding` is optional. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/server/s3-server-encoding.odcs.yaml b/docs/examples/server/s3-server-encoding.odcs.yaml new file mode 100644 index 0000000..f46b15c --- /dev/null +++ b/docs/examples/server/s3-server-encoding.odcs.yaml @@ -0,0 +1,32 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0043: the optional `encoding` field on server definitions. +# The same logical schema is exposed through two S3 servers with different +# physical character encodings. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 7b3f0c2e-1d4a-4e6b-9f2a-0a1b2c3d4e5f +status: active +servers: + - server: modern + type: s3 + location: s3://modern-bucket/customers/*.csv + format: csv + encoding: UTF-8 + - server: legacy + type: s3 + location: s3://legacy-bucket/customers/*.csv + format: csv + encoding: ISO-8859-1 +schema: + - name: customers + physicalType: table + properties: + - name: customer_id + logicalType: string + required: true + - name: customer_name + logicalType: string diff --git a/docs/infrastructure-servers.md b/docs/infrastructure-servers.md index d4f4280..ad2823b 100644 --- a/docs/infrastructure-servers.md +++ b/docs/infrastructure-servers.md @@ -84,6 +84,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | Key | Type | UX Label | Required | Description | |-----------|--------|-----------|----------|-----------------------------------------------------------------------------------------------| | delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | format | string | Format | Yes | File format. | | location | string | Location | Yes | Fully qualified path to Azure Blob Storage or Azure Data Lake Storage (ADLS), supports globs. | @@ -165,6 +166,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge |----------|--------|----------|----------|------------------------------------------------| | account | string | Account | Yes | The AWS Glue account | | database | string | Database | Yes | The AWS Glue database name | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | format | string | Format | No | The format of the files | | location | string | Location | No | The AWS S3 path. Must be in the form of a URL. | @@ -202,6 +204,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | Key | Type | UX Label | Required | Description | | ------ | ------ | -------- | -------- | ------------------------------------------ | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | format | string | Format | No | The format of the messages. | | host | string | Host | Yes | The bootstrap server of the kafka cluster. | @@ -209,6 +212,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | Key | Type | UX Label | Required | Description | | ------ | ------ | -------- | -------- | ------------------------------------ | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | format | string | Format | No | The format of the record | | region | string | Region | No | AWS region. | | stream | string | Stream | Yes | The name of the Kinesis data stream. | @@ -217,6 +221,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | Key | Type | UX Label | Required | Description | | ------ | ------ | -------- | -------- | -------------------------------------------------- | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | format | string | Format | Yes | The format of the file(s) | | path | string | Path | Yes | The relative or absolute path to the data file(s). | @@ -282,6 +287,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | Key | Type | UX Label | Required | Description | | ----------- | ------ | ------------ | -------- | --------------------------------------------------------------------------------- | | delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | endpointUrl | string | Endpoint URL | No | The server endpoint for S3-compatible servers. | | format | string | Format | No | File format. | | location | string | Location | Yes | S3 URL, starting with `s3://` | @@ -293,6 +299,7 @@ Secure File Transfer Protocol (SFTP) is a network protocol that enables secure a | Key | Type | UX Label | Required | Description | | --------- | ------ | --------- | -------- | --------------------------------------------------------------------------------- | | delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | format | string | Format | No | File format. | | location | string | Location | Yes | SFTP URL, starting with `sftp://`. The URL should include the port number. | @@ -363,6 +370,7 @@ Actian Zen (formerly Btrieve, later named Pervasive PSQL until version 13) is an | database | string | Database | No | Name of the database. | | dataset | string | Dataset | No | Name of the dataset. | | delimiter | string | Delimiter | No | Delimiter. | +| encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | | endpointUrl | string | Endpoint URL | No | Server endpoint. | | format | string | Format | No | File format. | | host | string | Host | No | Host name or IP address. | diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 7b22e95..150ead2 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -782,6 +782,16 @@ ], "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "delimiter": { "type": "string", "examples": [ @@ -958,6 +968,16 @@ "json", "delta" ] + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } }, "required": [ @@ -1138,6 +1158,16 @@ "type": "string", "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "host": { "type": "string", "description": "Host name or IP address." @@ -1203,6 +1233,16 @@ "description": "The format of the messages.", "examples": ["json", "avro", "protobuf", "xml"], "default": "json" + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } }, "required": [ @@ -1229,6 +1269,16 @@ "avro", "protobuf" ] + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } } }, @@ -1253,6 +1303,16 @@ "delta", "csv" ] + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } }, "required": [ @@ -1445,6 +1505,16 @@ ], "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "delimiter": { "type": "string", "examples": [ @@ -1481,6 +1551,16 @@ ], "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "delimiter": { "type": "string", "examples": [ diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index 48d8ed4..3a43f53 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -781,6 +781,16 @@ ], "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "delimiter": { "type": "string", "examples": [ @@ -957,6 +967,16 @@ "json", "delta" ] + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } }, "required": [ @@ -1137,6 +1157,16 @@ "type": "string", "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "host": { "type": "string", "description": "Host name or IP address." @@ -1202,6 +1232,16 @@ "description": "The format of the messages.", "examples": ["json", "avro", "protobuf", "xml"], "default": "json" + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } }, "required": [ @@ -1228,6 +1268,16 @@ "avro", "protobuf" ] + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } } }, @@ -1252,6 +1302,16 @@ "delta", "csv" ] + }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] } }, "required": [ @@ -1444,6 +1504,16 @@ ], "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "delimiter": { "type": "string", "examples": [ @@ -1480,6 +1550,16 @@ ], "description": "File format." }, + "encoding": { + "type": "string", + "description": "The expected character encoding for data payloads exposed through this server, for example UTF-8, ISO-8859-1, ASCII or UTF-16.", + "examples": [ + "UTF-8", + "ISO-8859-1", + "ASCII", + "UTF-16" + ] + }, "delimiter": { "type": "string", "examples": [ diff --git a/src/script/negative-tests/server-encoding-wrong-type.odcs.yaml b/src/script/negative-tests/server-encoding-wrong-type.odcs.yaml new file mode 100644 index 0000000..21495da --- /dev/null +++ b/src/script/negative-tests/server-encoding-wrong-type.odcs.yaml @@ -0,0 +1,23 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0043 negative test: `encoding` must be a string. Here it is an integer, +# so the schema MUST reject this contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 9c2d4e6f-2a1b-4c3d-8e5f-1a2b3c4d5e6f +status: active +servers: + - server: production + type: s3 + location: s3://my-bucket/customer_export/*.csv + format: csv + encoding: 65001 +schema: + - name: customers + physicalType: table + properties: + - name: customer_id + logicalType: string From b213a1c6d5d08c167edf7a6a493c326c1d7109c1 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 14:28:48 -0400 Subject: [PATCH 14/29] Update infrastructure-servers.md --- docs/infrastructure-servers.md | 162 ++++++++++++++++----------------- 1 file changed, 81 insertions(+), 81 deletions(-) diff --git a/docs/infrastructure-servers.md b/docs/infrastructure-servers.md index ad2823b..1d7f732 100644 --- a/docs/infrastructure-servers.md +++ b/docs/infrastructure-servers.md @@ -44,7 +44,7 @@ servers: ### Common Server Properties | Key | Type | UX label | Required | Description | -|------------------|--------|-------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| ---------------- | ------ | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | description | string | Description | No | Description of the server. | | environment | string | Environment | No | Environment of the server. Examples includes: prod, preprod, dev, uat. | | id | string | ID | No | A unique identifier used to reduce the risk of collisions, such as a UUID. | @@ -63,16 +63,16 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### API Server -| Key | Type | UX Label | Required | Description | -|----------------|--------|------------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **location** | string | Location | Yes | URL to the API | +| Key | Type | UX Label | Required | Description | +| ------------ | ------ | -------- | -------- | -------------- | +| **location** | string | Location | Yes | URL to the API | ### Amazon Athena Server [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is an interactive query service that makes it easy to analyze data directly in Amazon Simple Storage Service (Amazon S3) using standard SQL. With a few actions in the AWS Management Console, you can point Athena at your data stored in Amazon S3 and begin using standard SQL to run ad-hoc queries and get results in seconds. | Key | Type | UX Label | Required | Description | -|------------|--------|-------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| ---------- | ------ | ----------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | catalog | string | Catalog | No | Identify the name of the Data Source, also referred to as a Catalog. | | regionName | string | Region Name | No | The region your AWS account uses. | | schema | string | Schema | Yes | Identify the schema in the data source in which your tables exist. | @@ -81,19 +81,19 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Azure Server -| Key | Type | UX Label | Required | Description | -|-----------|--------|-----------|----------|-----------------------------------------------------------------------------------------------| -| delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | +| Key | Type | UX Label | Required | Description | +| --------- | ------ | --------- | -------- | ----------------------------------------------------------------------------------------------------- | +| delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| format | string | Format | Yes | File format. | -| location | string | Location | Yes | Fully qualified path to Azure Blob Storage or Azure Data Lake Storage (ADLS), supports globs. | +| format | string | Format | Yes | File format. | +| location | string | Location | Yes | Fully qualified path to Azure Blob Storage or Azure Data Lake Storage (ADLS), supports globs. | ### Google BigQuery [BigQuery](https://cloud.google.com/bigquery) is a fully managed, AI-ready data analytics platform that helps you maximize value from your data and is designed to be multi-engine, multi-format, and multi-cloud. | Key | Type | UX Label | Required | Description | -|---------|--------|----------|----------|-----------------------------------------------| +| ------- | ------ | -------- | -------- | --------------------------------------------- | | dataset | string | Dataset | Yes | The GCP dataset name. | | project | string | Project | Yes | The Google Cloud Platform (GCP) project name. | @@ -102,7 +102,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge [ClickHouse](https://clickhouse.com/) is an open-source column-oriented database management system that allows generating analytical data reports in real-time. | Key | Type | UX Label | Required | Description | -|----------|---------|----------|----------|------------------------------------| +| -------- | ------- | -------- | -------- | ---------------------------------- | | database | string | Database | Yes | The name of the database. | | host | string | Host | Yes | The host of the ClickHouse server. | | port | integer | Port | Yes | The port to the ClickHouse server. | @@ -112,7 +112,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge [Google Cloud SQL](https://cloud.google.com/sql) is a fully managed, cost-effective relational database service for PostgreSQL, MySQL, and SQL Server. | Key | Type | UX Label | Required | Description | -|----------|---------|----------|----------|------------------------------------------| +| -------- | ------- | -------- | -------- | ---------------------------------------- | | database | string | Database | Yes | The name of the database. | | host | string | Host | Yes | The host of the Google Cloud SQL server. | | port | integer | Port | Yes | The port of the Google Cloud SQL server. | @@ -121,7 +121,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Databricks Server | Key | Type | UX Label | Required | Description | -|---------|--------|----------|----------|---------------------------------------| +| ------- | ------ | -------- | -------- | ------------------------------------- | | catalog | string | Catalog | Yes | The name of the Hive or Unity catalog | | host | string | Host | No | The Databricks host | | schema | string | Schema | Yes | The schema name in the catalog | @@ -129,7 +129,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### IBM Db2 Server | Key | Type | UX Label | Required | Description | -|----------|---------|----------|----------|---------------------------------| +| -------- | ------- | -------- | -------- | ------------------------------- | | database | string | Database | Yes | The name of the database. | | host | string | Host | Yes | The host of the IBM DB2 server. | | port | integer | Port | Yes | The port of the IBM DB2 server. | @@ -138,7 +138,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Denodo Server | Key | Type | UX Label | Required | Description | -|----------|---------|----------|----------|--------------------------------| +| -------- | ------- | -------- | -------- | ------------------------------ | | database | string | Database | No | The name of the database. | | host | string | Host | Yes | The host of the Denodo server. | | port | integer | Port | Yes | The port of the Denodo server. | @@ -146,7 +146,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Dremio Server | Key | Type | UX Label | Required | Description | -|--------|---------|----------|----------|--------------------------------| +| ------ | ------- | -------- | -------- | ------------------------------ | | host | string | Host | Yes | The host of the Dremio server. | | port | integer | Port | Yes | The port of the Dremio server. | | schema | string | Schema | No | The name of the schema. | @@ -156,39 +156,39 @@ If your server is not in the list, please use [custom](#custom-server) and sugge [DuckDB](https://duckdb.org/) supports a feature-rich SQL dialect complemented with deep integrations into client APIs. | Key | Type | UX Label | Required | Description | -|----------|--------|----------|----------|-------------------------------| +| -------- | ------ | -------- | -------- | ----------------------------- | | database | string | Database | Yes | Path to duckdb database file. | | schema | string | Schema | No | The name of the schema. | ### Amazon Glue -| Key | Type | UX Label | Required | Description | -|----------|--------|----------|----------|------------------------------------------------| -| account | string | Account | Yes | The AWS Glue account | -| database | string | Database | Yes | The AWS Glue database name | +| Key | Type | UX Label | Required | Description | +| -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | +| account | string | Account | Yes | The AWS Glue account | +| database | string | Database | Yes | The AWS Glue database name | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| format | string | Format | No | The format of the files | -| location | string | Location | No | The AWS S3 path. Must be in the form of a URL. | +| format | string | Format | No | The format of the files | +| location | string | Location | No | The AWS S3 path. Must be in the form of a URL. | ### Hive [Apache Hive](https://hive.apache.org/) is a distributed, fault-tolerant data warehouse system that enables analytics at massive scale. Built on top of Apache Hadoop, Hive allows users to read, write, and manage petabytes of data using SQL-like queries through HiveQL, with native support for cloud storage systems and enterprise-grade security features. -| Key | Type | UX Label | Required | Description | -|--------------|---------|-----------------|------------|-------------------------------------------------| -| database | string | Database | Yes | The name of the Hive database. | -| host | string | Host | Yes | The host to the Hive server. | -| port | integer | Port | No | The port to the Hive server. Defaults to 10000. | +| Key | Type | UX Label | Required | Description | +| -------- | ------- | -------- | -------- | ----------------------------------------------- | +| database | string | Database | Yes | The name of the Hive database. | +| host | string | Host | Yes | The host to the Hive server. | +| port | integer | Port | No | The port to the Hive server. Defaults to 10000. | ### Apache Impala [Apache Impala](https://impala.apache.org/) is a massively parallel processing (MPP) SQL query engine for data stored in Apache Hadoop clusters. Impala provides high-performance, low-latency SQL queries on data stored in HDFS and Apache HBase, enabling interactive exploration and analytics without data movement or transformation. -| Key | Type | UX Label | Required | Description | -|--------------|---------|-----------------|------------|---------------------------------------------------| -| database | string | Database | Yes | The name of the Impala database. | -| host | string | Host | Yes | The host to the Impala server. | -| port | integer | Port | No | The port to the Impala server. Defaults to 21050. | +| Key | Type | UX Label | Required | Description | +| -------- | ------- | -------- | -------- | ------------------------------------------------- | +| database | string | Database | Yes | The name of the Impala database. | +| host | string | Host | Yes | The host to the Impala server. | +| port | integer | Port | No | The port to the Impala server. Defaults to 21050. | ### IBM Informix and HCL Informix @@ -202,28 +202,28 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Kafka Server -| Key | Type | UX Label | Required | Description | -| ------ | ------ | -------- | -------- | ------------------------------------------ | +| Key | Type | UX Label | Required | Description | +| -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| format | string | Format | No | The format of the messages. | -| host | string | Host | Yes | The bootstrap server of the kafka cluster. | +| format | string | Format | No | The format of the messages. | +| host | string | Host | Yes | The bootstrap server of the kafka cluster. | ### Amazon Kinesis -| Key | Type | UX Label | Required | Description | -| ------ | ------ | -------- | -------- | ------------------------------------ | +| Key | Type | UX Label | Required | Description | +| -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| format | string | Format | No | The format of the record | -| region | string | Region | No | AWS region. | -| stream | string | Stream | Yes | The name of the Kinesis data stream. | +| format | string | Format | No | The format of the record | +| region | string | Region | No | AWS region. | +| stream | string | Stream | Yes | The name of the Kinesis data stream. | ### Local Files -| Key | Type | UX Label | Required | Description | -| ------ | ------ | -------- | -------- | -------------------------------------------------- | +| Key | Type | UX Label | Required | Description | +| -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| format | string | Format | Yes | The format of the file(s) | -| path | string | Path | Yes | The relative or absolute path to the data file(s). | +| format | string | Format | Yes | The format of the file(s) | +| path | string | Path | Yes | The relative or absolute path to the data file(s). | ### MySQL Server @@ -236,7 +236,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Oracle | Key | Type | UX Label | Required | Description | -|-------------|---------|--------------|----------|--------------------------------| +| ----------- | ------- | ------------ | -------- | ------------------------------ | | host | string | Host | Yes | The host to the Oracle server | | port | integer | Port | Yes | The port to the Oracle server. | | serviceName | string | Service Name | Yes | The name of the service. | @@ -265,7 +265,7 @@ If your server is not in the list, please use [custom](#custom-server) and sugge [Google Cloud](https://cloud.google.com/pubsub) service to Ingest events for streaming into BigQuery, data lakes or operational databases. | Key | Type | UX Label | Required | Description | -|---------|--------|----------|----------|-----------------------| +| ------- | ------ | -------- | -------- | --------------------- | | project | string | Project | Yes | The GCP project name. | ### Amazon Redshift Server @@ -284,24 +284,24 @@ If your server is not in the list, please use [custom](#custom-server) and sugge [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) is an object storage service offering industry-leading scalability, data availability, security, and performance. Millions of customers of all sizes and industries store, manage, analyze, and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps. Other vendors have implemented a compatible implementation of S3. -| Key | Type | UX Label | Required | Description | -| ----------- | ------ | ------------ | -------- | --------------------------------------------------------------------------------- | -| delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | +| Key | Type | UX Label | Required | Description | +| ----------- | ------ | ------------ | -------- | ----------------------------------------------------------------------------------------------------- | +| delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| endpointUrl | string | Endpoint URL | No | The server endpoint for S3-compatible servers. | -| format | string | Format | No | File format. | -| location | string | Location | Yes | S3 URL, starting with `s3://` | +| endpointUrl | string | Endpoint URL | No | The server endpoint for S3-compatible servers. | +| format | string | Format | No | File format. | +| location | string | Location | Yes | S3 URL, starting with `s3://` | ### SFTP Server Secure File Transfer Protocol (SFTP) is a network protocol that enables secure and encrypted file transfers between a client and a server. -| Key | Type | UX Label | Required | Description | -| --------- | ------ | --------- | -------- | --------------------------------------------------------------------------------- | -| delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | +| Key | Type | UX Label | Required | Description | +| --------- | ------ | --------- | -------- | ----------------------------------------------------------------------------------------------------- | +| delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| format | string | Format | No | File format. | -| location | string | Location | Yes | SFTP URL, starting with `sftp://`. The URL should include the port number. | +| format | string | Format | No | File format. | +| location | string | Location | Yes | SFTP URL, starting with `sftp://`. The URL should include the port number. | ### Snowflake @@ -363,28 +363,28 @@ Actian Zen (formerly Btrieve, later named Pervasive PSQL until version 13) is an ### Custom Server -| Key | Type | UX Label | Required | Description | -|-------------|---------|-------------------|----------|---------------------------------------------------------------------| -| account | string | Account | No | Account used by the server. | -| catalog | string | Catalog | No | Name of the catalog. | -| database | string | Database | No | Name of the database. | -| dataset | string | Dataset | No | Name of the dataset. | -| delimiter | string | Delimiter | No | Delimiter. | +| Key | Type | UX Label | Required | Description | +| ----------- | ------- | ----------------- | -------- | ----------------------------------------------------------------------------------------------------- | +| account | string | Account | No | Account used by the server. | +| catalog | string | Catalog | No | Name of the catalog. | +| database | string | Database | No | Name of the database. | +| dataset | string | Dataset | No | Name of the dataset. | +| delimiter | string | Delimiter | No | Delimiter. | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | -| endpointUrl | string | Endpoint URL | No | Server endpoint. | -| format | string | Format | No | File format. | -| host | string | Host | No | Host name or IP address. | -| location | string | Location | No | A URL to a location. | -| path | string | Path | No | Relative or absolute path to the data file(s). | -| port | integer | Port | No | Port to the server. No default value is assumed for custom servers. | -| project | string | Project | No | Project name. | -| region | string | Region | No | Cloud region. | -| regionName | string | Region Name | No | Region name. | -| schema | string | Schema | No | Name of the schema. | -| serviceName | string | Service Name | No | Name of the service. | -| stagingDir | string | Staging Directory | No | Staging directory. | -| stream | string | Stream | No | Name of the data stream. | -| warehouse | string | Warehouse | No | Name of the cluster or warehouse. | +| endpointUrl | string | Endpoint URL | No | Server endpoint. | +| format | string | Format | No | File format. | +| host | string | Host | No | Host name or IP address. | +| location | string | Location | No | A URL to a location. | +| path | string | Path | No | Relative or absolute path to the data file(s). | +| port | integer | Port | No | Port to the server. No default value is assumed for custom servers. | +| project | string | Project | No | Project name. | +| region | string | Region | No | Cloud region. | +| regionName | string | Region Name | No | Region name. | +| schema | string | Schema | No | Name of the schema. | +| serviceName | string | Service Name | No | Name of the service. | +| stagingDir | string | Staging Directory | No | Staging directory. | +| stream | string | Stream | No | Name of the data stream. | +| warehouse | string | Warehouse | No | Name of the cluster or warehouse. | If you need another property, use [custom properties](./custom-other-properties.md). From 04bf598c56fe65d6282a701eb731cb9e8567b249 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 14:35:41 -0400 Subject: [PATCH 15/29] feat(rfc-0028): add optional `deprecated` flag to schema objects and properties (#294) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements approved RFC-0028 (Deprecated flag), shared with ODPS v1.1.0, for ODCS v3.2.0. Adds an optional `deprecated` boolean (default false) on SchemaElement, which both schema objects and properties (including nested) inherit — mirroring the synonyms placement. - schema: deprecated on SchemaElement in both latest + v3.2.0 (lockstep) - docs: deprecated row in schema.md + a Deprecated section - example: docs/examples/schema/deprecated.odcs.yaml (validated) - negative test: deprecated-wrong-type.odcs.yaml (schema rejects non-boolean) - CHANGELOG entry Approved RFC: https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0028-deprecated-flag.md Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 3 ++ docs/examples/schema/deprecated.odcs.yaml | 33 +++++++++++++++++++ docs/schema.md | 27 +++++++++++++++ schema/odcs-json-schema-latest.json | 5 +++ schema/odcs-json-schema-v3.2.0.json | 5 +++ .../deprecated-wrong-type.odcs.yaml | 18 ++++++++++ 6 files changed, 91 insertions(+) create mode 100644 docs/examples/schema/deprecated.odcs.yaml create mode 100644 src/script/negative-tests/deprecated-wrong-type.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 730b390..88f7868 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,9 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * **Adds** Physical data encoding ([RFC 0043](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0043-physical-data-encoding.md)): * New optional `encoding` string field on server definitions that expose serialized payloads (Azure, Glue, Custom, Kafka, Kinesis, Local, S3, SFTP), declaring the expected character encoding of the data, e.g. `UTF-8`, `ISO-8859-1`, `ASCII`, `UTF-16`. * Free-form string (no enum), default `UTF-8`; documents physical-payload encoding separately from the ODCS document encoding. Non-breaking, as `encoding` is optional. +* **Adds** Deprecated flag ([RFC 0028](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0028-deprecated-flag.md), shared with ODPS v1.1.0): + * New optional `deprecated` boolean on schema objects and properties (including nested properties), indicating an element is no longer recommended for use. + * Defaults to `false`; deprecated elements remain documented and validated for backward compatibility. Non-breaking, as `deprecated` is optional. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/schema/deprecated.odcs.yaml b/docs/examples/schema/deprecated.odcs.yaml new file mode 100644 index 0000000..855cb5b --- /dev/null +++ b/docs/examples/schema/deprecated.odcs.yaml @@ -0,0 +1,33 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0028: the optional `deprecated` boolean on schema objects +# and properties (including nested properties). Defaults to false. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 3e8a1b6c-4d2f-4a90-b1c7-5e9f0a2b3c4d +status: active +schema: + - name: legacy_orders + physicalType: table + deprecated: true + description: "DEPRECATED: use 'orders' instead. Will be removed in the next major version." + properties: + - name: order_id + logicalType: integer + required: true + - name: category_id + logicalType: integer + deprecated: true + description: "DEPRECATED: use the nested 'category' object instead." + - name: category + logicalType: object + properties: + - name: id + logicalType: integer + - name: legacy_code + logicalType: string + deprecated: true + description: "DEPRECATED: retained for backward compatibility only." diff --git a/docs/schema.md b/docs/schema.md index bcaea4b..bc28215 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -157,6 +157,7 @@ schema: | ------------------------ | ------ | ------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | businessName | string | Business Name | No | The business name of the element. | | description | string | Description | No | Description of the element. | +| deprecated | boolean | Deprecated | No | Indicates this element is deprecated and should not be used in new implementations. Defaults to `false`. See [Deprecated](#deprecated). | | id | string | ID | No | A unique identifier for the element used to create stable, refactor-safe references. Recommended for elements that will be referenced. See [References](./references.md) for more details. | | name | string | Name | Yes | Name of the element. | | physicalName | string | Physical Name | No | Physical name. | @@ -480,3 +481,29 @@ schema: `synonyms` was introduced in ODCS v3.2.0 ([RFC 0041](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0041-synonyms.md)). [Back to TOC](README.md) + +## Deprecated + +Any element (a schema object or a property, including nested properties) may set the optional `deprecated` boolean to signal that it is no longer recommended for use. It defaults to `false` when not specified. Deprecated elements remain documented and validated for backward compatibility; implementations MAY warn when they are used. Use the `description` field to point to a replacement and provide migration guidance. + +```yaml +schema: + - name: customers + logicalType: object + properties: + - name: email_address + logicalType: string + deprecated: true + description: "DEPRECATED: use 'primary_email' instead. Will be removed in the next major version." + - name: primary_email + logicalType: string + description: "Primary email address for the customer." +``` + +| Key | Type | UX label | Required | Description | +| ------------ | ------- | ---------- | -------- | ---------------------------------------------------------------------------------------------------- | +| deprecated | boolean | Deprecated | No | Indicates this element is deprecated and should not be used in new implementations. Defaults to `false`. | + +`deprecated` was introduced in ODCS v3.2.0 ([RFC 0028](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0028-deprecated-flag.md)). + +[Back to TOC](README.md) diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 150ead2..19019d9 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -1761,6 +1761,11 @@ "type": "string", "description": "The business name of the element." }, + "deprecated": { + "type": "boolean", + "description": "Indicates this element is deprecated and should not be used in new implementations. Defaults to false.", + "default": false + }, "synonyms": { "$ref": "#/$defs/Synonyms" }, diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index 3a43f53..cd15174 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -1760,6 +1760,11 @@ "type": "string", "description": "The business name of the element." }, + "deprecated": { + "type": "boolean", + "description": "Indicates this element is deprecated and should not be used in new implementations. Defaults to false.", + "default": false + }, "synonyms": { "$ref": "#/$defs/Synonyms" }, diff --git a/src/script/negative-tests/deprecated-wrong-type.odcs.yaml b/src/script/negative-tests/deprecated-wrong-type.odcs.yaml new file mode 100644 index 0000000..ee418df --- /dev/null +++ b/src/script/negative-tests/deprecated-wrong-type.odcs.yaml @@ -0,0 +1,18 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0028 negative test: `deprecated` must be a boolean. Here it is a string, +# so the schema MUST reject this contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 5f1c7a2d-8b3e-4c6a-9d0f-2e4b6a8c0d1e +status: active +schema: + - name: customers + physicalType: table + properties: + - name: email_address + logicalType: string + deprecated: "maybe" From 6f03298f72148d49fe036cf02cea21b403d238fd Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 15:01:05 -0400 Subject: [PATCH 16/29] feat(rfc-0035): add optional `vendor` field to customProperties (#295) Implements approved RFC-0035 (Vendor Attribution), shared with ODPS v1.1.0 and OORS v1.0.0, for ODCS v3.2.0. Adds an optional `vendor` string to CustomProperty. Per the RFC, vendor SHOULD be lowercase / match ^[a-z0-9][a-z0-9-]*$ but this is NOT enforced as a schema pattern (tools MUST preserve unknown vendor values); the recommended format is documented. - schema: vendor on CustomProperty in both latest + v3.2.0 (lockstep) - docs: vendor row in custom-other-properties.md - example: docs/examples/custom-other-properties/vendor.odcs.yaml (validated) - negative test: vendor-wrong-type.odcs.yaml (schema rejects non-string) - CHANGELOG entry Approved RFC: https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0035-extensions.md Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 3 ++ docs/custom-other-properties.md | 1 + .../custom-other-properties/vendor.odcs.yaml | 33 +++++++++++++++++++ schema/odcs-json-schema-latest.json | 5 +++ schema/odcs-json-schema-v3.2.0.json | 5 +++ .../vendor-wrong-type.odcs.yaml | 18 ++++++++++ 6 files changed, 65 insertions(+) create mode 100644 docs/examples/custom-other-properties/vendor.odcs.yaml create mode 100644 src/script/negative-tests/vendor-wrong-type.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 88f7868..d5922de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,9 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * **Adds** Deprecated flag ([RFC 0028](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0028-deprecated-flag.md), shared with ODPS v1.1.0): * New optional `deprecated` boolean on schema objects and properties (including nested properties), indicating an element is no longer recommended for use. * Defaults to `false`; deprecated elements remain documented and validated for backward compatibility. Non-breaking, as `deprecated` is optional. +* **Adds** Vendor attribution for custom properties ([RFC 0035](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0035-extensions.md), shared with ODPS v1.1.0 and OORS v1.0.0): + * New optional `vendor` string on `customProperties` items, associating a custom property with a specific vendor, provider, or external system. + * SHOULD be a stable, lowercase identifier (`^[a-z0-9][a-z0-9-]*$`); not enforced, and tools MUST preserve unknown vendor values. Non-breaking, as `vendor` is optional. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/custom-other-properties.md b/docs/custom-other-properties.md index 2653f61..9b4b3ed 100644 --- a/docs/custom-other-properties.md +++ b/docs/custom-other-properties.md @@ -44,6 +44,7 @@ customProperties: | customProperties[].id | string | ID | No | A unique identifier for the element used to create stable, refactor-safe references. Recommended for elements that will be referenced. See [References](./references.md) for more details. | | customProperties[].property | string | Property | No | The name of the key. Names should be in camel case–the same as if they were permanent properties in the contract. | | customProperties[].value | any | Value | No | The value of the key. It can be an array. | +| customProperties[].vendor | string | Vendor | No | Identifies the vendor, provider, or external system associated with this custom property. SHOULD be a stable, lowercase identifier matching `^[a-z0-9][a-z0-9-]*$` (e.g. `confluent`, `zeenea`). Tools MUST preserve unknown vendor values. (Added in v3.2.0, [RFC 0035](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0035-extensions.md).) | Avis: With version 3.2.0 the Description of 'customProperties.property' will be updated to "The name of the key. Could be in any notation. If this field is used for referencing it should be in camel case–the same as if they were permanent properties in the contract. Note that since ODCS version 3.1 the field 'customProperties.id' should be used for referencing purposes. In this case the name of the key could be human-readable and self-explanatory to the greatest extent." diff --git a/docs/examples/custom-other-properties/vendor.odcs.yaml b/docs/examples/custom-other-properties/vendor.odcs.yaml new file mode 100644 index 0000000..2c61bd0 --- /dev/null +++ b/docs/examples/custom-other-properties/vendor.odcs.yaml @@ -0,0 +1,33 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0035: the optional `vendor` field on customProperties, +# attributing a custom property to a specific vendor or external system. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 6a2f8c1d-3b4e-4f9a-8c7d-0e1f2a3b4c5d +status: active +customProperties: + - id: prop_gov_object + property: propGovObject + vendor: confluent + value: + myObject: + mox: abc + moy: 123 + - property: glossaryRefs + vendor: zeenea + value: + - category: KPI + def: Number of delivered doses + - property: genericFlag + value: true +schema: + - name: orders + physicalType: table + properties: + - name: order_id + logicalType: integer + required: true diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 19019d9..e0c2057 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -2993,6 +2993,11 @@ "description": { "type": "string", "description": "Description of the custom property." + }, + "vendor": { + "type": "string", + "description": "Identifies the vendor, provider, or external system associated with this custom property. SHOULD be a stable, lowercase identifier matching ^[a-z0-9][a-z0-9-]*$ (e.g. confluent, zeenea, atlan, soda). Tools MUST preserve unknown vendor values.", + "examples": ["confluent", "zeenea", "atlan", "soda"] } }, "required": ["property", "value"], diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index cd15174..ab7b918 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -2992,6 +2992,11 @@ "description": { "type": "string", "description": "Description of the custom property." + }, + "vendor": { + "type": "string", + "description": "Identifies the vendor, provider, or external system associated with this custom property. SHOULD be a stable, lowercase identifier matching ^[a-z0-9][a-z0-9-]*$ (e.g. confluent, zeenea, atlan, soda). Tools MUST preserve unknown vendor values.", + "examples": ["confluent", "zeenea", "atlan", "soda"] } }, "required": ["property", "value"], diff --git a/src/script/negative-tests/vendor-wrong-type.odcs.yaml b/src/script/negative-tests/vendor-wrong-type.odcs.yaml new file mode 100644 index 0000000..a9e0031 --- /dev/null +++ b/src/script/negative-tests/vendor-wrong-type.odcs.yaml @@ -0,0 +1,18 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0035 negative test: customProperties[].vendor must be a string. Here it +# is a number, so the schema MUST reject this contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 7b3e9d2f-4c5a-4e0b-9f1c-2d3e4f5a6b7c +status: active +customProperties: + - property: propGovObject + value: abc + vendor: 12345 +schema: + - name: orders + physicalType: table From b70d585a97912a89e1d6d1918fc24d5355500048 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 15:07:05 -0400 Subject: [PATCH 17/29] =?UTF-8?q?chore:=20renumber=20deprecated-flag=20RFC?= =?UTF-8?q?=200028=20=E2=86=92=200051=20(#296)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update all ODCS references to the deprecated-flag RFC from 0028 to 0051 after the number was renumbered in bitol-io/tsc to resolve a collision with the earlier RFC-0028 (Permalink). Claude-Session: https://claude.ai/code/session_01FRgkVBM2ry2NQecEAeddnN Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- docs/examples/schema/deprecated.odcs.yaml | 2 +- docs/schema.md | 2 +- src/script/negative-tests/deprecated-wrong-type.odcs.yaml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d5922de..76645c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,7 +34,7 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * **Adds** Physical data encoding ([RFC 0043](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0043-physical-data-encoding.md)): * New optional `encoding` string field on server definitions that expose serialized payloads (Azure, Glue, Custom, Kafka, Kinesis, Local, S3, SFTP), declaring the expected character encoding of the data, e.g. `UTF-8`, `ISO-8859-1`, `ASCII`, `UTF-16`. * Free-form string (no enum), default `UTF-8`; documents physical-payload encoding separately from the ODCS document encoding. Non-breaking, as `encoding` is optional. -* **Adds** Deprecated flag ([RFC 0028](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0028-deprecated-flag.md), shared with ODPS v1.1.0): +* **Adds** Deprecated flag ([RFC 0051](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0051-deprecated-flag.md), shared with ODPS v1.1.0): * New optional `deprecated` boolean on schema objects and properties (including nested properties), indicating an element is no longer recommended for use. * Defaults to `false`; deprecated elements remain documented and validated for backward compatibility. Non-breaking, as `deprecated` is optional. * **Adds** Vendor attribution for custom properties ([RFC 0035](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0035-extensions.md), shared with ODPS v1.1.0 and OORS v1.0.0): diff --git a/docs/examples/schema/deprecated.odcs.yaml b/docs/examples/schema/deprecated.odcs.yaml index 855cb5b..599549a 100644 --- a/docs/examples/schema/deprecated.odcs.yaml +++ b/docs/examples/schema/deprecated.odcs.yaml @@ -1,7 +1,7 @@ # Copyright 2026 The Bitol Contributors # SPDX-License-Identifier: Apache-2.0 -# Demonstrates RFC-0028: the optional `deprecated` boolean on schema objects +# Demonstrates RFC-0051: the optional `deprecated` boolean on schema objects # and properties (including nested properties). Defaults to false. version: 1.0.0 diff --git a/docs/schema.md b/docs/schema.md index bc28215..d07fd85 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -504,6 +504,6 @@ schema: | ------------ | ------- | ---------- | -------- | ---------------------------------------------------------------------------------------------------- | | deprecated | boolean | Deprecated | No | Indicates this element is deprecated and should not be used in new implementations. Defaults to `false`. | -`deprecated` was introduced in ODCS v3.2.0 ([RFC 0028](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0028-deprecated-flag.md)). +`deprecated` was introduced in ODCS v3.2.0 ([RFC 0051](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0051-deprecated-flag.md)). [Back to TOC](README.md) diff --git a/src/script/negative-tests/deprecated-wrong-type.odcs.yaml b/src/script/negative-tests/deprecated-wrong-type.odcs.yaml index ee418df..9f60e3b 100644 --- a/src/script/negative-tests/deprecated-wrong-type.odcs.yaml +++ b/src/script/negative-tests/deprecated-wrong-type.odcs.yaml @@ -1,7 +1,7 @@ # Copyright 2026 The Bitol Contributors # SPDX-License-Identifier: Apache-2.0 -# RFC-0028 negative test: `deprecated` must be a boolean. Here it is a string, +# RFC-0051 negative test: `deprecated` must be a boolean. Here it is a string, # so the schema MUST reject this contract. version: 1.0.0 From 16341fe1832bb1d71ebf966fbbfdf5b654adfea0 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 15:16:52 -0400 Subject: [PATCH 18/29] feat(rfc-0042): add `vector` logicalType for embeddings (#297) Implements approved RFC-0042 (Vector Type) for ODCS v3.2.0. Adds a `vector` value to the logicalType enum and a conditional logicalTypeOptions block (if logicalType==vector) with required `dimensions` plus optional `elementType`, `distanceMetric`, `normalized`, `embeddingModel`, and `embeddingModelVersion`. Field names are those the TSC ratified at approval (`dimensions`, `elementType`). - schema: vector enum value + conditional options block in both latest + v3.2.0 (lockstep); `dimensions` required when logicalType==vector - docs: vector rows in Logical Type Options table + a Vectors section in schema.md - example: docs/examples/schema/vector.odcs.yaml (validated) - negative test: vector-missing-dimensions.odcs.yaml (schema rejects: missing dimensions) - CHANGELOG entry Approved RFC: https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0042-vector-type.md Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 4 ++ docs/examples/schema/vector.odcs.yaml | 39 ++++++++++++++ docs/schema.md | 47 +++++++++++++++- schema/odcs-json-schema-latest.json | 53 ++++++++++++++++++- schema/odcs-json-schema-v3.2.0.json | 53 ++++++++++++++++++- .../vector-missing-dimensions.odcs.yaml | 20 +++++++ 6 files changed, 213 insertions(+), 3 deletions(-) create mode 100644 docs/examples/schema/vector.odcs.yaml create mode 100644 src/script/negative-tests/vector-missing-dimensions.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 76645c3..4d79273 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,10 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * **Adds** Vendor attribution for custom properties ([RFC 0035](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0035-extensions.md), shared with ODPS v1.1.0 and OORS v1.0.0): * New optional `vendor` string on `customProperties` items, associating a custom property with a specific vendor, provider, or external system. * SHOULD be a stable, lowercase identifier (`^[a-z0-9][a-z0-9-]*$`); not enforced, and tools MUST preserve unknown vendor values. Non-breaking, as `vendor` is optional. +* **Adds** Vector type ([RFC 0042](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0042-vector-type.md)): + * New `vector` value for `logicalType`, describing a fixed-dimension dense numeric array for embeddings and similarity search. + * Dedicated `logicalTypeOptions` for `vector`: required `dimensions` (positive integer) plus optional `elementType`, `distanceMetric`, `normalized`, `embeddingModel`, and `embeddingModelVersion`. + * Non-breaking: `vector` is a new optional `logicalType` value. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/schema/vector.odcs.yaml b/docs/examples/schema/vector.odcs.yaml new file mode 100644 index 0000000..d28531f --- /dev/null +++ b/docs/examples/schema/vector.odcs.yaml @@ -0,0 +1,39 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0042: the `vector` logicalType and its logicalTypeOptions +# (dimensions required; elementType, distanceMetric, normalized, embeddingModel, +# embeddingModelVersion optional). + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 8c1d4e7f-2a3b-4c5d-9e0f-1a2b3c4d5e6f +status: active +schema: + - name: tickets + physicalName: ticket_embeddings + properties: + - name: ticket_id + logicalType: string + primaryKey: true + required: true + - name: body + logicalType: string + - name: body_embedding + logicalType: vector + physicalType: vector(1536) + required: true + logicalTypeOptions: + dimensions: 1536 + elementType: float32 + distanceMetric: cosine + normalized: true + embeddingModel: openai/text-embedding-3-small + embeddingModelVersion: "2024-01-25" + - name: body_embedding_int8 + logicalType: vector + logicalTypeOptions: + dimensions: 1536 + elementType: int8 + distanceMetric: cosine diff --git a/docs/schema.md b/docs/schema.md index d07fd85..be23126 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -162,7 +162,7 @@ schema: | name | string | Name | Yes | Name of the element. | | physicalName | string | Physical Name | No | Physical name. | | physicalType | string | Physical Type | No | The physical element data type in the data source. For objects: `table`, `view`, `topic`, `file`. For properties: `VARCHAR(2)`, `DOUBLE`, `INT`, etc. | -| logicalType | string | Logical Type | No | The logical data type of the element. One of `string`, `date`, `timestamp`, `time`, `number`, `integer`, `object`, `array`, `boolean`, or `map`. At the object level this is typically `object` (or `array` for an array of objects); the data-type keywords above apply to properties. | +| logicalType | string | Logical Type | No | The logical data type of the element. One of `string`, `date`, `timestamp`, `time`, `number`, `integer`, `object`, `array`, `boolean`, `map`, or `vector`. At the object level this is typically `object` (or `array` for an array of objects); the data-type keywords above apply to properties. | | quality | array | Quality | No | List of data quality attributes. | | synonyms | array | Synonyms | No | A list of alternative names for the element (object or property), helping catalogs, AI/LLM tools, and natural language interfaces resolve business vocabulary to the underlying object. See [Synonyms](#synonyms). | | authoritativeDefinitions | array | Authoritative Definitions | No | List of links to sources that provide more details on the element; examples would be a link to privacy statement, terms and conditions, license agreements, data catalog, or another tool. | @@ -232,6 +232,12 @@ Additional metadata options to more accurately define the data type. | string | maxLength | integer | Maximum Length | No | Maximum length of the string. | | string | minLength | integer | Minimum Length | No | Minimum length of the string. | | string | pattern | string | Pattern | No | Regular expression pattern to define valid value. Follows regular expression syntax from ECMA-262 (). | +| vector | dimensions | integer | Dimensions | Yes | The fixed length of the vector. Positive integer. Examples: `384`, `768`, `1024`, `1536`, `3072`. See [Vectors](#vectors). | +| vector | elementType | string | Element Type | No | Numeric type of each element. One of `bfloat16`, `binary`, `float16`, `float32` (default), `float64`, `int8`, `uint8`. `binary` means each element is one bit (binary-quantized vectors). | +| vector | distanceMetric | string | Distance Metric | No | Intended similarity metric. One of `cosine`, `dotProduct`, `euclidean`, `hamming`, `manhattan`. Advisory — the physical index may differ. | +| vector | embeddingModel | string | Embedding Model | No | Identifier of the model used to produce the vectors (e.g., `openai/text-embedding-3-small`, `cohere/embed-english-v3.0`). | +| vector | embeddingModelVersion | string | Embedding Model Version | No | Version or revision of the embedding model, when the model identifier does not already carry one. | +| vector | normalized | boolean | Normalized | No | `true` if vectors are L2-normalized before storage, which makes `cosine` and `dotProduct` equivalent. Default `false`. | ### Expressing Date / Datetime / Timezone information @@ -377,6 +383,45 @@ schema: `logicalType: map` was introduced in ODCS v3.2.0 ([RFC 0030](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0030-maps.md)). +## Vectors + +A property can declare `logicalType: vector` to describe a fixed-dimension dense numeric array used for embeddings and similarity search (RAG, semantic matching). The shape is captured with `logicalTypeOptions`: `dimensions` (required), and the optional `elementType`, `distanceMetric`, `normalized`, `embeddingModel`, and `embeddingModelVersion`. The outer `physicalType` still carries the target system's native column syntax (e.g. `vector(1536)`, `VECTOR(FLOAT, 1536)`). + +### Examples + +**Minimal — a single embedding column:** + +```yaml +schema: + - name: products + properties: + - name: description_embedding + logicalType: vector + required: true + logicalTypeOptions: + dimensions: 1536 +``` + +**Detailed — a normalized OpenAI embedding with cosine similarity:** + +```yaml +- name: body_embedding + logicalType: vector + physicalType: vector(1536) + required: true + logicalTypeOptions: + dimensions: 1536 + elementType: float32 + distanceMetric: cosine + normalized: true + embeddingModel: openai/text-embedding-3-small + embeddingModelVersion: "2024-01-25" +``` + +`dimensions` is required whenever `logicalType: vector`. See [Logical Type Options](#logical-type-options) for the full list of options. + +`logicalType: vector` was introduced in ODCS v3.2.0 ([RFC 0042](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0042-vector-type.md)). + ## Enumerations A property can declare an `enum` to constrain its value to a fixed set of allowed entries. Each entry is an object with at least a `value` and may carry a label, identifier, description, tags, custom properties, and authoritative definitions. diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index e0c2057..e54f087 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -1842,7 +1842,7 @@ "logicalType": { "type": "string", "description": "The logical element data type.", - "enum": ["string", "date", "timestamp", "time", "number", "integer", "object", "array", "boolean", "map"] + "enum": ["string", "date", "timestamp", "time", "number", "integer", "object", "array", "boolean", "map", "vector"] }, "logicalTypeOptions": { "type": "object", @@ -2295,6 +2295,57 @@ }, "required": ["map"] } + }, + { + "if": { + "properties": { + "logicalType": { + "const": "vector" + } + } + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "dimensions": { + "type": "integer", + "minimum": 1, + "description": "The fixed length of the vector. Positive integer (only applicable when `logicalType: vector`). See RFC 0042.", + "examples": [384, 768, 1024, 1536, 3072] + }, + "elementType": { + "type": "string", + "enum": ["bfloat16", "binary", "float16", "float32", "float64", "int8", "uint8"], + "default": "float32", + "description": "Numeric type of each vector element. `binary` means each element is one bit (binary-quantized vectors)." + }, + "distanceMetric": { + "type": "string", + "enum": ["cosine", "dotProduct", "euclidean", "hamming", "manhattan"], + "description": "Intended similarity metric. Advisory \u2014 the physical index may differ." + }, + "embeddingModel": { + "type": "string", + "description": "Identifier of the model used to produce the vectors.", + "examples": ["openai/text-embedding-3-small", "cohere/embed-english-v3.0"] + }, + "embeddingModelVersion": { + "type": "string", + "description": "Version or revision of the embedding model, when the model identifier does not already carry one." + }, + "normalized": { + "type": "boolean", + "default": false, + "description": "True if vectors are L2-normalized before storage, which makes cosine and dotProduct equivalent." + } + }, + "required": ["dimensions"] + } + } + } } ], "unevaluatedProperties": false diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index ab7b918..20d13c9 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -1841,7 +1841,7 @@ "logicalType": { "type": "string", "description": "The logical element data type.", - "enum": ["string", "date", "timestamp", "time", "number", "integer", "object", "array", "boolean", "map"] + "enum": ["string", "date", "timestamp", "time", "number", "integer", "object", "array", "boolean", "map", "vector"] }, "logicalTypeOptions": { "type": "object", @@ -2294,6 +2294,57 @@ }, "required": ["map"] } + }, + { + "if": { + "properties": { + "logicalType": { + "const": "vector" + } + } + }, + "then": { + "properties": { + "logicalTypeOptions": { + "type": "object", + "additionalProperties": false, + "properties": { + "dimensions": { + "type": "integer", + "minimum": 1, + "description": "The fixed length of the vector. Positive integer (only applicable when `logicalType: vector`). See RFC 0042.", + "examples": [384, 768, 1024, 1536, 3072] + }, + "elementType": { + "type": "string", + "enum": ["bfloat16", "binary", "float16", "float32", "float64", "int8", "uint8"], + "default": "float32", + "description": "Numeric type of each vector element. `binary` means each element is one bit (binary-quantized vectors)." + }, + "distanceMetric": { + "type": "string", + "enum": ["cosine", "dotProduct", "euclidean", "hamming", "manhattan"], + "description": "Intended similarity metric. Advisory \u2014 the physical index may differ." + }, + "embeddingModel": { + "type": "string", + "description": "Identifier of the model used to produce the vectors.", + "examples": ["openai/text-embedding-3-small", "cohere/embed-english-v3.0"] + }, + "embeddingModelVersion": { + "type": "string", + "description": "Version or revision of the embedding model, when the model identifier does not already carry one." + }, + "normalized": { + "type": "boolean", + "default": false, + "description": "True if vectors are L2-normalized before storage, which makes cosine and dotProduct equivalent." + } + }, + "required": ["dimensions"] + } + } + } } ], "unevaluatedProperties": false diff --git a/src/script/negative-tests/vector-missing-dimensions.odcs.yaml b/src/script/negative-tests/vector-missing-dimensions.odcs.yaml new file mode 100644 index 0000000..ff08616 --- /dev/null +++ b/src/script/negative-tests/vector-missing-dimensions.odcs.yaml @@ -0,0 +1,20 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0042 negative test: `dimensions` is required whenever logicalType is +# vector. Here it is omitted, so the schema MUST reject this contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 9d2e5f8a-3b4c-4d6e-8f1a-2b3c4d5e6f7a +status: active +schema: + - name: products + physicalType: table + properties: + - name: description_embedding + logicalType: vector + logicalTypeOptions: + elementType: float32 + distanceMetric: cosine From 09b06f4effaceaa218fbe94ad67b4225937c2827 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 15:23:33 -0400 Subject: [PATCH 19/29] feat(rfc-0045): add `hana` server type for SAP HANA (#298) Implements approved RFC-0045 for ODCS v3.2.0. Adds `hana` to the server type enum, a HanaServer ServerSource def, and the type->$ref mapping. Required `host` plus optional `port`, `database` (tenant), and `schema`. Note: the RFC lists `port` as "number"; implemented as integer to match every other server type (a port is an integer). - schema: hana enum value + HanaServer def + oneOf mapping in both latest + v3.2.0 (lockstep); only `host` required - docs: SAP HANA section in infrastructure-servers.md - example: docs/examples/server/hana-server.odcs.yaml (validated) - negative test: hana-missing-host.odcs.yaml (schema rejects: missing host) - CHANGELOG entry Approved RFC: https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0045-hana-server-type.md Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 3 ++ docs/examples/server/hana-server.odcs.yaml | 24 +++++++++++ docs/infrastructure-servers.md | 11 +++++ schema/odcs-json-schema-latest.json | 40 ++++++++++++++++++- schema/odcs-json-schema-v3.2.0.json | 40 ++++++++++++++++++- .../hana-missing-host.odcs.yaml | 19 +++++++++ 6 files changed, 135 insertions(+), 2 deletions(-) create mode 100644 docs/examples/server/hana-server.odcs.yaml create mode 100644 src/script/negative-tests/hana-missing-host.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d79273..bcb35cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,9 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * New `vector` value for `logicalType`, describing a fixed-dimension dense numeric array for embeddings and similarity search. * Dedicated `logicalTypeOptions` for `vector`: required `dimensions` (positive integer) plus optional `elementType`, `distanceMetric`, `normalized`, `embeddingModel`, and `embeddingModelVersion`. * Non-breaking: `vector` is a new optional `logicalType` value. +* **Adds** SAP HANA server type ([RFC 0045](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0045-hana-server-type.md)): + * New `hana` server `type` for SAP HANA, with required `host` plus optional `port`, `database` (tenant), and `schema`. + * Non-breaking: adds a new optional server type. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/server/hana-server.odcs.yaml b/docs/examples/server/hana-server.odcs.yaml new file mode 100644 index 0000000..a3c2684 --- /dev/null +++ b/docs/examples/server/hana-server.odcs.yaml @@ -0,0 +1,24 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0045: the `hana` server type for SAP HANA. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 5d8b2a1c-7e3f-4a6d-9b0c-1e2f3a4b5c6d +status: active +servers: + - server: prod + type: hana + host: hana.acme.com + port: 30015 + database: HXE + schema: SALES +schema: + - name: sales + physicalType: table + properties: + - name: sale_id + logicalType: string + required: true diff --git a/docs/infrastructure-servers.md b/docs/infrastructure-servers.md index 1d7f732..b8cf638 100644 --- a/docs/infrastructure-servers.md +++ b/docs/infrastructure-servers.md @@ -292,6 +292,17 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | format | string | Format | No | File format. | | location | string | Location | Yes | S3 URL, starting with `s3://` | +### SAP HANA + +[SAP HANA](https://www.sap.com/products/technology-platform/hana.html) is an in-memory, column-oriented relational database used as an enterprise data platform. + +| Key | Type | UX Label | Required | Description | +| -------- | ------- | -------- | -------- | ------------------------------ | +| host | string | Host | Yes | Host of the HANA server. | +| port | integer | Port | No | Port of the HANA server. | +| database | string | Database | No | Name of the database (tenant). | +| schema | string | Schema | No | Name of the schema. | + ### SFTP Server Secure File Transfer Protocol (SFTP) is a network protocol that enables secure and encrypted file transfers between a client and a server. diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index e54f087..3887499 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -233,7 +233,7 @@ "description": "Type of the server.", "enum": [ "api", "athena", "azure", "bigquery", "clickhouse", "databricks", "denodo", "dremio", - "duckdb", "glue", "cloudsql", "db2", "hive", "impala", "informix", "kafka", "kinesis", "local", + "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "impala", "informix", "kafka", "kinesis", "local", "mysql", "oracle", "postgresql", "postgres", "presto", "pubsub", "redshift", "s3", "sftp", "snowflake", "sqlserver", "synapse", "trino", "vertica", "zen", "custom" ] @@ -545,6 +545,19 @@ "$ref": "#/$defs/ServerSource/OracleServer" } }, + { + "if": { + "properties": { + "type": { + "const": "hana" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/HanaServer" + } + }, { "if": { "properties": { @@ -1403,6 +1416,31 @@ "schema" ] }, + "HanaServer": { + "type": "object", + "title": "HanaServer", + "properties": { + "host": { + "type": "string", + "description": "Host of the HANA server." + }, + "port": { + "type": "integer", + "description": "Port of the HANA server." + }, + "database": { + "type": "string", + "description": "Name of the database (tenant)." + }, + "schema": { + "type": "string", + "description": "Name of the schema." + } + }, + "required": [ + "host" + ] + }, "PrestoServer": { "type": "object", "title": "PrestoServer", diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index 20d13c9..a4df61c 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -232,7 +232,7 @@ "description": "Type of the server.", "enum": [ "api", "athena", "azure", "bigquery", "clickhouse", "databricks", "denodo", "dremio", - "duckdb", "glue", "cloudsql", "db2", "hive", "impala", "informix", "kafka", "kinesis", "local", + "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "impala", "informix", "kafka", "kinesis", "local", "mysql", "oracle", "postgresql", "postgres", "presto", "pubsub", "redshift", "s3", "sftp", "snowflake", "sqlserver", "synapse", "trino", "vertica", "zen", "custom" ] @@ -544,6 +544,19 @@ "$ref": "#/$defs/ServerSource/OracleServer" } }, + { + "if": { + "properties": { + "type": { + "const": "hana" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/HanaServer" + } + }, { "if": { "properties": { @@ -1402,6 +1415,31 @@ "schema" ] }, + "HanaServer": { + "type": "object", + "title": "HanaServer", + "properties": { + "host": { + "type": "string", + "description": "Host of the HANA server." + }, + "port": { + "type": "integer", + "description": "Port of the HANA server." + }, + "database": { + "type": "string", + "description": "Name of the database (tenant)." + }, + "schema": { + "type": "string", + "description": "Name of the schema." + } + }, + "required": [ + "host" + ] + }, "PrestoServer": { "type": "object", "title": "PrestoServer", diff --git a/src/script/negative-tests/hana-missing-host.odcs.yaml b/src/script/negative-tests/hana-missing-host.odcs.yaml new file mode 100644 index 0000000..18cdc0b --- /dev/null +++ b/src/script/negative-tests/hana-missing-host.odcs.yaml @@ -0,0 +1,19 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0045 negative test: a `hana` server requires `host`. Here it is omitted, +# so the schema MUST reject this contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 6e9c3b2d-8f4a-4b7e-9c1d-2f3a4b5c6d7e +status: active +servers: + - server: prod + type: hana + port: 30015 + database: HXE +schema: + - name: sales + physicalType: table From 3e6427733956d4bcbf3abe96e63e8e897c5eba23 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 30 Jun 2026 15:36:17 -0400 Subject: [PATCH 20/29] feat(rfc-0046): allow customProperties and authoritativeDefinitions on SLA properties (#299) Implements approved RFC-0046 for ODCS v3.2.0. Adds optional `customProperties` and `authoritativeDefinitions` (reusing the existing $defs) to each slaProperties[] entry, consistent with other ODCS objects. - schema: both fields on ServiceLevelAgreementProperty in both latest + v3.2.0 (lockstep) - docs: two rows in service-level-agreement.md - example: docs/examples/sla/sla-extensions.odcs.yaml (validated) - negative test: sla-authoritativedefinition-missing-type.odcs.yaml (schema rejects: authoritativeDefinitions entry missing required `type`) - CHANGELOG entry Approved RFC: https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0046-sla-custom-properties-and-authoritative-definitions.md Claude-Session: https://claude.ai/code/session_01Adou6Wv1WCxD3JzmMkewvY Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 3 ++ docs/examples/sla/sla-extensions.odcs.yaml | 31 +++++++++++++++++++ docs/service-level-agreement.md | 2 ++ schema/odcs-json-schema-latest.json | 6 ++++ schema/odcs-json-schema-v3.2.0.json | 6 ++++ ...oritativedefinition-missing-type.odcs.yaml | 19 ++++++++++++ 6 files changed, 67 insertions(+) create mode 100644 docs/examples/sla/sla-extensions.odcs.yaml create mode 100644 src/script/negative-tests/sla-authoritativedefinition-missing-type.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index bcb35cb..455736d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,9 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * **Adds** SAP HANA server type ([RFC 0045](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0045-hana-server-type.md)): * New `hana` server `type` for SAP HANA, with required `host` plus optional `port`, `database` (tenant), and `schema`. * Non-breaking: adds a new optional server type. +* **Adds** SLA custom properties and authoritative definitions ([RFC 0046](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0046-sla-custom-properties-and-authoritative-definitions.md)): + * Each `slaProperties[]` entry may now carry optional `customProperties` and `authoritativeDefinitions`, consistent with other ODCS objects. + * Non-breaking: both fields are optional. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/sla/sla-extensions.odcs.yaml b/docs/examples/sla/sla-extensions.odcs.yaml new file mode 100644 index 0000000..cf38737 --- /dev/null +++ b/docs/examples/sla/sla-extensions.odcs.yaml @@ -0,0 +1,31 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0046: customProperties and authoritativeDefinitions on +# individual slaProperties entries. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 7f2a9c4d-1b6e-4a8d-9c0f-3e4a5b6c7d8e +status: active +name: my_table +schema: [] +slaProperties: + - property: availability + value: 99.9 + unit: "%" + customProperties: + - property: alertPolicyId + value: pol-123 + - property: latency + value: 100 + unit: ms + authoritativeDefinitions: + - url: https://wiki.acme.com/sla/latency + type: businessDefinition + customProperties: + - property: alertPolicyId + value: pol-123 + - property: owner + value: data-platform-team diff --git a/docs/service-level-agreement.md b/docs/service-level-agreement.md index d4a7be8..035c3d3 100644 --- a/docs/service-level-agreement.md +++ b/docs/service-level-agreement.md @@ -68,6 +68,8 @@ slaProperties: | ---------------------------------- | ------ | ----------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ~~slaDefaultElement~~ (Deprecated) | string | Default SLA element(s) | No | Element (using the element path notation) to do the checks on. DEPRECATED SINCE v3.1.0. WILL BE REMOVED IN ODCS v4.0.0. | | slaProperties | array | SLA | No | A list of key/value pairs for SLA specific properties. There is no limit on the type of properties. | +| slaProperties[].authoritativeDefinitions | array | Authoritative Definitions | No | Links to external definitions for this SLA property (e.g. the formal SLA document or measurement method). Same structure as elsewhere in ODCS. (Added in v3.2.0, [RFC 0046](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0046-sla-custom-properties-and-authoritative-definitions.md).) | +| slaProperties[].customProperties | array | Custom Properties | No | Vendor- or organization-specific key/value pairs for this SLA property. Same structure as elsewhere in ODCS. (Added in v3.2.0, [RFC 0046](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0046-sla-custom-properties-and-authoritative-definitions.md).) | | slaProperties[].description | string | Description | No | Description of the SLA for humans. | | slaProperties[].driver | string | Driver | No | Describes the importance of the SLA from the list of: `regulatory`, `analytics`, or `operational`. | | slaProperties[].element | string | Element(s) | No | Element(s) to check on. Multiple elements should be extremely rare and, if so, separated by commas. | diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 3887499..e5e6ef9 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -2976,6 +2976,12 @@ "type": "string", "description": "Configuration information for the scheduling tool, for cron a possible value is 0 20 * * *.", "examples": ["0 20 * * *"] + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" } }, "required": ["property", "value"], diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index a4df61c..f6f4569 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -2975,6 +2975,12 @@ "type": "string", "description": "Configuration information for the scheduling tool, for cron a possible value is 0 20 * * *.", "examples": ["0 20 * * *"] + }, + "customProperties": { + "$ref": "#/$defs/CustomProperties" + }, + "authoritativeDefinitions": { + "$ref": "#/$defs/AuthoritativeDefinitions" } }, "required": ["property", "value"], diff --git a/src/script/negative-tests/sla-authoritativedefinition-missing-type.odcs.yaml b/src/script/negative-tests/sla-authoritativedefinition-missing-type.odcs.yaml new file mode 100644 index 0000000..d29a9c9 --- /dev/null +++ b/src/script/negative-tests/sla-authoritativedefinition-missing-type.odcs.yaml @@ -0,0 +1,19 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0046 negative test: an slaProperties[].authoritativeDefinitions entry +# requires both `url` and `type`. Here `type` is omitted, so the schema MUST +# reject this contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 8a3b0c5e-2c7f-4b9e-8d1a-4f5b6c7d8e9f +status: active +schema: [] +slaProperties: + - property: latency + value: 100 + unit: ms + authoritativeDefinitions: + - url: https://wiki.acme.com/sla/latency From 28a06f50043b464dd881b28a23cf668ccc2cd80c Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Mon, 13 Jul 2026 08:31:14 -0400 Subject: [PATCH 21/29] =?UTF-8?q?feat(rfc-0050):=20variables=20=E2=80=94?= =?UTF-8?q?=20${VAR=5FNAME}=20interpolation,=20incl.=20POSIX=20defaults=20?= =?UTF-8?q?(#300)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies approved RFC-0050 (Variables) to ODCS v3.2.0: any string value MAY contain ${VAR_NAME} references resolved at runtime by tooling, with the POSIX ${VAR_NAME:-default} form for inline defaults (Option B, TSC 2026-07-13). No schema change — the RFC adds no section or field. New docs page, example, and a negative test (variable token in an integer-typed field is rejected). Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 5 ++ docs/README.md | 1 + .../examples/fundamentals/variables.odcs.yaml | 35 +++++++++++ docs/variables.md | 59 +++++++++++++++++++ mkdocs.yml | 1 + .../variables-in-integer-field.odcs.yaml | 23 ++++++++ 6 files changed, 124 insertions(+) create mode 100644 docs/examples/fundamentals/variables.odcs.yaml create mode 100644 docs/variables.md create mode 100644 src/script/negative-tests/variables-in-integer-field.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 455736d..60bee1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,11 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * **Adds** SLA custom properties and authoritative definitions ([RFC 0046](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0046-sla-custom-properties-and-authoritative-definitions.md)): * Each `slaProperties[]` entry may now carry optional `customProperties` and `authoritativeDefinitions`, consistent with other ODCS objects. * Non-breaking: both fields are optional. +* **Adds** Variables ([RFC 0050](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0050-variables.md), shared with ODPS v1.1.0 and OORS v1.0.0): + * Any string value in a contract MAY contain `${VAR_NAME}` references, resolved at runtime by tooling, keeping secrets and environment-specific values (hostnames, bucket paths, credentials) out of the document itself. + * The POSIX `${VAR_NAME:-default}` form supplies an inline default, used when the variable is unset or empty. + * Tools MUST resolve references before using a value, SHOULD error on unresolvable references (never silently substitute an empty string), and MUST preserve unresolved tokens verbatim when serializing back to YAML. + * Non-breaking: no new section or field is added to the standard; interpolation applies to string values only. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/README.md b/docs/README.md index e6fa6fc..02bb3d4 100644 --- a/docs/README.md +++ b/docs/README.md @@ -31,6 +31,7 @@ For more details, see the sections below: 12. [Custom & Other Properties](./custom-other-properties.md) 13. [Authoritative Definitions](./authoritative-definitions.md) 14. [Tags](./tags.md) +15. [Variables](./variables.md) ## Notes diff --git a/docs/examples/fundamentals/variables.odcs.yaml b/docs/examples/fundamentals/variables.odcs.yaml new file mode 100644 index 0000000..86a8705 --- /dev/null +++ b/docs/examples/fundamentals/variables.odcs.yaml @@ -0,0 +1,35 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0050 example: any string value MAY contain ${VAR_NAME} references, +# resolved at runtime by tooling. The POSIX ${VAR_NAME:-default} form supplies +# an inline default used when the variable is unset or empty. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 4c1cbb2a-1c86-4e0e-8f0e-6d7c9b1a5e42 +status: active +name: orders + +servers: + - server: prod_db + environment: prod + type: postgresql + host: ${DB_HOST} + port: 5432 + database: ${DB_NAME:-orders} + schema: ${DB_SCHEMA:-public} + +schema: + - name: orders + physicalType: table + properties: + - name: order_id + logicalType: string + physicalType: varchar(36) + quality: + - type: sql + query: "SELECT COUNT(*) FROM ${TARGET_TABLE} WHERE created_at > ${CUTOFF_DATE}" + mustBe: 0 + description: Variable references are valid in any string value, including as substrings. diff --git a/docs/variables.md b/docs/variables.md new file mode 100644 index 0000000..69585a0 --- /dev/null +++ b/docs/variables.md @@ -0,0 +1,59 @@ +--- +title: "Variables" +description: "Variable interpolation in ODCS: keep secrets and environment-specific values out of the data contract with ${VAR_NAME} references resolved at runtime by tooling." +--- + + + +# Variables + +Any string value in a data contract MAY contain one or more variable references of the form `${VAR_NAME}`, resolved at runtime by tooling ([RFC 0050](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0050-variables.md), shared with ODPS and OORS). This keeps secrets and environment-specific values — hostnames, bucket paths, credentials — out of the contract itself, so the same contract works across environments and is safe to store in version control. + +References MAY appear as a whole value or as a substring. A reference MAY carry an inline default using the POSIX `${VAR_NAME:-default}` form: the text between `:-` and the closing `}` is used verbatim when the variable is unset or empty. + +`VAR_NAME` is an identifier chosen by the contract author. Resolution is intentionally left to tooling; common sources include OS environment variables, `.env` files, secret managers, and CI/CD pipeline variables. + +## Example + +```yaml +servers: + - server: prod_db + environment: prod + type: postgresql + host: ${DB_HOST} + port: 5432 + database: ${DB_NAME:-orders} + schema: ${DB_SCHEMA:-public} +``` + +A variable reference is valid in any string value, not just `servers`, and MAY appear as a substring of a larger value — for example in a quality rule query: + +```yaml +quality: + - type: sql + query: "SELECT COUNT(*) FROM ${TARGET_TABLE} WHERE created_at > ${CUTOFF_DATE}" +``` + +## Syntax + +| Form | Behavior when the variable is set | Behavior when the variable is unset or empty | +|-------------------------|-----------------------------------|----------------------------------------------| +| `${VAR_NAME}` | Replaced by the variable's value. | Tooling SHOULD surface an error; it MUST NOT silently substitute an empty string. | +| `${VAR_NAME:-default}` | Replaced by the variable's value. | Replaced by `default`, verbatim. | + +## Tooling behavior + +* Tools MUST resolve `${VAR_NAME}` references before using the value for any purpose. +* If a referenced variable cannot be resolved (and no default is supplied), tools SHOULD surface an error and MUST NOT silently substitute an empty string. +* Tools MUST preserve unresolved `${VAR_NAME}` and `${VAR_NAME:-default}` tokens verbatim when serializing a contract back to YAML (round-trip safety). +* Tools MAY define their own resolution order across sources (for example, OS environment variable before `.env` file). + +## Notes + +* Interpolation applies to **string** values only. A field typed as an integer or boolean in the JSON schema (such as a server `port`) cannot hold a variable reference: the unresolved token is a string and the schema rejects it. +* No new section or field is added to the standard: a contract using variables validates against the standard JSON schema as-is. + +[Back to TOC](README.md) diff --git a/mkdocs.yml b/mkdocs.yml index f9cfd83..093f301 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -71,6 +71,7 @@ nav: - Custom: 'custom-other-properties.md' - Authoritative Definitions: 'authoritative-definitions.md' - Tags: 'tags.md' + - Variables: 'variables.md' - Examples: - 'examples/README.md' - '...' diff --git a/src/script/negative-tests/variables-in-integer-field.odcs.yaml b/src/script/negative-tests/variables-in-integer-field.odcs.yaml new file mode 100644 index 0000000..cd971f1 --- /dev/null +++ b/src/script/negative-tests/variables-in-integer-field.odcs.yaml @@ -0,0 +1,23 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0050 negative test: variable interpolation applies to STRING values only. +# A `postgresql` server `port` is typed as an integer, so the unresolved +# ${DB_PORT} token (a string) MUST be rejected by the schema. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 9d8f1c3a-5b2e-4f7d-a6c4-0e1b2d3c4f5a +status: active +servers: + - server: prod_db + environment: prod + type: postgresql + host: ${DB_HOST} + port: ${DB_PORT} + database: orders + schema: public +schema: + - name: orders + physicalType: table From 76e124e153ca44a28cf9a628c6ea54d75cb3c7b9 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 14 Jul 2026 11:57:19 -0400 Subject: [PATCH 22/29] feat(rfc-0047): add optional `id` to relationship objects (#302) Applies approved RFC-0047 to ODCS v3.2.0: a new optional `id` string on RelationshipBase (surfaced on schema-level and property-level relationships), completing RFC-0026a's stable-identifier coverage. Unique within its relationships array; SHOULD be stable; cannot contain . # / \ @ ! % & ^. Both schema files updated in lockstep; references.md field table + example extended. No negative-test fixture: the RFC adds only an optional string with no schema-enforced pattern/uniqueness (id-format rules are textual, like every other ODCS id), so there is no schema-level violation to assert. Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 4 ++++ docs/examples/references/relationships.odcs.yaml | 11 +++++++---- docs/references.md | 1 + schema/odcs-json-schema-latest.json | 4 ++++ schema/odcs-json-schema-v3.2.0.json | 4 ++++ 5 files changed, 20 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 60bee1c..bb5f35e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,10 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * The POSIX `${VAR_NAME:-default}` form supplies an inline default, used when the variable is unset or empty. * Tools MUST resolve references before using a value, SHOULD error on unresolvable references (never silently substitute an empty string), and MUST preserve unresolved tokens verbatim when serializing back to YAML. * Non-breaking: no new section or field is added to the standard; interpolation applies to string values only. +* **Adds** `id` to relationship objects ([RFC 0047](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0047-relationship-id.md)): + * New optional `id` string on `RelationshipBase` (surfaced on both schema-level and property-level relationships), completing the stable-identifier work of RFC-0026a for the last referenceable array-item object that lacked one. + * MUST be unique within its containing `relationships` array; SHOULD be stable across contract versions; cannot contain `.` `#` `/` `\` `@` `!` `%` `&` `^`. + * Non-breaking, as `id` is optional. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/references/relationships.odcs.yaml b/docs/examples/references/relationships.odcs.yaml index 4e2095a..a160f4b 100644 --- a/docs/examples/references/relationships.odcs.yaml +++ b/docs/examples/references/relationships.odcs.yaml @@ -8,7 +8,7 @@ apiVersion: v3.2.0 status: active name: orders_with_relationships description: - purpose: Demonstrate property-level, schema-level, composite, and shorthand foreign-key relationships (RFC 0026). + purpose: Demonstrate property-level, schema-level, composite, and shorthand foreign-key relationships (RFC 0026), each carrying a stable relationship id (RFC 0047). schema: - id: customers_obj @@ -62,7 +62,8 @@ schema: logicalType: string physicalType: VARCHAR(36) relationships: - - to: customers.id + - id: fk_order_customer + to: customers.id type: foreignKey customProperties: - property: cardinality @@ -73,14 +74,16 @@ schema: name: shipping_address_street logicalType: string relationships: - - to: addresses.address_street + - id: fk_order_shipping_address + to: addresses.address_street type: foreignKey # Schema-level relationships: explicit `from` and `to`, # including a composite (multi-column) foreign key. # Both sides use the shorthand `schema.property` notation. relationships: - - type: foreignKey + - id: fk_orders_address_composite + type: foreignKey from: ['orders.customer_id', 'orders.shipping_address_street'] to: ['addresses.customer_id', 'addresses.address_street'] customProperties: diff --git a/docs/references.md b/docs/references.md index 67adcc7..bc6bd9c 100644 --- a/docs/references.md +++ b/docs/references.md @@ -121,6 +121,7 @@ Relationships can be defined in two ways: | -------------------------------- | ------ | ----------------- | ----------------- | --------------------------------------------------------------------------------- | | relationships | array | Relationships | No | Array of relationship definitions | | relationships[].from | string | From | Context-dependent | Source property reference - Required at schema level, forbidden at property level | +| relationships[].id | string | ID | No | Optional stable identifier for the relationship, unique within its containing `relationships` array. Recommended for elements that will be referenced. Cannot contain: `.` `#` `/` `\` `@` `!` `%` `&` `^` (RFC 0047) | | relationships[].to | string | To | Yes | Target property reference using `schema.property` notation | | relationships[].type | string | Type | No | Type of relationship (defaults to `foreignKey`) | | relationships[].customProperties | array | Custom Properties | No | Additional metadata about the relationship | diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index e5e6ef9..1798e31 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -3146,6 +3146,10 @@ "type": "object", "description": "Base definition for relationships between properties, typically for foreign key constraints.", "properties": { + "id": { + "type": "string", + "description": "Optional stable identifier for this relationship, unique within its containing relationships array. SHOULD remain stable across contract versions. Cannot contain: . # / \\ @ ! % & ^" + }, "type": { "type": "string", "description": "The type of relationship. Defaults to 'foreignKey'.", diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index f6f4569..f0bc4f5 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -3145,6 +3145,10 @@ "type": "object", "description": "Base definition for relationships between properties, typically for foreign key constraints.", "properties": { + "id": { + "type": "string", + "description": "Optional stable identifier for this relationship, unique within its containing relationships array. SHOULD remain stable across contract versions. Cannot contain: . # / \\ @ ! % & ^" + }, "type": { "type": "string", "description": "The type of relationship. Defaults to 'foreignKey'.", From 7357f056b7a53caf28a1e66ae1b91f78f30fa19d Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 14 Jul 2026 12:01:24 -0400 Subject: [PATCH 23/29] feat(rfc-0049): add `iceberg` server type (#303) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies approved RFC-0049 to ODCS v3.2.0: new `iceberg` server type for Apache Iceberg catalogs via the Iceberg REST API. Both schema files updated in lockstep — enum value, conditional dispatch, and IcebergServer def with required catalog + catalogUrl (per the RFC field table / TSC decision) and optional namespace + warehouse. Docs section, positive example, and a negative test (missing catalogUrl → rejected). Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 3 ++ docs/examples/server/iceberg-server.odcs.yaml | 25 +++++++++++ docs/infrastructure-servers.md | 11 +++++ schema/odcs-json-schema-latest.json | 45 ++++++++++++++++++- schema/odcs-json-schema-v3.2.0.json | 45 ++++++++++++++++++- .../iceberg-missing-catalogurl.odcs.yaml | 20 +++++++++ 6 files changed, 147 insertions(+), 2 deletions(-) create mode 100644 docs/examples/server/iceberg-server.odcs.yaml create mode 100644 src/script/negative-tests/iceberg-missing-catalogurl.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index bb5f35e..ffac301 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -59,6 +59,9 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * New optional `id` string on `RelationshipBase` (surfaced on both schema-level and property-level relationships), completing the stable-identifier work of RFC-0026a for the last referenceable array-item object that lacked one. * MUST be unique within its containing `relationships` array; SHOULD be stable across contract versions; cannot contain `.` `#` `/` `\` `@` `!` `%` `&` `^`. * Non-breaking, as `id` is optional. +* **Adds** Apache Iceberg server type ([RFC 0049](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0049-iceberg-server-type.md)): + * New `iceberg` server `type` describing access to Apache Iceberg catalogs through the standardized Iceberg REST API, with required `catalog` and `catalogUrl` plus optional `namespace` and `warehouse`. + * Non-breaking: adds a new optional server type. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/server/iceberg-server.odcs.yaml b/docs/examples/server/iceberg-server.odcs.yaml new file mode 100644 index 0000000..ea01d93 --- /dev/null +++ b/docs/examples/server/iceberg-server.odcs.yaml @@ -0,0 +1,25 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0049: the `iceberg` server type for Apache Iceberg catalogs +# accessed through the standardized Iceberg REST API. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 7f2e4c91-3a6d-4b8e-9c0f-1d2e3f4a5b6c +status: active +servers: + - server: prod + type: iceberg + catalog: prod_catalog + catalogUrl: https://catalog.acme.com + namespace: acme_db.sales + warehouse: s3://acme-warehouse/sales/ +schema: + - name: sales + physicalType: table + properties: + - name: sale_id + logicalType: string + required: true diff --git a/docs/infrastructure-servers.md b/docs/infrastructure-servers.md index b8cf638..d616d6d 100644 --- a/docs/infrastructure-servers.md +++ b/docs/infrastructure-servers.md @@ -180,6 +180,17 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | host | string | Host | Yes | The host to the Hive server. | | port | integer | Port | No | The port to the Hive server. Defaults to 10000. | +### Apache Iceberg + +[Apache Iceberg](https://iceberg.apache.org/) is an open table format for large analytic datasets, accessed through the standardized Iceberg REST catalog API (Polaris, S3 Tables, Nessie, Unity Catalog, Glue, etc.). Added in ODCS v3.2.0 ([RFC 0049](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0049-iceberg-server-type.md)). + +| Key | Type | UX Label | Required | Description | +| ---------- | ------ | ----------- | -------- | ----------------------------------------------------------------------------------------------------------- | +| catalog | string | Catalog | Yes | Catalog name as registered in the query engine or catalog service (e.g. `my_catalog`). | +| catalogUrl | string | Catalog URL | Yes | URL of the Iceberg compatible REST catalog service (Polaris, S3 Tables, Nessie, Unity Catalog, Glue, etc.). | +| namespace | string | Namespace | No | Dot-separated namespace path within the catalog (e.g. `db.schema` or just `db`). | +| warehouse | string | Warehouse | No | Base storage location of the warehouse (e.g. `s3://my-bucket/warehouse/`). | + ### Apache Impala [Apache Impala](https://impala.apache.org/) is a massively parallel processing (MPP) SQL query engine for data stored in Apache Hadoop clusters. Impala provides high-performance, low-latency SQL queries on data stored in HDFS and Apache HBase, enabling interactive exploration and analytics without data movement or transformation. diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 1798e31..1063ef6 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -233,7 +233,7 @@ "description": "Type of the server.", "enum": [ "api", "athena", "azure", "bigquery", "clickhouse", "databricks", "denodo", "dremio", - "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "impala", "informix", "kafka", "kinesis", "local", + "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "iceberg", "impala", "informix", "kafka", "kinesis", "local", "mysql", "oracle", "postgresql", "postgres", "presto", "pubsub", "redshift", "s3", "sftp", "snowflake", "sqlserver", "synapse", "trino", "vertica", "zen", "custom" ] @@ -558,6 +558,19 @@ "$ref": "#/$defs/ServerSource/HanaServer" } }, + { + "if": { + "properties": { + "type": { + "const": "iceberg" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/IcebergServer" + } + }, { "if": { "properties": { @@ -1441,6 +1454,36 @@ "host" ] }, + "IcebergServer": { + "type": "object", + "title": "IcebergServer", + "properties": { + "catalog": { + "type": "string", + "description": "Catalog name as registered in the query engine or catalog service.", + "examples": ["my_catalog"] + }, + "catalogUrl": { + "type": "string", + "format": "uri", + "description": "URL of the Iceberg compatible REST catalog service (Polaris, S3 Tables, Nessie, Unity Catalog, Glue, etc.)", + "examples": ["https://my-catalog-service.com"] + }, + "namespace": { + "type": "string", + "description": "Dot-separated namespace/schema path within the catalog (e.g. db.schema or just db)." + }, + "warehouse": { + "type": "string", + "description": "Logical catalog or base location for the catalog to use.", + "examples": ["s3://my-bucket/warehouse/sales/", "production", "development"] + } + }, + "required": [ + "catalog", + "catalogUrl" + ] + }, "PrestoServer": { "type": "object", "title": "PrestoServer", diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index f0bc4f5..63ad8a5 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -232,7 +232,7 @@ "description": "Type of the server.", "enum": [ "api", "athena", "azure", "bigquery", "clickhouse", "databricks", "denodo", "dremio", - "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "impala", "informix", "kafka", "kinesis", "local", + "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "iceberg", "impala", "informix", "kafka", "kinesis", "local", "mysql", "oracle", "postgresql", "postgres", "presto", "pubsub", "redshift", "s3", "sftp", "snowflake", "sqlserver", "synapse", "trino", "vertica", "zen", "custom" ] @@ -557,6 +557,19 @@ "$ref": "#/$defs/ServerSource/HanaServer" } }, + { + "if": { + "properties": { + "type": { + "const": "iceberg" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/IcebergServer" + } + }, { "if": { "properties": { @@ -1440,6 +1453,36 @@ "host" ] }, + "IcebergServer": { + "type": "object", + "title": "IcebergServer", + "properties": { + "catalog": { + "type": "string", + "description": "Catalog name as registered in the query engine or catalog service.", + "examples": ["my_catalog"] + }, + "catalogUrl": { + "type": "string", + "format": "uri", + "description": "URL of the Iceberg compatible REST catalog service (Polaris, S3 Tables, Nessie, Unity Catalog, Glue, etc.)", + "examples": ["https://my-catalog-service.com"] + }, + "namespace": { + "type": "string", + "description": "Dot-separated namespace/schema path within the catalog (e.g. db.schema or just db)." + }, + "warehouse": { + "type": "string", + "description": "Logical catalog or base location for the catalog to use.", + "examples": ["s3://my-bucket/warehouse/sales/", "production", "development"] + } + }, + "required": [ + "catalog", + "catalogUrl" + ] + }, "PrestoServer": { "type": "object", "title": "PrestoServer", diff --git a/src/script/negative-tests/iceberg-missing-catalogurl.odcs.yaml b/src/script/negative-tests/iceberg-missing-catalogurl.odcs.yaml new file mode 100644 index 0000000..a235918 --- /dev/null +++ b/src/script/negative-tests/iceberg-missing-catalogurl.odcs.yaml @@ -0,0 +1,20 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0049 negative test: an `iceberg` server requires both `catalog` and +# `catalogUrl`. Here `catalogUrl` is omitted, so the schema MUST reject this +# contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 8a3f5d02-4b7e-4c9f-a0d1-2e3f4a5b6c7d +status: active +servers: + - server: prod + type: iceberg + catalog: prod_catalog + namespace: acme_db.sales +schema: + - name: sales + physicalType: table From cb036d24865fe5302f9822f75c08cef830de0fff Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Tue, 14 Jul 2026 12:02:11 -0400 Subject: [PATCH 24/29] docs(context): restore authoritativeDefinitions example (ontology/glossary/taxonomy) (#304) The contract-level context example in docs/context.md documented authoritativeDefinitions in the Definitions table but no longer showed it in use. Re-add the Ontology/Glossary/Taxonomy authoritativeDefinitions block on a constraint, matching the approved RFC-0038 example. Co-authored-by: Claude Fable 5 --- docs/context.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/context.md b/docs/context.md index 3dc89f8..2c550f9 100644 --- a/docs/context.md +++ b/docs/context.md @@ -40,6 +40,16 @@ context: - id: no-individual-order-exposure constraint: "Do not expose individual order details; aggregate to at least country level." tags: ['gdpr', 'pii'] + authoritativeDefinitions: + - url: https://example.com/MyGlobalAndMarvelousOntology + type: Ontology + description: Link to the ontology + - url: https://example.com/MySpecificAndWonderfulGlossary + type: Glossary + description: Link to the glossary + - url: https://example.com/OneOfManyTaxonomy + type: Taxonomy + description: Link to the taxonomy - constraint: "Do not join with customer PII tables without explicit data access approval." ``` From acb7f79c4594f72483f1fba0d401eb1cb16793c2 Mon Sep 17 00:00:00 2001 From: Jochen Christ Date: Sun, 19 Jul 2026 18:15:58 +0200 Subject: [PATCH 25/29] feat(rfc-0050): allow string values in server port fields (#307) The variables RFC allows ${VAR_NAME} references in string values only, which made it impossible to parameterize server ports: the unresolved token is a string and the integer-typed port rejected it. Relax all 19 server port definitions via a shared Port $def typed as integer or string, so a port can hold a variable reference such as ${DB_PORT} or ${DB_PORT:-5432}. Discussed in the bitol-wg Slack thread; other integer fields remain integer-only. --- CHANGELOG.md | 1 + .../examples/fundamentals/variables.odcs.yaml | 2 +- docs/variables.md | 5 ++- schema/odcs-json-schema-latest.json | 42 ++++++++++--------- schema/odcs-json-schema-v3.2.0.json | 42 ++++++++++--------- .../variables-in-integer-field.odcs.yaml | 12 +++++- 6 files changed, 61 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffac301..3138519 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,7 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * The POSIX `${VAR_NAME:-default}` form supplies an inline default, used when the variable is unset or empty. * Tools MUST resolve references before using a value, SHOULD error on unresolvable references (never silently substitute an empty string), and MUST preserve unresolved tokens verbatim when serializing back to YAML. * Non-breaking: no new section or field is added to the standard; interpolation applies to string values only. + * Server `port` fields now accept a string in addition to an integer, so they can hold a variable reference such as `${DB_PORT}` or `${DB_PORT:-5432}`, which the previous integer-only type rejected. * **Adds** `id` to relationship objects ([RFC 0047](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0047-relationship-id.md)): * New optional `id` string on `RelationshipBase` (surfaced on both schema-level and property-level relationships), completing the stable-identifier work of RFC-0026a for the last referenceable array-item object that lacked one. * MUST be unique within its containing `relationships` array; SHOULD be stable across contract versions; cannot contain `.` `#` `/` `\` `@` `!` `%` `&` `^`. diff --git a/docs/examples/fundamentals/variables.odcs.yaml b/docs/examples/fundamentals/variables.odcs.yaml index 86a8705..a16567f 100644 --- a/docs/examples/fundamentals/variables.odcs.yaml +++ b/docs/examples/fundamentals/variables.odcs.yaml @@ -17,7 +17,7 @@ servers: environment: prod type: postgresql host: ${DB_HOST} - port: 5432 + port: ${DB_PORT:-5432} database: ${DB_NAME:-orders} schema: ${DB_SCHEMA:-public} diff --git a/docs/variables.md b/docs/variables.md index 69585a0..28b814b 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -24,7 +24,7 @@ servers: environment: prod type: postgresql host: ${DB_HOST} - port: 5432 + port: ${DB_PORT:-5432} database: ${DB_NAME:-orders} schema: ${DB_SCHEMA:-public} ``` @@ -53,7 +53,8 @@ quality: ## Notes -* Interpolation applies to **string** values only. A field typed as an integer or boolean in the JSON schema (such as a server `port`) cannot hold a variable reference: the unresolved token is a string and the schema rejects it. +* Interpolation applies to **string** values only. A field typed as an integer or boolean in the JSON schema cannot hold a variable reference: the unresolved token is a string and the schema rejects it. +* The server `port` is an exception: the schema accepts an integer or a string, so it can hold a variable reference such as `${DB_PORT}` or `${DB_PORT:-5432}`. * No new section or field is added to the standard: a contract using variables validates against the standard JSON schema as-is. [Back to TOC](README.md) diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 1063ef6..9ebf72e 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -217,6 +217,10 @@ "description": "Stable technical identifier for references. Must be unique within its containing array. Cannot contain special characters ('-', '_' allowed).", "pattern": "^[A-Za-z0-9_-]+$" }, + "Port": { + "type": ["integer", "string"], + "description": "A network port: an integer, or a string, e.g. to hold a variable reference such as ${DB_PORT} resolved at runtime (RFC 0050)." + }, "Server": { "type": "object", "description": "Data source details of where data is physically stored.", @@ -859,7 +863,7 @@ "description": "The host of the ClickHouse server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the ClickHouse server." }, "database": { @@ -907,7 +911,7 @@ "description": "The host of the Denodo server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Denodo server." }, "database": { @@ -929,7 +933,7 @@ "description": "The host of the Dremio server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Dremio server." }, "schema": { @@ -1020,7 +1024,7 @@ "description": "The host of the Google Cloud Sql server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Google Cloud Sql server." }, "database": { @@ -1048,7 +1052,7 @@ "description": "The host of the IBM DB2 server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the IBM DB2 server." }, "database": { @@ -1075,7 +1079,7 @@ "description": "The host to the Hive server. " }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Hive server. Defaults to 10000." }, "database": { @@ -1097,7 +1101,7 @@ "description": "The host to the Impala server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Impala server. Defaults to 21050." }, "database": { @@ -1119,7 +1123,7 @@ "description": "The host to the Informix server. " }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Informix server. Defaults to 9088." }, "database": { @@ -1141,7 +1145,7 @@ "description": "Hostname or IP address of the Zen server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "Zen server SQL connections port. Defaults to 1583." }, "database": { @@ -1208,7 +1212,7 @@ "description": "Relative or absolute path to the data file(s)." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "Port to the server. No default value is assumed for custom servers." }, "project": { @@ -1355,7 +1359,7 @@ "description": "The host of the MySql server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the MySql server." }, "database": { @@ -1381,7 +1385,7 @@ ] }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the oracle server.", "examples": [ 1523 @@ -1410,7 +1414,7 @@ "description": "The host to the Postgres server" }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Postgres server." }, "database": { @@ -1438,7 +1442,7 @@ "description": "Host of the HANA server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "Port of the HANA server." }, "database": { @@ -1664,7 +1668,7 @@ "description": "The host to the Snowflake server" }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Snowflake server." }, "account": { @@ -1702,7 +1706,7 @@ ] }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the database server.", "default": 1433, "examples": [ @@ -1739,7 +1743,7 @@ "description": "The host of the Synapse server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Synapse server." }, "database": { @@ -1765,7 +1769,7 @@ ] }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The Trino port." }, "catalog": { @@ -1799,7 +1803,7 @@ "description": "The host of the Vertica server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Vertica server." }, "database": { diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index 63ad8a5..4235b52 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -216,6 +216,10 @@ "description": "Stable technical identifier for references. Must be unique within its containing array. Cannot contain special characters ('-', '_' allowed).", "pattern": "^[A-Za-z0-9_-]+$" }, + "Port": { + "type": ["integer", "string"], + "description": "A network port: an integer, or a string, e.g. to hold a variable reference such as ${DB_PORT} resolved at runtime (RFC 0050)." + }, "Server": { "type": "object", "description": "Data source details of where data is physically stored.", @@ -858,7 +862,7 @@ "description": "The host of the ClickHouse server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the ClickHouse server." }, "database": { @@ -906,7 +910,7 @@ "description": "The host of the Denodo server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Denodo server." }, "database": { @@ -928,7 +932,7 @@ "description": "The host of the Dremio server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Dremio server." }, "schema": { @@ -1019,7 +1023,7 @@ "description": "The host of the Google Cloud Sql server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Google Cloud Sql server." }, "database": { @@ -1047,7 +1051,7 @@ "description": "The host of the IBM DB2 server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the IBM DB2 server." }, "database": { @@ -1074,7 +1078,7 @@ "description": "The host to the Hive server. " }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Hive server. Defaults to 10000." }, "database": { @@ -1096,7 +1100,7 @@ "description": "The host to the Impala server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Impala server. Defaults to 21050." }, "database": { @@ -1118,7 +1122,7 @@ "description": "The host to the Informix server. " }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Informix server. Defaults to 9088." }, "database": { @@ -1140,7 +1144,7 @@ "description": "Hostname or IP address of the Zen server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "Zen server SQL connections port. Defaults to 1583." }, "database": { @@ -1207,7 +1211,7 @@ "description": "Relative or absolute path to the data file(s)." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "Port to the server. No default value is assumed for custom servers." }, "project": { @@ -1354,7 +1358,7 @@ "description": "The host of the MySql server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the MySql server." }, "database": { @@ -1380,7 +1384,7 @@ ] }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the oracle server.", "examples": [ 1523 @@ -1409,7 +1413,7 @@ "description": "The host to the Postgres server" }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Postgres server." }, "database": { @@ -1437,7 +1441,7 @@ "description": "Host of the HANA server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "Port of the HANA server." }, "database": { @@ -1663,7 +1667,7 @@ "description": "The host to the Snowflake server" }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the Snowflake server." }, "account": { @@ -1701,7 +1705,7 @@ ] }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port to the database server.", "default": 1433, "examples": [ @@ -1738,7 +1742,7 @@ "description": "The host of the Synapse server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Synapse server." }, "database": { @@ -1764,7 +1768,7 @@ ] }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The Trino port." }, "catalog": { @@ -1798,7 +1802,7 @@ "description": "The host of the Vertica server." }, "port": { - "type": "integer", + "$ref": "#/$defs/Port", "description": "The port of the Vertica server." }, "database": { diff --git a/src/script/negative-tests/variables-in-integer-field.odcs.yaml b/src/script/negative-tests/variables-in-integer-field.odcs.yaml index cd971f1..321380f 100644 --- a/src/script/negative-tests/variables-in-integer-field.odcs.yaml +++ b/src/script/negative-tests/variables-in-integer-field.odcs.yaml @@ -2,8 +2,10 @@ # SPDX-License-Identifier: Apache-2.0 # RFC-0050 negative test: variable interpolation applies to STRING values only. -# A `postgresql` server `port` is typed as an integer, so the unresolved -# ${DB_PORT} token (a string) MUST be rejected by the schema. +# `primaryKeyPosition` is typed as an integer, so the unresolved ${KEY_POS} +# token (a string) MUST be rejected by the schema. (The server `port` is an +# explicit exception: it accepts integer or string, so it can hold a +# variable reference.) version: 1.0.0 apiVersion: v3.2.0 @@ -21,3 +23,9 @@ servers: schema: - name: orders physicalType: table + properties: + - name: order_id + logicalType: string + physicalType: varchar(36) + primaryKey: true + primaryKeyPosition: ${KEY_POS} From b217657569d4e6f460470c08e81f6c65a1562f70 Mon Sep 17 00:00:00 2001 From: "Dr. Simon Harrer" Date: Thu, 23 Jul 2026 13:22:28 +0200 Subject: [PATCH 26/29] Update infrastructure-servers.md Remove the suggestion of using multiple server technologies per data contract, as this is no longer a best practice. --- docs/infrastructure-servers.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/infrastructure-servers.md b/docs/infrastructure-servers.md index 5409492..cf2c350 100644 --- a/docs/infrastructure-servers.md +++ b/docs/infrastructure-servers.md @@ -13,8 +13,6 @@ The typical ways of using the top level `servers` element are as follows: * **Single Server:** The data contract protects a specific dataset at a specific location. *Example:* a CSV file on an SFTP server. * **Multiple Environments:** The data contract makes sure that the data is protected in all environments. *Example:* a data product with data in a **dev**(elopment), UAT, and **prod**(uction) environment on Databricks. -* **Different Technologies:** The data contract makes sure that regardless of the offered technology, it still holds. *Example:* a data product offers its data in a Kafka topic and in a BigQuery table that should have the same structure and content. -* **Different Technologies and Multiple Environments:** The data contract makes sure that regardless of the offered technology and environment, it still holds. *Example:* a data product offers its data in a Kafka topic and in a BigQuery table that should have the same structure and content in **dev**(elopment), UAT, and **prod**(uction). [Back to TOC](README.md) From 7082ab4d58b39c625029edc39476af3bd5631417 Mon Sep 17 00:00:00 2001 From: Simon Harrer Date: Wed, 29 Jul 2026 08:58:45 +0200 Subject: [PATCH 27/29] Add context7.json --- context7.json | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 context7.json diff --git a/context7.json b/context7.json new file mode 100644 index 0000000..101a312 --- /dev/null +++ b/context7.json @@ -0,0 +1,4 @@ +{ + "url": "https://context7.com/bitol-io/open-data-contract-standard", + "public_key": "pk_C11eRTOix9LEfxZ9vwIdq" +} From 7c01759d98b6ee7a0ef2dd854e066af08d6000e4 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Wed, 5 Aug 2026 11:06:36 +1000 Subject: [PATCH 28/29] feat(rfc-0058): add `exasol` server type (#313) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies approved RFC-0058 to ODCS v3.2.0: new `exasol` server type for Exasol, an in-memory MPP analytics database. Both schema files updated in lockstep — enum value, conditional dispatch, and ExasolServer def with required host and optional port (default 8563) + schema. No `database` field: an Exasol cluster runs a single database and the schema is the namespace. Docs section, positive example, and a negative test (missing host -> rejected). Co-authored-by: Claude Opus 5 (1M context) --- CHANGELOG.md | 4 ++ docs/examples/server/exasol-server.odcs.yaml | 25 +++++++++++ docs/infrastructure-servers.md | 14 +++++- schema/odcs-json-schema-latest.json | 44 ++++++++++++++++++- schema/odcs-json-schema-v3.2.0.json | 44 ++++++++++++++++++- .../exasol-missing-host.odcs.yaml | 19 ++++++++ 6 files changed, 147 insertions(+), 3 deletions(-) create mode 100644 docs/examples/server/exasol-server.odcs.yaml create mode 100644 src/script/negative-tests/exasol-missing-host.odcs.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 3138519..6196e60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,6 +63,10 @@ RFCs targeting v3.2.0 are tracked under [`tsc/rfcs/`](https://github.com/bitol-i * **Adds** Apache Iceberg server type ([RFC 0049](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0049-iceberg-server-type.md)): * New `iceberg` server `type` describing access to Apache Iceberg catalogs through the standardized Iceberg REST API, with required `catalog` and `catalogUrl` plus optional `namespace` and `warehouse`. * Non-breaking: adds a new optional server type. +* **Adds** Exasol server type ([RFC 0058](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0058-exasol-server-type.md)): + * New `exasol` server `type` describing data served from Exasol, an in-memory MPP analytics database, with required `host` plus optional `port` (defaults to `8563`) and `schema`. + * No `database` field: an Exasol cluster runs a single database and the schema is the namespace. `host` may be a cluster connection range, e.g. `n11..14.acme.com`. + * Non-breaking: adds a new optional server type. * **Changes** to Servers: * Add optional Athena Server `workgroup` field and fix `stagingDir` to be optional in schema. diff --git a/docs/examples/server/exasol-server.odcs.yaml b/docs/examples/server/exasol-server.odcs.yaml new file mode 100644 index 0000000..37af66f --- /dev/null +++ b/docs/examples/server/exasol-server.odcs.yaml @@ -0,0 +1,25 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# Demonstrates RFC-0058: the `exasol` server type for Exasol, an in-memory MPP +# analytics database. An Exasol cluster runs a single database, so the schema is +# the namespace and there is no `database` field. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 5c9a1e73-8d24-4f60-b1a7-6e0c3b5d9f42 +status: active +servers: + - server: prod + type: exasol + host: exasol.acme.com + port: 8563 + schema: SALES +schema: + - name: sales + physicalType: table + properties: + - name: sale_id + logicalType: string + required: true diff --git a/docs/infrastructure-servers.md b/docs/infrastructure-servers.md index d616d6d..11db7fb 100644 --- a/docs/infrastructure-servers.md +++ b/docs/infrastructure-servers.md @@ -50,7 +50,7 @@ servers: | id | string | ID | No | A unique identifier used to reduce the risk of collisions, such as a UUID. | | roles | array | Roles | No | List of roles that have access to the server. Check [roles](./roles.md) section for more details. | | server | string | Server | Yes | Identifier of the server. | -| type | string | Type | Yes | Type of the server. Can be one of: api, athena, azure, bigquery, clickhouse, cloudsql, custom, databricks, db2, denodo, dremio, duckdb, glue, hive, impala, informix, kafka, kinesis, local, mysql, oracle, postgres, postgresql, presto, pubsub, redshift, s3, sftp, snowflake, sqlserver, synapse, trino, vertica, zen. | +| type | string | Type | Yes | Type of the server. Can be one of: api, athena, azure, bigquery, clickhouse, cloudsql, custom, databricks, db2, denodo, dremio, duckdb, exasol, glue, hive, impala, informix, kafka, kinesis, local, mysql, oracle, postgres, postgresql, presto, pubsub, redshift, s3, sftp, snowflake, sqlserver, synapse, trino, vertica, zen. | | customProperties | array | Custom Properties | No | Custom properties that are not part of the standard. | ## Specific Server Properties @@ -160,6 +160,18 @@ If your server is not in the list, please use [custom](#custom-server) and sugge | database | string | Database | Yes | Path to duckdb database file. | | schema | string | Schema | No | The name of the schema. | +### Exasol + +[Exasol](https://www.exasol.com/) is an in-memory, massively parallel processing (MPP) analytics database used as an enterprise data warehouse. Added in ODCS v3.2.0 ([RFC 0058](https://github.com/bitol-io/tsc/blob/main/rfcs/approved/odcs-v3.2.0/0058-exasol-server-type.md)). + +An Exasol cluster runs a single database and the schema is the namespace, so there is no `database` field. + +| Key | Type | UX Label | Required | Description | +| ------ | ------- | -------- | -------- | -------------------------------------------------------------------------------------- | +| host | string | Host | Yes | Host of the Exasol server. May be a cluster connection range, e.g. `n11..14.acme.com`. | +| port | integer | Port | No | Port of the Exasol server. Defaults to 8563. | +| schema | string | Schema | No | Name of the schema. | + ### Amazon Glue | Key | Type | UX Label | Required | Description | diff --git a/schema/odcs-json-schema-latest.json b/schema/odcs-json-schema-latest.json index 9ebf72e..c676385 100644 --- a/schema/odcs-json-schema-latest.json +++ b/schema/odcs-json-schema-latest.json @@ -237,7 +237,7 @@ "description": "Type of the server.", "enum": [ "api", "athena", "azure", "bigquery", "clickhouse", "databricks", "denodo", "dremio", - "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "iceberg", "impala", "informix", "kafka", "kinesis", "local", + "duckdb", "exasol", "glue", "hana", "cloudsql", "db2", "hive", "iceberg", "impala", "informix", "kafka", "kinesis", "local", "mysql", "oracle", "postgresql", "postgres", "presto", "pubsub", "redshift", "s3", "sftp", "snowflake", "sqlserver", "synapse", "trino", "vertica", "zen", "custom" ] @@ -380,6 +380,19 @@ "$ref": "#/$defs/ServerSource/DuckdbServer" } }, + { + "if": { + "properties": { + "type": { + "const": "exasol" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/ExasolServer" + } + }, { "if": { "properties": { @@ -963,6 +976,35 @@ "database" ] }, + "ExasolServer": { + "type": "object", + "title": "ExasolServer", + "properties": { + "host": { + "type": "string", + "description": "Host of the Exasol server. May be a cluster connection range.", + "examples": [ + "exasol.acme.com", + "n11..14.acme.com" + ] + }, + "port": { + "$ref": "#/$defs/Port", + "description": "Port of the Exasol server.", + "default": 8563, + "examples": [ + 8563 + ] + }, + "schema": { + "type": "string", + "description": "Name of the schema." + } + }, + "required": [ + "host" + ] + }, "GlueServer": { "type": "object", "title": "GlueServer", diff --git a/schema/odcs-json-schema-v3.2.0.json b/schema/odcs-json-schema-v3.2.0.json index 4235b52..efbf54e 100644 --- a/schema/odcs-json-schema-v3.2.0.json +++ b/schema/odcs-json-schema-v3.2.0.json @@ -236,7 +236,7 @@ "description": "Type of the server.", "enum": [ "api", "athena", "azure", "bigquery", "clickhouse", "databricks", "denodo", "dremio", - "duckdb", "glue", "hana", "cloudsql", "db2", "hive", "iceberg", "impala", "informix", "kafka", "kinesis", "local", + "duckdb", "exasol", "glue", "hana", "cloudsql", "db2", "hive", "iceberg", "impala", "informix", "kafka", "kinesis", "local", "mysql", "oracle", "postgresql", "postgres", "presto", "pubsub", "redshift", "s3", "sftp", "snowflake", "sqlserver", "synapse", "trino", "vertica", "zen", "custom" ] @@ -379,6 +379,19 @@ "$ref": "#/$defs/ServerSource/DuckdbServer" } }, + { + "if": { + "properties": { + "type": { + "const": "exasol" + } + }, + "required": ["type"] + }, + "then": { + "$ref": "#/$defs/ServerSource/ExasolServer" + } + }, { "if": { "properties": { @@ -962,6 +975,35 @@ "database" ] }, + "ExasolServer": { + "type": "object", + "title": "ExasolServer", + "properties": { + "host": { + "type": "string", + "description": "Host of the Exasol server. May be a cluster connection range.", + "examples": [ + "exasol.acme.com", + "n11..14.acme.com" + ] + }, + "port": { + "$ref": "#/$defs/Port", + "description": "Port of the Exasol server.", + "default": 8563, + "examples": [ + 8563 + ] + }, + "schema": { + "type": "string", + "description": "Name of the schema." + } + }, + "required": [ + "host" + ] + }, "GlueServer": { "type": "object", "title": "GlueServer", diff --git a/src/script/negative-tests/exasol-missing-host.odcs.yaml b/src/script/negative-tests/exasol-missing-host.odcs.yaml new file mode 100644 index 0000000..7339e22 --- /dev/null +++ b/src/script/negative-tests/exasol-missing-host.odcs.yaml @@ -0,0 +1,19 @@ +# Copyright 2026 The Bitol Contributors +# SPDX-License-Identifier: Apache-2.0 + +# RFC-0058 negative test: an `exasol` server requires `host`. Here `host` is +# omitted, so the schema MUST reject this contract. + +version: 1.0.0 +apiVersion: v3.2.0 +kind: DataContract +id: 6d0b2f84-9e35-4a71-c2b8-7f1d4c6e0a53 +status: active +servers: + - server: prod + type: exasol + port: 8563 + schema: SALES +schema: + - name: sales + physicalType: table From da17cea7c4e2832ff732e08e1575c8885583f047 Mon Sep 17 00:00:00 2001 From: "Jean-Georges \"jgp\" Perrin" Date: Wed, 5 Aug 2026 11:17:08 +1000 Subject: [PATCH 29/29] docs(servers): add a definition to every server type, spell out HCL/IBM Informix (#314) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consistency pass over docs/infrastructure-servers.md. Eighteen server sections jumped straight from the heading to the property table while the rest opened with a one-line definition of the technology. Adds that definition (linked to the vendor/project page, matching the existing style) for: API, Azure, Databricks, IBM Db2, Denodo, Dremio, Amazon Glue, Kafka, Amazon Kinesis, Local Files, MySQL, Oracle, Presto, Snowflake, Synapse, Trino, Vertica, and Custom. All 36 server sections now have one. Informix is always named as "HCL Informix and IBM Informix" — heading, intro, and the host/port descriptions, which previously said just "Informix". The lowercase `informix` in the `type` value list is the schema enum value and is left as is. Also re-aligns the three tables whose pipes had drifted (Common Server Properties, Exasol, Informix) — no content change. Co-authored-by: Claude Opus 5 (1M context) --- docs/infrastructure-servers.md | 72 +++++++++++++++++++++++++--------- 1 file changed, 54 insertions(+), 18 deletions(-) diff --git a/docs/infrastructure-servers.md b/docs/infrastructure-servers.md index 11db7fb..67e9e8c 100644 --- a/docs/infrastructure-servers.md +++ b/docs/infrastructure-servers.md @@ -43,15 +43,15 @@ servers: ### Common Server Properties -| Key | Type | UX label | Required | Description | -| ---------------- | ------ | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| description | string | Description | No | Description of the server. | -| environment | string | Environment | No | Environment of the server. Examples includes: prod, preprod, dev, uat. | -| id | string | ID | No | A unique identifier used to reduce the risk of collisions, such as a UUID. | -| roles | array | Roles | No | List of roles that have access to the server. Check [roles](./roles.md) section for more details. | -| server | string | Server | Yes | Identifier of the server. | +| Key | Type | UX label | Required | Description | +| ---------------- | ------ | ----------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| description | string | Description | No | Description of the server. | +| environment | string | Environment | No | Environment of the server. Examples includes: prod, preprod, dev, uat. | +| id | string | ID | No | A unique identifier used to reduce the risk of collisions, such as a UUID. | +| roles | array | Roles | No | List of roles that have access to the server. Check [roles](./roles.md) section for more details. | +| server | string | Server | Yes | Identifier of the server. | | type | string | Type | Yes | Type of the server. Can be one of: api, athena, azure, bigquery, clickhouse, cloudsql, custom, databricks, db2, denodo, dremio, duckdb, exasol, glue, hive, impala, informix, kafka, kinesis, local, mysql, oracle, postgres, postgresql, presto, pubsub, redshift, s3, sftp, snowflake, sqlserver, synapse, trino, vertica, zen. | -| customProperties | array | Custom Properties | No | Custom properties that are not part of the standard. | +| customProperties | array | Custom Properties | No | Custom properties that are not part of the standard. | ## Specific Server Properties @@ -63,6 +63,8 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### API Server +An API server describes data that is exposed through a network API rather than served from a database or file storage. The endpoint a consumer calls is identified by its URL. + | Key | Type | UX Label | Required | Description | | ------------ | ------ | -------- | -------- | -------------- | | **location** | string | Location | Yes | URL to the API | @@ -81,6 +83,8 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Azure Server +[Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) and [Azure Data Lake Storage (ADLS)](https://azure.microsoft.com/en-us/products/storage/data-lake-storage) are the Microsoft Azure object storage services for unstructured data and large-scale analytics workloads. + | Key | Type | UX Label | Required | Description | | --------- | ------ | --------- | -------- | ----------------------------------------------------------------------------------------------------- | | delimiter | string | Delimiter | No | Only for format = json. How multiple json documents are delimited within one file | @@ -120,6 +124,8 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Databricks Server +[Databricks](https://www.databricks.com/) is a cloud-based data platform, built on Apache Spark, that unifies data warehousing and data lakes under the lakehouse architecture. + | Key | Type | UX Label | Required | Description | | ------- | ------ | -------- | -------- | ------------------------------------- | | catalog | string | Catalog | Yes | The name of the Hive or Unity catalog | @@ -128,6 +134,8 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### IBM Db2 Server +[IBM Db2](https://www.ibm.com/products/db2) is a family of relational database management systems for transactional and analytical workloads, available both on cloud and on-premises. + | Key | Type | UX Label | Required | Description | | -------- | ------- | -------- | -------- | ------------------------------- | | database | string | Database | Yes | The name of the database. | @@ -137,6 +145,8 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Denodo Server +[Denodo](https://www.denodo.com/) is a data virtualization platform that provides unified, real-time access to data spread across disparate sources, without replicating it. + | Key | Type | UX Label | Required | Description | | -------- | ------- | -------- | -------- | ------------------------------ | | database | string | Database | No | The name of the database. | @@ -145,6 +155,8 @@ If your server is not in the list, please use [custom](#custom-server) and sugge ### Dremio Server +[Dremio](https://www.dremio.com/) is a lakehouse platform that runs SQL queries directly against data lake storage, built on Apache Arrow and Apache Iceberg. + | Key | Type | UX Label | Required | Description | | ------ | ------- | -------- | -------- | ------------------------------ | | host | string | Host | Yes | The host of the Dremio server. | @@ -166,14 +178,16 @@ If your server is not in the list, please use [custom](#custom-server) and sugge An Exasol cluster runs a single database and the schema is the namespace, so there is no `database` field. -| Key | Type | UX Label | Required | Description | +| Key | Type | UX Label | Required | Description | | ------ | ------- | -------- | -------- | -------------------------------------------------------------------------------------- | | host | string | Host | Yes | Host of the Exasol server. May be a cluster connection range, e.g. `n11..14.acme.com`. | -| port | integer | Port | No | Port of the Exasol server. Defaults to 8563. | -| schema | string | Schema | No | Name of the schema. | +| port | integer | Port | No | Port of the Exasol server. Defaults to 8563. | +| schema | string | Schema | No | Name of the schema. | ### Amazon Glue +[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service. Its Data Catalog holds the table definitions and schema metadata describing data stored in Amazon S3 and other sources. + | Key | Type | UX Label | Required | Description | | -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | | account | string | Account | Yes | The AWS Glue account | @@ -213,18 +227,20 @@ An Exasol cluster runs a single database and the schema is the namespace, so the | host | string | Host | Yes | The host to the Impala server. | | port | integer | Port | No | The port to the Impala server. Defaults to 21050. | -### IBM Informix and HCL Informix +### HCL Informix and IBM Informix -[IBM Informix](https://www.ibm.com/products/informix) is a high performance, always-on, highly scalable and easily embeddable enterprise-class database optimized for the most demanding transactional and analytics workloads. As an object-relational engine, IBM Informix seamlessly integrates the best of relational and object-oriented capabilities enabling the flexible modeling of complex data structures and relationships. +[HCL Informix](https://www.hcl-software.com/informix) and [IBM Informix](https://www.ibm.com/products/informix) are high performance, always-on, highly scalable and easily embeddable enterprise-class databases optimized for the most demanding transactional and analytics workloads. As object-relational engines, HCL Informix and IBM Informix seamlessly integrate the best of relational and object-oriented capabilities, enabling the flexible modeling of complex data structures and relationships. -| Key | Type | UX Label | Required | Description | -| -------- | ------- | -------- | -------- | -------------------------------------------------- | -| database | string | Database | Yes | The name of the database. | -| host | string | Host | Yes | The host to the Informix server. | -| port | integer | Port | No | The port to the Informix server. Defaults to 9088. | +| Key | Type | UX Label | Required | Description | +| -------- | ------- | -------- | -------- | ----------------------------------------------------------------------- | +| database | string | Database | Yes | The name of the database. | +| host | string | Host | Yes | The host to the HCL Informix and IBM Informix server. | +| port | integer | Port | No | The port to the HCL Informix and IBM Informix server. Defaults to 9088. | ### Kafka Server +[Apache Kafka](https://kafka.apache.org/) is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, and event-driven applications. + | Key | Type | UX Label | Required | Description | | -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | @@ -233,6 +249,8 @@ An Exasol cluster runs a single database and the schema is the namespace, so the ### Amazon Kinesis +[Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/) is a serverless streaming data service for collecting, processing, and analyzing large streams of records in real time. + | Key | Type | UX Label | Required | Description | | -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | @@ -242,6 +260,8 @@ An Exasol cluster runs a single database and the schema is the namespace, so the ### Local Files +A local server describes data stored as one or more files on the local file system, addressed by a relative or absolute path. It is typically used for development, testing, and small datasets. + | Key | Type | UX Label | Required | Description | | -------- | ------ | -------- | -------- | ----------------------------------------------------------------------------------------------------- | | encoding | string | Encoding | No | Expected character encoding of the payload, e.g. UTF-8, ISO-8859-1, ASCII, UTF-16. Defaults to UTF-8. | @@ -250,6 +270,8 @@ An Exasol cluster runs a single database and the schema is the namespace, so the ### MySQL Server +[MySQL](https://www.mysql.com/) is an open-source relational database management system, widely used for transactional and web applications. + | Key | Type | UX Label | Required | Description | | -------- | ------- | -------- | -------- | ----------------------------------------------- | | database | string | Database | Yes | The name of the database. | @@ -258,6 +280,8 @@ An Exasol cluster runs a single database and the schema is the namespace, so the ### Oracle +[Oracle Database](https://www.oracle.com/database/) is a multi-model relational database management system used for transactional and analytical enterprise workloads. Clients connect to a named service rather than directly to a database. + | Key | Type | UX Label | Required | Description | | ----------- | ------- | ------------ | -------- | ------------------------------ | | host | string | Host | Yes | The host to the Oracle server | @@ -277,6 +301,8 @@ An Exasol cluster runs a single database and the schema is the namespace, so the ### Presto Server +[Presto](https://prestodb.io/) is an open-source distributed SQL query engine for running interactive analytic queries against data sources of any size, from gigabytes to petabytes. + | Key | Type | UX Label | Required | Description | | ------- | ------ | -------- | -------- | ----------------------------- | | catalog | string | Catalog | No | The name of the catalog. | @@ -339,6 +365,8 @@ Secure File Transfer Protocol (SFTP) is a network protocol that enables secure a ### Snowflake +[Snowflake](https://www.snowflake.com/) is a fully managed cloud data platform that separates storage from compute, where compute is provided by virtual warehouses that can be sized and scaled independently. + | Key | Type | UX Label | Required | Description | | --------- | ------- | --------- | -------- | --------------------------------------------------------------------------- | | account | string | Account | Yes | The Snowflake account used by the server. | @@ -361,6 +389,8 @@ Secure File Transfer Protocol (SFTP) is a network protocol that enables secure a ### Synapse Server +[Azure Synapse Analytics](https://azure.microsoft.com/en-us/products/synapse-analytics) is the Microsoft Azure analytics service that brings together enterprise data warehousing and big data analytics. + | Key | Type | UX Label | Required | Description | | -------- | ------- | -------- | -------- | ------------------------------- | | database | string | Database | Yes | The name of the database. | @@ -369,6 +399,8 @@ Secure File Transfer Protocol (SFTP) is a network protocol that enables secure a ### Trino Server +[Trino](https://trino.io/) is an open-source distributed SQL query engine designed to query large datasets across one or more heterogeneous data sources. + | Key | Type | UX Label | Required | Description | | ------- | ------- | -------- | -------- | --------------------------------------- | | catalog | string | Catalog | Yes | The name of the catalog. | @@ -378,6 +410,8 @@ Secure File Transfer Protocol (SFTP) is a network protocol that enables secure a ### Vertica Server +[Vertica](https://docs.vertica.com/) is a column-oriented, massively parallel processing (MPP) analytical database for large-scale data warehousing. + | Key | Type | UX Label | Required | Description | | -------- | ------- | -------- | -------- | ------------------------------- | | database | string | Database | Yes | The name of the database. | @@ -397,6 +431,8 @@ Actian Zen (formerly Btrieve, later named Pervasive PSQL until version 13) is an ### Custom Server +A custom server describes any technology that does not have a dedicated type in ODCS yet. It accepts the union of the properties defined by the other server types, so connection details can still be expressed in a structured way. + | Key | Type | UX Label | Required | Description | | ----------- | ------- | ----------------- | -------- | ----------------------------------------------------------------------------------------------------- | | account | string | Account | No | Account used by the server. |