Implement function materialization

CHANGELOG.md

@@ -124,7 +124,7 @@ For example:
   -- test_ttl.sql
   {{ config(order_by='(ts)', engine='MergeTree()', materialized='table') }}
 
-  SELECT now() AS ts, 
+  SELECT now() AS ts,
         'Some value that should expire!' AS col_ttl
   ```
 
@@ -160,7 +160,7 @@ For example:
 #### Improvements
 * It is now possible to configure a TLS client certificate using `client_cert` and `client_cert_key` profile parameters. ([#413](https://github.com/ClickHouse/dbt-clickhouse/pull/413))
 * Added Support of insert_overwrite in cluster setup with incremental and distributed_incremental materializations ([#394](https://github.com/ClickHouse/dbt-clickhouse/pull/394))
-* Improve index and projections creation process ([#421](https://github.com/ClickHouse/dbt-clickhouse/pull/421)) 
+* Improve index and projections creation process ([#421](https://github.com/ClickHouse/dbt-clickhouse/pull/421))
 
 #### Bugs
 * Reverted breaking changes in MV materialization ([#416](https://github.com/ClickHouse/dbt-clickhouse/pull/416))
@@ -179,7 +179,7 @@ For example:
 
 #### New Features
 * [ClickHouse indexes](https://clickhouse.com/docs/en/optimize/sparse-primary-indexes) are now fully supported for `table` materialization.
-The index config should be added to the model config. for instance: 
+The index config should be added to the model config. for instance:
   ```python
   {{ config(
          materialized='%s',
@@ -203,7 +203,7 @@ The index config should be added to the model config. for instance:
 * Removed support in python 3.8 as it is no longer supported by dbt ([#402](https://github.com/ClickHouse/dbt-clickhouse/pull/402)
 
 ### Bug Fixes
-* Fix a minor bug related to validating existence of an old hanging mv ([#396]()) 
+* Fix a minor bug related to validating existence of an old hanging mv ([#396]())
 
 ### Release [1.8.6], 2024-12-05
 
@@ -243,7 +243,7 @@ The index config should be added to the model config. for instance:
 ### Release [1.8.2], 2024-08-22
 #### New Features
 * [ClickHouse projections](https://clickhouse.com/docs/en/sql-reference/statements/alter/projection) are now fully supported for `table` materialization, and partly supported for `distributed_table` materialization.
-The projection config should be added to the model config ([#342](https://github.com/ClickHouse/dbt-clickhouse/pull/342)), for instance: 
+The projection config should be added to the model config ([#342](https://github.com/ClickHouse/dbt-clickhouse/pull/342)), for instance:
   ```python
   {{ config(
          materialized='%s',
@@ -255,7 +255,7 @@ The projection config should be added to the model config ([#342](https://github
          ]
   ) }}
   ```
- 
+
 #### Bug Fixes
 * Until this release, when writing tests, it was not possible to pass empty seed data. ([#341](https://github.com/ClickHouse/dbt-clickhouse/pull/341))
 * When a cluster was used, the adapter left a few `__dbt_backup` tables in the schema. After the fix, these backup tables are now properly dropped. ([#326](https://github.com/ClickHouse/dbt-clickhouse/pull/326))
@@ -271,9 +271,9 @@ The projection config should be added to the model config ([#342](https://github
 * Enhanced the clickhouse__listagg macro to support single field ordering with optional direction, ensuring compatibility with ClickHouse's sorting limitations. Added validation to prevent the use of multiple order-by fields. ([#318](https://github.com/ClickHouse/dbt-clickhouse/pull/318))
 
 #### Documentation
-* Update the docs with the new `insert_overwrite` incremental strategy. ([#331](https://github.com/ClickHouse/dbt-clickhouse/pull/331)) 
+* Update the docs with the new `insert_overwrite` incremental strategy. ([#331](https://github.com/ClickHouse/dbt-clickhouse/pull/331))
 * Add documentation for codec column configuration. ([#317](https://github.com/ClickHouse/dbt-clickhouse/pull/317))
- 
+
 
 ### Release [1.8.1], 2024-07-11
 #### Bug Fix
@@ -330,7 +330,7 @@ for the PR!
 ### Release [1.7.4], 2024-03-23
 #### Improvement
 - Adds support for materializing ClickHouse dictionaries.  Thanks to [Rory Sawyer](https://github.com/SoryRawyer) for the contribution!
-See his excellent [tests](https://github.com/ClickHouse/dbt-clickhouse/blob/main/tests/integration/adapter/dictionary/test_dictionary.py) 
+See his excellent [tests](https://github.com/ClickHouse/dbt-clickhouse/blob/main/tests/integration/adapter/dictionary/test_dictionary.py)
 for example usage.
 
 ### Release [1.7.3], 2024-03-11
@@ -387,7 +387,7 @@ do not support `CREATE TABLE AS SELECT ... EMPTY`, since the automatic deduplica
 inserts in Replicated tables on those older versions.  Fixes https://github.com/ClickHouse/dbt-clickhouse/issues/216.
 
 ### Release [1.6.0], 2023-11-30
-#### Improvements 
+#### Improvements
 - Compatible with dbt 1.6.x.  Note that dbt new `clone` feature is not supported, as ClickHouse has no native "light weight"
 clone functionality, and copying tables without actual data transfer is not possible in ClickHouse (barring file manipulation
 outside ClickHouse itself).
@@ -476,7 +476,7 @@ https://github.com/ClickHouse/dbt-clickhouse/issues/167
 
 #### Improvement
 - Added macros for creating distributed tables.  See the `distributed_table.sql` include file.  Thanks to
-[gladkikhtutu](https://github.com/gladkikhtutu) for the contribution.  
+[gladkikhtutu](https://github.com/gladkikhtutu) for the contribution.
 
 ### Release [1.4.2], 2023-05-14
 #### Bug fixes
@@ -500,7 +500,7 @@ to non-overlapping data ranges.  Closes https://github.com/ClickHouse/dbt-clickh
   - Adds additional dbt 1.4.0 tests
   - Adds support for incremental_predicates.  This only applies to `delete+insert` incremental strategy.  Note that incremental
 predicates that depend on "non-deterministic" data (such as a subquery using a table that is accepting inserts) could lead to
-unexpected results for ReplicatedMergeTree tables depending on the timing of the incremental materialization.  
+unexpected results for ReplicatedMergeTree tables depending on the timing of the incremental materialization.
   - Replaces deprecated Exception classes
 - Setting the `use_lw_deletes` profile value to True will now attempt to enable the `allow_experimental_lightweight_delete`
 setting for the dbt session (if user has such permissions on the ClickHouse server).  See https://github.com/ClickHouse/dbt-clickhouse/issues/133
@@ -513,7 +513,7 @@ setting for the dbt session (if user has such permissions on the ClickHouse serv
 #### Documentation Update
 - The documentation has been updated to reflect that dbt-clickhouse does support ephemeral models, and ephemeral model tests do pass.
 However, due to a [ClickHouse limitation](https://github.com/ClickHouse/ClickHouse/issues/30323), CTEs will not work directly
-with INSERT statements so table models will fail if they include ephemeral models in the SELECT.  View models and other SQL 
+with INSERT statements so table models will fail if they include ephemeral models in the SELECT.  View models and other SQL
 statements using ephemeral models should work correctly.
 
 #### Bug Fix
@@ -537,7 +537,7 @@ slower "legacy" strategy.
 [tests](https://github.com/ClickHouse/dbt-clickhouse/blob/main/tests/integration/adapter/test_s3.py) for example usage.
 
 #### Bug Fixes
-- The ON CLUSTER clause has been added to additional DDL statements including incremental models processing. 
+- The ON CLUSTER clause has been added to additional DDL statements including incremental models processing.
 Closes https://github.com/ClickHouse/dbt-clickhouse/issues/117 and should close https://github.com/ClickHouse/dbt-clickhouse/issues/95
 for Replicated tables that use the `{uuid}` macro in the path to avoid name conflicts.  Thanks to [Saurabh Bikram](https://github.com/saurabhbikram)
 - The `apply` and `revoke` grants macros now correctly work with roles as well as user.  Again thanks to [Saurabh Bikram](https://github.com/saurabhbikram)

dbt/adapters/clickhouse/__version__.py

@@ -1 +1 @@
-version = '1.10.0'
+version = "1.11.0"

dbt/adapters/clickhouse/connections.py

@@ -3,10 +3,15 @@
 from contextlib import contextmanager
 from typing import TYPE_CHECKING, Any, Optional, Tuple, Union
 
+from dbt_common.events.contextvars import get_node_info
+from dbt_common.events.functions import fire_event
+from dbt_common.utils import cast_to_str
+
 import dbt.exceptions
 from dbt.adapters.clickhouse.dbclient import ChRetryableException, get_db_client
 from dbt.adapters.clickhouse.logger import logger
 from dbt.adapters.contracts.connection import AdapterResponse, Connection
+from dbt.adapters.events.types import SQLQuery
 from dbt.adapters.sql import SQLConnectionManager
 
 if TYPE_CHECKING:
@@ -87,7 +92,14 @@ def execute(
         client = conn.handle
 
         with self.exception_handler(sql):
-            logger.debug(f'On {conn.name}: {sql}...')
+            fire_event(
+                SQLQuery(
+                    conn_name=cast_to_str(conn.name),
+                    node_info=get_node_info(),
+                    sql=sql,
+                )
+            )
+            logger.debug(f"On {conn.name}: {sql}...")
             pre = time.time()
             if fetch:
                 query_result = client.query(sql)
@@ -116,7 +128,20 @@ def add_query(
         conn = self.get_thread_connection()
         client = conn.handle
         with self.exception_handler(sql):
-            logger.debug(f'On {conn.name}: {sql}...')
+            if abridge_sql_log:
+                log_sql = "{}...".format(sql[:512])
+            else:
+                log_sql = sql
+
+            fire_event(
+                SQLQuery(
+                    conn_name=cast_to_str(conn.name),
+                    node_info=get_node_info(),
+                    sql=log_sql,
+                )
+            )
+
+            logger.debug(f"On {conn.name}: {sql}...")
             pre = time.time()
             client.command(sql)
             status = self.get_status(client)

dbt/adapters/clickhouse/relation.py

@@ -1,14 +1,16 @@
 from dataclasses import dataclass, field
 from typing import Any, Dict, List, Optional, Type
 
-from dbt.adapters.base.relation import BaseRelation, EventTimeFilter, Path, Policy, Self
-from dbt.adapters.clickhouse.query import quote_identifier
-from dbt.adapters.contracts.relation import HasQuoting, RelationConfig
 from dbt_common.dataclass_schema import StrEnum
 from dbt_common.exceptions import DbtRuntimeError
 from dbt_common.utils import deep_merge
 
-NODE_TYPE_SOURCE = 'source'
+from dbt.adapters.base.relation import BaseRelation, EventTimeFilter, Path, Policy, Self
+from dbt.adapters.clickhouse.query import quote_identifier
+from dbt.adapters.contracts.relation import HasQuoting, RelationConfig
+
+NODE_TYPE_SOURCE = "source"
+NODE_TYPE_FUNCTION = "function"
 
 
 @dataclass
@@ -33,6 +35,7 @@ class ClickHouseRelationType(StrEnum):
     External = "external"
     Ephemeral = "ephemeral"
     Dictionary = "dictionary"
+    Function = "function"
 
 
 @dataclass(frozen=True, eq=False, repr=False)
@@ -146,6 +149,8 @@ def create_from(
         if relation_config.resource_type == NODE_TYPE_SOURCE:
             if schema == relation_config.source_name and relation_config.database:
                 schema = relation_config.database
+        elif relation_config.resource_type == NODE_TYPE_FUNCTION:
+            schema = None
         else:
             # quoting is only available for non-source nodes
             cluster = quoting.credentials.cluster or ""

dbt/include/clickhouse/macros/materializations/functions/scalar.sql

@@ -0,0 +1,15 @@
+{% macro clickhouse__formatted_scalar_function_args_sql() %}
+    {% set args = [] %}
+    {% for arg in model.arguments -%}
+      {%- do args.append(arg.name) -%}
+    {%- endfor %}
+    {{ args | join(', ') }}
+{% endmacro %}
+
+{% macro clickhouse__scalar_function_create_replace_signature_sql(target_relation) %}
+  CREATE OR REPLACE FUNCTION {{ target_relation.include(database=false, schema=false) }} {{ on_cluster_clause(this) }} AS ({{ clickhouse__formatted_scalar_function_args_sql() }}) ->
+{% endmacro %}
+
+{% macro clickhouse__scalar_function_body_sql() %}
+  {{ model.compiled_code }}
+{% endmacro %}

dev_requirements.txt

@@ -1,4 +1,4 @@
-dbt-core>=1.10.0,<1.11
+dbt-core>=1.11.0
 dbt-tests-adapter>=1.10,<2.0
 pytest>=7.2.0
 pytest-dotenv==0.5.2

tests/integration/adapter/udf/test_udf.py

@@ -0,0 +1,23 @@
+"""
+test UDF creation support for dbt-clickhouse
+"""
+
+import pytest
+
+import dbt.tests.adapter.functions.files as files
+from dbt.tests.adapter.functions.test_udfs import UDFsBasic
+
+# we'll import helper used by the core test project to write directories
+
+MY_UDF_SQL = """
+price * 2
+""".strip()
+
+
+class TestUDFBasics(UDFsBasic):
+    @pytest.fixture(scope="class")
+    def functions(self):
+        return {
+            "price_for_xlarge.sql": MY_UDF_SQL,
+            "price_for_xlarge.yml": files.MY_UDF_YML,
+        }