feat: [clickhouse] native binary protocol + parser + grafana v4 plugin

.github/codeql/codeql-config.yml

@@ -0,0 +1,11 @@
+name: "trickster CodeQL config"
+
+query-filters:
+  # trickster is a passthrough proxy; backends forward client SQL verbatim
+  # to the upstream. companion to the gosec G701 suppression in code.
+  - exclude:
+      id: go/sql-injection
+
+paths-ignore:
+  - vendor
+  - testdata

.github/workflows/codeql-analysis.yml

@@ -21,6 +21,7 @@ on:
 jobs:
   analyze:
     name: Analyze
+    if: github.repository_owner == 'trickstercache'
     runs-on: ubuntu-latest
     permissions:
       actions: read
@@ -44,10 +45,7 @@ jobs:
       uses: github/codeql-action/init@v3
       with:
         languages: ${{ matrix.language }}
-        # If you wish to specify custom queries, you can do so here or in a config file.
-        # By default, queries listed here will override any specified in a config file.
-        # Prefix the list here with "+" to use these queries and those in the config file.
-        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+        config-file: ./.github/codeql/codeql-config.yml
 
     # ℹ️ Command-line programs to run using the OS shell.
     # 📚 https://git.io/JvXDl

docs/clickhouse.md

@@ -4,9 +4,40 @@ Trickster will accelerate ClickHouse queries that return time series data normal
 
 ## Scope of Support
 
-Trickster is tested with the [ClickHouse DataSource Plugin for Grafana](https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource) v1.9.3 by Vertamedia, and supports acceleration of queries constructed by this plugin using the plugin's built-in `$timeSeries` macro.  Trickster also supports several other query formats that return "time series like" data.
+Trickster is tested with the [ClickHouse DataSource Plugin for Grafana](https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource) v1.9.3 by Vertamedia, and supports acceleration of queries constructed by this plugin using the plugin's built-in `$timeSeries` macro. Trickster also supports the official [Grafana ClickHouse plugin](https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/) (v4+), including `toDateTime64()` in WHERE clauses. Trickster also supports several other query formats that return "time series like" data.
 
-Trickster also supports the ClickHouse Go SDK (`clickhouse-go/v2`). Queries made through the SDK's HTTP protocol — including those using `clickhouse.OpenDB` — are proxied and cached through Trickster. The SDK's Native binary protocol is supported via transparent proxying (OPC).
+Trickster also supports the ClickHouse Go SDK (`clickhouse-go/v2`). Queries made through the SDK's HTTP protocol — including those using `clickhouse.OpenDB` — are proxied and cached through Trickster.
+
+### Native Binary Protocol Support
+
+Trickster supports the ClickHouse native binary protocol (port 9000) in two configurations:
+
+**Flow 1 — Native Protocol Listener:** Trickster can accept native protocol connections directly via a protocol listener. Clients connect using the native binary protocol, and Trickster proxies queries through its caching engine. Configure a protocol listener in the `frontend` section:
+
+```yaml
+frontend:
+  protocol_listeners:
+    - name: clickhouse-native
+      protocol: clickhouse-native
+      listen_port: 9000
+      backend: click1
+```
+
+**Flow 2 — Native Upstream:** Trickster can speak native protocol to ClickHouse upstream while accepting HTTP from clients. Set `protocol: native` on the backend:
+
+```yaml
+backends:
+  click1:
+    provider: clickhouse
+    origin_url: 'http://127.0.0.1:9000'
+    protocol: native
+```
+
+The native protocol implementation supports SELECT queries, ping/pong, handshake with addendum, and LZ4 block compression. INSERT operations over the native protocol are proxied as SQL but inline data blocks are not yet supported.
+
+Supported native protocol data types: all integer types (8–256 bit), Float32/64, String, FixedString(N), DateTime, DateTime64, Date, Date32, UUID, IPv4, IPv6, Enum8/16, Bool, Nullable(T), Array(T), Map(K,V), Tuple(T1,T2,...), LowCardinality(T), and Decimal.
+
+**Native Format in DPC:** When a client sends `default_format=Native` (as the official Grafana ClickHouse plugin does), Trickster's DPC automatically requests Native format from ClickHouse, deserializes it into the internal DataSet representation for caching and delta merging, and returns Native binary to the client. This is detected via the `X-ClickHouse-Format` response header — no configuration needed.
 
 Because ClickHouse does not ship an official Go query parser, Trickster includes its own custom SQL parser and lexer to deconstruct incoming ClickHouse queries, determine if they are cacheable and, if so, what elements are factored into the cache key derivation. Trickster also determines the requested time range and step based on the provided absolute values, in order to normalize the query before hashing the cache key.
 
@@ -20,44 +51,74 @@ SELECT intDiv(toUInt32(time_col, 60) * 60) [* 1000] [as] [alias]
 ```
 This is the approach used by the Grafana plugin.  The time_col and/or alias is used to determine the requested time range from the WHERE or PREWHERE clause of the query.  The argument to the ClickHouse intDiv function is the step value in seconds, since the toUInt32 function on a datetime column returns the Unix epoch seconds.
 
-#### ClickHouse Time Grouping Function
+#### ClickHouse Time Grouping Functions
 ```sql
 SELECT toStartOf[Period](time_col) [as] [alias]
 ```
-This is the approach that uses the following optimized ClickHouse functions to group timeseries queries:
+Supported `toStartOf` functions:
 ```
-toStartOfMinute
-toStartOfFiveMinute
-toStartOfTenMinutes
-toStartOfFifteenMinutes
-toStartOfHour
-toDate
+toStartOfNanosecond     toStartOfMicrosecond    toStartOfMillisecond
+toStartOfSecond         toStartOfMinute         toStartOfFiveMinute
+toStartOfTenMinutes     toStartOfFifteenMinutes toStartOfHour
+toStartOfDay            toStartOfWeek           toStartOfMonth
+toStartOfQuarter        toStartOfYear           toMonday
 ```
-Again the time_col and/or alias is used to determine the request time range from the WHERE or PREWHERE clause, and the step is derived from the function name.
+
+#### Custom Interval Grouping
+```sql
+SELECT toStartOfInterval(time_col, INTERVAL N unit) [as] [alias]
+```
+Supported units: `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`, `millisecond`, `microsecond`, `nanosecond`.
+
+#### SQL-Standard date_trunc
+```sql
+SELECT date_trunc('unit', time_col) [as] [alias]
+```
+Supported units: `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`, `millisecond`.
+
+#### timeSlot
+```sql
+SELECT timeSlot(time_col) [as] [alias]
+```
+Rounds to 30-minute boundaries.
+
+The time_col and/or alias is used to determine the request time range from the WHERE or PREWHERE clause, and the step is derived from the function name or interval.
 
 #### Determining the requested time range
 
 Once the time column (or alias) and step are derived, Trickster parses each WHERE or PREWHERE clause to find comparison operations 
 that mark the requested time range.  To be cacheable, the WHERE clause must contain either a `[timecol|alias] BETWEEN` phrase or 
 a `[time_col|alias] >[=]` phrase.  The BETWEEN or >= arguments must be a parsable ClickHouse string date in the form `2006-01-02 15:04:05`, 
-a ten digit integer representing epoch seconds, or the `now()` ClickHouse function with optional subtraction.
+a ten digit integer representing epoch seconds, or the `now()` / `now64()` ClickHouse functions with optional subtraction.
 
 If a `>` phrase is used, a similar `<` phrase can be used to specify the end of the time period.  If none is found, Trickster will still cache results up to
 the current time, but future queries must also have no end time phrase, or Trickster will be unable to find the correct cache key.
 
 Examples of cacheable time range WHERE clauses:
 ```sql
-WHERE t >= "2020-10-15 00:00:00" and t <= "2020-10-16 12:00:00"
-WHERE t >= "2020-10-15 12:00:00" and t < now() - 60 * 60
+WHERE t >= '2020-10-15 00:00:00' AND t <= '2020-10-16 12:00:00'
+WHERE t >= '2020-10-15 12:00:00' AND t < now() - 60 * 60
 WHERE datetime BETWEEN 1574686300 AND 1574689900
+WHERE datetime >= toDateTime64(1589904000, 3) AND datetime <= toDateTime64(1589997600, 3)
 ```
 
-Note that these values can be wrapped in the ClickHouse toDateTime function, but ClickHouse will make that conversion implicitly and it is not required.   All string times are assumed to be UTC.
+These values can be wrapped in `toDateTime()` or `toDateTime64()` (including with a precision argument). All string times are assumed to be UTC.
+
+Queries using `toStartOfMonth`, `toStartOfQuarter`, or `toStartOfYear` (which return Date rather than DateTime) automatically wrap time range boundaries in `toDate()` during interpolation.
 
 ### Non-Time-Series Queries
 
 Queries that are not cacheable as time series — such as `LIMIT`-based queries, `SELECT 1` health checks, or SDK handshake requests — are transparently proxied to the upstream ClickHouse server. These requests are cached using the Object Proxy Cache (OPC) with per-query cache keys derived from the `query` and `database` URL parameters, ensuring that different SQL statements receive distinct cache entries.
 
+When a query falls back from DPC to OPC, Trickster logs a warning at the `warn` level to help identify queries that aren't getting time series acceleration. This can be suppressed per-backend:
+
+```yaml
+backends:
+  click1:
+    provider: clickhouse
+    dpc_fallback_warning: false
+```
+
 ### Health and Ping Endpoint
 
 Trickster exposes a `/ping` endpoint that returns a health check response, matching the endpoint provided by ClickHouse itself. This enables compatibility with clients and SDKs that probe `/ping` during connection initialization.

docs/developer/environment/docker-compose-data/dashboards/trickster-clickhouse-grafana.json

@@ -0,0 +1,183 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {
+        "type": "grafana-clickhouse-datasource",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": { "legend": false, "tooltip": false, "viz": false },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": { "type": "linear" },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": { "group": "A", "mode": "none" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green" },
+              { "color": "red", "value": 80 }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 22, "w": 12, "x": 0, "y": 0 },
+      "id": 1,
+      "options": {
+        "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true },
+        "tooltip": { "hideZeros": false, "mode": "single", "sort": "none" }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-clickhouse-datasource",
+            "uid": "${datasource}"
+          },
+          "editorType": "sql",
+          "format": 1,
+          "meta": { "builderOptions": { "database": "default", "table": "trips" } },
+          "queryType": "sql",
+          "rawSql": "SELECT toStartOfFiveMinute(pickup_datetime) AS t, count() AS Count FROM default.trips WHERE $__timeFilter(pickup_datetime) GROUP BY t ORDER BY t",
+          "refId": "A"
+        }
+      ],
+      "title": "# Trips Over Time",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "grafana-clickhouse-datasource",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": { "legend": false, "tooltip": false, "viz": false },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": { "type": "linear" },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": { "group": "A", "mode": "none" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green" },
+              { "color": "red", "value": 80 }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 22, "w": 12, "x": 12, "y": 0 },
+      "id": 2,
+      "options": {
+        "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true },
+        "tooltip": { "hideZeros": false, "mode": "single", "sort": "none" }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-clickhouse-datasource",
+            "uid": "${datasource}"
+          },
+          "editorType": "sql",
+          "format": 1,
+          "meta": { "builderOptions": { "database": "default", "table": "trips" } },
+          "queryType": "sql",
+          "rawSql": "WITH sumIf(trip_id, payment_type = 'CSH') AS cash_trips, sumIf(trip_id, payment_type != 'CSH') AS non_cash_trips SELECT toStartOfFiveMinute(pickup_datetime) AS t, non_cash_trips / (cash_trips + non_cash_trips) * 100 AS \"Card Use Rate\" FROM default.trips WHERE $__timeFilter(pickup_datetime) GROUP BY t ORDER BY t",
+          "refId": "A"
+        }
+      ],
+      "title": "% Trips Paid w/ Credit Card",
+      "type": "timeseries"
+    }
+  ],
+  "preload": false,
+  "refresh": "5s",
+  "schemaVersion": 41,
+  "tags": [],
+  "templating": {
+    "list": [
+      {
+        "allowCustomValue": false,
+        "current": {
+          "text": "clickhouse-grafana-direct",
+          "value": "clickhouse-grafana-direct"
+        },
+        "description": "",
+        "name": "datasource",
+        "options": [],
+        "query": "grafana-clickhouse-datasource",
+        "refresh": 1,
+        "regex": "",
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-15m",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "ClickHouse (Grafana Plugin)",
+  "uid": "clickhouse-grafana-plugin",
+  "version": 1
+}

docs/developer/environment/docker-compose-data/grafana-config/provisioning/datasources/datasource.yaml

@@ -230,4 +230,30 @@ datasources:
     defaultDatabase: default
     username: default
     password: ""
-    usePOST: true
+    usePOST: true
+
+# ClickHouse (Official Grafana Plugin)
+- name: clickhouse-grafana-direct
+  type: grafana-clickhouse-datasource
+  access: proxy
+  editable: true
+  isDefault: false
+  jsonData:
+    host: clickhouse
+    port: 8123
+    protocol: http
+    defaultDatabase: default
+    username: default
+
+- name: clickhouse-grafana-trickster
+  type: grafana-clickhouse-datasource
+  access: proxy
+  editable: true
+  isDefault: false
+  jsonData:
+    host: host.docker.internal
+    port: 8480
+    path: /click1/
+    protocol: http
+    defaultDatabase: default
+    username: default

docs/developer/environment/docker-compose.yml

@@ -22,7 +22,7 @@ services:
     image: grafana/grafana:11.6.0
     user: nobody
     environment:
-      - GF_INSTALL_PLUGINS=vertamedia-clickhouse-datasource
+      - GF_INSTALL_PLUGINS=vertamedia-clickhouse-datasource,grafana-clickhouse-datasource
     volumes:
       - ./docker-compose-data/grafana-config:/etc/grafana
       - ./docker-compose-data/dashboards:/var/lib/grafana/dashboards

docs/supported-backend-providers.md

@@ -22,6 +22,6 @@ See the [InfluxDB Support Document](./influxdb.md) for more information.
 
 ### <img src="./images/external/clickhouse_logo.png" width=16 /> ClickHouse
 
-Trickster supports accelerating ClickHouse time series. Specify `'clickhouse'` as the Provider when configuring Trickster.
+Trickster supports accelerating ClickHouse time series over both HTTP and the ClickHouse native binary protocol (port 9000), and is tested against the Vertamedia and official Grafana ClickHouse (v4+) datasource plugins. Specify `'clickhouse'` as the Provider when configuring Trickster.
 
 See the [ClickHouse Support Document](./clickhouse.md) for more information.