Skip to content

SQL lowering: run DataFusion's SimplifyExpressions (needs IR timestamp literals and regex operators) #650

Description

@zzylol

Problem

The SQL front end lowers DataFusion's plan right after type coercion and does not run DataFusion's expression simplification (SimplifyExpressions). A DataFusion query reaches its executor simplified, and ours does not, so the lowered IR can differ from what DataFusion would execute. Today only one piece is matched by hand: #539 folds a numeric cast of a literal (latency < CAST(100 AS DOUBLE) → latency < 100.0) with DataFusion's ScalarValue::cast_to.

Running the whole rule (tried on the #646 restack, 2026-10-10) breaks the SQL corpora (bgp_analytics, bgp_jan2024_workload, netflow, tpch_deequ) in two ways:

  1. Timestamp literals. time >= CAST('2018-05-17 13:10:01' AS TIMESTAMP) folds to TimestampMillisecond(1526562601000). The IR has no timestamp literal, so lowering fails with unsupported scalar. Nearly every BGP and NetFlow query has a time filter.
  2. Regex. REGEXP_LIKE(col, pat) is rewritten into the operator col ~ pat (RegexMatch), which lowering does not support (unsupported feature: operator: RegexMatch).

The rule rewrites more than these two; the corpora only hit these.

Proposal

Run DataFusion's SimplifyExpressions after TypeCoercion, with an OptimizerContext without a query start time (without_query_execution_start_time()), so now() stays symbolic and still lowers to CurrentTimestamp. To make that work:

  • Add a timestamp literal to the IR's ScalarValue (types, planner, executor).
  • Lower RegexMatch / RegexIMatch / RegexNotMatch / RegexNotIMatch, and any other shape the simplifier produces, found by running the corpora.
  • Drop the hand-written numeric cast fold from feat: lower SQL through unified operators and scalars #539.

A timestamp literal also lets summary coverage put an absolute time filter (ts >= t0 AND ts < t1) into the selection, which #573 §4.2.2 lists as "later": today it stays in the definition.

Related

#539 (SQL lowering), #646 / #573 (summary coverage).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions