feat: expand date/time expression support using codegen dispatcher

[andygrove]

Which issue does this PR close?

Part of #3202. Supersedes and consolidates #4373.

Expressions covered

Ten Spark date/time expressions are routed through the Arrow-direct codegen dispatcher in this PR. All run inside the Comet pipeline (no operator-level Spark fallback) when spark.comet.exec.scalaUDF.codegen.enabled=true is set. Default behavior is unchanged when the flag is off.

• date_format (originally feat: route date_format through codegen dispatcher for non-native cases #4373; native to_char path retained for UTC + whitelisted-format cases, dispatcher path for everything else)
• add_months
• months_between
• make_timestamp
• timestamp_millis (MillisToTimestamp)
• timestamp_micros (MicrosToTimestamp)
• unix_seconds
• unix_millis
• unix_micros
• to_unix_timestamp

Rationale for this change

Comet's plan rules fall back to Spark for any expression that lacks a native serde, breaking up the Comet pipeline at the operator boundary. The Arrow-direct codegen dispatcher already in tree (CometScalaUDFCodegen, behind spark.comet.exec.scalaUDF.codegen.enabled) closure-serializes a bound Catalyst expression, ships it through a JvmScalarUdf proto, and Janino-compiles Spark's own doGenCode into a per-batch kernel that reads Arrow vectors and writes an Arrow output vector directly. For any expression whose doGenCode is real (not CodegenFallback) and whose input/output types fit CometBatchKernelCodegen.isSupportedDataType, routing through this dispatcher reproduces Spark behavior exactly without a bespoke UDF class.

This PR establishes a reusable helper for that routing pattern and applies it to the ten expressions above in two waves: DateFormatClass first (originally #4373, folded in here), then nine currently-unsupported expressions that share the same shape.

What changes are included in this PR?

fix(codegen) — Pre-existing bug surfaced while wiring date_format. CometBatchKernelCodegen.defaultBody emitted this.col$ord.isNull(i) for every NullIntolerant input, but primitive Arrow vectors are wrapped in CometPlainVector at input-cast time and expose isNullAt, not isNull. Janino rejected the kernel with "method isNull not declared". emitTypedGetters already knew the right method name via nullCheckMethod; the fix exposes that helper so defaultBody picks the same name per column ordinal. New source test pins the chosen method for TimeStampMicroTZVector so the regression can't recur.

feat — dispatcher helper extraction — Extract the closure-serialize + JvmScalarUdf emission from CometScalaUDF.convert into CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding) so other serdes can reuse the path. CometDateFormat.convert keeps the native to_char path for UTC + whitelisted-format cases and now calls the helper for everything else. Gated by spark.comet.exec.scalaUDF.codegen.enabled (default false, experimental); default behavior is unchanged.

feat — CometCodegenDispatch[T] base class — A one-class wrapper around emitJvmCodegenDispatch lets any expression with a real doGenCode slot in with a single-line object declaration. The helper marks the expression Compatible() because the dispatcher runs Spark's own doGenCode inside the kernel; behavior matches Spark exactly when the flag is on, and the operator falls back cleanly when it is off.

feat — nine new datetime serdes — Each of the nine new expressions becomes a one-line singleton extends CometCodegenDispatch[T] and is registered in temporalExpressions. Three are ANSI-sensitive (MakeTimestamp, MillisToTimestamp, ToUnixTimestamp carry failOnError); the dispatcher inherits the throw site from Spark's own doGenCode, so exception semantics propagate without any serde-level branching.

test — MaxSparkVersion annotation — Mirror the existing MinSparkVersion parser gate with a MaxSparkVersion ceiling. The CometSqlFileTestSuite skip logic now reports whether a constraint was a floor or ceiling. This lets fixtures pair a 3.5+ variant against a 3.4 variant where the expected error class wording differs.

Interval-producing expressions (MakeInterval/MakeYMInterval/MakeDTInterval) are explicitly out of scope: the dispatcher's isSupportedDataType does not include Spark's interval types. Version-conditional expressions (TimestampAdd/TimestampDiff 3.4+, DayName 3.5+, MonthName 4.0+) are deferred to a follow-on so this PR avoids touching the CometExprShim files.

Scaffolded with the superpowers:brainstorming and superpowers:writing-plans skills.

How are these changes tested?

• CometTemporalExpressionSuite date_format tests: 10/10 pass. Three "falls back to Spark" tests are paired with a "routes via codegen dispatcher" sibling that enables the flag and asserts in-Comet execution. date_format - timestamp_ntz input runs checkSparkAnswerAndOperator for every timezone under the codegen flag.
• CometSqlFileTestSuite: nine new per-expression SQL fixtures (add_months.sql, months_between.sql, make_timestamp.sql, timestamp_millis.sql, timestamp_micros.sql, unix_seconds.sql, unix_millis.sql, unix_micros.sql, to_unix_timestamp.sql) pin a non-UTC session timezone and the codegen flag at file scope. date_format.sql from feat: route date_format through codegen dispatcher for non-native cases #4373 is included.
• ANSI exception coverage: paired fixtures make_timestamp_ansi.sql / make_timestamp_ansi_spark34.sql, to_unix_timestamp_ansi.sql / to_unix_timestamp_ansi_spark34.sql, plus timestamp_millis_ansi.sql. The Spark 3.5+ files match the DATETIME_FIELD_OUT_OF_BOUNDS / CANNOT_PARSE_TIMESTAMP error classes; the 3.4 files match the underlying JDK java.time exception text (MonthOfYear, Invalid date, HourOfDay, could not be parsed). Each ANSI file includes a sentinel non-error query that uses checkSparkAnswerAndOperator so a silent dispatcher fallback would fail the file (the expect_error path alone passes vacuously because Spark fallback also throws).
• CometCodegenSourceSuite: includes the new NullIntolerant short-circuit uses isNullAt for CometPlainVector-wrapped columns regression test plus a parameterized Bucket 4 datetime expressions produce non-empty generated kernel source test covering all nine new expressions.
• CometCodegenSuite: no regressions in the dispatcher's existing surface.
• Local Spark 3.5 run: 284/284 SQL fixtures pass (3.4 variants correctly skip). Local Spark 3.4 run with -Pspark-3.4: 284/284 SQL fixtures pass (3.5+ variants correctly skip).

[mbutrovich]

Awesome to see the codegen framework being put to good use, and bugs being found and fixed!

A couple of things on the broader dispatcher pattern worth thinking about before the second wave of expressions lands.

The sentinel query after each expect_error block is cool. I checked make_timestamp_ansi.sql, timestamp_millis_ansi.sql, and to_unix_timestamp_ansi.sql and the sentinel is in all three. The concern is that nothing in CometSqlFileTestSuite or SqlFileTestParser enforces "any fixture combining expect_error with spark.comet.exec.scalaUDF.codegen.enabled=true must include a non-error sentinel using checkSparkAnswerAndOperator." As more codegen-dispatched ANSI expressions adopt this fixture pattern, a forgotten sentinel would let a regression silently pass. Could the parser flag that shape, or at minimum could a comment in SqlFileTestParser document the requirement so the next contributor isn't relying on conventions absorbed by osmosis?

On the cache-key side, three of the new expressions are failOnError-sensitive (MakeTimestamp, MillisToTimestamp, ToUnixTimestamp), MonthsBetween carries a roundOff flag, and several are timezone-aware. The dispatcher caches kernels by closure-serialized bytes, which should naturally differentiate them, but I don't see a test pinning that two plans of the same expression class differing only in failOnError (or roundOff, or session timezone) compile to distinct kernels rather than colliding in the cache. A small unit test in CometScalaUDFCodegen's suite that asserts the serialized bytes diverge for those pairs would lock in the invariant cheaply. Worth adding while the shapes are still fresh?

[mbutrovich]

I think expressions.md needs updating, but that's just a docs PR, no reasont to kick CI again on this PR. Awesome to see more use for the codegen path!

[andygrove]

Merged. Thanks @mbutrovich. I will follow up with docs PR.

[parthchandra]

Some post merge comments -

• This really should have been two PRs.
• The codegen dispatcher is incredibly powerful but we must be careful to use it only when necessary. It's terrific that the implemented functions cause a host of issues, but perhaps we can restrict future conversions to those cases where compatible native implementations are impossible?

[andygrove]

Some post merge comments -

• This really should have been two PRs.
• The codegen dispatcher is incredibly powerful but we must be careful to use it only when necessary. It's terrific that the implemented functions cause a host of issues, but perhaps we can restrict future conversions to those cases where compatible native implementations are impossible?

This is just adding a cheaper fallback for expressions that are not yet implemented. I'm happy to review PRs to add native implementations and remove the fallbacks.