[SPARK-55910][SQL][TESTS] Merge SQLTestUtilsBase/SQLTestUtils into QueryTestBase/QueryTest by zhengruifeng · Pull Request #55381 · apache/spark

zhengruifeng · 2026-04-17T00:44:14Z

What changes were proposed in this pull request?

Merge both test utility hierarchies into one:

Move all SQLTestUtilsBase methods into QueryTestBase
Change QueryTest from abstract class to trait, move all SQLTestUtils methods into it
Make SQLTestUtilsBase and SQLTestUtils thin empty aliases

Changes in QueryTest.scala:

QueryTestBase (trait):

Add Eventually, BeforeAndAfterAll, SQLTestData mixins
Move all methods from SQLTestUtilsBase: testImplicits, sql, sparkContext, withSQLConf, withTable, withView, withTempView, withGlobalTempView, withDatabase, withNamespace, withCache, uncacheTable, withTempDatabase, withCurrentCatalogAndNamespace, withLocale, withSessionVariable, withUserDefinedFunction, activateDatabase, stripSparkFilter, logicalPlanToSparkQuery, makeQualifiedPath, testFile, getLocalDirSize

QueryTest (changed from abstract class to trait):

Add PlanTest mixin
Move all methods from SQLTestUtils: setupTestData, loadTestData, beforeAll, withTempDir, testWithWholeStageCodegenOnAndOff, testQuietly, test override (DisableAdaptiveExecution), testWithUninterruptibleThread, withResourceTempPath, waitForTasksToFinish, withTempPaths
Fix getCurrentClassCallSitePattern and getNextLineCallSitePattern to use lastIndexWhere for stack frame lookup — trait default methods may add mixin forwarder frames that shift the stack layout

Changes in SQLTestUtils.scala:

SQLTestUtilsBase → empty trait alias of QueryTestBase
SQLTestUtils → empty trait alias of QueryTest
object SQLTestUtils.compareAnswers kept as-is

Unused import fixes:

AlterTableTests.scala — removed unused QueryTest.checkAnswer static import
HivePlanTest.scala — removed unused import spark.sql (now inherited)

Why are the changes needed?

Consolidates two parallel test utility hierarchies (QueryTestBase/QueryTest and SQLTestUtilsBase/SQLTestUtils) into one. Previously they shared PlanTestBase and SparkSessionProvider but were maintained as separate hierarchies, causing confusion about which to extend.

After this PR:

QueryTestBase (trait — has all base utilities from both old QueryTestBase + SQLTestUtilsBase)
└── QueryTest (trait — has all test-registration methods from both old QueryTest + SQLTestUtils)
    ├── SQLTestUtils (empty alias)
    └── SQLTestUtilsBase (empty alias, also extends QueryTestBase)

Changing QueryTest from abstract class to trait enables SQLTestUtils to extend it directly, and allows test suites to mix in QueryTest alongside SparkFunSuite or SparkPlanTest without inheritance conflicts.

Semantic change: `DisableAdaptiveExecution` tag handling

The test() override that honors @DisableAdaptiveExecution previously lived only on SQLTestUtils. Moving it onto QueryTest means every extends QueryTest suite now honors the tag — including ones that previously didn't (e.g., HivePlanTest and a handful of connector suites that extend QueryTest directly without going through SharedSparkSession). No live regression today since all current @DisableAdaptiveExecution usages go through SharedSparkSession, but this is a semantic expansion beyond just moving methods between traits.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI compilation across all modules
Locally verified affected tests pass (DataFrameFunctionsSuite, ReplaceIntegerLiteralsWithOrdinalsDataframeSuite, QueryTestSuite)

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-opus-4-6)

…tBase Move all methods from `SQLTestUtilsBase` into `QueryTestBase`, making `SQLTestUtilsBase` a thin alias that extends `QueryTestBase`. This is the first step toward consolidating the test utility class hierarchy by merging `SQLTestUtils` into `QueryTest`. Co-authored-by: Isaac

…erge Co-authored-by: Isaac

…sBase Use fully qualified name to avoid unused import warning while ensuring the type is resolved correctly from the sub-package. Co-authored-by: Isaac

…Utils Co-authored-by: Isaac

…nTest Co-authored-by: Isaac

Move all methods from `SQLTestUtils` into `QueryTest`, making `SQLTestUtils` a thin alias. Also move `compareAnswers` into `object QueryTest`. Co-authored-by: Isaac

… abstract class SQLTestUtils must remain a trait extending only traits (QueryTestBase, PlanTest) rather than the abstract class QueryTest, because many test suites mix in SQLTestUtils with SparkFunSuite or SparkPlanTest directly. Co-authored-by: Isaac

SQLTestUtils cannot extend QueryTest (abstract class) because many test suites mix SQLTestUtils with SparkFunSuite or SparkPlanTest directly. Methods that call test() must stay in a class/trait that extends SparkFunSuite. Reverting to keep SQLTestUtils unchanged. This PR now only merges SQLTestUtilsBase into QueryTestBase. Co-authored-by: Isaac

…sBase Co-authored-by: Isaac

…for QueryTestBase Co-authored-by: Isaac

…Utils into it Change QueryTest from abstract class to trait, enabling SQLTestUtils to extend it directly. Move all SQLTestUtils methods into QueryTest. Both SQLTestUtils and SQLTestUtilsBase are now thin aliases. Co-authored-by: Isaac

…QLTestUtilsBase Co-authored-by: Isaac

Co-authored-by: Isaac

…QueryTest When QueryTest was an abstract class, getStackTrace()(2) reliably returned the caller's frame. As a trait with default methods, the stack layout may differ. Use dynamic stack walking to find the first frame outside QueryTest.scala instead of a hardcoded index. Co-authored-by: Isaac

… filename Find the current method's frame by name, then take the next frame as the caller. No hardcoded filenames or indices. Co-authored-by: Isaac

…thods Use lastIndexWhere instead of indexWhere to find the method's frame, skipping any mixin forwarder frames that trait compilation may add. Co-authored-by: Isaac

cloud-fan

This PR unifies the parallel QueryTestBase/QueryTest and SQLTestUtilsBase/SQLTestUtils hierarchies. QueryTestBase absorbs everything from SQLTestUtilsBase (gaining Eventually, BeforeAndAfterAll, SQLTestData mixins plus ~22 helpers), QueryTest absorbs everything from SQLTestUtils and is converted from abstract class to trait so it can be mixed in alongside SparkFunSuite/SparkPlanTest. SQLTestUtilsBase and SQLTestUtils are left as empty aliases for backward compatibility.

The mechanical move looks clean. The trait conversion is backward compatible — every existing suite uses extends QueryTest with ..., so SparkFunSuite stays anchored in the linearization. The lastIndexWhere stack-trace fix correctly handles both plain trait-default and mixin-forwarder frame layouts. The two unused-import cleanups (AlterTableTests, HivePlanTest) are correct — checkAnswer and sql are now inherited through QueryTestBase.

A few things to address:

Keeping SQLTestUtils / SQLTestUtilsBase as empty aliases for backward compatibility is fine, but their scaladoc should say so explicitly (and steer new code to QueryTest/QueryTestBase), and the useful guidance that was previously on SQLTestUtils should move onto QueryTest rather than getting dropped.
Moving the DisableAdaptiveExecution-aware test() override from SQLTestUtils onto QueryTest is a behavior expansion that isn't called out in the PR description — see the inline comment.
A couple of small cleanups (redundant import, redundant self-type).

cloud-fan · 2026-04-20T05:26:55Z

- *
- * Subclasses should *not* create `SparkSession`s in the test suite constructor, which is
- * prone to leaving multiple overlapping [[org.apache.spark.SparkContext]]s in the same JVM.
+ * code base. Now a thin alias for [[QueryTest]].


Since the traits are kept as empty aliases for backward compatibility, their scaladoc should explain that explicitly — e.g. "Kept as an empty alias of QueryTest for backward compatibility with existing subclasses. New test suites should extend QueryTest/QueryTestBase directly." Right now "Now a thin alias for [[QueryTest]]" tells a reader about the aliasing but not the intent (backward compat, don't use for new code). Same for SQLTestUtilsBase on line 33.

Separately, the old SQLTestUtils scaladoc carried useful guidance that shouldn't be lost — "Subclasses should not create SparkSessions in the test suite constructor, which is prone to leaving multiple overlapping SparkContexts…" and "import testImplicits._ instead of through the SparkSession." Please lift that onto the QueryTest trait (QueryTest.scala:541) so future readers still see it.

updated the comments

cloud-fan · 2026-04-20T05:26:55Z

-      .map(_.toFile.length).sum
-  }
-}
+private[sql] trait SQLTestUtilsBase extends QueryTestBase { self: Suite => }


QueryTestBase already has { self: Suite => }, so the explicit self-type here is redundant.

Suggested change

private[sql] trait SQLTestUtilsBase extends QueryTestBase { self: Suite => }

private[sql] trait SQLTestUtilsBase extends QueryTestBase

it is needed, otherwise

[error] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala:33:45: illegal inheritance; [error] self-type org.apache.spark.sql.test.SQLTestUtilsBase does not conform to org.apache.spark.sql.QueryTestBase's selftype org.apache.spark.sql.QueryTestBase with org.scalatest.Suite [error] private[sql] trait SQLTestUtilsBase extends QueryTestBase [error] ^ [warn] two warnings found [error] one error found [error] (sql / Test / compileIncremental) Compilation failed [error] Total time: 452 s (0:07:32.0), completed Apr 20, 2026, 10:26:54 AM

cloud-fan · 2026-04-20T05:26:55Z

+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
+import org.apache.spark.sql.catalyst.catalog.SessionCatalog.DEFAULT_DATABASE
 import org.apache.spark.sql.catalyst.plans._
+import org.apache.spark.sql.catalyst.plans.PlanTest


The wildcard import org.apache.spark.sql.catalyst.plans._ on line 41 already brings in PlanTest, so this explicit import is redundant.

cloud-fan · 2026-04-20T05:26:55Z

+    }
+  }
+
+  override protected def test(testName: String, testTags: Tag*)(testFun: => Any)


This override previously lived on SQLTestUtils, so only suites extending SQLTestUtils/SharedSparkSession honored DisableAdaptiveExecution. Moving it onto QueryTest means every extends QueryTest suite now honors the tag — including ones that previously didn't (HivePlanTest, a handful of connector suites). No live regression today (all current @DisableAdaptiveExecution users already go through SharedSparkSession), but worth mentioning in the PR description since it's a semantic expansion beyond "move methods between traits."

mentioned in PR desc

- Improve scaladoc for SQLTestUtils/SQLTestUtilsBase to explain backward compat intent - Lift the SparkSession usage guidance onto QueryTest trait - Remove redundant self-type on SQLTestUtilsBase - Remove redundant PlanTest import (already covered by wildcard) Co-authored-by: Isaac

Co-authored-by: Isaac

zhengruifeng · 2026-04-21T00:22:58Z

thanks, merged into master

zhengruifeng changed the title ~~Sql merge utils into query~~ [SPARK-55910][SQL][TESTS] Sql merge utils into query Apr 17, 2026

zhengruifeng changed the title ~~[SPARK-55910][SQL][TESTS] Sql merge utils into query~~ [SPARK-55910][SQL][TESTS] Move SQLTestUtilsBase content into QueryTestBase Apr 17, 2026

zhengruifeng force-pushed the sql_merge_utils_into_query branch from 0983e20 to b120fb6 Compare April 17, 2026 01:03

zhengruifeng changed the title ~~[SPARK-55910][SQL][TESTS] Move SQLTestUtilsBase content into QueryTestBase~~ [SPARK-55910][SQL][TESTS] Merge SQLTestUtilsBase into QueryTestBase and SQLTestUtils into QueryTest Apr 17, 2026

zhengruifeng changed the title ~~[SPARK-55910][SQL][TESTS] Merge SQLTestUtilsBase into QueryTestBase and SQLTestUtils into QueryTest~~ [SPARK-55910][SQL][TESTS] Merge SQLTestUtilsBase/SQLTestUtils into QueryTestBase/QueryTest Apr 17, 2026

zhengruifeng added 17 commits April 17, 2026 07:49

[SPARK-XXXXX][SQL][TESTS] Fix unused imports after SQLTestUtilsBase m…

93f154a

…erge Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Fix QueryTestBase resolution in SQLTestUtil…

f1ec729

…sBase Use fully qualified name to avoid unused import warning while ensuring the type is resolved correctly from the sub-package. Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Remove unused QueryTest import from SQLTest…

5d2fa2e

…Utils Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Remove unused spark.sql import from HivePla…

f6031c7

…nTest Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Move SQLTestUtils content into QueryTest

50073ed

Move all methods from `SQLTestUtils` into `QueryTest`, making `SQLTestUtils` a thin alias. Also move `compareAnswers` into `object QueryTest`. Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Remove redundant self-type from SQLTestUtil…

73b030b

…sBase Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Use import instead of fully qualified name …

b0ef9b4

…for QueryTestBase Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Fix import order and restore self-type on S…

e726a7c

…QLTestUtilsBase Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Add missing Suite import in SQLTestUtils

84f2ae6

Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Fix import order for PlanTest in QueryTest

29142a1

Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Use method name lookup instead of hardcoded…

ef759f0

… filename Find the current method's frame by name, then take the next frame as the caller. No hardcoded filenames or indices. Co-authored-by: Isaac

[SPARK-XXXXX][SQL][TESTS] Fix stack trace lookup for trait default me…

df22c24

…thods Use lastIndexWhere instead of indexWhere to find the method's frame, skipping any mixin forwarder frames that trait compilation may add. Co-authored-by: Isaac

zhengruifeng force-pushed the sql_merge_utils_into_query branch from 52fec98 to df22c24 Compare April 17, 2026 07:49

zhengruifeng marked this pull request as ready for review April 17, 2026 09:38

zhengruifeng requested a review from cloud-fan April 17, 2026 09:39

cloud-fan reviewed Apr 20, 2026

View reviewed changes

zhengruifeng added 2 commits April 20, 2026 09:41

[SPARK-55910][SQL][TESTS] Restore self-type on SQLTestUtilsBase

5d3654f

Co-authored-by: Isaac

cloud-fan approved these changes Apr 20, 2026

View reviewed changes

zhengruifeng closed this in 4a471e0 Apr 21, 2026

zhengruifeng deleted the sql_merge_utils_into_query branch April 21, 2026 00:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-55910][SQL][TESTS] Merge SQLTestUtilsBase/SQLTestUtils into QueryTestBase/QueryTest#55381

[SPARK-55910][SQL][TESTS] Merge SQLTestUtilsBase/SQLTestUtils into QueryTestBase/QueryTest#55381
zhengruifeng wants to merge 19 commits intoapache:masterfrom
zhengruifeng:sql_merge_utils_into_query

zhengruifeng commented Apr 17, 2026 •

edited

Loading

Uh oh!

cloud-fan left a comment

Uh oh!

cloud-fan Apr 20, 2026

Uh oh!

zhengruifeng Apr 20, 2026

Uh oh!

cloud-fan Apr 20, 2026

Uh oh!

zhengruifeng Apr 20, 2026

Uh oh!

zhengruifeng Apr 20, 2026

Uh oh!

cloud-fan Apr 20, 2026

Uh oh!

zhengruifeng Apr 20, 2026

Uh oh!

cloud-fan Apr 20, 2026

Uh oh!

zhengruifeng Apr 20, 2026

Uh oh!

zhengruifeng commented Apr 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	private[sql] trait SQLTestUtilsBase extends QueryTestBase { self: Suite => }
	private[sql] trait SQLTestUtilsBase extends QueryTestBase

Conversation

zhengruifeng commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Semantic change: DisableAdaptiveExecution tag handling

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng commented Apr 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhengruifeng commented Apr 17, 2026 •

edited

Loading

Semantic change: `DisableAdaptiveExecution` tag handling