ci: scope Spark SQL trigger paths to per-version shims and diff#4415
Merged
Conversation
Each spark_sql_test_<v>.yml currently triggers on `spark/src/main/**` and `dev/diffs/**`, so a change confined to one Spark version's shim or diff still fans out and runs the other versions' jobs. Tighten the path allow-list per version: - exclude unrelated `spark/src/main/spark-3.4/`, `spark-3.5/`, `spark-3.x/`, `spark-4.0/`, `spark-4.1/`, `spark-4.2/`, `spark-4.x/` directories so a 3.4-only shim edit never fires the 4.x workflows, a 4.1-only shim edit never fires the 3.x workflows, and a future `spark-4.3/` shim won't trigger any 3.x workflow either - replace `dev/diffs/**` with the single `dev/diffs/<full-version>.diff` the workflow actually applies, which also stops `dev/diffs/iceberg/` edits from triggering the Spark SQL test workflows
mbutrovich
approved these changes
May 26, 2026
Contributor
mbutrovich
left a comment
mbutrovich
left a comment
There was a problem hiding this comment.
I think this looks right to me, thanks @andygrove!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tighten the path allow-list on each
spark_sql_test_<v>.ymlso a change confined to one Spark version no longer fans out to the others.Each workflow currently triggers on
spark/src/main/**anddev/diffs/**. Today that means:spark/src/main/spark-4.1/runs the 3.5 and 4.0 workflows for no reasondev/diffs/3.4.3.diffruns the 4.0 workflow for no reasondev/diffs/iceberg/*.diffruns all four Spark SQL workflows for no reasonspark/src/main/spark-4.3/shim it will run the 3.x workflows for no reasonThe change is purely to the
paths:filters; job logic is untouched.Per-version mapping (from
pom.xml)spark-3.4,spark-3.xdev/diffs/3.4.3.diffspark-3.5,spark-3.xdev/diffs/3.5.8.diffspark-4.0,spark-4.xdev/diffs/4.0.2.diffspark-4.1,spark-4.xdev/diffs/4.1.1.diffEach workflow keeps
spark/src/main/**(so unrelated Java/Scala/resources still trigger) but adds!-exclusions for the shim dirs that don't apply, and replacesdev/diffs/**with the single applicable diff file.Both
push:andpull_request:filters are updated where present (3.4 and 4.1 only havepush:paths since their PR trigger islabeled).Test plan
spark/src/main/spark-4.1/...in a follow-up PR and confirm only the 4.1 workflow is queueddev/diffs/3.5.8.diffand confirm only the 3.5 workflow is queueddev/diffs/iceberg/1.10.0.diffand confirm none of the Spark SQL workflows are queued