Skip to content

[SPARK-56018][INFRA][FOLLOW-UP] Install black for older branches in Python CodeGen check#55407

Closed
LuciferYang wants to merge 1 commit intoapache:masterfrom
LuciferYang:SPARK-56018-FOLLOWUP
Closed

[SPARK-56018][INFRA][FOLLOW-UP] Install black for older branches in Python CodeGen check#55407
LuciferYang wants to merge 1 commit intoapache:masterfrom
LuciferYang:SPARK-56018-FOLLOWUP

Conversation

@LuciferYang
Copy link
Copy Markdown
Contributor

@LuciferYang LuciferYang commented Apr 18, 2026

What changes were proposed in this pull request?

This PR adds a conditional step in the buf job of build_and_test.yml to install black for older branches while keeping ruff for master and newer branches.

Why are the changes needed?

SPARK-56018 replaced black with ruff as the Python formatter in the "Install dependencies for Python CodeGen check" step. However, this workflow is also used by scheduled builds for older branches, whose gen-protos.sh still calls black for formatting. This causes those builds to fail with:

+ black --config /home/runner/work/spark/spark/dev/pyproject.toml gen/proto/python
/home/runner/work/spark/spark/dev/gen-protos.sh: line 119: black: command not found
Traceback (most recent call last):
  File "/home/runner/work/spark/spark/./dev/check-protos.py", line 84, in <module>
Start checking the generated codes in pyspark-connect.
RUN: /home/runner/work/spark/spark/dev/gen-protos.sh connect /tmp/check_connect__protos6imuoe88
    check_protos(
  File "/home/runner/work/spark/spark/./dev/check-protos.py", line 49, in check_protos
    run_cmd(f"{SPARK_HOME}/dev/gen-protos.sh {module_name} {tmp}")
  File "/home/runner/work/spark/spark/./dev/check-protos.py", line 43, in run_cmd
    return subprocess.check_output(cmd.split(" ")).decode("utf-8")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/runner/work/spark/spark/dev/gen-protos.sh', 'connect', '/tmp/check_connect__protos6imuoe88']' returned non-zero exit status 127.

For example: https://github.com/apache/spark/actions/runs/24597946594/job/71931532850

Does this PR introduce any user-facing change?

No

How was this patch tested?

Monitor the daily test results for branch-4.1 after merging.

Was this patch authored or co-authored using generative AI tooling?

No

…en check

SPARK-56018 replaced `black` with `ruff` in the Python CodeGen check step, but older branches (branch-3.5, branch-4.0, branch-4.1) still use `black` in their `gen-protos.sh`. This causes scheduled builds for those branches to fail with `black: command not found`.

This patch adds a conditional step to install `black` for older branches while keeping `ruff` for master and newer branches.
@HyukjinKwon HyukjinKwon changed the title [SPARK-56018][INFRA][FOLLOWUP] Install black for older branches in Python CodeGen check [SPARK-56018][INFRA][FOLLOW-UP] Install black for older branches in Python CodeGen check Apr 19, 2026
@LuciferYang
Copy link
Copy Markdown
Contributor Author

Merged into master.Thanks @dongjoon-hyun and @HyukjinKwon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants