Skip to content

Add the remaining pre-commit hooks, adopt sphinx-scrapy - #26

Merged
wRAR merged 7 commits into
scrapy-plugins:mainfrom
kishan3:pre-commit-hooks
Oct 15, 2026
Merged

wRAR merged 7 commits into
scrapy-plugins:mainfrom
kishan3:pre-commit-hooks

Conversation

@kishan3

@kishan3 kishan3 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Aligns the pre-commit config with Scrapy's: adds blacken-docs, pre-commit-hooks (end-of-file-fixer, trailing-whitespace), sphinx-lint, sphinx-scrapy and zizmor.

sphinx-scrapy adoption

Following scrapy/parsel#380:

  • [tool.sphinx-scrapy] in pyproject.toml, sphinx_scrapy extension in docs/conf.py
  • docs/requirements.txt pins sphinx-scrapy==0.13.0; tox.ini uses sphinx-scrapy[tox] and the custom docs env is removed
  • .readthedocs.yml builds via tox; docs/.gitignore added
  • docs/_ext removed: it registered the same directives as sphinx-scrapy, which fails the build
  • html_theme dropped, as sphinx_rtd_theme is no longer installed (docs now use the default theme)

Hook fixes

  • blacken-docs formatting in docs, EOF newline in MANIFEST.in
  • zizmor: actions pinned to SHAs, persist-credentials: false, minimal permissions
  • The trusted-publishing finding in publish.yml is suppressed inline; moving to trusted publishing needs PyPI-side setup.

Verification

pre-commit run --all-files and tox -e docs pass.

Jira: DC-1353

🤖 Generated with Claude Code

Add blacken-docs, pre-commit-hooks, sphinx-lint, sphinx-scrapy and zizmor,
matching the pre-commit configuration of Scrapy. Adopt sphinx-scrapy for the
docs and fix the findings of the new hooks.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kishan3

kishan3 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.09%. Comparing base (63b3991) to head (702e878).

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #26       +/-   ##
===========================================
+ Coverage   85.71%   98.09%   +12.38%     
===========================================
  Files           4        4               
  Lines         105      105               
  Branches       19       19               
===========================================
+ Hits           90      103       +13     
+ Misses         11        0       -11     
+ Partials        4        2        -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread .github/workflows/publish.yml Outdated
push:
tags:
- "[0-9]+.[0-9]+.[0-9]+"
permissions: {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line seems unnecessary. I also wonder why the permissions line is nested under jobs instead of here.

Comment thread docs/requirements.txt
@@ -1,2 +1 @@
Sphinx==7.2.4 # https://github.com/sphinx-doc/sphinx/issues/11662#issuecomment-1699989128
sphinx-rtd-theme==1.3.0
sphinx-scrapy==0.13.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you switch the project to use requirements.in / requirements.txt?

Comment thread .pre-commit-config.yaml Resolved
- Add the Dependabot config used in Scrapy to upgrade GitHub Actions.
- Add docs/requirements.in and compile docs/requirements.txt from it with uv.
- Set permissions (contents: read) at the workflow level instead of per job.
@AdrianAtZyte

Copy link
Copy Markdown

Could you also do zytedata/url-matcher#30 here? It may feel a bit out of scope, but given the CI failure, it may be easiest to handle it now here.

- Require Python 3.10+; CI and classifiers now cover 3.10-3.15. This also
  fixes the 3.9 CI jobs, as sphinx-scrapy requires 3.10+.
- Replace the nonexistent "min" tox env in CI with min-1x and min-2x, and
  pin scrapy==2.6.0 and w3lib<2 there, since Scrapy 1.0.0 cannot be
  imported on Python 3.10.
- Bump ruff (hook id is now ruff-check), black, mypy, pytest, twine and
  build, and ignore PLC0415 in tests/test_params.py.
- Add the "Typing :: Typed" classifier.
@kishan3

kishan3 commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Could you also do zytedata/url-matcher#30 here? It may feel a bit out of scope, but given the CI failure, it may be easiest to handle it now here.

Thanks Adrian.
I just made some adjustments according to it. Please check.

Comment thread .github/workflows/publish.yml Outdated
python -m build
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@v1.10.2
uses: pypa/gh-action-pypi-publish@897895f1e160c830e369f9779632ebc134688e1b # v1.10.2 # zizmor: ignore[use-trusted-publishing]

@AdrianAtZyte AdrianAtZyte Oct 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s switch to trusted publishing. (I’ll handle the PyPI setup)

Grant the deploy job id-token: write and drop the PYPI_TOKEN secret and
the zizmor ignore for use-trusted-publishing.
@kishan3

kishan3 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

@AdrianAtZyte thanks, I have fixed tox errors.

- Run the check jobs on 3.14, as the docs env requires it (matches
  .readthedocs.yml).
- Bump pytest-mypy-testing to 0.2.0; 0.1.3 requires pytest<9.
- Remove the extra blank line at the end of publish.yml.
Comment thread .pre-commit-config.yaml Resolved
Comment thread docs/requirements.txt Resolved

@wRAR wRAR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@wRAR
wRAR merged commit 36de44b into scrapy-plugins:main Oct 15, 2026
15 checks passed
@kishan3
kishan3 deleted the pre-commit-hooks branch October 15, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants