Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,25 @@ on:
push:
tags:
- "[0-9]+.[0-9]+.[0-9]+"
permissions: {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line seems unnecessary. I also wonder why the permissions line is nested under jobs instead of here.

jobs:
deploy:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: '3.13'
- name: Install dependencies
run: |
pip install --upgrade build twine
python -m build
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@v1.10.2
uses: pypa/gh-action-pypi-publish@897895f1e160c830e369f9779632ebc134688e1b # v1.10.2 # zizmor: ignore[use-trusted-publishing]

@AdrianAtZyte AdrianAtZyte Oct 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s switch to trusted publishing. (I’ll handle the PyPI setup)

with:
password: ${{ secrets.PYPI_TOKEN }}
32 changes: 25 additions & 7 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,48 +3,66 @@ on:
pull_request:
push:
branches: [ main ]
permissions: {}
jobs:
test:
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- python-version: "3.9"
toxenv: "min"
- python-version: "3.9"
toxenv: "py39"
- python-version: "3.10"
toxenv: "py310"
- python-version: "3.11"
toxenv: "py311"
- python-version: "3.12"
toxenv: "py312"
- python-version: "3.13"
toxenv: "py313"
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: ${{ matrix.python-version }}
# The first version runs tox itself (sphinx-scrapy requires 3.10+),
# the second one is used by the test environment.
python-version: |
3.13
${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox
- name: tox
run: |
tox -e ${{ matrix.toxenv || 'py' }}
tox -e ${{ matrix.toxenv }}
- name: coverage
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
check:
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
fail-fast: false
matrix:
python-version: ['3.13'] # Keep in sync with .readthedocs.yml
tox-job: ["pre-commit", "mypy", "types", "docs", "twinecheck"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand Down
24 changes: 24 additions & 0 deletions .pre-commit-config.yaml
Comment thread
wRAR marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,27 @@ repos:
- id: ruff
args: [ --fix ]
- id: ruff-format
- repo: https://github.com/adamchainz/blacken-docs
rev: 1.20.0
hooks:
- id: blacken-docs
additional_dependencies:
- black==26.5.1
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/sphinx-contrib/sphinx-lint
rev: v1.0.2
hooks:
- id: sphinx-lint
- repo: https://github.com/scrapy/sphinx-scrapy
rev: 0.13.0
hooks:
- id: sphinx-scrapy
- repo: https://github.com/zizmorcore/zizmor-pre-commit
rev: v1.30.1
hooks:
- id: zizmor
args: [--no-progress, --fix]
Comment thread
wRAR marked this conversation as resolved.
Resolved
12 changes: 4 additions & 8 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
version: 2
formats: all
sphinx:
configuration: docs/conf.py
build:
os: ubuntu-24.04
tools:
python: "3.13" # Keep in sync with .github/workflows/tests.yml
python:
install:
- requirements: docs/requirements.txt
- path: .
python: "3.13"
commands:
- pip install tox
- tox -e docs -- $READTHEDOCS_OUTPUT/html
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1 +1 @@
recursive-include tests *
recursive-include tests *
1 change: 1 addition & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/_build/
25 changes: 0 additions & 25 deletions docs/_ext/__init__.py

This file was deleted.

22 changes: 1 addition & 21 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -1,30 +1,10 @@
import sys
from pathlib import Path

project = "scrapy-spider-metadata"
copyright = "2023, Zyte Group Ltd"
author = "Zyte Group Ltd"
release = "0.0.0"

sys.path.insert(0, str(Path(__file__).parent.absolute())) # _ext
extensions = [
"_ext",
"sphinx.ext.autodoc",
"sphinx.ext.intersphinx",
"sphinx.ext.viewcode",
"sphinx_scrapy",
]

exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]

html_theme = "sphinx_rtd_theme"

intersphinx_mapping = {
"python": (
"https://docs.python.org/3",
None,
),
"scrapy": (
"https://docs.scrapy.org/en/latest",
None,
),
}
4 changes: 4 additions & 0 deletions docs/metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ attribute named ``metadata``:

from scrapy import Spider


class MySpider(Spider):
name = "my_spider"
metadata = {
Expand All @@ -32,18 +33,21 @@ add or change some values in a subclass:

from scrapy import Spider


class BaseSpider(Spider):
metadata = {
"description": "Base spider.",
"category": "Base spiders",
}


class BaseNewsSpider(BaseSpider):
metadata = {
**BaseSpider.metadata,
"description": "Base news spider.",
}


class CNNSpider(BaseNewsSpider):
metadata = {
**BaseNewsSpider.metadata,
Expand Down
2 changes: 2 additions & 0 deletions docs/params.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,11 @@ as its parameter:
from scrapy import Request, Spider
from scrapy_spider_metadata import Args


class MyParams(BaseModel):
pages: int


class MySpider(Args[MyParams], Spider):
name = "my_spider"

Expand Down
3 changes: 1 addition & 2 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
Sphinx==7.2.4 # https://github.com/sphinx-doc/sphinx/issues/11662#issuecomment-1699989128
sphinx-rtd-theme==1.3.0
sphinx-scrapy==0.13.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you switch the project to use requirements.in / requirements.txt?

3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -189,3 +189,6 @@ runtime-evaluated-base-classes = ["pydantic.BaseModel"]

[tool.ruff.lint.pydocstyle]
convention = "pep257"

[tool.sphinx-scrapy]
python-version = "3.13"
10 changes: 2 additions & 8 deletions tox.ini
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
[tox]
requires =
sphinx-scrapy[tox]==0.13.0
envlist = py,pre-commit,mypy,types,docs,twinecheck

[pytest]
Expand Down Expand Up @@ -59,14 +61,6 @@ deps =
commands =
pytest {posargs:tests_typing}

[testenv:docs]
basepython = python3
changedir = docs
deps =
-rdocs/requirements.txt
commands =
sphinx-build -W -b html . {envtmpdir}/html

[testenv:twinecheck]
basepython = python3
deps =
Expand Down