Contributions are always welcome! Use the Scraping Configuration help section to get started and stop by the Discord #scrapers channel with any questions.
If possible, XPath or JSON based yml scrapers are always preferred over python scrapers unless explicitly required
The scrapers in this repository can be validated against a schema and checked for common errors.
Deno is used as a drop-in, sandboxed NodeJS alternative
# check all scrapers
deno task validate
# check specific scraper
deno task validate scrapers/foo.yml scrapers/bar.ymlThe Deno version used in CI is pinned in .dvmrc
Instead of Deno, Docker can be used to run the validator
docker run --rm -v .:/app denoland/deno:distroless run -R=/app/ -E /app/validate.js --ci