Skip to content

v0.4.11 #374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
D4Vinci merged 13 commits into from
Jul 12, 2026
Merged

v0.4.11 #374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
2548fbc
fix(ai): strip control characters before extraction (#366)
yetval Jul 6, 2026
aeb0f87
perf: wrap elements lazily in find_by_text/find_by_regex so first_mat…
yetval Jul 7, 2026
37365b9
Merge branch 'dev' into fix/366-control-characters
D4Vinci Jul 11, 2026
338ab58
fix(ai): strip control characters before extraction (#368)
D4Vinci Jul 11, 2026
76f1e3f
build: bump version and deps up
D4Vinci Jul 11, 2026
be48e66
Merge branch 'dev' into perf/lazy-wrap-first-match
D4Vinci Jul 11, 2026
97ef582
perf: lazily wrap elements in find_by_text/find_by_regex so first_mat…
D4Vinci Jul 11, 2026
c19c9bf
docs: updating the benchmarks
D4Vinci Jul 11, 2026
4022dac
feat: add a template for Shopify websites
D4Vinci Jul 12, 2026
a58fd35
docs: update contribution rules
D4Vinci Jul 12, 2026
b1c250f
docs: add a new page for the Shopify template
D4Vinci Jul 12, 2026
bea8c71
feat: Add option to set executable path in the CLI
D4Vinci Jul 12, 2026
027a7cd
docs: update docs and skill with the new cli option
D4Vinci Jul 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ To ensure that your PR gets accepted, please make sure that your PR is based on
- **Make your changes, keep the code clean with an explanation of any part that might be vague, and remember to create a separate virtual environment for this project.**
- If you are adding a new feature, please add tests for it.
- If you are fixing a bug, please add code with the PR that reproduces the bug.
- Spider platform templates are welcome when the platform exposes a uniform structure across many independent domains; single-site scrapers never belong in the library.
- Please follow the rules and coding style rules we explain below.


Expand Down
20 changes: 10 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -428,14 +428,14 @@ Scrapling isn't just powerful-it's also blazing fast. The following benchmarks c

| # | Library | Time (ms) | vs Scrapling |
|---|:-----------------:|:---------:|:------------:|
| 1 | Scrapling | 2.02 | 1.0x |
| 2 | Parsel/Scrapy | 2.04 | 1.01 |
| 3 | Raw Lxml | 2.54 | 1.257 |
| 4 | PyQuery | 24.17 | ~12x |
| 5 | Selectolax | 82.63 | ~41x |
| 6 | MechanicalSoup | 1549.71 | ~767.1x |
| 7 | BS4 with Lxml | 1584.31 | ~784.3x |
| 8 | BS4 with html5lib | 3391.91 | ~1679.1x |
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |


### Element Similarity & Text Search Performance
Expand All @@ -444,8 +444,8 @@ Scrapling's adaptive element finding capabilities significantly outperform alter

| Library | Time (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
| Scrapling | 2.39 | 1.0x |
| AutoScraper | 12.45 | 5.209x |
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |


> All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.
Expand Down
Binary file modified agent-skill/Scrapling-Skill.zip
View file
Open in desktop
Binary file not shown.
7 changes: 5 additions & 2 deletions agent-skill/Scrapling-Skill/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: scrapling-official
description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
version: "0.4.10"
version: "0.4.11"
license: Complete terms in LICENSE.txt
metadata:
homepage: "https://scrapling.readthedocs.io/en/latest/index.html"
Expand Down Expand Up @@ -40,7 +40,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo

Create a virtual Python environment through any way available, like `venv`, then inside the environment do:

`pip install "scrapling[all]>=0.4.10"`
`pip install "scrapling[all]>=0.4.11"`

Then do this to download all the browsers' dependencies:

Expand Down Expand Up @@ -158,6 +158,7 @@ Both (`fetch` / `stealthy-fetch`) share options:
| -H, --extra-headers | TEXT | Extra headers in format "Key: Value" (can be used multiple times) |
| --dns-over-https / --no-dns-over-https | None | Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False) |
| --block-ads / --no-block-ads | None | Block requests to ~3,500 known ad and tracker domains (default: False) |
| --executable-path | TEXT | Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set. |
| --ai-targeted | None | Extract only main content and sanitize hidden elements for AI consumption (default: False). Also enables ad blocking automatically. |

This option is specific to `fetch` only:
Expand Down Expand Up @@ -325,6 +326,8 @@ class BlogCrawler(CrawlSpider):
```
For sitemap-driven crawls, use `SitemapSpider` with the same `rules()` API. It fetches `sitemap_urls`, descends into sitemap indexes, and dispatches each URL through your rules. Put a `robots.txt` URL directly in `sitemap_urls` and the spider extracts each `Sitemap:` directive from it automatically. See `references/spiders/generic-templates.md` for the full reference, including `LinkExtractor`'s allow/deny/restrict_css/canonicalize options.

For Shopify-powered stores, subclass `ShopifySpider` and set `target_website` to the store's domain; it extracts every product variant through Shopify's JSON API without touching the HTML. See `references/spiders/platform-templates.md`.

### Advanced Parsing & Navigation
```python
from scrapling.fetchers import Fetcher
Expand Down
2 changes: 1 addition & 1 deletion agent-skill/Scrapling-Skill/examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.
Make sure Scrapling is installed:

```bash
pip install "scrapling[all]>=0.4.10"
pip install "scrapling[all]>=0.4.11"
scrapling install --force
```

Expand Down
2 changes: 1 addition & 1 deletion agent-skill/Scrapling-Skill/references/mcp-server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,6 @@ In a Claude Desktop configuration, add the option to the server arguments:
}
```

You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Tool calls can still pass `executable_path` directly when a single request or session needs a different browser executable.
You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Tool calls can still pass `executable_path` directly when a single request or session needs a different browser executable. The `scrapling extract fetch` and `scrapling extract stealthy-fetch` CLI commands support the same `--executable-path` option and environment variable fallback.

The MCP server name when registering with a client is `ScraplingServer`. The command is the path to the `scrapling` binary and the argument is `mcp`.
54 changes: 54 additions & 0 deletions agent-skill/Scrapling-Skill/references/spiders/platform-templates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Platform Spider Templates

Generic templates cover crawl *patterns* (follow links, walk sitemaps). Platform templates cover *platforms*: site builders that expose the same machine-readable structure across many independent websites, so the spider already knows where the data lives. You only point it at a domain.

## ShopifySpider

`ShopifySpider` extracts every product from any Shopify-powered store through Shopify's JSON API, without touching the website's HTML.

```python
from scrapling.spiders import ShopifySpider

class MyStore(ShopifySpider):
target_website = "example.com"

result = MyStore().start()
print(result.items[0])
```

Set `target_website` to the store's domain. When it's empty, the spider falls back to the first entry in `start_urls`, then `allowed_domains`, and raises `ValueError` if all three are empty. Full URLs are normalized to their domain automatically.

### How it works

1. Pages through `https://<store>/collections.json` (250 collections per page, the platform's cap).
2. For every collection that reports products, pages through `/collections/<handle>/products.json`.
3. Yields one item per product variant, deduplicating variants that appear in multiple collections.

### Item fields

| Field | Source |
|---------------|---------------------------------------------------------------------------|
| `name` | Product title, plus the variant title when it isn't the default one |
| `price` | Variant price (a string, exactly as Shopify returns it) |
| `category` | Collection handle, title-cased (`summer-sale` becomes `Summer Sale`) |
| `brand` | Product vendor |
| `identifier` | Variant id |
| `sku` | Variant SKU, or `""` when the store doesn't set one |
| `stock` | `None` when the variant is available, `0` when it's out of stock |
| `image_url` | First product image, or `""` |
| `url` | The product's page inside the collection |
| `description` | Product `body_html` with the HTML tags stripped |
| `old_price` | `compare_at_price` when it's a real pre-sale price, else `""` |
| `barcode` | Variant barcode, or `""` (most stores don't expose it in these endpoints) |

These fields are not mandatory; override the `_process_product()` method in your subclass to change the item structure or extract different fields from the product data.

### Notes and limits

- The JSON endpoints only expose products published to the online-store channel, so a collection's `products_count` can be higher than what's actually retrievable. The spider treats it as "nonzero means fetch this collection", never as an expected total.
- This template most likely won't work on stores behind extra protections or password-protected ones, and if it can work at all, it will need a lot of overrides from your side.
- Everything from `Spider` still applies: concurrency settings, delays, robots.txt compliance, checkpoints, and the lifecycle hooks.

### What qualifies as a platform template

Platform templates are accepted for platforms that expose a uniform, machine-readable structure across many independent websites, like Shopify does. Spiders for one specific website don't belong in the library, no matter how popular the website is.
20 changes: 10 additions & 10 deletions docs/README_AR.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,14 @@ Scrapling ليس قوياً فحسب - بل هو أيضاً سريع بشكل م

| # | المكتبة | الوقت (ms) | vs Scrapling |
|---|:-----------------:|:----------:|:------------:|
| 1 | Scrapling | 2.02 | 1.0x |
| 2 | Parsel/Scrapy | 2.04 | 1.01 |
| 3 | Raw Lxml | 2.54 | 1.257 |
| 4 | PyQuery | 24.17 | ~12x |
| 5 | Selectolax | 82.63 | ~41x |
| 6 | MechanicalSoup | 1549.71 | ~767.1x |
| 7 | BS4 with Lxml | 1584.31 | ~784.3x |
| 8 | BS4 with html5lib | 3391.91 | ~1679.1x |
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |


### أداء تشابه العناصر والبحث النصي
Expand All @@ -440,8 +440,8 @@ Scrapling ليس قوياً فحسب - بل هو أيضاً سريع بشكل م

| المكتبة | الوقت (ms) | vs Scrapling |
|-------------|:----------:|:------------:|
| Scrapling | 2.39 | 1.0x |
| AutoScraper | 12.45 | 5.209x |
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |


> تمثل جميع المعايير متوسطات أكثر من 100 تشغيل. انظر [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) للمنهجية.
Expand Down
20 changes: 10 additions & 10 deletions docs/README_CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,14 @@ Scrapling 不仅功能强大--它还速度极快。以下基准测试将 Scrapli

| # | 库 | 时间 (ms) | vs Scrapling |
|---|:-----------------:|:---------:|:------------:|
| 1 | Scrapling | 2.02 | 1.0x |
| 2 | Parsel/Scrapy | 2.04 | 1.01 |
| 3 | Raw Lxml | 2.54 | 1.257 |
| 4 | PyQuery | 24.17 | ~12x |
| 5 | Selectolax | 82.63 | ~41x |
| 6 | MechanicalSoup | 1549.71 | ~767.1x |
| 7 | BS4 with Lxml | 1584.31 | ~784.3x |
| 8 | BS4 with html5lib | 3391.91 | ~1679.1x |
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |


### 元素相似性和文本搜索性能
Expand All @@ -440,8 +440,8 @@ Scrapling 的自适应元素查找功能明显优于替代方案：

| 库 | 时间 (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
| Scrapling | 2.39 | 1.0x |
| AutoScraper | 12.45 | 5.209x |
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |


> 所有基准测试代表 100+ 次运行的平均值。请参阅 [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) 了解方法。
Expand Down
20 changes: 10 additions & 10 deletions docs/README_DE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,14 @@ Scrapling ist nicht nur leistungsstark -- es ist auch blitzschnell. Die folgende

| # | Bibliothek | Zeit (ms) | vs Scrapling |
|---|:-----------------:|:---------:|:------------:|
| 1 | Scrapling | 2.02 | 1.0x |
| 2 | Parsel/Scrapy | 2.04 | 1.01 |
| 3 | Raw Lxml | 2.54 | 1.257 |
| 4 | PyQuery | 24.17 | ~12x |
| 5 | Selectolax | 82.63 | ~41x |
| 6 | MechanicalSoup | 1549.71 | ~767.1x |
| 7 | BS4 with Lxml | 1584.31 | ~784.3x |
| 8 | BS4 with html5lib | 3391.91 | ~1679.1x |
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |


### Element-Ähnlichkeit & Textsuche-Leistung
Expand All @@ -440,8 +440,8 @@ Scraplings adaptive Element-Finding-Fähigkeiten übertreffen Alternativen deutl

| Bibliothek | Zeit (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
| Scrapling | 2.39 | 1.0x |
| AutoScraper | 12.45 | 5.209x |
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |


> Alle Benchmarks stellen Durchschnittswerte von über 100 Durchläufen dar. Siehe [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) für die Methodik.
Expand Down
20 changes: 10 additions & 10 deletions docs/README_ES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,14 @@ Scrapling no solo es potente, también es ultrarrápido. Los siguientes benchmar

| # | Biblioteca | Tiempo (ms) | vs Scrapling |
|---|:-----------------:|:-----------:|:------------:|
| 1 | Scrapling | 2.02 | 1.0x |
| 2 | Parsel/Scrapy | 2.04 | 1.01 |
| 3 | Raw Lxml | 2.54 | 1.257 |
| 4 | PyQuery | 24.17 | ~12x |
| 5 | Selectolax | 82.63 | ~41x |
| 6 | MechanicalSoup | 1549.71 | ~767.1x |
| 7 | BS4 with Lxml | 1584.31 | ~784.3x |
| 8 | BS4 with html5lib | 3391.91 | ~1679.1x |
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |


### Rendimiento de Similitud de Elementos y Búsqueda de Texto
Expand All @@ -440,8 +440,8 @@ Las capacidades de búsqueda adaptativa de elementos de Scrapling superan signif

| Biblioteca | Tiempo (ms) | vs Scrapling |
|-------------|:-----------:|:------------:|
| Scrapling | 2.39 | 1.0x |
| AutoScraper | 12.45 | 5.209x |
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |


> Todos los benchmarks representan promedios de más de 100 ejecuciones. Ver [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) para la metodología.
Expand Down
20 changes: 10 additions & 10 deletions docs/README_FR.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,14 @@ Scrapling n'est pas seulement puissant - il est aussi ultra rapide. Les benchmar

| # | Bibliothèque | Temps (ms) | vs Scrapling |
|---|:-----------------:|:----------:|:------------:|
| 1 | Scrapling | 2.02 | 1.0x |
| 2 | Parsel/Scrapy | 2.04 | 1.01 |
| 3 | Raw Lxml | 2.54 | 1.257 |
| 4 | PyQuery | 24.17 | ~12x |
| 5 | Selectolax | 82.63 | ~41x |
| 6 | MechanicalSoup | 1549.71 | ~767.1x |
| 7 | BS4 with Lxml | 1584.31 | ~784.3x |
| 8 | BS4 with html5lib | 3391.91 | ~1679.1x |
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |


### Performance de similarité d'éléments & recherche textuelle
Expand All @@ -440,8 +440,8 @@ Les capacités adaptatives de recherche d'éléments de Scrapling surpassent sig

| Bibliothèque | Temps (ms) | vs Scrapling |
|--------------|:----------:|:------------:|
| Scrapling | 2.39 | 1.0x |
| AutoScraper | 12.45 | 5.209x |
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |


> Tous les benchmarks représentent des moyennes de plus de 100 exécutions. Voir [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) pour la méthodologie.
Expand Down
20 changes: 10 additions & 10 deletions docs/README_JP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,14 @@ Scrapling は強力であるだけでなく、超高速です。以下のベン

| # | ライブラリ | 時間 (ms) | vs Scrapling |
|---|:-----------------:|:---------:|:------------:|
| 1 | Scrapling | 2.02 | 1.0x |
| 2 | Parsel/Scrapy | 2.04 | 1.01 |
| 3 | Raw Lxml | 2.54 | 1.257 |
| 4 | PyQuery | 24.17 | ~12x |
| 5 | Selectolax | 82.63 | ~41x |
| 6 | MechanicalSoup | 1549.71 | ~767.1x |
| 7 | BS4 with Lxml | 1584.31 | ~784.3x |
| 8 | BS4 with html5lib | 3391.91 | ~1679.1x |
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |


### 要素類似性とテキスト検索のパフォーマンス
Expand All @@ -440,8 +440,8 @@ Scrapling の適応型要素検索機能は代替手段を大幅に上回りま

| ライブラリ | 時間 (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
| Scrapling | 2.39 | 1.0x |
| AutoScraper | 12.45 | 5.209x |
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |


> すべてのベンチマークは 100 回以上の実行の平均を表します。方法論については[benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py)を参照してください。
Expand Down
Loading
Loading