v0.4.12

[D4Vinci]

A release focused on making your spiders smarter about the websites they crawl

Note

Follow us on X for daily tips and tricks

🚀 New Stuff and quality of life changes

• Spiders can now tune their own speed with AutoThrottle. Instead of guessing a download_delay that's either too slow or gets you banned, the spider measures how fast each website answers and adjusts the delay of every domain on its own. When a website starts blocking or rate-limiting you, it doubles the delay (or waits exactly what the Retry-After header asks for) until that stops, then speeds back up. Your download_delay and any robots.txt Crawl-delay are still respected as the minimum. (Check the docs)

  class MySpider(Spider):
      name = "adaptive"
      start_urls = ["https://example.com"]
      autothrottle_enabled = True

• Export your results to CSV and XML, next to the JSON/JSONL exporters you already had. Items that don't all share the same keys are still exported without losing anything, and nested values are written as JSON. (Check the docs)

  result = MySpider().start()
  result.items.to_csv("products.csv")
  result.items.to_xml("products.xml")

• The MCP server can now require authentication, so you can safely expose it instead of keeping it on your own machine. Any request without the token is rejected, and you can also restrict which hostnames the server answers to. (Check the docs)

  scrapling mcp --http --auth-token "$(openssl rand -hex 32)"

• Browsers now accept CDP URLs over HTTP, not just WebSocket ones. So next to the wss:// endpoints managed browser providers hand out, you can now point any browser fetcher or MCP session at a Chrome you started yourself with --remote-debugging-port=9222.

• Published Docker images are now tagged with their release version instead of only latest, so you can pin the exact version you want, by @JanRK in #384.

🐛 Bug Fixes

• Fixed cached responses losing all their cookies when the response came from a browser engine, which silently broke any session or auth logic relying on them while using the spiders' development mode, by @amitvijapur in #379. (Fixes #376)

• Fixed StealthyFetcher forcing the en-US locale on every browser instead of following your system's, which made websites see a mismatch between your locale and your IP address and treat you as suspicious, like Google answering with 429s. (Fixes #381)

• Fixed a misleading error message in the storage system and removed a dead call left after inserts, by @fix2015 in #377.

Performance

• get_all_text() is now O(nodes) instead of walking up the ancestors of every single text node, which makes it around 5-6x faster on deeply nested pages, by @yetval in #378.

🙏 Special thanks to the community for all the continuous testing and feedback

---

Big shoutout to our Platinum Sponsors