fix(fingerprints): fallback gracefully when browserforge raises ValueError on unsupported Chrome versions - #397
Conversation
There was a problem hiding this comment.
Pull request overview
This PR prevents scrapling from crashing at import time when browserforge can’t generate headers for a hardcoded (too-new) Chrome/Chromium version by adding a graceful fallback path in the header generation helper.
Changes:
- Wrap
HeaderGenerator(...).generate()intry/except ValueErrorinsidegenerate_headers(). - On
ValueError, retry header generation with unconstrained browser definitions (no explicit version constraints). - Preserve existing behavior differences between “browser mode” vs non-browser mode while ensuring import-time header generation doesn’t fail.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Nice idea, buddy, but it's not the optimal solution. I will make scrapling access Apify's neural network right away to skip the generation part, so it's faster, and also I want it now to be not limited to desktop devices only. This whole issue was caused by an issue with |
Description
When
chromium_versionorchrome_versionhardcoded inscrapling/engines/toolbelt/fingerprints.pyexceeds the maximum version supported inbrowserforge's header dataset (apify-fingerprint-datapoints),HeaderGenerator.generate()raises aValueError:Because
generate_headers(browser_mode=False)is executed at module import time inscrapling/engines/_browsers/_config_tools.py, thisValueErrorcrashes any application importingscraplingat boot time.Fix
HeaderGenerator(...).generate()in atry...except ValueErrorblock ingenerate_headers().min_version=ver, max_version=verconstraint is not present inbrowserforge's dataset, it falls back to unconstrained browser definitions.generate_headers()always returns valid headers without crashing.