fix: derive chrome fingerprint version from installed datapack at runtime - #398
Open
albatrossflyon-coder wants to merge 1 commit into
Open
fix: derive chrome fingerprint version from installed datapack at runtime#398albatrossflyon-coder wants to merge 1 commit into
albatrossflyon-coder wants to merge 1 commit into
Conversation
…time The hardcoded `chromium_version = 149` / `chrome_version = 149` constants were newer than what the installed `apify-fingerprint-datapoints` release actually ships fingerprint data for (max 143 on 0.14.0). Playwright doesn't expose the real installed browser version without launching it, so instead of hardcoding a number that drifts ahead of the datapack on every release, derive the newest supported version directly from the datapack's own browser-helper-file.json at import time. Fixes D4Vinci#396. Reproduced and verified live on Windows (issue previously reported as Linux/macOS-only) - generate_headers(browser_mode="chrome") raised ValueError before this change, generates real headers after it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #396
Root cause
The module-level
chromium_version = 149/chrome_version = 149constants inscrapling/engines/toolbelt/fingerprints.pywere hardcoded above the newest Chrome version the installedapify-fingerprint-datapointsrelease actually ships fingerprint data for (max143onapify-fingerprint-datapoints==0.14.0). Since__default_useragent__is computed at module import time,browserforge'sHeaderGeneratorraisesValueError: No headers based on this input can be generatedas soon as Scrapling is imported.This is not platform-specific. The issue was originally reported on Linux and confirmed on macOS in the comments, but the same mismatch reproduces identically on Windows — I verified this live before writing the fix.
Fix
Instead of hardcoding a version number that will drift ahead of the installed datapack on every future Scrapling/browserforge release (as pointed out in the issue thread), derive the newest supported Chrome version directly from the installed datapack's own
browser-helper-file.jsonat import time via a small cached helper,_max_supported_version(). This is the same data browserforge's ownHeaderGenerator._load_unique_browsers()already reads, so it's guaranteed to stay in sync with whatever datapack version is actually installed — no more hardcoded floor to bump on every release.Testing
ValueErrorongenerate_headers(browser_mode="chrome")) before the fix, confirmed it's resolved aftertests/fetchers/test_utils.py: one asserting the version constants match the datapack's actual max, one asserting header generation in chrome mode no longer raisestests/fetchers/test_utils.pypasses (5/5, including the 2 new tests)ruff check/ruff format --check: cleanbandit: no issuesmypy/pyright: no issuesvermin -t=3.10-: confirms3.10, matchingrequires-pythonNote
Added a defensive check so
_max_supported_version()raises a clear error instead of an opaquemax() arg is an empty sequenceif it's ever called with a browser name absent from the datapack (not reachable today since it's only called with"chrome", which is always present, but worth guarding since it's a general-purpose helper).