Skip to content

fix: defer browser headers until session startup - #395

Open
SulimanAbdulrazzaq wants to merge 1 commit into
D4Vinci:devfrom
SulimanAbdulrazzaq:agent/fix-mcp-header-import
Open

fix: defer browser headers until session startup#395
SulimanAbdulrazzaq wants to merge 1 commit into
D4Vinci:devfrom
SulimanAbdulrazzaq:agent/fix-mcp-header-import

Conversation

@SulimanAbdulrazzaq

Copy link
Copy Markdown

Proposed change

Scrapling currently generates browser headers while importing _config_tools. If browserforge cannot produce headers for the detected browser version, that exception prevents unrelated entry points such as scrapling mcp from starting.

This change defers default user-agent generation until a headless browser session actually needs it and caches one value per browser mode. Explicit user agents and headed sessions keep their existing behavior. Regression tests verify that importing the configuration module does not call browserforge and that on-demand generation preserves the Chromium/Chrome modes and caching behavior.

Validation:

  • pytest -o addopts="" tests/fetchers/test_browser_config.py -q (2 passed)
  • Ruff check and format check for the three changed files
  • mypy and pyright for the two changed source files
  • Bandit and Vermin repository checks for the changed files
  • real scrapling mcp stdio initialization returned a valid JSON-RPC result
  • git diff --check

The exact browserforge exception was reproduced locally with Scrapling 0.4.12, browserforge 1.2.4, and MCP 1.29.0. A macOS runner was not available, so the platform-independent failure path is additionally covered with a deterministic mock.

Type of change:

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests
  • Add or change doctests? -- Note: Please avoid changing both code and tests in a single pull request.
  • Documentation change?

Additional information

Checklist:

  • I have read CONTRIBUTING.md.
  • This pull request is all my own work -- I have not plagiarized.
  • I know that pull requests will not be merged if they fail the automated tests.
  • All new Python files are placed inside an existing directory.
  • All filenames are in all lowercase characters with no spaces or dashes.
  • All functions and variable names follow Python naming conventions.
  • All function parameters and return values are annotated with Python type hints.
  • All functions have doc-strings.

Generate default browser user agents only when a headless session needs one, so MCP startup does not depend on browserforge header data. Add regression coverage for import safety and per-mode caching.
@SulimanAbdulrazzaq

Copy link
Copy Markdown
Author

Hi @D4Vinci, when you have a chance, could you please review this PR? Thank you.

@yetval yetval left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thoughtful fix! LGTM I confirmed it resolves the startup issue in #394. One small note: the underlying browser issue can still appear when fetching, and #397 looks like a good complementary fix. This change is still a worthwhile improvement on its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants