webscout-mcp

Self-healing web access layer for AI Agents. Search, fetch, extract, and read long web content progressively — with provider routing, deterministic recovery, browser fallback, caching, and observability over the Model Context Protocol (MCP). Everything stays on your machine.

🎯 Project Positioning: webscout-mcp is a self-healing web access layer for AI agents. The core MCP server exposes 11 tools: web_search, web_fetch, web_crawl, web_extract, metadata_extract, rss_parse, content_quality, broken_links, cache_stats, cache_clear, and search_health. AI content understanding, vector search/RAG, headless browser automation, monitoring, SEO, and the optional Jev shadow evaluator are available as Python libraries / optional extras and are not required for core use. See Module Status for detailed stability and integration status.

中文版本简介 | 快速了解项目，适合中文用户阅读

🎯 What's Included in MCP (Right Now)

The MCP server currently exposes these 11 tools:

Tool	Description	Stability
web_search	Self-healing multi-provider search with deterministic recovery and automatic fallback	✅ Stable
web_fetch	Fetch a URL and return one window of extracted main content, with progressive continuation (start_char)	✅ Stable
web_crawl	Concurrent website crawling with depth limits	✅ Stable
web_extract	Structured content extraction with CSS selectors	✅ Stable
metadata_extract	Extract metadata (JSON-LD, OpenGraph, Twitter cards) from a page	🔶 Beta
rss_parse	Parse an RSS or Atom feed and return its entries	🔶 Beta
content_quality	Analyze content quality of a fetched page	🧪 Experimental
broken_links	Check for broken links on a web page	🧪 Experimental
cache_stats	View cache statistics and hit rates	✅ Stable
cache_clear	Clear the search/fetch cache	✅ Stable
search_health	Health report for all search backends	🔶 Beta

Available as Python libraries / optional extras (not required for core use): AI content understanding, vector search & RAG, headless browser automation, web monitoring & alerting, SEO analysis, OCR, PDF processing, knowledge graphs, and the optional Jev shadow evaluator (pip install "webscout-mcp[jev]", disabled by default, BYOK). See Module Status for the full list.

✨ Features

🔍 Core Web Tools

• Self-healing multi-provider search — Bing, DuckDuckGo, SearXNG, Tavily with health-based ranking, deterministic recovery, and automatic fallback
• Smart content extraction — trafilatura + readability-lxml + html2text fallback, clean article content
• Concurrent crawler — BFS crawl with depth/page limits, robots.txt compliance, retry on failures
• Structured data extraction — CSS selectors, attributes, regex extraction
• Metadata extraction — JSON-LD, OpenGraph, Twitter Cards, article metadata, images, links
• RSS/Atom support — parse feeds and feed indexes

🤖 AI Content Understanding

• Text summarization — automatic article and page summarization
• Question answering — ask questions about fetched content
• Key points extraction — extract main ideas and takeaways
• Content classification — categorize content into custom categories
• Tag generation — auto-generate relevant tags
• Sentiment analysis — analyze text sentiment
• Document comparison — compare two documents side by side
• Entity extraction — extract people, places, organizations, dates
• Multiple LLM backends — Ollama (local/free), OpenAI, Doubao, custom OpenAI-compatible

🧠 Vector Search & RAG

• Semantic search — search by meaning, not just keywords
• RAG (Retrieval-Augmented Generation) — answer questions based on your crawled content
• Local vector database — ChromaDB persistent storage
• Multiple embedding backends — local sentence-transformers (free), OpenAI, custom
• Document chunking — automatic text splitting with overlap
• Similarity threshold — configurable relevance filtering

🌐 Headless Browser Automation

• JavaScript rendering — fetch modern SPAs and dynamic content
• User interaction simulation — scroll, click, fill forms
• Screenshot capture — full-page screenshots
• PDF export — convert web pages to PDF
• Login state management — cookie persistence across sessions
• Anti-detection stealth mode — navigator.webdriver, plugins, languages spoofing
• Resource blocking — block images, media, CSS, fonts for faster loading
• Proxy support — HTTP/HTTPS proxy configuration
• Multiple browsers — Chromium, Firefox, WebKit

📡 Web Monitoring & Alerting

• Scheduled monitoring — configurable check intervals
• Content change detection — text, HTML, specific element changes
• Keyword monitoring — appearance, disappearance, count changes
• Price monitoring — track price changes with threshold alerts
• Change history — persistent history with diff generation
• Multi-channel alerts — Webhook, Email (SMTP), DingTalk, WeCom
• Configurable thresholds — minimum change size, similarity thresholds

⚡ Performance & Security

• Smart caching — SQLite cache with TTL, size limits, automatic eviction
• Rate limiting — per-domain token-bucket rate limiting
• SSRF protection — blocks localhost, sensitive ports, invalid schemes
• Browser fingerprint rotation — random User-Agents + realistic headers
• TLS fingerprint simulation — realistic TLS ClientHello fingerprints
• Connection pooling — persistent HTTP connections
• Cookie management — automatic cookie handling and persistence

🚀 Easy Setup & Deployment

• One-click setup — webscout-mcp setup auto-installs all dependencies
• System detection — auto-detects OS, CPU, memory, GPU
• Smart recommendations — suggests optimal configuration based on hardware
• Docker support — pre-built images for amd64 and arm64
• Docker Compose — one-command deployment
• systemd service — Linux service file for production
• Kubernetes — deployment manifests for container orchestration
• Configuration hot-reload — reload config without restart

🔍 Website Analysis & Optimization

• SEO analyzer — comprehensive SEO audit: meta tags, headings, images, links, URL structure, content length, Open Graph, Twitter Cards, Schema markup, with multi-dimensional scoring and actionable recommendations
• Broken link checker — detect broken links, redirect chains, invalid URLs, mixed content; classify internal/external/mailto/tel/javascript links; detailed reporting with statistics
• Performance analyzer — page performance audit: HTML size, DOM size, resource counts, render-blocking resources, inline CSS/JS, optimization techniques (lazy loading, preconnect, preload), compression/cache detection, performance scoring
• Content quality assessor — readability scores (Flesch-Kincaid, Gunning Fog, SMOG), keyword density, content structure analysis, duplicate content detection, quality scoring

📊 Export & Integration

• Multiple export formats — JSON, CSV, Excel, Parquet, SQLite, Markdown, HTML
• Field selection & ordering — export only specified fields with custom column order
• Append mode — incremental exports for CSV and SQLite
• MCP server — native Model Context Protocol support
• CLI interface — command-line tools for search, fetch, crawl
• Python API — full programmatic access to all features
• Sitemap support — parse sitemap.xml and sitemap indexes
• Incremental crawling — only re-fetch changed pages via ETag/Last-Modified

⚠️ Search Backend Stability Notice

webscout-mcp uses direct HTML scraping for search backends (Bing, DuckDuckGo, Google, Brave) by default — no API keys required. This makes it free to use, but please be aware of the stability trade-offs:

What can go wrong

• DOM changes: Search engines frequently update their HTML structure, which can break scrapers
• CAPTCHAs: Automated requests may trigger CAPTCHAs (especially Google and Brave)
• Bot detection: Advanced bot detection may block or rate-limit requests
• IP blocking: Sustained automated requests can lead to IP bans
• Parameter changes: Search engines may change request parameters or headers

Mitigations built in

• ✅ Multi-backend failover: If one backend fails, automatically try the next one
• ✅ Realistic browser headers: Random User-Agents and realistic request headers
• ✅ Rate limiting: Per-domain rate limiting to avoid overwhelming search engines
• ✅ Caching: SQLite cache reduces repeated requests to the same queries
• ✅ Retry with backoff: Exponential backoff on transient failures

For production use

For production workloads requiring higher reliability, consider:

1. Using official search APIs (Bing Search API, SerpAPI, etc.) — planned for future releases
2. Deploying with rotating proxies
3. Increasing cache TTL to reduce request frequency
4. Monitoring search backend health and adjusting backends accordingly

Bottom line: webscout-mcp's default search is great for development, personal use, and low-volume workloads. For high-volume production use, plan for additional reliability measures.

📦 Installation

Quick Install

pip install webscout-mcp

Requires Python 3.10+.

One-Click Full Setup (Recommended)

# Install core package
pip install webscout-mcp

# Run setup to install all optional dependencies
webscout-mcp setup --playwright --ollama --vector-store

The setup command will:

• Detect your system configuration (OS, CPU, memory, GPU)
• Install Playwright and Chromium browser
• Install Ollama and download a local LLM (optional)
• Install ChromaDB and sentence-transformers for vector search (optional)
• Generate a configuration file
• Run a health check to verify everything works

Optional Dependencies

# Browser automation (Playwright)
pip install webscout-mcp[browser]
playwright install chromium

# Vector search & RAG (ChromaDB + sentence-transformers)
pip install webscout-mcp[vector]

# AI content understanding (OpenAI client)
pip install webscout-mcp[ai]

# All features
pip install webscout-mcp[all]

Docker

docker pull wxslang/webscout-mcp:latest
docker run -p 8000:8000 wxslang/webscout-mcp:latest

🚀 Quick Start

MCP Client Configuration

Add to your MCP client config:

Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows):

{
  "mcpServers": {
    "webscout": {
      "command": "webscout-mcp",
      "args": []
    }
  }
}

Cursor (Settings → MCP → Add new MCP server):

{
  "mcpServers": {
    "webscout": {
      "command": "webscout-mcp",
      "args": []
    }
  }
}

CLI Usage

# Search the web
webscout-mcp search "python web scraping" --max-results 10

# Fetch a page
webscout-mcp fetch https://example.com --extract --format markdown

# Crawl a site
webscout-mcp crawl https://example.com --depth 2 --pages 50

# Run setup
webscout-mcp setup --playwright

# Start MCP server
webscout-mcp serve

Python API

from webscout_mcp import WebScout

# Initialize
scout = WebScout()

# Search
results = scout.search("AI agents", max_results=5)
for result in results:
    print(result.title, result.url)

# Fetch and extract
page = scout.fetch("https://example.com/article")
print(page.title)
print(page.content)  # Clean article text

# Crawl
pages = scout.crawl("https://example.com", depth=2, max_pages=20)

🤖 AI Content Understanding

from webscout_mcp.ai_processor import AIProcessor, AIConfig

# Use local Ollama (free, no API key needed)
config = AIConfig(backend="ollama", model="qwen2.5:7b")
ai = AIProcessor(config=config)

# Summarize
summary = ai.summarize(page.content, max_length=500)
print(summary.content)

# Ask questions
answer = ai.answer_question(page.content, "What are the main points?")
print(answer.content)

# Extract key points
points = ai.extract_key_points(page.content, num_points=5)
print(points.content)

# Analyze sentiment
sentiment = ai.analyze_sentiment(page.content)
print(sentiment.content)

Using OpenAI API:

config = AIConfig(
    backend="openai",
    model="gpt-4o",
    api_key="your-api-key",
)

Using Doubao (豆包):

config = AIConfig(
    backend="doubao",
    model="ep-20240101",
    api_key="your-api-key",
)

🧠 Vector Search & RAG

from webscout_mcp.vector_store import VectorStore, RAGEngine, Document

# Initialize vector store (local, free)
store = VectorStore()

# Add documents
doc = Document.from_text(page.content, source=page.url)
store.add_document(doc)

# Semantic search
results = store.search("how to build AI agents", n_results=5)
for result in results:
    print(f"[{result.score:.2f}] {result.document.content[:100]}")

# RAG: Ask questions based on your documents
rag = RAGEngine(vector_store=store)
answer = rag.query("What is the best approach for web scraping?")
print(answer["answer"])
print("Sources:", answer["sources"])

🌐 Headless Browser Automation

from webscout_mcp.browser_fetcher import BrowserFetcher, BrowserConfig

# Initialize
config = BrowserConfig(headless=True, block_media=True)
browser = BrowserFetcher(config=config)

# Fetch JS-rendered page
result = browser.fetch(
    "https://example.com/spa",
    wait_for_selector=".content",
    scroll_to_bottom=True,
)
print(result.title)
print(result.content)

# Take screenshot
browser.take_screenshot("https://example.com", "screenshot.png", full_page=True)

# Export to PDF
browser.export_pdf("https://example.com", "page.pdf")

# Click element
result = browser.click_element("https://example.com", "button.load-more")

# Fill form
result = browser.fill_form(
    "https://example.com/login",
    {"#username": "user", "#password": "pass"},
    submit_selector="button[type=submit]",
)

browser.close()

📡 Web Monitoring & Alerting

from webscout_mcp.monitor import WebMonitor, MonitorConfig, WebhookAlert, EmailAlert

# Initialize
config = MonitorConfig(check_interval=300, min_change_size=10)
monitor = WebMonitor(config=config)

# Add alert channels
monitor.add_alert_channel(WebhookAlert("https://hooks.example.com/alert"))
monitor.add_alert_channel(EmailAlert(
    smtp_server="smtp.gmail.com",
    smtp_port=587,
    username="you@gmail.com",
    password="app-password",
    from_addr="you@gmail.com",
    to_addrs=["recipient@example.com"],
))

# Check for changes
changes = monitor.check_url("https://example.com/pricing")
for change in changes:
    print(f"{change.change_type}: {change.old_value} -> {change.new_value}")

# Get history
history = monitor.get_history("https://example.com/pricing")

🔍 Website Analysis & Optimization

SEO Analysis

from webscout_mcp.seo_analyzer import SEOAnalyzer

# Initialize
analyzer = SEOAnalyzer()

# Analyze a page
html = "<html>...</html>"
metrics = analyzer.analyze(html, url="https://example.com")

# Check scores
print(f"Overall SEO Score: {metrics.overall_score}/100")
print(f"Meta Score: {metrics.meta_score}")
print(f"Heading Score: {metrics.heading_score}")
print(f"Image Score: {metrics.image_score}")

# Check issues and recommendations
print("Issues:", metrics.issues)
print("Recommendations:", metrics.recommendations)

Broken Link Checking

from webscout_mcp.broken_link_checker import BrokenLinkChecker

# Initialize
checker = BrokenLinkChecker(timeout=10.0, max_redirects=5)

# Check all links on a page
html = "<html>...</html>"
report = checker.check_page(html, base_url="https://example.com")

# Check statistics
print(f"Total links: {report.total_links}")
print(f"OK: {report.ok_links}")
print(f"Broken: {report.broken_links}")
print(f"Redirects: {report.redirect_links}")
print(f"Broken percentage: {report.broken_link_percentage}%")

# Get only broken links
broken = checker.get_broken_links(report)
for link in broken:
    print(f"[{link.status}] {link.url} - {link.error_message}")

# Generate human-readable summary
print(checker.generate_summary(report))

Performance Analysis

from webscout_mcp.performance_analyzer import PerformanceAnalyzer

# Initialize
analyzer = PerformanceAnalyzer()

# Analyze page performance
html = "<html>...</html>"
headers = {"Content-Encoding": "gzip", "Cache-Control": "max-age=3600"}
metrics = analyzer.analyze(html, url="https://example.com", response_headers=headers)

# Check scores
print(f"Overall Performance Score: {metrics.overall_score}/100")
print(f"HTML Size: {metrics.html_size_kb}KB (score: {metrics.html_size_score})")
print(f"DOM Nodes: {metrics.dom_node_count} (score: {metrics.dom_size_score})")
print(f"Requests: {metrics.request_count} (score: {metrics.request_count_score})")

# Check optimization techniques
print(f"Has gzip: {metrics.has_gzip}")
print(f"Has brotli: {metrics.has_brotli}")
print(f"Has lazy loading: {metrics.has_lazy_loading}")
print(f"Has preconnect: {metrics.has_preconnect}")

# Check issues and recommendations
print("Issues:", metrics.issues)
print("Warnings:", metrics.warnings)
print("Recommendations:", metrics.recommendations)

Enhanced Data Export

from webscout_mcp.data_exporter import DataExporter, ExportConfig

# Sample data
data = [
    {"title": "Result 1", "url": "https://example.com/1", "score": 0.95},
    {"title": "Result 2", "url": "https://example.com/2", "score": 0.85},
]

# Export to JSON
config = ExportConfig(format="json", output_path="results.json", pretty_json=True)
exporter = DataExporter(config=config)
result = exporter.export(data)
print(f"Exported {result.record_count} records to {result.output_path}")

# Export to CSV with field selection
config = ExportConfig(
    format="csv",
    output_path="results.csv",
    fields=["title", "url"],  # Only export these fields
    csv_delimiter=",",
)
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to Excel
config = ExportConfig(format="excel", output_path="results.xlsx", excel_sheet_name="Results")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to SQLite
config = ExportConfig(format="sqlite", output_path="results.db", sqlite_table_name="search_results")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to Parquet (columnar storage)
config = ExportConfig(format="parquet", output_path="results.parquet")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to Markdown
config = ExportConfig(format="markdown", output_path="results.md")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to HTML
config = ExportConfig(format="html", output_path="results.html")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Using convenience function
from webscout_mcp.data_exporter import export_data
result = export_data(data, "results.json", export_format="json", fields=["title", "url"])

⚙️ Configuration

Environment Variables

# Core
WEBSCOUT_CACHE_ENABLED=true
WEBSCOUT_CACHE_TTL=3600
WEBSCOUT_RATE_LIMIT_ENABLED=true

# Search
WEBSCOUT_SEARCH_DEFAULT_BACKEND=bing
WEBSCOUT_SEARCH_MAX_RESULTS=10

# AI
WEBSCOUT_AI_BACKEND=ollama
WEBSCOUT_AI_MODEL=qwen2.5:7b
WEBSCOUT_AI_API_KEY=your-key

# Vector Store
WEBSCOUT_VECTOR_DB=chroma
WEBSCOUT_EMBEDDING_BACKEND=local
WEBSCOUT_EMBEDDING_MODEL=BAAI/bge-small-zh-v1.5

# Browser
WEBSCOUT_BROWSER_TYPE=chromium
WEBSCOUT_BROWSER_HEADLESS=true
WEBSCOUT_BROWSER_BLOCK_MEDIA=true

# Monitor
WEBSCOUT_MONITOR_INTERVAL=300
WEBSCOUT_MONITOR_MIN_CHANGE=10

# Setup
WEBSCOUT_SETUP_PLAYWRIGHT=true
WEBSCOUT_SETUP_OLLAMA=false
WEBSCOUT_SETUP_CHROMADB=false

Config File

Create ~/.config/webscout/config.toml:

[server]
host = "127.0.0.1"
port = 8000

[cache]
enabled = true
ttl = 3600

[search]
default_backend = "bing"
max_results = 10

[ai]
backend = "ollama"
model = "qwen2.5:7b"

[vector_store]
enabled = true
vector_db = "chroma"
embedding_backend = "local"

[browser]
enabled = true
headless = true

[monitor]
check_interval = 300

🔧 Troubleshooting: pages that need a browser

Some sites answer a plain HTTP request with a page that says "checking your browser" instead of the content you asked for. web_fetch detects this, and if a browser is available it retries with one.

What it looks like when there is no browser

The payload does not look like a failure:

{
  "status_code": 200,
  "error": null,
  "error_code": null,
  "extracted": true,
  "extraction_status": "ok",
  "content": "火山引擎\n\n正在进行安全检测...\n\n为保障您的访问安全..."
}

Status 200, no error, extraction "ok" - because a readable page was returned. It is just not the page you wanted. The signals that identify it are:

Signal	Value for a challenge page
recovery_decision.reason	CHALLENGE, JS_REQUIRED, LOW_CONTENT_DENSITY, or SOFT_BLOCK
recovery_outcome	browser_unavailable:fallback_unavailable
content length	very short (tens of characters)
status_code	often 200, not 403

If you only check status_code and error, you will treat a challenge page as success. Check recovery_outcome when content looks thin.

Enabling browser escalation

pip install 'webscout-mcp[browser]'
playwright install chromium

That is all. With the extra installed, web_fetch escalates automatically when a page needs rendering - no sidecar, no Docker, no configuration.

Verify it took effect:

python -c "from webscout_mcp.local_browser_backend import LocalBrowserBackend; print(LocalBrowserBackend().is_available)"
# True

When a browser is not enough

Measured against sixteen real sites, browser escalation fixes client-rendered pages but not all anti-bot walls. Sites that reject by IP reputation rather than by rendering keep returning 403/202 even through a real browser:

Site	Response with browser	What would actually help
medium.com	200, but navigation only	client-side rendering
stackoverflow.com	403	residential IP
zhihu.com	403	residential IP, login session
amazon.com	202	JS + cookie handshake
reuters.com	200, full article	already works (TLS impersonation)

A proxy can be configured for the browser:

from webscout_mcp.browser_fetcher import BrowserConfig, BrowserFetcher

fetcher = BrowserFetcher(BrowserConfig(proxy_server="http://host:port"))

If a fetch returns unreadable text

content_quality grades each response and reports garbled when the bytes are not readable prose - typically a decoding fault rather than a blocked page. The most common historical cause was a brotli response decoded without the brotli package (fixed in v1.13.0; upgrade if you see it).

To grade a payload yourself:

from webscout_mcp.content_grading import assess

print(assess(content).to_dict())
# {'grade': 'usable' | 'thin' | 'empty' | 'garbled', 'chars': ..., 'reason': ...}

Diagnosing why an escalation did not happen

web_fetch returns a route_trace describing every decision:

{"stage": "recovery_classification", "reason": "CHALLENGE", "action": "BROWSER"}
{"capability": "browser", "provider": null, "result": "unavailable"}
{"stage": "recovery_execution", "outcome": "browser_unavailable:fallback_unavailable"}

provider: null with result: unavailable means no browser backend was registered - install the extra as above. A populated provider with result: failure means the browser ran and the site rejected it, which points at the IP-reputation cases in the table.

📚 Documentation

• README — This file
• Module Status — Module stability levels and MCP integration status
• Project Introduction — Detailed project overview and architecture
• Deployment Guide — Docker, systemd, Kubernetes deployment
• Examples — Usage examples and sample code
• CHANGELOG — Version history
• CONTRIBUTING — Contributing guidelines
• CODE OF CONDUCT — Community code of conduct
• SECURITY — Security policy and vulnerability reporting

🧪 Testing

# Install dev dependencies
pip install webscout-mcp[dev]

# Run all tests
pytest tests/

# Run with coverage
pytest tests/ --cov=webscout_mcp --cov-report=html

Test coverage: 395+ tests covering all modules.

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

MIT License — see LICENSE for details.

🙏 Acknowledgments

• trafilatura — content extraction
• readability-lxml — readability fallback
• Playwright — browser automation
• ChromaDB — vector database
• sentence-transformers — text embeddings
• Ollama — local LLM runtime
• httpx — HTTP client
• BeautifulSoup — HTML parsing

---

Built with ❤️ for the AI agent community.