Quick Scan feature (v1.1.0 candidate)

[osintph]

Adds an ad-hoc Quick Scan feature for target investigation, independent of the existing project/keyword/IOC pipeline. Delivered as six layered, reviewable commits. Full suite green: 133 passed, 2 skipped (71/2 baseline + 62 new, zero regressions). New files pass ruff with no new lint errors.

Files changed, grouped by commit

Commit 1 — storage models and Storage methods

• src/darkweb_scanner/storage.py (+146): QuickScanSession + QuickScanFinding models and 7 Storage methods.

Commit 2 — source config module and normalization logic

• src/darkweb_scanner/quick_scan_sources.py (+180, new): typed QuickScanSource registry.
• src/darkweb_scanner/quick_scan.py (+164, new): pure detection / normalization / high-signal / context logic.

Commit 3 — orchestrator run_quick_scan

• src/darkweb_scanner/quick_scan.py (+229): async orchestrator with Tor + safe_fetch dispatch, depth/URL/timeout caps.

Commit 4 — /api/quick-scan/ routes*

• src/darkweb_scanner/dashboard/dashboard_routes.py (+169): start/status/sessions/findings + /quick-scan page.
• src/darkweb_scanner/dashboard/http_client.py (+4): allowlist entry.

Commit 5 — Quick Scan tab UI

• src/darkweb_scanner/dashboard/templates/quick_scan.html (+357, new): standalone page + vanilla-JS polling.
• src/darkweb_scanner/dashboard/templates/index.html (+1): sidebar nav link.

Commit 6 — unit and integration test suite

• tests/unit/test_quick_scan_target_detection.py (+80)
• tests/unit/test_quick_scan_high_signal.py (+76)
• tests/integration/test_quick_scan_orchestration.py (+155)
• tests/integration/test_quick_scan_routes.py (+179)

Deviations from spec, with justification

1. UI is a standalone page, not an in-SPA panel. The dashboard is a single index.html with no static/ dir. This honors the spec's literal GET /quick-scan → quick_scan.html requirement; the sidebar entry navigates there.
2. dpaste.org and rentry.co ship disabled. Neither exposes a documented public search endpoint, so per the spec's own "otherwise skip cleanly" clause both are registered with empty url_templates. The clearnet safe_fetch path is fully implemented and tested (test_clearnet_path_uses_safe_fetch); enabling a paste source is a one-line change once an endpoint exists.
3. Unauthenticated requests return 302, not 401. The platform-wide @require_login decorator redirects to the login page; shared auth behavior was not altered. Cross-user (403) and double-start (409) match the spec exactly.
4. Single-active-scan guard is DB-backed (has_active_quick_scan over pending/running) rather than an in-process global like the existing crawler. Correct across gunicorn workers — addresses SECURITY_REVIEW_FABLE finding #5 rather than replicating that flaw.
5. 10-minute total cap is enforced cooperatively at fetch granularity (deadline checked before each batch and each fetch), so worst-case overshoot is one 30s per-source timeout rather than a hard mid-fetch kill that would lose accumulated findings. Verified by test_total_timeout_marks_session_completed_with_warning (marks completed + warning).

Allowlist additions to http_client.py

Exactly one host added to ALLOWED_EXTERNAL_HOSTS:

• dpaste.org — clearnet paste source, fetched via safe_fetch (HTTPS-only, IP-blocklisted). Added with a comment noting the source ships disabled pending a search endpoint.

rentry.co was not added (disabled source, no outbound calls). Onion sources (Ahmia, Torch, Haystak, ransomware seeds) need no allowlist entry — they route through Tor.

Smoke test pending

Manual smoke test against a benign domain (e.g. example.com) on a deployed instance with a live Tor daemon is not yet run — it requires the running stack. Expected: UI updates live, findings are stored, and the session completes cleanly.

🤖 Generated with Claude Code

[osintph]

Smoke tested on the VPS against the feature branch. Verified:

• Fresh scan starts and completes (example.com, 88 URLs, 0 findings, ~4 min)
• Cancel button interrupts a running scan (bpi.com.ph, status transitioned to cancelled cleanly)
• Previous sessions table renders both completed and cancelled correctly
• Three-restart race check on the singleton lock passed with no create_tables errors

Merging as v1.1.0 candidate. Documentation and release commit to follow in a separate PR.