feat: preprocess fetch+format module (PR4 of OpenAI Agents SDK migration)

[keli-wen]

Summary

PR4 of the OpenAI Agents SDK migration. Lands quantmind/preprocess/ as a leaf module that turns raw inputs (arxiv id, URL, DOI, local file path) into LLM-ready text/markdown plus typed metadata, with no LLM calls of its own. Removes the now-superseded quantmind/parsers/ + quantmind/sources/ modules and quantmind/utils/tmp.py.

Part of #71.

What landed

quantmind/preprocess/

File	Surface
fetch/_types.py	Fetched, RawPaper frozen slotted dataclasses
fetch/arxiv.py	async fetch_arxiv(id_or_url) -> RawPaper (arxiv lib via to_thread + httpx for PDF)
fetch/http.py	async fetch_url(url, *, timeout, max_bytes) with body cap + UA + selected headers
fetch/doi.py	async resolve_doi(doi) -> CrossrefMetadata (Crossref API, no key)
fetch/local.py	async read_local_file(path) -> Fetched with content-type by suffix
format/pdf.py	async pdf_to_markdown(pdf_bytes) -> str via PyMuPDF
format/html.py	async html_to_markdown(html, *, strip_boilerplate) -> str via trafilatura
clean.py	normalize_unicode / collapse_whitespace / dedupe_lines (sync, pure)
time.py	to_utc / parse_filing_date / business_days_between (sync, pure)

Errors: stdlib + httpx + 2 narrow ValueError subclasses (ArxivIdParseError, PdfParseError). No PreprocessError hierarchy.

Architectural compliance

• import-linter: 4th contract added — quantmind.preprocess is a leaf, may only depend on quantmind.utils. Existing 3 contracts updated to swap deleted parsers/sources entries for preprocess.
• basedpyright: drops parsers/ and sources/ from exclude (deleted); preprocess/ auto-included at standard mode.
• coverage: floor 60 → 65; transitional modules slated for PR5 deletion (flow/, llm/, four files inside config/ and models/analysis.py) get added to coverage.run.omit to mirror the basedpyright exclude philosophy. Floor ratchets to 75 in PR5 once those are gone.
• deps: add trafilatura>=1.10 (HTML extraction); add dev respx>=0.21 + pytest-asyncio>=0.23. Drop llama-cloud-services and marker (only callers were the deleted parsers/ files); drop top-level requests (only callers were parsers/sources).

Deletions

• quantmind/parsers/ — replaced by preprocess/format/pdf.py
• quantmind/sources/ — replaced by preprocess/fetch/{arxiv,http,doi,local}.py
• quantmind/utils/tmp.py — QuantMindTemplate had no surviving callers
• tests/parsers/, tests/sources/, tests/flow/, tests/llm/ — imported deleted modules

quantmind/flow/, quantmind/llm/, quantmind/config/, and quantmind/models/{content,paper,analysis}.py stay in the tree until PR5 (their migration target is the new flows/ surface).

Out of scope (follow-up issues to open on merge)

1. Add format/markdown.py helpers (split_by_heading, strip_frontmatter, normalize_links) — defer until PR5 paper_flow has a real consumer
2. Add preprocess/chunk.py with tiktoken — ship with PR7 storage layer
3. Add marker-pdf engine option to format/pdf.py — high-quality alternative
4. Add llama-parse engine option to format/pdf.py — paid API
5. Add unpaywall fallback in fetch/doi.py — OA PDF discovery
6. Holiday-aware business_days_between in time.py

Test plan

• bash scripts/verify.sh exits 0 (5/5 green)
• 252 tests pass; total branch coverage 86.01% (preprocess/ at >95% line)
• import-linter: 4/4 contracts kept
• grep -r "from quantmind.parsers\|from quantmind.sources" quantmind/ tests/ returns 0 results
• No test touches the network — respx mocks every httpx call; arxiv lib metadata fetch is patched at the module helper level
• CLAUDE.md state table + roadmap updated; README runbook example replaced with a working preprocess.fetch_arxiv snippet

Commit-by-commit

1. feat(preprocess): add fetch + format two-layer module — new files + tests
2. refactor: remove parsers/, sources/, utils/tmp.py and stale tests
3. chore(verify): update toolchain config for preprocess landing — pyproject deps, basedpyright exclude, 4th import-linter contract, coverage floor + omit list
4. docs: mark preprocess landed; refresh runbook example for PR4 — CLAUDE.md + README.md