Track search engine in doc attrs

[ppinchuk]

Track what search engine(s) each doc was found with

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.18519% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.61%. Comparing base (477e9d6) to head (f7785e8).

Files with missing lines	Patch %	Lines
compass/scripts/download.py	50.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #544      +/-   ##
==========================================
+ Coverage   66.48%   66.61%   +0.13%     
==========================================
  Files          78       78              
  Lines        7692     7707      +15     
  Branches      774      779       +5     
==========================================
+ Hits         5114     5134      +20     
+ Misses       2408     2400       -8     
- Partials      170      173       +3     

Flag	Coverage Δ
unittests	66.61% <85.18%> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds search-engine provenance tracking to collected documents so downstream pipeline outputs can record which search engine(s) surfaced each URL/document.

Changes:

• Switched simple search to use ELM’s attribute-preserving search API and updated duplicate-collapsing logic to accumulate search_engines.
• Propagated search_engines into downloaded document attrs and into threaded service “doc info” compilation for reporting.
• Renamed collection_workflow/extraction_workflow to collection/extraction within the jurisdiction pipeline wiring, and bumped nlr-elm to 0.0.50.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
compass/web/search.py	Preserve engine metadata in simple search; track search_engines across deduped results.
compass/scripts/download.py	Copy search_engines (and rank) from selected search results into document attrs.
compass/services/threaded.py	Include search_engines in compiled document metadata emitted by the threaded services layer.
compass/pipeline/jurisdiction.py	Rename/rewire workflow components to collection and extraction.
compass/pipeline/collection/base.py	Attach from_steps provenance to docs when returning early via eager extraction.
compass/pipeline/collection/dedupe.py	Add DocumentDeDuplicator.info() accessor used to retrieve per-doc provenance.
compass/utilities/costs.py	Small docstring wording correction.
compass/services/openai.py	Docstring wording corrections to reflect timed_tracker.
tests/python/unit/web/test_web_search.py	Add/extend assertions validating search_engines propagation and merging behavior.
tests/python/unit/scripts/test_download.py	Add test ensuring _docs_from_web_search copies search_engines into doc attrs.
tests/python/unit/services/test_services_threaded.py	Assert compiled doc info includes from_steps and search_engines.
tests/python/unit/pipeline/test_pipeline_jurisdiction.py	Update test wiring for renamed collection attribute.
tests/python/unit/pipeline/test_pipeline_collection.py	Update test wiring for renamed extraction attribute.
tests/python/unit/pipeline/test_pipeline_collection_dedupe.py	Add assertions around DocumentDeDuplicator.info() behavior.
pyproject.toml	Bump nlr-elm dependency to >=0.0.50,<1.
pixi.lock	Lockfile update corresponding to the nlr-elm bump.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.