refactor(news): add source-explicit collection and live verification

.agents/skills/quantmind-dev/SKILL.md

@@ -11,7 +11,9 @@ Development workflow for contributing to the QuantMind codebase.
 
 1. Read the repository root `AGENTS.md` or `CLAUDE.md` for the stable
    architecture constraints and the module map.
-2. Pick exactly one workflow reference below; do not load the others.
+2. Read `docs/README.md` when the task adds, changes, or uses a public
+   operation or public-network source.
+3. Pick exactly one workflow reference below; do not load the others.
 
 ## Select Workflow
 
@@ -24,8 +26,11 @@ A feature task usually chains all three: develop → commit → pull request.
 
 ## Rules
 
-- `bash scripts/verify.sh` is the single "shippable" gate. CI runs the same
-  script; run it before every push and before marking a PR ready.
+- `bash scripts/verify.sh` is the deterministic offline golden gate. Run it
+  before every push and before marking a PR ready.
+- Public-network integrations have separate bounded live component gates.
+  Run every applicable gate listed in `docs/README.md` when changing that
+  component and before publishing.
 - Never bypass pre-commit / pre-push hooks (`--no-verify`) unless the user
   explicitly authorizes it.
 - New features ship with a unit test and a focused example (see

.agents/skills/quantmind-dev/references/develop-components.md

@@ -18,10 +18,12 @@ apply throughout.
    or a side effect beyond the call it wraps). No premature abstractions —
    extract shared code when the second real caller appears, not before.
 4. **Add the unit test and the example** (sections below).
-5. **Update the public surface** if needed: module `__init__.py` exports
-   and user-facing docs (`README.md`, `docs/`) for user-visible features.
-6. **Verify**: targeted `pytest tests/<module>/` while iterating;
-   `bash scripts/verify.sh` before handoff.
+5. **Update the public surface** if needed: package exports, the relevant
+   design or guide, and the catalog in `docs/README.md`. Update the root
+   README only when its overview or quick start changes.
+6. **Verify**: run targeted tests while iterating, the offline golden gate
+   before handoff, and every applicable live component gate for changed
+   public-network integrations.
 
 ## Module Routing
 
@@ -46,9 +48,9 @@ apply throughout.
   objects are `TreeKnowledge` even when a flatten card exists alongside
   (e.g. `Paper` vs `PaperKnowledgeCard`).
 
-### `quantmind/configs/` — flow cfg + typed inputs
+### `quantmind/configs/` — operation cfg + typed inputs
 
-- Extend `BaseFlowCfg`; inputs are discriminated-union Pydantic types.
+- Extend `BaseFlowCfg`; inputs are Pydantic models or discriminated unions.
 - Never `Dict[str, Any]` in signatures — model it.
 
 ### `quantmind/preprocess/` — deterministic data prep
@@ -60,10 +62,11 @@ apply throughout.
 
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
-- Flows are pure `async def` functions, not classes; state passes as
-  arguments; side effects go through explicit hooks.
-- Use the OpenAI Agents SDK directly (`Agent`, `@function_tool`,
-  `output_type=`); never wrap `from agents import ...` in a facade.
+- Public operations are `async def` functions, not classes; state passes
+  as arguments and side effects are explicit.
+- Semantic operations use the OpenAI Agents SDK directly (`Agent`,
+  `@function_tool`, `output_type=`); deterministic operations do not add an
+  LLM. Never wrap `from agents import ...` in a facade.
 - Fan-out goes through `batch_run` (bounded concurrency, error policy);
   `batch_run` rejects `memory=` at the signature layer by design.
 
@@ -79,6 +82,35 @@ apply throughout.
 - Logger only. New general-purpose helpers need maintainer sign-off via an
   issue first; the default answer is "put it in the module that uses it".
 
+## Public Operation Checklist
+
+A public operation is complete only when all of these agree:
+
+1. A stage and name consistent with `docs/design/en/operations.md`.
+2. Typed input and config models, exported from `quantmind.configs`.
+3. One intent-oriented `async def` operation exported from `quantmind.flows`,
+   with its result contract exported from the canonical owning layer.
+4. Offline success and failure tests plus a magic-introspection test when the
+   operation follows the `(input, *, cfg)` convention.
+5. One runnable common-path example under `examples/<module>/`.
+6. A relevant design or guide and one row in `docs/README.md`.
+
+Do not add a registry solely for discovery; package exports and the component
+catalog are the discovery surfaces.
+
+## Public-Network Source Checklist
+
+When adding a source to an existing operation:
+
+1. Update the typed source selection and the operation's direct dispatch.
+2. Keep acquisition policy internal. Add a shared provider abstraction only
+   after a second implementation demonstrates shared behavior.
+3. Add offline mocked tests for parsing, boundaries, continuation after item
+   failures, and completeness semantics.
+4. Update the source table and design under `docs/`.
+5. Add or update a bounded live verifier and GitHub workflow. Keep live
+   network work out of `scripts/verify.sh`.
+
 ## Tests
 
 - Location mirrors the module: `tests/<module>/test_<topic>.py`.
@@ -101,5 +133,6 @@ apply throughout.
 
 - Docstrings: English, Google style, required on public functions and
   models.
-- User-visible behavior changes update `README.md` (usage section) and
-  `docs/` where applicable.
+- Public behavior changes update the relevant design or guide and the catalog
+  in `docs/README.md`. Update the root README only for top-level positioning
+  or quick-start changes.

.agents/skills/quantmind-dev/references/pull-request.md

@@ -7,14 +7,16 @@ How to open and maintain a pull request against QuantMind.
 1. Branch from `master`; keep the PR small and focused — one PR equals one
    reviewable change (see
    [Google eng practices on small CLs](https://google.github.io/eng-practices/review/developer/small-cls.html)).
-2. Run the canonical gate and make sure it is green:
+2. Run the deterministic offline golden gate:
 
    ```bash
    bash scripts/verify.sh
    ```
 
-   CI runs the exact same script; do not open (or mark ready) a PR with a
-   red local run.
+3. If the change affects a public-network integration, run every applicable
+   live component gate listed in `docs/README.md`.
+
+Do not open or mark ready a PR with a red offline or applicable live gate.
 
 ## Title
 
@@ -37,9 +39,10 @@ and make sure the body covers:
    if one exists.
 2. **Related issue** — reference it when one exists (`Closes #NN` /
    `Part of #NN`).
-3. **Verification performed** — state what you ran (e.g.
-   `bash scripts/verify.sh` green; targeted `pytest tests/<module>/`;
-   manual example run) so the reviewer does not have to guess.
+3. **Verification performed** — state the exact offline and applicable live
+   commands you ran (e.g. `bash scripts/verify.sh`; targeted
+   `pytest tests/<module>/`; a component verifier; manual example run) so the
+   reviewer does not have to guess.
 
 Keep the template checklist and remove items that do not apply.
 

.claude/skills/quantmind-dev/SKILL.md

@@ -11,7 +11,9 @@ Development workflow for contributing to the QuantMind codebase.
 
 1. Read the repository root `AGENTS.md` or `CLAUDE.md` for the stable
    architecture constraints and the module map.
-2. Pick exactly one workflow reference below; do not load the others.
+2. Read `docs/README.md` when the task adds, changes, or uses a public
+   operation or public-network source.
+3. Pick exactly one workflow reference below; do not load the others.
 
 ## Select Workflow
 
@@ -24,8 +26,11 @@ A feature task usually chains all three: develop → commit → pull request.
 
 ## Rules
 
-- `bash scripts/verify.sh` is the single "shippable" gate. CI runs the same
-  script; run it before every push and before marking a PR ready.
+- `bash scripts/verify.sh` is the deterministic offline golden gate. Run it
+  before every push and before marking a PR ready.
+- Public-network integrations have separate bounded live component gates.
+  Run every applicable gate listed in `docs/README.md` when changing that
+  component and before publishing.
 - Never bypass pre-commit / pre-push hooks (`--no-verify`) unless the user
   explicitly authorizes it.
 - New features ship with a unit test and a focused example (see

.claude/skills/quantmind-dev/references/develop-components.md

@@ -18,10 +18,12 @@ apply throughout.
    or a side effect beyond the call it wraps). No premature abstractions —
    extract shared code when the second real caller appears, not before.
 4. **Add the unit test and the example** (sections below).
-5. **Update the public surface** if needed: module `__init__.py` exports
-   and user-facing docs (`README.md`, `docs/`) for user-visible features.
-6. **Verify**: targeted `pytest tests/<module>/` while iterating;
-   `bash scripts/verify.sh` before handoff.
+5. **Update the public surface** if needed: package exports, the relevant
+   design or guide, and the catalog in `docs/README.md`. Update the root
+   README only when its overview or quick start changes.
+6. **Verify**: run targeted tests while iterating, the offline golden gate
+   before handoff, and every applicable live component gate for changed
+   public-network integrations.
 
 ## Module Routing
 
@@ -46,9 +48,9 @@ apply throughout.
   objects are `TreeKnowledge` even when a flatten card exists alongside
   (e.g. `Paper` vs `PaperKnowledgeCard`).
 
-### `quantmind/configs/` — flow cfg + typed inputs
+### `quantmind/configs/` — operation cfg + typed inputs
 
-- Extend `BaseFlowCfg`; inputs are discriminated-union Pydantic types.
+- Extend `BaseFlowCfg`; inputs are Pydantic models or discriminated unions.
 - Never `Dict[str, Any]` in signatures — model it.
 
 ### `quantmind/preprocess/` — deterministic data prep
@@ -60,10 +62,11 @@ apply throughout.
 
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
-- Flows are pure `async def` functions, not classes; state passes as
-  arguments; side effects go through explicit hooks.
-- Use the OpenAI Agents SDK directly (`Agent`, `@function_tool`,
-  `output_type=`); never wrap `from agents import ...` in a facade.
+- Public operations are `async def` functions, not classes; state passes
+  as arguments and side effects are explicit.
+- Semantic operations use the OpenAI Agents SDK directly (`Agent`,
+  `@function_tool`, `output_type=`); deterministic operations do not add an
+  LLM. Never wrap `from agents import ...` in a facade.
 - Fan-out goes through `batch_run` (bounded concurrency, error policy);
   `batch_run` rejects `memory=` at the signature layer by design.
 
@@ -79,6 +82,35 @@ apply throughout.
 - Logger only. New general-purpose helpers need maintainer sign-off via an
   issue first; the default answer is "put it in the module that uses it".
 
+## Public Operation Checklist
+
+A public operation is complete only when all of these agree:
+
+1. A stage and name consistent with `docs/design/en/operations.md`.
+2. Typed input and config models, exported from `quantmind.configs`.
+3. One intent-oriented `async def` operation exported from `quantmind.flows`,
+   with its result contract exported from the canonical owning layer.
+4. Offline success and failure tests plus a magic-introspection test when the
+   operation follows the `(input, *, cfg)` convention.
+5. One runnable common-path example under `examples/<module>/`.
+6. A relevant design or guide and one row in `docs/README.md`.
+
+Do not add a registry solely for discovery; package exports and the component
+catalog are the discovery surfaces.
+
+## Public-Network Source Checklist
+
+When adding a source to an existing operation:
+
+1. Update the typed source selection and the operation's direct dispatch.
+2. Keep acquisition policy internal. Add a shared provider abstraction only
+   after a second implementation demonstrates shared behavior.
+3. Add offline mocked tests for parsing, boundaries, continuation after item
+   failures, and completeness semantics.
+4. Update the source table and design under `docs/`.
+5. Add or update a bounded live verifier and GitHub workflow. Keep live
+   network work out of `scripts/verify.sh`.
+
 ## Tests
 
 - Location mirrors the module: `tests/<module>/test_<topic>.py`.
@@ -101,5 +133,6 @@ apply throughout.
 
 - Docstrings: English, Google style, required on public functions and
   models.
-- User-visible behavior changes update `README.md` (usage section) and
-  `docs/` where applicable.
+- Public behavior changes update the relevant design or guide and the catalog
+  in `docs/README.md`. Update the root README only for top-level positioning
+  or quick-start changes.

.claude/skills/quantmind-dev/references/pull-request.md

@@ -7,14 +7,16 @@ How to open and maintain a pull request against QuantMind.
 1. Branch from `master`; keep the PR small and focused — one PR equals one
    reviewable change (see
    [Google eng practices on small CLs](https://google.github.io/eng-practices/review/developer/small-cls.html)).
-2. Run the canonical gate and make sure it is green:
+2. Run the deterministic offline golden gate:
 
    ```bash
    bash scripts/verify.sh
    ```
 
-   CI runs the exact same script; do not open (or mark ready) a PR with a
-   red local run.
+3. If the change affects a public-network integration, run every applicable
+   live component gate listed in `docs/README.md`.
+
+Do not open or mark ready a PR with a red offline or applicable live gate.
 
 ## Title
 
@@ -37,9 +39,10 @@ and make sure the body covers:
    if one exists.
 2. **Related issue** — reference it when one exists (`Closes #NN` /
    `Part of #NN`).
-3. **Verification performed** — state what you ran (e.g.
-   `bash scripts/verify.sh` green; targeted `pytest tests/<module>/`;
-   manual example run) so the reviewer does not have to guess.
+3. **Verification performed** — state the exact offline and applicable live
+   commands you ran (e.g. `bash scripts/verify.sh`; targeted
+   `pytest tests/<module>/`; a component verifier; manual example run) so the
+   reviewer does not have to guess.
 
 Keep the template checklist and remove items that do not apply.
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,14 +1,20 @@
-## Description
+## Summary
 
-<!-- Brief description. Refer to the related issues if existed.
-It'll be great if relevant reviewers can be assigned as well.-->
+<!-- What changed and why? Keep this concise and link the design when useful. -->
 
-## Checklist
+## Related Issue
+
+<!-- Use `Closes #NN`, `Part of #NN`, or explain why no issue applies. -->
+
+## Verification
 
-Please feel free to remove inapplicable items for your PR.
+<!-- List the exact offline and applicable live commands you ran. -->
+
+## Checklist
 
-- [ ] The PR title starts with `$CATEGORY(xx): xxx` (such as `feat(tool): xxx`, `fix(source): xxx`, `docs(README): xxx`)
-- [ ] Related issue is referred in this PR
-- [ ] The markdown and latex are rendered correctly.
-- [ ] The code in PR is well-documented.
-- [ ] The PR is complete and small, read the [Google eng practice (CL equals to PR)](https://google.github.io/eng-practices/review/developer/small-cls.html) to understand more about small PR.
+- [ ] The title uses English Conventional Commit format: `type(scope): summary`.
+- [ ] The related issue or design discussion is linked when applicable.
+- [ ] `bash scripts/verify.sh` passes.
+- [ ] Every applicable live component gate passes, or this PR states why none applies.
+- [ ] Public behavior has focused tests, an example, and documentation where applicable.
+- [ ] The PR is complete, small, and contains no unrelated changes.

.github/workflows/news-e2e.yml

@@ -0,0 +1,44 @@
+name: news-e2e
+
+on:
+  pull_request:
+    branches: [master]
+  schedule:
+    - cron: "17 3 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  news-e2e:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+          cache-dependency-glob: pyproject.toml
+
+      - name: Create virtual environment
+        run: uv venv
+
+      - name: Install project runtime dependencies
+        run: uv pip install --python .venv/bin/python -e .
+
+      - name: Run live news E2E
+        run: .venv/bin/python scripts/verify_news_e2e.py