diff --git a/.agents/skills/quantmind-dev/SKILL.md b/.agents/skills/quantmind-dev/SKILL.md new file mode 100644 index 0000000..a05338c --- /dev/null +++ b/.agents/skills/quantmind-dev/SKILL.md @@ -0,0 +1,42 @@ +--- +name: quantmind-dev +description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code. +--- + +# QuantMind Dev + +Development workflow for contributing to the QuantMind codebase. + +## Start Here + +1. Read the repository root `AGENTS.md` or `CLAUDE.md` for the stable + architecture constraints and the module map. +2. Pick exactly one workflow reference below; do not load the others. + +## Select Workflow + +- Committing staged work → `references/commit.md` +- Opening or updating a pull request → `references/pull-request.md` +- Implementing or refactoring anything under `quantmind/` → + `references/develop-components.md` (read it **before** writing code) + +A feature task usually chains all three: develop → commit → pull request. + +## Rules + +- `bash scripts/verify.sh` is the single "shippable" gate. CI runs the same + script; run it before every push and before marking a PR ready. +- Never bypass pre-commit / pre-push hooks (`--no-verify`) unless the user + explicitly authorizes it. +- New features ship with a unit test and a focused example (see + `references/develop-components.md`). +- This skill is mirrored under `.agents/skills/quantmind-dev/` and + `.claude/skills/quantmind-dev/`; the copies are identical. When changing + the skill, update both in the same change. + +## Boundaries + +- This skill is for contributing to QuantMind itself. It does not cover + using QuantMind as a library in your own project. +- Product decisions (new modules, new dependencies, API redesigns) need an + issue and maintainer discussion first; do not encode them here. diff --git a/.agents/skills/quantmind-dev/references/commit.md b/.agents/skills/quantmind-dev/references/commit.md new file mode 100644 index 0000000..32737e9 --- /dev/null +++ b/.agents/skills/quantmind-dev/references/commit.md @@ -0,0 +1,57 @@ +# Commit Workflow + +How to commit work in the QuantMind repository. + +## Message Format + +English, [Conventional Commits](https://www.conventionalcommits.org/): + +```text +(): +``` + +- **type**: `feat` | `fix` | `refactor` | `docs` | `test` | `chore` +- **scope**: optional but preferred — use the module or area the change + touches, matching existing history (`feat(memory): ...`, + `chore(verify): ...`, `docs(news): ...`). Omit the scope only for + repo-wide changes (`feat: flows + magic apex layer`). +- **summary**: imperative, lower-case start, no trailing period. + +Examples from history: + +```text +feat: preprocess fetch+format module (PR4 of OpenAI Agents SDK migration) (#75) +chore(verify): add scripts/verify.sh + ruff/basedpyright/import-linter/pytest --cov harness (#73) +fix(test): Prevent creation of `test_data` directory during unit tests (#41) +``` + +## One Logical Change per Commit + +Keep each commit focused on a single logical change. Split unrelated edits +(e.g. a bug fix discovered while building a feature) into separate commits. +Do not mix mechanical reformatting with behavior changes. + +## Before Committing + +1. Inspect exactly what you are about to commit: + + ```bash + git status + git diff --staged + ``` + + Confirm no unintended files (scratch scripts, local configs, large + artifacts) are staged. + +2. Run verification appropriate to the change: + - during development: targeted tests, e.g. `pytest tests//` + - before push / handoff: `bash scripts/verify.sh` (canonical gate) + +3. Commit. Pre-commit hooks run ruff format/check and file hygiene checks; + the pre-push hook runs the full `scripts/verify.sh`. + +## Hooks + +If a hook fails, fix the underlying issue and commit again. Never use +`git commit --no-verify` or otherwise bypass hooks unless the user has +explicitly authorized it for that specific commit. diff --git a/.agents/skills/quantmind-dev/references/develop-components.md b/.agents/skills/quantmind-dev/references/develop-components.md new file mode 100644 index 0000000..b04a647 --- /dev/null +++ b/.agents/skills/quantmind-dev/references/develop-components.md @@ -0,0 +1,105 @@ +# Component Development Workflow + +How to implement or refactor code under `quantmind/`. Read this before +writing code; the architecture constraints in root `AGENTS.md` / `CLAUDE.md` +apply throughout. + +## General Loop + +1. **Find the nearest existing pattern.** Locate the closest analogous + implementation in the target module and follow its structure, naming, + and test layout. Consistency beats novelty. +2. **Check the dependency contract** for your module (table below) before + adding any import. `lint-imports` enforces these; if your design needs a + forbidden import, the design is wrong — restructure, do not work around + the contract. +3. **Implement small and flat.** Pure functions over classes; `Protocol` + over ABC; no meaningless wrappers (a method must add logic, abstraction, + or a side effect beyond the call it wraps). No premature abstractions — + extract shared code when the second real caller appears, not before. +4. **Add the unit test and the example** (sections below). +5. **Update the public surface** if needed: module `__init__.py` exports + and user-facing docs (`README.md`, `docs/`) for user-visible features. +6. **Verify**: targeted `pytest tests//` while iterating; + `bash scripts/verify.sh` before handoff. + +## Module Routing + +### Dependency contracts (enforced by import-linter) + +| Module | May import from `quantmind.*` | +|--------|-------------------------------| +| `quantmind/utils/` | nothing (leaf) | +| `quantmind/knowledge/` | nothing (leaf) | +| `quantmind/configs/` | `knowledge` only | +| `quantmind/preprocess/` | `utils` only | +| `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above | + +### `quantmind/knowledge/` — data standard + +- Pydantic models, `frozen=True`, `extra="forbid"`. +- Every `BaseKnowledge` subclass **must** require `as_of: datetime` + (financial time-sensitivity is mandatory) and a typed `source: SourceRef` + (no bare strings), and **must** override `embedding_text()`. +- Pick one shape: `FlattenKnowledge` (atomic card), `TreeKnowledge` + (hierarchical artifact), or `GraphKnowledge` (placeholder). Whole-document + objects are `TreeKnowledge` even when a flatten card exists alongside + (e.g. `Paper` vs `PaperKnowledgeCard`). + +### `quantmind/configs/` — flow cfg + typed inputs + +- Extend `BaseFlowCfg`; inputs are discriminated-union Pydantic types. +- Never `Dict[str, Any]` in signatures — model it. + +### `quantmind/preprocess/` — deterministic data prep + +- Fetch / format / clean / time utilities: deterministic, no LLM calls. +- Return frozen dataclasses (`Fetched`, `RawPaper`, ...), not dicts. +- Surface the common path at the package root (`from quantmind.preprocess + import fetch_arxiv`), keep explicit submodule paths working. + +### `quantmind/flows/` and `quantmind/magic.py` — apex layer + +- Flows are pure `async def` functions, not classes; state passes as + arguments; side effects go through explicit hooks. +- Use the OpenAI Agents SDK directly (`Agent`, `@function_tool`, + `output_type=`); never wrap `from agents import ...` in a facade. +- Fan-out goes through `batch_run` (bounded concurrency, error policy); + `batch_run` rejects `memory=` at the signature layer by design. + +### `quantmind/mind/` — cognitive layer + +- Lands via the Agents SDK migration (tracking: #71). Backends implement + the `Memory` Protocol with granular `tools()`, `mcp_servers()`, + `run_hooks()`, `reset()` — each may return an empty list; do not force + MCP on every implementation. + +### `quantmind/utils/` + +- Logger only. New general-purpose helpers need maintainer sign-off via an + issue first; the default answer is "put it in the module that uses it". + +## Tests + +- Location mirrors the module: `tests//test_.py`. +- Subclass `unittest.TestCase` (run via pytest). +- Mock external services (network, LLM APIs, filesystem where practical); + tests must pass offline. +- Cover the success path **and** at least one failure path per public + function. +- Coverage floor is enforced by `pytest --cov` in `scripts/verify.sh`; new + code should not lower branch coverage. + +## Examples + +- One focused example per new feature under `examples//` + (create the directory if it does not exist yet). +- Demo the simple, common usage — one scenario per file, runnable as + `python examples//.py`, minimal setup. + +## Documentation + +- Docstrings: English, Google style, required on public functions and + models. +- User-visible behavior changes update `README.md` (usage section) and + `docs/` where applicable. diff --git a/.agents/skills/quantmind-dev/references/pull-request.md b/.agents/skills/quantmind-dev/references/pull-request.md new file mode 100644 index 0000000..8f64552 --- /dev/null +++ b/.agents/skills/quantmind-dev/references/pull-request.md @@ -0,0 +1,51 @@ +# Pull Request Workflow + +How to open and maintain a pull request against QuantMind. + +## Before Opening + +1. Branch from `master`; keep the PR small and focused — one PR equals one + reviewable change (see + [Google eng practices on small CLs](https://google.github.io/eng-practices/review/developer/small-cls.html)). +2. Run the canonical gate and make sure it is green: + + ```bash + bash scripts/verify.sh + ``` + + CI runs the exact same script; do not open (or mark ready) a PR with a + red local run. + +## Title + +English, Conventional Commit style, same format as commit messages: + +```text +(): +``` + +Examples: `feat(preprocess): add RSS fetcher`, `fix(flows): handle empty +batch input`, `docs(readme): update quick start`. + +## Body + +Written in English (external audiences read it: contributors, search +indexers, future maintainers). Follow `.github/PULL_REQUEST_TEMPLATE.md` +and make sure the body covers: + +1. **What changed and why** — a short summary; link the design discussion + if one exists. +2. **Related issue** — reference it when one exists (`Closes #NN` / + `Part of #NN`). +3. **Verification performed** — state what you ran (e.g. + `bash scripts/verify.sh` green; targeted `pytest tests//`; + manual example run) so the reviewer does not have to guess. + +Keep the template checklist and remove items that do not apply. + +## Review and Merge + +- Assign relevant reviewers when you know who owns the area. +- Address review comments with follow-up commits (no force-push rewrites of + reviewed history unless asked). +- After merge: switch back to `master`, pull, and delete the feature branch. diff --git a/.claude/skills/quantmind-dev/SKILL.md b/.claude/skills/quantmind-dev/SKILL.md new file mode 100644 index 0000000..a05338c --- /dev/null +++ b/.claude/skills/quantmind-dev/SKILL.md @@ -0,0 +1,42 @@ +--- +name: quantmind-dev +description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code. +--- + +# QuantMind Dev + +Development workflow for contributing to the QuantMind codebase. + +## Start Here + +1. Read the repository root `AGENTS.md` or `CLAUDE.md` for the stable + architecture constraints and the module map. +2. Pick exactly one workflow reference below; do not load the others. + +## Select Workflow + +- Committing staged work → `references/commit.md` +- Opening or updating a pull request → `references/pull-request.md` +- Implementing or refactoring anything under `quantmind/` → + `references/develop-components.md` (read it **before** writing code) + +A feature task usually chains all three: develop → commit → pull request. + +## Rules + +- `bash scripts/verify.sh` is the single "shippable" gate. CI runs the same + script; run it before every push and before marking a PR ready. +- Never bypass pre-commit / pre-push hooks (`--no-verify`) unless the user + explicitly authorizes it. +- New features ship with a unit test and a focused example (see + `references/develop-components.md`). +- This skill is mirrored under `.agents/skills/quantmind-dev/` and + `.claude/skills/quantmind-dev/`; the copies are identical. When changing + the skill, update both in the same change. + +## Boundaries + +- This skill is for contributing to QuantMind itself. It does not cover + using QuantMind as a library in your own project. +- Product decisions (new modules, new dependencies, API redesigns) need an + issue and maintainer discussion first; do not encode them here. diff --git a/.claude/skills/quantmind-dev/references/commit.md b/.claude/skills/quantmind-dev/references/commit.md new file mode 100644 index 0000000..32737e9 --- /dev/null +++ b/.claude/skills/quantmind-dev/references/commit.md @@ -0,0 +1,57 @@ +# Commit Workflow + +How to commit work in the QuantMind repository. + +## Message Format + +English, [Conventional Commits](https://www.conventionalcommits.org/): + +```text +(): +``` + +- **type**: `feat` | `fix` | `refactor` | `docs` | `test` | `chore` +- **scope**: optional but preferred — use the module or area the change + touches, matching existing history (`feat(memory): ...`, + `chore(verify): ...`, `docs(news): ...`). Omit the scope only for + repo-wide changes (`feat: flows + magic apex layer`). +- **summary**: imperative, lower-case start, no trailing period. + +Examples from history: + +```text +feat: preprocess fetch+format module (PR4 of OpenAI Agents SDK migration) (#75) +chore(verify): add scripts/verify.sh + ruff/basedpyright/import-linter/pytest --cov harness (#73) +fix(test): Prevent creation of `test_data` directory during unit tests (#41) +``` + +## One Logical Change per Commit + +Keep each commit focused on a single logical change. Split unrelated edits +(e.g. a bug fix discovered while building a feature) into separate commits. +Do not mix mechanical reformatting with behavior changes. + +## Before Committing + +1. Inspect exactly what you are about to commit: + + ```bash + git status + git diff --staged + ``` + + Confirm no unintended files (scratch scripts, local configs, large + artifacts) are staged. + +2. Run verification appropriate to the change: + - during development: targeted tests, e.g. `pytest tests//` + - before push / handoff: `bash scripts/verify.sh` (canonical gate) + +3. Commit. Pre-commit hooks run ruff format/check and file hygiene checks; + the pre-push hook runs the full `scripts/verify.sh`. + +## Hooks + +If a hook fails, fix the underlying issue and commit again. Never use +`git commit --no-verify` or otherwise bypass hooks unless the user has +explicitly authorized it for that specific commit. diff --git a/.claude/skills/quantmind-dev/references/develop-components.md b/.claude/skills/quantmind-dev/references/develop-components.md new file mode 100644 index 0000000..b04a647 --- /dev/null +++ b/.claude/skills/quantmind-dev/references/develop-components.md @@ -0,0 +1,105 @@ +# Component Development Workflow + +How to implement or refactor code under `quantmind/`. Read this before +writing code; the architecture constraints in root `AGENTS.md` / `CLAUDE.md` +apply throughout. + +## General Loop + +1. **Find the nearest existing pattern.** Locate the closest analogous + implementation in the target module and follow its structure, naming, + and test layout. Consistency beats novelty. +2. **Check the dependency contract** for your module (table below) before + adding any import. `lint-imports` enforces these; if your design needs a + forbidden import, the design is wrong — restructure, do not work around + the contract. +3. **Implement small and flat.** Pure functions over classes; `Protocol` + over ABC; no meaningless wrappers (a method must add logic, abstraction, + or a side effect beyond the call it wraps). No premature abstractions — + extract shared code when the second real caller appears, not before. +4. **Add the unit test and the example** (sections below). +5. **Update the public surface** if needed: module `__init__.py` exports + and user-facing docs (`README.md`, `docs/`) for user-visible features. +6. **Verify**: targeted `pytest tests//` while iterating; + `bash scripts/verify.sh` before handoff. + +## Module Routing + +### Dependency contracts (enforced by import-linter) + +| Module | May import from `quantmind.*` | +|--------|-------------------------------| +| `quantmind/utils/` | nothing (leaf) | +| `quantmind/knowledge/` | nothing (leaf) | +| `quantmind/configs/` | `knowledge` only | +| `quantmind/preprocess/` | `utils` only | +| `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above | + +### `quantmind/knowledge/` — data standard + +- Pydantic models, `frozen=True`, `extra="forbid"`. +- Every `BaseKnowledge` subclass **must** require `as_of: datetime` + (financial time-sensitivity is mandatory) and a typed `source: SourceRef` + (no bare strings), and **must** override `embedding_text()`. +- Pick one shape: `FlattenKnowledge` (atomic card), `TreeKnowledge` + (hierarchical artifact), or `GraphKnowledge` (placeholder). Whole-document + objects are `TreeKnowledge` even when a flatten card exists alongside + (e.g. `Paper` vs `PaperKnowledgeCard`). + +### `quantmind/configs/` — flow cfg + typed inputs + +- Extend `BaseFlowCfg`; inputs are discriminated-union Pydantic types. +- Never `Dict[str, Any]` in signatures — model it. + +### `quantmind/preprocess/` — deterministic data prep + +- Fetch / format / clean / time utilities: deterministic, no LLM calls. +- Return frozen dataclasses (`Fetched`, `RawPaper`, ...), not dicts. +- Surface the common path at the package root (`from quantmind.preprocess + import fetch_arxiv`), keep explicit submodule paths working. + +### `quantmind/flows/` and `quantmind/magic.py` — apex layer + +- Flows are pure `async def` functions, not classes; state passes as + arguments; side effects go through explicit hooks. +- Use the OpenAI Agents SDK directly (`Agent`, `@function_tool`, + `output_type=`); never wrap `from agents import ...` in a facade. +- Fan-out goes through `batch_run` (bounded concurrency, error policy); + `batch_run` rejects `memory=` at the signature layer by design. + +### `quantmind/mind/` — cognitive layer + +- Lands via the Agents SDK migration (tracking: #71). Backends implement + the `Memory` Protocol with granular `tools()`, `mcp_servers()`, + `run_hooks()`, `reset()` — each may return an empty list; do not force + MCP on every implementation. + +### `quantmind/utils/` + +- Logger only. New general-purpose helpers need maintainer sign-off via an + issue first; the default answer is "put it in the module that uses it". + +## Tests + +- Location mirrors the module: `tests//test_.py`. +- Subclass `unittest.TestCase` (run via pytest). +- Mock external services (network, LLM APIs, filesystem where practical); + tests must pass offline. +- Cover the success path **and** at least one failure path per public + function. +- Coverage floor is enforced by `pytest --cov` in `scripts/verify.sh`; new + code should not lower branch coverage. + +## Examples + +- One focused example per new feature under `examples//` + (create the directory if it does not exist yet). +- Demo the simple, common usage — one scenario per file, runnable as + `python examples//.py`, minimal setup. + +## Documentation + +- Docstrings: English, Google style, required on public functions and + models. +- User-visible behavior changes update `README.md` (usage section) and + `docs/` where applicable. diff --git a/.claude/skills/quantmind-dev/references/pull-request.md b/.claude/skills/quantmind-dev/references/pull-request.md new file mode 100644 index 0000000..8f64552 --- /dev/null +++ b/.claude/skills/quantmind-dev/references/pull-request.md @@ -0,0 +1,51 @@ +# Pull Request Workflow + +How to open and maintain a pull request against QuantMind. + +## Before Opening + +1. Branch from `master`; keep the PR small and focused — one PR equals one + reviewable change (see + [Google eng practices on small CLs](https://google.github.io/eng-practices/review/developer/small-cls.html)). +2. Run the canonical gate and make sure it is green: + + ```bash + bash scripts/verify.sh + ``` + + CI runs the exact same script; do not open (or mark ready) a PR with a + red local run. + +## Title + +English, Conventional Commit style, same format as commit messages: + +```text +(): +``` + +Examples: `feat(preprocess): add RSS fetcher`, `fix(flows): handle empty +batch input`, `docs(readme): update quick start`. + +## Body + +Written in English (external audiences read it: contributors, search +indexers, future maintainers). Follow `.github/PULL_REQUEST_TEMPLATE.md` +and make sure the body covers: + +1. **What changed and why** — a short summary; link the design discussion + if one exists. +2. **Related issue** — reference it when one exists (`Closes #NN` / + `Part of #NN`). +3. **Verification performed** — state what you ran (e.g. + `bash scripts/verify.sh` green; targeted `pytest tests//`; + manual example run) so the reviewer does not have to guess. + +Keep the template checklist and remove items that do not apply. + +## Review and Merge + +- Assign relevant reviewers when you know who owns the area. +- Address review comments with follow-up commits (no force-push rewrites of + reviewed history unless asked). +- After merge: switch back to `master`, pull, and delete the feature branch. diff --git a/AGENTS.md b/AGENTS.md index fd94e54..4a7e302 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,42 +1,82 @@ -# AI Development Guide +# QuantMind — Agent Instructions -> The following context is a bundle of best practices for AI development. Please follow the guidance strictly. +Guidance for coding agents contributing to this repository. Keep this file +aligned with `CLAUDE.md` (same core rules); update both in the same change. -## Avoid Meaningless Wrapper Methods +## What This Is -**Core Rule:** If a method only does simple data access + basic error checking, use it inline directly. +QuantMind is a knowledge extraction and retrieval library for quantitative +finance, built **on top of** the OpenAI Agents SDK. It is a domain library, +not an agent framework: runtime, tracing, tool scaffolding, and multi-agent +handoff all come from `openai-agents`. -One-sentence Decision Criteria: **"Does this method do any actual work beyond wrapping the call?"** +## Module Map -### Simplified Comparison +| Module | Role | +|--------|------| +| `quantmind/knowledge/` | Pydantic data standard (`FlattenKnowledge` / `TreeKnowledge` / `GraphKnowledge`) — dependency leaf | +| `quantmind/configs/` | Flow cfg + typed inputs (`BaseFlowCfg`, discriminated unions) — depends only on `knowledge` | +| `quantmind/preprocess/` | Deterministic fetch / format / clean / time utilities — depends only on `utils` | +| `quantmind/flows/` | Apex layer: end-to-end pipeline functions (`paper_flow`, `batch_run`) | +| `quantmind/magic.py` | `resolve_magic_input`: natural language → `(input, cfg)` | +| `quantmind/mind/` | Cognitive layer (memory protocol); landing via the Agents SDK migration (#71) | +| `quantmind/utils/` | Logger only — keep it that way | -```python -# ❌ Meaningless wrapper -def _get_llm(self, identifier: str): - if identifier not in self._llm_blocks: - raise KeyError(f"LLM block '{identifier}' not found") - return self._llm_blocks[identifier] +The pre-migration agent runtime was removed and archived on the +`archive/agent-runtime-final` branch. Reference it for history; never +resurrect it into master. -# ✅ Direct usage -llm_block = self._llm_blocks[identifier] # KeyError naturally thrown +## Setup and Verification + +```bash +uv venv && source .venv/bin/activate +uv pip install -e ".[dev]" +bash scripts/verify.sh # canonical "is this branch shippable" check ``` -### Keep wrappers when they provide +`scripts/verify.sh` runs five fast-fail steps (`ruff format --check`, +`ruff check`, `basedpyright`, `lint-imports`, `pytest --cov`). CI runs the +exact same script, so a green local run means a green PR. Do not bypass +pre-commit / pre-push hooks unless the user explicitly authorizes it — fix +the underlying issue instead. + +## Architecture Constraints (stable) + +1. **Library, not framework** — functions over classes, `Protocol` over ABC, + no plugin registries, no hook discovery, no CLI. +2. **Do not rebuild the agent runtime** — use `openai-agents` directly; no + QuantMind-side facades over `from agents import ...`. +3. **Pydantic at boundaries, frozen dataclass internally** — Pydantic + (`frozen=True`, `extra="forbid"`) for anything exposed to an LLM or a + user; frozen dataclasses for internal value types. +4. **Import boundaries are contracts** — `import-linter` (configured in + `pyproject.toml`) pins the dependency graph; never work around a failing + contract. +5. **Absolute imports** across module boundaries. +6. **No meaningless wrappers** — a method must add logic, abstraction, or a + side effect beyond the call it wraps; otherwise inline it. + +## Tests and Examples + +A new feature ships with a unit test **and** a focused example: -- **Complex logic**: Conditional logic, data transformation, loops -- **Abstract interfaces**: Abstract methods, public APIs -- **Side effects**: Logging, state changes, external calls +- Tests: `tests//`, subclass `unittest.TestCase`, mock external + services, cover success and failure paths. +- Examples: `examples//`, one simple usage per file. -**Remember:** Encapsulation should hide complexity, not add complexity. +## Communication -## Impl with Unit Test and Example +- Commit messages: English, Conventional Commits. +- PR titles, PR bodies, and issue bodies: English. +- Code comments and docstrings: English, Google style. -If you are implementing a new feature, please implement the unit test and example. +## Development Workflows -- For unit test, add in `tests/`, and inherit the `unittest.TestCase` class. -- For example, add in `examples/`, and just demo the simple usage. (do not add too many use cases in single file) +For commit, pull-request, or component-implementation tasks, load the +`quantmind-dev` skill and follow the matching reference: -## Comment Style +- `.agents/skills/quantmind-dev/SKILL.md` (this toolchain) +- `.claude/skills/quantmind-dev/SKILL.md` (Claude Code) -- All comments should be in English. -- All comments should be in the Google style. +The two copies are identical; when changing the skill, update both in the +same change. diff --git a/CLAUDE.md b/CLAUDE.md index 2c790ec..477b577 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,179 +1,83 @@ # CLAUDE.md -This file provides guidance to Claude Code (claude.ai/code) when working in this repository. +This file provides guidance to Claude Code (claude.ai/code) when working in +this repository. Keep this file aligned with `AGENTS.md` (same core rules); +update both in the same change. -## Project Overview +## What This Is -QuantMind is an intelligent knowledge extraction and retrieval framework for quantitative -finance. As of 2026-04, it is being **repositioned as a domain library that runs on top -of OpenAI Agents SDK**, rather than as a self-contained agent framework. +QuantMind is a knowledge extraction and retrieval library for quantitative +finance, built **on top of** the OpenAI Agents SDK. It is a domain library, +not an agent framework: runtime, tracing, tool scaffolding, and multi-agent +handoff all come from `openai-agents`. -The pre-pivot agent runtime (`brain/`, `tools/`, `storage/`, `tagger/`, custom Tool ABC, -custom MultiStepAgent / Memory) was removed in PR #70. A full snapshot of the removed -code is preserved on the `archive/agent-runtime-final` branch on origin — reference it -if you need historical context, never resurrect it into master. +## Module Map -## Target Architecture (post-migration) +| Module | Role | +|--------|------| +| `quantmind/knowledge/` | Pydantic data standard (`FlattenKnowledge` / `TreeKnowledge` / `GraphKnowledge`) — dependency leaf | +| `quantmind/configs/` | Flow cfg + typed inputs (`BaseFlowCfg`, discriminated unions) — depends only on `knowledge` | +| `quantmind/preprocess/` | Deterministic fetch / format / clean / time utilities — depends only on `utils` | +| `quantmind/flows/` | Apex layer: end-to-end pipeline functions (`paper_flow`, `batch_run`) | +| `quantmind/magic.py` | `resolve_magic_input`: natural language → `(input, cfg)` | +| `quantmind/mind/` | Cognitive layer (memory protocol); landing via the Agents SDK migration (#71) | +| `quantmind/utils/` | Logger only — keep it that way | -``` -quantmind/ -├── flows/ # e2e pipeline functions (paper_flow, news_flow, ...) -├── knowledge/ # Pydantic schemas (KnowledgeItem subclasses: Paper, News, ...) -├── preprocess/ # fetch (arxiv/http/doi/local) + format (pdf/html/markdown) -├── mind/ # cognitive layer; mind/memory/ is the MVP (filesystem-backed) -├── configs/ # centralized cfg + input types (BaseFlowCfg + per-flow types) -├── magic.py # resolve_magic_input: natural language -> (input, cfg) -└── utils/ # logger only +The pre-migration agent runtime was removed and archived on the +`archive/agent-runtime-final` branch. Reference it for history; never +resurrect it into master. + +## Setup and Verification + +```bash +uv venv && source .venv/bin/activate +uv pip install -e ".[dev]" +bash scripts/verify.sh # canonical "is this branch shippable" check ``` -Key principle: QuantMind does NOT rebuild Agent runtime, lifecycle hooks, tracing, -multi-agent handoff, or tool framework. Those come from `openai-agents`. +`scripts/verify.sh` runs five fast-fail steps (`ruff format --check`, +`ruff check`, `basedpyright`, `lint-imports`, `pytest --cov`). CI runs the +exact same script, so a green local run means a green PR. Do not bypass +pre-commit / pre-push hooks unless the user explicitly authorizes it — fix +the underlying issue instead. -## Current Repository State (after PR #70 / #73 / #74 / #75 / PR5) +## Architecture Constraints (stable) -| Module | Status | Notes | -|--------|--------|-------| -| `quantmind/knowledge/` | landed (PR3) | data standard with three shapes: `FlattenKnowledge` (`News` / `Earnings` / `PaperKnowledgeCard`), `TreeKnowledge` (`Paper`), `GraphKnowledge` (placeholder); shared base = `BaseKnowledge` with typed `SourceRef` / `ExtractionRef` provenance + `embedding_text()` contract | -| `quantmind/configs/` | landed (PR3) | `BaseFlowCfg` / `BaseInput` + per-flow cfg + discriminated-union input types | -| `quantmind/preprocess/` | landed (PR4) | `fetch/` (`fetch_arxiv` / `fetch_url` / `resolve_doi` / `read_local_file` returning `Fetched` / `RawPaper` / `CrossrefMetadata` frozen dataclasses) + `format/` (`pdf_to_markdown` via PyMuPDF, `html_to_markdown` via trafilatura) + `clean.py` + `time.py`; leaf module — only depends on `quantmind.utils` | -| `quantmind/flows/` | landed (PR5) | apex layer: `paper_flow` (`PaperInput` → `Paper` via SDK Agent), `batch_run` + `BatchResult` (bounded-concurrency fan-out, `memory=` rejected by design), `_runner.run_with_observability` + `_compose_hooks` + `_archive_run_artifacts` (PR6 stub); only depends on configs/knowledge/preprocess/utils + `agents` SDK | -| `quantmind/magic.py` | landed (PR5) | `resolve_magic_input(natural_language, *, target_flow, ...) -> (input, cfg)` plus `preview_resolve` debug helper; introspects flow signatures and runs a lightweight resolver Agent with `output_type=ResolvedFlowConfig[InputT, CfgT]` | -| `quantmind/utils/logger.py` | permanent | only general-purpose utility | +1. **Library, not framework** — functions over classes, `Protocol` over ABC, + no plugin registries, no hook discovery, no CLI. +2. **Do not rebuild the agent runtime** — use `openai-agents` directly; no + QuantMind-side facades over `from agents import ...`. +3. **Pydantic at boundaries, frozen dataclass internally** — Pydantic + (`frozen=True`, `extra="forbid"`) for anything exposed to an LLM or a + user; frozen dataclasses for internal value types. +4. **Import boundaries are contracts** — `import-linter` (configured in + `pyproject.toml`) pins the dependency graph; never work around a failing + contract. +5. **Absolute imports** across module boundaries. +6. **No meaningless wrappers** — a method must add logic, abstraction, or a + side effect beyond the call it wraps; otherwise inline it. -PR5 removed the transitional packages (`quantmind/{flow,llm,config,models}/` -and their tests under `tests/{config,models}/`); PR4 had already removed -`quantmind/parsers/`, `quantmind/sources/`, and `quantmind/utils/tmp.py`. -The codebase has now converged to the five permanent module roots -(`flows/`, `configs/`, `knowledge/`, `preprocess/`, `mind/`) plus -`magic.py` and `utils/`. +## Tests and Examples -`basedpyright` runs in standard mode across the whole `quantmind/` -package — there are no per-module exclusions left. Five `import-linter` -contracts pin the dependency graph: `utils` and `knowledge` are leaves, -`configs` only depends on `knowledge`, `preprocess` only depends on -`utils`, and `flows + magic` is the apex (cannot import the deleted -transitional packages, which are listed in the contract as a tripwire -against accidental re-introduction). +A new feature ships with a unit test **and** a focused example: -## Development Commands +- Tests: `tests//`, subclass `unittest.TestCase`, mock external + services, cover success and failure paths. +- Examples: `examples//`, one simple usage per file. -### Environment +## Communication -```bash -uv venv -source .venv/bin/activate -uv pip install -e ".[dev]" -``` +- Commit messages: English, Conventional Commits. +- PR titles, PR bodies, and issue bodies: English. +- Code comments and docstrings: English, Google style. -### Verify (canonical local check) +## Development Workflows -`scripts/verify.sh` is the single source of truth for "is this branch -shippable". CI (`.github/workflows/verify.yml`) runs the exact same script, -so a green local run means a green PR. Run it before every push: +For commit, pull-request, or component-implementation tasks, load the +`quantmind-dev` skill and follow the matching reference: -```bash -bash scripts/verify.sh -``` +- `.claude/skills/quantmind-dev/SKILL.md` (Claude Code) +- `.agents/skills/quantmind-dev/SKILL.md` (other agent toolchains) -It runs five steps in fixed order, fast-failing on the first error: - -1. `ruff format --check` — formatting must be clean -2. `ruff check` — lint (D, E, F, I, W, B, W505) must pass -3. `basedpyright` — standard-mode type check on permanent + new modules -4. `lint-imports` — architectural boundary contracts must hold -5. `pytest --cov` — tests pass with ≥ 75% branch coverage (raised from 65 - in PR5 after the transitional packages were deleted) - -Pre-commit hooks (`.pre-commit-config.yaml`): -- pre-commit stage: trailing whitespace / EOF / ruff / ruff-format (fast) -- pre-push stage: full `scripts/verify.sh` - -Don't bypass hooks unless the user explicitly authorizes — fix the underlying -issue instead. - -## Architecture Principles - -1. **No framework, just lib** — Functions over classes; Protocol over ABC; no plugin - registries or hook discovery -2. **Pure functions** — Flows are `async def run(...)`, not classes; state passed as - args; side effects via explicit hooks -3. **Pydantic at boundaries, frozen dataclass internally** — Pydantic for anything - exposed to LLM (`output_type=`, cfg, input); frozen dataclass for internal value - types -4. **Batch is first-class** — `batch_run(flow_fn, inputs, ...)` will land in PR4 - (concurrency + error handling + progress aggregation). Users do NOT write - `asyncio.gather` boilerplate themselves -5. **Customization 3 layers** — cfg (YAML/CLI), kwargs (Python `extra_*` flow args), - building blocks (fork the flow file). Each layer has explicit extension points -6. **Observability 3 layers** — SDK auto-tracing, external processors via - `add_trace_processor()`, local trajectory archive under `/runs/` -7. **No CLI** — User-facing entry is a runbook script (5 lines of Python), not a - framework command. Magic input is the loose-input UX, resolved by an Agent -8. **Magic input first** — Users describe intent in natural language; - `magic.resolve_magic_input(...)` returns a structured `(input, cfg)` tuple - -## Conventions When Editing - -- **Schemas**: Pydantic, `extra="forbid"`, `frozen=True`. All `BaseKnowledge` - subclasses must require `as_of: datetime` (financial time-sensitivity is mandatory) - and provide a typed `source: SourceRef` (no bare strings). Subclasses MUST - override `embedding_text()` so the store layer knows what to embed. -- **Knowledge shapes**: pick one of `FlattenKnowledge` (atomic card), - `TreeKnowledge` (hierarchical artifact), or wait for `GraphKnowledge` - (placeholder). Whole-document objects are `TreeKnowledge` even when a - flatten card exists alongside (e.g. `Paper` vs `PaperKnowledgeCard`). -- **Configs**: Extend `BaseFlowCfg` (lands in PR2); never use `Dict[str, Any]` in - init signatures -- **Tools**: SDK's `@function_tool` decorator; do NOT subclass anything -- **Memory backends**: Implement the `Memory` Protocol with granular `tools()`, - `mcp_servers()`, `run_hooks()`, `reset()` — each may return an empty list. Do not - force MCP on every implementation -- **Tests**: Subclasses of `unittest.TestCase` in `tests//`. Mock external - dependencies; cover both success and failure paths -- **Imports**: Absolute (`from quantmind.knowledge import Paper`); no relative - imports across module boundaries - -## Communication Conventions - -- **PR descriptions and issue bodies must be written in English**, regardless of the - language of the conversation that triggered them. They are read by external audiences - (search indexers, future maintainers, contributors who don't read Chinese). -- Commit messages: English, conventional-commit style (`feat:` / `fix:` / `refactor:` / - `docs:` / `chore:` ...). -- Inline PR review comments and issue discussion threads may be in whichever language - fits the participants. - -## Things NOT to Do - -- ❌ Rebuild Agent runtime / Tool ABC / lifecycle hook abstraction -- ❌ Add a CLI (`argparse`/`typer`/`click`); users run Python runbook scripts -- ❌ Introduce class-based `BaseFlow` / plugin registry / hook discovery -- ❌ Wrap `from agents import ...` in a QuantMind-side facade — use the SDK directly -- ❌ Mix `batch_run` and `memory` (mutually exclusive in MVP; `batch_run` rejects - `memory=` at the signature layer — design doc §4.3.5) -- ❌ Use `Dict[str, Any]` in init functions; use Pydantic models -- ❌ Add hard deps on observability platforms (Langfuse / Logfire / etc.); document - integration via `add_trace_processor()` in user-facing cookbook only -- ❌ Build embedding-based memory before filesystem memory has shipped and stabilized - -## Reference Material - -- OpenAI Agents SDK docs: -- Lifecycle / RunHooks API: -- MCP integration (filesystem server): -- Tracing (auto-capture, processors, disable): -- Original SDK announcement: -- Removed agent runtime snapshot: `archive/agent-runtime-final` branch on origin - -## Roadmap (post-PR1) - -| PR | Focus | -|----|-------| -| #70 (merged) | Clean removal of self-built agent runtime | -| #73 (merged) | Golden Harness — `scripts/verify.sh` with ruff + basedpyright + import-linter + pytest --cov, plus matching CI | -| #74 (merged) | `knowledge/` data standard (Flatten / Tree / Graph shapes) + `configs/` skeleton; `openai-agents>=0.14` introduced for `BaseFlowCfg.model_settings` | -| #75 (merged) | `preprocess/` (fetch + format two layers); deletes `parsers/` + `sources/` + `utils/tmp.py`; coverage floor 60→65; 4th import-linter contract | -| PR5 (this PR) | `flows/` (`paper_flow` + `batch_run` + `BatchResult` + `_runner`) + `magic.py`; deletes `quantmind/{flow,llm,config,models}/`; coverage floor 65→75; 5th import-linter contract pins `flows + magic` as apex | -| PR6 | `mind/memory/filesystem` MVP + trajectory archive (fills `_archive_run_artifacts` stub) | -| PR7 | `mind/store/` + SQLite + `sqlite-vec` MVP; introduces `preprocess/chunk.py` with `tiktoken` | -| PR8+ | Second flow (news/earnings) / observability cookbook / longer-term modules | +The two copies are identical; when changing the skill, update both in the +same change. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 014f892..e5eb4d7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,144 +1,91 @@ # Contributing to QuantMind -Thank you for contributing to QuantMind! This guide provides essential information for developers. +Thank you for contributing to QuantMind! This guide covers environment +setup and the contribution process. The canonical development workflow +(commit format, PR format, component development) lives in the +`quantmind-dev` skill — `.claude/skills/quantmind-dev/SKILL.md` / +`.agents/skills/quantmind-dev/SKILL.md` — and the repository-wide rules +live in `AGENTS.md` / `CLAUDE.md`. If you develop with a coding agent, it +will pick these up automatically. ## 🚀 Quick Setup 1. **Fork and clone** the repository 2. **Set up environment**: + ```bash uv venv && source .venv/bin/activate - uv pip install -e . + uv pip install -e ".[dev]" ``` + 3. **Install pre-commit hooks**: + ```bash ./scripts/pre-commit-setup.sh ``` -## 🛠️ Development Setup - -### Pre-commit Hooks +## ✅ Verification -We use pre-commit hooks to ensure code quality and consistency. These hooks automatically format code, run linting, and perform other quality checks before each commit. - -**Install pre-commit hooks:** +`scripts/verify.sh` is the single source of truth for "is this branch +shippable". CI runs the exact same script, so a green local run means a +green PR: ```bash -# Automated setup (recommended) -./scripts/pre-commit-setup.sh - -# Or manual setup -pip install pre-commit -pre-commit install -pre-commit install --hook-type pre-push +bash scripts/verify.sh ``` -**What the hooks do:** - -- **On every commit:** - - Code formatting with `ruff format` (80-char line length) - - Linting with `ruff check --fix` (auto-fixes issues) - - File quality checks (trailing whitespace, EOF, YAML syntax) - - Safety checks (large files, merge conflicts) +It runs five fast-fail steps: `ruff format --check`, `ruff check`, +`basedpyright`, `lint-imports`, `pytest --cov` (with a branch-coverage +floor configured in `pyproject.toml`). -- **On push to remote:** - - Full unit test suite via `scripts/unittest.sh` - -**Manual execution:** +**Hooks**: the pre-commit stage runs formatting/lint and file hygiene +checks; the pre-push stage runs the full `scripts/verify.sh`. If a hook +fails, fix the issue — don't bypass with `--no-verify`. ```bash -# Run formatting and linting -./scripts/lint.sh - -# Run all pre-commit hooks on all files +# Run all pre-commit hooks manually pre-commit run --all-files -# Run specific tests -./scripts/unittest.sh tests/quantmind/sources/ -./scripts/unittest.sh all # Run all tests +# Run targeted tests while iterating +pytest tests// ``` -**Troubleshooting:** - -- If hooks fail, fix the issues and commit again -- To skip hooks temporarily (not recommended): `git commit --no-verify` -- Update hooks: `pre-commit autoupdate` - ## 📝 Development Standards -### Code Requirements -- **Location**: All new code in `quantmind/` module -- **Style**: Google-style docstrings, 80-char line length -- **Architecture**: Abstract base classes + dependency injection -- **Type Safety**: Pydantic models + comprehensive type hints - -### Testing -- **Unit tests**: Required in `tests/quantmind/` (mirror module structure) -- **Coverage**: Test success and error cases -- **Mocking**: Mock external APIs and file systems - -### Documentation -- **Examples**: Add to `examples/quantmind/` for new features -- **Docstrings**: Google-style format for all public methods - -## 🏗️ Contribution Types - -### New Sources -- Extend `BaseSource[ContentType]` in `quantmind/sources/` -- Add config in `quantmind/config/sources.py` -- Include tests and usage example - -### New Parsers -- Extend `BaseParser` in `quantmind/parsers/` -- Handle multiple content formats with error handling - -### New Taggers -- Extend `BaseTagger` in `quantmind/tagger/` -- Support rule-based and ML approaches - -### Storage Backends -- Extend `BaseStorage` in `quantmind/storage/` -- Implement indexing, querying, and concurrent access +- **Architecture**: QuantMind is a domain library on top of the OpenAI + Agents SDK — functions over classes, `Protocol` over ABC, no CLI, no + agent-runtime rebuilding. See `AGENTS.md` / `CLAUDE.md` for the stable + constraints and the module map. +- **Style**: Google-style docstrings, English comments, 80-char lines + (enforced by ruff). +- **Types**: Pydantic models at boundaries, frozen dataclasses internally; + comprehensive type hints (`basedpyright` runs in standard mode). +- **Dependency boundaries**: enforced by `import-linter` + (`pyproject.toml`); don't work around a failing contract. +- **Tests**: required under `tests//` (mirror the module + structure), `unittest.TestCase`, mock external services, cover success + and failure paths. +- **Examples**: one focused example under `examples//` for each + new feature. ## 🔄 Pull Request Process -1. **Create feature branch** from `master` -2. **Follow conventional commits**: `type(scope): description` -3. **Pre-commit hooks** run automatically on commit/push -4. **Before submitting**: - ```bash - pre-commit run --all-files - ./scripts/unittest.sh all - ``` -5. **Submit PR** using our template +1. **Create a feature branch** from `master`. +2. **Follow Conventional Commits**: `type(scope): description`, in English. +3. **Verify before submitting**: `bash scripts/verify.sh` must be green. +4. **Submit the PR** using the template — English body, reference the + related issue, and state the verification you performed. +5. Keep PRs small and focused + ([Google eng practices](https://google.github.io/eng-practices/review/developer/small-cls.html)). -### PR Checklist -- [ ] Code in `quantmind/` following architecture patterns -- [ ] Unit tests with comprehensive coverage -- [ ] Usage example (for new features) -- [ ] All pre-commit hooks pass -- [ ] Conventional commit format - -## 💡 Development Tips - -```bash -# Run specific tests -pytest tests/quantmind/sources/ -pytest tests/quantmind/models/ - -# Test CLI functionality -quantmind extract "test query" --max-papers 5 -quantmind config show - -# Check code quality -./scripts/lint.sh -``` +For significant changes (new modules, new dependencies, API redesigns), +open an [issue](https://github.com/LLMQuant/quant-mind/issues) to discuss +first. ## ❓ Questions? - Check existing [issues](https://github.com/LLMQuant/quant-mind/issues) - Review architecture patterns in existing code -- Look at `examples/` for usage patterns -- See `CLAUDE.md` for detailed architecture +- See `AGENTS.md` / `CLAUDE.md` for repository-wide rules Thank you for contributing! 🚀