itdove/lore#37: DocRepoIngester: LLM-powered documentation repo ingestion

[itdove]

Here's the filled PR template:

---

Jira Issue: N/A

Description

Adds DocRepoIngester — a new ingester that scans git repositories of unstructured documentation (.md, .rst, .txt, .adoc) and extracts structured knowledge entries via LLM. Resolves #37.

Core changes:

• New DocRepoIngester class (src/lore/ingest/doc_repo.py) implementing the LoreIngester ABC — orchestrates DocIngester per file with repo-level sync, path filtering (doc_paths/exclude_paths), content-hash-based incremental sync, and provenance tracking
• SyncEngine updated to detect ingester: "doc-repo" in hierarchy config and route to DocRepoIngester instead of the default frontmatter parser; repos without the field use existing behavior (backward compatible)
• Write isolation unified: all levels 1+ now use PR-only flow — no direct DB writes
• CLI (lore ingest) extended with --repo, --doc-paths, and --exclude-paths flags for one-off doc repo ingestion
• Config models updated with doc_paths, exclude_paths, and ingester fields on hierarchy entries
• New dashboard pages: Capture page for manual knowledge capture, Logs page for ingestion activity
• Dashboard browser and entry detail pages updated with doc-repo entry support
• Demo config and scripts updated for level 3 (docs) with doc-repo ingestion setup

Assisted-by: Claude

Testing

Steps to test

1. Pull down the PR
2. Install dependencies: pip install -e ".[dev]"
3. Run unit tests: pytest tests/unit/test_ingest_doc_repo.py tests/unit/test_sync_engine.py tests/unit/test_cli.py tests/unit/test_mcp_tools.py -v
4. Set up a demo config: bash scripts/create-demo-config.sh
5. Run lore sync and verify doc-repo hierarchy entries are processed via DocRepoIngester
6. Run lore ingest --repo <doc-repo-url> --doc-paths docs/guides for one-off ingestion
7. Start dashboard (lore dashboard) and verify Capture and Logs pages render

Scenarios tested

• Path filtering: only files under doc_paths are processed; exclude_paths are skipped
• Incremental sync: unchanged files (same content hash) are not reprocessed
• DocRepoIngester routes correctly when ingester: "doc-repo" is set in config
• Backward compatibility: hierarchy entries without ingester field use existing frontmatter parser
• Unit tests cover path filtering, sync state, dedup, CLI flags, and MCP tool registration

Deployment considerations

• This code change is ready for deployment on its own
• This code change requires the following considerations before being deployed: