Oversight architecture: reviewer≠corrector, modern component taxonomy, methodology - #31
Open
crbazevedo wants to merge 8 commits into
Open
Oversight architecture: reviewer≠corrector, modern component taxonomy, methodology#31crbazevedo wants to merge 8 commits into
crbazevedo wants to merge 8 commits into
Conversation
…er ≠ corrector)
Equation 6 previously folded detection and repair into one number c
(catch_rate), implicitly assuming every caught error is perfectly fixed.
This generalizes it:
σ_corr = σ_raw + (1 − σ_raw) · c_eff, c_eff = catch_rate × fix_rate
- catch_rate (c): the REVIEWER's probability of *detecting* an error
(a human reviewer, or a model acting as LLM-as-judge — verdict only).
- fix_rate (f): the CORRECTOR's probability of *repairing* a flagged error
by re-doing the task or patching, using the reviewer's feedback.
The paper's single-number model is exactly the special case fix_rate=1, so
all existing values are unchanged (parity preserved by default).
Changes:
- _formulae.sigma_corr_fixed_point gains fix_rate=1.0; threaded through
recursive_chain_quality, corrector_capacity_threshold, max_pipeline_depth.
- models.Node gains fix_rate field; docstring reframes reviewer vs corrector.
- capacity.compute_node_capacity / compute_pipeline_capacity read node.fix_rate.
- estimation: estimate_catch_rate now documented as estimating the *effective*
correction c_eff (outcomes alone can't separate detect from repair); new
estimate_fix_rate(flagged, repaired) recovers f from the reviewer's flag log.
- web/mso-core.js mirrors the generalization (fixRate defaults to 1).
- Parity: scorr cases gain 3-arg form; new PIPE_FIX pipeline with fix_rate<1.
- New tests: TestCorrectorDecomposition (formulae) + tests/test_estimation.py.
All 95 tests pass; JS↔Python parity verified through Node incl. fix_rate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Provenance for every cold-start σ band, with confidence tags (CONFIRMED / SOURCED / INTERPOLATED) and citations. Discloses sources per operator request. Covers the five research gaps: 1. Frontier-model faithfulness — Vectara HHEM-2.3 leaderboard (CONFIRMED): reasoning-vs-faithfulness tradeoff (DeepSeek-R1 11.3% vs V3 6.1%), top tier (GPT-5.4-nano 3.1%, Gemini-2.5-flash-lite 3.3%, Llama-3.3-70B 4.1%). Honest "no public benchmark" for Fable / exact GPT-5.5 / Opus 4.8. 2. Embedding/retrieval models (MTEB/BEIR) — retrieval is embedder + vector search, NOT an LLM. Qwen3-Embedding, OpenAI-3, BGE-M3, mGTE, Granite. 3. Rerankers — cross-encoder vs listwise-LLM uplift; recency penalty. 4. Reviewer catch_rate (RewardBench-2) + corrector fix_rate (self-correction papers): intrinsic self-correction without external feedback is unreliable — validates the detection × fix-success split. 5. Modern RAG + agentic component taxonomy (Berkeley compound-AI, ReAct). Added to mkdocs nav under a new Methodology section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ectors 78 → 120 cells, 43 models, 9 task types. Single source of truth is now priors.yaml; web/mso-priors.js is generated by scripts/gen_priors_js.py and pinned by a drift-guard test. Provenance for every band: priors-evidence.md. Architecture corrections (per operator feedback): - LLMs do NOT do retrieval. The 13 LLM "retrieval" cells were grounded- generation (RAG reader) faithfulness proxies → moved to a new `grounded_generation` task. `retrieval` is now embedder-only. - Reviewer ≠ corrector: new `review` (catch_rate, detection) stays; new `correction` task (fix_rate, repair) added with the seed_node + schema support (schema_version 2). New task types: grounded_generation, retrieval (embedders), reranking, correction. New models: - Frontier faithfulness (Vectara HHEM-2.3, CONFIRMED): gpt-5.4-nano, gemini-2.5-flash-lite/pro, phi-4, qwen3-8b, kimi-k2.5, o3-pro, o4-mini-high (reasoning models carry the CONFIRMED faithfulness penalty). - User-named-but-unbenchmarked (INTERPOLATED, flagged, NOT invented): gpt-5.5, claude-opus-4.8, kimi-2.6, fable-5 (no public benchmark). - Embedders (MTEB/BEIR): qwen3-embedding-8b, openai-3-large/small, cohere-embed-v3, e5-mistral-7b, bge-m3, granite-embedding-r2. - Rerankers (cross-encoder + listwise LLM): qwen3-reranker, cohere-rerank-3, bge-reranker-v2-m3, rankzephyr-7b, rankgpt-gpt4. - Reviewers: llm-judge-single / llm-judge-ensemble / human-reviewer. - Correctors: corrector-no-feedback (low fix_rate — self-correction without external feedback is unreliable) / with-feedback / with-oracle. Adds gen_priors_js.py (--check drift mode), tasksForModel/modelsForTask JS helpers, and tests: correction-seed, no-feedback<with-feedback fix_rate, JS-bundle-drift guard, parity for all v2 component types. 97 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clean, reader-facing methodology page (operator request). Explains, concept- first with symbols in parentheses: - the three distinct competences (raw σ_raw / catch rate c / fix rate f) and why detection and repair must be separated (σ_corr = σ_raw + (1−σ_raw)·c·f); - corrected ≠ autonomous competence → the masking index; - how a benchmark becomes a seed: σ_skill = clamp(σ_raw_mid/γ, .05, .98) so the node reproduces the prior at the return-operator fixed point; reviewers seed c, correctors seed f directly; - band width → confidence (humility meter); - normalization rules (absolute→σ with discount; relative→ranking only, never inverted; widen on doubt; INTERPOLATED never invented); - the component taxonomy (LLM does NOT retrieve; reviewer ≠ corrector); - honest caveats (judge catch softest, retrieval corpus-dependent, reasoning≠ faithful, self-correction-without-feedback unreliable). Linked from mkdocs nav under Methodology, alongside the evidence ledger. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reworks the node palette to match real RAG/agentic architecture and adds the detection×fix-success oversight model to the UI (per operator feedback). Taxonomy (palette now grouped Generate / Retrieve / Govern / Human / Tools): - GENERATE: LLM drafter, Code generator, Extractor, Classifier/router, RAG reader (grounded generation). - RETRIEVE (NOT language models — drawn as parallelograms, cyan): Embedder / retriever, Reranker (post-rank). Fixes "LLMs do not do retrieval": an LLM's only retrieval roles are reranking and grounded generation. - GOVERN: Model reviewer (LLM-as-judge) and Corrector (re-do/patch) as SEPARATE components — reviewer detects (catch rate c), corrector repairs (fix rate f). - RAG template rebuilt: embedder → reranker → grounded reader → groundedness judge → answer (was a chain of mislabeled "rag"/llm nodes). Oversight model: - Every node gets a fix-rate (f) slider; σ_corr uses c·f end-to-end. Catch-rate floor lowered to 0 (embedders/retrievers don't self-correct). - New lesson "Reviewer ≠ corrector": a judge that detects but a corrector that can't repair leaves quality stuck; raising f crosses B_eff above zero. Verified: C_op 0.331 → 0.459 when f→0.8. - Seeder defaults the model to one that actually has the node's task, filters the task dropdown to valid (model,task) combos, and seeds fix_rate for correction cells. Prior chip + preview label catch-rate / fix-rate / σ_skill distinctly. Vocabulary (concept-first, symbol in parentheses): - Toolbar: "Quality target (p_min)", "Process complexity (H(W))". - Cards: "Capacity ceiling (C_op)", "Autonomy buffer (B_eff)", "Worst masking (M*)", "Fragile motifs" — each with an explanatory tooltip. - Inspector sliders renamed; legend updated (generator / retriever-not-an-LLM / reviewer-corrector / human / tool-data). Footer links the methodology page; package pin bumped to 0.1.3. Browser-verified: 20-component palette, RAG architecture, fix lever, seeder defaulting, inspector fix slider — no console errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframes the Prescriptions panel to answer the paper's allocation questions directly, and adds the missing positive direction of the Minimum Sufficient Oversight principle — where autonomy can EXPAND: - "Block autonomy / redesign" when infeasible (check feasibility before expanding autonomy). - "Keep review here" at high-masking, central nodes (where oversight has the highest marginal value; corrected ≠ autonomous competence). - "Intervene upstream" on a diamond's shared source (early intervention prevents downstream correction load). - NEW "Autonomy can expand here" when a reviewed node's M*≈1 and the pipeline has buffer — reclaim that oversight for higher-masking nodes (the water-filling reallocation; minimize governance burden). Verified in-browser: all four fire on the right conditions; "expand" gated on feasible && B_eff>0 && reviewed node with low masking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"The Questions It Answers" — ties each allocation question and prescription from the paper/LinkedIn framing to the exact metric, lever, prescription, or lesson that answers it: where autonomy expands, where review remains, intervene- upstream, feasibility-before-autonomy, fragile motifs, process entropy vs the autonomy cliff, and corrected≠autonomous competence. Added to mkdocs nav. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`mkdocs` deploy (docs.yml) stages web/ → docs/app so the cockpit serves at
/app/widgets/. Mirror that locally for an integrated preview where in-page
links (e.g. the methodology link) resolve exactly as in production:
cp -r web docs/app && rm -f docs/app/README.md
mkdocs serve -a 127.0.0.1:8055 # → http://127.0.0.1:8055/delegation-lab/
Ignore the staged copy so it never lands in git.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Responds to the feedback batch on the cockpit. Stacked on #30 (cold-start priors) — base retargets to
mainonce #30 merges.What changed
1. Reviewer ≠ corrector (detection × fix-success). Equation 6 generalized to
σ_corr = σ_raw + (1−σ_raw)·c·f: a reviewer detects (catch ratec), a corrector repairs (fix ratef). The paper's single-number model is thef=1special case (parity preserved). Threaded through formulae,Node, capacity, estimation (estimate_fix_ratefrom the reviewer flag log), and the JS core.2. "LLMs do not do retrieval" — corrected. Retrieval is an embedder + vector search, not an LLM. The cockpit palette now groups Generate / Retrieve / Govern: retrievers & rerankers are drawn as non-LLM components; an LLM's retrieval roles are reranking (post-ranking) and grounded generation (RAG reader). The RAG template was rebuilt: embedder → reranker → grounded reader → groundedness judge.
3. Model reviewer (LLM-as-judge) vs human reviewer; correctors that consume feedback. New
Model reviewerandCorrectorcomponents; new lesson "Reviewer ≠ corrector" — a judge that detects but a corrector that can't repair leaves quality stuck until you raisef.4. Priors table v2 (78 → 120 cells, 43 models, 9 task types). New frontier models from the Vectara HHEM faithfulness leaderboard (incl. the confirmed reasoning-vs-faithfulness penalty), embedders (MTEB/BEIR), rerankers (cross-encoder + listwise LLM), LLM-judge catch rates (RewardBench-2), corrector fix rates (self-correction papers). User-named models with no public benchmark (Fable 5, exact GPT-5.5 / Opus 4.8 / Kimi 2.6) are flagged INTERPOLATED, never invented.
priors.yamlis now the single source of truth;web/mso-priors.jsis generated byscripts/gen_priors_js.pyand drift-guarded by a test.5. Methodology disclosed. New docs: How the Priors Are Built (the γ fixed-point inversion, c/f definitions, normalization rules, confidence proxy, taxonomy, caveats) and a Priors Evidence Ledger (every band cited, tagged CONFIRMED / SOURCED / INTERPOLATED).
6. Concept-first vocabulary. "Capacity ceiling (C_op)", "Autonomy buffer (B_eff)", "Worst masking (M*)", "Quality target (p_min)", "Process complexity (H(W))", and the inspector sliders — concept first, symbol in parentheses.
7. Every paper question → a cockpit lever. Prescriptions reframed to the allocation questions (keep review / intervene upstream / block autonomy / expand autonomy), plus a guide — The Questions It Answers — mapping each claim to its metric/lever/lesson.
Test plan
uv run pytest) — incl. JS↔Python parity for the newfix_rateand all v2 component types, and the JS-bundle drift guard.mkdocs build --strictclean (no broken links/anchors).🤖 Generated with Claude Code