diff --git a/.gitignore b/.gitignore index 6530ecd..4015f9a 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,5 @@ venv/ *.ipynb_checkpoints/ .DS_Store site/ +# staged copy of web/ for the integrated docs preview (mirrors CI deploy) +docs/app/ diff --git a/docs/guides/questions.md b/docs/guides/questions.md new file mode 100644 index 0000000..859e887 --- /dev/null +++ b/docs/guides/questions.md @@ -0,0 +1,98 @@ +# The questions the cockpit answers + +The paper frames oversight as an **allocation problem**: once work is delegated, +governance is no longer "how accurate is the model?" but "where should oversight +go?" This page maps each question the framework poses to the exact place you can +*read the answer off* the [cockpit](https://crbazevedo.github.io/delegation-lab/app/widgets/cockpit.html) — +through a metric, a lever, a prescription, or a lesson. + +Every value is computed by the same equations as the `minimal-oversight` package +(pinned by a parity test), so the cockpit is the paper made interactive, not a +mock-up. + +## Where should autonomy expand? + +**Read it:** the **Prescriptions** panel surfaces *"Autonomy can expand at X — +review is barely lifting it (M\*≈1)"* whenever a reviewed node has slack and the +pipeline has a positive **Autonomy buffer (B_eff)**. That is the water-filling +rule running in reverse: reclaim oversight where its marginal value is near zero. +**Lever:** lower a node's catch rate (c) and watch B_eff stay positive. + +## Where should review remain? + +**Read it:** **Risk ranking** orders nodes by `S = DC · M* · κ`, and the +Prescriptions panel says *"Keep review at X"* for high-**masking (M\*)**, central +nodes. High masking means corrected quality is propping up weak raw competence — +exactly where removing review would bite. + +## Where should intervention happen earlier (upstream)? + +**Read it:** **Delegation centrality (DC)** in the risk ranking measures how much +downstream work depends on a node. The **diamond** motif triggers the +prescription *"Intervene upstream — correct the shared source, not the merge."* +**Lesson:** *Reviewer placement* shows a reviewer on one branch leaving the other +raw; moving it above the fan-out fixes both. + +## Where is the workflow too uncertain / complex / drift-sensitive to delegate? + +**Read it:** the **feasibility banner**. If **Capacity ceiling (C_op)** falls +below the **Quality target (p_min)** the pipeline is INFEASIBLE — no local policy +rescues it. Raise **Process complexity (H(W))** and watch the buffer collapse +toward the capacity cliff (H_crit). + +## Which regions consume oversight without improving outcomes? + +**Read it:** **masking (M\*)** near 1.0 on a *reviewed* node — review is running +but barely changing the output. The "Autonomy can expand" prescription points +straight at it. + +## Which task motifs are structurally fragile under delegation? + +**Read it:** the **Motifs** panel tags `chain`, `fan_out`, `merge`, and +`diamond`. **Lessons:** *Merge gate type* (all-required vs vote) and *Chain depth +& masking* show how each motif degrades and how to redesign it. + +## Where does process entropy increase as autonomy expands? + +**Read it:** the **Process complexity (H(W))** slider feeds the **Autonomy buffer +(B_eff = C_op − p_min − λ·H(W))**. More routing/branching ⇒ higher H(W) ⇒ smaller +buffer ⇒ closer to the cliff. + +## Where does early intervention prevent downstream correction load? + +**Read it:** high **delegation centrality (DC)** marks nodes whose errors +compound downstream. Fixing them upstream (vs. correcting every consequence) is +the cheaper allocation — the SOTA priority score `S` ranks exactly these. + +## Where should autonomy be *blocked* until feasibility improves? + +**Read it:** the **INFEASIBLE** banner names the bottleneck and the prescription +says *"Block autonomy / redesign … before expanding autonomy."* Feasibility is a +gate, not a goal. + +## Is "corrected" performance evidence of autonomous competence? + +**No — and the cockpit makes the trap visible.** A node can show high corrected +quality (σ_corr) while its **raw competence (σ_raw)** is poor; the gap is the +**masking index (M\* = σ_corr / σ_raw)**. **Lesson:** *Reviewer ≠ corrector* +shows that a reviewer which only *detects* (catch rate c) without a corrector +that *repairs* (fix rate f) leaves quality stuck — corrected ≠ autonomous. + +--- + +### The framework's prescriptions, and where they live + +| Prescription (from the paper) | In the cockpit | +|---|---| +| Allocate oversight where marginal value is highest | Risk ranking (S = DC·M*·κ) + "Keep review" prescriptions | +| Intervene upstream when downstream correction compounds | Delegation centrality + diamond motif + "Intervene upstream" | +| Identify motifs that degrade under delegation | Motifs panel + Merge-gate / Chain-depth lessons | +| Check feasibility before expanding autonomy | Feasibility banner (C_op vs p_min), H_crit cliff | +| Treat corrected performance as insufficient evidence | Masking index + Reviewer ≠ corrector lesson | +| Minimize governance burden (the MSO principle) | "Autonomy can expand here" prescription | + +Start from a template, slide the **Quality target** and **Process complexity**, +seed nodes from public benchmarks (the ⊕ button — see +[How the priors are built](../methodology/priors.md)), then refine every number +with your own traces. The cockpit is a calculator for the autonomy ↔ uncertainty +↔ process-structure ↔ intervention ↔ capacity tradeoff — not a verdict. diff --git a/docs/methodology/priors-evidence.md b/docs/methodology/priors-evidence.md new file mode 100644 index 0000000..f211461 --- /dev/null +++ b/docs/methodology/priors-evidence.md @@ -0,0 +1,191 @@ +# Priors Evidence Ledger + +This page is the **provenance ledger** for the cold-start priors in +`minimal_oversight/data/priors.yaml`. Every σ band the cockpit seeds a node +with traces back to a row here. Priors are *starting points to refine with your +own traces* — not ground truth. See [Methodology](priors.md) for how raw +benchmark numbers become σ. + +**Confidence tags.** Each row is tagged by how it was verified: + +- **CONFIRMED** — independently corroborated (≥2 adversarial verifier votes). +- **SOURCED** — drawn from a primary leaderboard / benchmark paper, but the + automated cross-verification pass was rate-limited before it could vote. + Treated as cited-but-unverified → **wider bands**. +- **INTERPOLATED** — no direct public benchmark for that exact (model × task); + a conservative wide band set by tier interpolation, flagged in-cell. + +Last research sweep: **2026-06-18** (5 angles, 26 sources, 121 raw claims). + +--- + +## Normalization rules (applied to every cell) + +- **Absolute metrics** (pass@1, accuracy, F1, nDCG@10, faithfulness rate, + judge↔human agreement) ≈ success probability → mapped to σ with a modest + domain-shift discount (leaderboard conditions are friendlier than production). +- **Relative metrics** (Arena Elo, win-rates, preference rates) are **rankings, + not probabilities** — never inverted into σ. Used only to *order* models that + lack an absolute number, anchored to the scale set by absolute benchmarks. +- **Bands widen** when evidence is relative, stale, contaminated, or + verification-limited. Retrieval σ is corpus-dependent; judge catch-rate is the + softest column; self-correction *without external feedback* is unreliable. + +--- + +## GAP 1 — Frontier-model faithfulness (drafting / retrieval) + +Source: **Vectara HHEM-2.3 hallucination leaderboard** — summarization +factual-consistency over 7,700+ articles, temperature 0, "use only the passage." +Factual-consistency = 100% − hallucination rate. Explicitly RAG/agentic-relevant. +Updated 2026-05-11. + · + + +| Model | Hallucination % | Faithfulness σ (drafting/RAG) | Confidence | +|---|---|---|---| +| GPT-5.4-nano (2026-03-17) | 3.1% | 0.90 / 0.93 / 0.96 | CONFIRMED | +| Gemini-2.5-flash-lite | 3.3% | 0.90 / 0.93 / 0.96 | CONFIRMED | +| Phi-4 | 3.7% | 0.89 / 0.93 / 0.96 | SOURCED | +| Llama-3.3-70B-Instruct-Turbo | 4.1% | 0.88 / 0.92 / 0.96 | CONFIRMED | +| Qwen3-8B | 4.8% | 0.86 / 0.91 / 0.95 | SOURCED | +| GPT-4.1 | 5.6% | 0.85 / 0.90 / 0.94 | SOURCED | +| DeepSeek-V3 | 6.1% | 0.84 / 0.90 / 0.94 | CONFIRMED | +| DeepSeek-V3.2 | 6.3% | 0.84 / 0.90 / 0.94 | CONFIRMED | +| Gemini-2.5-pro | 7.0% | 0.82 / 0.89 / 0.94 | SOURCED | +| Claude Sonnet 4 | 10.3% | 0.78 / 0.86 / 0.92 | SOURCED | +| DeepSeek-R1 (reasoning) | 11.3% | 0.74 / 0.83 / 0.90 | CONFIRMED | +| Claude Opus 4 | 12.0% | 0.74 / 0.82 / 0.89 | SOURCED | +| Kimi-K2.5 | 14.2% | 0.70 / 0.80 / 0.88 | SOURCED | +| o4-Mini-High (reasoning) | 18.6% | 0.64 / 0.75 / 0.85 | CONFIRMED | +| o3-Pro (reasoning) | 23.3% | 0.58 / 0.70 / 0.82 | CONFIRMED | + +**Reasoning-vs-faithfulness tradeoff (CONFIRMED 3-0).** Reasoning variants +hallucinate ~2× their non-reasoning siblings (R1 11.3% vs V3 6.1%). A reasoning +model is a *worse* drafting/RAG faithfulness prior than its base sibling — model +the reasoning variant with a lower faithfulness σ even when it is "smarter." + +**Not found / no public benchmark.** No "Claude Fable" model and no exact +"GPT-5.5 / Opus 4.8 / Kimi 2.6" appears on a public faithfulness leaderboard as +of this sweep. Those names, if used in the cockpit, are **INTERPOLATED** from +the nearest benchmarked sibling and flagged in-cell — never invented numbers. + +--- + +## GAP 2 — Embedding / retrieval models (NOT LLMs) + +Retrieval is **embedding + vector search**, not an LLM generating text. The σ +here is "P(relevant item retrieved in top-k)", anchored to MTEB-v2 Retrieval / +BEIR nDCG@10. Leaderboard numbers are upper-bound-ish and **corpus-dependent** — +bands are wide. All SOURCED (verification rate-limited). + +| Embedder | Anchor | Retrieval σ | Source | +|---|---|---|---| +| Qwen3-Embedding-8B | MTEB-En v2 75.2; Retrieval 70.9 | 0.62 / 0.68 / 0.74 | arXiv:2506.05176 | +| Qwen3-Embedding-4B | MTEB-En v2 74.6 | 0.60 / 0.66 / 0.72 | arXiv:2506.05176 | +| Qwen3-Embedding-0.6B | MTEB-En v2 70.7 | 0.55 / 0.62 / 0.68 | arXiv:2506.05176 | +| OpenAI text-embedding-3-large | BEIR nDCG@10 55.4; MTEB-En 64.6 | 0.52 / 0.58 / 0.64 | arXiv:2407.19669 | +| OpenAI text-embedding-3-small | MTEB-En 62.3 | 0.48 / 0.55 / 0.62 | arXiv:2407.19669 | +| Cohere Embed v3 (multilingual) | MTEB-En 64.0 | 0.50 / 0.57 / 0.63 | arXiv:2407.19669 | +| E5-mistral-7b | MTEB-En 66.6 | 0.52 / 0.59 / 0.65 | arXiv:2407.19669 | +| BGE-M3 (dense) | BEIR 48.7; MIRACL 67.7 | 0.46 / 0.53 / 0.60 | arXiv:2407.19669 | +| mGTE-TRM (dense, 304M) | BEIR 51.1; MIRACL 62.1 | 0.47 / 0.54 / 0.61 | arXiv:2407.19669 | +| granite-embedding-english-r2 (149M) | BEIR 53.1; MTEB-v2 Ret. 56.4 | 0.48 / 0.55 / 0.62 | arXiv:2508.21085 | +| granite-embedding-small-r2 (47M) | BEIR 50.9 | 0.45 / 0.52 / 0.59 | arXiv:2508.21085 | + +Relative-only ordering (NOT invertible to σ): Qwen3-Embedding-8B > Gemini-Embedding +(70.6 vs 68.4 MTEB multilingual); NV-Embed-v2 56.3, GritLM-7B 60.9, gte-Qwen2-7B 62.5 trail. + +--- + +## GAP 3 — Rerankers (post-ranking) + +A reranker has **no standalone σ_raw** — it transforms a candidate list. We +record a *ranking-quality uplift* and a post-rerank "P(correct item in top-k)". +Two families: **cross-encoders** (pointwise) and **listwise LLM rerankers**. + +| Reranker | Family | Uplift (nDCG@10 / MTEB-R) | Post-rerank σ | Source | +|---|---|---|---|---| +| Qwen3-Reranker (4B/8B) | cross-encoder | MTEB-R 61.8 → ~69 (+7) | 0.65 / 0.71 / 0.76 | arXiv:2506.05176 | +| Qwen3-Reranker-0.6B | cross-encoder | MTEB-R 61.8 → 65.8 (+4) | 0.62 / 0.68 / 0.73 | arXiv:2506.05176 | +| granite-reranker-r2 (149M) | cross-encoder | BEIR 53.1 → 55.4 (+2.3) | 0.55 / 0.62 / 0.68 | arXiv:2508.21085 | +| BGE-reranker-v2-m3 | cross-encoder | (MTEB-R 57.0 baseline) | 0.55 / 0.61 / 0.67 | arXiv:2506.05176 | +| Jina-reranker-v2-multilingual | cross-encoder | (MTEB-R 58.2) | 0.55 / 0.61 / 0.67 | arXiv:2506.05176 | +| RankGPT (GPT-4, listwise) | LLM listwise | TREC-DL19 75.6 nDCG@10 | 0.66 / 0.72 / 0.78 | arXiv:2508.16757 | +| RankZephyr-7B (listwise) | LLM listwise | TREC-DL19 74.2; novel 62.7 | 0.62 / 0.69 / 0.75 | arXiv:2508.16757 | +| MonoT5-3B | cross-encoder | TREC-DL19 71.8; novel 60.8 | 0.60 / 0.67 / 0.73 | arXiv:2508.16757 | + +**Recency penalty (SOURCED).** On FutureQueryEval (post-April-2025 novel +queries) reranker nDCG@10 drops 5–15% vs standard benchmarks — leaderboard +numbers are optimistic; production bands should sit at the low end. + +--- + +## GAP 4 — Reviewers (LLM-as-judge) and correctors (fix-success) + +These are **two distinct roles**, and the cockpit now models them as such +(detection × fix-success, see [Methodology](priors.md)). + +### (a) Reviewer catch_rate (softest column) + +LLM-as-judge agreement with human preference. Source: RewardBench-2 +(arXiv:2604.13717), CriticGPT (arXiv:2407.00215), MT-Bench (arXiv:2306.05685). + +| Reviewer setup | Anchor | catch_rate σ | Source | +|---|---|---|---| +| Single LLM judge (k=1) | RewardBench-2 71.7% acc (N=1729) | 0.60 / 0.68 / 0.74 | arXiv:2604.13717 | +| Ensemble judge (k=8) | RewardBench-2 81.5% (+9.8pp) | 0.72 / 0.80 / 0.85 | arXiv:2604.13717 | +| Ensemble + scoring criteria | RewardBench-2 83.6% | 0.74 / 0.82 / 0.87 | arXiv:2604.13717 | +| CriticGPT (code review) | 63% preferred over human critiques | 0.55 / 0.65 / 0.75 | arXiv:2407.00215 | +| Human reviewer (domain expert) | — (paper assumption) | 0.75 / 0.85 / 0.92 | model card default | + +CriticGPT's "63% preferred" is a **relative win-rate over humans**, NOT an +absolute catch fraction — used as ranking evidence only (catch_rate is mid-high +for code-error detection), never inverted to σ. + +### (b) Corrector fix_rate + +P(repair succeeds | error was flagged). Source: "LLMs Cannot Self-Correct +Reasoning Yet" (arXiv:2310.01798), "When Can LLMs Actually Correct Their Own +Mistakes" (TACL 2024), Self-Refine (Madaan 2023), Reflexion (Shinn 2023). + +| Corrector setup | Finding | fix_rate σ | Source | +|---|---|---|---| +| Self-correction, NO external feedback | at/below baseline; can degrade | 0.10 / 0.30 / 0.55 | arXiv:2310.01798 | +| Correction WITH reviewer feedback | materially improves over baseline | 0.55 / 0.70 / 0.85 | TACL 2024 | +| Agentic re-do with test/oracle signal | high when signal is reliable | 0.65 / 0.78 / 0.90 | Reflexion 2023 | + +**Key result (SOURCED, multiple papers agree).** Intrinsic self-correction +without an external signal is unreliable — `fix_rate` collapses toward baseline. +This is exactly why the cockpit splits *detection* (reviewer) from *repair* +(corrector): a corrector with no reviewer feedback has a low `fix_rate`, and +`σ_corr = σ_raw + (1−σ_raw)·catch_rate·fix_rate` makes that visible. + +--- + +## GAP 5 — Modern RAG + agentic component taxonomy + +Anchors: Lewis et al. RAG (2020), ReAct (arXiv:2210.03629), Self-Refine, +Reflexion, Berkeley "Compound AI Systems" +(), agentic +surveys (arXiv:2404.11584, 2506.04565). The node types the cockpit should model: + +| Component | LLM-backed? | Quality metric | Bucket | +|---|---|---|---| +| Generator (draft/code/extract) | yes | pass@1 / faithfulness / F1 | generator | +| Query rewriter / expander | yes | retrieval uplift | generator | +| Embedder | **no** (vector model) | MTEB/BEIR nDCG@10 | retriever | +| Vector retriever | **no** (ANN search) | recall@k / nDCG@10 | retriever | +| Reranker | either | post-rerank nDCG@10 | reranker | +| Router / classifier / intent | yes | accuracy / F1 | router/classifier | +| Reviewer / judge | yes or human | catch_rate (detect) | reviewer/judge | +| Corrector / refiner | yes | fix_rate (repair) | corrector | +| Tool / function caller | yes | call success / BFCL | tool-caller | +| Planner / decomposer | yes | plan validity | planner | +| Memory | **no** (store) | retrieval accuracy | memory | +| Groundedness / citation verifier | yes | faithfulness | verifier | + +**The two corrections the cockpit needed:** (1) retrieval is an embedder + +vector search, *not* an LLM — an LLM only enters retrieval as a *reranker* +(post-ranking) or *generator* (reader); (2) reviewer ≠ corrector — detection and +repair are separate competences whose **product** governs corrected quality. diff --git a/docs/methodology/priors.md b/docs/methodology/priors.md new file mode 100644 index 0000000..d5c2b2e --- /dev/null +++ b/docs/methodology/priors.md @@ -0,0 +1,158 @@ +# How the priors are built + +When you have no traces yet, the cockpit still needs *starting numbers* for each +node. This page explains exactly how a public benchmark becomes the number a node +is seeded with — so you can judge it, not just trust it. Every prior is a +**hypothesis to refine with your own traces**, never ground truth. The raw +evidence and citations live in the [Priors Evidence Ledger](priors-evidence.md). + +We write the concept first and the symbol in parentheses, e.g. *raw competence +(σ_raw)*. You never need the symbols to use the cockpit; they're there if you +want to follow the math. + +--- + +## The two competences a node has + +A node does work, and that work may be reviewed and repaired. We track three +distinct probabilities, because conflating them is the most common governance +mistake: + +| Concept (symbol) | Plain meaning | Who owns it | +|---|---|---| +| **Raw competence (σ_raw)** | P(the node is right *on its own*, before any review) | the worker | +| **Reviewer catch rate (c)** | P(a reviewer *detects* an error that exists) | the reviewer | +| **Corrector fix rate (f)** | P(a flagged error is *actually repaired*) | the corrector | + +The **corrected quality (σ_corr)** the rest of the workflow sees is: + +$$\sigma_{corr} = \sigma_{raw} + (1 - \sigma_{raw}) \cdot c \cdot f$$ + +In words: start from what the worker gets right on its own, then recover the +errors that are *both* caught *and* fixed. The product **c · f** is the +**effective correction**. + +### Why detection and repair are separate + +A reviewer that flags every error but hands it to a corrector that can't fix it +(`f` ≈ 0) lifts quality by *nothing*. The published evidence is blunt about this: +intrinsic self-correction *without external feedback* is unreliable and sometimes +makes answers worse, while correction *with* a reliable reviewer signal helps +([evidence](priors-evidence.md#gap-4-reviewers-llm-as-judge-and-correctors-fix-success)). +So the cockpit models a **reviewer** (detection, `c`) and a **corrector** +(repair, `f`) as separate competences. The classic single-number model — +"the reviewer catches *and* perfectly fixes" — is just the special case **f = 1**. + +> **Lesson built into the framework:** *corrected* performance is not evidence of +> *autonomous* competence. A node can look healthy (high σ_corr) while its raw +> competence (σ_raw) is poor — the gap is hidden by review. We surface that gap as +> the **masking index (M\* = σ_corr / σ_raw)**. + +--- + +## From a benchmark number to a seed + +### Generators → seed the skill (σ_skill) + +For a node that *produces* work (drafting, code, extraction, classification, +retrieval, reranking) the prior is a **raw-competence band** σ_raw = (low / mid / +high). But a node in the model is parameterized by its underlying **skill +(σ_skill)**, which the *return operator* damps toward an equilibrium by the gain +**γ = η / (η + δ) = 10/12 ≈ 0.833** (observation rate η over observation + decay). +At that equilibrium, raw competence is γ · σ_skill. So to make a freshly-seeded +node *reproduce the prior's σ_raw*, we invert the relationship: + +$$\sigma_{skill} = \mathrm{clamp}\!\left(\frac{\sigma_{raw}^{mid}}{\gamma},\; 0.05,\; 0.98\right)$$ + +Then γ · σ_skill = σ_raw at the fixed point, by construction. (The clamp keeps the +seed inside a sane range; when it bites, the round-trip is approximate and the +cockpit says so.) + +### Reviewers → seed the catch rate (c); correctors → seed the fix rate (f) + +A **reviewer** node is seeded directly with the **catch rate (c)** from its prior +band; a **corrector** node with the **fix rate (f)**. No γ inversion — these are +already probabilities of an event (detect / repair), not competences subject to +the return-operator dynamics. + +### How wide is the band? → confidence + +Each prior is a band, and band width is a crude evidence-strength signal. The +cockpit shows a **confidence = 1 − (high − low)**: a tight band (strong, recent, +absolute evidence) reads as high confidence; a wide band (relative, stale, or +verification-limited evidence) reads as low. Confidence is a *humility meter*, not +a guarantee. + +--- + +## Turning benchmarks into σ: the normalization rules + +Not all benchmark numbers mean the same thing. We apply three rules, and record +which one was used in every cell's note. + +1. **Absolute metrics → σ directly (with a discount).** pass@1, accuracy, F1, + nDCG@10, faithfulness rate, and judge↔human agreement are already + "probability of being right"-shaped. We map them to σ with a modest + *domain-shift discount*, because leaderboard conditions are friendlier than + your production data. +2. **Relative metrics → ranking only, never σ.** Arena Elo, win-rates, and + preference rates order models; they are *not* success probabilities. We never + invert an Elo into a σ. Relative evidence is used only to *rank* models whose + scale is already pinned by an absolute benchmark, and any cell resting on it + carries a wider band. +3. **Bands widen with doubt.** Relative, stale, contaminated, or + verification-limited evidence → wider band. No public evidence → a deliberately + wide conservative band flagged **INTERPOLATED**, *never* an invented number. + +--- + +## The component taxonomy + +Modern RAG and agentic systems are *compound* — many specialized parts, not one +model. Two corrections matter most for governance: + +- **An LLM does not do retrieval.** Retrieval is an **embedder + vector search** — + not a language model generating text. An LLM's legitimate roles in a retrieval + pipeline are *reranking* (post-ranking candidates) and *grounded generation* + (the reader that writes an answer from already-retrieved context). +- **A reviewer is not a corrector.** Detection and repair are different jobs with + different success rates (see above). + +| Node type | Language model? | What it's scored on | Seeds | +|---|---|---|---| +| Generator (draft / code / extract) | yes | pass@1 / faithfulness / F1 | σ_skill | +| Classifier / router | yes | accuracy / F1 | σ_skill | +| **Embedder / retriever** | **no** — vector model + ANN search | MTEB / BEIR nDCG@10 | σ_skill | +| Reranker | cross-encoder *or* listwise LLM | post-rerank nDCG@10 | σ_skill | +| Grounded generation (RAG reader) | yes | faithfulness | σ_skill | +| **Model reviewer (LLM-as-judge)** | yes | catch rate (detection) | c | +| Human reviewer | no (person) | catch rate (detection) | c | +| **Corrector / refiner** | yes | fix rate (repair) | f | + +The retrieval, reranking, and grounded-generation distinctions all flow from the +[component-taxonomy evidence](priors-evidence.md#gap-5-modern-rag-agentic-component-taxonomy). + +--- + +## Honest caveats + +- **Judge catch rate is the softest column.** LLM-as-judge agreement with humans + is noisy and biased (position, verbosity, self-preference). Ensembling several + judge calls helps materially, but treat every catch-rate prior as a wide guess + until your traces say otherwise. +- **Retrieval σ is corpus-dependent.** MTEB/BEIR numbers are leaderboard + upper-bounds; your corpus and query distribution will differ. Recency hurts: + rerankers drop measurably on novel queries. +- **Reasoning ≠ faithful.** Reasoning model variants hallucinate *more* than their + base siblings on grounded tasks; a "smarter" model can be a *worse* drafting + prior. The priors encode this. +- **Self-correction without feedback is unreliable.** A corrector with no reviewer + signal has a low fix rate by design. +- **INTERPOLATED cells are not measurements.** Where a model has no public + benchmark (e.g. an unreleased name), its band is interpolated from the nearest + benchmarked sibling and flagged. Refine it from your own traces first. + +**The one-line method:** map a *cited* benchmark to a σ band under explicit +normalization rules, invert through the return-operator gain to seed the node, +keep detection and repair separate, and widen the band whenever the evidence is +weak — then let your traces overwrite all of it. diff --git a/mkdocs.yml b/mkdocs.yml index debdcb9..5f1f74b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -65,12 +65,16 @@ nav: - Autonomy Time: concepts/autonomy.md - Topology as Governance: concepts/topology.md - Guides: + - The Questions It Answers: guides/questions.md - Analyzing a Pipeline: guides/analyze.md - Interactive Companion: guides/interactive.md - LangGraph Integration: guides/langgraph.md - ADK Integration: guides/adk.md - Calibrating from Traces: guides/traces.md - Reading the Report: guides/interpret.md + - Methodology: + - How the Priors Are Built: methodology/priors.md + - Priors Evidence Ledger: methodology/priors-evidence.md - Paper: - Validation (8 Experiments): paper/validation.md - Equation-to-Code Reference: paper/references.md diff --git a/scripts/gen_priors_js.py b/scripts/gen_priors_js.py new file mode 100644 index 0000000..b1f7d0d --- /dev/null +++ b/scripts/gen_priors_js.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +"""Generate web/mso-priors.js from src/minimal_oversight/data/priors.yaml. + +The YAML is the single source of truth. This script loads it through the +package's own (validating) loader and emits the browser bundle, so the cockpit +seeder and the Python reference can never drift. The parity test +(tests/test_parity.py) pins seedNode JS ↔ Python to 1e-9 on top of this. + +Usage: python scripts/gen_priors_js.py [--check] + (no args) rewrite web/mso-priors.js + --check exit 1 if the file on disk differs from freshly-generated output +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) + +from minimal_oversight import priors as P # noqa: E402 + +OUT = REPO / "web" / "mso-priors.js" + + +def _num(x: float) -> str: + """Compact JS number literal (shortest round-trip, matches JSON.stringify).""" + return json.dumps(round(float(x), 6)) + + +def _band_for(cell) -> tuple[str, object]: + """Return (kind, band) — which payload this cell seeds from.""" + if cell.task_type == P.REVIEW_TASK: + return "review", cell.catch_rate + if cell.task_type == P.CORRECTION_TASK: + return "correction", cell.fix_rate + return "generator", cell.sigma_raw + + +def build() -> str: + data = P.load_priors() + cells = data["cells"] + models = sorted({m for (m, _t) in cells}) + task_types = list(data["task_types"]) + + lines: list[str] = [] + for (model, task) in cells: + cell = cells[(model, task)] + kind, band = _band_for(cell) + obj = ( + "{kind:" + json.dumps(kind) + + ",band:{low:" + _num(band.low) + + ",mid:" + _num(band.mid) + + ",high:" + _num(band.high) + "}" + + ",benchmark:" + json.dumps(cell.primary_benchmark or "") + + ",metric_kind:" + json.dumps(cell.metric_kind or "") + + ",note:" + json.dumps(cell.normalization_note or "") + + "}" + ) + lines.append(' ' + json.dumps(model + "|" + task) + ": " + obj + ",") + cells_block = "\n".join(lines) + + return f'''/** + * mso-priors.js — cold-start σ_raw / catch_rate / fix_rate priors for cockpit nodes. + * + * GENERATED FROM src/minimal_oversight/data/priors.yaml — DO NOT EDIT BY HAND. + * Regenerate with: python scripts/gen_priors_js.py + * + * Mirrors minimal_oversight.priors.seed_node (Python) exactly: + * - generator / retrieval / reranking task-types → sigma_skill = clamp(mid/γ, 0.05, 0.98) + * so gamma * sigma_skill == band.mid at the return-operator fixed point. + * - review task-type → catch_rate = clamp(mid, 0, 1) (reviewer error-detection) + * - correction task-type → fix_rate = clamp(mid, 0, 1) (corrector repair-success) + * - provenance.confidence = 1 − band_width (a crude evidence-strength proxy). + * + * Provenance for every band: docs/methodology/priors-evidence.md. + * Parity: tests/test_parity.py pins JS ↔ Python to within 1e-9 on seedNode outputs. + */ +(function (root, factory) {{ + if (typeof module === "object" && module.exports) module.exports = factory(); + else root.MSO_Priors = factory(); +}})(typeof self !== "undefined" ? self : this, function () {{ + "use strict"; + + var GAMMA = 10 / 12; // η/(η+δ) return-operator fixed-point gain + + function clamp(x, a, b) {{ return Math.max(a, Math.min(b, x)); }} + + // ---- bundled prior table (generated from data/priors.yaml) ----------------- + var CELLS = {{ +{cells_block} + }}; + + var MODELS = {json.dumps(models)}; + var TASK_TYPES = {json.dumps(task_types)}; + + // ---- public API ------------------------------------------------------------ + + /** List all model names in the table. */ + function listModels() {{ return MODELS.slice(); }} + + /** List all task-type names in the table. */ + function listTaskTypes() {{ return TASK_TYPES.slice(); }} + + /** List the task-types that have a prior for the given model. */ + function tasksForModel(model) {{ + return TASK_TYPES.filter(function (t) {{ return !!CELLS[model + "|" + t]; }}); + }} + + /** List the models that have a prior for the given task-type. */ + function modelsForTask(taskType) {{ + return MODELS.filter(function (m) {{ return !!CELLS[m + "|" + taskType]; }}); + }} + + /** Return true if a prior exists for the (model, taskType) pair. */ + function hasCell(model, taskType) {{ return !!(CELLS[model + "|" + taskType]); }} + + /** + * Seed a cockpit Node from a (model, taskType) prior. + * + * Returns an object ready to be merged onto a node: + * {{ model, task_type, is_prior: true, seeds, sigma_skill?, catch_rate?, fix_rate?, provenance }} + * + * Mirrors Python priors.seed_node exactly. Throws if no cell exists. + */ + function seedNode(model, taskType) {{ + var cell = CELLS[model + "|" + taskType]; + if (!cell) throw new Error("no prior for model=" + model + " task_type=" + taskType); + + var b = cell.band; + var out = {{ model: model, task_type: taskType, is_prior: true }}; + + if (taskType === "review") {{ + out.catch_rate = clamp(b.mid, 0, 1); + out.seeds = "catch_rate"; + }} else if (taskType === "correction") {{ + out.fix_rate = clamp(b.mid, 0, 1); + out.seeds = "fix_rate"; + }} else {{ + out.sigma_skill = clamp(b.mid / GAMMA, 0.05, 0.98); + out.catch_rate = 0.0; + out.seeds = "sigma_skill"; + }} + + out.provenance = {{ + band: {{ low: b.low, mid: b.mid, high: b.high }}, + confidence: Math.round((1 - (b.high - b.low)) * 1000) / 1000, + benchmark: cell.benchmark, + metric_kind: cell.metric_kind, + note: cell.note + }}; + return out; + }} + + return {{ + listModels: listModels, listTaskTypes: listTaskTypes, + tasksForModel: tasksForModel, modelsForTask: modelsForTask, + hasCell: hasCell, seedNode: seedNode, GAMMA: GAMMA + }}; +}}); +''' + + +def main() -> int: + generated = build() + if "--check" in sys.argv: + current = OUT.read_text() if OUT.exists() else "" + if current != generated: + print(f"DRIFT: {OUT} is stale. Run: python scripts/gen_priors_js.py") + return 1 + print(f"OK: {OUT} matches priors.yaml") + return 0 + OUT.write_text(generated) + n = generated.count('": {kind:') + print(f"wrote {OUT} ({n} cells)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/parity_runner.js b/scripts/parity_runner.js index 5dda02c..616be41 100644 --- a/scripts/parity_runner.js +++ b/scripts/parity_runner.js @@ -20,7 +20,7 @@ const out = {}; out.fisher = cases.fisher.map((s) => MSO.fisherInformation(s)); out.volume = cases.volume.map((s) => MSO.fisherVolumeElement(s)); out.sraw_fp = cases.sraw_fp.map((a) => MSO.sigmaRawFixedPoint(a[0], a[1], a[2], a[3])); -out.scorr = cases.scorr.map((a) => MSO.sigmaCorrFixedPoint(a[0], a[1])); +out.scorr = cases.scorr.map((a) => MSO.sigmaCorrFixedPoint(a[0], a[1], a[2])); out.masking = cases.masking.map((a) => MSO.maskingIndex(a[0], a[1])); out.estimate = (cases.estimate || []).map((c) => { const e = MSOEstimate.estimate(c.raw, c.corr); @@ -66,7 +66,8 @@ out.seed_node = (cases.seed_node || []).map((c) => { return { seeds: s.seeds, sigma_skill: s.sigma_skill != null ? s.sigma_skill : null, - catch_rate: s.catch_rate, + catch_rate: s.catch_rate != null ? s.catch_rate : null, + fix_rate: s.fix_rate != null ? s.fix_rate : null, confidence: p.confidence, band_low: p.band.low, band_mid: p.band.mid, diff --git a/src/minimal_oversight/_formulae.py b/src/minimal_oversight/_formulae.py index 02385c8..6ca9953 100644 --- a/src/minimal_oversight/_formulae.py +++ b/src/minimal_oversight/_formulae.py @@ -76,16 +76,33 @@ def sigma_raw_fixed_point( def sigma_corr_fixed_point( sigma_raw_star: float, catch_rate: float, + fix_rate: float = 1.0, ) -> float: - """Fixed-point corrected quality: σ*_corr = σ*_raw + (1 − σ*_raw) × c. + """Fixed-point corrected quality: σ*_corr = σ*_raw + (1 − σ*_raw) × c_eff. - Ref: Equation 6. + The effective correction is decomposed into two independent stages: + + c_eff = catch_rate × fix_rate + + where ``catch_rate`` (*c*) is the **reviewer's** probability of *detecting* + an error, and ``fix_rate`` (*f*) is the **corrector's** probability of + *successfully repairing* an error that was flagged. The paper's single-number + correction model is the special case ``fix_rate = 1`` — i.e. every caught + error is perfectly fixed. Separating detection from repair lets a workflow + model a reviewer (human or LLM-as-judge) that only produces a verdict, paired + with a distinct corrector that consumes that verdict and re-does or patches + the work. + + Ref: Equation 6 (with the detection × fix-success generalization). Args: sigma_raw_star: Fixed-point raw competence. - catch_rate: Corrector catch probability *c*. + catch_rate: Reviewer error-detection probability *c*. + fix_rate: Corrector repair-success probability *f* given a flagged error. + Defaults to 1.0 (reviewer's catch is perfectly corrected). """ - return sigma_raw_star + (1.0 - sigma_raw_star) * catch_rate + c_eff = catch_rate * fix_rate + return sigma_raw_star + (1.0 - sigma_raw_star) * c_eff def masking_index(sigma_corr: float, sigma_raw: float) -> float: @@ -239,6 +256,7 @@ def recursive_chain_quality( eta: float, delta: float, sigma_0: float = 0.0, + fix_rate: float = 1.0, ) -> float: """Recursive chain quality C_op(D) for a linear chain of identical layers. @@ -252,7 +270,9 @@ def recursive_chain_quality( sigma_raw_star = sigma_raw_fixed_point( sigma_skill_eff, eta, delta, sigma_0=sigma_0 ) - sigma_corr_prev = sigma_corr_fixed_point(sigma_raw_star, catch_rate) + sigma_corr_prev = sigma_corr_fixed_point( + sigma_raw_star, catch_rate, fix_rate + ) return sigma_corr_prev @@ -345,6 +365,7 @@ def max_pipeline_depth( delta: float = 2.0, sigma_0: float = 0.0, max_depth: int = 100, + fix_rate: float = 1.0, ) -> float: """Maximum target-feasible depth under the recursive chain model. @@ -362,7 +383,8 @@ def max_pipeline_depth( d_max = 0 for depth in range(1, max_depth + 1): quality = recursive_chain_quality( - depth, sigma_skill, catch_rate, eta, delta, sigma_0=sigma_0 + depth, sigma_skill, catch_rate, eta, delta, + sigma_0=sigma_0, fix_rate=fix_rate, ) if quality >= p_min: d_max = depth @@ -371,7 +393,8 @@ def max_pipeline_depth( if d_max == max_depth: next_quality = recursive_chain_quality( - max_depth + 1, sigma_skill, catch_rate, eta, delta, sigma_0=sigma_0 + max_depth + 1, sigma_skill, catch_rate, eta, delta, + sigma_0=sigma_0, fix_rate=fix_rate, ) if next_quality >= p_min: return float("inf") @@ -386,19 +409,25 @@ def corrector_capacity_threshold( p_min: float, sigma_raw_star: float, catch_rate: float, + fix_rate: float = 1.0, ) -> float: """Minimum review fraction K/N for a target to be feasible. - K/N >= max(0, (p_min − σ*_raw) / [(1 − σ*_raw) × c]) + K/N >= max(0, (p_min − σ*_raw) / [(1 − σ*_raw) × c_eff]) + + where ``c_eff = catch_rate × fix_rate``. A corrector that only repairs a + fraction of the errors its reviewer flags (``fix_rate < 1``) needs a + proportionally larger review fraction to hit the same target. Ref: Section 1, Euler-Lagrange Solution. """ sigma_star = float(np.clip(sigma_raw_star, 0.0, 1.0)) if p_min <= sigma_star: return 0.0 - if catch_rate <= 0 or (1 - sigma_star) <= 0: + c_eff = catch_rate * fix_rate + if c_eff <= 0 or (1 - sigma_star) <= 0: return float("inf") - threshold = (p_min - sigma_star) / ((1 - sigma_star) * catch_rate) + threshold = (p_min - sigma_star) / ((1 - sigma_star) * c_eff) return max(0.0, threshold) diff --git a/src/minimal_oversight/capacity.py b/src/minimal_oversight/capacity.py index 3f19a99..3b605d2 100644 --- a/src/minimal_oversight/capacity.py +++ b/src/minimal_oversight/capacity.py @@ -50,8 +50,9 @@ def compute_node_capacity( """ sigma_skill = node.sigma_skill if node.sigma_skill is not None else 0.55 catch_rate = node.catch_rate if node.catch_rate is not None else 0.65 + fix_rate = node.fix_rate if node.fix_rate is not None else 1.0 sigma_raw_star = F.sigma_raw_fixed_point(sigma_skill, eta, delta, sigma_0=sigma_0) - return F.sigma_corr_fixed_point(sigma_raw_star, catch_rate) + return F.sigma_corr_fixed_point(sigma_raw_star, catch_rate, fix_rate) def compute_pipeline_capacity( @@ -74,6 +75,7 @@ def compute_pipeline_capacity( node = pipeline.get_node(name) sigma_skill = node.sigma_skill if node.sigma_skill is not None else 0.55 catch_rate = node.catch_rate if node.catch_rate is not None else 0.65 + fix_rate = node.fix_rate if node.fix_rate is not None else 1.0 # Effective skill incorporates parent corrected qualities parents = pipeline.parents(name) @@ -87,7 +89,9 @@ def compute_pipeline_capacity( sigma_raw_star = F.sigma_raw_fixed_point( sigma_skill_eff, eta, delta, sigma_0=sigma_0 ) - sigma_corr_star = F.sigma_corr_fixed_point(sigma_raw_star, catch_rate) + sigma_corr_star = F.sigma_corr_fixed_point( + sigma_raw_star, catch_rate, fix_rate + ) node_sigma_corr[name] = sigma_corr_star capacities[name] = sigma_corr_star diff --git a/src/minimal_oversight/data/priors.yaml b/src/minimal_oversight/data/priors.yaml index 25d4bb2..4d0ab62 100644 --- a/src/minimal_oversight/data/priors.yaml +++ b/src/minimal_oversight/data/priors.yaml @@ -31,19 +31,23 @@ # FaithJudge: https://arxiv.org/abs/2505.04847 meta: - schema_version: 1 + schema_version: 2 # v2: + grounded_generation/retrieval/reranking/correction tasks gamma: 0.8333333333333334 # return-operator fixed-point gain eta/(eta+delta)=10/12 - status: curated # placeholder -> curated after research pass (2026-06-17) - observed_date: "2026-06-17" # date this table was compiled; check for staleness + status: curated # placeholder -> curated after research pass (2026-06-17/18) + observed_date: "2026-06-18" # date this table was compiled; check for staleness generator_default_catch_rate: 0.0 # a generator does not correct its parent + evidence_ledger: docs/methodology/priors-evidence.md # provenance + confidence tags task_types: - code_generation - classification - extraction - drafting - - retrieval - - review # catch-rate proxy + - grounded_generation # RAG reader: LLM generating a grounded answer (faithfulness) + - retrieval # embedder + vector search (NOT an LLM) — P(relevant item in top-k) + - reranking # post-ranking of candidates (cross-encoder or listwise LLM) + - review # reviewer error-DETECTION (catch_rate); human or LLM-as-judge + - correction # corrector error-REPAIR (fix_rate); consumes reviewer feedback # ---- CODE GENERATION ---------------------------------------------------------- # Anchor: LiveBench Coding (contamination-limited, ground-truth scored, no LLM @@ -704,7 +708,11 @@ priors: source_url: https://github.com/vectara/hallucination-leaderboard normalization_note: interpolated; mid tier faithfulness; wide band -# ---- RETRIEVAL / RAG-ANSWER CORRECTNESS ---------------------------------------- +# ---- GROUNDED GENERATION / RAG READER (LLM faithfulness) ---------------------- +# IMPORTANT: an LLM does NOT do retrieval. Retrieval is embedding + vector search +# (see the RETRIEVAL section below, populated by embedder models). What an LLM +# does in a RAG pipeline is GROUNDED GENERATION — the "reader" that writes an +# answer from already-retrieved context. These cells score that reader role. # Proxy: Vectara HHEM (grounded-summarization faithfulness) + FaithBench/RAGTruth. # NOTE: this measures FAITHFULNESS (groundedness), not full end-to-end RAG answer- # correctness. The two can diverge — DeepSeek-R1 scores low on faithfulness but @@ -715,7 +723,7 @@ priors: # https://arxiv.org/abs/2505.04847 (FaithBench Table 7) - model: claude-opus-4 - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.83, mid: 0.89, high: 0.94 } primary_benchmark: Vectara-HHEM + FaithBench/RAGTruth metric: factual-consistency / faithfulness rate (PROXY for RAG correctness) @@ -727,7 +735,7 @@ priors: normalization_note: faithfulness PROXY — not full RAG answer-correctness; interpolated above Sonnet anchor - model: claude-sonnet-4 - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.80, mid: 0.87, high: 0.93 } primary_benchmark: FaithBench/RAGTruth metric: factual-consistency rate (faithfulness proxy for RAG) @@ -739,7 +747,7 @@ priors: normalization_note: direct FaithBench measurement; faithfulness PROXY only; Sonnet-4.5 confirmed >10% hallu — use low end of band - model: claude-haiku-4 - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.75, mid: 0.83, high: 0.90 } primary_benchmark: Vectara-HHEM (interpolated) metric: factual-consistency rate (faithfulness proxy) @@ -751,7 +759,7 @@ priors: normalization_note: interpolated; small tier; faithfulness proxy only; refine from traces - model: gpt-4o - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.80, mid: 0.84, high: 0.90 } primary_benchmark: FaithBench/RAGTruth metric: factual-consistency rate (faithfulness proxy for RAG) @@ -763,7 +771,7 @@ priors: normalization_note: direct FaithBench measurement; faithfulness PROXY only - model: gpt-4o-mini - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.72, mid: 0.80, high: 0.88 } primary_benchmark: Vectara-HHEM (interpolated) metric: factual-consistency rate (faithfulness proxy) @@ -775,7 +783,7 @@ priors: normalization_note: interpolated; small tier; faithfulness proxy only - model: openai-o3 - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.80, mid: 0.87, high: 0.93 } primary_benchmark: Vectara next-gen (interpolated) metric: factual-consistency rate (faithfulness proxy) @@ -787,7 +795,7 @@ priors: normalization_note: reasoning models WORSE on faithfulness than Flash; wide band; faithfulness proxy only - model: gemini-2-pro - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.88, mid: 0.93, high: 0.97 } primary_benchmark: FaithBench/RAGTruth metric: factual-consistency rate (faithfulness proxy for RAG) @@ -799,7 +807,7 @@ priors: normalization_note: best faithfulness in FaithBench set; faithfulness PROXY only; note HHEM conflict-of-interest - model: gemini-2-flash - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.88, mid: 0.93, high: 0.97 } primary_benchmark: Vectara-HHEM + FaithBench metric: factual-consistency rate (faithfulness proxy for RAG) @@ -811,7 +819,7 @@ priors: normalization_note: Flash-class tops HHEM; FaithBench confirms strong faithfulness; PROXY only - model: llama-3.3-70b - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.78, mid: 0.84, high: 0.90 } primary_benchmark: FaithBench/RAGTruth metric: factual-consistency rate (faithfulness proxy for RAG) @@ -823,7 +831,7 @@ priors: normalization_note: direct FaithBench measurement; faithfulness PROXY only; 70B tier - model: deepseek-v3 - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.88, mid: 0.93, high: 0.96 } primary_benchmark: Vectara-HHEM + DeepSeek blog metric: factual-consistency rate (faithfulness proxy for RAG) @@ -835,7 +843,7 @@ priors: normalization_note: V3 outstanding faithfulness; faithfulness PROXY only; does not predict end-to-end RAG correctness - model: deepseek-r1 - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.78, mid: 0.86, high: 0.91 } primary_benchmark: Vectara-HHEM + DeepSeek blog metric: factual-consistency rate (faithfulness proxy for RAG) @@ -847,7 +855,7 @@ priors: normalization_note: REASONING PENALTY on faithfulness; R1+RAG can hit 86% clinical accuracy despite poor faithfulness — proxy gap CRITICAL - model: qwen2.5-72b - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.75, mid: 0.85, high: 0.92 } primary_benchmark: Vectara-HHEM (interpolated) metric: factual-consistency rate (faithfulness proxy) @@ -859,7 +867,7 @@ priors: normalization_note: interpolated; mid tier faithfulness proxy; wide band - model: mistral-large - task_type: retrieval + task_type: grounded_generation sigma_raw: { low: 0.75, mid: 0.85, high: 0.92 } primary_benchmark: Vectara-HHEM (interpolated) metric: factual-consistency rate (faithfulness proxy) @@ -1043,3 +1051,519 @@ priors: observed_date: null source_url: null normalization_note: SOFTEST COLUMN; conservative wide-band fallback per research guidance; no reliable evidence + +# =============================================================================== +# v2 ADDITIONS (2026-06-18 research sweep). Provenance + confidence tags: +# docs/methodology/priors-evidence.md. +# =============================================================================== + +# ---- NEW FRONTIER MODELS · faithfulness (drafting + grounded_generation) ------ +# Source: Vectara HHEM-2.3 leaderboard (updated 2026-05-11), summarization +# factual-consistency = 100% − hallucination%. CONFIRMED (3-0) for the reasoning +# penalty and the top tier. faithfulness σ ≈ consistency, modest domain discount. +# https://github.com/vectara/hallucination-leaderboard + + - model: gpt-5.4-nano + task_type: drafting + sigma_raw: { low: 0.90, mid: 0.93, high: 0.96 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "3.1% hallucination (96.9% consistency)" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: CONFIRMED top-tier faithfulness; faithfulness subdim only; style/IF quality is relative-only and NOT captured + - model: gpt-5.4-nano + task_type: grounded_generation + sigma_raw: { low: 0.90, mid: 0.93, high: 0.96 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY for RAG reader) + metric_kind: absolute + reported_value: "3.1% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: RAG reader faithfulness proxy; retrieval STEP is an embedder (see retrieval task) + + - model: gemini-2.5-flash-lite + task_type: drafting + sigma_raw: { low: 0.90, mid: 0.93, high: 0.96 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "3.3% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: CONFIRMED top-tier faithfulness; faithfulness subdim only + - model: gemini-2.5-flash-lite + task_type: grounded_generation + sigma_raw: { low: 0.90, mid: 0.93, high: 0.96 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY) + metric_kind: absolute + reported_value: "3.3% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: RAG reader faithfulness proxy + + - model: gemini-2.5-pro + task_type: drafting + sigma_raw: { low: 0.82, mid: 0.89, high: 0.94 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "7.0% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: SOURCED (verification rate-limited); faithfulness subdim only; wider band + - model: gemini-2.5-pro + task_type: grounded_generation + sigma_raw: { low: 0.82, mid: 0.89, high: 0.94 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY) + metric_kind: absolute + reported_value: "7.0% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: RAG reader faithfulness proxy; SOURCED + + - model: phi-4 + task_type: drafting + sigma_raw: { low: 0.89, mid: 0.93, high: 0.96 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "3.7% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: SOURCED; small-model strong faithfulness; faithfulness subdim only + - model: phi-4 + task_type: grounded_generation + sigma_raw: { low: 0.89, mid: 0.93, high: 0.96 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY) + metric_kind: absolute + reported_value: "3.7% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: RAG reader faithfulness proxy; SOURCED + + - model: qwen3-8b + task_type: drafting + sigma_raw: { low: 0.86, mid: 0.91, high: 0.95 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "4.8% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: SOURCED; faithfulness subdim only + - model: qwen3-8b + task_type: grounded_generation + sigma_raw: { low: 0.86, mid: 0.91, high: 0.95 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY) + metric_kind: absolute + reported_value: "4.8% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: RAG reader faithfulness proxy; SOURCED + + - model: kimi-k2.5 + task_type: drafting + sigma_raw: { low: 0.70, mid: 0.80, high: 0.88 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "14.2% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: SOURCED; mid-low faithfulness; faithfulness subdim only; wide band + - model: kimi-k2.5 + task_type: grounded_generation + sigma_raw: { low: 0.70, mid: 0.80, high: 0.88 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY) + metric_kind: absolute + reported_value: "14.2% hallucination" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: RAG reader faithfulness proxy; SOURCED + + - model: o3-pro + task_type: drafting + sigma_raw: { low: 0.58, mid: 0.70, high: 0.82 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "23.3% hallucination (reasoning model — among worst)" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: CONFIRMED reasoning-vs-faithfulness penalty; a reasoning model is a WORSE drafting prior than its base sibling; wide band + - model: o3-pro + task_type: grounded_generation + sigma_raw: { low: 0.58, mid: 0.70, high: 0.82 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY) + metric_kind: absolute + reported_value: "23.3% hallucination (reasoning model)" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: CONFIRMED reasoning penalty; reader faithfulness proxy; wide band + + - model: o4-mini-high + task_type: drafting + sigma_raw: { low: 0.64, mid: 0.75, high: 0.85 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness) + metric_kind: absolute + reported_value: "18.6% hallucination (reasoning model)" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: CONFIRMED reasoning penalty; faithfulness subdim only; wide band + - model: o4-mini-high + task_type: grounded_generation + sigma_raw: { low: 0.64, mid: 0.75, high: 0.85 } + primary_benchmark: Vectara-HHEM-2.3 + metric: factual-consistency rate (faithfulness PROXY) + metric_kind: absolute + reported_value: "18.6% hallucination (reasoning model)" + sample_size: "7700+ docs" + observed_date: "2026-05" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: CONFIRMED reasoning penalty; reader faithfulness proxy; wide band + +# ---- USER-NAMED MODELS WITHOUT A PUBLIC BENCHMARK (INTERPOLATED, flagged) ------ +# These exact names did not appear on a public faithfulness leaderboard as of the +# 2026-06-18 sweep. Bands are interpolated from the nearest benchmarked sibling +# and flagged — NOT invented measurements. Refine from your own traces first. + + - model: gpt-5.5 + task_type: drafting + sigma_raw: { low: 0.86, mid: 0.92, high: 0.96 } + primary_benchmark: NONE (interpolated from gpt-5.4-nano) + metric: factual-consistency rate (faithfulness) + metric_kind: relative + reported_value: "no public benchmark for exact name; sibling gpt-5.4-nano 3.1%" + sample_size: null + observed_date: "2026-06" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: INTERPOLATED from gpt-5.4-nano tier; no public benchmark for this exact name; wide band; refine from traces + - model: gpt-5.5 + task_type: grounded_generation + sigma_raw: { low: 0.86, mid: 0.92, high: 0.96 } + primary_benchmark: NONE (interpolated from gpt-5.4-nano) + metric: faithfulness PROXY + metric_kind: relative + reported_value: "no public benchmark for exact name" + sample_size: null + observed_date: "2026-06" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: INTERPOLATED; reader faithfulness proxy; no public benchmark for exact name + + - model: claude-opus-4.8 + task_type: drafting + sigma_raw: { low: 0.78, mid: 0.86, high: 0.93 } + primary_benchmark: NONE (interpolated from claude-opus-4) + metric: factual-consistency rate (faithfulness) + metric_kind: relative + reported_value: "no public benchmark for exact name; sibling Claude Opus 4 ~12% hallu" + sample_size: null + observed_date: "2026-06" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: INTERPOLATED from Claude Opus 4 (modest gen-over-gen lift assumed); no public benchmark for exact name; wide band + - model: claude-opus-4.8 + task_type: grounded_generation + sigma_raw: { low: 0.78, mid: 0.86, high: 0.93 } + primary_benchmark: NONE (interpolated from claude-opus-4) + metric: faithfulness PROXY + metric_kind: relative + reported_value: "no public benchmark for exact name" + sample_size: null + observed_date: "2026-06" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: INTERPOLATED; reader faithfulness proxy; no public benchmark for exact name + + - model: kimi-2.6 + task_type: drafting + sigma_raw: { low: 0.70, mid: 0.81, high: 0.89 } + primary_benchmark: NONE (interpolated from kimi-k2.5) + metric: factual-consistency rate (faithfulness) + metric_kind: relative + reported_value: "no public benchmark for exact name; sibling Kimi-K2.5 14.2% hallu" + sample_size: null + observed_date: "2026-06" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: INTERPOLATED from Kimi-K2.5; no public benchmark for exact name; wide band + - model: kimi-2.6 + task_type: grounded_generation + sigma_raw: { low: 0.70, mid: 0.81, high: 0.89 } + primary_benchmark: NONE (interpolated from kimi-k2.5) + metric: faithfulness PROXY + metric_kind: relative + reported_value: "no public benchmark for exact name" + sample_size: null + observed_date: "2026-06" + source_url: https://github.com/vectara/hallucination-leaderboard + normalization_note: INTERPOLATED; reader faithfulness proxy; no public benchmark for exact name + + - model: fable-5 + task_type: drafting + sigma_raw: { low: 0.65, mid: 0.80, high: 0.92 } + primary_benchmark: NONE (no public evidence) + metric: factual-consistency rate (faithfulness) + metric_kind: relative + reported_value: "no reliable public evidence — model not on any public leaderboard" + sample_size: null + observed_date: null + source_url: null + normalization_note: NO PUBLIC BENCHMARK FOUND; deliberately wide conservative band; this is a placeholder to refine from your own traces, not a measurement + - model: fable-5 + task_type: grounded_generation + sigma_raw: { low: 0.65, mid: 0.80, high: 0.92 } + primary_benchmark: NONE (no public evidence) + metric: faithfulness PROXY + metric_kind: relative + reported_value: "no reliable public evidence" + sample_size: null + observed_date: null + source_url: null + normalization_note: NO PUBLIC BENCHMARK FOUND; wide conservative band; refine from traces + +# ---- RETRIEVAL · EMBEDDING MODELS (NOT LLMs) ---------------------------------- +# Retrieval is embedding + vector search. σ here ≈ P(relevant item in top-k), +# anchored to MTEB-v2 Retrieval / BEIR nDCG@10. Leaderboard numbers are upper- +# bound-ish and CORPUS-DEPENDENT → wide bands. SOURCED (verification rate-limited). +# Qwen3-Embedding arXiv:2506.05176 · mGTE/BGE/OpenAI arXiv:2407.19669 · Granite arXiv:2508.21085 + + - model: qwen3-embedding-8b + task_type: retrieval + sigma_raw: { low: 0.62, mid: 0.68, high: 0.74 } + primary_benchmark: MTEB-English-v2 (Retrieval) + metric: nDCG@10 / Mean(Task) + metric_kind: absolute + reported_value: "MTEB-En v2 75.2; Retrieval 70.9" + sample_size: "MTEB-v2 task suite" + observed_date: "2025-06" + source_url: https://arxiv.org/abs/2506.05176 + normalization_note: retrieval σ ≈ P(relevant in top-k); leaderboard upper-bound, corpus-dependent; SOURCED + - model: openai-text-embedding-3-large + task_type: retrieval + sigma_raw: { low: 0.52, mid: 0.58, high: 0.64 } + primary_benchmark: BEIR / MTEB-English + metric: nDCG@10 + metric_kind: absolute + reported_value: "BEIR nDCG@10 55.4; MTEB-En 64.6" + sample_size: "BEIR 13-task avg" + observed_date: "2024-07" + source_url: https://arxiv.org/abs/2407.19669 + normalization_note: retrieval σ from BEIR nDCG@10; corpus-dependent; SOURCED + - model: openai-text-embedding-3-small + task_type: retrieval + sigma_raw: { low: 0.48, mid: 0.55, high: 0.62 } + primary_benchmark: MTEB-English + metric: MTEB-En average + metric_kind: absolute + reported_value: "MTEB-En 62.3" + sample_size: "MTEB suite" + observed_date: "2024-07" + source_url: https://arxiv.org/abs/2407.19669 + normalization_note: cheaper/smaller embedder; corpus-dependent; SOURCED + - model: cohere-embed-v3 + task_type: retrieval + sigma_raw: { low: 0.50, mid: 0.57, high: 0.63 } + primary_benchmark: MTEB-English + metric: MTEB-En average + metric_kind: absolute + reported_value: "MTEB-En 64.0 (multilingual-v3.0)" + sample_size: "MTEB suite" + observed_date: "2024-07" + source_url: https://arxiv.org/abs/2407.19669 + normalization_note: corpus-dependent; SOURCED + - model: e5-mistral-7b + task_type: retrieval + sigma_raw: { low: 0.52, mid: 0.59, high: 0.65 } + primary_benchmark: MTEB-English + metric: MTEB-En average + metric_kind: absolute + reported_value: "MTEB-En 66.6" + sample_size: "MTEB suite" + observed_date: "2024-07" + source_url: https://arxiv.org/abs/2407.19669 + normalization_note: 7B LLM-initialized embedder; corpus-dependent; SOURCED + - model: bge-m3 + task_type: retrieval + sigma_raw: { low: 0.46, mid: 0.53, high: 0.60 } + primary_benchmark: BEIR / MIRACL + metric: nDCG@10 + metric_kind: absolute + reported_value: "BEIR 48.7; MIRACL 67.7 (dense)" + sample_size: "BEIR / MIRACL" + observed_date: "2024-07" + source_url: https://arxiv.org/abs/2407.19669 + normalization_note: strong open multilingual embedder; corpus-dependent; SOURCED + - model: granite-embedding-r2 + task_type: retrieval + sigma_raw: { low: 0.48, mid: 0.55, high: 0.62 } + primary_benchmark: BEIR / MTEB-v2 Retrieval + metric: nDCG@10 + metric_kind: absolute + reported_value: "BEIR 53.1; MTEB-v2 Retrieval 56.4 (149M)" + sample_size: "BEIR 13-task" + observed_date: "2025-08" + source_url: https://arxiv.org/abs/2508.21085 + normalization_note: compact ModernBERT embedder; corpus-dependent; SOURCED + +# ---- RERANKING · POST-RANKING (cross-encoder or listwise LLM) ------------------ +# A reranker transforms a candidate list. σ ≈ P(correct item in top-k AFTER rerank). +# Recency penalty: nDCG@10 drops 5-15% on novel queries (FutureQueryEval) → low end. +# Qwen3/Granite rerankers arXiv:2506.05176 / 2508.21085 · listwise LLMs arXiv:2508.16757 + + - model: qwen3-reranker + task_type: reranking + sigma_raw: { low: 0.65, mid: 0.71, high: 0.76 } + primary_benchmark: MTEB-R (over Qwen3-Embedding) + metric: nDCG@10 / MTEB-R uplift + metric_kind: absolute + reported_value: "MTEB-R 61.8 → ~69 (+7) with 4B/8B reranker" + sample_size: "MTEB-R" + observed_date: "2025-06" + source_url: https://arxiv.org/abs/2506.05176 + normalization_note: cross-encoder; post-rerank P(correct in top-k); corpus-dependent; SOURCED + - model: cohere-rerank-3 + task_type: reranking + sigma_raw: { low: 0.58, mid: 0.65, high: 0.72 } + primary_benchmark: BEIR / TREC-DL (cross-encoder) + metric: nDCG@10 uplift + metric_kind: relative + reported_value: "proprietary cross-encoder; no clean public absolute row" + sample_size: null + observed_date: "2026-06" + source_url: https://arxiv.org/abs/2508.16757 + normalization_note: cross-encoder; relative-only public evidence; wide band; SOURCED + - model: bge-reranker-v2-m3 + task_type: reranking + sigma_raw: { low: 0.55, mid: 0.61, high: 0.67 } + primary_benchmark: MTEB-R (cross-encoder) + metric: nDCG@10 + metric_kind: absolute + reported_value: "MTEB-R ~57.0" + sample_size: "MTEB-R" + observed_date: "2025-06" + source_url: https://arxiv.org/abs/2506.05176 + normalization_note: open cross-encoder reranker; corpus-dependent; SOURCED + - model: rankzephyr-7b + task_type: reranking + sigma_raw: { low: 0.62, mid: 0.69, high: 0.75 } + primary_benchmark: TREC-DL19 / FutureQueryEval (listwise LLM) + metric: nDCG@10 + metric_kind: absolute + reported_value: "TREC-DL19 74.2; novel-query 62.7" + sample_size: "TREC-DL / FutureQueryEval" + observed_date: "2025-08" + source_url: https://arxiv.org/abs/2508.16757 + normalization_note: listwise LLM reranker (post-ranking, NOT retrieval); recency penalty on novel queries; SOURCED + - model: rankgpt-gpt4 + task_type: reranking + sigma_raw: { low: 0.66, mid: 0.72, high: 0.78 } + primary_benchmark: TREC-DL19 (listwise LLM) + metric: nDCG@10 + metric_kind: absolute + reported_value: "TREC-DL19 75.6" + sample_size: "TREC-DL19" + observed_date: "2025-08" + source_url: https://arxiv.org/abs/2508.16757 + normalization_note: listwise LLM post-ranking; an LLM's legitimate role in retrieval is reranking, NOT the retrieval step; SOURCED + +# ---- REVIEW · LLM-as-judge vs human reviewer (catch_rate) --------------------- +# Reviewer error-DETECTION. Distinguishes a model reviewer (LLM-as-judge) from a +# human reviewer. RewardBench-2 (arXiv:2506.01937 / sweep 2604.13717), CriticGPT +# (arXiv:2407.00215). SOFTEST COLUMN — wide bands. + + - model: llm-judge-single + task_type: review + catch_rate: { low: 0.60, mid: 0.68, high: 0.74 } + primary_benchmark: RewardBench-2 (single LLM judge, k=1) + metric: judge↔human preference accuracy + metric_kind: absolute + reported_value: "71.7% accuracy (N=1729)" + sample_size: "1729" + observed_date: "2026" + source_url: https://arxiv.org/abs/2506.01937 + normalization_note: a single LLM-as-judge call; SOFTEST COLUMN; subject to position/verbosity/self-preference bias; SOURCED + - model: llm-judge-ensemble + task_type: review + catch_rate: { low: 0.72, mid: 0.80, high: 0.85 } + primary_benchmark: RewardBench-2 (ensemble k=8) + metric: judge↔human preference accuracy + metric_kind: absolute + reported_value: "81.5% (k=8); 83.6% with scoring criteria" + sample_size: "1729" + observed_date: "2026" + source_url: https://arxiv.org/abs/2506.01937 + normalization_note: ensembling N independent judge calls materially raises catch_rate (+~10pp); still bias-prone; SOURCED + - model: human-reviewer + task_type: review + catch_rate: { low: 0.75, mid: 0.85, high: 0.92 } + primary_benchmark: paper assumption (domain-expert reviewer) + metric: error-detection rate + metric_kind: relative + reported_value: "model-default; CriticGPT beats humans on 63% of code critiques" + sample_size: null + observed_date: "2024" + source_url: https://arxiv.org/abs/2407.00215 + normalization_note: human domain expert; higher catch than a single LLM judge but slower/costly; for code, a strong LLM judge can rival humans; refine from traces + +# ---- CORRECTION · corrector fix-success (fix_rate) ---------------------------- +# Corrector REPAIR-success given a flagged error: P(repair succeeds | flagged). +# KEY FINDING (multiple papers): intrinsic self-correction WITHOUT external +# feedback is unreliable (at/below baseline); WITH reviewer feedback it improves. +# arXiv:2310.01798 (LLMs Cannot Self-Correct Reasoning Yet) · TACL 2024. + + - model: corrector-no-feedback + task_type: correction + fix_rate: { low: 0.10, mid: 0.30, high: 0.55 } + primary_benchmark: "LLMs Cannot Self-Correct Reasoning Yet" + metric: P(repair succeeds | flagged), no external signal + metric_kind: absolute + reported_value: "intrinsic self-correction at/below baseline; can degrade" + sample_size: null + observed_date: "2023-10" + source_url: https://arxiv.org/abs/2310.01798 + normalization_note: corrector with NO reviewer feedback; unreliable → low/wide band; this is why detection (reviewer) and repair (corrector) must be modeled separately + - model: corrector-with-feedback + task_type: correction + fix_rate: { low: 0.55, mid: 0.70, high: 0.85 } + primary_benchmark: "When Can LLMs Actually Correct Their Own Mistakes (TACL 2024)" + metric: P(repair succeeds | flagged), with reviewer feedback + metric_kind: absolute + reported_value: "correction with reliable external feedback materially beats baseline" + sample_size: null + observed_date: "2024" + source_url: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00713/125177 + normalization_note: corrector that CONSUMES reviewer feedback and re-does/patches; fix_rate is high only when the feedback signal is reliable; SOURCED + - model: corrector-with-oracle + task_type: correction + fix_rate: { low: 0.65, mid: 0.78, high: 0.90 } + primary_benchmark: Reflexion / agentic re-do with test or oracle signal + metric: P(repair succeeds | flagged), with verifiable signal + metric_kind: absolute + reported_value: "agentic re-do with unit-test/oracle signal repairs reliably" + sample_size: null + observed_date: "2023" + source_url: https://arxiv.org/abs/2303.11366 + normalization_note: corrector with a verifiable signal (tests, exec results); highest fix_rate; degrades if the signal is noisy; SOURCED diff --git a/src/minimal_oversight/estimation.py b/src/minimal_oversight/estimation.py index b383d21..82c2e69 100644 --- a/src/minimal_oversight/estimation.py +++ b/src/minimal_oversight/estimation.py @@ -69,9 +69,16 @@ def estimate_catch_rate( raw_outcomes: list[float] | np.ndarray, corrected_outcomes: list[float] | np.ndarray, ) -> float | None: - """Infer corrector catch rate from paired pre/post-correction outcomes. + """Infer the *effective correction* from paired pre/post outcomes. - c = (σ_corr − σ_raw) / (1 − σ_raw) + c_eff = (σ_corr − σ_raw) / (1 − σ_raw) + + This is the inverse of Equation 6. Pre/post outcomes alone cannot separate + *detection* (catch_rate, *c*) from *repair* (fix_rate, *f*) — they only + reveal the product ``c_eff = c × f``. When the reviewer's flag log is also + available, use :func:`estimate_fix_rate` to decompose them. When no separate + corrector exists (the node self-reviews and self-fixes), ``c_eff`` *is* the + catch_rate with fix_rate ≡ 1. Returns None if σ_raw ≈ 1 (no errors to catch). """ @@ -79,8 +86,37 @@ def estimate_catch_rate( s_corr = float(np.mean(corrected_outcomes)) if (1.0 - s_raw) < 1e-8: return None - c = (s_corr - s_raw) / (1.0 - s_raw) - return float(np.clip(c, 0.0, 1.0)) + c_eff = (s_corr - s_raw) / (1.0 - s_raw) + return float(np.clip(c_eff, 0.0, 1.0)) + + +def estimate_fix_rate( + flagged: list[float] | np.ndarray, + repaired: list[float] | np.ndarray, +) -> float | None: + """Infer corrector repair-success rate *f* from the reviewer's flag log. + + Given, for each item the reviewer *flagged* as an error, whether the + corrector's re-do/patch actually *succeeded*: + + f = P(repair succeeds | error was flagged) + = mean(repaired over flagged items) + + Args: + flagged: 1 if the reviewer flagged this item as an error, else 0. + repaired: 1 if the corrector's fix succeeded on this item, else 0. + Only entries where ``flagged == 1`` are counted. + + Returns None if nothing was flagged (no repairs to score). + """ + flagged_arr = np.asarray(flagged, dtype=float) + repaired_arr = np.asarray(repaired, dtype=float) + mask = flagged_arr > 0.5 + n_flagged = int(np.sum(mask)) + if n_flagged == 0: + return None + f = float(np.sum(repaired_arr[mask]) / n_flagged) + return float(np.clip(f, 0.0, 1.0)) def estimate_process_entropy( diff --git a/src/minimal_oversight/models.py b/src/minimal_oversight/models.py index 2dddc06..be8fc8b 100644 --- a/src/minimal_oversight/models.py +++ b/src/minimal_oversight/models.py @@ -23,15 +23,24 @@ class AggregationType(Enum): @dataclass class Node: - """A single delegation node (agent + optional corrector). + """A single delegation node (an agent, with optional review + correction). + + Oversight is decomposed into two independent stages. A **reviewer** (human, + or a model acting as LLM-as-judge) *detects* errors with probability + ``catch_rate`` (*c*). A **corrector** then *repairs* a flagged error with + probability ``fix_rate`` (*f*), either by re-doing the task or applying a + patch using the reviewer's feedback. The effective correction is the product + ``c × f``; the paper's single-number model is the special case ``f = 1``. Attributes: name: Human-readable identifier. sigma_skill: True competence (if known); often estimated. sigma_raw: Observed raw competence (pre-correction success rate). sigma_corr: Observed corrected quality (post-correction success rate). - catch_rate: Corrector's error-catch probability *c*. - review_capacity: Fraction of outputs the corrector reviews (*K/N*). + catch_rate: Reviewer's error-detection probability *c*. + fix_rate: Corrector's repair-success probability *f* given a flagged + error. Defaults to None → treated as 1.0 (caught errors are fixed). + review_capacity: Fraction of outputs the reviewer inspects (*K/N*). drift_rate: Estimated skill degradation rate *mu_eff*. noise_var: Estimated noise variance *nu_eff^2*. aggregation: How this node merges inputs (only relevant for merge nodes). @@ -43,6 +52,7 @@ class Node: sigma_raw: float | None = None sigma_corr: float | None = None catch_rate: float | None = None + fix_rate: float | None = None review_capacity: float | None = None drift_rate: float | None = None noise_var: float | None = None diff --git a/src/minimal_oversight/priors.py b/src/minimal_oversight/priors.py index a059b53..fb05e3d 100644 --- a/src/minimal_oversight/priors.py +++ b/src/minimal_oversight/priors.py @@ -24,7 +24,8 @@ import yaml -REVIEW_TASK = "review" +REVIEW_TASK = "review" # payload: catch_rate (reviewer error-detection) +CORRECTION_TASK = "correction" # payload: fix_rate (corrector repair-success) @dataclass(frozen=True) @@ -50,8 +51,9 @@ class PriorCell: model: str task_type: str - sigma_raw: Band | None # generator task-types - catch_rate: Band | None # the `review` task-type + sigma_raw: Band | None # generator / retrieval / reranking task-types + catch_rate: Band | None # the `review` task-type (reviewer detection) + fix_rate: Band | None # the `correction` task-type (corrector repair) primary_benchmark: str | None metric: str | None metric_kind: str | None # "absolute" | "relative" @@ -95,11 +97,14 @@ def load_priors(path: str | None = None) -> dict[str, Any]: raise ValueError(f"unknown task_type {task!r} for model {model!r}") sigma = Band.from_mapping(row["sigma_raw"]) if "sigma_raw" in row else None catch = Band.from_mapping(row["catch_rate"]) if "catch_rate" in row else None + fix = Band.from_mapping(row["fix_rate"]) if "fix_rate" in row else None if task == REVIEW_TASK and catch is None: raise ValueError(f"review cell {model!r} must carry a catch_rate band") - if task != REVIEW_TASK and sigma is None: + elif task == CORRECTION_TASK and fix is None: + raise ValueError(f"correction cell {model!r} must carry a fix_rate band") + elif task not in (REVIEW_TASK, CORRECTION_TASK) and sigma is None: raise ValueError(f"generator cell {model!r}/{task!r} must carry a sigma_raw band") - for b in (sigma, catch): + for b in (sigma, catch, fix): if b is not None and not (0.0 <= b.low <= b.mid <= b.high <= 1.0): raise ValueError(f"non-monotone or out-of-range band for {model!r}/{task!r}") key = (model, task) @@ -110,6 +115,7 @@ def load_priors(path: str | None = None) -> dict[str, Any]: task_type=task, sigma_raw=sigma, catch_rate=catch, + fix_rate=fix, primary_benchmark=row.get("primary_benchmark"), metric=row.get("metric"), metric_kind=row.get("metric_kind"), @@ -160,6 +166,10 @@ def seed_node(model: str, task_type: str, path: str | None = None) -> dict[str, band = cell.catch_rate out["catch_rate"] = _clamp(band.mid, 0.0, 1.0) out["seeds"] = "catch_rate" + elif task_type == CORRECTION_TASK: + band = cell.fix_rate + out["fix_rate"] = _clamp(band.mid, 0.0, 1.0) + out["seeds"] = "fix_rate" else: band = cell.sigma_raw meta = data["meta"] diff --git a/tests/test_estimation.py b/tests/test_estimation.py new file mode 100644 index 0000000..dc511f0 --- /dev/null +++ b/tests/test_estimation.py @@ -0,0 +1,44 @@ +"""Tests for the estimation module — inferring quantities from traces. + +Focus on the detection × fix-success decomposition: pre/post outcomes reveal +only the *effective* correction; the reviewer's flag log is needed to recover +the corrector's fix_rate. +""" + +import pytest + +from minimal_oversight.estimation import ( + estimate_catch_rate, + estimate_fix_rate, +) + + +class TestEffectiveCorrection: + def test_catch_rate_inverts_equation_6(self): + # raw=0.6, corr=0.86 → c_eff = (0.86-0.6)/(1-0.6) = 0.65 + raw = [1, 1, 1, 0, 0] # mean 0.6 + corr = [1, 1, 1, 1, 0] # mean 0.8 + c_eff = estimate_catch_rate(raw, corr) + expected = (0.8 - 0.6) / (1 - 0.6) + assert pytest.approx(c_eff, rel=1e-9) == expected + + def test_none_when_no_errors(self): + assert estimate_catch_rate([1, 1, 1], [1, 1, 1]) is None + + +class TestFixRate: + def test_fix_rate_is_conditional_on_flagged(self): + # 4 items flagged as errors; 3 repairs succeeded → f = 0.75. + flagged = [1, 1, 1, 1, 0, 0] + repaired = [1, 1, 1, 0, 0, 0] + f = estimate_fix_rate(flagged, repaired) + assert pytest.approx(f, rel=1e-9) == 0.75 + + def test_unflagged_items_ignored(self): + # A "success" on an unflagged item must not inflate fix_rate. + flagged = [1, 0, 0, 0] + repaired = [0, 1, 1, 1] + assert estimate_fix_rate(flagged, repaired) == 0.0 + + def test_none_when_nothing_flagged(self): + assert estimate_fix_rate([0, 0, 0], [0, 0, 0]) is None diff --git a/tests/test_formulae.py b/tests/test_formulae.py index 6735bf0..7a96973 100644 --- a/tests/test_formulae.py +++ b/tests/test_formulae.py @@ -144,6 +144,45 @@ def test_threshold_is_zero_when_raw_support_suffices(self): assert threshold == 0.0 +class TestCorrectorDecomposition: + """Detection × fix-success generalization of Equation 6. + + A reviewer detects errors (catch_rate); a corrector repairs flagged errors + (fix_rate). Effective correction is the product; fix_rate=1 recovers the + paper's single-number model. + """ + + def test_fix_rate_one_recovers_legacy(self): + # Default fix_rate=1.0 must reproduce the paper's worked example. + legacy = sigma_corr_fixed_point(0.667, catch_rate=0.70) + explicit = sigma_corr_fixed_point(0.667, catch_rate=0.70, fix_rate=1.0) + assert explicit == legacy + assert pytest.approx(explicit, rel=1e-2) == 0.900 + + def test_effective_correction_is_product(self): + # catch=0.70, fix=0.50 → c_eff=0.35 → σ_corr = 0.667 + 0.333*0.35 + sc = sigma_corr_fixed_point(0.667, catch_rate=0.70, fix_rate=0.50) + expected = 0.667 + (1 - 0.667) * (0.70 * 0.50) + assert pytest.approx(sc, rel=1e-9) == expected + + def test_zero_fix_rate_means_no_correction(self): + # A reviewer that catches everything but a corrector that never repairs + # leaves quality at raw. + sc = sigma_corr_fixed_point(0.60, catch_rate=1.0, fix_rate=0.0) + assert sc == 0.60 + + def test_partial_fix_needs_more_review(self): + # Halving fix_rate doubles the review fraction needed for the same target. + full = corrector_capacity_threshold(0.80, 0.60, catch_rate=0.50, fix_rate=1.0) + half = corrector_capacity_threshold(0.80, 0.60, catch_rate=0.50, fix_rate=0.5) + assert pytest.approx(half, rel=1e-9) == 2 * full + + def test_chain_quality_degrades_with_lower_fix_rate(self): + strong = recursive_chain_quality(3, 0.55, 0.65, 10, 2, fix_rate=1.0) + weak = recursive_chain_quality(3, 0.55, 0.65, 10, 2, fix_rate=0.5) + assert strong > weak + + class TestSOTA: def test_score_increases_with_centrality(self): s1 = sota_priority_score(1.0, 1.35, 0.45) diff --git a/tests/test_parity.py b/tests/test_parity.py index 0f19b1b..85fd72c 100644 --- a/tests/test_parity.py +++ b/tests/test_parity.py @@ -42,11 +42,12 @@ } -def _nd(node_id, skill, catch, parents, agg="product"): +def _nd(node_id, skill, catch, parents, agg="product", fix=None): return { "id": node_id, "sigma_skill": skill, "catch_rate": catch, + "fix_rate": fix, "parents": parents, "aggregation": agg, } @@ -80,12 +81,24 @@ def _nd(node_id, skill, catch, parents, agg="product"): _nd("d", 0.66, 0.6, ["c"]), ] } +# Reviewer detects (catch_rate) but the corrector only repairs a fraction +# (fix_rate < 1). Exercises the detection × fix-success generalization of Eq. 6. +PIPE_FIX = { + "nodes": [ + _nd("gen", 0.62, 0.70, [], fix=0.5), + _nd("review_correct", 0.80, 0.65, ["gen"], fix=0.8), + ] +} CASES = { "fisher": SIGMAS, "volume": SIGMAS, "sraw_fp": [[0.55, 10, 2, 0], [0.8, 10, 2, 0], [0.45, 10, 2, 0.1], [0.62, 8, 3, 0]], - "scorr": [[0.5167, 0.65], [0.375, 0.70], [0.708, 0.70], [0.2, 0.9]], + "scorr": [ + [0.5167, 0.65], [0.375, 0.70], [0.708, 0.70], [0.2, 0.9], + # 3-arg form: fix_rate < 1 (corrector repairs only a fraction of catches) + [0.5167, 0.65, 0.5], [0.375, 0.70, 0.8], [0.2, 0.9, 1.0], [0.6, 0.5, 0.0], + ], "masking": [[0.83, 0.5167], [0.8125, 0.375], [0.9125, 0.708]], "eff_skill": [ [0.85, [0.7012, 0.7908], "product"], @@ -102,6 +115,7 @@ def _nd(node_id, skill, catch, parents, agg="product"): {"pipeline": PIPE_SALES, "p_min": 0.80, "hw": 0.0}, {"pipeline": PIPE_SALES, "p_min": 0.70, "hw": 1.5}, {"pipeline": PIPE_CHAIN, "p_min": 0.75, "hw": 0.5}, + {"pipeline": PIPE_FIX, "p_min": 0.70, "hw": 0.0}, ], "centrality": [ {"pipeline": PIPE_SALES, "names": [n["id"] for n in PIPE_SALES["nodes"]]}, @@ -130,6 +144,12 @@ def _nd(node_id, skill, catch, parents, agg="product"): {"model": "gpt-4o", "task_type": "drafting"}, {"model": "gemini-2-flash", "task_type": "review"}, {"model": "deepseek-r1", "task_type": "extraction"}, + # v2 component types: embedder retrieval, reranker, judge, corrector + {"model": "qwen3-embedding-8b", "task_type": "retrieval"}, + {"model": "rankgpt-gpt4", "task_type": "reranking"}, + {"model": "llm-judge-ensemble", "task_type": "review"}, + {"model": "corrector-with-feedback", "task_type": "correction"}, + {"model": "gpt-5.4-nano", "task_type": "grounded_generation"}, ], # web/mso-estimate.js turns a practitioner's real outcomes into per-node # sigma_raw / sigma_corr / catch / masking. Pin those to estimation.py. @@ -161,6 +181,7 @@ def _build_graph(spec: dict) -> PipelineGraph: n["id"], sigma_skill=n["sigma_skill"], catch_rate=n["catch_rate"], + fix_rate=n.get("fix_rate"), aggregation=_AGG[n["aggregation"]], ) for n in spec["nodes"] @@ -175,8 +196,10 @@ def _build_graph(spec: dict) -> PipelineGraph: def _node_masking(spec: dict) -> dict: masking = {} for n in spec["nodes"]: + fix = n.get("fix_rate") + fix = 1.0 if fix is None else fix lsr = F.sigma_raw_fixed_point(n["sigma_skill"], 10, 2, 0) - lsc = F.sigma_corr_fixed_point(lsr, n["catch_rate"]) + lsc = F.sigma_corr_fixed_point(lsr, n["catch_rate"], fix) masking[n["id"]] = F.masking_index(lsc, lsr) return masking @@ -266,7 +289,8 @@ def _python_expected() -> dict: exp["seed_node"].append({ "seeds": s["seeds"], "sigma_skill": s.get("sigma_skill"), - "catch_rate": s["catch_rate"], + "catch_rate": s.get("catch_rate"), + "fix_rate": s.get("fix_rate"), "confidence": prov["confidence"], "band_low": prov["band"]["low"], "band_mid": prov["band"]["mid"], @@ -347,6 +371,8 @@ def test_browser_port_matches_python_reference(): assert g["seeds"] == e["seeds"], f"seed_node seeds mismatch for {label}" for k in ["band_low", "band_mid", "band_high", "confidence"]: assert _close(g[k], e[k]), f"seed_node {k} mismatch for {label}" - if e["sigma_skill"] is not None: - assert _close(g["sigma_skill"], e["sigma_skill"]), f"seed_node sigma_skill mismatch for {label}" - assert _close(g["catch_rate"], e["catch_rate"]), f"seed_node catch_rate mismatch for {label}" + for k in ["sigma_skill", "catch_rate", "fix_rate"]: + if e[k] is not None: + assert g[k] is not None and _close(g[k], e[k]), f"seed_node {k} mismatch for {label}" + else: + assert g[k] is None, f"seed_node {k} should be null for {label}" diff --git a/tests/test_priors.py b/tests/test_priors.py index 348d86e..9f1ead5 100644 --- a/tests/test_priors.py +++ b/tests/test_priors.py @@ -2,12 +2,33 @@ from __future__ import annotations +import importlib.util import textwrap +from pathlib import Path import pytest from minimal_oversight import priors as P +_REPO = Path(__file__).resolve().parents[1] + + +def test_js_bundle_is_regenerated_from_yaml(): + """web/mso-priors.js must be freshly generated from priors.yaml. + + Guards against editing priors.yaml without running scripts/gen_priors_js.py. + """ + spec = importlib.util.spec_from_file_location( + "gen_priors_js", _REPO / "scripts" / "gen_priors_js.py" + ) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + generated = mod.build() + on_disk = (_REPO / "web" / "mso-priors.js").read_text() + assert generated == on_disk, ( + "web/mso-priors.js is stale — run: python scripts/gen_priors_js.py" + ) + def test_table_loads_and_lists(): models = P.list_models() @@ -22,7 +43,7 @@ def test_generator_seed_reproduces_sigma_raw_at_fixed_point(): data = P.load_priors() gamma = data["gamma"] for (model, task), cell in data["cells"].items(): - if task == "review": + if task in ("review", "correction"): continue seed = P.seed_node(model, task) assert seed["seeds"] == "sigma_skill" @@ -44,6 +65,24 @@ def test_review_seed_yields_catch_rate(): assert "sigma_skill" not in seed +def test_correction_seed_yields_fix_rate(): + correction_cells = [(m, t) for (m, t) in P.load_priors()["cells"] if t == "correction"] + assert correction_cells, "expected at least one correction cell" + for model, _ in correction_cells: + seed = P.seed_node(model, "correction") + assert seed["seeds"] == "fix_rate" + assert 0.0 <= seed["fix_rate"] <= 1.0 + assert "sigma_skill" not in seed + assert "catch_rate" not in seed + + +def test_corrector_no_feedback_is_low_fix_rate(): + """The headline finding: a corrector with no reviewer feedback is unreliable.""" + no_fb = P.seed_node("corrector-no-feedback", "correction")["fix_rate"] + with_fb = P.seed_node("corrector-with-feedback", "correction")["fix_rate"] + assert no_fb < with_fb, "self-correction without feedback must be a weaker prior" + + def test_provenance_block_is_well_formed(): for (model, task) in P.load_priors()["cells"]: prov = P.seed_node(model, task)["provenance"] diff --git a/web/mso-core.js b/web/mso-core.js index b3a06c9..830282f 100644 --- a/web/mso-core.js +++ b/web/mso-core.js @@ -42,9 +42,12 @@ if (denom <= 0) throw new Error("eta + delta must be positive"); return (eta * sigmaSkill + delta * sigma0) / denom; } - // Fixed-point corrected quality: σ*_corr = σ_raw + (1−σ_raw)·c (Eq. 6) - function sigmaCorrFixedPoint(sigmaRaw, catchRate) { - return sigmaRaw + (1 - sigmaRaw) * catchRate; + // Fixed-point corrected quality: σ*_corr = σ_raw + (1−σ_raw)·c_eff (Eq. 6). + // c_eff = catchRate (reviewer detection) × fixRate (corrector repair). + // fixRate defaults to 1 → the paper's single-number model (caught == fixed). + function sigmaCorrFixedPoint(sigmaRaw, catchRate, fixRate) { + var f = fixRate == null ? 1 : fixRate; + return sigmaRaw + (1 - sigmaRaw) * catchRate * f; } // Masking index M* = σ_corr/σ_raw; > 1 indicates masking function maskingIndex(sigmaCorr, sigmaRaw) { @@ -185,6 +188,7 @@ var n = byId[id]; var skill = n.sigma_skill == null ? 0.55 : n.sigma_skill; var c = n.catch_rate == null ? 0.65 : n.catch_rate; + var f = n.fix_rate == null ? 1 : n.fix_rate; var parents = n.parents || []; var skillEff; if (parents.length) { @@ -192,7 +196,7 @@ skillEff = effectiveSkill(skill, pc, n.aggregation || "product"); } else skillEff = skill; var sr = sigmaRawFixedPoint(skillEff, eta, delta, s0); - var sc = sigmaCorrFixedPoint(sr, c); + var sc = sigmaCorrFixedPoint(sr, c, f); corr[id] = sc; caps[id] = sc; }); return caps; @@ -220,8 +224,9 @@ var perNode = {}; nodes.forEach(function (n) { var c = n.catch_rate == null ? 0.65 : n.catch_rate; + var f = n.fix_rate == null ? 1 : n.fix_rate; var lsr = sigmaRawFixedPoint(n.sigma_skill == null ? 0.55 : n.sigma_skill, eta, delta, s0); - var lsc = sigmaCorrFixedPoint(lsr, c); + var lsc = sigmaCorrFixedPoint(lsr, c, f); perNode[n.id] = { sigma_raw: lsr, sigma_corr: lsc, masking: maskingIndex(lsc, lsr), fisher: fisherInformation(lsr), capacity: caps[n.id] diff --git a/web/mso-priors.js b/web/mso-priors.js index 6e9d0d6..1feb3ec 100644 --- a/web/mso-priors.js +++ b/web/mso-priors.js @@ -1,19 +1,17 @@ /** - * mso-priors.js — cold-start σ_raw/catch_rate priors for MSO cockpit nodes. + * mso-priors.js — cold-start σ_raw / catch_rate / fix_rate priors for cockpit nodes. * - * Mirrors minimal_oversight.priors.seed_node (Python) exactly: - * - generator task-types → sigma_skill = clamp(band.mid / γ, 0.05, 0.98) - * so gamma * sigma_skill == band.mid at the fixed point. - * - review task-type → catch_rate = clamp(band.mid, 0, 1) - * - provenance.confidence = 1 − band_width (a crude evidence-strength proxy) + * GENERATED FROM src/minimal_oversight/data/priors.yaml — DO NOT EDIT BY HAND. + * Regenerate with: python scripts/gen_priors_js.py * - * Data sourced from a cited deep-research pass (2026-06-17) against: - * - LiveBench Coding (code_generation, absolute) - * - JSONSchemaBench arXiv:2501.10868 (extraction, absolute) - * - Vectara HHEM-2.3 + FaithBench/RAGTruth (drafting + retrieval, absolute) - * - FaithJudge / MT-Bench / RewardBench-2 / CriticBench (review, absolute/relative) - * - Conservative tier-interpolation (classification — no surviving absolute benchmark) + * Mirrors minimal_oversight.priors.seed_node (Python) exactly: + * - generator / retrieval / reranking task-types → sigma_skill = clamp(mid/γ, 0.05, 0.98) + * so gamma * sigma_skill == band.mid at the return-operator fixed point. + * - review task-type → catch_rate = clamp(mid, 0, 1) (reviewer error-detection) + * - correction task-type → fix_rate = clamp(mid, 0, 1) (corrector repair-success) + * - provenance.confidence = 1 − band_width (a crude evidence-strength proxy). * + * Provenance for every band: docs/methodology/priors-evidence.md. * Parity: tests/test_parity.py pins JS ↔ Python to within 1e-9 on seedNode outputs. */ (function (root, factory) { @@ -57,7 +55,7 @@ "claude-opus-4|extraction": {kind:"generator",band:{low:0.82,mid:0.91,high:0.98},benchmark:"JSONSchemaBench",metric_kind:"absolute",note:"moderate schema, plain prompting; frontier tier; schema difficulty dominates; wide band"}, "claude-sonnet-4|extraction": {kind:"generator",band:{low:0.75,mid:0.87,high:0.96},benchmark:"JSONSchemaBench",metric_kind:"absolute",note:"moderate schema, plain prompting; mid tier; wide band to capture difficulty variance"}, "claude-haiku-4|extraction": {kind:"generator",band:{low:0.5,mid:0.7,high:0.88},benchmark:"JSONSchemaBench",metric_kind:"absolute",note:"small/efficient tier; Llama-1B floor; Haiku expected above floor but degrades on hard schemas"}, - "gpt-4o|extraction": {kind:"generator",band:{low:0.8,mid:0.91,high:0.98},benchmark:"JSONSchemaBench",metric_kind:"absolute",note:"OpenAI JSON-mode raises compliance on accepted schemas; coverage drops on hard schemas; moderate schema"}, + "gpt-4o|extraction": {kind:"generator",band:{low:0.8,mid:0.91,high:0.98},benchmark:"JSONSchemaBench",metric_kind:"absolute",note:"OpenAI JSON-mode raises compliance on accepted schemas; coverage drops on hard schemas; moderate schema default"}, "gpt-4o-mini|extraction": {kind:"generator",band:{low:0.65,mid:0.8,high:0.93},benchmark:"JSONSchemaBench",metric_kind:"absolute",note:"smaller model; JSON-mode available but coverage limited; wide band"}, "openai-o3|extraction": {kind:"generator",band:{low:0.82,mid:0.92,high:0.99},benchmark:"JSONSchemaBench (interpolated)",metric_kind:"absolute",note:"frontier reasoning model; constrained API expected; interpolated upward; wide band"}, "gemini-2-pro|extraction": {kind:"generator",band:{low:0.8,mid:0.91,high:0.98},benchmark:"JSONSchemaBench",metric_kind:"absolute",note:"Gemini constrained-mode matches OpenAI on accepted schemas; coverage limited on hard schemas"}, @@ -77,23 +75,23 @@ "gemini-2-flash|drafting": {kind:"generator",band:{low:0.88,mid:0.93,high:0.97},benchmark:"Vectara-HHEM",metric_kind:"absolute",note:"Flash-class tops HHEM faithfulness leaderboard; faithfulness subdim only; note HHEM is Vectara's own judge"}, "llama-3.3-70b|drafting": {kind:"generator",band:{low:0.78,mid:0.84,high:0.9},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"direct FaithBench measurement; faithfulness subdim only; style relative-only"}, "deepseek-v3|drafting": {kind:"generator",band:{low:0.88,mid:0.93,high:0.97},benchmark:"Vectara-HHEM + DeepSeek blog",metric_kind:"absolute",note:"V3 is notably less hallucinatory than R1; one of the best base-model faithfulness rates"}, - "deepseek-r1|drafting": {kind:"generator",band:{low:0.8,mid:0.87,high:0.91},benchmark:"Vectara-HHEM + DeepSeek blog",metric_kind:"absolute",note:"REASONING PENALTY confirmed; R1 substantially worse than V3 on faithfulness; R1-0528 narrowed gap ~45-50%"}, + "deepseek-r1|drafting": {kind:"generator",band:{low:0.8,mid:0.87,high:0.91},benchmark:"Vectara-HHEM + DeepSeek blog",metric_kind:"absolute",note:"REASONING PENALTY confirmed; R1 substantially worse than V3 on faithfulness; R1-0528 narrowed gap ~45-50% \u2014 widen"}, "qwen2.5-72b|drafting": {kind:"generator",band:{low:0.78,mid:0.85,high:0.92},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; mid tier faithfulness; wide band"}, "mistral-large|drafting": {kind:"generator",band:{low:0.78,mid:0.85,high:0.92},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; mid tier faithfulness; wide band"}, - "claude-opus-4|retrieval": {kind:"generator",band:{low:0.83,mid:0.89,high:0.94},benchmark:"Vectara-HHEM + FaithBench/RAGTruth",metric_kind:"absolute",note:"faithfulness PROXY — not full RAG answer-correctness; interpolated above Sonnet anchor"}, - "claude-sonnet-4|retrieval": {kind:"generator",band:{low:0.8,mid:0.87,high:0.93},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"direct FaithBench measurement; faithfulness PROXY only; Sonnet-4.5 confirmed >10% hallu — use low end"}, - "claude-haiku-4|retrieval": {kind:"generator",band:{low:0.75,mid:0.83,high:0.9},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; small tier; faithfulness proxy only; refine from traces"}, - "gpt-4o|retrieval": {kind:"generator",band:{low:0.8,mid:0.84,high:0.9},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"direct FaithBench measurement; faithfulness PROXY only"}, - "gpt-4o-mini|retrieval": {kind:"generator",band:{low:0.72,mid:0.8,high:0.88},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; small tier; faithfulness proxy only"}, - "openai-o3|retrieval": {kind:"generator",band:{low:0.8,mid:0.87,high:0.93},benchmark:"Vectara next-gen (interpolated)",metric_kind:"relative",note:"reasoning models WORSE on faithfulness than Flash; wide band; faithfulness proxy only"}, - "gemini-2-pro|retrieval": {kind:"generator",band:{low:0.88,mid:0.93,high:0.97},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"best faithfulness in FaithBench set; faithfulness PROXY only; note HHEM conflict-of-interest"}, - "gemini-2-flash|retrieval": {kind:"generator",band:{low:0.88,mid:0.93,high:0.97},benchmark:"Vectara-HHEM + FaithBench",metric_kind:"absolute",note:"Flash-class tops HHEM; FaithBench confirms strong faithfulness; PROXY only"}, - "llama-3.3-70b|retrieval": {kind:"generator",band:{low:0.78,mid:0.84,high:0.9},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"direct FaithBench measurement; faithfulness PROXY only; 70B tier"}, - "deepseek-v3|retrieval": {kind:"generator",band:{low:0.88,mid:0.93,high:0.96},benchmark:"Vectara-HHEM + DeepSeek blog",metric_kind:"absolute",note:"V3 outstanding faithfulness; faithfulness PROXY only; does not predict end-to-end RAG correctness"}, - "deepseek-r1|retrieval": {kind:"generator",band:{low:0.78,mid:0.86,high:0.91},benchmark:"Vectara-HHEM + DeepSeek blog",metric_kind:"absolute",note:"REASONING PENALTY on faithfulness; R1+RAG can hit 86% clinical accuracy despite poor faithfulness — proxy gap CRITICAL"}, - "qwen2.5-72b|retrieval": {kind:"generator",band:{low:0.75,mid:0.85,high:0.92},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; mid tier faithfulness proxy; wide band"}, - "mistral-large|retrieval": {kind:"generator",band:{low:0.75,mid:0.85,high:0.92},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; mid tier faithfulness proxy; wide band"}, - "claude-opus-4|review": {kind:"review",band:{low:0.62,mid:0.72,high:0.84},benchmark:"FaithJudge / MT-Bench (zero-shot LLM judge)",metric_kind:"absolute",note:"SOFTEST COLUMN; strong frontier judge tier; zero-shot only; few-shot raises to ~84% (not assumed here); wide band"}, + "claude-opus-4|grounded_generation": {kind:"generator",band:{low:0.83,mid:0.89,high:0.94},benchmark:"Vectara-HHEM + FaithBench/RAGTruth",metric_kind:"absolute",note:"faithfulness PROXY \u2014 not full RAG answer-correctness; interpolated above Sonnet anchor"}, + "claude-sonnet-4|grounded_generation": {kind:"generator",band:{low:0.8,mid:0.87,high:0.93},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"direct FaithBench measurement; faithfulness PROXY only; Sonnet-4.5 confirmed >10% hallu \u2014 use low end of band"}, + "claude-haiku-4|grounded_generation": {kind:"generator",band:{low:0.75,mid:0.83,high:0.9},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; small tier; faithfulness proxy only; refine from traces"}, + "gpt-4o|grounded_generation": {kind:"generator",band:{low:0.8,mid:0.84,high:0.9},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"direct FaithBench measurement; faithfulness PROXY only"}, + "gpt-4o-mini|grounded_generation": {kind:"generator",band:{low:0.72,mid:0.8,high:0.88},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; small tier; faithfulness proxy only"}, + "openai-o3|grounded_generation": {kind:"generator",band:{low:0.8,mid:0.87,high:0.93},benchmark:"Vectara next-gen (interpolated)",metric_kind:"relative",note:"reasoning models WORSE on faithfulness than Flash; wide band; faithfulness proxy only"}, + "gemini-2-pro|grounded_generation": {kind:"generator",band:{low:0.88,mid:0.93,high:0.97},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"best faithfulness in FaithBench set; faithfulness PROXY only; note HHEM conflict-of-interest"}, + "gemini-2-flash|grounded_generation": {kind:"generator",band:{low:0.88,mid:0.93,high:0.97},benchmark:"Vectara-HHEM + FaithBench",metric_kind:"absolute",note:"Flash-class tops HHEM; FaithBench confirms strong faithfulness; PROXY only"}, + "llama-3.3-70b|grounded_generation": {kind:"generator",band:{low:0.78,mid:0.84,high:0.9},benchmark:"FaithBench/RAGTruth",metric_kind:"absolute",note:"direct FaithBench measurement; faithfulness PROXY only; 70B tier"}, + "deepseek-v3|grounded_generation": {kind:"generator",band:{low:0.88,mid:0.93,high:0.96},benchmark:"Vectara-HHEM + DeepSeek blog",metric_kind:"absolute",note:"V3 outstanding faithfulness; faithfulness PROXY only; does not predict end-to-end RAG correctness"}, + "deepseek-r1|grounded_generation": {kind:"generator",band:{low:0.78,mid:0.86,high:0.91},benchmark:"Vectara-HHEM + DeepSeek blog",metric_kind:"absolute",note:"REASONING PENALTY on faithfulness; R1+RAG can hit 86% clinical accuracy despite poor faithfulness \u2014 proxy gap CRITICAL"}, + "qwen2.5-72b|grounded_generation": {kind:"generator",band:{low:0.75,mid:0.85,high:0.92},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; mid tier faithfulness proxy; wide band"}, + "mistral-large|grounded_generation": {kind:"generator",band:{low:0.75,mid:0.85,high:0.92},benchmark:"Vectara-HHEM (interpolated)",metric_kind:"relative",note:"interpolated; mid tier faithfulness proxy; wide band"}, + "claude-opus-4|review": {kind:"review",band:{low:0.62,mid:0.72,high:0.84},benchmark:"FaithJudge / MT-Bench (zero-shot LLM judge)",metric_kind:"absolute",note:"SOFTEST COLUMN; strong frontier judge tier; zero-shot only; few-shot raises to ~84% (not assumed here); wide band mandatory"}, "claude-sonnet-4|review": {kind:"review",band:{low:0.55,mid:0.65,high:0.78},benchmark:"FaithJudge / MT-Bench (zero-shot LLM judge)",metric_kind:"absolute",note:"SOFTEST COLUMN; mid judge tier; zero-shot only; RewardBench-2 shows ~20pp drop on harder distributions; wide band"}, "claude-haiku-4|review": {kind:"review",band:{low:0.48,mid:0.58,high:0.7},benchmark:"FaithJudge / MT-Bench (zero-shot LLM judge)",metric_kind:"absolute",note:"SOFTEST COLUMN; small judge tier; near-random on hard distributions; wide band; do not use as sole quality gate"}, "gpt-4o|review": {kind:"review",band:{low:0.55,mid:0.65,high:0.78},benchmark:"MT-Bench + FaithJudge",metric_kind:"absolute",note:"pairwise preference easier than absolute error-catch; MT-Bench bias documented (position/verbosity/self-enhancement); mid judge tier"}, @@ -105,11 +103,53 @@ "deepseek-v3|review": {kind:"review",band:{low:0.52,mid:0.63,high:0.76},benchmark:"FaithJudge (interpolated)",metric_kind:"relative",note:"SOFTEST COLUMN; interpolated; mid tier; CriticBench pairing asymmetry applies"}, "deepseek-r1|review": {kind:"review",band:{low:0.5,mid:0.62,high:0.76},benchmark:"FaithJudge (interpolated)",metric_kind:"relative",note:"SOFTEST COLUMN; reasoning overhead uncertain for critique; conservative mid tier; wide band"}, "qwen2.5-72b|review": {kind:"review",band:{low:0.5,mid:0.6,high:0.74},benchmark:"FaithJudge / CriticBench (no reliable evidence)",metric_kind:"relative",note:"SOFTEST COLUMN; conservative wide-band fallback per research guidance; no reliable evidence"}, - "mistral-large|review": {kind:"review",band:{low:0.5,mid:0.6,high:0.74},benchmark:"FaithJudge / CriticBench (no reliable evidence)",metric_kind:"relative",note:"SOFTEST COLUMN; conservative wide-band fallback per research guidance; no reliable evidence"} + "mistral-large|review": {kind:"review",band:{low:0.5,mid:0.6,high:0.74},benchmark:"FaithJudge / CriticBench (no reliable evidence)",metric_kind:"relative",note:"SOFTEST COLUMN; conservative wide-band fallback per research guidance; no reliable evidence"}, + "gpt-5.4-nano|drafting": {kind:"generator",band:{low:0.9,mid:0.93,high:0.96},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"CONFIRMED top-tier faithfulness; faithfulness subdim only; style/IF quality is relative-only and NOT captured"}, + "gpt-5.4-nano|grounded_generation": {kind:"generator",band:{low:0.9,mid:0.93,high:0.96},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"RAG reader faithfulness proxy; retrieval STEP is an embedder (see retrieval task)"}, + "gemini-2.5-flash-lite|drafting": {kind:"generator",band:{low:0.9,mid:0.93,high:0.96},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"CONFIRMED top-tier faithfulness; faithfulness subdim only"}, + "gemini-2.5-flash-lite|grounded_generation": {kind:"generator",band:{low:0.9,mid:0.93,high:0.96},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"RAG reader faithfulness proxy"}, + "gemini-2.5-pro|drafting": {kind:"generator",band:{low:0.82,mid:0.89,high:0.94},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"SOURCED (verification rate-limited); faithfulness subdim only; wider band"}, + "gemini-2.5-pro|grounded_generation": {kind:"generator",band:{low:0.82,mid:0.89,high:0.94},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"RAG reader faithfulness proxy; SOURCED"}, + "phi-4|drafting": {kind:"generator",band:{low:0.89,mid:0.93,high:0.96},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"SOURCED; small-model strong faithfulness; faithfulness subdim only"}, + "phi-4|grounded_generation": {kind:"generator",band:{low:0.89,mid:0.93,high:0.96},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"RAG reader faithfulness proxy; SOURCED"}, + "qwen3-8b|drafting": {kind:"generator",band:{low:0.86,mid:0.91,high:0.95},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"SOURCED; faithfulness subdim only"}, + "qwen3-8b|grounded_generation": {kind:"generator",band:{low:0.86,mid:0.91,high:0.95},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"RAG reader faithfulness proxy; SOURCED"}, + "kimi-k2.5|drafting": {kind:"generator",band:{low:0.7,mid:0.8,high:0.88},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"SOURCED; mid-low faithfulness; faithfulness subdim only; wide band"}, + "kimi-k2.5|grounded_generation": {kind:"generator",band:{low:0.7,mid:0.8,high:0.88},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"RAG reader faithfulness proxy; SOURCED"}, + "o3-pro|drafting": {kind:"generator",band:{low:0.58,mid:0.7,high:0.82},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"CONFIRMED reasoning-vs-faithfulness penalty; a reasoning model is a WORSE drafting prior than its base sibling; wide band"}, + "o3-pro|grounded_generation": {kind:"generator",band:{low:0.58,mid:0.7,high:0.82},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"CONFIRMED reasoning penalty; reader faithfulness proxy; wide band"}, + "o4-mini-high|drafting": {kind:"generator",band:{low:0.64,mid:0.75,high:0.85},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"CONFIRMED reasoning penalty; faithfulness subdim only; wide band"}, + "o4-mini-high|grounded_generation": {kind:"generator",band:{low:0.64,mid:0.75,high:0.85},benchmark:"Vectara-HHEM-2.3",metric_kind:"absolute",note:"CONFIRMED reasoning penalty; reader faithfulness proxy; wide band"}, + "gpt-5.5|drafting": {kind:"generator",band:{low:0.86,mid:0.92,high:0.96},benchmark:"NONE (interpolated from gpt-5.4-nano)",metric_kind:"relative",note:"INTERPOLATED from gpt-5.4-nano tier; no public benchmark for this exact name; wide band; refine from traces"}, + "gpt-5.5|grounded_generation": {kind:"generator",band:{low:0.86,mid:0.92,high:0.96},benchmark:"NONE (interpolated from gpt-5.4-nano)",metric_kind:"relative",note:"INTERPOLATED; reader faithfulness proxy; no public benchmark for exact name"}, + "claude-opus-4.8|drafting": {kind:"generator",band:{low:0.78,mid:0.86,high:0.93},benchmark:"NONE (interpolated from claude-opus-4)",metric_kind:"relative",note:"INTERPOLATED from Claude Opus 4 (modest gen-over-gen lift assumed); no public benchmark for exact name; wide band"}, + "claude-opus-4.8|grounded_generation": {kind:"generator",band:{low:0.78,mid:0.86,high:0.93},benchmark:"NONE (interpolated from claude-opus-4)",metric_kind:"relative",note:"INTERPOLATED; reader faithfulness proxy; no public benchmark for exact name"}, + "kimi-2.6|drafting": {kind:"generator",band:{low:0.7,mid:0.81,high:0.89},benchmark:"NONE (interpolated from kimi-k2.5)",metric_kind:"relative",note:"INTERPOLATED from Kimi-K2.5; no public benchmark for exact name; wide band"}, + "kimi-2.6|grounded_generation": {kind:"generator",band:{low:0.7,mid:0.81,high:0.89},benchmark:"NONE (interpolated from kimi-k2.5)",metric_kind:"relative",note:"INTERPOLATED; reader faithfulness proxy; no public benchmark for exact name"}, + "fable-5|drafting": {kind:"generator",band:{low:0.65,mid:0.8,high:0.92},benchmark:"NONE (no public evidence)",metric_kind:"relative",note:"NO PUBLIC BENCHMARK FOUND; deliberately wide conservative band; this is a placeholder to refine from your own traces, not a measurement"}, + "fable-5|grounded_generation": {kind:"generator",band:{low:0.65,mid:0.8,high:0.92},benchmark:"NONE (no public evidence)",metric_kind:"relative",note:"NO PUBLIC BENCHMARK FOUND; wide conservative band; refine from traces"}, + "qwen3-embedding-8b|retrieval": {kind:"generator",band:{low:0.62,mid:0.68,high:0.74},benchmark:"MTEB-English-v2 (Retrieval)",metric_kind:"absolute",note:"retrieval \u03c3 \u2248 P(relevant in top-k); leaderboard upper-bound, corpus-dependent; SOURCED"}, + "openai-text-embedding-3-large|retrieval": {kind:"generator",band:{low:0.52,mid:0.58,high:0.64},benchmark:"BEIR / MTEB-English",metric_kind:"absolute",note:"retrieval \u03c3 from BEIR nDCG@10; corpus-dependent; SOURCED"}, + "openai-text-embedding-3-small|retrieval": {kind:"generator",band:{low:0.48,mid:0.55,high:0.62},benchmark:"MTEB-English",metric_kind:"absolute",note:"cheaper/smaller embedder; corpus-dependent; SOURCED"}, + "cohere-embed-v3|retrieval": {kind:"generator",band:{low:0.5,mid:0.57,high:0.63},benchmark:"MTEB-English",metric_kind:"absolute",note:"corpus-dependent; SOURCED"}, + "e5-mistral-7b|retrieval": {kind:"generator",band:{low:0.52,mid:0.59,high:0.65},benchmark:"MTEB-English",metric_kind:"absolute",note:"7B LLM-initialized embedder; corpus-dependent; SOURCED"}, + "bge-m3|retrieval": {kind:"generator",band:{low:0.46,mid:0.53,high:0.6},benchmark:"BEIR / MIRACL",metric_kind:"absolute",note:"strong open multilingual embedder; corpus-dependent; SOURCED"}, + "granite-embedding-r2|retrieval": {kind:"generator",band:{low:0.48,mid:0.55,high:0.62},benchmark:"BEIR / MTEB-v2 Retrieval",metric_kind:"absolute",note:"compact ModernBERT embedder; corpus-dependent; SOURCED"}, + "qwen3-reranker|reranking": {kind:"generator",band:{low:0.65,mid:0.71,high:0.76},benchmark:"MTEB-R (over Qwen3-Embedding)",metric_kind:"absolute",note:"cross-encoder; post-rerank P(correct in top-k); corpus-dependent; SOURCED"}, + "cohere-rerank-3|reranking": {kind:"generator",band:{low:0.58,mid:0.65,high:0.72},benchmark:"BEIR / TREC-DL (cross-encoder)",metric_kind:"relative",note:"cross-encoder; relative-only public evidence; wide band; SOURCED"}, + "bge-reranker-v2-m3|reranking": {kind:"generator",band:{low:0.55,mid:0.61,high:0.67},benchmark:"MTEB-R (cross-encoder)",metric_kind:"absolute",note:"open cross-encoder reranker; corpus-dependent; SOURCED"}, + "rankzephyr-7b|reranking": {kind:"generator",band:{low:0.62,mid:0.69,high:0.75},benchmark:"TREC-DL19 / FutureQueryEval (listwise LLM)",metric_kind:"absolute",note:"listwise LLM reranker (post-ranking, NOT retrieval); recency penalty on novel queries; SOURCED"}, + "rankgpt-gpt4|reranking": {kind:"generator",band:{low:0.66,mid:0.72,high:0.78},benchmark:"TREC-DL19 (listwise LLM)",metric_kind:"absolute",note:"listwise LLM post-ranking; an LLM's legitimate role in retrieval is reranking, NOT the retrieval step; SOURCED"}, + "llm-judge-single|review": {kind:"review",band:{low:0.6,mid:0.68,high:0.74},benchmark:"RewardBench-2 (single LLM judge, k=1)",metric_kind:"absolute",note:"a single LLM-as-judge call; SOFTEST COLUMN; subject to position/verbosity/self-preference bias; SOURCED"}, + "llm-judge-ensemble|review": {kind:"review",band:{low:0.72,mid:0.8,high:0.85},benchmark:"RewardBench-2 (ensemble k=8)",metric_kind:"absolute",note:"ensembling N independent judge calls materially raises catch_rate (+~10pp); still bias-prone; SOURCED"}, + "human-reviewer|review": {kind:"review",band:{low:0.75,mid:0.85,high:0.92},benchmark:"paper assumption (domain-expert reviewer)",metric_kind:"relative",note:"human domain expert; higher catch than a single LLM judge but slower/costly; for code, a strong LLM judge can rival humans; refine from traces"}, + "corrector-no-feedback|correction": {kind:"correction",band:{low:0.1,mid:0.3,high:0.55},benchmark:"LLMs Cannot Self-Correct Reasoning Yet",metric_kind:"absolute",note:"corrector with NO reviewer feedback; unreliable \u2192 low/wide band; this is why detection (reviewer) and repair (corrector) must be modeled separately"}, + "corrector-with-feedback|correction": {kind:"correction",band:{low:0.55,mid:0.7,high:0.85},benchmark:"When Can LLMs Actually Correct Their Own Mistakes (TACL 2024)",metric_kind:"absolute",note:"corrector that CONSUMES reviewer feedback and re-does/patches; fix_rate is high only when the feedback signal is reliable; SOURCED"}, + "corrector-with-oracle|correction": {kind:"correction",band:{low:0.65,mid:0.78,high:0.9},benchmark:"Reflexion / agentic re-do with test or oracle signal",metric_kind:"absolute",note:"corrector with a verifiable signal (tests, exec results); highest fix_rate; degrades if the signal is noisy; SOURCED"}, }; - var MODELS = ["claude-haiku-4","claude-opus-4","claude-sonnet-4","deepseek-r1","deepseek-v3","gemini-2-flash","gemini-2-pro","gpt-4o","gpt-4o-mini","llama-3.3-70b","mistral-large","openai-o3","qwen2.5-72b"]; - var TASK_TYPES = ["code_generation","classification","extraction","drafting","retrieval","review"]; + var MODELS = ["bge-m3", "bge-reranker-v2-m3", "claude-haiku-4", "claude-opus-4", "claude-opus-4.8", "claude-sonnet-4", "cohere-embed-v3", "cohere-rerank-3", "corrector-no-feedback", "corrector-with-feedback", "corrector-with-oracle", "deepseek-r1", "deepseek-v3", "e5-mistral-7b", "fable-5", "gemini-2-flash", "gemini-2-pro", "gemini-2.5-flash-lite", "gemini-2.5-pro", "gpt-4o", "gpt-4o-mini", "gpt-5.4-nano", "gpt-5.5", "granite-embedding-r2", "human-reviewer", "kimi-2.6", "kimi-k2.5", "llama-3.3-70b", "llm-judge-ensemble", "llm-judge-single", "mistral-large", "o3-pro", "o4-mini-high", "openai-o3", "openai-text-embedding-3-large", "openai-text-embedding-3-small", "phi-4", "qwen2.5-72b", "qwen3-8b", "qwen3-embedding-8b", "qwen3-reranker", "rankgpt-gpt4", "rankzephyr-7b"]; + var TASK_TYPES = ["code_generation", "classification", "extraction", "drafting", "grounded_generation", "retrieval", "reranking", "review", "correction"]; // ---- public API ------------------------------------------------------------ @@ -119,6 +159,16 @@ /** List all task-type names in the table. */ function listTaskTypes() { return TASK_TYPES.slice(); } + /** List the task-types that have a prior for the given model. */ + function tasksForModel(model) { + return TASK_TYPES.filter(function (t) { return !!CELLS[model + "|" + t]; }); + } + + /** List the models that have a prior for the given task-type. */ + function modelsForTask(taskType) { + return MODELS.filter(function (m) { return !!CELLS[m + "|" + taskType]; }); + } + /** Return true if a prior exists for the (model, taskType) pair. */ function hasCell(model, taskType) { return !!(CELLS[model + "|" + taskType]); } @@ -126,7 +176,7 @@ * Seed a cockpit Node from a (model, taskType) prior. * * Returns an object ready to be merged onto a node: - * { model, task_type, is_prior: true, seeds, sigma_skill?, catch_rate, provenance } + * { model, task_type, is_prior: true, seeds, sigma_skill?, catch_rate?, fix_rate?, provenance } * * Mirrors Python priors.seed_node exactly. Throws if no cell exists. */ @@ -140,6 +190,9 @@ if (taskType === "review") { out.catch_rate = clamp(b.mid, 0, 1); out.seeds = "catch_rate"; + } else if (taskType === "correction") { + out.fix_rate = clamp(b.mid, 0, 1); + out.seeds = "fix_rate"; } else { out.sigma_skill = clamp(b.mid / GAMMA, 0.05, 0.98); out.catch_rate = 0.0; @@ -156,5 +209,9 @@ return out; } - return { listModels: listModels, listTaskTypes: listTaskTypes, hasCell: hasCell, seedNode: seedNode, GAMMA: GAMMA }; + return { + listModels: listModels, listTaskTypes: listTaskTypes, + tasksForModel: tasksForModel, modelsForTask: modelsForTask, + hasCell: hasCell, seedNode: seedNode, GAMMA: GAMMA + }; }); diff --git a/web/widgets/cockpit.html b/web/widgets/cockpit.html index 7de6698..acae0c2 100644 --- a/web/widgets/cockpit.html +++ b/web/widgets/cockpit.html @@ -36,10 +36,10 @@

MSO governance cockpit

- p_min0.80 - H(W)0.0 + Quality target0.80 + Process complexity0.0 @@ -87,19 +87,20 @@

MSO governance cockpit

- AI agent - tool / connector + generator (LLM) + retriever / reranker (not an LLM) + reviewer / corrector human - data + tool / data
-
C_op
—
-
B_eff
—
-
worst M*
—
-
motifs
—
+
Capacity ceiling (C_op)
—
+
Autonomy buffer (B_eff)
—
+
Worst masking (M*)
—
+
Fragile motifs
—
Press Run tokens to simulate tasks through this graph and compare empirical success to the analytic C_op.
-

Every value is computed by mso-core.js (Fisher info, capacity propagation, masking, water-filling, motif detection, delegation centrality), pinned to minimal-oversight==0.1.2 by the parity test. Connector parameters are illustrative defaults — edit any node to match your system.

+

Every value is computed by mso-core.js (Fisher info, capacity propagation, masking, water-filling, motif detection, delegation centrality), pinned to minimal-oversight==0.1.3 by the parity test. Component parameters are illustrative defaults — seed any node from public benchmarks (⊕) or edit to match your system. How the priors are built ↗