feat(priors): cold-start σ_raw prior table — 78 cited per-model cells - #30
Open
crbazevedo wants to merge 3 commits into
Open
feat(priors): cold-start σ_raw prior table — 78 cited per-model cells#30crbazevedo wants to merge 3 commits into
crbazevedo wants to merge 3 commits into
Conversation
…aceholder data) Scaffolds the cold-start path so a node can be seeded from a (model, task_type) prior when the user has no traces yet — to be refined later by their own logged outcomes. - data/priors.yaml: schema for low/mid/high sigma_raw bands (generator tasks) / catch_rate bands (review task) with provenance (benchmark, metric, metric_kind, source_url, normalization_note). Currently PLACEHOLDER abstract-tier rows; cited per-model values from the research pass replace them next. - priors.py: load_priors/list_models/list_task_types/seed_node. seed_node maps a cell to Node kwargs — generator -> sigma_skill = mid/gamma (reproduces the prior's sigma_raw at the return-operator fixed point), review -> catch_rate — plus a provenance block with a (1 - band_width) confidence proxy. - tests: fixed-point round-trip, review->catch_rate, band/provenance validity, loader rejection of non-monotone bands / review-without-catch / unknown task. Hatchling bundles data/priors.yaml in the wheel (verified). Not wired into the cockpit/notebook yet — that lands with the curated numbers + a JS parity port. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cells) Replace placeholder rows with 78 per-model × task-type prior cells sourced from a multi-agent deep-research sweep (109 agents, 6435k tokens, 622 tool uses). Each cell carries low/mid/high σ_raw or catch_rate bands, primary benchmark, metric_kind, source URL, and a normalization note. Sources by task-type: - code_generation: LiveBench Coding (Table 1, Jun-2024 snapshot); cross-check HumanEval vs SWE-bench-Verified to justify wide bands - classification: NO surviving absolute benchmark (MMLU vendor proxy refuted 2-1); conservative tier-interpolation cells flagged metric_kind=relative - extraction: JSONSchemaBench (EPFL-DLab, Feb 2025, arXiv 2501.10868); schema- difficulty × constrained-decoding dominates over model family - drafting: Vectara HHEM-2.3 + FaithBench/RAGTruth faithfulness subdim (absolute) for grounded accuracy; style/IF quality is RELATIVE-ONLY and not captured in σ - retrieval: same HHEM + FaithBench source; explicit note that faithfulness ≠ end-to-end RAG correctness (DeepSeek-R1 divergence documented) - review (catch_rate): FaithJudge zero-shot (o3-mini-high 68.8% bal-acc ceiling), MT-Bench (GPT-4 ~80-85% pairwise), RewardBench-2 (~20pp harder-distribution drop), CriticBench pairing asymmetry — widest bands, softest column Key normalization decisions: - ABSOLUTE metrics (pass@1, faithfulness rate) → σ_raw with modest domain-shift discount; band widened when evidence is stale or interpolated - RELATIVE metrics (AlpacaEval, Arena Elo) NOT inverted to σ; used for ranking only - DeepSeek R1 reasoning penalty on faithfulness explicitly captured - Classification cells marked for immediate trace refinement All 8 test_priors tests pass; 85 total, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- web/mso-priors.js: JS port of priors.seed_node; bundles all 78 cited (model × task) cells inline; exports seedNode / listModels / listTaskTypes / hasCell / GAMMA; UMD-compatible (browser globals + Node require) - web/widgets/cockpit.html: "⊕ Seed from model & task…" button in node inspector; opens modal with model/task dropdowns auto-guessed from node label and connector type; live preview shows σ_skill or catch_rate + band + confidence + benchmark name + relative-evidence warning; applying the prior stores provenance on n._prior and renders a chip in the inspector; manually adjusting any slider clears the prior - scripts/parity_runner.js: added seed_node output block, requires mso-priors.js - tests/test_parity.py: added seed_node parity cases (4 cells covering generator + review branches); imports priors.seed_node; asserts JS ↔ Python agree to 1e-6 on sigma_skill / catch_rate / band / confidence All 85 tests pass, 1 skipped (no Node; parity test self-skips). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/minimal_oversight/data/priors.yaml— 78 cited per-model × task-type cells from adversarially-verified deep research (78 cells, 13 models × 6 task types). Each cell carries σ_raw band (or catch_rate band for review), primary_benchmark, metric_kind (absolute|relative), source URL, and normalization note.src/minimal_oversight/priors.py— Pythonseed_node(model, task_type)that maps a prior cell onto Node kwargs + provenance (σ_skill for generators via fixed-point GAMMA inversion; catch_rate for review nodes).web/mso-priors.js— JS port ofseed_node; all 78 cells bundled inline; UMD-compatible (browser + Node require).web/widgets/cockpit.html— "⊕ Seed from model & task…" button in node inspector: modal with auto-guessed model/task dropdowns, live preview (band + confidence + benchmark), prior chip displayed in inspector, slider edits clear the prior.scripts/parity_runner.js+tests/test_parity.py—seed_nodeparity cases (4 cells, generator + review branches); JS ↔ Python must agree to 1e-6.Cold-start problem solved
Users with no traces can now seed cockpit nodes from public benchmark evidence instead of guessing σ values. The seeder is labeled as a prior (confidence score, relative-evidence warning) to discourage over-trusting it.
Test plan
uv run pytestor.venv/bin/python -m pytest)seed_nodeJS/Python agree to 1e-6 on σ_skill, catch_rate, band, confidence🤖 Generated with Claude Code