Cost-aware allocation: model registry, budget optimizer, agentic templates - #32
Open
crbazevedo wants to merge 13 commits into
Open
Cost-aware allocation: model registry, budget optimizer, agentic templates#32crbazevedo wants to merge 13 commits into
crbazevedo wants to merge 13 commits into
Conversation
…ndation) Phase 1 of cost-aware allocation. Two pure-data layers the optimizer needs: - model_registry.yaml + registry.py: per-model provider, license (open-weights AND commercially-usable flag), modality (llm/embedding/reranker), input/output $/Mtok, and a DERIVED 1..100 log-scale cost_index (store dollars, derive the index). Helpers: cost_per_run, is_open_source, blended_cost, open_models, models_by_modality. 37 models; OSS cluster cheap-to-mid (index 1–47), proprietary frontier at the top (67–96). Costs provisional — refined from the pricing research sweep; Mistral-Large flagged open-weights-but-non-commercial. - complexity.py: complexity tiers (trivial→critical) as competence multipliers, plus tool-misuse penalty: σ_eff = σ_prior · factor · (1 − tool_misuse). required_prior_sigma inverts it — the prior σ a model needs to clear a node's bar (so hard/critical nodes demand stronger, usually proprietary, models). 17 new tests; full suite 115 pass. Optimizer (Phase 2) builds on these. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 2: a greedy structural local-search that runs both directions the
operator asked for, on cockpit-shaped node specs.
optimize.py:
- node_params / node_cost / evaluate: derive (σ_skill, catch, fix) and $/run for
a node from its model × task × complexity (+ tool-misuse), using the priors
and the registry. In-place oversight (a reviewer+corrector AT a node, not a
new delegation hop) lifts σ_corr without skill-damping — the faithful
"invest in review" lever.
- candidate_models / prescribe_node: the cheapest model that clears a node's bar,
open-source preferred; flags oss_can_do vs needs_proprietary (hard/critical
tasks → proprietary). Unpriced models are excluded from cost optimization.
- optimize_allocation: Phase A invests (model upgrade or in-place review, max
ΔC_op) until C_op ≥ p_min; Phase B divests (cheaper model, cheaper/dropped
review) while holding the bar and fitting the budget. Verified:
* cheap qwen3-8b draft below 0.80 → invests review, trims to phi-4 → 0.875 feasible
* opus+opus $0.105 → phi-4(OSS)+gemini-flash $0.00044, feasible, within budget
Pricing refreshed from the research sweep (most CONFIRMED 3-0): GPT-5.4-nano
$0.20/$1.25, GPT-5.5 $5/$30, Claude Opus 4.8 $5/$25 (newer Opus tier), Gemini
2.5 Flash $0.30/$2.50, Haiku 4.5 $1/$5, Kimi K2 $0.60/$2.50. Registry status →
sourced.
8 optimizer tests; full suite 123 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cites the 2026-06-18 pricing sweep behind model_registry.yaml — input/output $/Mtok + license per model, mostly CONFIRMED against official pricing pages — plus the cost-index derivation and caveats (per-search billing, batch tiers, self-host economics, unpriced models excluded from optimization). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 3a — the browser layer for cost-aware allocation, generated from the same YAMLs as Python so it can never drift: - web/mso-registry.js (gen_registry_js.py): model cost/license/index helpers. - web/mso-optimize.js: full port of optimize.py + complexity — nodeParams, nodeCost, evaluate, candidateModels, prescribeNode, optimizeAllocation. Reuses mso-core.js for capacity propagation. - mso-priors.js gains priorMid() (raw band mid for the optimizer). - Parity: tests/test_parity.py now pins effective_sigma, cost_per_run/index, evaluate, candidate-model ordering, AND the full optimizer outcome — JS ↔ Python agree on c_op, cost, feasibility, the step list, and the final per-node model assignment (verified identical on both directions). - Drift guards for both generated bundles. 124 tests pass. Cockpit UI (Phase 3b) builds on this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 3b — drives the cost-aware optimizer from the cockpit. Inspector (per node): - Model dropdown (cost-sorted, OSS-tagged) — assigning a model derives the node's σ_skill / catch / fix from the (model × task × complexity) prior. - Task-complexity selector (trivial→critical) — harder ⇒ needs a stronger model. - "Model & cost" readout: chosen model + OSS/proprietary badge + $/run, and the prescription "min model that clears target" with an open-source / needs- proprietary verdict. Analysis panel: - Total pipeline cost ($/run) + an optional budget input. - "⚡ Optimize allocation" — runs the optimizer (both directions): invests model upgrades / in-place review to reach the quality target at min cost, or trims an expensive pipeline to the budget while holding the bar. Reports new C_op, cost, Nx-cheaper, within-budget, and the step list. Model-less nodes (tools, hand-set) pass their explicit σ through node_params, so the optimizer's C_op agrees with the cockpit's analyze() (matching passthrough added to optimize.py + mso-optimize.js, parity preserved). Browser-verified: opus+opus $0.105 → phi-4(OSS)+gemini-flash $0.000445 (236× cheaper), C_op 0.814 ≥ target, within budget — no console errors. 124 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…suse
Phase 4 — the realistic workflow bank the optimizer was built for.
- New components: "Tool caller (MCP agent)" (Generate) and "MCP gateway" (Dev,
deterministic connector to real systems).
- Tool-misuse slider per node ("agent calls a tool wrong/unsafe") — feeds the
optimizer's effective σ (σ_eff = σ_prior · complexity · (1−tool_misuse)).
- nd() now carries model / complexity / tool_misuse and derives σ from the prior
when a model is pre-loaded; loadTemplate applies it.
- Two templates with models PRE-LOADED (deliberately expensive — claude-opus-4
across LLM nodes — so ⚡ Optimize has work to do):
* "Agentic RAG + rerank": query-rewrite → embed → KB search → rerank →
grounded reader → groundedness judge. Verified: opus-everywhere $0.196,
infeasible 0.735 → Optimize reaches feasible 0.800, 3.7× cheaper, swapping
to OSS (phi-4 / qwen3-embedding / qwen3-reranker) and investing review.
* "Tool-using agent (MCP)": planner → tool caller (tool_misuse 0.15) → MCP
gateway → output verifier.
Browser-verified end-to-end, no console errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Bigger node cards (156×54) render three lines: label, then model · cost ($/1k-runs) when a model is assigned (OSS shown in green), then component type + masking badge. shortModel() abbreviates long model names; truncW() clips each line to width so text never overflows the card. - Workflow templates now route through the layered auto-layout (left-to-right by depth) so the larger cards never overlap and edges stay clean. Lessons keep their hand-tuned coordinates (their action buttons reposition specific nodes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…onomy Connectors (realistic sources/sinks/APIs), grouped in the palette: - Trust: Auth/SSO (Auth0), KYC (Plaid), Payments (Stripe), Fraud/risk, Billing. - Apps: Salesforce, HubSpot, ERP (SAP/NetSuite), ITSM (ServiceNow), DocuSign, Zendesk. - Dev: GitHub, Jira, CI/CD, PagerDuty, Datadog, MCP gateway. - Comms: Slack, Email (SendGrid), SMS/WhatsApp (Twilio), Chat widget. - Data: Datastore, Vector DB (Pinecone), SQL/warehouse, Knowledge base, File store (S3), Web search, EHR/FHIR (Epic), External API, enrichment. 16 new templates with models PRE-LOADED, grouped by industry in the menu: - Commerce: Conversational commerce (14-node multi-agent: chat→auth→intent→ product search→rerank→reco→cart→checkout→fraud→payment→order→confirm→ post-sales), Returns & refunds. - Support/Knowledge: ticket triage, enterprise KB assistant. - Finance/Risk: loan underwriting, KYC/AML, insurance claims. - Healthcare/Legal: patient intake + clinical support, contract review. - Software/Ops: AI code review, SRE incident response, SOC alert triage. - Ops/Growth: HR resume screening, marketing content, analytics NL→SQL, supply-chain reorder. Each routes through the layered auto-layout (tn() helper, no hand coords) so the larger cards never overlap. Browser-verified: all 16 load, 0 errors, 0 overlaps, C_op spread 0.705–0.870 (several infeasible → real optimizer/review work). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…value) Each workflow now shows a one-line description under the toolbar explaining the flow and what to try (e.g. "infeasible at 0.80 — Optimize, then lower the target to find the cliff"). Helps users navigate the 16+ industry templates and points them at the optimizer. Hidden for lessons (which show their own hint box). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t-value) Land on "Support ticket triage" (RAG + judge + human escalation, models & cost visible, feasible) instead of the sparse sales demo, and sync the dropdown so the selection matches the canvas. Better first impression of the cost-aware features; the masking lessons remain one click away under Lessons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…port
Applies the 2026 UX/product-design research sweep (4 confirmed pillars):
- Optimizer trust (PAVED EuroVis 2020 + AutoML IUI 2020): "⚡ Optimize" no longer
auto-mutates the pipeline. It previews a small cost↔quality frontier ("Min cost
· meets target", "More headroom", "Fit budget"), each card LEADING with the
outcome (C_op, $/run, Nx-cheaper, open-source share) and DEMOTING the move list
to a collapsible "what changed" — the user applies one a-posteriori.
- Export (PAVED store/export): download the pipeline + its governance numbers
(C_op, B_eff, cost) as JSON, complementing Import.
- Time-to-first-value: realistic model-loaded default, per-template descriptions,
industry-grouped menu (shipped in prior commits).
Adds docs/methodology/cockpit-ux.md citing the sources (Munzner, ELK, Eiband
2021, Google PAIR, Drozdal IUI 2020, PAVED EuroVis 2020) — repo's sources-
disclosure discipline. Browser-verified: preview does not mutate until Apply;
export builds valid JSON; no console errors. 124 tests pass, mkdocs --strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surfaces the paper's full apparatus as practical, one-click heuristics — and exposes drift, which the cockpit had been ignoring entirely. Theory readout (live): - Autonomy runway T*_auto = B_eff / μ_eff (Eq. 17) with μ_eff = the worst-drifting node's rate — how long autonomy lasts before drift forces intervention. - Capacity cliff H_crit = (C_op − p_min)/λ (Demonstration 7) + headroom vs H(W). Recipe library (each grounded in an equation, with a computed value / apply): - "Run N× longer" — scales node drift by 1/N (stabler models / lower temp / caching), making T*_auto ≈ N× longer. (Your example.) - "Max safe process load" — sets H(W) to 80% of H_crit: the most routing/tool variety the workflow can absorb before autonomy is impossible. - "Min review to hit target" — corrector capacity threshold K/N for the bottleneck: the minimum fraction of outputs that must be reviewed. - "Where to invest oversight" — water-filling allocation α*(x) (Eq. 8), ranking nodes by Fisher-weighted marginal oversight value. - "Maximize autonomy buffer" — decomposes B_eff (Eq. 16) into its three levers and quantifies the drift-time bought per +0.01 of buffer. Browser-verified: readout renders, all five recipes return correct grounded output, "Run 2×" halves drift, no console errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A clean reference mapping each MSO quantity (σ_raw/σ_corr, masking, effective skill, Fisher info, water-filling α*, C_op, H(W), B_eff, H_crit, T*_auto, drift μ_eff, corrector K/N, delegation centrality, SOTA priority, motifs, cost) to exactly where it is seen and used in the cockpit, plus the recipe library and the equation each recipe applies. Linked from the cockpit footer and mkdocs nav. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delivers the cost/budget/optimizer feature plus a realistic workflow gallery and research-backed UX. Stacked on #31 (base retargets to
mainas #30/#31 merge).What you can do
Pick from 25+ workflow templates across industries (commerce, support, finance/risk, healthcare, legal, software/ops, HR, marketing, analytics, supply-chain, security), assign models per node, set a budget, and hit ⚡ Optimize allocation — it previews a cost↔quality frontier (open-source preferred) and you apply the one you want.
Engine (Phases 1–5)
σ_eff = σ_prior · complexity · (1 − tool_misuse).optimize.py↔mso-optimize.js, parity-pinned) — greedy structural search both directions;prescribe_node(cheapest clearing model + OSS verdict); in-place review investment (not a damping hop).Workflow gallery & cockpit polish
docs/methodology/cockpit-ux.md.Verified
mkdocs --strictclean.🤖 Generated with Claude Code