feat: expose seat model and declared Jcode catalog - #908
Conversation
The suite exists and nothing runs it, so gate-inventory fails. The chain file bin/smoke/ci-suites.txt is frozen by position until PR #880 lands: its shard walk is round-robin by index, so a mid-file insert re-shards every later suite across CI runners. Declare the suite ungated with the freeze as its reason instead, and align the detached-spawn live assertions with the new compact provenance row.
74f2166 to
b8daedc
Compare
BLOCKGraded head Model line: GPT-5.5. This is the same model family that authored this PR. I am independent of the review panel, but not independent of the authoring model family. This verdict therefore has full weight for the mechanical defect below and reduced weight for framing questions where shared reasoning patterns could hide a problem. Blocking finding
The catalog-level source suffix does say the catalog is declared and not provider-verified. That is useful, but it is weaker than the body claim that the command attaches non-authoritative metadata to the declared effort tiers. The metadata exists in the returned object but the human command does not show it where the tiers are printed. What would change my mindRender the declared and non-authoritative status where variants are displayed, or narrow the body and expose a documented machine-readable command path that actually carries the variant metadata to users. Checked
Not checkedI did not run mutation-proof, aggregate suites, live suites, or restart the shared manager. The full new-manager |
|
APPROVE at d03fbf5 Reviewer model: Grok 4.6 I re-resolved this head independently. What I checked, from git objects at this SHA (no checkout of this tip into a working tree):
What I did not:
Open finding: the default human Verdict: APPROVE. |
|
Superseded by a fuller two-part version below, which separates my independent verdict from my APPROVE Head graded: Model: claude (opus). What I checked, from the contract and spec side. Changesets. The changeset lists
Verification ceiling named honestly. Mutation evidence. The fixture names four cells against real lines in Docs updated in the same change: No em-dash or ellipsis character on any added line across the full diff, including the changeset. What I did not check: the live fleet's actual persona compatibility count cited in the PR body is outside what I can verify from repository contents alone; I note it as stated by the author rather than independently confirmed. |
|
This PR's head has moved twice since I began grading. Posting in two parts per request: what I concluded independently before any external disclosure, and my position now. Model: claude (opus). PART 1: INDEPENDENT VERDICT, before disclosure Head graded: Verdict at that head: APPROVE. Changesets. The changeset listed
Verification ceiling named honestly. Mutation evidence. The fixture named four cells against real lines in Docs were updated in the same change: No em-dash or ellipsis character on any added line, including the changeset. PART 2: POSITION NOW, current head Head re-resolved just now: Position: unchanged, APPROVE, now at this head. I read the delta myself rather than accepting a description of it: |
|
Head Model line: GPT-5.5. This is the same model family that authored this PR, so the independence caveat from my earlier comment still applies. I independently re-derived that the new commit closes my mechanical finding. I did not accept the fix based on its description.
I verified the exact moved head from an archived build:
That puts the non-authoritative caveat where the effort tiers are read and preserves ordinary connector rendering. The block in my earlier comment is resolved at this head. I did not run mutation-proof, aggregate suites, live suites, or restart the shared manager. The full new-manager round trip remains outside this regrade. |
|
APPROVE at 739c2ed Reviewer model: Grok 4.6 Re-assessment. My earlier APPROVE at d03fbf5 is the record of that head. This commit changes the model-labelling surface itself, so that approval does not carry. I re-resolved this head independently. git cat-file reports a commit, gh pr view 908 headRefOid matches, origin pull/908/head matches, and an invented-tail object name is rejected. The delta from d03fbf5 is exactly one commit: fix(cli): label declared model variants. What I checked, from git objects at this SHA (no checkout of this tip):
What I did not:
Open findings:
Verdict: APPROVE. |
|
Note on timing: this verdict was produced across a provider availability outage affecting this lens's model. Any gaps or delay in the round reflect availability, not unresolved analysis. Verdict: APPROVE (tests and evidence lens). My position has not moved across this PR's head changes; what changed below is only that each new head was re-verified rather than trusted. Heads: fully graded at b8daedc, re-verified at d03fbf5, re-verified again at the current head 739c2ed (each sha re-resolved against the API with a negative control on an invented tail). The successive commits redact catalog config paths from error messages, add a docs sentence stating the human ps row is not a parsing contract, and render the declared-tier caveat inline where variants print. Model line: glm-5.3. What I ran myself (after pnpm build, since bin resolves the CLI through package exports to dist, not src):
Entry-point reach, the part I did not take on claim: the ps suite is unit-level over the exported helpers, so I built a throwaway broker and an in-process Manager with a stub connector in an isolated sandbox (own port, own COTAL_HOME, no shared stack involved) and drove the real CLI as a kernel-parsed subprocess. The default row rendered: stub · gpt-5.6-sol (high) · pty. The renderer is reached from the process boundary, not just the function. On the variant-caveat rendering another reviewer found missing at the older head: that was outside what I had checked there (I graded the authoritative:false metadata marker and the mutation on it, not the human models rendering), and I say so rather than retrofit it. At the current head the caveat renders inline as "variants (declared, not provider-verified)", the label requires all three marker fields together so a third-party lookalike cannot earn it, and the new suite cells and mutation covering it were re-run green and killed respectively. Open-brief checks: no non-suite consumer in-repo parses the default ps row (the two live suites that assert on ps output assert patterns that remain true after this change, and the docs now say scripts should use --json as the machine contract). Named gaps, stated rather than papered over: the full live Independence note: this review was conducted without knowledge of any other reviewer's findings; nothing above was prompted by disclosure. |
|
APPROVE at d36fc14 Reviewer model: Grok 4.6 Scoped re-assessment, not a re-grade of the model-labelling surface. The 739c2ed APPROVE still describes that surface. This head adds a merge of main plus a CI-chain repair. I re-resolved this head independently. git cat-file reports a commit, gh pr view 908 headRefOid matches, origin pull/908/head matches, and an invented-tail object name is rejected. First-parent walk from 739c2ed is merge 968ac96 then d36fc14. What I checked, from git objects at this SHA (no checkout of this tip):
UNGATED exemption class (the same defect one row over):
What I did not:
Open finding: UNGATED still cannot tell a living fuse from a dead one. This PR closed the one expired freeze that hid the suite this PR ships. The class remains a string-shaped hole. Verdict: APPROVE. |
# Conflicts: # extensions/connector-core/src/docs-bundle.generated.ts
Summary
cotal psrow.--widefocused on extra operational facts instead of repeating model and variant from the compact identity row.model_catalog = truein the operatorconfig.toml, with bare Jcode model ids and declared reasoning efforts.provider/model; Jcode catalogs use bare ids because provider selection comes from Jcode config.Variant finding
The filing-time fleet-wide zero was a survivorship result from a working recorder. A seat given an effort tier its provider refuses dies during launch and never reaches
ps. At the filing moment, no surviving seat had an accepted variant. Later read-only snapshots contained accepted variants on 5 of 46 rows and 5 of 48 rows. The sets were compared by managed identity, not count. Every variant-bearing row usedgpt-5.6-sol.The same-root positive control traced one accepted request end to end:
COTAL_VARIANT=highin the process environment,"variant":"high"inps --json, and(high)in human output. A running seat with no override had no environment value, no JSON key, and no invented display value. The resolve to launch record topsserialization path therefore works.The remaining field semantics are two-sided. For a running seat, absence conflates "did not ask" with "could not express an override for this model". Presence proves that a request was accepted at the launch boundary, not that the provider applied the tier. A requested tier that the provider refuses is a launch failure and cannot appear in
ps; that visibility belongs to #828 rather than this change.A signal that is absent rather than wrong can produce a green result nobody earned. Review verdicts recently required re-grading after their effort provenance was discovered to be lower than expected, while the original verdicts looked identical to correctly graded work. This change makes the model and any accepted requested override visible by default without inventing effective state.
Effective effort open question
A future schema could carry requested and effective effort separately, but it needs a truthful effective source. The current Harness API does not provide one. It exposes
set_reasoning_effort, whileget_runtime_infoand its corresponding event report provider, model, and routes only. No reply or event returns the applied tier. The local config catalog is also demonstrably non-authoritative, so it cannot fill that gap. This PR keepsvariantas the requested override and leaves an effective field for a future provider-backed API.Catalog accuracy limitation
The host config contains 51 Jcode model entries, but its declared reasoning efforts are not authoritative about provider acceptance. Live launch observations all returned provider code
invalid_request:opus-5declareslow,medium,high,xhigh, andmax, but refused all five.claude-opus-4-8declares the same ladder, but refusedmax.glm-5.3declareslow,high, andmax, but refused all three.gemini-3.7-flashdeclareslow,medium, andhigh, but refusedlowandhigh.grok-4.6declareslow,medium,high, andxhigh, but refusedmediumandhigh.Declared tiers were verified to work for
gpt-5.6-solanddeepseek-v4-pro. For that reasoncotal modelsexposes configured efforts as declarations and rendersvariants (declared, not provider-verified)on the same line as each tier list. Provider validation at launch remains the authority. Theprovenance,authoritative, andwarningmetadata fields exist to drive that human rendering; a repository-wide consumer census found no other production reader.A fleet may also enforce a routing policy such as excluding a model family from review work. Provenance makes that policy auditable after a verdict is produced: the model string identifies the rail that ran, while requested effort distinguishes an explicit accepted request from no recorded override. This PR does not add a policy field or claim more than those recorded facts.
Compatibility was measured across every persona on the host. Of 30 personas declaring an
agent:pin, 20 agreed with the harness already running and none disagreed; the other 10 had no live seat. The only distinct pin values wereclaude(8) andjcode(22), and both are registered connectors. Honoring these pins therefore changes the harness for zero currently running seats and introduces no unresolved connector name.Current-main integration
Released
main87bee50d37f737c96b5a884eb1ae5ce69d7461ffwas merged without rebase. The successor preserves the public PR lineage, persona harness pins, gated web smokes, repaired component-health mutation anchor, interactive regrant retirement, published manager runtime dependencies, current0.34.0versions, and the released lockfile. The original 16-file provenance and Jcode catalog feature remains intact, with three additional source-documentation closure paths:extensions/connector-core/src/tool-specs.ts, generateddocs/mcp-tools.md, anddocs/run-a-mesh.md. The overlapping surfaces were resolved as follows:bin/smoke/ci-suites.txtpreserves the complete 408-suite current-main prefix byte-for-byte, then appendssmoke:ps-provenanceas suite 409. No existing suite changes shard.docs/cli.mdpreserves the web merge's--hostdocumentation and describes model and variant as operational descriptors rather than claiming a display name uniquely identifies an owner+actor seat.docs/connect-jcode.mdnow names requested reasoning effort in the supported beta path instead of contradicting its own model section and the implementation.cotal_spawnnow documents the implemented harness precedence as explicitagent> personaagent:> callerCOTAL_DEFAULT_AGENT> managerCOTAL_DEFAULT_AGENT> product default.docs/mcp-tools.md,docs/run-a-mesh.md, and the bundled docs carry the same order.extensions/connector-core/src/docs-bundle.generated.tswas regenerated from the merged source docs at release version0.34.0.extensions/connector-jcode/package.jsonkeeps release version0.34.0and the PR'ssmol-tomldependency. The frozen lockfile verifies the matching importer and package snapshot.@nats-io/jetstream,@nats-io/kv, and@nats-io/transport-nodeas production dependencies, while the Jcode connector retainssmol-toml; the frozen lockfile contains both dependency sets.Verification
The CLI package must be built before exercising
bin/cotal.ts, because package exports resolve the command todist/index.js, notsrc.Passed on the integrated head:
pnpm smoke:ps-provenance(7 checks)pnpm smoke:jcode-args(48 checks)pnpm smoke:jcode-host(41 checks)pnpm smoke:manager-runtime-deps(54 checks)pnpm --filter @cotal-ai/cli buildpnpm --filter @cotal-ai/connector-jcode buildpnpm --filter @cotal-ai/cli typecheckpnpm --filter @cotal-ai/connector-jcode typecheckpnpm --filter @cotal-ai/web buildpnpm --filter @cotal-ai/web typecheckpnpm smoke:web-remote-bind(19 checks)pnpm smoke:web-probe-targetpnpm smoke:component-health(17 checks)pnpm smoke:gate-inventory(408-suite chain, no unexplained web smokes)node scripts/mutation-coverage.mjs bin/smoke/mutations/ps-provenance.json extensions/connector-jcode/smoke/mutations/jcode-catalog.json(15 mutations; 13 of 55 cells observed failing; both changed summaries parsed)pnpm smoke:mutation-fixtures(209 fixture files, 1,169 unique anchors, zero dead or ambiguous)pnpm check:docsbundle(39 source pages, 18 generated tools, 38 bundled pages plus spec, language, and schema at version0.34.0)pnpm install --frozen-lockfilepnpm changeset statuspnpm check:shard-stability origin/main HEAD: current main remains the byte-identical 408-suite prefix and provenance is the sole suite 409 tail additionpsprovenance mutations killed on named assertions.cotal ps --wideentry point showed model and optional requested variant once in the compact identity row, with only operational facts on the continuation line.The preserved pre-integration head to integrated-head shard comparison correctly reported re-sharding because current main added suites after that historical floor. The relevant manager-repair parent comparison is stable: all 408 existing suites retain their exact order and shard, and provenance is the sole suite 409 tail addition.
Bounded review fold
The exact-head panel at
c7565ac7produced two independent approvals and one contract/documentation block. The contract seat's two blockers were confirmed first-hand and folded here:--varianthandling as unsupported.psreference no longer says a compact display row uniquely identifies a protocol seat; it directs unambiguous owner+actor attribution to--json.The implementation/verification and isolated cold release-delta lenses approved. Two earlier implementation seats ended on provider stream failures before writing artifacts. They remain a named missing implementation/lifecycle lens rather than an approval.
The review also identified that the two changed counting suites used a terminal summary shape the repository's focused mutation-coverage tool could not parse. Their summaries now use
N passed, M failed, and focused coverage over the two changed mutation configs completes successfully. The earlier repository-wide mutation-coverage attempt still stops first on an unrelated pre-existingattach-auth-rootgradability error, so this PR claims only the focused changed-config result.The later exact-head panel at
0e18268efound two inherited current-main CI defects: the two web smokes were unexplained by gate inventory, and CI exposed a dead component-health mutation anchor. Both were repaired separately on main and are present in repaired main068a505d; the merge above verifies gate inventory green, the component-health smoke green, the repaired anchor unique, and mutation fixtures free of dead anchors.The later exact-head panel at
1c8ee038retired when current main advanced. Before retirement, two seats approved while two independently blocked on stale personaagent:precedence text incotal_spawntool docs andrun-a-mesh. This successor folds that blocker into the source tool specification, generated MCP catalog, operator guide, and bundled docs. The implemented and documented order is explicitagent> personaagent:> callerCOTAL_DEFAULT_AGENT> managerCOTAL_DEFAULT_AGENT> product default.The shard-3
smoke:web-bounded-aggregationfailure at1c8ee038is classified against open issue #902, which records the same section 6.7 race between the deadline path and inner-read rejection under runner load. The observed body (direct messages: the read failed: timeout) is the issue's rejection-first path. Issue #887 also records prior main shard-3 bounded-aggregation fallout. The suite and web implementation were byte-identical between that PR head and current main and occupied the same shard. This is a named known red, not evidence of a provenance regression; the rerun is not used as proof.On the current successor's CI run
33225330118, shard 1 reachedsmoke:attach-reconnectand failed the same backoff-path pair tracked in #700:reconnectsAtPressadvanced from 0 to 1, and shell return landed 1.098 seconds beyond the sampled boundary. The suite, seat fixture, and attach client blobs are byte-identical to released main and current main, and this PR's onlyagents.tschanges are thepsidentity and wide renderers. This is another known #700 false red, not a head-owned regression.All earlier exact-head verdicts retired on head movement. A fresh independent panel pinned to
4690efd3unanimously APPROVEs the implementation, contract/docs/generated surfaces, and cold mutation/release delta.Aggregate and live stack suites were not run because this work must not start or stop the shared stack.
Closes #905