Repository navigation
test(roster): RT-07 Ask classification and bounded-slice planning #421
Description
RT-07: Ask classification and bounded-slice planning
Campaign: eve-post-development-20261011
EASY SUBSET AUTHORIZED: Ray authorized easy live tasks on 2026-10-11. Remaining cases and higher difficulty stay deferred.
| Field | Declaration |
|---|---|
| Work item | Acceptance scenario RT-07; not a runnable implementation issue |
| Board | Roster Test; In review |
| Priority / mode | P0 / Mixed |
| Candidate SHA / manifest | Easy subset: 1d53bda; minimal consumer baseline 8864bdb; full matrix not frozen |
| Attempts / result / evidence | 16 retained campaign samples; final candidate 3/3 reviewed / Pending review (partial subset) / Reviewed; no human acceptance |
| Owner / estimate | Copilot-operated easy subset authorized by Ray; measured samples linked below; remaining cases deferred |
Matrix: #414
Project: https://github.com/users/R4yHarris/projects/3
Spec: FEATURE_SPEC section 5.3: Work breakdown; honor sections 3 and 7.
Baseline: pending final reviewed/integrated candidate SHA and frozen configuration; never reuse an old uncommitted fingerprint.
Mode: Mixed | Priority: P0 | Initial result: Not run | Evidence stage: Not run
Outcome
Demonstrate the named operational behavior with attributable evidence, or record a failure, blocker or unresolved spec decision without claiming success.
Prerequisites
Operator procedure
- Submit a classification corpus through CLI issue intake and shell local asks: ambiguous ask, question, explicit bounded slice, broad feature and initiative.
- For the slice, provide a named outcome, concrete file allow-list and observable acceptance checks; inspect TASK, RECIPE and ESTIMATE before coder execution.
- For ambiguous and larger work, inspect clarification or PLAN-only handling and observe whether any product write or coder starts.
Acceptance checks
- Explicit outcome/files/checks produce a bounded slice regardless of the title; ambiguous work requests clarification.
- Large work stops at planning/splitting before coder execution. A planner cannot edit product code.
- Estimate records difficulty, minutes and available confidence before coding; acceptance checks and allow-list survive handoff without weakening.
- Record current clarify/slice/feature/initiative classifications separately from spec question/story/epic/incident terminology; unsupported incident semantics are a gap, not invented coverage.
Evidence and result
Use the matrix observation template. Append one redacted observation per actual attempt/case. Do not replace failed attempts with a later pass. Include run/seat IDs, exact interface, actual verification/review and human rating only when observed. Fixture results must be labeled fixture, not live operational evidence.
Fresh campaign starts at Not run. Previous comments, observations and evaluations are retained as historical evidence; they are not new-round results. Only observations tied to this candidate and campaign count. All required cases need new evidence and human review before Result becomes Pass. Unavailable prerequisites are recorded as fresh Blocked observations, not inherited old outcomes.
Safety and non-goals
This is TEST work, not a build task. No product/source/policy/workflow/vendor edits are requested; do not execute this issue as an implementation slice. Use disposable approved fixtures and separately approved bounded model tasks. No actual credentials, prompt/source bodies or transcripts may be posted.
No model calls, spend, publication, process termination or destructive cleanup are authorized by issue creation. Obtain explicit operator approval before those operations. Keep this issue only in Roster Test. Cross-link a separate Roster Build issue for any discovered defect; keep test failure evidence here and rerun after the fix.
Workflow and release gate
Backlog -> Ready only after dependencies, candidate freeze, disposable scope and budgets are approved. In progress means actual authorized execution; In review means evidence is awaiting independent/human review. Done/Pass requires all original required cases and genuine human acceptance, not a subset of green fixtures. A defect creates a separately scoped Roster Build issue; retain its failing observation here.
Plain-text Ask example
Write a short Markdown guide explaining the difference between a Git branch and a worktree, with one safe PowerShell example for listing each. Do not execute the examples or change existing files. Output it in a new "easy-git-guide" directory
Use: Easy-level live delivery sample approved by Ray on 2026-10-11; only this bounded subset is authorized now.
A prompt alone does not verify this scenario. Operator-controlled fixtures, actual evidence and genuine human review remain mandatory. Never execute a TEST issue as a product implementation issue.
Activity
r4yharris-agent-coder commentedon Oct 10, 2026
Fresh RT-07 observation
Fresh campaign eve-matrix-9f80afa-20261010; candidate 9f80afa, contracts 61b97b3e93e670a675c0301cce5412c73404d179. Windows, Node v20.20.2. Original matrix protocol #414; previous 28044dd observations are not carried forward.
Full existing suite: ROSTER_TEST_JOBS=2; npm exec --yes --package=node@20 -- node scripts\run-tests.mjs. Exit 0; 171 files, 1593 tests, 1586 passed, 0 failed, 7 skipped; 784.0s wall. Nine 60-second file-budget warnings retained. This is fresh fixture/regression evidence, NOT whole-scenario operational acceptance, idle benchmark or human AI-Eval. Missing platforms, actual TTY, multi-model corpus and publication variants are not replaced with fixtures.
Evidence stage denotes the highest evidenced subcase only; it does not certify every required gate. Human evaluation/reviewer: absent. No TEST issue was sent wholesale to roster run; no policy/workflow/vendor edits, saved-default changes, deployments or disposable-product publication. Board: https://github.com/users/R4yHarris/projects/3
Spec: FEATURE_SPEC 5.3, honoring sections 3 and 7.
Whole-scenario disposition: Blocked.
Classification/bounded planning fixtures pass. Supplementary live input failures retained; complete real CLI/TTY taxonomy corpus missing.
Fresh known lock failure is cross-linked to existing Build defect #441; Node20 code-planning failure is cross-linked to #454. Corrected live observations: one docs delivery/reviewer pass after automatic continuation, one code failure before coder; 1/2 reviewed yield, 0/2 first-pass yield, no human acceptance. Initial four experiment runtime/input mistakes and unavailable usage/exception evidence are retained, not relabeled as successful deliveries. See the new campaign summary on #414 for exact live outcomes and the complete 24-row disposition.
[-]TEST RT-07: Ask classification and bounded-slice planning[/-][+]test(roster): RT-07 Ask classification and bounded-slice planning[/+]r4yharris-agent-coder commentedon Oct 11, 2026
Easy subset E1 attempted on c57b0d3, task local-6c812b414308e124. Planner/critic stopped before coder: Plan critic revision failed: Repair did not emit valid tool_calls. Elapsed 162553ms. Live run, not a fixture; requested qwen3.8-27b, returned glm-5.3-flash. No publication, no human rating. Retain this failure; broader RT-07 cases remain unverified. Separate Build repair is being created; difficulty progression paused.
r4yharris-agent-coder commentedon Oct 11, 2026
Easy E1 remained unsuccessful on replay. | Sample | Task | Candidate | Seconds | Observation |
|---|---|---|---:|---|
| E1 | local-6c812b414308e124 | c57b0d3 | 162.6 | Failed before coder: critic protocol repair rejected JSON; repaired in #494/#495. |
| E1 | local-6895d9516fdf5d92 | 7e46115 | 165.3 | Failed before coder: proposed documentation checks rejected by action/content guard; not waived. |
| E2 | local-768f52ec4c6d2e39 | 7e46115 | 130.8 | Documentation created; model reviewer pass; independent content inspection; no suite run or human rating. |
| E3 | local-4a0d628808a59490 | 7e46115 | 49.2 | Failed in coder: empty function name, no product edits/tests/review; repaired in #496/#497. |
| E3 | local-709dc96db336bb60 | 7e46115 | 601.3 | Utility and seven tests written; scoped tests pass; cancelled at ten-minute deadline during final repository-wide verification; no reviewer. |
Documentation convergence blocker #498; do not mark this scenario Pass. Original required cases and human review remain unmet.
r4yharris-agent-coder commentedon Oct 11, 2026
Easy round completed: repaired candidate 1d53bda
Final candidate: 1d53bda. Latest easy corpus: 3/3 reviewed deliveries; human acceptance: 0.
All 16 attempts below are retained; overall yield across changing candidates/placements is not the final-candidate yield or a controlled benchmark.
Repairs and diagnosis
Limitations and progression
Minimal consumer baseline 8864bdb is local, disposable and has no remote; start is current/offline with freshness unknown. Real configured endpoint/models were used, not mocked model fixtures. All consumer runs explicitly warned that provenance events/sessions could not persist without a repository identity. Local logs, artifacts and this observation journal exist; these are not successful durable-history acceptance or a reason to invent a GitHub repository identity. No model default, endpoint, secret or policy changed; returned model labels remain response-backed.
Only easy actual tasks were run. No generated output publication, deployment, medium/hard task, concurrent workers or human AI-Eval. Full RT-07/RT-10 prerequisites and cases still require their own evidence; set Pending review, not scenario Pass/Done. Retain prior Fail observations.
Next difficulty gate: at least three distinct docs and three distinct code/test tasks per eligible profile with equivalent frozen inputs, repeated outcomes, measured review yield/rework/elapsed time and genuine human ratings as RT-18 requires. Current corpus is only two docs plus one code utility; no world-leading or stable multi-model performance claim. Advance one difficulty level at a time after that gate, not from these three green samples alone.
Scale-out gate later: GitHub parent/child issue outcomes and dependency waves; capacity-aware profile/hardware admission; isolated writer worktrees; response-backed context/memory/usage; cancellation/deadline/recovery evidence; independent review; App-only publication/merge. Reconcile #489/#476/#477 before authorizing simultaneous project execution. No new Kanban DB, competing runtime or cognitive role-play substitute for tested behavior.