feat(orchestrator): hierarchical GRPO for proposer-solver envs

[hallerite]

What

Adds the hierarchical_grpo algorithm — two-level GRPO baselines for task-generating envs like proposer-solver-v1, where the proposer is asked group_size times to invent a problem and each proposal fans out to env.n solvers:

• HierarchicalGRPOAlgorithm (algo/hierarchical_grpo.py): every trainable trace is mean-centered against its peer set. Agents in episode_agents (the solvers) center against the same-agent traces of their own episode — sibling attempts at the same minted task; every other agent (the proposer) centers against its same-agent traces across the group — parallel attempts at the same source task, each rewarded by what its minted problem did to the solvers (proposer-solver-v1's learnability, 4p(1−p)). Rewards never mix across agents or minted tasks; singleton peer sets center to zero (GRPO's convention — the zero-advantage filter drops them).
• HierarchicalGRPOAlgoConfig (type = "hierarchical_grpo", required episode_agents) in the discriminated union. The field is required on purpose: which agents are episode-scoped is env truth the algorithm must not guess.
• configs/debug/algo/hierarchical_grpo.toml — proposer-solver-v1 with Qwen3-4B, 4 proposals per topic (group_size = 4) × 4 solvers per proposal (env.n = 4), both sides trainable, eval pinned to temperature = 1.0 (at the model's greedy default the thinking proposer reliably fails to emit the JSON contract).
• Docs (algorithms.md incl. a "Hierarchical Advantage (Proposer–Solver)" section, training.md) and the configs skill updated.

Why

The proposer-solver episode tree breaks the flat-group assumption twice over: pooling solver rewards across episodes baselines them against attempts at different problems, and pooling proposer and solver traces mixes reward scales across agents (learnability vs. correctness). Plain GRPO does both silently. Hierarchical GRPO assigns each level its own GRPO baseline: solver advantages are computed GRPO-style within their proposed task, proposer advantages GRPO-style across its group_size proposals.

Stacked on #3137 (verifiers bump); like rae (#3136), this is a named Algorithm of the kind #3108's single-trainable-agent gate points multi-agent envs toward. Touches the same docs/config tables as #3136 — whichever merges second has trivial one-line conflicts.

Test plan

• uv run pytest tests/unit/orchestrator tests/unit/test_configs.py -q — 204 passed, including 3 new tests (solver centering within episode vs. pooled mis-crediting, proposer centering across the group with no cross-agent leakage, the full 2×2 proposer-solver shape).
• 2-GPU e2e smoke with Qwen3-4B on this branch: uv run rl @ configs/debug/algo/hierarchical_grpo.toml --max-steps 3 — 3 orchestrator + 3 trainer steps (train rewards 0.54 / 0.68 / 0.13), eval at temperature = 1.0 healthy (reward 0.63, 17% episode errors vs. 100% at the greedy default), exit 0.

🤖 Generated with Claude Code