feat(orchestrator): RAE — role-conditioned advantage estimation for self-play envs (SPIRAL)

[hallerite]

What

Adds the rae algorithm — SPIRAL's role-conditioned advantage estimation (arXiv:2506.24119) — as a named Algorithm for multi-agent self-play envs like kuhn-poker-v1:

• RAEAlgorithm (algo/rae.py): each trainable agent keeps an EMA baseline of its own rewards; every trace's advantage is reward − baseline, scored against the pre-update baseline (the reference implementation's unbiased order), then folded in at decay (SPIRAL's α, default 0.95). Baselines are keyed by agent_name on a per-env instance — the paper's per (game, role). No normalization, and group_size is free (RAE needs no sibling rollouts; group_size = 1 is fine).
• RAEAlgoConfig (type = "rae", decay) in the discriminated union.
• configs/debug/algo/rae.toml — shared-policy Kuhn poker self-play (both agents late-bound to the run's own model).
• Docs (algorithms.md incl. a "Self-Play Advantage (RAE)" section, training.md) and the configs skill updated.

Stacked on #3137 (verifiers bump — brings kuhn-poker-v1 and the interleaving-agents Interaction API); this PR carries only the RAE change.

Why

Group-relative baselines assume the group is exchangeable attempts by one agent. A self-play episode yields one trace per agent, all trainable, and in a zero-sum game the rewards sum to ~0 whatever the policy does — the group mean carries no information, and centering both agents against it converts structural asymmetries (the first-mover edge) into permanent credit for one agent. SPIRAL's ablations show self-play without role baselines collapses (thinking collapse, gradient spikes); per-agent EMA baselines measure each agent against its own expected reward.

This is the "named Algorithm" that #3108's single-trainable-agent gate points multi-agent envs toward — the two compose: grpo keeps refusing mixed-agent groups, rae is the estimator that handles them.

Test plan

• uv run pytest tests/unit/orchestrator tests/unit/test_configs.py -q — 204 passed, including 3 new RAE tests (zero-init baselines, per-agent separation + persistence across groups, pre-update scoring order).
• 2-GPU e2e smoke on this branch: uv run rl @ configs/debug/algo/rae.toml --max-steps 3 — 3 orchestrator steps with both seats training and 3 clean trainer steps, exit 0.

🤖 Generated with Claude Code