Add expected-goals stat via continuous threat model by colonelpanic8 · Pull Request #279 · rlrml/subtr-actor

colonelpanic8 · 2026-07-12T18:47:53Z

What

Adds an expected-goals (xG) stat to the Rust core, built on a continuous team-threat model rather than shot detection.

Every live-play frame, we model each team's threat as V(state) = P(this team scores within 5s), a logistic model over 17 attacking-normalized physics features (ball kinematics, gravity-aware on-target projection, goal open angle, attacker/defender context). Derived stats fall out of V:

threat_touch events — V after a touch minus V before, i.e. the threat a touch added or destroyed.
threat_episode events — an above-threshold V span; its xG is the within-episode time integral Σ V·dt/τ (peak V is kept as a separate peak_value display field), credited to the last attacking toucher (a goal always closes its episode).
Accumulators — per-player threat-added and xG (credited episode integrals), plus per-team xG totals (the full-match integral, which includes diffuse sub-threshold threat no player is credited for — ~37% of the total).

Both events are player-scoped and currently hidden (no timeline projection yet).

Why not shot-gated

In Rocket League a "shot" is fuzzy (air dribbles rolled in, bounces, deflections), and gating xG on a shot classifier injects selection bias — only classifier-flagged touches plus all goals would enter the denominator. A continuous per-frame model avoids that and yields ~2,400 supervision rows per replay instead of a handful of shots. Discussed and chosen deliberately.

Model — `trained-v2`

Feature extraction lives in exactly one place (compute_threat_features); the new threat_dataset_dump bin exports training rows through that same code path, so training and inference cannot diverge.

Fit on 10.3M live-play rows from 5,280 rank-stratified ranked replays (rocket-sense production corpus, tiers 1–22), grouped train/test split by replay:

Metric	trained-v1	baseline	GBT ceiling
log-loss	0.169	0.252	0.154
Brier	0.047	0.064	0.043
AUC	0.885	—	0.906

Calibration tracks observed frequency within ~10% relative across all 15 prediction quantiles and across rank tiers.

Aggregation is validated through the shipping code path (threat_dataset_dump --episode-summary): the full-match integral recovers 3.62 mean goals per team-game vs 3.68 actual (within 2%, corr 0.75). Summing episode peaks instead would over-count 2.7× (9.87 vs 3.68) — which is why the integral, not the peak, is the exported xG. Aggregate identity check: integrating V·dt/τ recovers 6.72 mean goals/game vs 6.67 actual, per-replay corr(xG, goals) = 0.78.

Per-rank models? No. A single rank-blind model calibrates acceptably across the whole ladder, so xG means the same thing at every rank. Rank-as-feature is documented as a revisit trigger if drift grows.

Reproducing / retraining

Full pipeline in scripts/threat_model/: corpus fetch (production API, rank-stratified) → threat_dataset_dump → train_threat_model.py (emits a paste-ready coefficient block + parity fixture) → embed + bump THREAT_MODEL_VERSION.

Tests / checks

just check clean (fmt, clippy --workspace --all-targets --all-features, JS style, rdme).
Full lib suite green (914 passing), including feature mirror-symmetry, hand-computed ballistic on-target, episode state machine, ΔV emission, accumulator, and a Rust↔Python parity test pinning embedded f32 inference to the training pipeline's predictions.
cargo test -p subtr-actor-bakkesmod --no-run compiles; regenerated event-catalog TS bindings + event-definition docs are included.

Follow-ups (not in this PR)

Un-hide the events and build the threat-curve timeline in js/stat-evaluation-player/; wire xG into rocket-sense (schema-version bump); optionally swap the logistic for the GBT to capture the last ~15% of signal.

🤖 Generated with Claude Code

colonelpanic8

The continuous state-value approach is promising, and sharing Rust feature extraction between training and inference is a strong foundation. I think the PR needs changes before merge: the exported xG aggregation is not the aggregation evaluated in the model notes, touch attribution has correctness problems, the stats are not exposed through normal collector outputs, CI has two integration gaps, and the training environment is not reproducible. Inline comments contain the concrete changes I recommend. I do not think a separate design document is needed; the operational training README is sufficient once it is reproducible.

colonelpanic8 · 2026-07-12T19:18:28Z

+                .record_episode(event);
+        }
+        let team = &mut self.team_stats[usize::from(!event.team_is_team_0)];
+        team.xg += event.xg;


This accumulator calls the sum of thresholded episode peaks xG, but the reported 6.72-versus-6.67 validation integrates V*dt/tau instead. Those are different estimators. Threshold oscillation can count one developing chance repeatedly, while all sub-threshold threat contributes zero. Please validate this exact episode-sum aggregate on held-out replays before exporting it as xG, or keep these values named as threat/episode peaks for now.

Confirmed empirically, and you were right that these were different estimators. I validated the episode-peak-sum through the shipping calculator (new --episode-summary mode on threat_dataset_dump) over the 5,280-replay corpus: peak sums averaged 9.87 per team-game vs 3.68 actual goals — 2.7× over-counting, exactly the failure you predicted (V is calibrated per 5s window, not per episode).

Rather than rename, I changed the estimator to the calibrated one: episode xg is now the within-episode time integral Σ V·dt/τ, team xg is the full-match integral, and the old peak moved to a peak_value display field. Re-validated end-to-end through the same shipping path: full-integral xG 3.62 vs 3.68 actual per team-game (within 2%), corr 0.75; the episode-attributed (player-credited) share is ~63% of the team integral, and that gap is documented on the accumulator — diffuse sub-threshold threat is deliberately unattributed. Rationale + numbers are in scripts/threat_model/README.md.

colonelpanic8 · 2026-07-12T19:18:28Z

        {
            self.state.controlled_play.apply_event(event);
        }
+        let expected_goals = ctx.get::<ExpectedGoalsCalculator>()?;


The values are accumulated into StatsProjectionState here, but expected_goals is not registered as a builtin stats module, builtin_module_json has no output arm for it, and timeline snapshots omit it. Normal Rust/Python/JS stats consumers therefore cannot request the advertised player/team xG fields. This also leaves the builtin analysis-node JSON path failing in the BakkesMod checks.

Completed on top of caec086 (which covered the analysis-node JSON arm): expected_goals is now in builtin_stats_module_names(), has builtin_module_json / snapshot-frame / snapshot-config arms, and TeamStatsSnapshot/PlayerStatsSnapshot gained expected_goals fields populated in both the stats-timeline frame and playback-frames paths (TS bindings regenerated). The three live_abi_exposes_every_builtin_* BakkesMod tests pass — they iterate every module through the module/frame/config JSON paths. One caveat, documented in statsTimelineDerivation.ts: the JS event-derived materialization keeps zeroed defaults for this module until the threat events get a timeline projection (they're still hidden), so hydrated values flow through the Rust module/snapshot surfaces.

colonelpanic8 · 2026-07-12T19:18:28Z

+    }
+
+    fn emit_touch_events(&mut self, frame: &FrameInfo, touch_state: &TouchState, values: [f32; 2]) {
+        for touch in &touch_state.touch_events {


Every touch in TouchState receives the complete previous-frame-to-current-frame team value change. TouchState intentionally supports multiple simultaneous contacts, including same-team candidates, so the accumulator can count one transition two or more times. Please define a primary-touch, split-credit, or team-transition attribution policy and add a multi-touch test.

Fixed with a primary-touch policy: at most one threat_touch per team per frame, and the team's primary touch (the same latest-contact/evidence ordering TouchState::primary_touch_event already encodes, via a new primary_touch_event_for_team) receives the whole previous-frame→current-frame transition. simultaneous_same_team_touches_credit_one_event_for_the_frame_transition proves two same-team contacts on one frame yield exactly one event carrying the full ΔV.

colonelpanic8 · 2026-07-12T19:18:28Z

+                .previous_values
+                .map(|previous| previous[index])
+                .unwrap_or(values[index]);
+            self.touch_events.push(ThreatTouchEvent {


This combines touch.time with the current processing frame, while the probability delta also brackets detection-frame values. Touches recovered from the recent candidate cache can be backdated, leaving time, frame, and V samples referring to different moments. Preserve touch.frame and preferably touch_id, then explicitly define contact-time versus detection-time attribution.

The event now separates the two moments explicitly: contact-time fields time/frame/touch_id come from the underlying TouchEvent (previously frame was the detection frame paired with contact time, which was incoherent), and new detection_frame/detection_time fields carry the processing moment. The ΔV bracketing deliberately anchors on detection — V is only evaluated on processed live frames, and detection is the first frame reflecting the touch — and the doc comment says so. backdated_touch_keeps_contact_fields_and_detection_fields_separate exercises a backdated touch through the real update path.

colonelpanic8 · 2026-07-12T19:18:28Z

+        } else {
+            1.0
+        },
+        defenders_goalside: (defenders_goalside as f32 / team_size_norm).clamp(0.0, 1.0),


defenders_goalside is normalized by the attacking roster count. In uneven-team or leaver frames this produces incorrect fractions. Please compute a separate defending-team denominator. Because this changes a trained input feature, the embedded coefficients need to be retrained afterward.

Fixed — normalized by the defending team's eligible roster (same non-demoed filter used to iterate defenders) with a zero guard, plus a dedicated 2v1 test (defenders_goalside_normalizes_by_defending_team_size). Audited the other defender features (nearest-dist/-to-goal/-boost, in-net): none use a denominator, so this was the only instance. Retrained afterward as trained-v2 — metrics unchanged to 3 decimals (held-out log-loss 0.169, AUC 0.885) since uneven-team frames are rare, but the embedded coefficients and the parity fixture are regenerated from the corrected feature.

colonelpanic8 · 2026-07-12T19:18:28Z

+        self.episode_events.begin_update();
+        self.last_frame = Some((frame.frame_number, frame.time));
+
+        self.detect_goals(frame, gameplay, events);


Goal detection runs before pending-episode expiry, so a same-team goal arriving after closed_at + PENDING_EPISODE_GOAL_GRACE_SECONDS can consume and upgrade the episode before it is expired. Resolve stale pending episodes first, or enforce the age inside goal attribution, and add a late-goal-after-expiry test.

Enforced inside goal attribution rather than by reordering (robust regardless of within-frame call order): close_episode_as_goal now takes the goal time, and a goal arriving more than the grace period after closed_at flushes the pending episode unupgraded. Test: goal_after_pending_grace_expiry_does_not_upgrade_episode.

colonelpanic8 · 2026-07-12T19:18:28Z

+   Two attacking-normalized rows per sampled live-play frame (one per team),
+   with τ-agnostic goal-time columns for downstream labeling/censoring.
+
+3. **Train and evaluate** (needs numpy/pandas/scikit-learn):


The documented training command imports NumPy, pandas, and scikit-learn, but this PR declares or locks none of them; the repository Nix/uv environment also lacks pandas and scikit-learn. That makes both reproduction and coefficient provenance dependent on the ambient machine. PEP 723 metadata plus a uv script lock would be a good small setup; a directory-local pyproject.toml and uv.lock would be better if this pipeline grows.

Done — train_threat_model.py now declares numpy/pandas/scikit-learn in a PEP 723 block, pinned by a committed train_threat_model.py.lock (uv lock --script), and the README documents uv run --script. trained-v2 was actually fit through that locked environment, so the shipped coefficients' provenance is already reproducible. fetch_corpus.py stays stdlib-only.

colonelpanic8 · 2026-07-12T19:18:28Z

+PER_STRATUM = int(os.environ.get("PER_STRATUM", "150"))  # per (playlist, tier)
+PLAYLISTS = {"ranked-doubles", "ranked-duels", "ranked-standard"}
+
+TOKEN = subprocess.run(


Resolving a hard-coded personal pass entry at module import makes even importing or inspecting this script workstation-specific. Please accept the token through an environment variable or configurable token command, resolve it inside main, and make the base URL/cache directory configurable. pass show rocket-sense/token can remain a local fallback.

Done — the token resolves inside main() from ROCKET_SENSE_API_TOKEN, falling back to a configurable ROCKET_SENSE_TOKEN_COMMAND (default remains pass show rocket-sense/token for local use); ROCKET_SENSE_BASE_URL and THREAT_CORPUS_CACHE are also env-configurable. Importing the module no longer touches pass (verified: import has no side effects, TOKEN stays unset until main).

colonelpanic8 · 2026-07-12T19:18:28Z

+        "Evaluate the versioned logistic threat model V(state) for both teams on every live-play frame from full ball and player physics state.",
+        "On each attributed touch, emit the toucher's team's V just before the touch (previous live frame) and just after (the touch's frame).",
+    ],
+    hidden = true,


Because these definitions are intentionally hidden and have no timeline stream, the generated catalog now fails eventCatalogParity.test.ts. Please either make that test automatically ignore hidden_from_review entries or add these two keys to its explicit unsurfaced set; the current PR leaves the stats-player check red.

Took the first option: eventCatalogParity.test.ts now skips hidden_from_review entries generically, so the Rust hidden flag is the single source of truth and future hidden events need no per-key exception. I also dropped the two hardcoded keys your interim 66d92b0 added, since the generic skip covers them. Full stat-evaluation-player suite passes (235/235).

Model each team's threat V(state) = P(score within 5s) every live-play frame with a logistic model over 17 attacking-normalized physics features (ball kinematics, gravity-aware on-target projection, goal open angle, attacker/defender context). Shots are not a gating event: touches emit threat-delta events (V after minus before), and above-threshold threat spans close as episode events whose peak V is the xG, credited to the last attacking toucher. Accumulators derive per-player threat-added and xG plus per-team totals. Feature extraction lives only in compute_threat_features; the new threat_dataset_dump bin exports training rows through that same path, so training and inference cannot diverge. trained-v1 coefficients were fit on 10.3M rows from 5,280 rank-stratified ranked replays (tiers 1-22, rocket-sense production corpus): held-out log-loss 0.169 vs 0.252 baseline, AUC 0.885, calibration within ~10% relative across prediction quantiles and rank tiers, integrated xG recovering 6.72 goals/game vs 6.67 actual. A single rank-blind model calibrates acceptably across the ladder, so no per-rank models. Pipeline (corpus fetch, dataset dump, training, embed) is documented under scripts/threat_model/, and a parity test pins Rust inference to the training pipeline's predictions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rework the xG aggregation to the calibrated estimator: V is a per-5s-window probability, so summing episode peaks over-counts goals ~2.7x (9.87 vs 3.68 actual per team-game on the 5,280-replay corpus). Episode xg is now the within-episode time integral of V*dt/tau, team xg is the full-match integral (validated end-to-end through the shipping calculator: 3.62 vs 3.68 actual, within 2%), and the old peak moves to a peak_value display field. Per-player xg sums below team xg by design: diffuse sub-threshold threat (~37% of the integral) is not attributed to any player. Touch attribution now emits at most one threat_touch per team per frame, credited to the team's primary toucher, so simultaneous contacts cannot double-count a frame transition; events carry contact time/frame/touch id separately from the detection frame that anchors the V bracketing. Goals arriving after the pending-episode grace window no longer upgrade expired episodes. defenders_goalside now normalizes by the defending roster; the model was retrained after that feature fix (trained-v2, held-out log-loss 0.169 / AUC 0.885, unchanged) with a fresh parity fixture. Expected goals is now a registered builtin stats module end to end (module JSON, snapshot frames, config, timeline snapshots, TS bindings), the event catalog parity test skips hidden-from-review definitions generically, and threat_dataset_dump grows an --episode-summary mode that validates the shipped estimator through the real calculator path. Training scripts are reproducible: PEP 723 metadata + uv lock, and the corpus fetcher takes its token/base URL/cache from the environment instead of a hard-coded pass entry resolved at import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The parity test now skips hidden_from_review entries generically, so the hardcoded threat_episode/threat_touch keys added in 66d92b0 are covered by declaration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

colonelpanic8 commented Jul 12, 2026

View reviewed changes

colonelpanic8 and others added 5 commits July 12, 2026 14:26

Expose expected goals through analysis node JSON

584f4dd

Except hidden threat events from player parity

01662c1

Drop redundant per-key parity exceptions for hidden events

c5afec3

The parity test now skips hidden_from_review entries generically, so the hardcoded threat_episode/threat_touch keys added in 66d92b0 are covered by declaration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

colonelpanic8 force-pushed the claude/expected-goals-stat-382a6b branch from 66d92b0 to c5afec3 Compare July 12, 2026 21:28

colonelpanic8 and others added 7 commits July 12, 2026 14:33

Regenerate stat definition docs for integral episode xG

7373d56

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Finalize expected goals review fixes

6866dbf

Make expected goals opt-in with symmetric 2v2 features

60ae49f

Show live threat in the stats player

45a6f7c

Add incident xG aggregation and timeline

0efa8af

Update timeline view source contract

95488f4

Fix live stats opt-in module exports

a82f751

Uh oh!

Conversation

colonelpanic8 commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why not shot-gated

Model — trained-v2

Reproducing / retraining

Tests / checks

Follow-ups (not in this PR)

Uh oh!

colonelpanic8 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

colonelpanic8 commented Jul 12, 2026 •

edited

Loading

Model — `trained-v2`