Skip to content

program.md

Latest commit

 

History

History
149 lines (116 loc) · 5.34 KB

program.md

File metadata and controls

149 lines (116 loc) · 5.34 KB
Raw

autoresearch — March Madness

Autonomous optimization of NCAA tournament spread predictions.

Setup

  1. Agree on a run tag: e.g. mar29
  2. Create the branch: git checkout -b autoresearch/<tag>
  3. Read these files for full context:
    • config.py — metric (MAE), guard rails, known dead ends
    • prepare.py — data loading (1,048 tournament games, 2009-2025) + evaluation
    • experiment.py — the spread formula you modify
  4. Verify data: ls ~/.cache/autoresearch/march-madness/data/. If missing: uv run prepare.py
  5. Initialize results.tsv with header row
  6. Confirm and go

Experimentation

Launch: uv run experiment.py

Modify only: experiment.py Do NOT modify: prepare.py, config.py Goal: lowest MAE (mean absolute error of predicted margin vs actual margin)

Secondary metrics (win_accuracy, ats_accuracy, per-bucket breakdown) are printed for context but MAE drives keep/discard.

Output format

---
mae:              9.1234
rmse:             11.5678
win_accuracy:     0.7500
ats_accuracy:     0.5200
total_seconds:    1.3

Bucket breakdown (test set):
  toss_up       n= 28  mae=8.2  win%=57.1%
  lean          n= 35  mae=7.8  win%=74.3%
  solid         n= 42  mae=9.1  win%=83.3%
  blowout       n= 15  mae=12.4 win%=93.3%

Extract: grep "^mae:" run.log

Logging results

results.tsv — tab-separated, 4 columns:

commit	mae	status	description
a1b2c3d	9.4500	keep	baseline — BL=1.1 * emDiff/100 * avgPace

The experiment loop

LOOP FOREVER:

  1. Check git state and read results.tsv + combat_log.md
  2. Think of an experimental idea (see "Research directions" below)
  3. Modify experiment.py
  4. git commit
  5. uv run experiment.py > run.log 2>&1
  6. grep "^mae:" run.log
  7. Record in results.tsv
  8. If MAE improved → keep. If worse → git reset --hard HEAD~1, log in combat_log.md

NEVER STOP. Run experiments continuously until interrupted.

Data structure

Each game in the dataset:

game = {
    "year": 2024,
    "date": "2024-03-21",
    "round": "r64",              # first_four, r64, r32, s16, e8, f4, championship
    "fav": {
        "name": "UConn Huskies",
        "seed": 1,
        "kp_rank": 1,
        "kp_em": 35.22,          # KenPom Adjusted Efficiency Margin (per 100 possessions)
        "kp_tempo": 68.5,        # KenPom Adjusted Tempo (possessions per game)
        "score": 87
    },
    "dog": {
        "name": "Stetson Hatters",
        "seed": 16,
        "kp_rank": 230,
        "kp_em": -5.41,
        "kp_tempo": 64.2,
        "score": 52
    },
    "seed_diff": -15,            # fav_seed - dog_seed (negative = bigger mismatch)
    "em_diff": 40.63,            # fav_kp_em - dog_kp_em (always positive)
    "actual_margin": 35          # fav_score - dog_score (positive = fav won)
}

Train: ~930 games (2009-2023). Test: ~120 games (2024-2025).

Research directions

Ideas to explore (not exhaustive — get creative):

Formula tweaks

  • Scaling factor (BLOWOUT): currently 1.1, try other values
  • Pace weighting: does pace matter equally for all games? High-pace vs low-pace dynamics
  • Nonlinear transforms: sqrt compression, log scaling, piecewise functions on em_diff
  • Per-round adjustments: R64 might behave differently than F4
  • Seed-based corrections: 1v16 might need different treatment than 5v12

Feature engineering

  • Rank difference (kp_rank): does ordinal rank add signal beyond em_diff?
  • Tempo mismatch: when fast team plays slow team, what happens?
  • Seed upset probability: historical upset rates by seed matchup
  • Round progression: do margins compress as tournament progresses?

ML approaches (numpy/pandas available)

  • Linear regression on multiple features
  • Polynomial features on em_diff
  • Binned predictions: learn average margin per em_diff bucket
  • Weighted training: recent years weighted more than older years

Ensemble / hybrid

  • Formula + correction: base formula + learned residual
  • Multi-model blend: average multiple approaches

Domain knowledge

  • KenPom AdjEM is per 100 possessions. Multiply by (avg_pace / 100) to get per-game margin.
  • All games are neutral site (tournament). No home court advantage.
  • "Favorite" is the team with higher KenPom AdjEM (not necessarily higher seed).
  • Tournament margins are nonlinear: KenPom overshoots on 1v16/2v15 (blowout predictions too high) and undershoots on 7v10/8v9 (toss-up predictions too low).
  • Sqrt compression (sign * k * sqrt(|spread|/k), k=15) was derived from this dataset and showed -0.8 MAE improvement in cross-validation. The agent may rediscover this or find something better.

Dead ends (do not retry)

These have been extensively tested in the main march-madness repo:

Approach Result Why it failed
ATS coverage optimization Smaller spreads, worse margin prediction Rewards timidity, not accuracy
BLOWOUT > 1.1 +noise, no improvement More aggression = worse on neutral games
Momentum factors Degrades upset prediction Refiner, not a flipper — and we have no momentum data here
Conference strength Added noise Not predictive in tournament setting
Rest day adjustments No signal Tournament games are scheduled, not random
MAE < 8.0 target Spreads hug margins, no betting edge Sweet spot is 8-10 for practical use