Doghouse project. It works, but the prediction is post factum: the bands have already fallen when it fires, and the few-minute "lead" is only against the official bulletin's publishing lag, not the flare. We beat latency, not the event. Full post-mortem in the awesome-ml-systems dog house. Kept public, and the UI is genuinely nice. Fork it if you can do better.
Can you hear a solar flare black out the shortwave bands before the space-weather bulletin says so? An X-ray flare ionizes the dayside D-layer within minutes and HF propagation collapses. Thousands of amateur WSPR beacons are an unintentional planetary ionosonde, and when the dayside bands fall silent together the collapse is visible in the firehose. dead-air reads that collapse live from the independent WSPR data, at best a few minutes ahead of the official bulletin's publishing lag.
The live scope: each frequency layer shows observed vs expected spot rate over the dayside, the terminator tracks the real sun, and the shield ruptures on the footprint region when the bands fall silent together.
Every dayside band dives together as the D-layer absorbs, minutes around the flare peak. That simultaneous multi-band collapse is the signature the detector fires on.
A normalcy model predicts the expected dayside spot rate per band, path geometry and hour. The scorer turns the shortfall into a robust per-band z, and flags a collapse when three or more dayside bands stay quiet together. The GOES M/X flare record is the truth the events are scored against, on a time-held-out window (train through 2026-02, validate 2026-03/04, hold out 2026-05 to 07). No shuffling.
| metric (held-out, vs GOES M1+, v5 regional) | value |
|---|---|
| event recall | 0.26 |
| event precision | 0.23 |
| median lead to flare peak | -10 min (fires after the peak) |
| per-bucket normalcy error (MAE, log spots) | 0.19 |
Read that honestly: this is post factum. The bands have already dropped when the detector fires, and on the held-out window the median lead is negative (it fires after the peak). Any apparent earliness is against the bulletin's publishing lag, not the event: we beat latency, not foresight. The multi-band signature is real (a held-out M6.7 fires two collapse windows, an M5.8 stays below the three-band bar), the operational value is not.
The row-level normalcy model hugs the diagonal (MAE 0.19 in log spots): the deficit the detector reads is a real shortfall against a calibrated expectation, not model error.
The detector localizes a collapse to a footprint cell. The path midpoint is binned
to a coarse geographic grid (45 by 45 degrees) and the deficit, the robust z and
the event flag are computed per (band, region), logged to dead_air_predictions
v2. The app shows the worst-hit region per band and places the shield rupture on
that cell. The GBM is unchanged: region derives from the path midpoint, already a
model feature, so only the deficit aggregation grain and the frozen z calibration
move to the regional level.
Regional calibration (held-out, vs GOES M1+, z=1.25, three bands per cell):
| metric | v1 global | current (v5 regional, 45 deg) |
|---|---|---|
| event recall | 0.41 | 0.26 |
| event precision | 0.15 | 0.23 |
| median lead to peak | 3 min | +6.5 val, -10 holdout |
| normalcy MAE (log spots) | 0.19 | 0.19 |
A finer cell isolates the footprint but splits a band stack across cells, so recall falls. The training pipeline searches the cell size, z and the bands-per-cell threshold on validation by F0.5 (precision-weighted), and the winning cell rides in the model bundle so serve bins identically. That search confirmed 45 deg cells, z=1.25, three bands: 30 splits the stack, 60 loses precision. Threshold and cell tuning are at their ceiling (~0.23 precision); the next lever is matching the collapse footprint to the subsolar patch, a detector rebuild, not a knob.
Read these before quoting the number anywhere.
- Recall is bounded by observing geometry. A flare blacks out the sunlit hemisphere. When the dayside sits over an empty ocean at local midnight there is almost no WSPR traffic under it to go quiet, so that flare cannot be seen from the firehose at all. Recall against every M1+ flare therefore has a hard ceiling no model reaches.
- The label is a proxy for the collapse. The detector flags propagation collapse. A GOES flare is one cause, the one with an official timestamped record to score against. Geomagnetic storms and absorption events also silence the bands, so some fired events with no matching flare are real collapses, not pure noise, which the precision number cannot separate.
- Precision is the weak point (~0.23). The detector scores each footprint cell on its own, so a natural regional lull can still trip it. Matching the collapse footprint to the subsolar patch, which would separate a flare from a local lull, is the real next lever and is NOT built. Threshold and cell tuning are maxed out.
- Mid-class flares are partial. An M1 to M4 flare hits the lowest bands first and often does not reach three bands. The three-band bar is deliberate: it holds out for broad collapses and lets narrow ones pass.
HF radio bounces off the ionosphere. An X-ray flare ionizes the D-layer on the sunlit side, which absorbs the lower bands first (7, 10 MHz) and the higher bands as the flare grows. The nightside is untouched. Thousands of amateur stations trading WSPR beacons report every path that completes, so a sudden multi-band dayside drop in that traffic is a sudden ionospheric disturbance in progress.
The model learns the normal spot rate per band, path geometry, hour and season, then reads the live deficit. The per-band z baseline is frozen at training time and stored with the model, so a three-hour serving window scores the deficit on the same scale as the training months, with no window-length skew.
An FTI (feature, training, inference) system on Hopsworks. Sources arrive on their own cadence and join through the feature store, with one shared feature module so training and serving cannot skew.
flowchart LR
wspr([wspr.live ClickHouse]):::ext
goes([NOAA SWPC + NASA DONKI]):::ext
psk([PSKReporter MQTT]):::ext
subgraph FE[Feature]
direction TB
f1[deadair-wspr, PySpark] --> fg1[(wspr_agg)]:::hops
f2[deadair-goes] --> fg2[(goes_flare, label)]:::hops
end
subgraph TR[Training]
direction TB
fv{{dead_air_fv}}:::hops --> t1[normalcy GBM + frozen detector] --> reg[(Model Registry)]:::hops
end
subgraph INF[Inference]
direction TB
sc[deadair-scorer, every 5 min] --> pr[(dead_air_predictions)]:::hops --> app[dead-air app]
end
wspr --> f1
goes --> f2
fg1 --> fv
fg2 --> fv
reg --> sc
fg1 --> sc
psk --> app
classDef hops fill:#10b98122,stroke:#34d399,color:#e5e7eb;
classDef ext fill:none,stroke:#6b7280,color:#9ca3af,stroke-dasharray:4 3;
The sources, each on a different cadence:
| source | cadence | role |
|---|---|---|
| wspr.live ClickHouse | ~1 min fresh | spot reports, training history and serving reads from one source, no skew |
| NOAA SWPC GOES | live | M/X flare record: the label, and the bulletin the detector aims to beat |
| NASA DONKI FLR | historical | flare backfill for the training window |
| PSKReporter MQTT | ~281 spots/s | the app wall, display only |
The file-by-file map:
deadair_features.py shared, skew-free: path geometry + history baseline + detector
collect/goes.py NOAA SWPC + NASA DONKI flare client
pipelines/wspr_pipeline.py F1 WSPR spots -> wspr_agg (Hopsworks job, PySpark)
pipelines/goes_pipeline.py F2 flares -> goes_flare (Hopsworks job)
pipelines/train.py T feature view -> dead_air_normalcy (Hopsworks job)
pipelines/score.py I1 wspr_agg -> dead_air_predictions, 5 min (Hopsworks job)
app/ I2 FastAPI app: per-band panel + GOES receipts + live wall
tools/job_resources.py set job memory via the SDK after a deploy
tools/make_banner.py the series banner generator
reqs/dead-air.md the FTI specification
The detector lives in deadair_features.py next to the geometry and the history
baseline, so pipelines/train.py and pipelines/score.py import the exact same
code and the scorer carries no dependency on the training pipeline.
Clone into a Hopsworks project on the /hopsfs/... FUSE mount. Paths self-derive,
nothing is hardcoded to a username. The committed job memory and chunk sizes are
modest so the pipelines schedule on a small cluster. On a larger box, override them
on the command line.
make wspr-backfill # F1 WSPR history -> wspr_agg
make goes-backfill # F2 flare history -> goes_flare
make train-job # T normalcy GBM + detector -> dead_air_normalcy
make wspr-schedule # F1 incremental, every 15 min
make goes-schedule # F2 top-up, hourly
# scorer + app deploy from pipelines/score.py and app/deploy_app.pymake train-job TRAIN_MEM=8192 # larger cluster
make wspr-backfill WSPR_MEM=8192 WSPR_CHUNK_DAYS=10A server-rendered wall. The finger is a per-band dayside health panel from
dead_air_predictions: each HF band shows its latest residual-deficit z and turns
red inside a fired collapse. The receipts are the recent GOES M/X flares, so when
the finger moves before a flare row appears, that gap is the lead. The wall is the
PSKReporter firehose relayed over SSE, raw spots scrolling, thinning on the paths
a collapse is killing. When the bands are quiet and GOES is silent the wall is
still a live picture of the planet talking to itself.