Restart implementation requirements

[infotroph]

Chunks of work to support model restarting:

• Crop layer harmonization give consistent sensible timeries [Alexey, in progress]
• Crop changes are passed through to management events [done?]
• Workflow setup creates needed inputs in each run directory
  • including parameter updates
• Model passes hop test
  • Will probably need to write out more trackers, esp. tillage rate multiplier
  • General approach: Any state that needs to be carried through gets an initial value parameter
• Ensemble run generates full timeseries output
  • with resource use remaining feasible
• Validation
• Downscaling site selection accounts for time series
  • Or ignores it for a principled reason and still works
• Downscaling runs succeed with restarted input

[infotroph]

Live discussion with @Alomir @dlebauer @hdpriest-ui:

• Any changes needed in Sipnet? TBD: Possible we'll want to write out 2-3 trackers (notably including the current state of the tillage decay factor) and add initialization parameters to read them back in.
• We already discussed and rejected moving the entire model restart process inside Sipnet (Should SIPNET handle its own restarts? PecanProject/pecan#3795). Would it make sense to consider a subset of that work to let us update Sipnet parameters midrun without reloading state? We think no, once we got that far we'd want to just keep going to update state as well.
• Do restarted runs get spread across a distributed workflow? No, restarts are done in serial inside each run directory.

[ashiklom]

Crop layer harmonization give consistent sensible timeries [Alexey, in progress]

This is done (or, at least, good enough for MVP) with LandIQ-harminized-v4.1 data. Relevant PR is ccmmf/cadwr-landuse#3. Data are on geo at /projectnb/dietzelab/ccmmf/LandIQ-harmonized-v4.1.

Crop changes are passed through to management events [done?]

The basic functionality here works via PecanProject/pecan#3821. It's basically just a big join of LandIQ parcels (from above), the HLS phenology data (/projectnb/dietzelab/ccmmf/management/phenology/matched_landiq_mslsp_v4.1), BISM Kc crop definitions and coefficients (already in PEcAn), and MAD and rooting depth values (new in PR 3821 above). I've implemented this in the context of irrigation, but not yet in the context of other events, but that should be straightforward since it's just a join relating LandIQ crop ID to PEcAn's crop definitions.

Next steps:

• Modify to produce event.json files instead of raw SIPNET irrigation outputs
• Modify to work statewide with all LandIQ polygons (rather than small subset of design points)
• Modify to produce ensembles with some uncertainty estimates (currently, just one set of events per parcel)

Workflow setup creates needed inputs in each run directory (including parameter updates)

In progress by me. Here's where I was last week: https://github.com/orgs/ccmmf/discussions/188#discussioncomment-16075946. The basic functionality is 90% there, but it's in the "I want to have a janky script so I understand how to do it" phase, not a "PR in progress" phase (though once the former is done, the latter should be very quick). Next up is:

• Parameter updates (key thing is matching crop types to PEcAn posteriors)
• Testing with real event files

• Model passes hop test
  • Will probably need to write out more trackers, esp. tillage rate multiplier
  • General approach: Any state that needs to be carried through gets an initial value parameter

Sounds like @infotroph is working on this? Also noting the existence of https://github.com/PecanProject/pecan/blob/develop/modules/assim.sequential/R/hop_test.R --- this is deeply integrated into the SDA code (using sda_enkf with no data), which to me feels like overkill for restarts in the context of events...but presumably, it does work.

All other steps are not started yet.

[mdietze]

Crop changes are passed through to management events [done?]

@ashiklom I wanted to clarify that @sarahkanee is responsible for the event files for planting and harvesting. Those are done statewide for planting and very close for harvesting (just need to finalize some non-RS numbers related to harvest efficiency and litter by plant pool). She should be able to point you to where these files are on the BU server. Note that we're leaving the final event file assembly (interleaving the different planting, harvesting, irrigation, tillage, fertilizer, and non-crop C events into a single per-site chronological json) to the modeling team.