Add CPS raw2Sv realtime processing flow

[LOCEANlloydizard]

• add a CPS-specific raw2Sv flow for NRT processing
• egister and deploy the CPS raw2Sv flow through the deployment framework
• add PostgreSQL RAW/Sv processing ledger support
• add RAW directory monitoring and Prefect event emission to trigger raw2Sv processing
• reconcile existing RAW files when the watcher starts and emit a startup event so pending files are picked up
• ensure RAW filesystem events are emitted even when the file is already registered in the processing ledger
• add configurable limits for the number of new RAW files processed per run
• integrate CPS raw2Sv processing with the existing CPS transect-processing workflow
• add/update tests for the processing ledger, RAW watcher, CPS raw2Sv flow, and transect simulation

[LOCEANlloydizard]

is a duplicate

[leewujung]

not sure what you mean - it's the only entry in the registry?

[LOCEANlloydizard]

there’s another one at L85! Not sure when it was added, my bad.. I think I added it at some point!

[leewujung]

maybe collapse lines like these throughout the codebase to a single line to avoid excessive vertical splitting

[LOCEANlloydizard]

done!

[leewujung]

Hey @LOCEANlloydizard : I went through the raw monitor behavior and made a couple small patches -- one of them is to prevent a single write to send event multiple times.

I think that there's a philosophical difference that would have practical consequences that we should discuss:

• if I understand correctly, the raw monitor right now would emit an event to trigger flow_raw2Sv_CPS every time when a .raw file is created, changed, or moved in to the watched directory
• my original thought is for the raw monitor to emit an event to only update the ledger but not trigger the flow_raw2Sv_CPS or flow_raw2Sv, this is because:
  • if we operate out of the back-up server (which we have been on all 3 vessels we have deployed the workflow on), this is fine, because the back-up server is only synced at a specified interval, so the event emission would be infrequent
  • however if we operate out of the actual drive that the EK80 software is writing data into (which is probably a bad practice), since the latest .raw file would be written into continuously until it is closed when reaching the configured file size limit, flow_raw2Sv_* will keep on getting triggered, which would cause a lot of collision and probably would crash
  • in addition, in flow_raw2Sv_* the .raw file conversion can be configured to executed in parallel or sequentially (right now we just do sequential conversion for the NRT deployment since that op is very fast, and we will likely switch to a much slimmer machine later). In the scheme of the current code, there would always only be 1 .raw file getting converted/processed to Sv.
  • updating the database ledger is a much more simplistic operation than the whole raw to Sv conversion

We can restrict the event emission only when a .raw is no longer changing (done via some ledger record logics), but this means that for shallow regions when each ping is short, we would miss a large chunk of most recent data, especially if the configured file size limit is large (for example, SWFSC used to produce 2GB files...).

It seems to me that it is more straightforward to go with the route to only update the ledger when watchdog detects changes, and run flow_raw2Sv_* at N min intervals. This does mean that we could have a lag of up to N mins in the workflow, but I think to make the "real" real-time workflow, we would have to allow echopype to accept real-time streams of the datagrams, or subscribe to the EK80 REST API to get the Sv values from the EK80 software directly.

Thoughts? Happy to discuss! I may have misunderstood the code or missed other considerations you have.

[LOCEANlloydizard]

hey @leewujung thank you very much for the inputs! yes agreed: i changed the RAW monitor so it only updates the ledger and no longer emits RAW events. raw2Sv_CPS now runs on a 5 min interval and checks the ledger for new files. I also tested the updated workflow end-to-end locally and it behaved as expected! i'm merging this now to check the deployment! cheers!