Skip to content

enh: Upgrade post-processing flows for edge components - #178

Open
leewujung wants to merge 9 commits into
echostack-org:mainfrom
leewujung:postproc_local_raw
Open

enh: Upgrade post-processing flows for edge components#178
leewujung wants to merge 9 commits into
echostack-org:mainfrom
leewujung:postproc_local_raw

Conversation

@leewujung

Copy link
Copy Markdown
Member

This PR is an upgrade from #173 and include the following changes:

  • change all raw2Sv flows to be based off a local .raw dataset
    • I ran the original version that download data from an OSN bucket and performs conversion, but the download part is sooooo slow
    • for the current implementation I downloaded all .raw data using rclone and the conversion flows are super fast locally
  • build the ledgers for raw2Sv, MVBS slices, and prediction slices on the first flow run
    • assign MVBS and prediction entries to be no_data for the times during which now .raw files were collected, such as during the night -- this allows the flow runs to quickly skip through these slices
    • since the dataset is fixed for post-processing, these can be all done ahead of time
  • added mechanism to control how many times a specific raw2Sv, MVBS slice creation, or prediction computation is performed on a specific file or slice -- if the task fails for more than the specified N times, the task entry would be marked as always_failed and skipped in later flow runs
  • use Prefect native flow conflict to cancel new flows of the same deployment

Notes:

@leewujung leewujung added the enhancement New feature or request label Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant