Skip to content

Flarelist anc - #431

Merged
nicHoch merged 31 commits into
i4Ds:masterfrom
nicHoch:flarelist_ANC
Oct 22, 2025
Merged

Flarelist anc#431
nicHoch merged 31 commits into
i4Ds:masterfrom
nicHoch:flarelist_ANC

Conversation

@nicHoch

@nicHoch nicHoch commented Sep 1, 2025

Copy link
Copy Markdown
Collaborator

handels flare lists
but also LL3 data

nicHoch added 19 commits December 19, 2023 09:38
@nicHoch
bump all product versions to 2
d9c5387
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
f0642f9
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
5d22466
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
2ae68fb
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
31cbb6d
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
af81758
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
47631c7
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
9f0dfb1
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
c020af4
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
bd2083b
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
f4f9c5b
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
5d0fe4f
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
9704ff9
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
4ec73cd
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
eb6b5be
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
af929ee
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
e716f41
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
3b3f968
@nicHoch
Merge branch 'master' of https://github.com/i4Ds/STIXCore
319e962
@codecov-commenter

codecov-commenter commented Sep 2, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 36.06557% with 624 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.25%. Comparing base (3ae9671) to head (8c06504).
⚠️ Report is 18 commits behind head on master.

Files with missing lines Patch % Lines
stixcore/products/level3/flarelist.py 34.93% 244 Missing ⚠️
stixcore/io/FlareListManager.py 13.15% 165 Missing ⚠️
stixcore/processing/FLtoFL.py 27.50% 58 Missing ⚠️
stixcore/processing/LL.py 31.34% 46 Missing ⚠️
stixcore/processing/FlareListL3.py 35.18% 35 Missing ⚠️
stixcore/io/ProcessingHistoryStorage.py 10.00% 18 Missing ⚠️
stixcore/products/level3/flarelistproduct.py 54.54% 15 Missing ⚠️
stixcore/products/lowlatency/quicklookLL.py 53.57% 13 Missing ⚠️
stixcore/products/product.py 65.00% 7 Missing ⚠️
stixcore/processing/AspectANC.py 14.28% 6 Missing ⚠️
... and 6 more
Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #431      +/-   ##
==========================================
- Coverage   72.23%   69.25%   -2.98%     
==========================================
  Files          78       76       -2     
  Lines        8085     8100      +15     
==========================================
- Hits         5840     5610     -230     
- Misses       2245     2490     +245

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nicHoch
nicHoch force-pushed the flarelist_ANC branch 2 times, most recently from 4fe20e4 to 88d949c Compare September 2, 2025 18:04
@nicHoch
bump all product versions to 2
2d810ad 
Add support for trigger scaling in compression mode S=0,k=0,M=7 (i4Ds#377)

* Unscale data when compressed with mode S=0,k=0,M=7
* QL - LC, BKG, SPECTRA
* BSD - CPD, SCPD, VIS and SPEC

bump all product versions to 2 (i4Ds#378)

* bump all product versions to 2

* roleout IDB version 2.26.27

* fix test with new version number in mock object

* fix test

* fix for latest IDB as fallback

Fix duplicated hk time bin bug

Ddpdupdates2 (i4Ds#380)

* bump all product versions to 2

* prep for L2 ddpd

* integrate IDB release 2.26.38

* update for common CONF submodule

* fir text for latest IDB to 2.26.38

Config fix (i4Ds#381)

* bump all product versions to 2

* fix module path

Fix bugs found when processing new scaled trigger data. (i4Ds#382)

* Fix bug processing new scaled trigger data.
* Allow for QL file to unscale by packet
* Fix bug that create extra (replica) triggers in Spectrogram

Publish_L2 / RID LUT Manager (i4Ds#385)

* bump all product versions to 2
* RID LUT data manager update and use in processing for trigger descaling
* Update stixcore/io/RidLutManager.py
* fixed API mock via static files on pup99

---------

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

Add warning if trigger scaling changes in request (i4Ds#388)

End2end rid lut manager (i4Ds#390)

* bump all product versions to 2

* add copy and parent option

* fix missing init of singleton

bump all product versions to 2 (i4Ds#389)

Bump conf version (i4Ds#391)

* bump all product versions to 2

* bump common version

Flexible way to fill DATAMAX/MIN BUNIT EXPOSURE in FITS header keywords (i4Ds#393)

* bump all product versions to 2

* add properties for dedicated fits header keywords with class inhe. / override

* add test and minor fixes after review

* fix XPOSURE/XPOMAX header error

* fix tests for dmax...

* relax end2end fitsdiff

* less tolerance in end2end test

Solo anc stix ephemeris (i4Ds#399)

* bump all product versions to 2

* add more FITS header keywords

* fix for ANC also get fits updated headers while publishing

Upgrade_watchdog (i4Ds#401)

* bump all product versions to 2

* bump version

* fix version label

* pin common CONF version for end2end test

* ignore VERS_CFG in end2end testing

* add test for TM folder observing

* bump watchdog==6.0.0

* integrate review fixes

Update docks (i4Ds#409)

* bump all product versions to 2

* update documentation

* add doc for end to end tetsing

* fix format

* add doc for manual (re)processing of fits products

E2e fix sym add (i4Ds#412)

* bump all product versions to 2

* dep np.bool8

* disable test

Get rid reason/scaling lookup error handling (i4Ds#411)

* bump all product versions to 2

* add global error detection in pipeline logger (not just main process)

* add exception handling for get_reason

daily processing pipeline (i4Ds#405)

* bump all product versions to 2

* initial

* daily processing task

* add singleStep

* move ANC-aspect from pipeline to daily_pipeline

* removed requirement for the moment

* fix i4Ds#283 and option to process only latest fits file versions

* update ddpd scripts

* fix circular imports

* test git dep

* skio test on win

* skip test on win/mac

* Update stixcore/io/FlareListManager.py

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

* fix format error

---------

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

Add Zenodo file (i4Ds#413)

* Add Zenodo author information

Update high resolution transmission to match IDL defaults (i4Ds#414)

Pin major and minor versions of some packages (i4Ds#415)

V1.5.1_fixes (i4Ds#420)

* bump all product versions to 2

* fix i4Ds#417

* fix  i4Ds#418

* daily report log files could have different names if the scripts run over multiple days

* fix i4Ds#419

* end2end test: better report if no corresponding file found

* rewrite pipeline to cronjob based approach i4Ds#422

* cleanup before merge

Fix bug in spectrogram time bin calculations (i4Ds#423)

* Only use the closing time offset (NIX00269) for the last science substrure

Update .zenodo.json

Maintenance update project and dependencies (i4Ds#428)

* Update project use project.toml and move config into own files

* Fix most of the tests

* ruff-check, ruff-format and codespell

* Some spelling mistakes left need to check the meaning

* More updates and fixes

* Few more small fixes

* Fix doc build

* Revert unintentional spelling correction in IDB sql script.

* Update tox test dep name

* Add version.py to git

* Skip unclear test add skipped test for later

* Update workflow python version

* Add remote data to github CI

bump all product versions to 2

Add support for trigger scaling in compression mode S=0,k=0,M=7 (i4Ds#377)

* Unscale data when compressed with mode S=0,k=0,M=7
* QL - LC, BKG, SPECTRA
* BSD - CPD, SCPD, VIS and SPEC

bump all product versions to 2 (i4Ds#378)

* bump all product versions to 2

* roleout IDB version 2.26.27

* fix test with new version number in mock object

* fix test

* fix for latest IDB as fallback

Fix duplicated hk time bin bug

Ddpdupdates2 (i4Ds#380)

* bump all product versions to 2

* prep for L2 ddpd

* integrate IDB release 2.26.38

* update for common CONF submodule

* fir text for latest IDB to 2.26.38

Config fix (i4Ds#381)

* bump all product versions to 2

* fix module path

Fix bugs found when processing new scaled trigger data. (i4Ds#382)

* Fix bug processing new scaled trigger data.
* Allow for QL file to unscale by packet
* Fix bug that create extra (replica) triggers in Spectrogram

Publish_L2 / RID LUT Manager (i4Ds#385)

* bump all product versions to 2
* RID LUT data manager update and use in processing for trigger descaling
* Update stixcore/io/RidLutManager.py
* fixed API mock via static files on pup99

---------

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

Add warning if trigger scaling changes in request (i4Ds#388)

End2end rid lut manager (i4Ds#390)

* bump all product versions to 2

* add copy and parent option

* fix missing init of singleton

bump all product versions to 2 (i4Ds#389)

Bump conf version (i4Ds#391)

* bump all product versions to 2

* bump common version

Flexible way to fill DATAMAX/MIN BUNIT EXPOSURE in FITS header keywords (i4Ds#393)

* bump all product versions to 2

* add properties for dedicated fits header keywords with class inhe. / override

* add test and minor fixes after review

* fix XPOSURE/XPOMAX header error

* fix tests for dmax...

* relax end2end fitsdiff

* less tolerance in end2end test

Solo anc stix ephemeris (i4Ds#399)

* bump all product versions to 2

* add more FITS header keywords

* fix for ANC also get fits updated headers while publishing

Upgrade_watchdog (i4Ds#401)

* bump all product versions to 2

* bump version

* fix version label

* pin common CONF version for end2end test

* ignore VERS_CFG in end2end testing

* add test for TM folder observing

* bump watchdog==6.0.0

* integrate review fixes

Update docks (i4Ds#409)

* bump all product versions to 2

* update documentation

* add doc for end to end tetsing

* fix format

* add doc for manual (re)processing of fits products

E2e fix sym add (i4Ds#412)

* bump all product versions to 2

* dep np.bool8

* disable test

Get rid reason/scaling lookup error handling (i4Ds#411)

* bump all product versions to 2

* add global error detection in pipeline logger (not just main process)

* add exception handling for get_reason

daily processing pipeline (i4Ds#405)

* bump all product versions to 2

* initial

* daily processing task

* add singleStep

* move ANC-aspect from pipeline to daily_pipeline

* removed requirement for the moment

* fix i4Ds#283 and option to process only latest fits file versions

* update ddpd scripts

* fix circular imports

* test git dep

* skio test on win

* skip test on win/mac

* Update stixcore/io/FlareListManager.py

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

* fix format error

---------

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

Add Zenodo file (i4Ds#413)

* Add Zenodo author information

Update high resolution transmission to match IDL defaults (i4Ds#414)

Pin major and minor versions of some packages (i4Ds#415)

fix i4Ds#417

fix  i4Ds#418

daily report log files could have different names if the scripts run over multiple days

fix i4Ds#419

end2end test: better report if no corresponding file found

rewrite pipeline to cronjob based approach i4Ds#422
@nicHoch
flare list, flare list with position, flare list with soop, flare lis…
ca4f701 
…t preview maps
@nicHoch
change flarelist to L3
ca8a9d8
@nicHoch
new naming for LL3 QL products
28d1ac5
@nicHoch
fixes after python3.11 upgrade
9d13144
@nicHoch
fix missing isnan from merge conflicts
34f27b8 
fix tests

fix tests

fix tests
samaloney
samaloney reviewed Oct 2, 2025
View reviewed changes
Comment thread processed.sqlite Outdated
samaloney
samaloney reviewed Oct 2, 2025
View reviewed changes
Comment thread stixcore/io/FlareListManager.py Outdated
samaloney
samaloney reviewed Oct 2, 2025
View reviewed changes
Comment thread stixcore/io/FlareListManager.py
samaloney
samaloney reviewed Oct 2, 2025
View reviewed changes
Comment thread stixcore/io/FlareListManager.py
samaloney
samaloney reviewed Oct 2, 2025
View reviewed changes
Comment thread stixcore/io/fits/processors.py Outdated
samaloney
samaloney reviewed Oct 2, 2025
View reviewed changes

@samaloney samaloney left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots going on here, some of it looks still in progress in particular the flare list stuff, so hard to fully understand but all seems reasonable

@nicHoch

nicHoch commented Oct 14, 2025

Copy link
Copy Markdown
Collaborator Author

@samaloney
so i cleaned this up a bit in preparation for the release.
all flare list processing in the process_daily runner are deactivated so only ASPECT files and LL03 ql lightcurve plots are generated and published

please do a quick review again and approve if possible so that that can be merged finally

samaloney
samaloney approved these changes Oct 16, 2025
View reviewed changes

@samaloney samaloney left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hard to check everything as it a large PR but looks reasonable.

@nicHoch
nicHoch merged commit de06abc into i4Ds:master Oct 22, 2025
8 of 9 checks passed
@nicHoch nicHoch added this to the v1.6.0 milestone Oct 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@samaloney samaloney samaloney approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v1.6.0

Development

Successfully merging this pull request may close these issues.

3 participants

@nicHoch @codecov-commenter @samaloney