Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
219 commits
Select commit Hold shift + click to select a range
debef9c
split into candidate and simple calc phases
markmac99 Jan 16, 2026
240a720
print candmode in logs
markmac99 Jan 16, 2026
b1040c6
tidy up logging
markmac99 Jan 17, 2026
aa20b49
bug in conversion step
markmac99 Jan 17, 2026
02a9865
avoid committing to sqlite after every update
markmac99 Jan 19, 2026
f569ebe
support for excluding already-failed candidates
markmac99 Jan 19, 2026
12dab86
start adding some standalone management to the sqlite db scripts
markmac99 Jan 19, 2026
9b18baa
add obs-date to database
markmac99 Jan 19, 2026
8208b77
got params in wrong order arg
markmac99 Jan 20, 2026
1dc6ae5
improve archiving process
markmac99 Jan 20, 2026
332c5f3
log how many obs got moved to sqlite
markmac99 Jan 20, 2026
db77938
make logging provisional
markmac99 Jan 20, 2026
1089a1b
remove unnecessary logging
markmac99 Jan 20, 2026
e4320a5
make obs database a class
markmac99 Jan 21, 2026
75f11ea
improve documentation
markmac99 Jan 21, 2026
405d6c0
some tidying up
markmac99 Jan 21, 2026
81a2969
simplify migration to sqlite
markmac99 Jan 21, 2026
9a71d85
keep track of which traj we're doing
markmac99 Jan 21, 2026
297da7c
improve archiving and add merging support
markmac99 Jan 21, 2026
f1f5f5e
updates to improve handling of operation modes and remote data
markmac99 Jan 24, 2026
6eb44fa
improve documentation
markmac99 Jan 25, 2026
9024741
Merge branch 'wmpg:master' into distrib_processing
markmac99 Jan 25, 2026
1afaff1
bugfix to create phase1 trajs properly
markmac99 Jan 26, 2026
625ee4d
bugfix in when to save phase1
markmac99 Jan 26, 2026
3684014
be a bit less verbose
markmac99 Jan 27, 2026
7f5677a
remove unnecessary message
markmac99 Jan 27, 2026
21fa47e
reinstate prior behaviour to skip ignored stations when retrying solu…
markmac99 Jan 27, 2026
2c1ab8b
simplify calls to addTrajectory
markmac99 Jan 27, 2026
610cfa5
create trajectories db in sqlite
markmac99 Jan 27, 2026
22e9932
add trajectories db as json file
markmac99 Jan 27, 2026
558ce81
bugfixes
markmac99 Jan 27, 2026
fd228a9
updates to handle trajectory sqlite db better
markmac99 Jan 28, 2026
b82689b
dont load the traj db in MCMODE2 as we don't need it
markmac99 Jan 28, 2026
8331148
some support to analyse the traj db
markmac99 Jan 28, 2026
62b45cd
improve logging
markmac99 Jan 29, 2026
02b6c0e
handle verbose flags better
markmac99 Jan 29, 2026
13e6836
more debug
markmac99 Jan 29, 2026
9ec2eaa
remove commitnow as its causing issues
markmac99 Jan 29, 2026
7a8d4e3
debug
markmac99 Jan 29, 2026
36a9f0b
debug
markmac99 Jan 29, 2026
ebf5473
typo
markmac99 Jan 29, 2026
7e88468
more doc and debug to track down issue
markmac99 Jan 29, 2026
2eaeb91
fix bug in fails table
markmac99 Jan 29, 2026
ad4eec4
bugfix
markmac99 Jan 29, 2026
d27a13b
simplify code
markmac99 Jan 29, 2026
6c28d9d
bugfixes in correlateDB
markmac99 Jan 29, 2026
863d7e2
avoid removing trajectories incorrectly
markmac99 Jan 29, 2026
50eb9df
ting bug in reporting of removed traj
markmac99 Jan 29, 2026
36a880b
Client-mode remote data processing
markmac99 Jan 30, 2026
bb7b502
reduce amount of legacy data thats loaded
markmac99 Jan 30, 2026
23b258b
make dates tz aware
markmac99 Jan 30, 2026
be29951
improve logging
markmac99 Jan 30, 2026
5eebd9d
more tweaks to copying legacy data
markmac99 Jan 30, 2026
a87fb16
bugfix
markmac99 Jan 30, 2026
71e20aa
bugfix
markmac99 Jan 30, 2026
6083b46
remove unncessary debug
markmac99 Jan 31, 2026
5f986c7
don't try to save an unopened database
markmac99 Jan 31, 2026
6e8bb88
Updates for remote client datahandling
markmac99 Feb 1, 2026
9cd2d00
support to signal when a client is ready for data
markmac99 Feb 3, 2026
458170d
more work on master mode
markmac99 Feb 3, 2026
138d672
fix bug in unpairing if db is empty
markmac99 Feb 3, 2026
69c99ce
fix bug in unpairing if db is empty
markmac99 Feb 3, 2026
fd38bd5
make uploaded database names unique
markmac99 Feb 3, 2026
6aac11f
oops, make sure phase1 dir exists even in mc mode
markmac99 Feb 3, 2026
ce2877b
small bugfixes
markmac99 Feb 4, 2026
3622d0c
remove some excess verbosity
markmac99 Feb 4, 2026
67fefbe
indicate how many are being solved
markmac99 Feb 4, 2026
c4eecac
bugfix to handle nans in the database
markmac99 Feb 4, 2026
92f865a
add log suffix
markmac99 Feb 4, 2026
b271037
add log suffix properly
markmac99 Feb 4, 2026
1d366bf
catch SFTP fails
markmac99 Feb 4, 2026
333b48e
more small changes to protect against sftp fails
markmac99 Feb 4, 2026
c224ac1
defend against sftp failures
markmac99 Feb 11, 2026
47a2d27
bugfix to prevent phase2 crash merging data
markmac99 Feb 11, 2026
6da8bbb
add function to copy json data over
markmac99 Feb 11, 2026
83a2b72
bugfix: only move phase1 if client was doing phase1
markmac99 Feb 11, 2026
6ece9ab
make sure folders exist
markmac99 Feb 12, 2026
c9bb9bd
initialise remote data handling in the right place for phase2
markmac99 Feb 12, 2026
c9ede51
improve logging
markmac99 Feb 12, 2026
cda60ba
prevent crash in mcmode 2
markmac99 Feb 12, 2026
fbd7899
print operation mode more frequently
markmac99 Feb 12, 2026
7677acc
update remote data handling to actually move trajectories!
markmac99 Feb 12, 2026
21c94b4
slight improvement in shutdown messaging
markmac99 Feb 12, 2026
1b73810
bugfix
markmac99 Feb 12, 2026
8d8a325
chmod upload folders as needed
markmac99 Feb 12, 2026
cb20be5
fix perms on upload folders
markmac99 Feb 12, 2026
db2f17c
add some verbosity for debug
markmac99 Feb 13, 2026
e753a1e
bugfix in pairing routine
markmac99 Feb 13, 2026
b2cf365
Merge branch 'wmpg:master' into distrib_processing
markmac99 Feb 14, 2026
b12504a
rename archiveoldrecords to archivemonths
markmac99 Feb 14, 2026
379a5e9
Merge branch 'distrib_processing' of github.com:markmac99/WesternMete…
markmac99 Feb 14, 2026
4d212af
remove unnecessasry flag
markmac99 Feb 14, 2026
22dadc5
whoops changed param name incorrectly
markmac99 Feb 14, 2026
86721cd
initialise obs and traj db
markmac99 Feb 14, 2026
508f40e
a bit of documentation
markmac99 Feb 16, 2026
bd31fa0
check status from merge functions
markmac99 Feb 16, 2026
8de1c47
if connection fails, close gracefully and retry later
markmac99 Feb 16, 2026
e04f9f4
write pidfile
markmac99 Feb 16, 2026
17ef66d
remove unnecessary saving of temp pickles in ph2
markmac99 Feb 16, 2026
0a102b9
tidying up the date formats when reporting candidates, fails etc
markmac99 Feb 18, 2026
b73f07d
bugfix failed to define ref_dt
markmac99 Feb 19, 2026
7b45db1
add some comments
markmac99 Feb 26, 2026
91ab59e
create function to merge broken so we can call it elsewhere if needed
markmac99 Feb 26, 2026
615ea33
commit the write-ahead-log after every candidate save op
markmac99 Feb 27, 2026
43a4e8d
commit the obs database after saving candidates
markmac99 Feb 27, 2026
1a5d592
don't load stations or scan folders if in phase1 or phase2 only modes
markmac99 Feb 27, 2026
4ba68fc
support fractional bins
markmac99 Feb 27, 2026
3c0ae48
small correction in correlateDB
markmac99 Feb 27, 2026
f51e56c
remove unnecessary cursors
markmac99 Feb 27, 2026
d898e1f
support to purge local processed data
markmac99 Feb 27, 2026
704aa0a
make sure dbs are closed in the maintenance section
markmac99 Feb 27, 2026
51b637a
add verbosity
markmac99 Feb 27, 2026
3e7ac46
fix verbosity
markmac99 Feb 27, 2026
7c16e37
error handling in db merge process
markmac99 Feb 27, 2026
739ae25
remote remote WAL dbs after merger
markmac99 Feb 27, 2026
dcc904a
avoid trying to merge empty databases
markmac99 Feb 27, 2026
6415260
more fixes for database or table not existing
markmac99 Feb 27, 2026
d46b06f
documentation
markmac99 Feb 27, 2026
70b59ce
add more debug
markmac99 Feb 27, 2026
8071f49
delete rather than update when unpairing
markmac99 Feb 27, 2026
f663308
handle unpairing more efficiently with multi-insert
markmac99 Feb 27, 2026
6e20cf7
rename function more sensibly
markmac99 Mar 3, 2026
821bf68
switch logic round to mark paired after succeessful phase1. This redu…
markmac99 Mar 3, 2026
5fe13e0
remove unnecessay commit
markmac99 Mar 3, 2026
cfc0b81
pass new obs to solveTrajectory so they can be marked paired if needed
markmac99 Mar 4, 2026
7db6bdf
bug in phase1 processing
markmac99 Mar 5, 2026
ac19e59
working on ref_dt bug
markmac99 Mar 5, 2026
ced94c2
add debug messages to help testing
markmac99 Mar 5, 2026
9db92f7
tidying up
markmac99 Mar 6, 2026
e9d3c6e
avoid reprocessing already processed candidates
markmac99 Mar 7, 2026
c2572de
bugfixes and performance improvements
markmac99 Mar 8, 2026
e9794a8
remove incorrect location of Obs DB
markmac99 Mar 8, 2026
9612432
various improvements
markmac99 Mar 9, 2026
b9d9747
tidying up
markmac99 Mar 9, 2026
d7fabbd
fixes for issues #86, #87, #88, #94
markmac99 Mar 10, 2026
88bcd09
Add candidate database and use it
markmac99 Mar 11, 2026
48a35b0
purge older candidate data to keep database size reasonable
markmac99 Mar 11, 2026
cfd7645
save cands using jdt_ref to make checks easier
markmac99 Mar 11, 2026
c820387
small bug in cands db purge process
markmac99 Mar 11, 2026
bcaa042
move cand loading to within datahandle
markmac99 Mar 11, 2026
f234e8e
avoid slightly misleading message in log
markmac99 Mar 11, 2026
16a3db2
make sure to close and commit databases on exit
markmac99 Mar 11, 2026
2416d4b
bugfix
markmac99 Mar 12, 2026
1e44569
bugfixes and tidying up
markmac99 Mar 12, 2026
87c5109
add obs id to traj data
markmac99 Mar 15, 2026
21ab115
remove some debug
markmac99 Mar 16, 2026
8f6eaf6
add obs_ids to failed traj if convergence angle too small
markmac99 Mar 16, 2026
d9505a1
add ignored obs to database
markmac99 Mar 17, 2026
92e42bd
add ignored obs to traj object
markmac99 Mar 17, 2026
323d4c9
simplify updating the traj database
markmac99 Mar 17, 2026
6fe3f86
bugfix in trajdb
markmac99 Mar 17, 2026
a17a3a6
Merge branch 'wmpg:master' into distrib_processing
markmac99 Mar 20, 2026
205b9a3
add functionality to remove a trajectory by ID and to retrieve obsids…
markmac99 Mar 20, 2026
e74b8d7
remove some debugging
markmac99 Mar 21, 2026
830cce1
better remote data error management
markmac99 Mar 24, 2026
5bbf10e
improved duplicate management
markmac99 Mar 24, 2026
0fbfb3e
be sure to remove both traj with an overlap in obs so they get reanal…
markmac99 Mar 24, 2026
4a0c828
possible dupes: don't delete both trajs if not in auto mode
markmac99 Mar 26, 2026
c317cd6
less aggressive purging of processed phase1
markmac99 Mar 26, 2026
37cb56a
bugfixes in process to copy data from Json file
markmac99 Mar 27, 2026
e269d94
bugfix in allocation of data to children
markmac99 Mar 27, 2026
be840b8
log where phase1 files are saved
markmac99 Mar 27, 2026
8aa22e4
adding remote processing config file
markmac99 Mar 27, 2026
c12138c
tweaking debug
markmac99 Mar 27, 2026
b592ac8
support conversion of legacy trajectories as well as failed to sqlite
markmac99 Mar 28, 2026
9a3a4ed
more work on getting buckets right
markmac99 Mar 29, 2026
f421533
bugfix
markmac99 Mar 29, 2026
4fb4433
fix error in cand/phase1 saving logic
markmac99 Mar 29, 2026
83098f3
reworking on obs loading
markmac99 Mar 29, 2026
704bbb4
improve exception handling
markmac99 Mar 29, 2026
af31def
fixing timebucket calcs
markmac99 Mar 29, 2026
7521455
remove some debugging
markmac99 Mar 29, 2026
8b22845
improvements in data selection to avoid
markmac99 Mar 30, 2026
a89601a
performance bugfix in finding unprocessed data
markmac99 Mar 30, 2026
5785fed
bugfix when iterating through testing stations to ignore
markmac99 Mar 31, 2026
9433235
remove some debug and add some informational messages
markmac99 Mar 31, 2026
e6a8679
fix bug that was creating duplicates when reprocessing existing traj …
markmac99 Apr 1, 2026
8c8356b
updating some inline documentation
markmac99 Apr 3, 2026
68154de
Adding documentation on new capabilities
markmac99 Apr 3, 2026
0dfb9b4
Move documentation to right place
markmac99 Apr 3, 2026
047f784
More work on documentation
markmac99 Apr 3, 2026
bb872b7
More doco tweaks
markmac99 Apr 3, 2026
2807a0b
don't load json db if its not present (doh)
markmac99 Apr 4, 2026
5c09521
add merge process for canddb and fix small bug
markmac99 Apr 7, 2026
f437bca
bugfix in addTrajectory
markmac99 Apr 11, 2026
bae0d29
avoid closing closed databases
markmac99 Apr 11, 2026
8c5ce08
filter out legacy traj that can't be checked for dupes as easily
markmac99 Apr 11, 2026
5e44e3c
make dbdir and logdir if they dont exist
markmac99 Apr 16, 2026
b4daaa1
make sure traj objects include obs_ids when persisted
markmac99 Apr 20, 2026
300376b
bugfix - set candidate id and ref_ts in UTC
markmac99 Apr 20, 2026
7996132
sort trajbasics by jdt_ref
markmac99 Apr 21, 2026
cb989e3
bugfix for zero-length obsids
markmac99 Apr 21, 2026
58d7dea
correct return value from addTrajectory
markmac99 Apr 21, 2026
2d7a586
fix issues with updateTrajectoryDatabase
markmac99 Apr 21, 2026
ed9137d
missing verbose parameter
markmac99 Apr 24, 2026
781639e
bugfix to avoid crash if dupe traj arlready removed
markmac99 May 1, 2026
ca3fdf7
tiny tweak to avoid reprocessing the same events twice
markmac99 May 4, 2026
610b902
trying to fix bug #110
markmac99 May 5, 2026
2f55828
add some debug
markmac99 May 6, 2026
4ce305c
adding a little commentary
markmac99 May 6, 2026
4bbc6cb
Resolving issue where deduplication would crash if there were no
markmac99 May 6, 2026
32b9059
cosmetic tweak to remove irrelevant comment
markmac99 May 7, 2026
ba5e525
improvements in archiving/purging process
markmac99 May 13, 2026
89b9c7b
increase threshold for switching to only one timing estimation model.
markmac99 May 16, 2026
f4c6943
provide some additional logging when verbose is true
markmac99 May 16, 2026
04ee755
set default archivemonths to zero, meaning purge anything older than …
markmac99 May 28, 2026
9bf71d2
improve documentation
markmac99 May 28, 2026
f42cbca
missed an f-string
markmac99 May 28, 2026
68da3d4
update docs a little
markmac99 Jun 2, 2026
88258d6
minor tweak
markmac99 Jun 2, 2026
e77e398
Fix for issue 3 identified by Codex CLI
markmac99 Jun 3, 2026
0addda1
fix for issues 1,2,4 identified by Codex.
markmac99 Jun 3, 2026
4f1770c
fix for issue 6 identified by codex in an unused function. Also renam…
markmac99 Jun 3, 2026
c63d7b3
interestingly, this worked BEFORE my correction, because mcmode is a …
markmac99 Jun 3, 2026
9f67b33
improve handling of situations where the remote 'processed' file exists
markmac99 Jun 4, 2026
d2fc408
slight improvement in documentation
markmac99 Jun 4, 2026
02057b9
small doco changes
markmac99 Jun 4, 2026
11a431b
move processed phase1 from child nodes
markmac99 Jun 4, 2026
2f4fc0a
more clarification of the docs
markmac99 Jun 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 218 additions & 0 deletions Documentation/WMPL_Upgrades_2026April.md

Large diffs are not rendered by default.

Binary file added Documentation/node_structure.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion wmpl/Rebound/REBOUND.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
REBOUND_FOUND = True

except ImportError:
print("REBOUND package not found. Install REBOUND and reboundx packages to use the REBOUND functions.")
# don't print a message here as its already printed whenever REBOUND_FOUND is False
REBOUND_FOUND = False

from wmpl.Utils.TrajConversions import (
Expand Down
Loading