Lazy handling and saving of timestamps (and remove load_from/save_to folder_metadata)#4689
Lazy handling and saving of timestamps (and remove load_from/save_to folder_metadata)#4689alejoe91 wants to merge 24 commits into
load_from/save_to folder_metadata)#4689Conversation
load_from/save_to folder_metadata in save/load operationsload_from/save_to folder_metadata)
|
@chrishalcrow my feeiling was correct: in-memory timestamps are not propagated when saving in parallel...I'll propose a fix. |
| else: | ||
| d["time_vector"] = d["time_vector"][start_frame:end_frame] |
There was a problem hiding this comment.
Now, the time_vector of a FrameSliceRecording is the time vector of the parent: e.g. the full recordings time_vector.
But if you save a FrameSliceRecording as a binary file, it will save using get_times.
So if you load the sliced recording, this will have the reduced time_vector. So the save/load loop doesn't preserve all information.
I think the FrameSliceRecording should have the sliced time vector? This should still work lazily with memmap?
There was a problem hiding this comment.
this is correct because the save dumps a snapshot of the current state. I rewrote the logic to make it clearer though
…time_vector/has_time_vector functions
In the direction of #4671, this PR starts by rmeoving the unused mechanism to save to / load from metadata.
In addition, this PR also fixes saving timestamps in proper binary format.
For recording objects, the
_extra_metadata_to_folderwas saving timestamps tonpyfiles, but this is now handled by thewrite_binaryandBinaryRecordingExtractor, which writes in chunk to proper binary files. ThebinaryRecordingExtractoracceprs afile_timestamps_pathwith the location of the raw timestamps. Note that this is absolutely necessary for chronic data, since the old version was saving the entire timestamps array with numpy and this cannot be done with very long recordings. Old datasets are handled with the_handle_extractor_backward_compatibility.To further cleanup, this PR adds
save_properties_to/load_properties_from_binary_folderto remove duplicated code for saving and loading properties to numpy arrays.The cross serialization check was extended to always run against major past versions, to test a recording with timestamps, and to save recordings in parallel for
zarr/binary_folder.