Skip to content

Lazy handling and saving of timestamps (and remove load_from/save_to folder_metadata)#4689

Open
alejoe91 wants to merge 24 commits into
SpikeInterface:mainfrom
alejoe91:save-timestamps
Open

Lazy handling and saving of timestamps (and remove load_from/save_to folder_metadata)#4689
alejoe91 wants to merge 24 commits into
SpikeInterface:mainfrom
alejoe91:save-timestamps

Conversation

@alejoe91

@alejoe91 alejoe91 commented Jul 14, 2026

Copy link
Copy Markdown
Member

In the direction of #4671, this PR starts by rmeoving the unused mechanism to save to / load from metadata.

In addition, this PR also fixes saving timestamps in proper binary format.
For recording objects, the _extra_metadata_to_folder was saving timestamps to npy files, but this is now handled by the write_binary and BinaryRecordingExtractor, which writes in chunk to proper binary files. The binaryRecordingExtractor acceprs a file_timestamps_path with the location of the raw timestamps. Note that this is absolutely necessary for chronic data, since the old version was saving the entire timestamps array with numpy and this cannot be done with very long recordings. Old datasets are handled with the _handle_extractor_backward_compatibility.

To further cleanup, this PR adds save_properties_to/load_properties_from_binary_folder to remove duplicated code for saving and loading properties to numpy arrays.

The cross serialization check was extended to always run against major past versions, to test a recording with timestamps, and to save recordings in parallel for zarr/binary_folder.

@alejoe91 alejoe91 added core Changes to core module refactor Refactor of code, with no change to functionality labels Jul 14, 2026
@alejoe91 alejoe91 added this to the 0.105.0 milestone Jul 15, 2026
Comment thread src/spikeinterface/core/tests/test_baserecording.py Outdated
Comment thread src/spikeinterface/core/tests/test_baserecording.py Outdated
Comment thread src/spikeinterface/core/baserecording.py Outdated
@alejoe91 alejoe91 changed the title Remove load_from/save_to folder_metadata in save/load operations Lazy handling and saving of timestamps (and remove load_from/save_to folder_metadata) Jul 20, 2026
@alejoe91

Copy link
Copy Markdown
Member Author

@chrishalcrow my feeiling was correct: in-memory timestamps are not propagated when saving in parallel...I'll propose a fix.

Comment thread src/spikeinterface/core/baserecording.py Outdated
Comment thread src/spikeinterface/core/baserecording.py
Comment thread src/spikeinterface/core/core_tools.py Outdated
Comment on lines -71 to -72
else:
d["time_vector"] = d["time_vector"][start_frame:end_frame]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, the time_vector of a FrameSliceRecording is the time vector of the parent: e.g. the full recordings time_vector.

But if you save a FrameSliceRecording as a binary file, it will save using get_times.

So if you load the sliced recording, this will have the reduced time_vector. So the save/load loop doesn't preserve all information.

I think the FrameSliceRecording should have the sliced time vector? This should still work lazily with memmap?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is correct because the save dumps a snapshot of the current state. I rewrote the logic to make it clearer though

@alejoe91 alejoe91 added the performance Performance issues/improvements label Jul 22, 2026
Comment thread src/spikeinterface/core/time_series_tools.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Changes to core module performance Performance issues/improvements refactor Refactor of code, with no change to functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants