Skip to content

Lazy-mode for loading Sortinganalyzer - #4623

Merged
chrishalcrow merged 17 commits into
SpikeInterface:mainfrom
alejoe91:lazy-extensions
Jul 22, 2026
Merged

Lazy-mode for loading Sortinganalyzer#4623
chrishalcrow merged 17 commits into
SpikeInterface:mainfrom
alejoe91:lazy-extensions

Conversation

@alejoe91

@alejoe91 alejoe91 commented Jun 22, 2026

Copy link
Copy Markdown
Member

The SortingAnalyzer, especially for chronic data, can be very memory hungry. This PR adds a lazy mode at load that:

  • Reads the spike vector in lazy mode (memmaps memory folder, virtual memmap for zarr - TODO)
  • Reads vector-extensions as memmaps or keeps them as zarr array
  • Implements a memmap-like ZarrSpikeVector

NOTE: in lazy mode, save/select/merge/split operations are not permitted!

@alejoe91 alejoe91 added core Changes to core module performance Performance issues/improvements labels Jun 22, 2026
Comment thread src/spikeinterface/core/sortingfolder.py
@alejoe91
alejoe91 marked this pull request as ready for review June 24, 2026 15:19
@alejoe91
alejoe91 requested a review from samuelgarcia June 24, 2026 15:23
@alejoe91 alejoe91 added this to the 0.105.0 milestone Jul 15, 2026
@chrishalcrow

chrishalcrow commented Jul 21, 2026

Copy link
Copy Markdown
Member

Hello, when I do a get_data() on any of the spike-vector extensions (e.g. spikeamplitudes) on a zarr'd analyzer I get an error here:

return all_data.copy() # return a copy to avoid modification

as follows:

Traceback (most recent call last):
  File "<python-input-10>", line 1, in <module>
    sa.get_extension('spike_amplitudes').get_data()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/nolanlab/fromgit/spikeinterface/src/spikeinterface/core/sortinganalyzer.py", line 3465, in get_data
    return self._get_data(*args, **kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nolanlab/fromgit/spikeinterface/src/spikeinterface/core/analyzer_extension_core.py", line 1622, in _get_data
    return all_data.copy()  # return a copy to avoid modification
           ^^^^^^^^^^^^^
AttributeError: 'Array' object has no attribute 'copy'

@alejoe91

Copy link
Copy Markdown
Member Author

.get_data()

So when it's lazy we should force copy=False. Does it make sense?

Comment thread src/spikeinterface/core/sortinganalyzer.py Outdated
Comment thread src/spikeinterface/core/sortingfolder.py
Co-authored-by: Chris Halcrow <57948917+chrishalcrow@users.noreply.github.com>

@chrishalcrow chrishalcrow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me, and doesnt't change any defaults.
Note: it's pretty conservative. If you use lazy mode, you can't save/recompute/anything!

@chrishalcrow
chrishalcrow merged commit fe501e8 into SpikeInterface:main Jul 22, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Changes to core module performance Performance issues/improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants