Add optional mtscomp support to Open Ephys binary reader

[arturoptophys]

Summary

This PR adds optional, transparent support for losslessly compressed Open Ephys
continuous signals stored as:

continuous.cbin
continuous.ch

OpenEphysBinaryRawIO continues to support the existing continuous.dat
layout. When both representations are present, the raw .dat file takes
precedence.

Why

mtscomp provides a lossless compression for ephys data streams. In its current implementation,
its mostly designed to work with SpikeGLX files and when using SpikeInterface read_cbin_ibl is implemented there.

However, mtscomp can also compress the .dat files from open ephys providing a 50-60 % reduction of file size. A possible usage scenario is compressing the file in-place so replacing the continuous.dat with continuous.cbin and continuous.ch.

Before this change, OpenEphysBinaryRawIO always expected continuous.dat. Reading a compressed recording therefore required an out-of-band full decompression step, duplicated the signal data on disk, and prevented consumers from treating compressed and uncompressed recordings through the same Neo interface.

This implementation keeps mtscomp optional and decompresses only the requested chunks in memory. It does not create a temporary decompressed .dat file. All the meta data from openephys are still read via the normal path.

What was implemented

Generic mtscomp buffer backend

• Added an mtscomp buffer type to BaseRawWithBufferApiIO.
• Imports mtscomp lazily, only when a compressed signal chunk is requested.
• Caches one mtscomp.Reader per (block_index, seg_index, buffer_id).
• Reuses the existing shared stream and requested-channel slicing logic, so
  Open Ephys ADC and SYNC separation remains outside the generic backend.
• Closes cached readers defensively during RawIO cleanup.
• Raises an actionable error when compressed traces are requested without the
  optional dependency.

Open Ephys storage detection

• Detects continuous.dat or a continuous.cbin/continuous.ch pair without
  wildcard matching.
• Preserves .dat precedence when both formats exist.
• Reports clear errors for a missing .ch file or missing signal data.
• Adds cbin and ch to the recognized extensions.
• Exposes a read-only uses_mtscomp property for downstream integrations.

Metadata validation

The .ch JSON metadata is read during header parsing without importing
mtscomp. The reader validates:

• required n_channels, sample_rate, dtype, and chunk_bounds fields;
• positive channel count;
• finite, positive sample rate;
• valid NumPy dtype;
• integer, zero-based, monotonically increasing chunk bounds.

For each compressed stream, these values are cross-checked against
structure.oebin. The final chunk bound is used as the compressed buffer's
sample count.

Backward compatibility

• The valid raw .dat path still uses the existing raw_filename field,
  memmap shape calculation, and raw buffer-description keys.
• raw_filename remains present in the dictionaries returned by
  explore_folder() for raw recordings.
• Compressed streams deliberately do not expose a misleading raw_filename
  pointing to a .cbin file.
• Invalid compressed artifacts are ignored when a valid continuous.dat
  exists.
• Events and timestamps continue to be read from the existing Open Ephys
  metadata paths.
• Existing ADC, SYNC, stream, block, segment, and channel-selection behavior
  continues through the shared code paths.

Optional dependencies

• Added the neo[mtscomp] optional extra.
• Selected mtscomp>=1.0.2 after a compatibility smoke test.
• Included tqdm in the extra because mtscomp 1.0.2 imports it at runtime but
  does not declare package dependencies.
• Added mtscomp to the test and all extras; it remains outside Neo's
  mandatory dependencies.

Tests added

The Open Ephys RawIO tests now build small synthetic recordings, copy them, and
compress the copies without modifying shared fixtures. Coverage includes:

• .dat-only, .dat plus compressed files, .cbin/.ch, missing .ch,
  orphaned .ch, and missing-data storage selection;
• raw-versus-compressed header and byte-for-byte signal equality;
• first sample, last sample, middle slices, and reads crossing mtscomp chunk
  boundaries;
• full-channel, contiguous-channel, and non-contiguous-channel access;
• neural, ADC, and SYNC stream separation and exact values;
• multiple blocks, segments, and signal buffers, including cache-key
  isolation;
• reader reuse across repeated requests and explicit reader cleanup;
• malformed or mismatched channel count, dtype, sample rate, and chunk bounds;
• header and event parsing while mtscomp imports are blocked;
• actionable failure only when a compressed signal chunk is requested without
  mtscomp;
• preservation of the legacy raw buffer-description contract and
  raw_filename;
• confirmation that signal access does not create a decompressed .dat file.

Validation

• Open Ephys focused suite: 14 passed, 6 skipped
• Other BaseRawWithBufferApiIO reader tests available locally: 1 passed, 33 skipped
• Direct synthetic RawBinarySignalRawIO read/cleanup smoke test: passed
• Python compilation checks: passed
• Optional-dependency metadata checks: passed
• git diff --check: passed

The skipped tests require Datalad-backed external fixtures, which were not available in the local environment.

The mtscomp compatibility spike used mtscomp 1.0.2 with Python 3.12.9 and
NumPy 2.4.4 and verified Reader.open(), shape, dtype, arbitrary frame and
channel slicing, and slices crossing compression chunk boundaries. The full
Python 3.10-3.13 matrix is left to CI.

[alejoe91]

Hi @arturoptophys

So you guys compress the Open Ephys .dat files with mtscomp? I thought that the main usage was from IBL, which created cbin for spikeglx.

Normally, we'd like to keep NEO as a library to read "default" file formats. This is especially true in this case, as we need core changes to the buffer logic. A potential alternative would be to add a separate reader either here or in SpikeInterface (if you guys use it). This is what we have done for IBL cbin files: https://github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/extractors/cbin_ibl.py

What do you think?

[arturoptophys]

@alejoe91 Well, we have not been using in on a large scale yet, but it is a consideration given that it helps a bit with the disc space.

And yes, our pipeline is based on SpikeInterface. It just seemed an easier task to modify the reader at the very base instead of a bit more complex implementation within SI. But I will take a look if it would be possible to create a custom reader which combines openephys meta-data extraction with the cbin - raw data extraction.

[alejoe91]

@alejoe91 Well, we have not been using in on a large scale yet, but it is a consideration given that it helps a bit with the disc space.

And yes, our pipeline is based on SpikeInterface. It just seemed an easier task to modify the reader at the very base instead of a bit more complex implementation within SI. But I will take a look if it would be possible to create a custom reader which combines openephys meta-data extraction with the cbin - raw data extraction.

If you are just starting with this, I strongly recommend using SpikeInterface compression directly!

It uses zarr and it's natively chunked and cloud compatible :)

See paper and docs

[arturoptophys]

@alejoe91 Thx i will look into it, is there a reverse operation for SI-compression (to restore original files) ? otherwise it becomes a bit of a forever buy-in in SI :)

[alejoe91]

@alejoe91 Thx i will look into it, is there a reverse operation for SI-compression (to restore original files) ? otherwise it becomes a bit of a forever buy-in in SI :)

Everything is saved to zarr, so it can be read directly through their API. I argue that this is less of a buy-in than mtscomp!

[h-mayorquin]

I agree with @alejoe91 here.