---
title: "Read 'Blackrock' Neural Event & Signals"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Read 'Blackrock' Neural Event & Signals}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  eval = FALSE,
  comment = "#>"
)
```


## Introduction to `readNSx`

`readNSx` is an R package that converts `Blackrock-Microsystems` files (`.nev`, `.nsx`) into commonly used formats that are compatible with `R`, `Python`, `Matlab`. The package is designed for neuroscientist who study intracranial Electroencephalography and use systems like `Blackrock`, `Ripple`, etc. Support versions are `2.2`, `2.3`, `3.0`.

To install `readNSx` from `CRAN`,

```r
install.packages("readNSx")
```

To install development version

```r
install.packages("readNSx", repos = c(
  dipterix = "https://dipterix.r-universe.dev",
  CRAN = "https://cloud.r-project.org"))
```


## Basic Usage

*(Well, this package has no advanced usage.)*

### One-time import for the first time 

If you have never converted data, use the following one-time function call to import `.nev` and `.nsx` data:

```r
readNSx::import_nsp(
  path = "YAB_Datafile_001.nev", 
  prefix = "YAB_Converted_001", 
  exclude_events = "spike", 
  partition_prefix = "/part"
)
```

* `path`: path to `.nev` or any `nsx` file (such as `ns3, ns5`)
* `prefix`: path prefix of imported files (where the data will be saved)
* `exclude_events`: events to exclude from importing; default is "spike" (spike-sorting waveform and classes)
* `partition_prefix`: see Section "Anatomy of imported files".

Please check help function `?readNSx::import_nsp`

### Load imported data from R

*(If you have never imported data before, please check previous sub-section first)*

#### Example 1: load epoch from `NEV` comment data packet events

The following example loads the trial epoch table from `NEV` comment events. The `time_in_seconds` is the second of stimuli relative to `time_origin`.
Column `comment` is whatever comments sent by the task script (sent from `psychtoolbox` or related software that you use when collecting data).

```r
# prefix <- "YAB_Converted_001"
nev <- readNSx::get_nev(prefix)

# `time_in_seconds` is relative to time-origin
nev$header_basic$time_origin
#>        Year       Month   DayofWeek         Day        Hour   
#>        2022           8           5          26          15   
#>      Minute      Second Millisecond 
#>           4          10         156 

readNSx::get_event(nev, "comment")
#>     timestamp packet_id char_set flag data   comment   event time_in_seconds 
#> 1      683033     65535        0    0  255  audio-ba comment        22.76777 
#> 2      753242     65535        0    0  255  video-ba comment        25.10807 
#> ...
```

#### Example 2: get channel information and data

```r
# prefix <- "YAB_Converted_001"

# Gather information of channel 10
loaded <- readNSx::get_channel(prefix, channel_id = 10)

# Get NSx configurations, 
loaded$nsx
#> Basic header information (NSx):
#>   Internal type: NEURALCD
#>   Channel count: 152
#>   Sample rate: 2000 Hz
#>   Time origin: 2022-08-26 15:04:10 158ms
#> Extended header information (NSx):
#>   - CC (152 x 16, channels: 1-152): type, electrode_id, electrode_label, ...
#> Cache status:
#>   Prefix: ...
#>   Number of partitions: 1


# E.g. number of partitions (i.e. unpaused continuous recordings)
loaded$nsx$nparts
#> 1

# Get channel information
loaded$channel_info
#>    type electrode_id electrode_label physical_connector connector_pin
#> 10   CC           10        RA10-010                  1            10
#>    min_digital_value max_digital_value min_analog_value
#> 10            -32764             32764            -8191
#>    max_analog_value units high_freq_corner high_freq_order
#> 10             8191    uV              300               1
#>    high_freq_type low_freq_corner low_freq_order low_freq_type
#> 10              1         7500000              3             1
#>    sample_rate_signal sample_rate_timestamp which_nsp    filename
#> 10               2000                 30000         3 RA10-010.h5

# Get channel data
channel_signal <- loaded$channel_detail$part1$data; channel_signal
#> Class: H5D
#> Dataset: /data
#> Filename: ...
#> Access type: H5F_ACC_RDONLY
#> Datatype: H5T_IEEE_F64LE
#> Space: Type=Simple     Dims=798264     Maxdims=Inf
#> Chunk: 16384

# Get the actual numbers
channel_signal[]
```

Please use square bracket `channel_signal[]` to load the data into the memory.



## Anatomy of imported files

The imported file paths will start with `prefix`, which is specified by you, my dear users. In the following demonstration, I'll use a placeholder `<prefix>` to represent your inputs.

```
partition_info         - Name of continuous recording within the block, 
                          sample rates, starting time per partition per NSx
<prefix>_scans         - Basic information for current block
<prefix>_channels      - Electrode channel information ( ID, Label, ... )
<prefix>_events/       - NEV setting headers and data packets (events)
  - DIGLABEL           - Digital input setup
  - NEUEVLBL           - Channel labels
  - NEUEVWAVE          - Spike waveform settings
  - ...                - Other settings
  - event-***          - Data packets (digital inputs, comments...)
  - waveforms.h5       - Spike waveforms & cluster
<prefix>_ieeg          - NSx data folder
  - configurations.rds - NSx basic headers (versions, number of partitions, ...)
  - partition_info     - Continuous recording duration, start time, sample rates
  - nsx_summary.rds    - Internally used
  - part1/, part2/, ...- Channel folder
    - XXXX-001.h5      - Channel data file, each file correspond to a channel.
    - XXXX-002.h5         The file name ALWAYS ends with channel ID.
    - ...                 Each HDF5 file contains a "meta" and a "data" part,
                            "meta": JSON string of channel information
                            "data": numerical signal voltage data (in `uV`)
```

The signal data is stored at `<prefix>_ieeg/part*` by default, where `*` are positive integers representing the partition number. The `Blackrock` system allows users to pause their recordings and resume later without having to start a new block. A partition in `readNSx` represents one continuous recording within a "block". In most of cases, when there is no pause within a block, you will see only one partition. In some experimental settings, there could be one or more pauses, `readNSx` will store each continuous recordings in separate folders to make sure each partition is always continuous. The start time of the partitions will be stored in `partition_info.tsv`.

Please be aware that partition pattern `<prefix>_ieeg/part*` is not fixed. You can change the pattern via parameter `partition_prefix` when importing the data. For example, `partition_prefix="_part` will create partition files within directory `<prefix>_ieeg_part*`.

When `readNSx` stores the data, the channels are saved individually. For example, channel `1 (LA-2)` and `2 (LA-2)` are stored in separate `HDF5` files. This arrangement is out of the file-size and computational considerations:

* When storing all channels together, the file size will become super big. Reading one channel might resulting in reading the whole file (if handled poorly, or if using network drive)
* In some cases, `HDF5` files can only have one file pointers (file locked by some software). This could limit batch computing algorithms that can be paralleled at channel-level
* My personal research requires fast approach to copy or share data for quick inspection/analysis/visualizations (don't want to move large files around)

### File formats

The following file types will be generated:

* `.h5` (`HDF5` file): common file format that can include one or more data within a single file. 
  - `Matlab`: use `h5disp` to get enclosing data names; use `h5read` to read specific data
  - `Python`: use `h5py` package to load the files
  - `R`: if you want to load them individually, use `raveio::h5_names` to get enclosing data names; use `raveio::load_h5` to read specific data. `readNSx` also provides high-level functions to load them (see `?readNSx::get_channel`).
* `.tsv` (tab-separated values file): plain text files that can be easily read by many languages. You can open them in `Microsoft Office: Excel`.
* `.rds` (R object file): can only be read from R, internally used by `readNSx` to store data objects. Users do not need to read from these files (but also do NOT delete them, or `readNSx` will break). They serve as redundant files in case the `tsv` files are altered accidentally (for example, `Excel` might alter data formats automatically). If you see an `.rds` file sharing the same name as `.tsv`, they share the same information. 

