Realtime analysis - Offline and online processing of simulated and real data

This tutorial presents a number of example scripts which will show how to use the realtime module in practice. We move from simple to more complex examples. All the examples make use of the rt_process function, which processes incoming data in realtime using a specified trialfun and bcifun. The trialfun specifies how to extract trials from the incoming data whereas the bcifun performs an analysis on individual trials (e.g., preprocessing, feature extraction, and classification).

In this example, we will use an offline MEG dataset and show how a neurofeedback application can be implemented using the realtime module. Note that this is just for demonstration purposes since we run the system offline (pretending that the data is acquired online). In this demonstration, we clarify some fundamental concepts that will be needed in the implementation of an online neurofeedback system. We use a dataset where the subject was required to covertly attend to the left and right visual field.

The first thing to do in order to perform an offline 'simulated' realtime analysis is to specify both the trialfun and the bcifun.

As our trialfun, we use the trialfun_realtime function which can be found in the Fieldtrip trialfun folder. This trialfun runs in synchronous or asynchronous mode, where the first only extracts parts of the data linked to a stimulus trigger and where the latter extracts data in a continuous fashion, without paying attention to the triggers. Which mode you should use depends on the particular application. E.g., when a BCI system uses a cue to determine the start of a trial, we would use a synchronous mode, whereas a neurofeedback application would probably use an asynchronous mode. For our example we specify the following options:

cfg = [];
cfg.readevent = 'no'; % indicates asynchronous mode
cfg.blocksize = 1; % block size of each trial in seconds
cfg.offset = 0; % offset w.r.t. the end of the previous trial in seconds
cfg.bufferdata = 'first'; % start to read from the beginning or end of data (typically 'first' for offline data, 'last' for online data) 

As our bcifun we use the bcifun_latidx function, which computes a lateralization index between left and right hemisphere occipital alpha power. You may study its contents by opening the file in Matlab.

For our example we specify the following options:

cfg.foilim = [8 14]; % the frequency band over which to average
cfg.avgchan = { [1:19] [20:38] }; % the channel indices over which to average; we expect exactly two groups in order to compute the lateralization index  

Now, we may call rt_process using these two functions and our offline CTF dataset as follows:

cfg.trialfun = @trialfun_realtime;
cfg.bcifun   = @bcifun_latidx;
cfg.readevent = 'no'; % we only need to read events for synchronous setups to extract the triggers
cfg.dataset = '/Volumes/Elements/data/alphalat/subject11/Sander_AlphaLat_20080530_01.ds'; % path to our dataset
cfg.channel = {'MRO' 'MLO'}; % left and right occipital channels; these correspond to the channel indices in avgchan

rt_process(cfg);

The output of this simple script is something like the following:

processing 25300 trials using bcifun_latidx
processing segment 1 from sample 1 to 120, condition = NaN
0.1023
processing segment 2 from sample 120 to 240, condition = NaN
0.10383
processing segment 3 from sample 240 to 360, condition = NaN
0.10377
...

Hence, it will print the output of the bcifun, namely the computed lateralization index. Typically, we wouldn't want to print the output but rather send it to some other application that is able to feed the command back to the user. This is realized by the ''cfg.ostream' option of rt_process. This will specify how the command is send via ft_write_event. Since we assume that all processes run on the same machine for this demonstration, we may use a first-in first-out (FIFO) protocol, which blocks the application until a new command is received. This is realized as follows:

system('unlink /tmp/tmp.pipe'); % make sure the pipe is empty
cfg.ostream = 'fifo:///tmp/tmp.pipe'; % use /tmp/tmp.pipe as a fifo pipe (note that the pipe should be empty at startup)
rt_process(cfg);

however, the pipe will block since there is no reading process. Hence, immediately after calling rt_process, we should start a process which reads from the pipe. Here, we will use app_neurofeedback, which is a simple example application written in Psychtoolbox. So, in a separate Matlab session, we call:

cfg.istream = 'fifo:///tmp/tmp.pipe';
app_neurofeedback(cfg);

It is convenient to run these processes using Matlab and shell scripts such as the following:

% nf_processing script
cfg.blocksize = 1;
cfg.offset = 0;
cfg.bufferdata = 'first';
cfg.foilim = [8 14];
cfg.avgchan = { [1:19] [20:38] };
cfg.trialfun = @trialfun_realtime;
cfg.bcifun   = @bcifun_latidx;
cfg.readevent = 'no';
cfg.dataset =  '/Volumes/Elements/data/alphalat/subject11/Sander_AlphaLat_20080530_01.ds';
cfg.channel = {'MRO' 'MLO'};
cfg.ostream = 'fifo:///tmp/tmp.pipe';
rt_process(cfg);

#!/bin/sh
# shell script to spawn the processes

# clear and create the pipe
unlink /home/electromag/marvger/tmp.pipe
mkfifo -m 0666 /home/electromag/marvger/tmp.pipe

( ( xterm -title 'nf_processing' -e matlab -nodesktop -nosplash -r "nf_processing;" &) && sleep 1)
( ( xterm -title 'nf_application' -e matlab -maci -nodesktop -nosplash -r "cfg.istream = 'fifo:///tmp/tmp.pipe'; app_neurofeedback(cfg)" &) && sleep 1) 

# Note: -maci is only relevant for 64-bit Macs since Psychtoolbox only runs in 32-bit mode

Note that an alternative to the multiple processes is to define the application behaviour within the bcifun. With some tricks (e.g., using persistent states), this is possible. In practice however, acquisition, analysis and presentation may all take place on separate machines.

In this example we proceed similarly to the previous example but now the data will analyzed in synchronous mode. Furthermore, in BCI we have the additional complication that a session can be divided into a training session and a test session (which may possibly run in parallel in adaptive BCIs) and the classifier state should remembered during the experiments. This behavior is captured by the bcifun using persistent states. Here, we use bcifun_frqclf which classifies frequency data. Note that we make use of cfg.count which is passed on as the current example number by rt_process. Furthermore, trialfun_realtime will give us the observed triggers for each trial. We need this information to label our training data.

A shell script which runs the simulated realtime BCI without introducing additional Matlab scripts could look like this

#!/bin/sh
INIT="cfg.triggers = [4 2];"
INIT="$INIT cfg.blocksize = 1;"
INIT="$INIT cfg.offset = 1;" # collect trial from 1 - 2 seconds after the trigger (cue)
INIT="$INIT cfg.bufferdata = 'first';"
INIT="$INIT cfg.foilim = [8 14];";
INIT="$INIT cfg.clfproc = clfproc({standardizer() svmmethod()});"
INIT="$INIT cfg.trialfun = @trialfun_realtime;"
INIT="$INIT cfg.bcifun   = @bcifun_frqclf;"
INIT="$INIT cfg.readevent = 'yes';"
INIT="$INIT cfg.dataset = '/Volumes/Elements/data/alphalat/subject11/Sander_AlphaLat_20080530_01.ds';"
INIT="$INIT cfg.channel = {'MRO' 'MLO'};"
INIT="$INIT cfg.ostream = 'fifo:///tmp/tmp.pipe';"

unlink /tmp/tmp.pipe
mkfifo -m 0666 /tmp/tmp.pipe

( ( xterm -title 'rt_process' -e matlab -nodesktop -nosplash -r "${INIT} rt_process(cfg);" & ) && sleep 1) || ( echo ERROR 'rt_process' exit 1 )
( ( xterm -title 'app_neurofeedback' -e matlab -maci -nodesktop -nosplash -r "cfg.istream='fifo:///tmp/tmp.pipe'; app_neurofeedback(cfg);" & ) && sleep 1) || ( echo ERROR 'app_neurofeedback' exit 1 )

Note that we have specified our triggers and set cfg.readevent = 'yes' since we use a synchronous instead of an asynchronous setup.

There are multiple ways to define your own BCI behaviour. For instance, you might consider separating training and test behaviour in different bcifuns that get called using separate processes, or you could load your pretrained classifier from disk, or ….

Sending the commands to an application and specification of the application proceeds in the same way as for the previously encountered neurofeedback example.

In previous examples we have seen how to process simulated realtime data. Processing of actual realtime data proceeds in the same way although there are some additional complications. Here, we discuss how to set up an online BCI using the CTF system. First, make sure the latest version of Fieltrip is available on all machines you are going to use. Then, turn on the MEG system and start CommandCenter on the acquisition machine. Here, you need to go through all the standard initialization steps. It is very important to select a suitable sampling frequency etc. as this may heavily influence the behaviour of the online BCI. As a default you could consider selecting the realtime study. Acquisition does not need to be started explicitly if you don't want to collect the data on disk.

The following step is to start AcqBuffer on the acquisition machine. AcqBuffer reads in the data and writes it to a ring buffer (in case of a segmentation fault please restart Odin). This ring buffer can then be used to transfer data to other communicating processes.

One of the simplest online setups is outlined above. Here, we read directly from the AcqBuffer on the acquisition machine and send commands to the presentation machine using TCP/IP by means of the following input and output streams

<code matlab>
cfg.istream = 'shm://';
cfg.ostream = 'tcp://presentation011:1976';
</code>

On the presentation machine, we use

cfg.istream = 'tcp://localhost:1976';

to indicate how commands are received. Note that the acquisition machine is now also responsible for the analysing the data (converting the data into commands). Other than this, the setup is the same as in the offline case.

A problem with direct reading of the AcqBuffer is that acquisition and analysis takes place on the same machine. This can be undesirable for reasons of efficiency. For instance, at the DCCN, the acquisition machine does not run the latest version of Matlab and hence, does not support the classification module. Furthermore, it might be useful to have a dedicated machine to do the analysis. For example, at the DCCN we may want to use the mentat cluster.

At the DCCN, we can use the FieldTrip buffer as follows. The acquisition machine send the AcqBuffer data to a FieldTrip buffer using rt_ctfproxy. This can be achieved with the following shell script:

#!/bin/sh
INIT="addpath(genpath('/home/common/matlab/fieldtrip'));" # make sure the latest version is added to the path
INIT="$INIT cfg.jumptoeof='yes';" 
if [ -n "$1" ]; then
    INIT="$INIT cfg.channel = $1;"
fi
( ( xterm -title 'rt_ctfproxy' -e /opt/matlab73/bin/matlab -nodesktop -nosplash -r "${INIT} rt_ctfproxy(cfg);" & ) && sleep 2 ) || ( echo ERROR 'rt_ctfproxy' exit 1 )

This shell script uses

buffer://localhost:1972

as the default communication protocol. Furthermore, cfg.jumptoeof is used to immediately start analyzing the latest data. We may perform channel selection at this level by calling the shell script with the channel names as an additional argument; e.g., by calling myscript ”{'MLO34' 'MRO34'}”.

This takes care of data transmission to other processes. Now, we still need to analyse this data and send commands to the presentation machine. This is achieved by calling rt_process on our analysis machine and an application program (such as app_neurofeedback) on the presentation machine. However, communication is now via the Fieldtrip buffer, so the analysis machine uses

cfg.dataset = 'buffer://odin:1972'

where odin is the acquisition machine. Furthermore, events are written using the TCP/IP protocol. Hence, we use

cfg.ostream = 'tcp://presentation011:1976';

on the analysis machine, and

cfg.istream = 'tcp://localhost:1976';

on the presentation machine. Running

rt_process([])

on the analysis machine demonstrates readout of the fieldtrip buffer using bci_latidx. Running

rt_process(cfg)

and

app_neurofeedback(cfg)

with the described cfg options demonstrates the full BCI cycle using a separate acquisition, analysis and presentation machine based on the neurofeedback application.