The FieldTrip buffer for real-time access to EEG/MEG data

This page documents the implementation and possible use of the FieldTrip buffer. The audience is co-developers of FieldTrip and developers of other real-time applications.

The MATLAB implementation (i.e. the mex file) is by default included in the normal FieldTrip release and in the fileio toolbox. If you just want to use the FieldTrip buffer from within MATLAB, most of the information you'll find here is not relevant for you.

Processing EEG/MEG data in real-time in MATLAB first and foremost requires access to that data. The data is acquired in real-time by the acquisition system (EEG/MEG hardware and acquisition software) and can be streamed into a processing pipeline. To allow for full control of the timing in MATLAB and to implement the processing pipeline in MATLAB, the stream of data has to be buffered. The FieldTrip buffer is a network transparent TCP server that allows the acquisition client to stream data to it per sample or in small blocks, while at the same time previous data can be analyzed.

More generally, the FieldTrip buffer is used to communicate between separate application programs. One application program is responsible for the acquisition of the data, and it will write the data (and optionally also trigger events) to the buffer. Other application programs can connect to read the data and the events from the buffer and optionally also write new events (e.g. as output of some BCI classification algorithm) to the buffer. The buffer is implemented as a multithreaded application in C/C++ and it allows multiple clients to connect and read/write the data and events simultaneously.

The source code of the buffer can be integrated in any EEG/MEG acquisition or analysis system. The buffer server is instantiated by the acquisition software by first writing the header information to it, describing the number of channels and the sampling frequency. Subsequently, data and/or events can be written to it. There is also a command to flush (or empty) the buffer, which is for example needed when a new experimental run is started.

The buffer is compiled into a MATLAB mex file, which means that you can read from and write to the buffer in MATLAB. This allows you to process small segments of the real-time streaming EEG data in MATLAB, while incoming new data is buffered in a separate thread. Since the buffer allows multiple concurrent read connections, multiple MATLAB clients can connect to it, each analyzing a specific aspect of the data concurrently. The MATLAB mex file can be used to

• access a remote buffer, linked to the acquisition software which runs as a seperate program and possibly even on a seperate computer
• instantiate a local buffer, linked to the Matlab process as seperate thread within the MATLAB process

The binary TCP/IP network protocol allows client applications to be developed in an arbitrary programming language, e.g. C, Matlab, Java or Python. The network protocol documents the low-level protocol used for serializing requests and responses.

There is no fundamental problem in reading data that is acquired in real-time into Matlab. Even if that data comes streaming directly from an EEG amplifier, you still could read one sample at a time and collect it all in one matrix. But since Matlab is a single threaded application, collecting the continuously streaming data AND processing it at the same time is challenging. Imagine acquiring some samples of the EEG stream and then do a big computation on those samples, like an fft, during the time that Matlab needs to compute the fft, the new data that comes streaming from your amplifier would get lost. Assuming that the data is streamed over the network, the low-level TCP/IP network stack of your operating system will buffer it for some time, but duration that the data remains in that network stack buffer is not guaranteed.

Therefore we have decided to link the streaming of the data with a buffer, i.e. instead of having Matlab read one sample at a time, we created another standalone application that buffers the data. In Matlab, you can read any data from this buffer at a time that suits you. E.g. you can read the last 100 miliseconds of data (as a Nchans X Nsamples matrix) and then do your computation on that. During the computation Matlab will be busy, but the standalone buffer still captures the new data that comes in.

Acccess to the real-time buffer provided by the standard FieldTrip reading functions from the fileio module. The API allows for reading header information, event information, and blocks of data.

hdr    = read_header('buffer://localhost:port')       % this returns a structure with header information
event  = read_event ('buffer://localhost:port')       % this returns a structure with event/trigger information
dat    = read_data  ('buffer://localhost:port', ...)  % this returns a Nchans X Nsamples matrix with the data

The interface between Matlab and the real-time buffer is network transparent. That means that the buffering can be done on one computer, e.g. the one attached to the EEG amplifier, and the computations in Matlab can be done on another.

The low-level code for the realtime buffering is implemented in C. Streaming of the data from the EEG/MEG acquisition system to the buffer is system-dependent. The access to the data in Matlab is realized by mex files. The outline of the acquisition client, the buffer server and a Matlab client that processes the data is given below. The names in the flowcharts correspond to the functions in the C-code.

The leftmost box represents the application that is writing data into the buffer. The rightmost box represents the application that is doing the analysis. You can see that reading and writing from/to the buffer is completely symmetric and that there is no fundamental difference between a client that is continuously streaming data to the buffer and another client that is processing segments of the data.

The box in the middle represents the actual buffer, with the actual data being stored in computer memory/RAM represented by the blue cloud at the bottom. The header, data and event information is managed by the dmarequest function, where dma refers to direct memory access. The tcpserver function at the top waits for incoming connections. If a connection comes in, it is handed over to the tcpsocket function which reads the incoming request from the network, hands it over to dmarequest, and subsequently writes the response to the client back onto the network.

Given the symmetry in the design, it is possible to have a standalone buffer, i.e. have the tcpserver run in its own application with its own memory. It is also possible to link the tcpserver with the associated buffer memory (the middle box) to the acquisition software. Finally it is also possible to link the tcpserver with the associated buffer memory to the application that is doing the processing. The MATLAB mex file includes the tcpserver, which can optionally be started as a seperate thread.

If you want to implement the buffer in your own application, you should consider whether your application should maintain the buffer itself, or whether your application would be writing to a buffer that is maintained by another application.

The best way to get started with incorporating the FieldTrip buffer in your own software project, is by looking at the example applications in the buffer/test directory. It contains a number of examples

• demo_buffer implements a standalone buffer (middle box in the figure above)
• demo_sinewave implements a standalone data acquisition client (left box in the figure above)
• demo_combined implements both the buffer and the data acquisition in a single executable (left+middle box)
• demo_event writes events to the buffer, c.f. demo_sinewave which writes only data

There are also some test applications that you can also look at. These test applications each highlight a particular feature.

Your application is responsible for starting up the tcpserver. The tcpserver is a multithreaded function, which will spawn a seperate tcpsocket thread for every incoming connection. The tcpsocket thread reads a request from the network, processes the request, and returns a response. Possible requests that the buffer can process are PUT_HDR, PUT_DAT, PUT_EVT, GET_HRD, GET_DAT, GET_EVT, FLUSH_HDR, FLUSH_DAT, FLUSH_EVT. The tcpserver is instantiated using the following pseudo-code:

... start the multithreading...
tcpserver((void *)(&host));

or alternatively you can start the tcpserver in its own thread like this

... start the multithreading...
pthread_create(&tid, &attr, tcpserver, (void *)(&host));

A real example for starting the tcpserver can be found in the code in buffer/test/demo_buffer.c (only the buffer) or in buffer/test/demo_combined.c (which starts the tcpserver and subsequently writes simulated data to it).

After starting the tcpserver, your application can start writing to the buffer. The writing to the buffer should be done like in this pseudo code:

... construct a PUT_HDR request ...
connection = open_connection(hostname, port);
status = clientrequest(connection, request, &response);
close_connection(connection);

while (newdata)
  ... construct a PUT_DAT request ...
  connection = open_connection(hostname, port);
  status = clientrequest(connection, request, &response);
  close_connection(connection);
end

In the example above, the network transparent communication is kept stateless. It is also possible to keep the connection open for multiple subsequent requests, like this:

connection = open_connection(hostname, port);
... construct a PUT_HDR request ...
status = clientrequest(connection, request, &response);

while (newdata)
  ... construct a PUT_DAT request ...
  status = clientrequest(connection, request, &response);
end
close_connection(connection);

If your want your application only to write to a remote buffer, you can use the following pseudo code:

connection = open_connection(hostname, port);
... construct a PUT_HDR request ...
status = clientrequest(connection, request, &response);

while (newdata)
  ... construct a PUT_DAT request ...
  status = clientrequest(connection, request, &response);
end
close_connection(connection);

Depending on how the host is specified, the open_connection function will open a network socket (remote buffer) or will do nothing (local buffer). The clientrequest function will automatically determine whether it should write to a remote or to a local buffer, and it will call tcprequest or dmarequest (dma = direct memory access) respectively.

The dmarequest function always deals with the low-level memory management. If you want to keep all multithreaded code out of your application, which is only possible if you write to a remote buffer, then you can use the tcprequest function instead of the clientrequest function. The only c-functions that require multithreading are tcpserver which start a thread for every incoming connection, and dmarequest which uses a mutex to ensure that multiple memcopy requests are not interfering.

In principle the only support that has to be implemented specifically in Matlab is to allow it connect as client to the buffer server. That could be done using the pnet function from the tcp_udp_ip toolbox. For efficiency reasons and for simplicity, we also have made a direct implementation of the buffer source code in MATLAB by wrappint it into a mex file.

The primary responsibility of the mex file is to allow you from within MATLAB to make a connection to a local or remote buffer and read or write header information, data and/or events.

Furthermore, it is possible to instantiate the buffer as network transparent server, linked to the MATLAB application (the “tcpserver” thread). It is even possible to start a second thread that emulates an EEG acquisition system (the “sinewave” thread).

The interface of that mex function is documented here.

%  BUFFER manages and accesses the realtime data acquisition buffer
%  This function is implented as mex file.
% 
%  Use as
%    retval = buffer(cmd, detail, host, port)
% 
%  To read data from a buffer server over the network
%    hdr = buffer('get_hdr', [],     host, port)
%    dat = buffer('get_dat', datsel, host, port)
%    evt = buffer('get_evt', evtsel, host, port)
% 
%  To write data to a buffer server over the network
%    buffer('put_hdr', hdr, host, port)
%    buffer('put_dat', dat, host, port)
%    buffer('put_evt', evt, host, port)
% 
%  To implement a local buffer server and have other clients
%  connect to it at a specified network port
%    buffer('tcpserver', 'init', [], port)
%    buffer('tcpserver', 'exit', [], port)
% 
%  To implement a local acquisition client and have it buffer the
%  data locally 
%    buffer(acqclient, 'init')
%    buffer(acqclient, 'exit')
% 
%  To implement a local acquisition client and have it send the data
%  through the network to a buffer server
%    buffer(acqclient, 'init', host, port)
%    buffer(acqclient, 'exit', host, port)

An example for using the buffer function in Matlab would be the following. It creates a local tcpserver, a sinewave acquisition client, and subsequently runs a loop as fast as possible in which it reads the last second of data data and plots it in a figure.

buffer('tcpserver', 'init', 'localhost', 1234);
buffer('sinewave',  'init', 'localhost', 1234);
pause(2); % wait some time to fill the buffer

while (1)
  hdr = read_header('buffer:/localhost:1234');
  sel.begsample = hdr.nSamples - hdr.Fs + 1;
  sel.endsample = hdr.nSamples;
  dat = read_data('buffer:/localhost:1234');
  plot(dat');
end

Note that in the example above the MATLAB functions calls to get the header and data use the fileio interface to the buffer. This fileio interface (read_header/read_data/read_event) takes care of the reformatting of the low-level TCP messages. It also allows your MATLAB code to work both with a realtime buffer and with an offline file.

Building the source code on different platforms can appear challenging. The buffer has been successfully compiled and tested on Linux (32 and 64 bit), Windows (32 bit) and Mac OS X (PowerPC and Intel platform). We try to facilitate the compilation of the source code by supplying various build methods.

Generally, please note that no matter which platform, there are three different parts of the code, which are compiled in different steps.

• …/buffer/src contains the core buffer functions written in C. These are compiled into a library libbuffer.[a/lib] by using Makefiles or project files outside of Matlab.
• …/buffer/test contains demos and test applications written in C. These are also compiled outside of Matlab, but since they depend on libbuffer, they can only be compiled after that.
• …/buffer/matlab contains the sources of the MEX file buffer.mex???, which also depend on libbuffer. Since Matlab installations vary so much, we rely on the command mex to compile this part inside Matlab.

The best tested method for compilation is by using the Makefile (only for Linux, OS X and MinGW). For Windows, Borland C++ (version 6.0) project files are supplied. The buffer can also be compiled on Linux, Mac OSX, and Windows using cmake. Compiling on Linux and OSX is pretty straight forward while doing that on windows is a bit more tricky.

On Linux and OS X almost all dependencies should be pre-installed. You need a build-environment. This should be available by default on OSX. On Linux it can be installed by installing a package called something like “build-essentials”, which is probably available through the distribution's package-system (like apt on debian).

Compiling the buffer library should simply work by changing into the directory …realtime/buffer/src and typing make, with the expected outcome of a new file libbuffer.a. You can also build the demos and test applications by changing into …realtime/buffer/test and typing make again.

The MEX-file is compiled within Matlab. Just change into …realtime/buffer/matlab and type compile. This should only fail in case your version of GCC is either too old or (more likely) too recent, and Matlab will give you a corresponding warning. In particular, GCC>=4.2 doesn't seem to be supported by Matlab versions as new as 2009b.

If you have an unsupported GCC version, you should check whether your Linux distribution offers older packages of GCC. You can also compile GCC from source and install multiple versions of GCC alongside, but please refer to http://gcc.gnu.org for more information on this. If you have multiple versions of GCC, you will also have to tweak the file ~/.matlab/matlabXXXX/mexopts.sh where XXXX denotes the version of Matlab you are using: First locate the right “case” segment for your operation system, e.g. glnx86 for 32-bit Linux flavours. Then, modify the variables CC and CXX such that they point to the binaries of the right version. As a hint, these are often called gcc-4.2 and g++-4.2, that is, the version number forms part of the name. Maybe try and find the right files on the command line first.

The following was tested from 16-03-2010 to 18-03-2010.

Fields marked with “n.a.” refer to unavailable or non-functioning Matlab configurations. Note that although you can run 32-bit Matlab versions on a 64-bit machine, you will not be able to compile MEX-files with your native 64-bit compiler in this case.

Cross-version compatibility: It seems MEX-files compiled on any version >= 7.2 can be run on any other version >= 7.2 on the same type of machine. Possible trap: If you're trying to compile both 32 and 64 bit versions from the same source directory, make sure you always compile libbuffer.a using the same platform. If you get strange build errors, you might try to link a 64 bit MEX file to a 32 bit library, or vice versa.

On OSX the cmake software has to be downloaded and installed from here, or using FinkCommander.

In the top-level directory of the source-tree (the place where you see e.g. the “src” folder) create a new directory called e.g. “build” and enter it. Now issue cmake ../. This checks for dependencies. If cmake complains about not finding something, it must be installed first.

Typing make compiles and links the libaries and some executables all of which can be found in the “src” directory (n.b. not the “src” directory where the sources are but the newly created one in the folder cmake was called from).

One possibility of using the buffer on Windows is by downloading and installing the cygwin environment. In the setup.exe of cygwin you have to mark “cmake” and “make” to be installed. Afterwards it's the same as for Linux and Mac. The downside of this approach is that the libraries and executables depend on the cygwin-dll.

MinGW is a port of GCC that produces executables without special dependencies (or rather, they only depend on the Microsoft C run-time which is present on any Windows system). For compiling the code in both the src and test directories, you can just use the same Makefile as the one on Linux and MacOS X.

Unfortunately, Matlab doesn't recognise MinGW by itself, so for compiling the MEX-file, you will need to get Gnumex. This is a small collection of tools that wrap the MinGW utilities for usage by the mex command. Both MinGW and Gnumex are relatively easy to install, and you should stick to the defaults (in particular, please place MinGW in C:\MinGW). You should also put C:\MinGW\bin into your path. After installing, you do the following:

1. In a command prompt window, change to the src directory and type mingw32-make. If everything works, you should get a libbuffer.a file
2. Change to the test directory and type mingw32-make. This should produce demo_buffer.exe and further executables.
3. Within Matlab, change to the …buffer\matlab directory and type compile('mingw'). This should produce the file buffer.mexw32 in the …\fileio\private directory.

Assuming you've installed the tools and put the corresponding bin directory (e.g., C:\BCC55\bin) on the path, you may need to select BCC as the compiler in Matlab using mex -setup. Then you can use Makefile_BCC for compiling the core buffer functions and test applications, and compile.m for compiling the buffer.

1. In a command prompt window, change to the src directory and type make -f Makefile_BCC. If everything works, you should get a libbuffer.lib file
2. Change to the test directory and type the same. This should produce demo_buffer.exe and further executables.
3. Within Matlab, change to the …buffer\matlab directory and type compile('bcc55'). This should produce the file buffer.mexw32 in the …\fileio\private directory.

The src directory contains a VC2008 project and solution file, which you can use to compile the core functions. You should get a libbuffer.lib file as a result, which needs to be present directly within src. If for some reason it shows up somewhere else (e.g., in src\Debug or src\Release), you need to copy it. Compilation of the MEX file is again done in Matlab. Make sure that Visual C++ is selected as the compiler (mex -setup), change to ..buffer\matlab and type compile('vc').

If your favorite compiler is not listed above, or you have some special non-standard installation that gives you problems compiling the MEX file, you can tweak realtime/buffer/compile.m to suit your needs. You should try to only modify the contents of the variables 'cflags', 'extra_cflags', and 'ldflags' for compilation and linking options, ideally by adding a new case inside the switch statement. In any case, you should first make sure that you are able to compile the buffer C library and build/run the demos.

The best example for integrating the FieldTrip buffer in your own application is in the demo_sinewave and the demo_combined command-line applications. They implement a sinewave signal generator that writes its data to the buffer. Subsequently another application (e.g. implemented in MATLAB) can read from the buffer.

You should start by studying the example scripts in the 'realtime' directory, e.g. 'rt_signalproxy' as an example of how to write random data to a buffer, or 'rt_pooraudioproxy' as a simple example for reading data from some hardware device and writing into a buffer.

If you find you're getting problems you might want to think about the following:

• It's useful to distinguish between reading your data (say, from a hardware device), processing steps, and writing the data (say, to a buffer). Try to separate these things both in your head and your program.
• Can you nail down the source of the problems, or rather, are you really sure which of the components fails? If not, try to swap your components with something simpler. For example, in Matlab you can initially try to write the data to 'empty://' instead of a FieldTrip buffer. This will help you determining whether your acquisition part works correctly. The other way round, you can use the 'rt_signalproxy' (or ideas from it) to check whether the writing part works.
• Are you having performance problems? Maybe you can initially try to reduce the number of channels or the sample rate, and check whether the program logic is right.
• How much data are you trying to move? If for example you want to write to a buffer on a remote machine, you need to be aware of the physical limits of the network connection.
• You can try moving the actual buffer to another machine, and you do not need to have the buffer attached to a Matlab session (try using 'demo_buffer').
• You can try to report and compare the wall clock time with the sample time. Here, wall clock time refers to the real time your computer has run since you started the acquisition loop, while sample time is given by the amount of data processed so far, divided by your sampling frequency. These two numbers should ideally be the same or have a small constant offset, but they should not drift apart. Successive sample and clock times should also match your blocksize, that is, if you always read 500 samples at a time from a device, and your sampling rate is 1000 Hz, your measured clock time and calculated sample time should increase roughly by 0.5 seconds after each block.

• replace property handling with events on server side, keep properties for client side

• specification network messages (done)
• implementation network buffer (done, needs some debugging and auditing)
• implementation network communication over TCP (done)
• implementation direct memory access in similar fashion as network protocol (done)
• implementation acquisition client for simulated sinewave data (done)
• implementation acquisition client for CTF MEG system (done)
• compilation of code on Linux, testing and debugging (done)
• implementation buffering in matlab (done)
• compilation of code on Windows, testing and debugging (done)
• extend header content with channel labels (done)
• extend header content with calibration values (partly)
• replicate benchmark results in matlab (done/ongoing)

As of 18-05-2010, the FieldTrip buffer TCP server provides an automatic adaptation of requests and its responses to the endianness of the client. For example, if the buffer resides on an x86 PC, and data is written to it from a G4, the server will automatically convert the incoming packets (data/events/header information) to its own (little-endian) format, process the request, and then convert the response back to the (big-endian) format of the client. The opposite happens if the G4 is the server, and the PC the client. No conversion is done if both the server and the client run on the same type of machine (which includes the dmarequest as a notable case).

Portability note: non-unix systems may not allow read()/write() on sockets, but recv()/send() are usually ok. This is true on Windows and OS/2, for example.

On a 32 bits linux/RHEL4 machine (mentat068) I got the following error “libgcc_s.so.1 must be installed for pthread_cancel to work” Searching the internet did not give much helpfull suggestions, but using strace did help. It turned out that there was a version of libgcc_s on the LD_LIBRARY_PATH that was loaded prior to the system one. Changing it to first having /lib and then the rest solved it.

The following macro names are defined at compile time and may be handy for debugging:

__LINE__    Integer value representing the current line in the source code file being compiled.
__FILE__    A string literal containing the presumed name of the source file being compiled.
__DATE__    A string literal in the form "Mmm dd yyyy" containing the date in which the compilation process began.
__TIME__    A string literal in the form "hh:mm:ss" containing the time at which the compilation process began.
__cplusplus An integer value. All C++ compilers have this constant defined to some value. If the compiler is fully 
            compliant with the C++ standard its value is equal or greater than 199711L depending on the version 
            of the standard they comply.

The threaded functions should be checked with http://en.wikipedia.org/wiki/Reentrant in mind

The FieldTrip buffer source code is available from ftp://ftp.fcdonders.nl/pub/fieldtrip/modules.

The FieldTrip buffer source code is licensed under both the GNU General Public License (GPL) and the Berkeley Software Distribution License (BSD). You can choose the license that has the most advantages for you:

1. Use the BSD License to use the FieldTrip buffer commercially or
2. Use the GNU General Public License to use the FieldTrip buffer into your open source project.

A plain-text version of the GNU General Public License is included with the FieldTrip buffer source code release in the file LICENSE_GPL. It can also be found here.

A plain-text version of the BSD License is included with the FieldTrip buffer source code release in the file LICENSE_BSD. It can also be found here.

The FieldTrip buffer makes use of POSIX threads. This is available by default on UNIX-like operating systems. A well-tested implementation of POSIX threads is also available for windows at http://sourceware.org/pthreads-win32/. The pthreads-win32 library is released under the GNU Lesser General Public License (LPGL), which explicitely allows it to be used in closed-source commercial applications.