This page is part of the documentation series of the Fieldtrip buffer for realtime aquisition. The FieldTrip buffer is a standard that defines a central hub (the FieldTrip buffer) that facilitates realtime exchange of neurophysiological data. The documentation is organized in four main sections, being:

1. description and general overview of the buffer,
2. definition of the buffer protocol,
3. the reference implementation, and
4. specific implementations that interface with acquisition software, or software platforms.

This page provides a formal description of how the communication over the network is performed between the FieldTrip buffer server and clients.

The MATLAB implementation (i.e. the mex file) is by default included in the normal FieldTrip toolbox release. If you just want to use the FieldTrip buffer from within MATLAB, most of the information you'll find here is not relevant for you.

The FieldTrip buffer is designed to facilitate transporting of data samples and events (also called markers) from an acquisition device (e.g., an EEG amplifier) to one or more (analysis) programs in real-time. Hereafter, we will refer to both the acquisition part and the analysis part(s) as clients, and we do not distinguish between clients that mostly write and clients that mostly read. Since we explicitly wish to include Matlab scripts and other single-threaded programming environments as possible clients, it is apparent that we need some sort of buffering of data and events. The application (or part thereof) that does this is refered to as the server.

The FieldTrip buffer represents three elements: the header structure, the data matrix and a list of event structures. Since we target real-time usage, both the data matrix and the list of events will usually be implemented as a ring buffer, which means that after a time, old data samples and events will not be accessible anymore.

The binary TCP/IP network protocol allows client applications to be developed in an arbitrary programming language, e.g. C, Matlab, Java or Python. The network protocol documents the low-level protocol used for serializing requests and responses.

We do not aim to provide or specify:

• any kind of remote procedure calls, or direct communication between clients,
• an explicit way of setting up experiments,
• merging data from different acquisition systems,
• writing data to disk,
• a complete implementation for all possible programming languages (e.g. FORTRAN, COBOL),
• a complete implementation for all possible operating systems (e.g. Amiga, OS/2 Warp, Windows 95).

For the communication the client sends a request, which always results in a response from the server. The request can for example be GET_DAT, and the response would contain the data. In case the request cannot be fulfilled (e.g. the requested data is not available), the server responds with an error. The request and the response consist of a binary message sent over TCP, which is described in detail below.

The buffer represents three elements: the header structure, the data matrix and a list of event structures. The data matrix and the event list are both implemented as a ring buffer.

For the communication the client sends a request, which always results in a response from the server. The request can for example be GET_DAT, and the response would contain the data. In case the request cannot be fulfilled (e.g. the requested data is not available), the server responds with an error. The request and the response consist of a binary message sent over TCP, which is described in detail below.

We are still unsure about how to support backwards-compatibility in the buffer. Currently (V1) request will only succeed if the client and the server use the same protocol version number.

We can stick to the fixed 8-byte prefix containing version, command and bufsize (indicating the size of the remaining message), but one of the more important issues is a strategy that allows for both flexibility and compatibility when it comes to fixes and extensions.

TODO: short text describing the current state of different versions.

Every request and response starts with the following fixed-size 8 byte structure which corresponds to the definition of messagedef_t in message.h in the reference implementation.

The buffer always stores data in its native format, that is, on a x86 compatible processor all numbers are stored in little-endian format, and on a PowerPC (G4/G5/…) numbers are stored in big-endian format. The server will automatically convert incoming requests and its responses to the endianness of the client, which is detected by the 16-bit version field. Clients can always transmit requests in their own native format, and can assume that the data they receive is in their native format.

Because we are relying on detecting endianness by looking at the 16-bit version number, we will get problems if we ever want to bump the version number to 256 or bigger.

This request is used for initialising the buffer and setting header information like the number of channels and the sampling frequency. Clients need to transmit a fixed structure and optionally a set of chunks (chunk_t in message.h). The command number of this request is 0x101 (=257 in decimal notation).

The fixed part (24 bytes) consists of the following (headerdef_t in message.h)

The fixed part of the header is sufficient to interpret the data as a feature vector of length nchans, which is sampled at regular intervals (fsamp). Additional information that are system specific such as channel names for EEG/MEG, or calibration values (in case the EEG/MEG data type is int16 or int32) are not represented in the fixed part of the header. In order to transmit this type of extended header information, one needs to use the backwards-compatible extension of chunks (see at the bottom of this page).

For this request, the buffer server will only return a fixed 8-byte message consisting of the already known version;command;bufsize triple. If the request was successful, the returned command will have the value 0x104 for PUT_OK. Otherwise, for example if the server could not allocated the required memory, command will contain the value 0x105 (=261) for PUT_ERR. Since there is no additional information attached to the response, bufsize should be 0. Every other response means that a communication error occurred and should be treated as such.

Suppose you want to write header information that corresponds to a NIFTI-1 file for transmitting fMRI scans. For this you could pick the dedicated NIFTI-1 chunk type (FT_CHUNK_NIFTI1=5), the length of which is always 348 bytes. Lets assume the data is given as 16-bit integers (DATATYPE_INT16=6), the sampling frequency is 0.5 Hz (for a repetition time of 2 sec.) and that your volumes contain 64x64x20 = 81920 voxels. The complete request would then look like this

Note that for every additional chunk, you need to increase the bufsize field of both the message definition and the header definition by the size of the chunk (including the 8 bytes for its type and size field).

This request is the opposite of PUT_HDR and uses the same data structures. Its command number is 0x201 (=513). The client just sends the 8-byte triple

and on success will receive the fixed message definition with command containing 0x204 (=516) for GET_OK, followed by the fixed header definition structure and optional chunks. The client should determine whether chunks are present by looking at the bufsize fields. In contrast to the PUT_HDR request, the returned header definition structure will contain the actual number of samples and events that have been written to the buffer so far.

If an error occurs, or no header information has been written yet, the buffer server will only return the 8-byte triplet version;command;bufsize, with the command value set to 0x205 (=517) for GET_ERR, and bufsize=0 to indicate no extra message payload.

As an example, suppose another client reads back the header information we wrote in the PUT_HDR example, but in the meantime 4 samples and 3 events have been written. The response would then look like this:

This request is used for storing samples in the buffer by appending them to those already present. Clients need to transmit a fixed structure followed by the actual data samples, where samples (= one value each from multiple channels) need to be transmitted contiguously. The command number of this request is 0x102 (=258 in decimal notation).

The fixed part (16 bytes) consists of the following (datadef_t in message.h)

The response will be the 8-byte triple version=1,command,bufsize=0. On success, command will contain 0x104 (=260) for PUT_OK. In case of an error, for example if the data_type or nchans fields do not match the values previously written together with the header information, or if no header information has been written at all so far, command will contain 0x105 (=261) for PUT_ERR. Again, every other response should be treated as a communication error.

Suppose you want to append 200 samples from 32 channels of single precision data. In this case, the data_type field contains the value 9 (DATATYPE_FLOAT32 in message.h), and the size of all samples is 200*32*4 = 25600 bytes. The complete request would look like this:

This request is used for retrieving data samples from the buffer and comes in two flavours. The first variant just asks for all samples that are currently present in the buffer (note that this does not necessarily correspond to all samples that have been written so far, since old samples might have fallen out of the internally used ring buffer already). For this, the client just sends 8 bytes like the following:

The client can also request a specific interval of samples by transmitting 2 indices as unsigned 32-bit integers (see datasel_t in message.h). For example, to retrieve samples with index 4 up to (and including) 15, you would send

Note that changed bufsize reflects the extra message payload, and that indices are zero-offset. That is, the first sample ever written has the number 0, and the request above thus asks for 12 samples, starting with the 5th!

On success, the server will respond by the fixed message definition and the fixed data definition structures, followed by the actual data samples. Assuming 32 channel single precision data again, you would get

If begsample and endsample have not been specified with the request, the response has the same form, and it will be hard to determine which samples the server actually returned.

If an error occurs, for example if the requested indices are outside of the range that is currently present in the ring buffer, the server responds with the triple version=1;command;bufsize=0, where command=0x205 (=517) for GET_ERR.

Using this request, clients can store events (in a ringbuffer similar to that used for the data samples). The command number of this request is 0x103 (=259). As for the PUT_DAT request, you can only append events, and at some point old events will fall out of the ring buffer. Every event is described by a fixed structure followed by a variable-length field that contains the event's type and value.

The fixed part (32 bytes) consists of the following fields (eventdef_t in message.h):

After the fixed part, you need to first transmit the type in the form specified by type_type and type_numel, and after that the value of this event. Multiple events can be transmitted one after another. Please note that the bufsize field above should always contain type_numel times the size per type element plus the same product for the value field.

The response of the buffer server will be the usual triple version=1,command,bufsize=0, with command equal to 0x104 (=260 / PUT_OK) on sucess, or 0x105 (=261 / PUT_ERR) in case an error occured.

Suppose you want to add two events that relate to sample 10 and 12, respectively, whose type is the string “Button” and whose value is “Left” and “Right”. We'll use offset=duration=0, and since both type and value are given as strings, the type_type and value_type fields both have the value 0 (for DATATYPE_CHAR, see message.h). The type_numel and value_numel fields contain the lengths of the respective strings, and since a character only takes one byte, the bufsize field is the sum of both string lengths. All in all, the complete request for this would be:

This request is used for retrieving events from the buffer. Similarly to GET_DAT, it comes in two flavours. The first variant asks for all events that are currently present in the buffer, and is composed of an 8-byte triple as follows:

The client can also request a specific interval of events by transmitting 2 indices as unsigned 32-bit integers (see eventsel_t in message.h). For example, to retrieve samples with index 8 up to (and including) 10, you would send

Note that the changed bufsize reflects the extra message payload, and that indices are zero-offset. That is, the first event ever written has the number 0, and the request above thus asks for 3 events, starting with the 9th!

On success, the server will respond by the fixed message definition followed by the actual events in the same format as for PUT_EVT. The only difference to PUT_EVT is indeed that on success, the returned command value has the code 0x204 (GET_OK). If an error occurs, the server does not transmit any events and just responds with the triple version=1;command;bufsize=0, where command=0x205 (=517) for GET_ERR.

Using this request, the client can ask to remove all data from the buffer. All events and the header information are kept, although of course the nsamples field of the fixed header structure will be reset to 0.

The client sends the following 8 bytes

and on success the server responds with

This request is for removing all events from the buffer. All data samples and the header information are kept, although of course the nevents field of the fixed header structure will be reset to 0.

The client sends the following 8 bytes

and on success the server responds with

This request if for clearing all contents of the buffer, including samples, events, and any chunks present in the header.

The client sends the following 8 bytes

and the server responds with

This request is intended to be used in a realtime processing loop to poll for newly arrived samples or events. The client sends a fixed message consisting of

where timeout is given in milliseconds. The server will respond only after either

• the sample count of the buffer is higher than nsamples, or
• the event count of the buffer is higher than nevents, or
• more than timeout milliseconds have passed since the receipt of the request.

Then, the response consists of the fixed sequence

where nsamples and nevents contain the sample and event count of the buffer at the time the response is sent. Note that depending on timeout conditions, either or both of these quantities can be below the given threshold.

If no header information is present in the buffer, the server replies immediately with the triple

This request can also be used as a light-weight GET_HDR replacement where no chunks (see below) are transmitted. Just set timeout=0 and the server will respond with the nsamples and nevents quantities immediately.

If you only want to wait for new events, and do not care about data samples (yet), you can set the nsamples field in the request to a very high number (2^32-1 as the biggest uint32). The same works for the opposite case where you're interested in samples, not events.

As already mentioned, the PUT_HDR request can contain a variable part consisting of chunks. These are transmitted one after another (chunk_t in message.h). Their structure is

The chunk type and the content of the chunk are system specific. If the client application does not recognize the chunk type, it can skip over it.

If chunks are present, they will be transmitted in every GET_HDR request, using the same format. Care must be taken to adapt the processing logic of the client in case the header contains a large chunk (such as a ~3MB big CTF .res4 file), that is, the GET_HDR request should be made only as often as necessary, and replaced by WAIT_DAT.

The following is a list of currently defined and used chunk types:

This chunk can represent unspecified binary data, which can for example be used during development of site-specific protocols. The buffer server will make no attempt to interpret the contents.

This chunk is used for labelling the channels (or elements of the feature vector) that are represented in the buffer. The form of the representation is inspired by the BrainProducts RDA protocol.

This chunk is useful for specifying that a channel can have one of a discrete number of different types, e.g. data = “meg_ad_eog\0\1\1\1\1\3\3\2\2” could be used for a system with 8 channels, the first four of which are for MEG, then 2 channels EOG, then 2 channels A/D.

This chunk is used for labelling the channels (or elements of the feature vector) that are represented in the buffer.

This chunk is also inspired by the BrainProducts RDA protocol, and describes the mapping from A/D values to physical quantities such as micro-Volts in EEG.

This contains an arbitrary number of key/value pairs, each of which is given as a 0-terminated string. An empty key (=double 0) indicates the end of the list. Example: “amplifier_gain\0high\0noise_reduction\0active\0\0”.

Not used so far.

Used for transporting a NIFTI-1 header structure for specifying fMRI data.

Used for transporting the sequence protocol used in Siemens MR scanners (VB17). This is also part of the DICOM header (private tag 0029:0120) that these scanners write.

This chunk contains a .res4 file as written by the CTF MEG acquisition software in its normal binary (big-endian!) format.

The client sends an 8-byte triple with version=2, command=GET_REQ_VER (number to be specified later) and bufsize=0. The server responds with the 8-byte message definition, followed by a list of triples in the form command,oldest_version,newest_version, that is, for each command the server might be able to handle different versions. In this way, clients can check whether the server understands all the requests that will be used on the client side.

Having this scheme means that we can easily add a new request without changing any of those already specified and used by clients, e.g., we could introduce a “version 3” variant of a GET_EVT call, but leave all other calls unchanged (including the V2 GET_EVT call). In this way V2 clients would be able to talk to the V2-3 server without problems, but a newer client would be able to make use of the new call (and, say, talk V2 for all other requests). As an extreme, we could actually keep all V1 requests as they are, and V1 clients would never notice a change to the server (SK: I would advise against this, because this means we also need to stick to the limited error reporting we have in V1). Newer clients could send this directly after connecting, and then either report an error (if the server does not provide all the requests they need), or adapt their protocol to the capabilities of the server.

Currently (V1) the server sends an error identifier that is specific to each request, e.g., GET_ERR for GET_XXX requests, and PUT_ERR for PUT_XXX requests, but it does not specify the type of error. We can drop the GET/PUT/… distinction since this provides almost no value, and should rather define more informative symbols like this (numerical values to follow):

When retrieving events, the server should always send the index of each event along with the response, as this makes filtering and housekeeping much easier. With a minimal change versus V1, we can just amend the GET_EVT request so it trasmits the event index before the rest of its definition, so for this request only, events would be transported by the following structure (fixed part = 36 bytes now):

After this, we would transmit the type and value fields in the same way as V1.

On top of that, we can think about providing filter mechanisms on the server side. For the sample, offset and duration field, we can easily support filtering on a (min/max) condition. type and value could be filtered for an exact match. All filter conditions would be combined with a logical AND by the server.

How to serialize this request? After a fixed 8-byte header, you could send

Remark 1: Why is the sample field defined to be a (signed) int32 originally? Remark 2: Note that a logical OR can still be achieved by sending multiple requests and then filtering out duplicate events on the client side. AND seems more useful.

Some of the offline file formats supported by FieldTrip have a timestamp field for events. We should think about including this in the FieldTrip buffer as well. The type should be 64-bit double precision, with the IEEE standard NaN (not a number) indicating that this field is not filled. Otherwise, the content can be application specific, or for example contain the system time of a specific machine at which an event happened, maybe encoded as UNIX time (seconds and fractions thereof since the epoch / 1970). Currently the timing is based on samples alone, which makes it hard to fuse data from different sources.

Instead of reporting the number of samples inside the datadef_t field in the response to a GET_DAT (as in V1), the server should rather send the index of the first and last sample that is being transmitted. The number of samples can be inferred from that, but the opposite direction is not possible. This is useful because the ring buffer will loose old samples after some time, where the V1 GET_DAT without a (begsample;endsample) tuple makes it impossible for the client to infer which samples it got. Another possible extension is to include a filter condition to grab samples within certain limits (as opposed to grabbing the specific interval in V1).

(This would also be more consistent with the new GET_EVT call, where you get event indices as well).

Many acquisition systems (e.g. Biosemi EEG + CTF MEG) provide continuously sampled trigger channels. Markers contained in these channels will be turned into discrete events by the acquisition tools. However, while originally the continuous samples and the markers in the trigger channels come from the same data block (e.g., from the USB driver or a shared memory segment), it is not well defined whether the tools should first write the events or the samples to the FieldTrip buffer. This has consequences for the analysis side: For example, if a client polls for new data, and does so just in the time between the acquisition tool writing the samples and the events to the buffer, this client will only notice the new samples, might process them, and then move on to the next block. The events will only be visibile in the next polling operation. The reverse situation is also conceivable, but here the problem is less severe because the client “knows” that if there are new events present, the corresponding samples must at some point follow.

Things to think about: Should it be mandatory to always write events first, then samples? Or should we add a request for writing samples and events at the same time (atomically with respect to access from other clients)? Should this request replace the old PUT_DAT call?

We should think about setting the desired size of the ring buffers for both events and samples in this request, since this is where the memory gets allocated (also depends on the number of channels), and as such also where possible errors will be reported.

On the other hand, often the acquisition client will care less about the length of the buffer than the processing client, and in this case it would be better to have a separate call (from the processing client) to define the desired ring buffer size, before the acquisition client puts the header.

It would also be useful to include something like a running ID in the header information, which automatically gets incremented for every PUT_HDR call. Currently, it's possible that clients A writes a header, client B reads it, client A writes another header, client B goes on without noticing. Re-writing the header often occurs when acquisition devices are restarted (e.g., acq2ft will re-write the header when the user enables head-localization in Acq).

Chunks have been introduced recently in version 1 to enable transmitting meta information such as channel names, calibration values, or MR protocol data. The current (V1) way of handling those is simple: they are always written and retrieved together with the header. In fact, the buffer server does not care at all about the contents and types of the chunks. As a downside, GET_HDR requests get slow if large chunks are present (e.g., CTF system).

Robert has proposed adding seperate requests for adding and reading chunks. Issues to think about:

• Clients need a way to determine which chunks are present - this is easy.
• Uniqueness of chunks: Do we allow multiple chunks of the same type (e.g., for representing different general purpose key/value pairs)?
• Do we allow a chunk to change over time? If so, clients need a way to determine whether a certain chunk has been updated since it was last read.

See draft header chunks.

See separate page on draft compatability across versions.

Currently, Matlab users type 'buffer://somehost:port' for talking to a buffer, and thanks to some intelligence in the MEX file, this translates into either opening a new TCP connection, reusing a connection, or talking directly to a buffer embedded in the MEX file.

The C interface, however, is less consistent. For example, a TCP connection is opened using open_connection(hostname, port), a connection using the new UNIX sockets facility is opened using open_unix_connection(pathname), and for direct memory access, no such call is made. Robert proposed to provide a unified API also on the C level, that is, we would have one function that is called like

con = open_connection("localhost:1972");   // open a TCP connection
con = open_connection("/some/unix/path");  // open a UNIX/local domain connection
con = open_connection("<dma>");            // "open" a direct memory "connection"

and that returns a data structure with information about the type of connection instead of a simple integer (or socket identifier). This would naturally extend to future implementations (e.g., Windows pipes).

One small difficulty remains: Currently there is always only one buffer per process, which is identified by a socket number=0. If we ever want to support having multiple buffers in the same process, we need some way to distinguish between them, and do so by using a string (as above). However, the most natural way would be to encapsulate all variables of an embedded buffer in one C struct, and refer to that struct by a pointer. How should we turn this into a string in a sensible way? Have a static array of pointers and use an index into that array? Use the address of the pointer as a hex-string?

C does not know namespaces, so we should try to avoid too short names like “append” which might yield problems when linking to other libraries at the same time. ft_ or ftb_ looks like a reasonable choice.

Currently every C program using the FieldTrip buffer needs to do the same steps (fill request data structure, check response data structure, clean up) over and over again. At least some of this could be made easier. This will also make it less painful to migrate to newer protocol versions. C++ users can already use wrapper classes in FtBuffer.h for handling requests and responses, which provide automatic cleanup.