CTF / VSM MedTech

CTF/VSM MedTech is the company that produces one of the three most widely used whole-head MEG systems and one of the few systems that has been used for real-time MEG analysis for the purpose of BCI (see Mellinger et al., NeuroImage 2007). There are 151 channel and 275 channel systems. The systems provide real-time access to the MEG data through shared memory.

The acquisition software runs on a linux computer. If prior to starting the acquisition software shared memory with the appropriate details is initialized, the acquisition software will write a copy of the data to that shared memory. The shared memory is split over 600 packets, each packet holding 28160 samples. With approximately 350 channels (MEG, EEG and status/trigger channels) in the typical MEG system, that amounts to approximately 80 samples per packet. So in total there are 600*80=48000 data samples stored in a shared memory buffer.

In FieldTrip it is possible to use the fileio module to read from shared memory. Because the shared memory also has to be freed to ensure that the Acq software continues writing to it, a daemon process has to be running in the background. The daemon process is called “AcqBuffer” and is started from the command line. It constantly loops over the 600 packets in shared memory, and if there are less than 20 packets free, it memcpy's the “setup” packet (containing the name of the res4 file that has the full header details in it) to the next packet, thereby freeing the packet previously containing the setup. This procedure ensures that the content of the setup packet can always be read, even while it is being copied.

Besides maintaining the copy of the setup packet and ensuring that there are always some packets free to receive the new data from Acq, the AcqBuffer application also performs trigger detection on indicated channels and stores these triggers in a convenient representation. This speeds up the trigger detection in the read_event function in MATLAB considerably.

In MATLAB the full header details of the MEG data set are determined by first reading the packet containing the setup information (i.e. the name of the res4 file) and subsequently by reading the details from that res4 file using the standard reading function. The number of samples available in the dataset is updated to reflect the amount of data present in the buffer.

After you start

>> AcqBuffer

on the linux command line, and in another terminal

>> Acq

You can access the data in MATLAB like this

hdr    = read_header(filename), this returns a structure with the header information
event  = read_event(filename), this returns a structure with the event information, (i.e. the triggers) 
dat    = read_data(filename, ...), this returns a 2-D or 3-D array with the data

where filename should be a string containing 'ctf_shm://', i.e. similar as a Universal Resource Identifier. In case you want to use the header information from another res4 file, you can specify the filename as 'ctf_shm://<example.res4>', i.e. including the full path and filename of the res4 header file.

The (inter-)operation of the three involved software components, all running on the same machine can be summarised with the following diagram.

The rt_ctfproxy function (part of the realtime module in FieldTrip) reads the MEG data from shared memory and writes to a FieldTrip buffer. The FieldTrip buffer is a multi-threaded and network transparent buffer that allows data to be streamed to it, while at the same time allowing another MATLAB session on the same or another computer to read data from the buffer for analysis.

Subsequently in another Matlab session you can read from the FieldTrip buffer using the read_header, read_data and read_event functions by specifying 'buffer://hostname:port' as the filename to the reading functions.

It seems that the default linux/redhat configuratino of the shared memory does not allow a sufficiently large memory block to be allocated. To change the setting in the operating system, you should do (as root user):

echo 1000000000 > /proc/sys/kernel/shmmax

There appears to be two ways of (re)defining the amount of shared memory in your system (tips thanks to Dave Glowacki of SSEC):

• add this line to your /etc/rc.d/rc.local file:

    echo shared_memory_size > /proc/sys/kernel/shmmax

(where shared_memory_size is the amount of shared memory you want to declare in bytes) and reboot.

• a more permanent solution would be to change the value of SHMMAX in /usr/src/linux/include/asm/shmparam.h and rebuild your kernel.

There is a problem in the CTF acquisition software that sometimes causes the shared memory interface to fail. The diagnosis of the problem is that the Acq software runs and writes the data to the shared-memory buffer, where it is detected by AcqBuffer, and that after a certain random amount of time (around one minute) the AcqBuffer stops. The problem seems to be caused by a memory buffer overrun. The shared memory consists of 600 packets, each defined as

typedef struct {
 ACQ_MessageType message_type; % this is 4 bytes long
 int messageId;
 int sampleNumber;
 int numSamples;
 int numChannels;
 int data[28160];
} ACQ_MessagePacketType;

So in total each packet is 5*4+28160*4 bytes long, and there are 600 of those in shared memory. If the numChannels*numSamples of the previous block is slightly larger than 28160, it means that Acq is trying to write more data points into the “data” section of that packet than fits in, causing the data to flow over into the next packet. The first couple of integers in the next packet (indicating the Type and other details) are therefore messed up, and Aqc thinks that that packet is already filled. Then it stops writing to shared memory altogether.

I have tested this idea with a specially tweaked version of my AcqBuffer shared memory “maintenance” program and indeed see this happen for a data block that has 91*310=28210 samples in it, which is 50 more than the 28160 that would fit in. The next block is therefore corrupt.

Now understanding the problem, we can start thinking about a solution. Somehow in the CTF code there is an incorrect estimate of the number of samples that fits into a block. Probably we can play with the channel number to circumvent the problem. Given a certain channel number, Acq will have to determine how many samples fit into a single block. I suspect the bug in the Acq code to be something like sampleNumber = round(28160/numChannels) where it should be sampleNumber = floor(28160/numChannels) i.e. rounding off to the bottom. To solve this, we can look at

for chancount=1:500
success(chancount) = ((round(28160/chancount).*chancount)<=28160);
end

>> find(success)
ans =
    1     2     4     5     6     8    10    11    13    14    15    16    17    18    19    20    22    23    24    25    26    29   31    32    33    34    36    37    38    39    40    42    44    46    47    50    51    53    54    55    57    59    60    62    64    65    67    68    69    70    72    75    78    79    80    82    83    84    85    86    88    89    91    92    95    96    97    98    99   102   103   105   107   109   110   112   113   114   124   125   126   128   129   132   134   136   138   140   142   145   148   151   152   153   157   158   159   160   176   177   178   179   180   184   185   186   190   191   194   195   198   201   202   204   205   207   208   210   211   213   216   218   220   221   223   225   227   230   232   234   236   238   240   242   244   246   247   249   251   253   255   256   258   260   262   263   265   267   268   270   273   275   276   278   281   284   286   287   289   290   292   293   295   296   298   299   302   305   306   308   309   312   315   316   319   320   322   323   326   327   330   331   334   335   338   339   342   343   346   347   350   351   352   355   356   359   360   361   364   365   369   370   373   374   375   378   379   380   384   385   389   390   391   394   395   396   400   401   402   406   407   408   412   413   414   418   419   420   424   425   426   430   431   432   433   437   438   439   440   444   445   446   451   452   453   454   458   459   460   461   466   467   468   469   474   475   476   477   482   483   484   485   490   491   492   493   494   499   500

Update: This calculation seems not to be 100% correct. For example, 359 channels do NOT work. 360 seems to be okay.

This is the table (up to 500 channels) that lists the number of channels for which it will work. The test that I performed happened to be with 310 channels, which is not in the list. Since we can always add a few (unused) EEG channels to the acquisition, we can work around the bug in the CF Acq software.

In order to overcome the aforementioned problems with non-functioning number of channels, a new tool was developed that combines the old AcqBuffer and rt_ctfproxy in one stand-alone application. This program is called acq2ft and resides in the realtime/ctf directory. It operates by grabbing one packet (setup or data) at a time out of the shared memory, and more or less directly transferring it into a FieldTrip buffer that can be embedded in acq2ft itself, or on any other location on the network.

In contrast to AcqBuffer, acq2ft also decodes header information from the .res4 file pointed to by the setup collection packet, and thus knows by itself which channels contain triggers instead of relying on a 3rd application or Matlab script to write that information back to shared memory. On top of that, acq2ft overallocates the shared memory by 1000 samples and is prepared to operate successfully even if the proprietary Acq application writes too much data into any slot. Instead of preparing the shared memory segment for access by a third application, it streams the data and events (decoded from the trigger channels) to a local or remote FieldTrip buffer, as depicted by the following diagram:

Similarly to AcqBuffer, you need to start acq2ft before starting Acq so that the shared memory interface can be detected and connected to by the latter. You have the option of spawning a FieldTrip buffer server directly within acq2ft, using

acq2ft

to use the default port 1972, or using (note the minus)

acq2ft - port

to spawn at server at the given port. You can also tell acq2ft to stream the data to a buffer provided by another application (possibly on another machine), using

acq2ft hostname port

Please note that for this to work, you will need to start up the remote buffer before acq2ft. Once acq2ft is happily up and running, you can start Acq from a different command line or using existing GUI tools.

Since streaming the data to a remote FieldTrip buffer might incur a delay due to network traffic, acq2ft employs an internal ring buffer for up to 10 data packets and a simple (socket pair) mechanism to synchronize between copying data packets from the shared memory to the internal ring buffer, and streaming the data from the ring buffer to the FieldTrip buffer. This means that small delays should not interfere with the time-critical operation of clearing up slots in the shared memory segment, as long as the average throughput is high enough.