development:preproc

FieldTrip has a consistent set of low-level functions for reprocessing of EEG and MEG data, such as filtering, baseline correction and re-referencing. This allows other projects to re-use the implemented methods seperate from FieldTrip (e.g. for realtime analysis of EEG and MEG data) and perhaps also to contribute to FieldTrip.

The low-level functions are combined in the preproc toolbox, which is available for download here.

Note that the FieldTrip toolbox comes with a user-friendly preprocessing function, which reads data from disk and applies preprocessing to that data. Regular users mainly interested in analyzing their data should be using that function. The functions described here are for people that are developing their own code and/or contributing to the development of FieldTrip.

All of the low-level preprocessing functions require data to be represented as a 2D matrix (Nchans X Nsamples). The following functions are implemented

preproc_bandpassfilter - applies a band-pass filter to the data
preproc_bandstopfilter - applies a band-stop filter to the data
preproc_highpassfilter - applies a high-pass filter to the data
preproc_lowpassfilter - applies a low-pass filter to the data
preproc_medianfilter - applies a median filter, i.e. a jump-preserving smoothing kernel
preproc_dftfilter - applies a notch filter to the data to remove the 50Hz noise
preproc_baselinecorrect - performs a baseline correction
preproc_detrend - removes linear or higher order polynomial trends
preproc_derivative - computes the temporal Nth order derivative
preproc_hilbert - computes the Hilbert transpose of the data
preproc_rectify - rectifies the data, usefull for EMG
preproc_rereference - rereferences EEG data using the average over all channels or selected channels

For real-time use of the functions in this module it is relevant to know how much time each function takes. The execution time depends on the number of channels and on the number of samples in the data block. Assuming that multiple functions are called as part of a larger processing pipeline, the time of the functions that are called has to be summed up.

The benchmarking results presented in the table below were determined with fieldtrip-20090107 and Matlab 7.5 running on a MacPro with 2×2.66 GHz dual-core Intel Xeon processors. Execution time was determined for three representative data block sizes and for various parameters for each function.

The full code for the benchmarking (one script per preproc function and some helper functions) is included in the release version of FieldTrip. You can repeat it on your own workstation and for your own data characteristics.

function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; order = 2; type = 'but'; dir = 'onepass';	1.40 ms	2.02 ms	7.10 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; order = 4; type = 'but'; dir = 'onepass';	1.68 ms	2.35 ms	7.80 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; order = 8; type = 'but'; dir = 'onepass';	2.31 ms	3.04 ms	8.82 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; order = 16; type = 'fir'; dir = 'onepass';	1.20 ms	1.75 ms	6.52 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; order = 32; type = 'fir'; dir = 'onepass';	1.34 ms	1.84 ms	7.05 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; order = 64; type = 'fir'; dir = 'onepass';	1.28 ms	1.97 ms	7.63 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_bandstopfilter; Fsample = 1000; Fbp = [ 45 55 ]; order = 2; type = 'but'; dir = 'onepass';	1.46 ms	2.11 ms	7.23 ms
preproc_bandstopfilter; Fsample = 1000; Fbp = [ 45 55 ]; order = 4; type = 'but'; dir = 'onepass';	1.78 ms	2.46 ms	7.97 ms
preproc_bandstopfilter; Fsample = 1000; Fbp = [ 45 55 ]; order = 8; type = 'but'; dir = 'onepass';	2.59 ms	3.31 ms	9.51 ms
preproc_bandstopfilter; Fsample = 1000; Fbp = [ 45 55 ]; order = 16; type = 'fir'; dir = 'onepass';	1.17 ms	1.73 ms	6.47 ms
preproc_bandstopfilter; Fsample = 1000; Fbp = [ 45 55 ]; order = 32; type = 'fir'; dir = 'onepass';	1.21 ms	1.79 ms	6.58 ms
preproc_bandstopfilter; Fsample = 1000; Fbp = [ 45 55 ]; order = 64; type = 'fir'; dir = 'onepass';	1.26 ms	1.94 ms	7.53 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_baselinecorrect; begsample = []; endsample = [];	0.11 ms	0.40 ms	0.93 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_derivative; order = 1; padding = 'end';	0.05 ms	0.18 ms	0.83 ms
preproc_derivative; order = 1; padding = 2;	0.05 ms	0.12 ms	0.51 ms
preproc_derivative; order = 1; padding = 3;	0.05 ms	0.12 ms	0.50 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_detrend; begsample = []; endsample = []; order = 1;	0.12 ms	0.47 ms	1.94 ms
preproc_detrend; begsample = []; endsample = []; order = 2;	0.12 ms	0.56 ms	2.18 ms
preproc_detrend; begsample = []; endsample = []; order = 3;	0.14 ms	0.74 ms	2.43 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_dftfilter; Fsample = 1000; Fline = 50;	0.13 ms	0.64 ms	2.42 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_highpassfilter; Fsample = 1000; Fh = 1; order = 2; type = 'but'; dir = 'onepass';	1.28 ms	1.92 ms	6.79 ms
preproc_highpassfilter; Fsample = 1000; Fh = 1; order = 4; type = 'but'; dir = 'onepass';	2.08 ms	2.23 ms	7.18 ms
preproc_highpassfilter; Fsample = 1000; Fh = 1; order = 8; type = 'but'; dir = 'onepass';	2.00 ms	2.68 ms	8.00 ms
preproc_highpassfilter; Fsample = 1000; Fh = 1; order = 16; type = 'fir'; dir = 'onepass';	1.09 ms	1.67 ms	6.03 ms
preproc_highpassfilter; Fsample = 1000; Fh = 1; order = 32; type = 'fir'; dir = 'onepass';	1.12 ms	1.84 ms	6.58 ms
preproc_highpassfilter; Fsample = 1000; Fh = 1; order = 64; type = 'fir'; dir = 'onepass';	1.16 ms	1.86 ms	7.56 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_hilbert; option = 'complex';	0.26 ms	1.32 ms	8.47 ms
preproc_hilbert; option = 'abs';	0.38 ms	1.61 ms	10.42 ms
preproc_hilbert; option = 'angle';	5.17 ms	25.37 ms	44.77 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_lowpassfilter; Fsample = 1000; Fl = 30; order = 2; type = 'but'; dir = 'onepass';	1.29 ms	1.89 ms	6.91 ms
preproc_lowpassfilter; Fsample = 1000; Fl = 30; order = 4; type = 'but'; dir = 'onepass';	1.47 ms	2.17 ms	7.59 ms
preproc_lowpassfilter; Fsample = 1000; Fl = 30; order = 8; type = 'but'; dir = 'onepass';	1.92 ms	2.69 ms	8.15 ms
preproc_lowpassfilter; Fsample = 1000; Fl = 30; order = 16; type = 'fir'; dir = 'onepass';	1.16 ms	1.65 ms	6.25 ms
preproc_lowpassfilter; Fsample = 1000; Fl = 30; order = 32; type = 'fir'; dir = 'onepass';	1.09 ms	1.73 ms	6.60 ms
preproc_lowpassfilter; Fsample = 1000; Fl = 30; order = 64; type = 'fir'; dir = 'onepass';	1.15 ms	1.90 ms	7.64 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; N = 2; type = 'but'; dir = 'onepass';	1.40 ms	2.03 ms	7.34 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; N = 4; type = 'but'; dir = 'onepass';	1.91 ms	2.42 ms	7.79 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; N = 8; type = 'but'; dir = 'onepass';	2.35 ms	3.06 ms	8.87 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; N = 16; type = 'fir'; dir = 'onepass';	1.33 ms	1.75 ms	6.34 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; N = 32; type = 'fir'; dir = 'onepass';	1.32 ms	1.83 ms	6.61 ms
preproc_bandpassfilter; Fsample = 1000; Fbp = [ 10 100 ]; N = 64; type = 'fir'; dir = 'onepass';	1.27 ms	1.91 ms	7.47 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_rectify;	0.03 ms	0.11 ms	0.55 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_rereference; refchan = 1;	0.07 ms	0.16 ms	0.55 ms
preproc_rereference; refchan = 'all';	0.10 ms	0.25 ms	1.07 ms
function name and algorithm details	8ch x 100smp	8ch x 500smp	64ch x 500smp
preproc_standardize; begsample = []; endsample = [];	0.25 ms	0.53 ms	1.73 ms

2010-05-20 12:12  marvger

	* ft_preproc_baselinecorrect.m: used bsxfun to increase efficiency

2010-05-19 08:58  jansch

	* ft_preproc_resample.m: added documentation with regard to the anti-aliasing
	  applied by default if using the resample-method

2010-04-21 17:56  roboos

	* ft_preproc_bandpassfilter.m, ft_preproc_bandstopfilter.m, ft_preproc_baselinecorrect.m,
	  ft_preproc_denoise.m, ft_preproc_derivative.m, ft_preproc_detrend.m,
	  ft_preproc_dftfilter.m, ft_preproc_highpassfilter.m, ft_preproc_hilbert.m,
	  ft_preproc_lowpassfilter.m, ft_preproc_medianfilter.m, ft_preproc_rectify.m,
	  ft_preproc_rereference.m, ft_preproc_resample.m, ft_preproc_standardize.m: added full GPL
	  copyrights and svn ID tag to preproc

2010-04-16 09:22  roboos

	* private: added compat/signal/filtfilt function from octave, put it into preproc/private as
	  file-external where it will become the default (instead of the signal processing toolbox
	  version)

2010-04-09 13:47  timeng

	* ft_preproc_bandpassfilter.m, ft_preproc_bandstopfilter.m, ft_preproc_baselinecorrect.m,
	  ft_preproc_denoise.m, ft_preproc_derivative.m, ft_preproc_detrend.m,
	  ft_preproc_dftfilter.m, ft_preproc_highpassfilter.m, ft_preproc_hilbert.m,
	  ft_preproc_lowpassfilter.m, ft_preproc_medianfilter.m, ft_preproc_rectify.m,
	  ft_preproc_rereference.m, ft_preproc_resample.m, ft_preproc_standardize.m: ft prefix added
	  to help sections and in the codes only for functions called within the preproc and forward
	  module. Compat now still necessary called from main FieldTrip functions

2010-04-08 11:03  roboos

	* compat/preproc_bandpassfilter.m, compat/preproc_bandstopfilter.m,
	  compat/preproc_baselinecorrect.m, compat/preproc_denoise.m, compat/preproc_derivative.m,
	  compat/preproc_detrend.m, compat/preproc_dftfilter.m, compat/preproc_highpassfilter.m,
	  compat/preproc_hilbert.m, compat/preproc_lowpassfilter.m, compat/preproc_medianfilter.m,
	  compat/preproc_rectify.m, compat/preproc_rereference.m, compat/preproc_resample.m,
	  compat/preproc_slidingavg.m, compat/preproc_standardize.m, ft_preproc_bandpassfilter.m,
	  ft_preproc_bandstopfilter.m, ft_preproc_baselinecorrect.m, ft_preproc_denoise.m,
	  ft_preproc_derivative.m, ft_preproc_detrend.m, ft_preproc_dftfilter.m,
	  ft_preproc_highpassfilter.m, ft_preproc_hilbert.m, ft_preproc_lowpassfilter.m,
	  ft_preproc_medianfilter.m, ft_preproc_rectify.m, ft_preproc_rereference.m,
	  ft_preproc_standardize.m: removed the unix executable flag from all *.m files (thanks to
	  Guillaume)
	  see http://fieldtrip.fcdonders.nl/development/svn#keeping_the_svn_repository_tidy for
	  details

2010-04-07 13:52  roboos

	* compat, compat/preproc_bandpassfilter.m, compat/preproc_bandstopfilter.m,
	  compat/preproc_baselinecorrect.m, compat/preproc_denoise.m, compat/preproc_derivative.m,
	  compat/preproc_detrend.m, compat/preproc_dftfilter.m, compat/preproc_highpassfilter.m,
	  compat/preproc_hilbert.m, compat/preproc_lowpassfilter.m, compat/preproc_medianfilter.m,
	  compat/preproc_rectify.m, compat/preproc_rereference.m, compat/preproc_resample.m,
	  compat/preproc_slidingavg.m, compat/preproc_standardize.m, ft_preproc_bandpassfilter.m,
	  ft_preproc_bandstopfilter.m, ft_preproc_baselinecorrect.m, ft_preproc_denoise.m,
	  ft_preproc_derivative.m, ft_preproc_detrend.m, ft_preproc_dftfilter.m,
	  ft_preproc_highpassfilter.m, ft_preproc_hilbert.m, ft_preproc_lowpassfilter.m,
	  ft_preproc_medianfilter.m, ft_preproc_rectify.m, ft_preproc_rereference.m,
	  ft_preproc_resample.m, ft_preproc_standardize.m, preproc_bandpassfilter.m,
	  preproc_bandstopfilter.m, preproc_baselinecorrect.m, preproc_denoise.m,
	  preproc_derivative.m, preproc_detrend.m, preproc_dftfilter.m, preproc_highpassfilter.m,
	  preproc_hilbert.m, preproc_lowpassfilter.m, preproc_medianfilter.m, preproc_rectify.m,
	  preproc_rereference.m, preproc_resample.m, preproc_standardize.m: renamed all preproc_xxx
	  functions into ft_preproc_xxx
	  added a preproc/compat directory with backward compatibility wrappers

2010-04-07 09:27  roboos

	* preproc_bandpassfilter.m: only change in whitespace

2010-04-06 21:01  roboos

	* preproc_resample.m: initial implementation, taken from spikedownsample/myresample

2010-02-17 15:58  roboos

	* private: added provate directory with replacement function for butterworth filter

2010-02-17 15:54  roboos

	* benchmark: Created a branches and a tags directory at the highest level of the
	  repository, moved preproc/benchmark from the trunk to
	  branches/benchmarking. In the future there could also be a
	  branches/testing directory.

2010-01-01 21:09  joagro

	* preproc_hilbert.m: fixed two bugs: 1) angle was computed on non-normalised complex values;
	  2) unwrapping was performed along wrong dimension; Joachim

2009-12-16 19:38  roboos

	* preproc_standardize.m: remember the state for updating the mean and variance estimate

2009-12-04 10:00  roboos

	* COPYING, README, benchmark/benchmark.m, benchmark/benchmark_all.m,
	  benchmark/benchmark_bandpassfilter.m, benchmark/benchmark_bandstopfilter.m,
	  benchmark/benchmark_baselinecorrect.m, benchmark/benchmark_derivative.m,
	  benchmark/benchmark_detrend.m, benchmark/benchmark_dftfilter.m,
	  benchmark/benchmark_highpassfilter.m, benchmark/benchmark_hilbert.m,
	  benchmark/benchmark_lowpassfilter.m, benchmark/benchmark_medianfilter.m,
	  benchmark/benchmark_rectify.m, benchmark/benchmark_rereference.m,
	  benchmark/benchmark_standardize.m, preproc_bandpassfilter.m, preproc_bandstopfilter.m,
	  preproc_baselinecorrect.m, preproc_denoise.m, preproc_derivative.m, preproc_detrend.m,
	  preproc_dftfilter.m, preproc_highpassfilter.m, preproc_hilbert.m, preproc_lowpassfilter.m,
	  preproc_medianfilter.m, preproc_rectify.m, preproc_rereference.m, preproc_standardize.m:
	  set the executable property to OFF on many files that should not be executable (m-files,
	  mex-files, readme, etc.)

2009-11-30 11:28  roboos

	* preproc_rereference.m: changed all occurences of tabs into 4 spaces

2009-11-30 08:53  roboos

	* benchmark/benchmark.m, benchmark/benchmark_all.m, benchmark/benchmark_bandpassfilter.m,
	  benchmark/benchmark_bandstopfilter.m, benchmark/benchmark_baselinecorrect.m,
	  benchmark/benchmark_derivative.m, benchmark/benchmark_detrend.m,
	  benchmark/benchmark_dftfilter.m, benchmark/benchmark_highpassfilter.m,
	  benchmark/benchmark_hilbert.m, benchmark/benchmark_lowpassfilter.m,
	  benchmark/benchmark_medianfilter.m, benchmark/benchmark_rectify.m,
	  benchmark/benchmark_rereference.m, benchmark/benchmark_standardize.m,
	  preproc_bandpassfilter.m, preproc_bandstopfilter.m, preproc_baselinecorrect.m,
	  preproc_denoise.m, preproc_derivative.m, preproc_detrend.m, preproc_dftfilter.m,
	  preproc_highpassfilter.m, preproc_hilbert.m, preproc_lowpassfilter.m,
	  preproc_medianfilter.m, preproc_rectify.m, preproc_rereference.m, preproc_standardize.m:
	  changed svn property for keywords Log and Rev

2009-11-30 08:49  roboos

	* benchmark/benchmark.m, benchmark/benchmark_all.m, benchmark/benchmark_bandpassfilter.m,
	  benchmark/benchmark_bandstopfilter.m, benchmark/benchmark_baselinecorrect.m,
	  benchmark/benchmark_derivative.m, benchmark/benchmark_detrend.m,
	  benchmark/benchmark_dftfilter.m, benchmark/benchmark_highpassfilter.m,
	  benchmark/benchmark_hilbert.m, benchmark/benchmark_lowpassfilter.m,
	  benchmark/benchmark_medianfilter.m, benchmark/benchmark_rectify.m,
	  benchmark/benchmark_rereference.m, benchmark/benchmark_standardize.m,
	  preproc_bandpassfilter.m, preproc_bandstopfilter.m, preproc_baselinecorrect.m,
	  preproc_denoise.m, preproc_derivative.m, preproc_detrend.m, preproc_dftfilter.m,
	  preproc_highpassfilter.m, preproc_hilbert.m, preproc_lowpassfilter.m,
	  preproc_medianfilter.m, preproc_rectify.m, preproc_rereference.m, preproc_standardize.m:
	  changed svn property eol-style to native to ensure consistent end-of-line behaviour on
	  windows and linux

2009-10-27 16:01  roboos

	* benchmark/benchmark.m, preproc_bandpassfilter.m, preproc_bandstopfilter.m,
	  preproc_baselinecorrect.m, preproc_denoise.m, preproc_derivative.m, preproc_detrend.m,
	  preproc_dftfilter.m, preproc_highpassfilter.m, preproc_hilbert.m, preproc_lowpassfilter.m,
	  preproc_medianfilter.m, preproc_rectify.m, preproc_rereference.m, preproc_standardize.m:
	  removed the old cvs log comments from each of the fieldtrip source code files
	  subversion (a.k.a. svn) does not make use of the special $Log$ keyword, which means that
	  the log messages do not reflect the latest changes
	  you can use 'svn log <filename.m>' or 'svn -v log | less' to get the same detailled
	  information as used to be in the cvs log

2009-10-21 10:25  roboos

	* ., COPYING, README, benchmark, benchmark/benchmark.m, benchmark/benchmark_all.m,
	  benchmark/benchmark_bandpassfilter.m, benchmark/benchmark_bandstopfilter.m,
	  benchmark/benchmark_baselinecorrect.m, benchmark/benchmark_derivative.m,
	  benchmark/benchmark_detrend.m, benchmark/benchmark_dftfilter.m,
	  benchmark/benchmark_highpassfilter.m, benchmark/benchmark_hilbert.m,
	  benchmark/benchmark_lowpassfilter.m, benchmark/benchmark_medianfilter.m,
	  benchmark/benchmark_rectify.m, benchmark/benchmark_rereference.m,
	  benchmark/benchmark_standardize.m, preproc_bandpassfilter.m, preproc_bandstopfilter.m,
	  preproc_baselinecorrect.m, preproc_denoise.m, preproc_derivative.m, preproc_detrend.m,
	  preproc_dftfilter.m, preproc_highpassfilter.m, preproc_hilbert.m, preproc_lowpassfilter.m,
	  preproc_medianfilter.m, preproc_rectify.m, preproc_rereference.m, preproc_standardize.m:
	  initial import

Functions for preprocessing of EEG/MEG data

Module layout

Interface to the preprocessing functions

Benchmarking

Recent changes to the code