Combining information from Electroencephalography (EEG) and Functional Magnetic Resonance Imaging (fMRI) has been a topic of increased interest recently. The main advantage of the EEG is its high temporal resolution, in the scale of milliseconds, while the main advantage of fMRI is the detection of functional activity with good spatial resolution. The advantages of each modality seem to complement each other, providing better insight in the neuronal activity of the brain. The main goal of combining information from both modalities is to increase the spatial and the temporal localization of the underlying neuronal activity captured by each modality. This paper presents a novel technique based on the combination of these two modalities (EEG, fMRI) that allow a better representation and understanding of brain activities in time. EEG is modeled as a sequence of topographies, based on the notion of microstates. Hidden Markov Models (HMMs) were used to model the temporal evolution of the topography of the average Event Related Potential (ERP). For each model the Fisher score of the sequence is calculated by taking the gradient of the trained model parameters. The Fisher score describes how this sequence deviates from the learned HMM. Canonical Partial Least Squares (CPLS) were used to decompose the two datasets and fuse the EEG and fMRI features. In order to test the effectiveness of this method, the results of this methodology were compared with the results of CPLS using the average ERP signal of a single channel. The presented methodology was able to derive components that co-vary between EEG and fMRI and present significant differences between the two tasks.
Keywords: EEG; Hidden markov models; classification; fMRI; fusion; partial least squares.