Multivariate Machine Learning Methods for Fusing Multimodal Functional Neuroimaging Data

Sven Dahne, Felix Bieszmann, Wojciech Samek, Stefan Haufe, Dominique Goltz, Christopher Gundlach, Arno Villringer, Siamac Fazli, Klaus Robert Muller

Research output: Contribution to journalArticle

43 Citations (Scopus)

Abstract

Multimodal data are ubiquitous in engineering, communications, robotics, computer vision, or more generally speaking in industry and the sciences. All disciplines have developed their respective sets of analytic tools to fuse the information that is available in all measured modalities. In this paper, we provide a review of classical as well as recent machine learning methods (specifically factor models) for fusing information from functional neuroimaging techniques such as: LFP, EEG, MEG, fNIRS, and fMRI. Early and late fusion scenarios are distinguished, and appropriate factor models for the respective scenarios are presented along with example applications from selected multimodal neuroimaging studies. Further emphasis is given to the interpretability of the resulting model parameters, in particular by highlighting how factor models relate to physical models needed for source localization. The methods we discuss allow for the extraction of information from neural data, which ultimately contributes to 1) better neuroscientific understanding; 2) enhance diagnostic performance; and 3) discover neural signals of interest that correlate maximally with a given cognitive paradigm. While we clearly study the multimodal functional neuroimaging challenge, the discussed machine learning techniques have a wide applicability, i.e., in general data fusion, and may thus be informative to the general interested reader.

Original languageEnglish
Article number7182735
Pages (from-to)1507-1530
Number of pages24
JournalProceedings of the IEEE
Volume103
Issue number9
DOIs
Publication statusPublished - Sep 1 2015
Externally publishedYes

    Fingerprint

Keywords

  • data fusion
  • EEG
  • fMRI
  • fNIRS
  • Machine learning
  • MEG
  • multimodal neuroimaging
  • review

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Cite this

Dahne, S., Bieszmann, F., Samek, W., Haufe, S., Goltz, D., Gundlach, C., Villringer, A., Fazli, S., & Muller, K. R. (2015). Multivariate Machine Learning Methods for Fusing Multimodal Functional Neuroimaging Data. Proceedings of the IEEE, 103(9), 1507-1530. [7182735]. https://doi.org/10.1109/JPROC.2015.2425807