<  Back to the Polytechnique Montréal portal

Multimodal autoencoder predicts fNIRS resting state from EEG signals

Parikshat Sirpal, Rafat Damseh, Ke Peng, Dang Khoa Nguyen and Frédéric Lesage

Article (2021)

Published Version
Terms of Use: Creative Commons Attribution .
Download (7MB)
Cite this document: Sirpal, P., Damseh, R., Peng, K., Nguyen, D. K. & Lesage, F. (2021). Multimodal autoencoder predicts fNIRS resting state from EEG signals. Neuroinformatics, 2021. doi:10.1007/s12021-021-09538-3
Show abstract Hide abstract


In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model’s fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals.

Uncontrolled Keywords

EEG-fNIRS, Functional brain imaging, Deep neural networks, Epilepsy, Resting state, Functional connectivity, Neurovascular coupling

Open Access document in PolyPublie
Subjects: 2500 Génie électrique et électronique > 2500 Génie électrique et électronique
9000 Sciences de la santé > 9000 Sciences de la santé
Department: Département de génie électrique
Research Center: Autre
Grant number: 239876, 87183
Date Deposited: 24 Mar 2022 11:40
Last Modified: 25 Mar 2022 01:20
PolyPublie URL: https://publications.polymtl.ca/9262/
Document issued by the official publisher
Journal Title: Neuroinformatics (vol. 2021)
Publisher: Springer Nature
Official URL: https://doi.org/10.1007/s12021-021-09538-3


Total downloads

Downloads per month in the last year

Origin of downloads


Repository Staff Only