<  Back to the Polytechnique Montréal portal

Multimodal autoencoder predicts fNIRS resting state from EEG signals

Parikshat Sirpal, Rafat Damseh, Ke Peng, Dang Khoa Nguyen, Frédéric Lesage

Article (2021)

Open Acess document in PolyPublie and at official publisher
Open Access to the full text of this document
Published Version
Terms of Use: Creative Commons Attribution
Download (2MB)
Show abstract
Hide abstract


In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model's fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals.

Uncontrolled Keywords

EEG-fNIRS, Functional brain imaging, Deep neural networks, Epilepsy, Resting state, Functional connectivity, Neurovascular coupling

Subjects: 2500 Electrical and electronic engineering > 2500 Electrical and electronic engineering
9000 Health sciences > 9000 Health sciences
Department: Department of Electrical Engineering
Research Center: Other
Grant number: 239876, 87183
PolyPublie URL: https://publications.polymtl.ca/9262/
Journal Title: Neuroinformatics (vol. 2021)
Publisher: Springer Nature
DOI: 10.1007/s12021-021-09538-3
Official URL: https://doi.org/10.1007/s12021-021-09538-3
Date Deposited: 24 Mar 2022 11:40
Last Modified: 11 Nov 2022 14:03
Cite in APA 7: Sirpal, P., Damseh, R., Peng, K., Nguyen, D. K., & Lesage, F. (2021). Multimodal autoencoder predicts fNIRS resting state from EEG signals. Neuroinformatics, 2021, 22 pages. https://doi.org/10.1007/s12021-021-09538-3


Total downloads

Downloads per month in the last year

Origin of downloads


Repository Staff Only

View Item View Item