Decoding Speech from Single Trial MEG Signals Using Convolutional Neural Networks and Transfer Learning

Abstract

Decoding speech directly from the brain has the potential for the development of the next generation, more efficient brain computer interfaces (BCIs) to assist in the communication of patients with locked-in syndrome (fully paralyzed but aware). In this study, we have explored the spectral and temporal features of the magnetoencephalography (MEG) signals and trained those features with convolutional neural networks (CNN) for the classification of neural signals corresponding to phrases. Experimental results demonstrated the effectiveness of CNNs in decoding speech during perception, imagination, and production tasks. Furthermore, to overcome the long training time issue of CNNs, we leveraged principal component analysis (PCA) for spatial dimension reduction of MEG data and transfer learning for model initialization. Both PCA and transfer learning were found to be highly beneficial for faster model training. The best configuration (50 principal coefficients + transfer learning) led to more than 10 times faster training than the original setting while the speech decoding accuracy remained at a similarly high level.