NeuroImage · 2023 · 90 citations · 29 references
MedicineComputer ScienceBiomedical ImagingDementiaMachine LearningAlzheimer DiseaseMultimodal ImagingMultimodal PhenotypingNeurodegenerative DiseasesNeuroimaging In DementiaMild Cognitive Impairment DiagnosisMild Cognitive Impairment ProgressionMulti-modal Data AnalysisMultimodal Classification FusionMultimodal Feature FusionMultimodal Feature RepresentationMultimodal Machine LearningTransformer-based Classification
Accurate medical classification requires a large number of multi-modal data, and in many cases, different feature types. Previous studies have shown promising results when using multi-modal data, outperforming single-modality models when classifying diseases such as Alzheimer's Disease (AD). However, those models are usually not flexible enough to handle missing modalities. Currently, the most common workaround is discarding samples with missing modalities which leads to considerable data under-utilisation. Adding to the fact that labelled medical images are already scarce, the performance of data-driven methods like deep learning can be severely hampered. Therefore, a multi-modal method that can handle missing data in various clinical settings is highly desirable. In this paper, we present Multi-Modal Mixing Transformer (3MT), a disease classification transformer that not only leverages multi-modal data but also handles missing data scenarios. In this work, we test 3MT for AD and Cognitively normal (CN) classification and mild cognitive impairment (MCI) conversion prediction to progressive MCI (pMCI) or stable MCI (sMCI) using clinical and neuroimaging data. The model uses a novel Cascaded Modality Transformers architecture with cross-attention to incorporate multi-modal information for more informed predictions. We propose a novel modality dropout mechanism to ensure an unprecedented level of modality independence and robustness to handle missing data scenarios. The result is a versatile network that enables the mixing of arbitrary numbers of modalities with different feature types and also ensures full data utilization in missing data scenarios. The model is trained and evaluated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset with the state-of-the-art performance and further evaluated with The Australian Imaging Biomarker & Lifestyle Flagship Study of Ageing (AIBL) dataset with missing data.
29
Clinical diagnosis of Alzheimer's disease
Guy M. McKhann, David A. Drachman, Marshall Folstein et al. · Neurology · 1984 · 28K citations · Full text
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
Guy M. McKhann, David S. Knopman, Howard Chertkow et al. · Alzheimer s & Dementia · 2011 · 18.8K citations · Full text