M²LC-Net: A multi-modal multi-disease long-tailed classification network for real clinical scenes

Abstract

Leveraging deep learning-based techniques to classify diseases has attracted extensive research interest in recent years. Nevertheless, most of the current studies only consider single-modal medical images, and the number of ophthalmic diseases that can be classified is relatively small. Moreover, imbalanced data distribution of different ophthalmic diseases is not taken into consideration, which limits the application of deep learning techniques in realistic clinical scenes. In this paper, we propose a Multimodal Multi-disease Long-tailed Classification Network (M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> LC-Net) in response to the challenges mentioned above. M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> LC-Net leverages ResNet18-CBAM to extract features from fundus images and Optical Coherence Tomography (OCT) images, respectively, and conduct feature fusion to classify 11 common ophthalmic diseases. Moreover, Class Activation Mapping (CAM) is employed to visualize each mode to improve interpretability of M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> LC-Net. We conduct comprehensive experiments on realistic dataset collected from a Grade III Level A ophthalmology hospital in China, including 34,396 images of 11 disease labels. Experimental results demonstrate effectiveness of our proposed model M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> LC-Net. Compared with the state-of-the-art, various performance metrics have been improved significantly. Specifically, Cohen's kappa coefficient κ has been improved by 3.21%, which is a remarkable improvement.