Seven-Point Checklist and Skin Lesion Classification Using Multitask Multimodal Neural Nets

TLDR

The study proposes a multitask deep convolutional neural network trained on multimodal data to classify the 7‑point melanoma checklist and diagnose skin lesions. The network uses multitask loss functions across clinical, dermoscopic, and metadata inputs, enabling robustness to missing modalities, and outputs checklist scores, diagnosis labels, feature vectors for retrieval, and localized discriminant regions. On 1,011 lesion cases, the model achieved comprehensive performance across all 7‑point criteria and diagnosis, and the dataset is publicly released at http://derm.cs.sfu.ca.

Abstract

We propose a multitask deep convolutional neural network, trained on multimodal data (clinical and dermoscopic images, and patient metadata), to classify the 7-point melanoma checklist criteria and perform skin lesion diagnosis. Our neural network is trained using several multitask loss functions, where each loss considers different combinations of the input modalities, which allows our model to be robust to missing data at inference time. Our final model classifies the 7-point checklist and skin condition diagnosis, produces multimodal feature vectors suitable for image retrieval, and localizes clinically discriminant regions. We benchmark our approach using 1011 lesion cases, and report comprehensive results over all 7-point criteria and diagnosis. We also make our dataset (images and metadata) publicly available online at http://derm.cs.sfu.ca.