An Attention-Based Mechanism to Combine Images and Metadata in Deep Learning Models Applied to Skin Cancer Classification

André G. C. Pacheco, Renato A. Krohling

IEEE Journal of Biomedical and Health Informatics · 2021 · 237 citations · 45 references

Concepts

TL;DR

Computer‑aided skin cancer classifiers typically rely solely on lesion images, yet incorporating patient demographics—an element human experts use—has the potential to boost performance. This study aims to integrate image and metadata features within deep learning models for skin cancer classification. We introduce the Metadata Processing Block (MetaBlock), which leverages metadata to reinforce the most relevant image‑derived features throughout the network, and benchmark it against MetaNet and feature‑concatenation baselines. Across two datasets, MetaBlock consistently improves classification accuracy for all tested models and outperforms the other combination methods in 6 of 10 evaluated scenarios.

Abstract

Computer-aided skin cancer classification systems built with deep neural networks usually yield predictions based only on images of skin lesions. Despite presenting promising results, it is possible to achieve higher performance by taking into account patient demographics, which are important clues that human experts consider during skin lesion screening. In this article, we deal with the problem of combining images and metadata features using deep learning models applied to skin cancer classification. We propose the Metadata Processing Block (MetaBlock), a novel algorithm that uses metadata to support data classification by enhancing the most relevant features extracted from the images throughout the classification pipeline. We compared the proposed method with two other combination approaches: the MetaNet and one based on features concatenation. Results obtained for two different skin lesion datasets show that our method improves classification for all tested models and performs better than the other combination approaches in 6 out of 10 scenarios.

References

45