Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)

TLDR

The 2018 ISIC challenge, held at MICCAI in Granada, presented a dataset of over 12,500 dermoscopic images across three tasks—segmentation, attribute detection, and disease classification—attracting 900 registrants and 399 submissions, and introduced novel evaluation protocols assessing both segmentation accuracy and algorithm generalization. Top segmentation algorithms still fail on more than 10 % of images on average, and algorithms with comparable test performance can differ markedly in generalization, underscoring the need for regulatory scrutiny and establishing a new benchmark for future healthcare challenges.

Abstract

This work summarizes the results of the largest skin image analysis challenge in the world, hosted by the International Skin Imaging Collaboration (ISIC), a global partnership that has organized the world's largest public repository of dermoscopic images of skin. The challenge was hosted in 2018 at the Medical Image Computing and Computer Assisted Intervention (MICCAI) conference in Granada, Spain. The dataset included over 12,500 images across 3 tasks. 900 users registered for data download, 115 submitted to the lesion segmentation task, 25 submitted to the lesion attribute detection task, and 159 submitted to the disease classification task. Novel evaluation protocols were established, including a new test for segmentation algorithm performance, and a test for algorithm ability to generalize. Results show that top segmentation algorithms still fail on over 10% of images on average, and algorithms with equal performance on test data can have different abilities to generalize. This is an important consideration for agencies regulating the growing set of machine learning tools in the healthcare domain, and sets a new standard for future public challenges in healthcare.