Real-world post-deployment performance of a novel machine learning-based digital health technology for skin lesion assessment and suggestions for post-market surveillance

Abstract

Introduction: Deep Ensemble for Recognition of Malignancy (DERM) is an artificial intelligence as a medical device (AIaMD) tool for skin lesion assessment. Methods: We report prospective real-world performance from its deployment within skin cancer pathways at two National Health Service hospitals (UK) between July 2021 and October 2022. Results: A total of 14,500 cases were seen, including patients 18-100 years old with Fitzpatrick skin types I-VI represented. Based on 8,571 lesions assessed by DERM with confirmed outcomes, versions A and B demonstrated very high sensitivity for detecting melanoma (95.0-100.0%) or malignancy (96.0-100.0%). Benign lesion specificity was 40.7-49.4% (DERM-vA) and 70.1-73.4% (DERM-vB). DERM identified 15.0-31.0% of cases as eligible for discharge. Discussion: We show DERM performance in-line with sensitivity targets and pre-marketing authorisation research, and it reduced the caseload for hospital specialists in two pathways. Based on our experience we offer suggestions on key elements of post-market surveillance for AIaMDs.