Real-world post-deployment performance of a novel machine learning-based digital health technology for skin lesion assessment and suggestions for post-market surveillance

Lucy Thomas; Chris Hyde; Dan Mullarkey; Jack Greenhalgh; Dilraj Kalsi; Justin Ko

doi:10.3389/fmed.2023.1264846

Real-world post-deployment performance of a novel machine learning-based digital health technology for skin lesion assessment and suggestions for post-market surveillance

Front Med (Lausanne). 2023 Oct 31:10:1264846. doi: 10.3389/fmed.2023.1264846. eCollection 2023.

Authors

Lucy Thomas¹, Chris Hyde², Dan Mullarkey³, Jack Greenhalgh³, Dilraj Kalsi³, Justin Ko⁴

Affiliations

¹ Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom.
² Exeter Test Group, Department of Health and Community Sciences, University of Exeter Medical School, Exeter, United Kingdom.
³ Skin Analytics Ltd., London, United Kingdom.
⁴ Department of Dermatology, Stanford Medicine, Stanford, CA, United States.

Abstract

Introduction: Deep Ensemble for Recognition of Malignancy (DERM) is an artificial intelligence as a medical device (AIaMD) tool for skin lesion assessment.

Methods: We report prospective real-world performance from its deployment within skin cancer pathways at two National Health Service hospitals (UK) between July 2021 and October 2022.

Results: A total of 14,500 cases were seen, including patients 18-100 years old with Fitzpatrick skin types I-VI represented. Based on 8,571 lesions assessed by DERM with confirmed outcomes, versions A and B demonstrated very high sensitivity for detecting melanoma (95.0-100.0%) or malignancy (96.0-100.0%). Benign lesion specificity was 40.7-49.4% (DERM-vA) and 70.1-73.4% (DERM-vB). DERM identified 15.0-31.0% of cases as eligible for discharge.

Discussion: We show DERM performance in-line with sensitivity targets and pre-marketing authorisation research, and it reduced the caseload for hospital specialists in two pathways. Based on our experience we offer suggestions on key elements of post-market surveillance for AIaMDs.

Keywords: AI as a medical device; AI for skin cancer; DERM; Skin Analytics; artificial intelligence; deep ensemble for the recognition of malignancy; skin cancer.