Use of Machine Learning to Assess Cataract Surgery Skill Level With Tool Detection

Jessica Ruzicki; Matthew Holden; Stephanie Cheon; Tamas Ungi; Rylan Egan; Christine Law

doi:10.1016/j.xops.2022.100235

Use of Machine Learning to Assess Cataract Surgery Skill Level With Tool Detection

Ophthalmol Sci. 2022 Oct 27;3(1):100235. doi: 10.1016/j.xops.2022.100235. eCollection 2023 Mar.

Authors

Jessica Ruzicki¹, Matthew Holden², Stephanie Cheon³, Tamas Ungi⁴, Rylan Egan⁵, Christine Law¹

Affiliations

¹ Department of Ophthalmology, Kingston Health Sciences Centre, Queen's University, Kingston, Ontario, Canada.
² School of Computer Science, Carleton University, Ottawa, Ontario, Canada.
³ School of Medicine, Faculty of Health Sciences, Queen's University, Kingston, Ontario, Canada.
⁴ Laboratory for Percutaneous Surgery, School of Computing, Queen's University, Kingston, Ontario, Canada.
⁵ Office of Health Sciences Education, Faculty of Health Sciences, Queen's University, Kingston, Ontario, Canada.

Abstract

Purpose: To develop a method for objective analysis of the reproducible steps in routine cataract surgery.

Design: Prospective study; machine learning.

Participants: Deidentified faculty and trainee surgical videos.

Methods: Consecutive cataract surgeries performed by a faculty or trainee surgeon in an ophthalmology residency program over 6 months were collected and labeled according to degrees of difficulty. An existing image classification network, ResNet 152, was fine-tuned for tool detection in cataract surgery to allow for automatic identification of each unique surgical instrument. Individual microscope video frame windows were subsequently encoded as a vector. The relation between vector encodings and perceived skill using k-fold user-out cross-validation was examined. Algorithms were evaluated using area under the receiver operating characteristic curve (AUC) and the classification accuracy.

Main outcome measures: Accuracy of tool detection and skill assessment.

Results: In total, 391 consecutive cataract procedures with 209 routine cases were used. Our model achieved an AUC ranging from 0.933 to 0.998 for tool detection. For skill classification, AUC was 0.550 (95% confidence interval [CI], 0.547-0.553) with an accuracy of 54.3% (95% CI, 53.9%-54.7%) for a single snippet, AUC was 0.570 (0.565-0.575) with an accuracy of 57.8% (56.8%-58.7%) for a single surgery, and AUC was 0.692 (0.659-0.758) with an accuracy of 63.3% (56.8%-69.8%) for a single user given all their trials.

Conclusions: Our research shows that machine learning can accurately and independently identify distinct cataract surgery tools in videos, which is crucial for comparing the use of the tool in a step. However, it is more challenging for machine learning to accurately differentiate overall and specific step skill to assess the level of training or expertise.

Financial disclosures: The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Keywords: AUC, area under the receiver operating characteristic curve; Artificial intelligence; CI, confidence interval; Cataract surgery; Education.