Mixed Reality and Artificial Intelligence: A Holistic Approach to Multimodal Visualization and Extended Interaction in Knee Osteotomy

Andrea Moglia; Luca Marsilio; Matteo Rossi; Maria Pinelli; Emanuele Lettieri; Luca Mainardi; Alfonso Manzotti; Pietro Cerveri

doi:10.1109/JTEHM.2023.3335608

Mixed Reality and Artificial Intelligence: A Holistic Approach to Multimodal Visualization and Extended Interaction in Knee Osteotomy

IEEE J Transl Eng Health Med. 2023 Nov 21:12:279-290. doi: 10.1109/JTEHM.2023.3335608. eCollection 2024.

Authors

Andrea Moglia¹, Luca Marsilio¹, Matteo Rossi^{1

2}, Maria Pinelli³, Emanuele Lettieri³, Luca Mainardi¹, Alfonso Manzotti⁴, Pietro Cerveri^{1

2}

Affiliations

¹ Department of ElectronicsInformation and BioengineeringPolitecnico di Milano 20133 Milan Italy.
² Istituto Auxologico Italiano IRCCS 20149 Milan Italy.
³ Department of Management, Economics and Industrial EngineeringPolitecnico di Milano 20133 Milan Italy.
⁴ Hospital ASST FBF-Sacco 20157 Milan Italy.

Abstract

Objective: Recent advancements in augmented reality led to planning and navigation systems for orthopedic surgery. However little is known about mixed reality (MR) in orthopedics. Furthermore, artificial intelligence (AI) has the potential to boost the capabilities of MR by enabling automation and personalization. The purpose of this work is to assess Holoknee prototype, based on AI and MR for multimodal data visualization and surgical planning in knee osteotomy, developed to run on the HoloLens 2 headset.

Methods: Two preclinical test sessions were performed with 11 participants (eight surgeons, two residents, and one medical student) executing three times six tasks, corresponding to a number of holographic data interactions and preoperative planning steps. At the end of each session, participants answered a questionnaire on user perception and usability.

Results: During the second trial, the participants were faster in all tasks than in the first one, while in the third one, the time of execution decreased only for two tasks ("Patient selection" and "Scrolling through radiograph") with respect to the second attempt, but without statistically significant difference (respectively [Formula: see text] = 0.14 and [Formula: see text] = 0.13, [Formula: see text]). All subjects strongly agreed that MR can be used effectively for surgical training, whereas 10 (90.9%) strongly agreed that it can be used effectively for preoperative planning. Six (54.5%) agreed and two of them (18.2%) strongly agreed that it can be used effectively for intraoperative guidance.

Discussion/conclusion: In this work, we presented Holoknee, the first holistic application of AI and MR for surgical planning for knee osteotomy. It reported promising results on its potential translation to surgical training, preoperative planning, and surgical guidance. Clinical and Translational Impact Statement - Holoknee can be helpful to support surgeons in the preoperative planning of knee osteotomy. It has the potential to impact positively the training of the future generation of residents and aid surgeons in the intraoperative stage.

Keywords: Lower limb osteotomy; deep neural networks; image segmentation; surgical planning.

MeSH terms

Artificial Intelligence
Augmented Reality*
Humans
Knee Joint / diagnostic imaging
Osteotomy / methods
Surgery, Computer-Assisted* / methods

Grants and funding

This work was supported in part by the Regione Lombardia within the Framework POR-FESR 2014-2020 (REFINE Project) and in part by PNRR-PE through Future Artificial Intelligence Research (FAIR)–Italian Ministry of University and Research.