Utilizing Computer Vision Technology for Automated Surgical Video Analysis to Improve Surgical Proficiency

Introduction: Operative video data can be predictive of surgical expertise, complication rates and patient outcomes.¹ More importantly, studies have shown that surgeons reviewing video recordings of their own surgeries can improve patient outcomes and surgery time.² As video capture technology improve and data storage less burdensome, there has been a surge in the amount of intraoperative video recordings. Realizing the full potential of these recordings, however, has significant barriers. Extensive time and labor commitment to transfer and organize the data, locate and edit critical sections, and development of systematic analyses are several significant ones. We proposed that many of these barriers can be overcome with automation, which will lead to saved surgeon time, wider utilization, and eventual standardized performance tracking.

Methods: We piloted our computer vision technologies from the context of robotic surgeries in phantom and ex-vivo animal models, and as a start are adopting them to endoscopic neurosurgery. The functions include surgical scene classification, surgical tool tracking, and camera motion estimation. Furthermore, we are developing a web-based platform aimed to streamline uploading, storing, cataloging, and sharing surgical videos, while concurrently capturing and tracking usage data.

Results: In terms of automated surgical scene classification, our algorithm was able to detect when the camera was outside of the body with 98.3% accuracy. It achieved accuracy of 99.627% at identifying the presence of the drill in the scene, which can be used to delineate start and end of bony exposure. Overall, our surgical tool tracking demonstrated a key point detection error of only <1.5%. In addition, we are able to track camera motion. Our algorithm had a highly accurate camera motion estimation, with a mean error of <0.05%. The precise tracking of surgical tools and camera position is the beginning step for surgical tool motion analysis and potentially surgical performance analysis.

Conclusion: Our preliminary results suggest it is feasible to automate surgical video analyses. Further studies are needed to investigate the potential for wider adaptation and to usage of the data in, performance tracking, enhancing surgical skill education, and individualized technical development.

Fig. 1 Confusion matrices for detecting different features of the surgical scene: when camera is outside of the body, when smoke is present, when the camera is flushed, and when the drill was being used. There is a high degree with accuracy with a mean error of <0.05%.

No conflict of interest has been declared by the author(s).

Publication History

Article published online:
05 February 2024