A new innovative imaging modality: “extended reality” in endoscopic ultrasonography

Endoscopic ultrasonography (EUS) is essential for management of pancreaticobiliary diseases; it involves operating the endoscope, interpreting the EUS images, comparing them with other images, and considering needle biopsy or surgery. In addition, the anatomy of each patient must be considered; however, attaining an accurate understanding is a major challenge for trainees.

Extended reality (XR) technology is increasingly being used in medical settings [1] [2] [3] [4]. It allows users to grasp and view a three-dimensional (3D) image from various angles, which enables a sensory understanding in conjunction with endoscopic imaging. We believe that XR support can significantly enhance EUS comprehension and training. We here report a case in which EUS was performed with XR navigation support.

A 67-year-old man underwent EUS for suspicion of a serous cystic neoplasm ([Fig. 1]). 3D models were constructed through contrast-enhanced computed tomography (CT) and magnetic resonance cholangiopancreatography (MRCP) using SYNAPSE VINCENT (Fujifilm Medical Co., Ltd., Tokyo, Japan). The models were uploaded into the Holoeyes MD system (Holoeyes Inc., Tokyo, Japan) and visualized as 3D holograms in real space through the see-through “HoloLens 2” goggles (Microsoft Corp., Redmond, Washington, USA), allowing simultaneous viewing of the ultrasound monitor and 3D hologram ([Fig. 2]; [Video 1]). Changing the angle of the 3D image to match that of the EUS image made understanding the depictions of the organs easier. Furthermore, trainees and experts could view the same 3D hologram simultaneously in a virtual session, which proved effective for teaching and reviewing ([Fig. 3]). By adding a hologram of the endoscope, understanding its movement using the goggles became possible ([Fig. 4]).

Fig. 1 Imaging of a suspected serous cystic neoplasm in the tail of the pancreas on: a magnetic resonance cholangiopancreatography; b contrast-enhanced computed tomography.

Fig. 2 Simultaneous display of the endoscopic ultrasonography image and a 3D hologram through the see-through goggles.

Fig. 3 Use of the 3D hologram for review and educational purposes.

Fig. 4 With the immersive goggles, various simulations can be experienced using a 3D hologram.

Extended reality navigation during endoscopic ultrasonography.Video 1

We performed EUS safely with the XR navigation system, allowing for an intuitive understanding of the 3D structure of the organs. This system was also effective for teaching and review purposes. The new method supported by XR could be a breakthrough in this field.

Endoscopy_UCTN_Code_TTT_1AS_2AD

Conflict of Interest

The authors declare that they have no conflict of interest.

• References

• 1 Tokunaga T, Sugimoto M, Saito Y. et al. Intraoperative holographic image-guided surgery in a transanal approach for rectal cancer. Langenbecks Arch Surg 2022; 407: 2579-2584
• 2 Saito Y, Sugimoto M, Imura S. et al. Intraoperative 3D hologram support with mixed reality techniques in liver surgery. Ann Surg 2020; 271: E4-E7
• 3 Ryu S, Kitagawa T, Goto K. et al. Intraoperative holographic guidance using virtual reality and mixed reality technology during laparoscopic colorectal cancer surgery. Anticancer Res 2022; 42: 4849-4856
• 4 Ishikawa-Kakiya Y, Maruyama H, Tanoue K. et al. Using a new, innovative “mixed reality” technology in endoscopic retrograde cholangiopancreatography. Endoscopy 2024; 56: E901-E902

Publikationsverlauf

Artikel online veröffentlicht:
31. Januar 2025