Are YouTube Videos a Reliable Training Method for Safe Laparoscopic Cholecystectomy? A Simulated Decision-Making Exercise to Assess the Critical View of Safety

Abstract

Background The present study assesses the educational value of laparoscopic cholecystectomy videos on YouTube regarding the correct application of the critical view of safety (CVS), and evaluates… surgical trainees' perceptions of the CVS criteria in a simulated, operative decision-making exercise.

Methods YouTube was systematically searched for laparoscopic cholecystectomy videos, explicitly reporting a satisfactory CVS. The top 30 most popular videos, by number of views, were identified and scored on the 6-point scale by three experienced consultants. After watching a training module on CVS rationale and criteria, 10 trainees, blinded to the consultants' assessment, were instructed to view the videos, score each criterion and answer the binary question “Would you divide the cystic structures?” by “yes” or “no.”

Results An inadequate CVS was found in 30% of the included videos. No statistical association was noted between number of views, likes, or dislikes with successful CVS rates. Inter-observer agreement between consultants and trainees ranged from minimal to moderate (k = 0.07–0.60). Discrepancy between trainees' CVS scores and their simulated decision to proceed to division of the cystic structures was found in 15% of assessments, with intra-observer agreement ranging from minimal to excellent (k = 0.27–1.0). For the CVS requirements, inter-observer agreement was minimal for the dissection of the cystic plate (k = 0.26) and triangle clearance (k = 0.39) and moderate for the identification of two and only two structures (k = 0.42).

Conclusion The CVS is central to the culture of safety in laparoscopic cholecystectomy. Surgical videos are a useful training tool as simulated, operative decision-making exercises. However, public video platforms should be used judiciously, since their content is not peer-reviewed or quality-controlled.

Keywords

surgical training - laparoscopic cholecystectomy - bile duct - critical view of safety (CVS).

Technological advances and web-based applications have truly revolutionized the field of surgical education. In an effort to enhance the learning experience, traditional tools, like the time-honored atlas of surgical operations, have given way to contemporary solutions, like simulators and virtual reality.[1] Undoubtedly, the audiovisual interaction offered by modern technology cannot be surpassed by books and amphitheater lectures.

Surgical videos are an example of the benefits of multimedia in the learning process and have been established as a modern educational module. Ninety percent of surgeons and trainees watch videos to prepare for surgical cases and studying, usually on public platforms like YouTube (www.youtube.com).[2] [3] [4] However, the true value of public video libraries, as source of medical information or surgical technique, has been questioned, due to concerns for low quality, non—peer-reviewed material, or nonadherence to guidelines.[5] [6] [7] [8] [9] [10]

To date, only a small number of studies have evaluated the quality of laparoscopic cholecystectomy videos on popular platforms (YouTube, Vimeo, etc.), with regard to the CVS principles.[1] [11] [12] [13] [14] They have found alarmingly low rates of correct CVS application, as low as 10 to 28%. However, selection of videos in these studies was based on the search term “laparoscopic cholecystectomy” alone, without any reference to the CVS itself.

The aim of our study was to assess the educational value of laparoscopic cholecystectomy videos on YouTube, that explicitly reported successful CVS achievement. These videos were used for a simulated decision-making exercise, aimed at evaluating surgical trainees' perceptions of the CVS criteria.

Materials and Methods

Introduced in 1995, the CVS is a method of target identification used in laparoscopic cholecystectomy, to avoid bile duct and vascular injuries, due to misidentification of anatomical structures in the hepatocystic triangle.[15] Secure identification of the cystic structures depends on three requirements: (1) clearance of the hepatocystic triangle of all fibrofatty tissue, (2) identification of two and only two tubular structures entering the gallbladder wall, and (3) dissection of the lower third of the gallbladder off the cystic plate. Each criterion is awarded 0 to 2 points, for a maximum of six points.[16] Scores of 5 or 6 are considered a satisfactory CVS, allowing safe ligation of the cystic duct and artery. Scores of 0 to 4 require further dissection or a bail-out technique.[17]

Using the keywords “critical view of safety” and “laparoscopic cholecystectomy,” the YouTube platform was searched on May 21^st, 2020, to identify operative videos, which explicitly reported a satisfactory CVS, either in the title, the description, or embedded on the video itself. Eligible were videos of live surgical procedures with adequate play time to enable assessment of the dissection of the hepatocystic triangle and demonstration of the CVS criteria. Animations, lectures, conference presentations, and educational material provided by scientific societies were excluded. No restrictions on age, gender, ethnicity, or experience of the primary surgeon were imposed. Both videos of elective and acute cholecystitis cases were acceptable. For each video, the following data were extracted: URL, number of views, likes, dislikes and comments, gender, and country of the primary surgeon.

Three consultant surgeons, trained on and exclusively performing the CVS approach, jointly scored the videos, using the six-point scale. Videos were judged as a whole, therefore, points were given if the relevant CVS requirement was present either in the anterior or the posterior view. Each video was subsequently characterized as either “Pass” (5–6) or “Fail” (0–4). A subanalysis was performed, to assess whether number of views, likes and dislikes were associated with rates of successful CVS.

The video URLs were then given to 10 trainees [five junior (years 1–3) and five senior (years 4–5) residents], blinded to the consultants' assessment. After watching a training module on CVS rationale and criteria with operative examples, they were instructed to view the videos, until the cystic structures were clipped and divided, to score each CVS criterion and answer the binary question “Would you divide the cystic structures?” by “Yes” or “No.”

Ιnter-observer agreement was calculated for each resident as percent agreement and Randolph's kappa, by comparison with consultants' score.[18] Intra-observer agreement among trainees was also calculated, comparing their CVS evaluation and decision to divide or not the cystic structures. To further determine which CVS criterion was the most difficult to identify, the gradings for each criterion were compared among residents by Randolph's kappa. Statistical analysis was performed on SPSS ver 20.0 (IBM Corp.). Values of kappa coefficient were interpreted as follows: no agreement (0–0.20), minimal agreement (0.21–0.40), weak agreement (0.41–0.60), moderate agreement (0.61–0.80), strong agreement (0.81–0.90), and almost perfect agreement (>0.90).

Results

The 30 highest ranking videos, by number of views, were included in the study ([Table 1]). Median number of views was 2,313 (range 331–58,541). All surgeons were males.

Table 1 Characteristics of included YouTube videos

No	Views	Likes	Dislikes	Comments	Surgeon gender	Country	Setting
1	58,541	368	46	64	Male	India	Elective
2	33,183	90	19	8	Male	United States	Elective
3	28,374	78	4	3	Male	Egypt	Elective
4	28,311	141	8	3	Male	India	Elective
5	10,171	46	6	15	Male	Turkey	Acute
6	9,657	35	2	5	Male	Turkey	Elective
7	7,800	54	6	0	Male	Egypt	Elective
8	6,049	29	7	4	Male	Italy	Elective
9	4,424	24	2	2	Male	Italy	Acute
10	3,257	6	0	3	Male	India	Elective
11	3,181	24	2	4	Male	Turkey	Acute
12	2,951	8	2	0	Male	UK	Elective
13	2,409	23	5	5	Male	India	Elective
14	2,408	18	1	1	Male	United States	Elective
15	2,374	51	1	4	Male	Argentina	Elective
16	2,251	28	0	3	Male	Argentina	Elective
17	1,914	11	2	0	Male	Turkey	Elective
18	1,515	17	0	5	Male	Italy	Acute
19	1,324	4	0	0	Male	India	Elective
20	1,080	1	2	0	Male	India	Elective
21	980	26	0	11	Male	India	Elective
22	925	32	2	6	Male	Italy	Elective
23	888	5	0	0	Male	Greece	Elective
24	885	32	1	4	Male	India	Elective
25	864	12	0	1	Male	United States	Elective
26	805	24	0	0	Male	Argentina	Elective
27	604	5	0	2	Male	Turkey	Acute
28	516	5	0	1	Male	Turkey	Elective
29	344	2	1	0	Male	India	Acute
30	331	2	0	0	Male	United States	Elective

Twenty-one videos (70%) were judged by the consultants as having properly obtained the CVS (scores of 5–6), whereas 9 (30%) were deemed unsatisfactory (scores of 0–4). Out of 24 elective and six acute cholecystitis cases, the CVS was not obtained in 5 (20.8%) and 4 (66.7%), respectively. No statistical association was observed between number of views, likes or dislikes with completion rates of CVS.

[Table 2] shows the trainees' evaluation. Overall “Pass” ratings ranged between 30 and 76.7%, while decision to proceed with division of the cystic structures ranged between 53.3 and 83.3% (discrepancy in 45/300 assessments). The inter-observer agreement between consultants and residents is shown in [Table 2] and [Fig. 1]. Percent agreement ranged between 53 and 80% (mean 69.7 ± 9.2%), whereas Randolph's kappa between 0.07 and 0.60 (mean 0.39 ± 0.18). Intra-observer agreement between the trainees' CVS scoring and the decision to divide the structures ranged between 63.3 and 100% (as percent agreement) and 0.27 to 1 (as Randolph's kappa). ([Fig. 2]). Level of trainee surgical experience was not associated with higher levels of inter- and intra-observer agreement. For the three CVS requirements, inter-observer agreement was minimal for dissection of the cystic plate (k = 0.26) and triangle clearance (k = 0.39) and weak for the identification of 2 structures (k = 0.42).

Table 2 Inter- and intraobserver agreement between consultants and trainees

	Experience	Pass (5–6) n (%)	Fail (0–4) n (%)	% Agreement	Inter-obsever agreement (kappa)	Would you clip? yes (%)	% Agreement	Intra-observer agreement (kappa)
Consultants		21 (70)	9 (30)
Trainee no 1	Senior	10 (33.3)	20 (66.7)	63.3%	0.27	21 (70)	63.3%	0.27
Trainee no 2	Senior	15 (50)	15 (50)	80%	0.60	16 (53.3)	96.7%	0.93
Trainee no 3	Senior	19 (63.3)	11 (36.7)	60%	0.20	19 (63.3)	93.3%	0.87
Trainee no 4	Senior	18 (60)	12 (40)	70%	0.40	18 (60)	100%	1
Trainee no 5	Senior	17 (56.7)	13 (43.3)	80%	0.60	25 (83.3)	73.3%	0.47
Trainee no 6	Junior	16 (53.3)	14 (46.7)	76.7%	0.53	19 (63.3)	90%	0.80
Trainee no 7	Junior	18 (60)	12 (40)	76.7%	0.53	18 (60)	100%	1
Trainee no 8	Junior	23 (76.7)	7 (23.3)	73.3%	0.47	23 (76.7)	93.3%	0.87
Trainee no 9	Junior	18 (60)	12 (40)	63.3%	0.27	25 (83.3)	76.7%	0.53
Trainee no 10	Junior	9 (30)	21 (70)	53.3%	0.07	20 (66.7)	63.3%	0.27

Fig. 1 Summary of trainees' assessment of the top-30 YouTube CVS videos [pass = adequate CVS (5–6), fail = inadequate CVS (0–4); as judged by consultants]. CVS, critical view of safety.

Fig. 2 Summary of trainees' responses to the binary question “Would you divide the cystic structures?” [pass = adequate CVS (5–6), fail = inadequate CVS (0–4); as judged by consultants]. CVS, critical view of safety.

Discussion

Previous studies assessing laparoscopic cholecystectomy videos on internet platforms found disappointing rates of proper CVS application. Lee et al analyzed 73 YouTube videos, grading 15% as “good,” 55% as “moderate” and 30% as “poor.”[11] However, demonstration of the CVS requirements was not reported by the 6-point scale, but rather on a 0 to 3 scale, and was incorporated into the overall scoring. Deal et al reviewed 40 representative videos on YouTube, Vimeo and the SAGES library, of variable technical performance.[13] Faculty expert ratings showed that only 12.5% achieved CVS scores of ≥5, while 85% scored ≤3. Moreover, an analysis of 139 YouTube videos by the same team, using crowd-sourcing, found no statistical correlation between number of views, likes, dislikes or subscribers and the completion of CVS.[12] Rodriguez et al analyzed the top ten listed YouTube videos and found only one to show a satisfactory CVS (≥5), while Chavira et al reviewed a total of 77 YouTube, WebSurg and GIBLIB videos, showing successful CVS rates in 27.7, 44.4, and 40%, respectively.[1] [14]

The aforementioned studies used the search term “laparoscopic cholecystectomy” during the selection process. By combining the keywords “critical view of safety” AND “laparoscopic cholecystectomy,” our analysis included only those videos that explicitly reported the successful application of the CVS. Of the top-30 most popular videos by number of views, 70% were evaluated by consultants as having conclusively achieved a CVS score of 5 or 6. Expectedly, this percentage was higher compared with similar studies of unselected cases. However, our results showed that a significant proportion of surgeons (30%), although claiming a satisfactory CVS, still misunderstood the core concept in practice.

In addition, similar to previous studies, the number of views, likes, and dislikes was not associated with successful CVS rates.[1] [12] Indeed, the significant difference in views between the first and last video (58,541 vs. 331) shows that viewers rarely scroll beyond the first few results.[14] Even more importantly, neither likes nor dislikes reflect the actual quality of the content. Public video libraries should therefore be used very judiciously, since the uploaded content is usually not peer-reviewed. Dedicated surgical websites and operative videos produced by academic institutions or official surgical societies are of higher educational value and should be preferred as training material.

The simulated decision-making exercise revealed certain interesting results. Between trainees and consultants, inter-observer agreement ranged from minimal to moderate. Generally, trainees tended to give lower marks overall, compared with consultants. To some extent, this fact may be attributed to a more cautious evaluation of the CVS criteria by surgeons in the beginnings of their learning curve. On the other hand, higher level of surgical training was not associated with higher inter-observer agreement rates, as might have been expected.

Yet, even more contradictory was the discordance between CVS scores and decision to divide the cystic structures. Despite the explanatory training module prior to the exercise, in 15% of cases the trainees would indeed proceed to ligation, even though their awarded CVS score was <5. This misconception of the CVS rationale is a hazardous gap in surgical training and could ultimately reflect an unsafe practice.

While the learning curve for mastering the CVS has yet to be determined, by analysis and comparison of operative notes and videos we do know that even experienced surgeons may lack full understanding of the three steps that constitute a proper CVS.[19] [20] Given the burden of BDI on the health care system and patients' long-term quality of life, education of surgeons toward the correct application of the CVS cannot be overemphasized.[19] [21] Tutorials with structured curriculum are necessary to highlight the rationale behind the CVS requirements and promote a culture of safety in laparoscopic cholecystectomy. Either in the form of lecture or video, they have been shown to increase rates of successful CVS and improve confidence among trainees.[21] [22] [23] [24] [25] [26]

Furthermore, we found considerable variation in the evaluation of each CVS criterion, with least agreement for adequate dissection of the cystic plate, similar to Mascagni et al.[27] Mobilization of the lower third of the gallbladder off the liver bed is essential, to secure that the purported cystic structures do not reenter the hepatic parenchyma at a higher level. However, there is not one single reason why surgeons fail to obtain a proper CVS. Nakazato et al found that the most common cause for an incomplete CVS was the inadequate cystic plate dissection, Carr et al found the inadequate clearance of the hepatocystic triangle, while Nijssen et al found the inability to recognize two and only two structures.[20] [25] [26] Equal emphasis on all three requirements is therefore necessary.

Inconsistency and subjective interpretation of the CVS criteria hide a dangerous trap, that could lead to vasculobiliary injuries. A conceptual framework, developed by expert academic surgeons, defined the essential competencies required to establish the proper CVS.[28] This framework includes cognitive elements and potential errors, related to situational awareness, decision-making and action-oriented subtasks, and may serve as the basis for surgical training, assessment tools, and quality-control metrics.

Our study was limited by the small number of participants, all of them trainees at a single surgical department, as well as small number of videos (n = 30). We also narrowed the selection of surgical videos to YouTube and chose not to include specialized, online surgical libraries. The videos were included solely on the basis of popularity and were not assessed for their technical quality or surgical competency, thus better reflecting real-world situations. Nevertheless, they were evaluated by experienced consultants using the recommended six-point CVS scale. Finally, our training module was similar in concept to the video tutorial by Deal et al, but has not been validated as an educational tool.[24]

Conclusion

Promotion of a culture of safety should be the very core of laparoscopic cholecystectomy training. And the CVS concept is central to this culture. Surgical videos are a useful educational tool, as simulated decision-making exercises. However, public video platforms should be used judiciously by trainees, since their content is not peer-reviewed or quality-controlled.

Conflict of Interest

None declared.

• References

• 1 Chavira AM, Rivas JF, Molina APRF. et al. The educational quality of the critical view of safety in videos on youtube® versus specialized platforms: which is better? Critical view of safety in virtual resources. Surg Endosc 2021; (e-pub ahead of print)
• 2 Rapp AK, Healy MG, Charlton ME, Keith JN, Rosenbaum ME, Kapadia MR. YouTube is the most frequently used educational video source for surgical preparation. J Surg Educ 2016; 73 (06) 1072-1076
• 3 Celentano V, Smart N, Cahill RA. et al. Use of laparoscopic videos amongst surgical trainees in the United Kingdom. Surgeon 2019; 17 (06) 334-339
• 4 Mota P, Carvalho N, Carvalho-Dias E, João Costa M, Correia-Pinto J, Lima E. Video-based surgical learning: improving trainee education and preparation for surgery. J Surg Educ 2018; 75 (03) 828-835
• 5 Madathil KC, Rivera-Rodriguez AJ, Greenstein JS, Gramopadhye AK. Healthcare information on YouTube: a systematic review. Health Informatics J 2015; 21 (03) 173-194
• 6 Karic B, Moino V, Nolin A, Andrews A, Brisson P. Evaluation of surgical educational videos available for third year medical students. Med Educ Online 2020; 25 (01) 1714197
• 7 Lee JS, Seo HS, Hong TH. YouTube as a source of patient information on gallstone disease. World J Gastroenterol 2014; 20 (14) 4066-4070
• 8 de'Angelis N, Gavriilidis P, Martínez-Pérez A. et al. Educational value of surgical videos on YouTube: quality assessment of laparoscopic appendectomy videos by senior surgeons vs. novice trainees. World J Emerg Surg 2019; 14: 22
• 9 Kanlioz M, Ekici U. Reliability and educational features of YouTube videos about hernia operations performed using laparoscopic TEP method. Surg Laparosc Endosc Percutan Tech 2020; 30 (01) 74-78
• 10 Frongia G, Mehrabi A, Fonouni H, Rennert H, Golriz M, Günther P. YouTube as a potential training resource for laparoscopic fundoplication. J Surg Educ 2016; 73 (06) 1066-1071
• 11 Lee JS, Seo HS, Hong TH. YouTube as a potential training method for laparoscopic cholecystectomy. Ann Surg Treat Res 2015; 89 (02) 92-97
• 12 Deal SB, Alseidi AA. Concerns of quality and safety in public domain surgical education videos: an assessment of the critical view of safety in frequently used laparoscopic cholecystectomy videos. J Am Coll Surg 2017; 225 (06) 725-730
• 13 Deal SB, Stefanidis D, Telem D. et al. Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy. Surg Endosc 2017; 31 (12) 5094-5100
• 14 Rodriguez HA, Young MT, Jackson HT, Oelschlager BK, Wright AS. Viewer discretion advised: is YouTube a friend or foe in surgical education?. Surg Endosc 2018; 32 (04) 1724-1728
• 15 Strasberg SM, Hertl M, Soper NJ. An analysis of the problem of biliary injury during laparoscopic cholecystectomy. J Am Coll Surg 1995; 180 (01) 101-125
• 16 Sanford DE, Strasberg SM. A simple effective method for generation of a permanent record of the critical view of safety during laparoscopic cholecystectomy by intraoperative “doublet” photography. J Am Coll Surg 2014; 218 (02) 170-178
• 17 Manatakis DK, Papageorgiou D, Antonopoulou M-I. et al. Ten-year audit of safe bail-out alternatives to the critical view of safety in laparoscopic cholecystectomy. World J Surg 2019; 43 (11) 2728-2733
• 18 Randolph JJ. Free-Marginal Multirater Kappa (multirater K): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa. Paper presented at: Joensuu Learning and Instruction Symposium, October 14–15. 2005 , Joensuu, Finland
• 19 Stefanidis D, Chintalapudi N, Anderson-Montoya B, Oommen B, Tobben D, Pimentel M. How often do surgeons obtain the critical view of safety during laparoscopic cholecystectomy?. Surg Endosc 2017; 31 (01) 142-146
• 20 Nijssen MA, Schreinemakers JM, Meyer Z, van der Schelling GP, Crolla RM, Rijken AM. Complications after laparoscopic cholecystectomy: a video evaluation study of whether the critical view of safety was reached. World J Surg 2015; 39 (07) 1798-1803
• 21 Niemann AC, Matusko N, Sandhu G, Varban OA. Assessing the effect of the critical view of safety criteria on simulated operative decision-making: a pilot study. Surg Endosc 2019; 33 (03) 911-916
• 22 Nijssen MA, Schreinemakers JM, van der Schelling GP, Crolla RM, Rijken AM. Improving critical view of safety in laparoscopic cholecystectomy by teaching interventions. J Surg Educ 2016; 73 (03) 442-447
• 23 Chen CB, Palazzo F, Doane SM. et al. Increasing resident utilization and recognition of the critical view of safety during laparoscopic cholecystectomy: a pilot study from an academic medical center. Surg Endosc 2017; 31 (04) 1627-1635
• 24 Deal SB, Stefanidis D, Brunt LM, Alseidi A. Development of a multimedia tutorial to educate how to assess the critical view of safety in laparoscopic cholecystectomy using expert review and crowd-sourcing. Am J Surg 2017; 213 (05) 988-990
• 25 Carr BD, Matusko N, Sandhu G, Varban OA. Cut or do not cut? Assessing perceptions of safety during laparoscopic cholecystectomy using surgical videos. J Surg Educ 2018; 75 (06) 1583-1588
• 26 Nakazato T, Su B, Novak S, Deal SB, Kuchta K, Ujiki M. Improving attainment of the critical view of safety during laparoscopic cholecystectomy. Surg Endosc 2020; 34 (09) 4115-4123
• 27 Mascagni P, Fiorillo C, Urade T. et al. Formalizing video documentation of the critical view of safety in laparoscopic cholecystectomy: a step towards artificial intelligence assistance to improve surgical safety. Surg Endosc 2020; 34 (06) 2709-2714
• 28 Madani A, Watanabe Y, Feldman LS. et al. Expert intraoperative judgment and decision-making: defining the cognitive competencies for safe laparoscopic cholecystectomy. J Am Coll Surg 2015; 221 (05) 931-940.e8

Publication History

Received: 22 September 2021

Accepted: 26 October 2021

Article published online:
23 December 2021