Introduction
Endoscopy training is a multidisciplinary activity, the quality of which is assured by the Joint Advisory Group on Gastrointestinal Endoscopy (JAG). Cross-specialty training is registered via the Joint Advisory Group on Gastrointestinal Endoscopy Training System (JETS), with certification awarded on completion of appropriate courses, direct observation of procedural skills (DOPS) assessments [1]
[2]
[3] , critical indicative procedures, and key performance indicators (KPIs).
JAG recommends weekly, adjusted training lists of eight points; with esophagogastroscopy and flexible sigmoidoscopy allocated one point, and colonoscopy two points, equating to 20 annual training lists. Currently the quality assurance of endoscopy training is assessed by the Global Rating Scale (GRS) in all UK endoscopy units. This is a biannual, self-assessed score rating three training domains (training environment, trainers, and assessment and appraisal), included in the JAG accreditation standards, assessed on JAG visits. Yet does this provide a sufficiently detailed and objective measure of the delivery of training at the unit level?
A UK 2015 training survey reported a disparity in access between medical gastroenterologists and surgeons (0.9 vs. 0.5 lists per week, and 3.0 vs. 1.2 ad hoc training opportunities) [4] and following the Department of Health’s initiative to improve service, the Clinical Endoscopy Training Programme was piloted [5] with a standard of two weekly sessions to promote faster certification.
In the surgical arena, accommodating endoscopy training within operative curricula and educational contracts, aimed at dual accreditation, is especially challenging. Moreover, the COVID-19 pandemic has resulted in sweeping cutbacks in endoscopy services: overall activity has fallen to 12 % of pre-COVID level, cancer detection by 58 %, and training by 93 % (mean 1930 to 133 procedures per week), performed by only 46 trainees [6]. The aim of this study was to assess the quality of endoscopy training in a single UK Statutory Educational Body (SEB) related to hospital units compared with JETS certification standards.
Methods
Data recorded by 211 consecutive cross-specialty trainee endoscopists registered with the JAG on Gastrointestinal Endoscopy Training System (JETS) across 18 hospital endoscopy units throughout 2019, amounting to 28,928 training procedures, were analyzed. Data included the numbers of trainers and trainees, frequency of training lists, procedures performed, Direct Observation of Procedural Skills (DOPS) completed, and Key Performance Indicators (KPI). To satisfy JAG certification guidelines, 20 completed DOPS should be obtained over the course of 200 procedures; DOPS per 200 procedures related to unit were estimated by dividing the total procedures performed per unit by the number 200. ERCP and endoluminal ultrasound were excluded from the analysis because of their status as advanced specialist endoscopic procedures, with fewer hospitals equipped to deliver such training across Wales. The JAG provided ethical approval.
Statistical analysis
One sample Wilcoxon-signed rank test was used to compare observed metrics against JAG standards. Descriptive statistics were used to describe differences in training activity across individual hospital units where no comparative standard existed. Data were tested for normality using a Shapiro-Wilks test and are presented using medians (interquartile range [IQR]). Statistical analysis was conducted using SPSS version 25 (IBM, New York, United States). P < 0.05 was considered statistically significant.
Hospitals were ranked by quartiles according to their median performance against JAG standards. These fall broadly into training activity (training list per trainee, points per training list and DOPS completed per 200 procedures) and training performance (upper and lower gastrointestinal endoscopy KPIs). The top-performing unit was awarded an additional point, and the lowest scoring unit received a score of 0, creating a scale of 0 to 5. Points per training list scores were adapted, because higher quartiles, representing more points per training list, did not correspond with better training performance. A JAG standard of 8 points equated to a score of 5 and graduated to a minimum score of 0 for maximal over- and under- booked lists. Scores were then used to generate a composite index with overall grading (A + to D) based on quartiles, adapted from Hopkins et al, comparing hospital surgical training unit performance in Wales [7].
A radar chart was created for each individual hospital unit, each arm representing the zero to five scale of each metric. These were plotted in Excel (Microsoft, Redmond, Washington, United States) to compare individual training units visually. Percentage coverage was calculated using standard trigonometry.
Results
Quality performance metric median values related to individual hospital units can be found in [Table 1], and further classified by hospital according to local health boards in [Table 2]. Median training list allocation per trainee was 13 (10–17), with only two hospital units (D17, D18) meeting the JAG standard of 20 training lists per year (P = 0.001, effect size d = –0.56). The JAG recommendation that training lists be adjusted to eight points was achieved by one hospital (D17). Median points per list were 11 (5–18, P = 0.064, d = 0.31). Six hospitals under-utilized training opportunities (D1, 7, 11, 12, 13, 14), and 11 over-booked lists, risking inappropriate training environments (T3, D2, 4, 5, 6, 8, 9,10,15,16,18). Median annual total procedure numbers were 1395 (465–2365), amounting to 115 (39–214) procedures per trainee. Dividing the total DOPS per unit by the number of trainees per unit; resulting in median three (1–6) DOPS per trainee per year.
Table 1
Endoscopy training quality performance metrics related to individual hospital units.
|
Metric
|
Median (IQR)
|
Mean (SD)
|
JAG standard
|
P value (effect size)
|
|
Annual training procedures
|
1395 (456–2365)
|
1607 (1270)
|
|
|
|
EGD
|
732 (237–1275)
|
811 (634)
|
|
|
|
Colonoscopy
|
386 (176–837)
|
517 (434)
|
|
|
|
Flexible sigmoidoscopy
|
178 (50–363)
|
279 (290)
|
|
|
|
No. training lists
|
143 (92–239)
|
156 (88)
|
|
|
|
No. trainees
|
12 (7–16)
|
12 (7)
|
|
|
|
No. trainers
|
11 (6–18)
|
11 (6)
|
|
|
|
Trainee:trainer
|
0.8 (0.7–1.3)
|
0.9 (0.4)
|
|
|
|
Training list per trainee
|
13 (10–17)
|
13 (6)
|
20
|
0.001* (d = -0.56)
|
|
Annual number of procedures per trainee
|
115 (39–214)
|
145 (140)
|
|
|
|
Points per adjusted training list
|
11 (5–18)
|
12 (8)
|
8
|
0.064 (d = 0.31)
|
|
DOPS competed per trainee
|
3 (1–6)
|
4 (3)
|
|
|
|
DOPS completed per trainer
|
4 (1–7)
|
4 (3)
|
|
|
|
DOPS completed per list
|
0.2 (0.1–0.4)
|
0.3 (0.2)
|
|
|
|
DOPS completed per 200 procedures
|
6 (2–12)
|
7 (6)
|
20
|
< 0.001* (d = -0.61)
|
|
D2 Intubation
|
93 % (91–96)
|
93 % (4)
|
> 95 %
|
0.068 (d = -0.30)
|
|
J maneuver
|
94 % (90–96)
|
93 % (4)
|
> 95 %
|
0.149 (d = -0.24)
|
|
Cecal Intubation
|
82 % (72–90)
|
80 % (15)
|
> 90 %
|
0.011* (d = -0.44)
|
|
Polyp detection rate
|
25 % (18–34)
|
25 % (10)
|
> 20 %
|
0.035* (d = 0.35)
|
IQR, interquartile range; SD, standard deviation; JAG, Joint Advisory Group on Gastrointestinal Endoscopy; EGD, esophagogastroduodenoscopy; DOPS, direct observation of procedural skills.
Table 2
Endoscopy training quality performance metrics related to local health boards.
|
Metric
|
Median (range)
|
Mean (SD)
|
JAG standard
|
P value (effect size)
|
|
Annual training procedure
|
3866 (2035–7865)
|
4817 (3657)
|
|
|
|
EGD
|
1510 (1510–3098)
|
1968 (1957)
|
|
|
|
Colonoscopy
|
1038 (704–1750)
|
1263 (905)
|
|
|
|
Flexible sigmoidoscopy
|
631 (214–2734)
|
1248 (1346)
|
|
|
|
Number of training lists
|
490 (356–580)
|
467 (140)
|
|
|
|
No. trainees
|
33 (23–48)
|
35 (15)
|
|
|
|
No. trainers
|
37 (21–46)
|
34 (13)
|
|
|
|
Trainee:trainer
|
1 (0.8–1.5)
|
1 (0.31)
|
|
|
|
Training list per trainee
|
13 (12–18)
|
14 (3)
|
20
|
0.028 (d = -0.64)
|
|
Annual number of procedures per trainee
|
117 (64–193)
|
135 (96)
|
|
|
|
Points per adjusted training list
|
10 (8–13)
|
11 (6)
|
8
|
0.249 (d = 0.33)
|
|
DOPS competed per trainee
|
3 (2–6)
|
4 (3)
|
|
|
|
DOPS completed per trainer
|
3 (2–7)
|
4 (3)
|
|
|
|
DOPS completed per list
|
0.24 (0.16–0.42)
|
0.28 (0.15)
|
|
|
|
DOPS completed per 200 procedures
|
9 (2–11)
|
8 (4)
|
20
|
0.028 (d = -0.64)
|
|
D2 Intubation
|
94 % (91–96)
|
93 % (3)
|
> 95 %
|
0.345 (d = -0.27)
|
|
J maneuver
|
94 % (92–97)
|
94 % (3)
|
> 95 %
|
0.249 (d = -0.33)
|
|
Cecal Intubation
|
85 % (81–90)
|
78 % (17)
|
> 90 %
|
0.075 (d = -0.51)
|
|
Polyp detection rate
|
24 % (21–30)
|
27 % (6)
|
> 20 %
|
0.028 (d = 0.64)
|
IQR, interquartile range; SD, standard deviation; JAG, Joint Advisory Group on Gastrointestinal Endoscopy; EGD, esophagogastroduodenoscopy; DOPS, direct observation of procedural skills.
Median DOPS per 200 procedures was six (2–12); significantly fewer than the JAG target (P < 0.001, d = –0.61), with only one hospital (D11) achieving the standard.
Hospitals were ranked by quartiles according to their median performance against KPI JAG standards ([Fig. 1]). Individual hospital performance can be found in Supplementary Table 1.
Fig. 1 Key performance indicators (KPIs) per hospital unit (ranked by quartiles). Bars represent median with interquartile range. Shaded section represents performance meeting/exceeding Joint Advisory Group on Gastrointestinal Endoscopy standard for each KPI.
Esophagogastroduodenoscopy KPIs
D2 intubation and J maneuver were 93 % (91–96) and 94 % (90–96) respectively, similar to the JAG standard of 95 % (P = 0.068 d = –0.30; P = 0.149 d = –0.24).
Colonoscopy KPIs
Caecal intubation was 82 % (72–90); 14 hospitals achieved lower rates than the JAG standard (90 %, P = 0.011, d = –0.44). polyp detection rate (PDR) was 24 % (21–30); 13 hospitals exceeded the JAG standard (20 %, P = 0.035, d = 0.35).
Compound-level unit quality analysis
Compound-level scores were calculated for all hospitals and can be found in [Fig. 2]. The median score was 17 (14–20), with a mean of 17 (standard deviation (SD) 5). Radar charts demonstrating the relative performance of the best versus the worst units can be found in [Fig. 3]. Median radar coverage was 22.9% (16.4 %–34.2 %), mean 27.1 % (SD 13.8 %).
Fig. 2 Compound scores related to hospital unit. D, district general hospital; T, tertiary hospital; red, quartile 1; orange, quartile 2; yellow, quartile 3; green, quartile 4.
Fig. 3 Radar chart of lowest and highest performing unit.
Discussion
The age of deliberate accountability has arrived, of bonus for measured performance, and of faith in the value of publishing outcomes to guarantee clarity. This is the first study to provide objective comparative data regarding hospital unit level endoscopy training quality metrics within a UK Statutory Education Body. The salient findings related to KPI variation and consequently training deficit.
Esophagogastroduodenoscopy (EGD) KPIs with regard to D2 intubation and J maneuver were 96 % in upper quartile units compared with 91 % and 90 % in the lower quartile: Colonoscopy KPIs with regards to cecal Intubation and PDR were 90 % and 30 % in upper quartile units compared with 72 % and 21 % in lower quartile units, translating to training metric radar plot areas that differed four-fold.
The surgical adage and prior common practice of “see one, do one, teach one” is long gone and fails contemporary scrutiny and outcome metrics in the modern medical world of audit, measured accountability, and patient safety and quality of care. With the European working time directive, reduction in working hours and “shape of training” shortening the length of specialist training [8], it is more important than ever to train effectively, and to recognize competency-based proficiency rather than arbitrary minimum thresholds. A recent review of training provision for Gastroenterology trainees in Wales, revealed a two-fold variation across health boards, at a rate inadequate to achieve full certification in gastroscopy and colonoscopy in the “shape of training” era [9]. Indeed, JAG considers both measures, with certification awarded following completion of competency-based assessments in addition to a minimum level of indicative certified numbers of procedures [10].
Dissatisfaction related to endoscopy training is most commonly attributed to poor access to training lists, with training progression heavily influenced by hospital placement on-call rotas and other general medical responsibilities [11] This issue amplified with respect to surgical trainees in view of competing scheduled elective surgical demands [4]
[12]
[13], resulting in as few as 18 % of Colorectal trainees achieving the minimum number of colonoscopy procedures by the end of training [14]. The most recent UK cross-specialty survey reinforced this, highlighting again a discrepancy in training, with 38.9 % of surgical trainees accessing one training list per week, and 70 % of gastroenterology trainees having two or more scheduled endoscopy lists per week [15]. Furthermore, the UK JAG census reported a 12 % to 15 % increase in service pressures between 2017 and 2019, with a clear adverse effect on training: dedicated training list numbers falling from 76 % in 2017 to 52 % in 2019 [16]. The introduction of the Clinical Endoscopy Training Programme in 2016 aimed to improve service provision and has increased training list requirement two-fold for clinical nurse endoscopists [17]. Hospital size and teaching status does not reflect the output through an endoscopy unit, with smaller hospitals potentially better equipped to train in endoscopy given the lack of alternative competing interests. In an increasingly competitive arena for training resource, increased focus on the practicalities of training provision for all groups of endoscopy trainees should ensure equity.
Quality metrics related to endoscopy training utilizing surrogate markers of hospital performance are conspicuous by their absence. JAG considers factors such as; simulation, course attendance, lifetime procedure numbers, and frequency of training and non-training lists, as markers of training quality [12]. What this study adds is training activity metrics, and in particular: numbers of training lists, adjustments to training needs, completion of satisfactory summative DOPS assessments, and training performance KPIs. Hospital, compound-level surgical training quality performance metrics illustrated by means of radar chart plots have been reported previously; specifically, to compare core surgical training quality related to hospital units in the Wales SEB. Important performance training metric variation was apparent for 6-month hospital placements, with trainees in upper quartile units achieving on average, 20 % more operations, 40 % more WBAs, and one more peer reviewed publication, than trainees in lower quartile units [7]. Compound-level surgical training and endoscopy training quality performance metrics were not consistent: with one bottom-quartile performing unit for surgical training in the top quartile for endoscopy training; seven units featuring in the same quartile, and seven district general hospitals providing endoscopy training without surgical training.
This study has a number of inherent limitations. The data pertain to a solitary UK SEB, and therefore, they must be interpreted with caution. Individual trainee outcomes were not assessed, and as a result, individual hospital unit training quality was unable to be directly compared based on trainee performance. Furthermore, relative learning curve trajectories of these trainees could not be determined. Factors affecting the learning curve in colonoscopy have been previously described [10], but the learning curve trajectory was not studied. In relation to six general surgical indicative procedures, the trajectory varied 5-fold [18]: endoscopy learning curve trajectories are worthy of further research but are arguably likely to vary similarly.
Moreover, this was a retrospective analysis of prospectively collected data from all cross-specialty trainees who recorded training activity on JETS: measuring competence related to trainees’ self-perceived and reported KPIs also has limitations. Surgical trainees are allocated rotations within individual hospital units; however, training opportunities may present themselves across sites within local health boards. It is possible that trainees and, or trainers are registered at more than one hospital which may risk duplication within the JETS database. Training lists and numbers of DOPS completed per 200 procedures were calculated related to individual trainees, and crude data inaccuracy could influence results. It is possible that endoscopy services may be unevenly distributed between hospitals within a local health board, this should be considered when interpreting the results and planning trainee allocation.
Through initiating GRS, JAG attempts to define clearly what a training list should be: with a qualified trainer and an appropriate case mix and number. However, it is known that some “training lists” exist that do not meet the definition. While JAG recommend adjusting a training list to 8 points and reflect an appropriate case mix, it should be recognized that this broad recommendation requires tailoring to the need of the individual trainee. An endoscopist at the start of their endoscopy training may initially require less than 8 points, while those more senior, approaching certification, should be performing at an independent level and therefore a 10- to 12-point training list would be more appropriate. Considering this recommendation as a fixed standard in this study does not reflect the need for adaptability and can only be generalized given the lack of specific trainee outcomes or stages of training. Furthermore, trainers and trainees alike highlight the importance of ad hoc lists, especially as trainees increase their experience and become more independent. This study does not capture these lists, in which opportunistic training occurs.
Recognizing the variation in training quality across hospital units is imperative to maximize available opportunities, however this study strengthens existing evidence that poor access to training lists remains a significant problem. Endoscopy training fellowships have been suggested as one option to compensate for these deficiencies [9]. In addition, the SPRINT program: Structured Programme for Induction and Training, an existing initiative to improve the training delivery in Wales, incorporates simulator and lesion recognition training, with endoscopic non-technical skills, and has been reported to shorten time to reach 200 procedures [19]. With the evolution of high-fidelity virtual reality simulators, this is an area requiring further research to develop and validate a structured simulator-based training curriculum as an adjunct to hands-on training.
Conclusions
Important disparities in hospital endoscopy unit performance were observed and disguised by the cloak of clinical pressures currently prevalent in the NHS. Compound hospital training quality varied three-fold. Trainees, trainers and training program directors alike should be aware of such data when planning educational programs, so that the quality of endoscopy training can be focused and optimized. JAG now considers simulation to be an important and integral marker of training. Adding simulation to the training armamentarium should be urgently recognized as a paramount constituent of the recovery-phase of COVID-19 training catch-up strategy, in order to overcome rationed front-line clinical training opportunities and also to address the pressing clinical service back-log of urgent suspected cancer referrals. Development of a nationally agreed and accredited curriculum allied to endoscopic virtual reality haptic feedback will be key to recovery and improved endoscopy training.
