Introduction
Endoscopic submucosal dissection (ESD) has become standard care for early gastrointestinal
cancers [1]
[2]
[3]. Master and Slave Transluminal Endoscopic Robot (MASTER) is a novel robotic endosurgical
system that has two operating arms equipped with multiple degrees of freedom, enabling
complex maneuvers [4]. The feasibility of performing MASTER ESD has been shown in previous studies [5]
[6]
[7]. Going forward, the value of MASTER will need to be evaluated in terms of the user’s
learning curve and training efficacy. Therefore, it is essential to develop an appropriate
training program and a suitable skill assessment tool.
To provide an objective assessment tool in minimally invasive surgery, the Global
Operative Assessment of Laparoscopic Skills (GOALS) was developed [8]. More recently, the Global Evaluative Assessment of Robotic Skills (GEARS), which
was modeled after GOALS, was also developed [9]. There has not been an objective assessment tool developed for endoluminal surgical
platforms such as MASTER. In this study, we explored use of a new scoring method,
the Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E), which was
modified from both GOALS and GEARS in evaluation of the learning curve of novice operators
using MASTER.
Methods
Procedures
This study involved in vivo and ex vivo experiments. Participating operators performed
ESD using MASTER on two adult female pigs, as well as on isolated pig tissues obtained
from a commercial source. All experiments were conducted at Innoheart Pte Ltd with
prior approval from the Institutional Animal Care and Use Committee. As for the in
vivo experiments, the endoscope was introduced under general anesthesia and the operator
telemanipulated the two robotic arms, one holding a cautery knife and the other a
grasper, via cables that passed through the two channels of the endoscope. For the
ex vivo experiments, the isolated colorectal segment was mounted on a stand to simulate
its normal orientation. Then, an imaginary lesion was marked on the mucosa and MASTER
ESD was performed. The pigs were euthanized after the procedures.
GEARS-E
In creating GEARS-E, we expanded the domains in GOALS and GEARS to encompass features
unique to MASTER procedures ([Table 1]). Modeled after GOALS and GEARS, GEARS-E comprises six domains representing various
skill-related variables (depth perception, bimanual dexterity, efficiency, tissue
handling, autonomy and endoscope control). Assessment of each domain is done by scoring
on a 5-point Likert scale with performance anchors at 1, 3 and 5. A rating of 1 denotes
the lowest proficiency while 5 is the highest. As the focus of this study was on the
novice’s training on robotic manipulators, we left the endoscope manipulation to a
skilled endoscopist independently. To minimize bias from this, we excluded the sixth
domain, endoscopic control, from the assessment. Therefore, the global rating score
for this study was the sum of the scores for each of the remaining five domains, the
maximum total score of which is 25.
Table 1
Six domains of GEARS-E.
|
Depth perception
|
|
1
|
2
|
3
|
4
|
5
|
|
Constantly overshoots target, wide swings, slow to correct
|
|
Some overshooting or missing of target, but quick to correct
|
|
Accurately directs instruments in the correct plane to target
|
|
Bimanual dexterity
|
|
1
|
2
|
3
|
4
|
5
|
|
Uses only one robotic arm, poor coordination
|
|
Uses both arms, but does not optimize interactions between arms
|
|
Expertly uses both arms in a complementary way to provide best exposure
|
|
Efficiency
|
|
1
|
2
|
3
|
4
|
5
|
|
Inefficient efforts, many uncertain movement, constantly changing focus or persisting
without progress
|
|
Slow, but planned movements are reasonably organized
|
|
Confident, efficient and safe conduct, maintains focus on task, fluid progression
|
|
Tissue handling
|
|
1
|
2
|
3
|
4
|
5
|
|
Rough movements, injures tissues, poor grasper control, lack of tension
|
|
Handles tissues reasonably well, minor trauma to tissues, not optimal tension
|
|
Handles tissues well, applies appropriate tension, negligible injury to tissue
|
|
Autonomy
|
|
1
|
2
|
3
|
4
|
5
|
|
Unable to complete entire task, even with verbal guidance
|
|
Able to complete task safety with moderate guidance
|
|
Able to complete task independently without prompting
|
|
Endoscope control
|
|
1
|
2
|
3
|
4
|
5
|
|
Consistently does not optimize view and scope position, extremely frequent and inefficient
adjustment
|
|
View and scope position is sometimes not optimal, needs to adjust scope position frequently
|
|
Controls view and scope position optimally and independently
|
Study design
Five procedures were performed by three operators (Novice-1, Novice-2 and Expert)
([Fig. 1]). Novice operators were overseas gastroenterologists who had no MASTER experience,
but had performed conventional ESD, with Novice-1 having accumulated experience with
approximately 500 ESD cases and Novice-2, more than 1000 such cases. Expert had previously
performed eight animal and human MASTER ESD. To begin with, novice operators watched
a video on the performance of a previous MASTER ESD and then underwent 15 minutes
of practice in manipulating the robotic arms under virtual reality simulation, as
well as in a dry laboratory setting. Then, each novice operator performed one ex vivo
MASTER ESD on the rectum. Subsequently, each novice operator performed one in vivo
MASTER ESD (a gastric ESD by Novice-1, and a sigmoid colon ESD by Novice-2). Expert
separately performed an in vivo MASTER ESD on the sigmoid colon. Every procedure was
self-evaluated using GEARS-E by all three participants, and by an independent evaluator,
who had previously performed three MASTER ESD but was not involved in the procedures
in this study. Statistically significant differences were evaluated by paired two-sided
t-test followed by multiple testing correction using Holm method (P < alpha = 0.05 /n).
Fig. 1 Study design of in vivo and ex vivo animal experiments. Five procedures in all were
performed by three operators (Novice-1, Novice-2 and Expert) and assessed by four
evaluators (including self-evaluations) using GEARS-E.
Results
A total of five procedures were successfully completed, of which two were done each
by Novice-1 and Novice-2, and one by Expert. Every procedure was assessed by the four
evaluators including the Novice-1, Novice-2, Expert and the independent evaluator.
The proficiency and performances of the operators as reflected by their mean scores
from 4 evaluators in the 5 domains is summarized in [Table 2]. The mean global rating scores for Novice-1 first attempt, Novice-1 second attempt,
Novice-2 first attempt, Novice-2 second attempt and Expert’s case were 13.0 (6 – 21),
16.0 (12 – 19), 13.3 (6 – 23), 15.5 (13 – 20) and 21.5 (14 – 25), respectively. The
mean scores for each of the five domains were statistically higher in the second attempts
compared to the first attempts for both Novice-1 (P = 0.0007 < alpha/4) and Novice-2 (P = 0.0486 < alpha/1) ([Fig. 2]). The results suggested that incremental changes in proficiency following repeated
procedure. On the other hand, the mean scores for each of the five domains for Expert’s
case were consistently higher than those for the two novice operators in both the
first and second attempts (P = 0.0012 < alpha/2, P = 0.0008 < alpha/3, respectively).
Table 2
Means and ranges of scores in the five domains and global rating scores.
|
GEARS-E
|
Novice-1 1st
|
Novice-1 2nd
|
Novice-2 1st
|
Novice-2 2nd
|
Expert
|
|
Depth perception
|
2.5 (1 – 5)
|
3 (2 – 4)
|
2.5 (1 – 5)
|
3.3 (3 – 4)
|
4.3 (3 – 5)
|
|
Bimanual dexterity
|
2.8 (1 – 4)
|
3.3 (3 – 4)
|
2.5 (1 – 4)
|
3.3 (3 – 4)
|
4.3 (3 – 5)
|
|
Efficiency
|
2.8 (1 – 4)
|
3.5 (3 – 4)
|
3 (1 – 5)
|
3.3 (3 – 4)
|
4.3 (3 – 5)
|
|
Tissue handling
|
2.5 (1 – 4)
|
3.3 (2 – 4)
|
2.8 (1 – 5)
|
2.8 (2 – 4)
|
4.3 (2 – 5)
|
|
Autonomy
|
2.5 (2 – 4)
|
3 (2 – 4)
|
2.5 (1 – 4)
|
3 (2 – 4)
|
4.5 (3 – 5)
|
|
Global rating score
|
13 (6 – 21)
|
16 (12 – 19)
|
13.3 (6 – 23)
|
15.5 (13 – 20)
|
21.5 (14 – 25)
|
Fig. 2 Mean scores for each of five domains were statistically higher in the second attempts
compared to the first attempts for both Novice-1 (P = 0.0007 < alpha/4) and Novice-2 (P = 0.0486 < alpha/1). Mean scores for each of five domains for Expert’s case were
higher than those for the two novice operators in both the first and second attempts
(P = 0.0012 < alpha/2, P = 0.0008 < alpha/3, respectively).
Discussion
While operation time, completeness of resection and adverse event rates are thought
to be good indices for objective assessment of ESD procedures [6]
[10]
[11]
[12], in reality, using them to compare operator skill or the effects of training across
various clinical conditions can be challenging. For tackling this, a number of assessment
tools have been generated [13]
[14]. Because MASTER is a novel and unique endosurgical platform, an objective and reliable
method of measuring training efficacy and outcome is needed.
In the current study, the means of the global rating scores and of each domain assessed
increased alongside the increases in surgical experiences of MASTER. Especially when
we put all domains together, statistically significant differences in parallel with
experiences were observed. We deduced, therefore, that there is substantial possibility
for GEARS-E to be an objective and consistent scale for assessment of proficiency
in performing MASTER procedure.
On the other hand, we noted the wide-ranging scores of individual evaluators. For
standardization, the anchoring method we applied probably didn’t work well. During
evaluator training, the evaluators in the GOALS study were encouraged to use the full
range of scores [8]. Another possible mean to help evaluators in standardizing their assessment of skills
based on different skill levels is to let them watch video recordings of typical procedures
performed by novice, intermediate and expert operators.
This study had some limitations. First, the evaluators were not blinded, and on the
contrary, we included self-evaluations besides evaluations by other independent evaluators.
Second, because this was a pilot study, sample size was small. Furthermore, due to
the nature of ESD, it is challenging to equalize difficulty factors such as lesion
location and endoscopic conditions. Finally, to focus on novice training, the sixth
domain was excluded from the assessment.
Conclusion
In conclusion, results using GEARS-E showed correlations with surgical experience
in MASTER ESD. As an assessment tool for evaluation of surgical skills, GEARS-E has
a great potential for application in MASTER procedures. For the next step, we will
design a training program that integrates objective GEARS-E skill assessment so that
we can promote MASTER safely and assess its effectiveness across centers around the
world.
