Introduction
Colorectal cancer (CRC) is the third most common cancer in both sexes and the second
leading cause of death in the world [1]. Screening for CRC and the removal of neoplastic polyps by colonoscopy has led to
a substantial improvement in survival [2].
Optical diagnosis aims to predict the histology of a polyp based on its endoscopic
features. This practice could avoid pathological analysis in several cases and reduce
the derived costs. The American Society for Gastrointestinal Endoscopy’s PIVI working
group established diagnostic thresholds for real-time implementation of optical diagnosis
for diminutive polyps (≤ 5 mm) [3]. However, PIVI criteria have not yet been met in community-based practices or in
non-expert hands [4]
[5]. In this regard, the European Society of Gastrointestinal Endoscopy (ESGE) emphasizes
the importance of being able to ensure and maintain competence in optical diagnosis
as well as considering only the proportion of high-confidence diagnosis as a benchmark
[6].
During the past few decades, considerable technological advances have been made in
the application of artificial intelligence (AI) to medicine. Computer-aided diagnosis
(CADx) is a promising solution to overcome human variation in characterization of
polyps by providing decision support. In this specific field, CADx approaches based
on deep learning (DL) represent an advantage over previous machine learning by combining
both the automatic extraction and classification of image characteristics using a
multilayered system called convolutional neural networks (CNNs) [7].
The quality and design of published CADx systems has varied over time. Initial studies
were carried out retrospectively and were tested ex vivo using selected stored images
[8]
[9]
[10]
[11]
[12]. More recent studies, most of them DL approaches, have been prospectively conducted
reporting higher accuracy with faster processing times, which allows diagnosis in
real time [13]
[14]
[15]
[16]. However, to date, only CADx systems using complex optical technologies as endocytoscopy
have been tested in vivo [13], that is, while the colonoscopy is being performed, which still hinders the adoption
of this technology in daily practice. Furthermore, the newest CADx systems are currently
using advanced imaging modalities, which clearly limits their implementation worldwide.
The aim of this study was to evaluate the efficacy of a new CADx system based on DL
called ATENEA for in vivo optical diagnosis in consecutive patients using only white
light endoscopy (WLE) and compare its performance with endoscopists.
Methods
Development of ATENEA
As a first step, images of any polyp from routine colonoscopies performed at Hospital
Clínic of Barcelona from January 2016 to December 2020 were prospectively collected.
The images were captured with high-definition colonoscopes (CF-HQ180, CF-HQ-185, CF-HQ-190
and EVIS EXERA III videoprocessor, Olympus Europe, Hamburg, Germany) using an external
computer with a frame grabber to ensure image acquisition with the highest quality.
Only white light images without magnification or chromoendoscopy were used. Data from
morphology, location, and size of the polyp were collected and periodically transferred
by an assistant into the database along with the histological category of the lesion
obtained after pathological analysis.
The development of ATENEA consisted of two stages: feature extraction and image classification.
Feature extraction was made by using a faster region-based CNN with ResNet50 as backbone.
For system training purposes, a region of interest (ROI) delineating the polyp was
manually defined by clinicians using a program called GTCreator ([Fig. 1]) [17]. The extracted features, ROI, and actual histologic class of the polyp were used
to train the system. ATENEA learned to classify images into adenoma or non-adenoma
categories using an 80 % (high confidence) threshold value (predictions with a confidence
value < 80 % were not considered valid to represent the actual performance of the
system as they could not guarantee a robust performance).
Fig. 1 Example of a manually defined region of interest (ROI) delineating the polyp using
GTCreator in the training phase.
ATENEA was trained and validated with a total of 1049 high-definition white light
images of 483 polyps from 354 patients, with a maximum of three images of the same
polyp (but with a different view or perspective). Images had variable quality but
all had a visible mucosal pattern and only blurred images and polyps covered by mucus
were excluded. Images containing patient data or without histological analysis were
also excluded. About two-thirds were adenomas and one-third non-adenomas, following
a similar proportion to what is found in real life, with a median size of 4.5 mm (range
2 mm-20 mm). These images were randomly split into training (n = 837) and validation
sets (n = 212), with the condition that all the images of the same polyp were in the
same set.
In vivo experiment
This observational prospective cohort study was performed in Hospital Clinic of Barcelona
between January and March 2021 and included individuals from the fecal immunochemical
test-based (cut-off of ≥ 20 μg of hemoglobin/g of feces) organized population CRC
screening program in which all individuals aged 50 to 69 years are invited to participate.
Colonoscopies were performed by four staff endoscopists with more than 1000 colonoscopies
and adenoma detection rates of 47 %, 48%, 51 %, and 55 %, respectively (average in
this program and time period was 49 %). No specific training in optical diagnosis
was performed for the study purpose. The study was conducted according to the guidelines
of the Declaration of Helsinki and approved by the Institutional Review Board of Hospital
Clínic of Barcelona (HCB2017/0506, 7/18/2017) and informed consent was obtained from
all patients involved in the study. It was also registered in ClinicalTrials.gov (NCT03775811).
All polyps detected and resected with a final pathological report were prospectively
included regardless of image quality (not centered polyps, blurred or covered by mucus).
Only the image that was considered adequate for the prediction (either by the orientation
of the polyp within the image or by the proximity for its correct assessment) was
selected by the endoscopist during the exploration and was analyzed. The following
variables were collected: estimated size (in mm), location (rectum-sigma, descending,
transverse, ascending) and morphology according to Paris classification [18], which were properly noted in the colonoscopy report.
In real time, ATENEA classified each polyp as an adenoma or non-adenoma and provided
the confidence value for each prediction. To avoid losing any prediction, we were
more flexible than in train/validation phases and the minimum threshold was reduced
to 50 %. Values between 50 % and 80 % were considered low-confidence predictions.
In this phase, the system was fully automatic (without delineation of ROI), providing
for each lesion a bounding box and its corresponding histological class. Time needed
to acquire and process each image to obtain the automatic prediction was of 40 milliseconds
(real time).
For the optical diagnosis process, the endoscopists were asked to categorize each
lesion into two categories (adenoma and non-adenoma) based on its surface pattern
and to provide its diagnostic confidence (high vs low) without any time limitation.
Endoscopists were blinded to the ATENEA output. The decision was intuitive and based
on their previous experience but without using chromoendoscopy or any of the existing
classifications. Serrated sessile lesions (SSLs) were included in the non-adenoma
category. [Fig. 2] shows the set-up in the exploration room.
Fig. 2 Setting in the endoscopy room showing the position of the assistant sitting in front
the computer: the endoscopist is blind to the image displayed in the computer and
ATENEA’s output.
Histopathology
All polyps were removed using the usual techniques and sent separately for further
evaluation to the Pathology Department, which was the gold standard. The diagnosis
of dysplasia in neoplastic polyps was made based on the Modified Vienna Classification
[19]. The pathologist was blinded to the predictions of both endoscopist and ATENEA.
Statistical analysis
All the polyps were globally analyzed, independently of the endoscopist who performed
the exploration. The numbers of polyps that were true positive (adenomatous polyps
predicted to be adenomatous), true negative (non-adenomatous polyps predicted to be
non-adenomatous), false positive (non-adenomatous polyps predicted to be adenomatous)
or false negative (adenomatous polyps predicted to be non-adenomatous) were calculated.
Sensitivity, specificity, positive predictive value, negative predictive value and
accuracy with 95 % confidence intervals were calculated. Comparisons of these metrics
between ATENEA and endoscopists were performed with Chi squared test.
Receiver operator characteristics (ROC) curves for use of different prediction confidence
values to determine ATENEA in vivo performance for diagnosis of adenoma were constructed
with Matlab. The area under the curve (AUC) and optimal operating point with its sensitivity
and specificity and 95 % confidence intervals were calculated.
Two-sided P < 0.05 was considered statistically significant. All calculations were performed
with R version 4.1.2 and Matlab for Windows version 2020.
Results
During the study period, 90 polyps from 31 consecutive patients were included. Sixty-nine
(76.7 %) were adenomas with a median size of 5.0 mm (range: 2–25 mm). Characteristics
of the polyps are described in [Table 1].
Table 1
Characteristics of the polyps included in the study.
|
In vivo test (n = 90)
|
|
Histological type
|
|
|
69 (76.7 %)
|
|
|
21 (23.3 %)
|
|
|
16
|
|
|
5
|
|
Size (in mm)
|
|
|
52 (57.8 %)
|
|
|
21 (23.3 %)
|
|
|
17 (18.8 %)
|
|
Location
|
|
|
22 (24.4 %)
|
|
|
68 (75.6 %)
|
|
Morphology
|
|
|
5 (5.5 %)
|
|
|
30 (33.3 %)
|
|
|
55 (61.1 %)
|
ATENEA provided an output for all 90 polyps and was able to correctly predict the
histology in 63 of 69 adenomas (sensitivity: 91.3 %, 95 % CI: 82 %–97 %) and 12 of
21 non-adenomas (specificity: 57.1 %, 95 % CI: 34 %–78 %). Endoscopists were able
to provide a correct optical diagnosis in 52 of 69 adenomas (sensitivity: 75.4 %,
95 % CI: 64 %–85 %) and 20 of 21 non-adenomas (specificity: 95.2 %, 95 % CI: 76 %–100 %),
with an accuracy of 83.3 % (95 % CI: 74 %–90 %) and 80 % (95 % CI: 70 %–88 %) for
CADx and endoscopists, respectively ([Table 2]).
Table 2
Performance characteristics of ATENEA and the endoscopists for diagnosis of adenoma.
|
TP
|
FP
|
TN
|
FN
|
PPV
|
SENS
|
NPV
|
Spec
|
Accuracy
|
|
ATENEA, n = 90
|
63
|
9
|
12
|
6
|
87.5 %
(95 % CI: 78 %–94%)
|
91.3 %
(95 % CI: 82 %–97%)
|
66.7 %
(95 % CI: 41 %–87%)
|
57.1 %
(95 % CI: 34 %–78%)
|
83.3 %
(95 % CI: 74 %–90%)
|
|
Endoscopists, n = 90
|
52
|
1
|
20
|
17
|
98.1 %
(95 % CI: 90 %–100%)
|
75.4 %
(95 % CI: 64 %–85%)
|
54.0 %
(95 % CI: 37 %–71%)
|
95.2 %
(95 % CI: 76 %–100%)
|
80.0 %
(95 % CI: 70 %–88%)
|
|
P value
|
|
0.07
|
0.02
|
0.55
|
0.01
|
0.7
|
|
ATENEA diminutive polyps
≤ 5 mm, n = 52
|
30
|
7
|
11
|
4
|
81.1 %
(95 % CI: 65 %–92%)
|
88.2 %
(95 % CI: 73 %–97%)
|
73.3 %
(95 % CI: 45 %–92%)
|
61.1 %
(95 % CI: 36 %–83%)
|
78.8 %
(95 % CI: 65 %–89%)
|
|
Endoscopists diminutive polyps
≤ 5 mm, n = 52
|
20
|
1
|
17
|
14
|
95.2 %
(95 % CI: 76 %–100%)
|
58.8 %
(95 % CI: 41 %–75%)
|
54.8 %
(95 % CI: 36 %–73%)
|
94.4 %
(95 % CI: 73 %–100%)
|
71.1 %
(95 % CI: 57 %–83%)
|
|
P value
|
|
0.27
|
0.01
|
0.38
|
0.04
|
0.5
|
|
ATENEA small polyps
< 10 mm, n = 73
|
47
|
9
|
12
|
5
|
83.9 %
(95 % CI: 72 %–92%)
|
90.4 %
(95 % CI: 79 %–97%)
|
70.6 %
(95 % CI: 44 %–90%)
|
57.2 %
(95 % CI: 34 %–78%)
|
80.8 %
(95 % CI: 70 %–89%)
|
|
Endoscopists small polyps
< 10 mm, n = 73
|
36
|
1
|
20
|
16
|
97.3 %
(95 % CI: 86 %–100%)
|
69.2 %
(95 % CI: 55 %–81%)
|
55.5 %
(95 % CI: 38 %–72%)
|
95.2 %
(95 % CI: 76 %–100%)
|
76.7 %
(95 % CI: 65 %–86%)
|
|
P value
|
|
|
|
|
0.09
|
0.02
|
0.46
|
0.01
|
0.69
|
TP, true positive; FP, false positive; TN, true negative; FN, false negative; PPV,
positive predictive value; SENS, sensitivity; NPV, negative predictive value; SPEC,
specificity.
With respect to diminutive polyps (≤ 5 mm), ATENEA was able to correctly predict the
histology in 31 of 35 adenomas (sensitivity: 88.2 %, 95 % CI: 73 %–97 %) and 11 of
18 non-adenomas (specificity: 61.1 %, 95 % CI: 36 %–83 %) and endoscopists in 20 of
35 adenomas (sensitivity: 58.8 %, 95 % CI: 41%–75 %) and 17 of 18 non-adenomas (specificity:
94.4 %, 95 % CI: 73 %–100 %), respectively, with an accuracy of 78.8 % (95 % CI: 65 %–89 %)
and 71.1 % (95 % CI: 57 %–83 %) for ATENEA and endoscopists, respectively ([Table 2]). Results of prediction for small polyps (< 10 mm) are also shown in [Table 2].
ATENEA and the endoscopists disagreed in their prediction in 31 of 90 cases (34.5 %).
Endoscopists made their prediction with high confidence in 79 cases (87 %) and with
low confidence in 11. In all these 11 cases, ATENEA made a good prediction ([Fig. 3]).
Fig. 3 ATENEA and endoscopists’ predictions for all polyps. Each circle represents a polyp
and colors correspond to a correct prediction (green) and incorrect prediction (red)
with high confidence (full circle) or low confidence (half circle).
The ROC curve for ATENEA in vivo performance ([Fig. 4]) showed an AUC of 0.782. The optimal operating point was achieved by using 74.2 %
as the threshold value with a sensitivity of 86.9 % (95 % CI: 79.9 %–93.9 %) and a
specificity of 66.7 % (95 % CI: 56.9 %–76.4 %).
Fig. 4 Receiver operating characteristic (ROC) curve for different prediction confidence
values for ATENEA. The optimal operating point (defined as the point in the curve
with better balance of specificity and sensitivity) was achieved by using 74.2 % as
the threshold value.
Discussion
This study demonstrates that ATENEA, a fully automatic optical diagnostic system,
can accurately predict polyp histology in an in vivo setting using only white light,
which answers one of the 10 key research questions identified by international experts
related to AI implementation in colonoscopy [20]. ATENEA is particularly useful for identifying adenomatous lesions, showing its
readiness to be used in a clinical environment.
Until now, the majority of CAD systems developed for characterizing colorectal polyps
that use AI have required advanced optical diagnostic equipment, such as narrow band
imaging (NBI) [9]
[10]
[11]
[12]
[21] and endocytoscopy [8]
[22]
[23]. However, WLE. which is the most common endoscopic modality, has not been extensively
studied in this context yet. For this reason, we used only white light images to allow
for widespread use of ATENEA regardless of the manufacturer and model of endoscope
used.
Our group previously developed a hand-crafted predictive model based on extraction
of surface patterns (textons) over white light images. This system was validated in
a small dataset containing only 225 high-quality images with a diagnostic accuracy
of over 90 % [24]. Unfortunately, the system was not fully automatic and could not perform in real
time and, consequently, it could not be used in an in vivo experiment. In contrast,
ATENEA can perform in vivo automatic polyp classification due to its capability of
calculating the result in less than 40 ms (real time). It is important to make a difference
between what real-time and in vivo means: the first refers to the time the system
takes to process an image, whereas the second is related to where and when the system
is applied (in vivo: in the exploration room versus ex vivo: off-line experiments).
These concepts are commonly confused in the literature and, in most of the cases,
real-time is used instead of in vivo without mentioning the actual processing time
[13]
[14].
The results obtained in the present experiment showed that ATENEA had better global
accuracy than endoscopists, particularly for adenomatous lesions. Conversely, clinicians
performed better for the non-adenoma category. It has to be pointed out that they
had more information than the system because they knew the location and size of the
lesion (variables for which ATENEA was blind), so this could lead to a major pretest
probability of better diagnosis of non-adenomatous lesions when faced with a small
or diminutive polyp in the rectosigmoid colon. In contrast, ATENEA performance depended
solely on the number and variability of examples from each of the classes in the training
set.
The lower performance of ATENEA for non-adenomatous lesions could limit implementation
of a “leave in situ” policy. However, it must be stated that the training of the AI
system was greatly affected by the reduced number of examples of this class in the
dataset. There are two reasons for this: first, the prevalence of non-adenomatous
lesions is generally lower and second, they are not systematically removed when located
in the rectosigmoid colon, making their collection difficult. Hence, it is necessary
to enlarge the dataset both in quantity and percentage of examples of the minority
class.
CADx systems are not intended to replace endoscopists, but rather, to help them in
their tasks. As stated by the ESGE review on advanced endoscopic imaging, the most
likely scenario is that the intelligent systems will be used as a “second reader”
to support the endoscopist’s final diagnosis [25]. In this sense, an important result from our study was that if endoscopists with
low confidence had followed the output of the system, all their predictions would
have been correct. This shows the potential of ATENEA to assist clinicians with less
expertise.
Similar to endoscopists, AI systems do also provide a confidence value in their predictions
using a percentage instead of a binary assessment (high versus low). By varying the
confidence value, we represented different performances of ATENEA using a ROC curve.
Our results show that if the confidence value is low, the sensitivity of the system
is lower as it provides more outputs, some of them erroneous. Conversely, if we increase
the allowed confidence value, less but more confident outputs will be provided, with
the risk of missing some of the lesions without a prediction. The so-called Simple
Optical Diagnosis Accuracy or SODA criteria, which were recently published by the
ESGE Curricula Working Group [26], are more flexible than the PIVI criteria and emphasize the importance of not leaving
any diminutive lesion with advanced neoplasia in situ. In accordance with SODA criteria,
we considered sensitivity to be the most important outcome of the intelligent system.
The AUC value obtained in our study is similar to the 0.84 reported in a recent meta-analysis
using only WLE [27] (which is usually lower than the AUC in studies using chromoendoscopy or magnification
techniques).
Unlike other studies in which low-quality images or the known “difficult for AI cases”
were not included, the main strength of this study is that it was performed under
real clinical practice conditions and polyps were included regardless of image quality
(not centered polyps, blurred or covered by mucus). If we had excluded these polyps
from the analysis, the ATENEA performance would have been better. Nevertheless, the
lack of publicly available annotated datasets does not allow for a fair comparison
between ATENEA and other CADx systems. In this sense, comparison of metrics in meta-analysis
are difficult to understand as the number, quality, and class of the images are different
in each study and affect training and testing stages of the validation of computational
systems.
The study has the following limitations. First, the training dataset was small in
terms of number of different polyps. To mitigate this, we used pre-trained weights
from ImageNet [28], which is a more general dataset and we applied data augmentation operations (color
transformation, rotation, horizontal and vertical flip) to enlarge its size. The collaboration
with other centers and the public availability of other datasets could also be of
great use to both enlarge the dataset and perform multicenter validations, increasing
the robustness of the results. Second, our study did not consider a separate class
for SSL due to the low number of examples in the dataset. The problem of SSLs has
not adequately been addressed in other previous studies. It is well known that SSLs
are neoplastic lesions and there is not an ideal optical method for their characterization
[29]. Due to the clinical relevance of SSLs, some studies propose a classification between
neoplastic and non-neoplastic lesions instead of using adenoma vs. non-adenoma. Following
this logic, if we had included all SSL in the same category as adenomas, four of five
SSLs would have been correctly classified.
Conclusions
In conclusion, ATENEA, a CADx system based only on WLE data, is ready to be used for
in vivo and real-time characterization of colorectal polyps, enabling the endoscopist
to make direct decisions. ATENEA achieved a global accuracy similar to endoscopists,
despite having lower performance for non-adenomatous lesions.
