Introduction
In recent years, advancements in image processing techniques have profoundly impacted the way gastrointestinal endoscopy is performed. High-definition and 4K video technologies now enable unprecedented visualization of vascular and mucosal details. With this enhanced image quality, the bottleneck in disease detection and diagnosis has shifted from visualization to interpretation by the endoscopist. Artificial intelligence (AI) has already shown great promise in assisting this interpretation [1 ]
[2 ]. Nonetheless, the same technological progress that empowers AI also introduces potential challenges that may frustrate successful implementation in daily practice.
Current endoscopy systems offer a wide range of post-processing enhancement settings that may improve image characteristics such as color, texture, and contrast ([Fig. 1 ]). Endoscopists can change these settings according to their own preferences or specific situations. Enhancement settings, while often subtle to the human eye, may significantly impact the accuracy of AI predictions. [Fig. 2 ] exemplifies this issue. This remarkable behavior can be attributed to a phenomenon known as “domain shift.” This term describes the tendency of deep learning systems to perform well on familiar data but deteriorate quickly on data that slightly deviate from the training dataset [3 ]
[4 ]. In the example shown in [Fig. 2 ], an AI system trained and validated solely on enhancement setting X may perform significantly less well when exposed to enhancement setting Y. In other fields of medicine, studies have shown that domain shift can significantly impact the reliability of AI systems [5 ]
[6 ]
[7 ]. The potential risk of domain shift is currently an underexplored issue for AI systems in gastrointestinal endoscopy.
Fig. 1 Examples of subtle differences in post-processing settings of endoscopy systems from different manufacturers: enhancement setting type A1 vs. A8 (left; Olympus, Tokyo, Japan), standard settings vs. tone and structure enhancement (middle; Fujifilm, Tokyo, Japan), and white-light endoscopy vs. I-SCAN 1 (right; Pentax, Tokyo, Japan; courtesy of Rehan Haidry).
Fig. 2 Two separate predictions of a computer-aided detection system for Barrett’s neoplasia. The same case is displayed twice with different levels of image enhancement. While the upper lesion is detected with high confidence, the same lesion projected with a different enhancement setting is missed.
One approach that can help mitigate the effects of domain shift is data augmentation. Data augmentation refers to the technique used to increase the diversity of a training dataset without actually collecting new data. It involves artificially creating new training images by applying transformations to existing images [8 ]. This can include operations such as rotating, flipping, and cropping images, adjusting contrast, or changing colors. The goal is to make AI systems more robust against new data that differ from the training dataset (i. e. domain shift).
A specialized form of data augmentation is image enhancement-based data augmentation. In this approach, the augmentations focus on transformations that closely align with the changes in post-processing enhancement settings available on endoscopy processors.
In this study, we evaluated the impact of domain shift due to post-processing enhancement settings on performance consistency of two endoscopic AI applications: computer-aided detection (CADe) of Barrett’s neoplasia and computer-aided diagnosis of colorectal polyps (CADx). We trained both CAD systems on datasets with limited variability of enhancement settings using standard data augmentation. We then tested these CAD systems across a wide range of enhancement settings resembling the heterogeneity encountered in daily clinical practice.
Subsequently, we evaluated image enhancement-based data augmentation methods to improve the performance consistency or robustness of both CAD systems against the image heterogeneity of enhancement settings.
Methods
Experimental setup
In this study we aimed to evaluate the impact of frequently used image enhancement settings on the performance of endoscopic AI systems. We investigated this based on two prevalent AI applications in endoscopy: CADe for primary detection of early Barrett’s neoplasia using white-light endoscopy, and CADx for optical diagnosis of colonic polyps using narrow-band imaging. For the current study, we redeveloped these two CAD systems based on datasets originating from published studies. First, we retrained these systems using their original datasets, which were acquired with limited variability in enhancement settings. These CAD systems were then tested across a wide range of test sets, each comprising exactly the same images, but displayed with different enhancement settings. Then, both CAD systems were retrained using specific data augmentation methods. The performance of these adjusted CAD systems was then re-evaluated on the same array of test sets.
Datasets for CAD systems
The CADe dataset consisted of Barrett’s esophagus images collected both retrospectively and prospectively for the development of a previously published CADe system by our own research group [2 ]
[9 ]. All data were acquired in 11 international Barrett’s referral centers. Images were carefully collected and curated according to standardized protocols to adhere to high quality standards. The dataset contained 3339 neoplastic images and 2884 nondysplastic images originating from 637 and 269 Barrett’s patients, respectively. All images were captured with the HQ190 gastroscope and the CV-190 processor (Olympus, Tokyo, Japan) using white-light endoscopy without magnification. The vast majority of images were captured with enhancement setting A5.
The CADx dataset consisted of colonoscopy images collected prospectively for the development of a CADx system for characterization of diminutive colorectal polyps, and is publicly available [10 ]. Data were acquired in eight Dutch hospitals. All images were collected following a specific workflow to maintain quality consistency across centers. The dataset was separated into a neoplastic group, with adenomas and sessile serrated lesions, and a non-neoplastic group, with hyperplastic polyps. The neoplastic group comprised 2746 images from 736 patients, while the non-neoplastic set contained 542 images originating from 233 patients. Images used in the current study were obtained with 190-series endoscopes and CV-190 processors (Olympus) using narrow-band imaging without magnification. Images were captured with enhancement settings A3 and A5.
Examples of both datasets are given in [Fig. 3 ].
Fig. 3 Cases featured in the dataset for computer-aided detection of Barrett’s neoplasia (upper row) and computer-aided diagnosis of diminutive colorectal polyps (lower row).
Enhancement settings
Both datasets were collected using the EXERA III CV190 processor from Olympus. This device offers multiple enhancement settings for both white-light endoscopy and narrow-band imaging. The most frequently used settings are:
enhancement type A: this setting enhances fine patterns in the image by increasing contrast and sharpness;
enhancement type B: this setting enhances even finer patterns than type A by more subtly increasing contrast and sharpness.
During an endoscopic procedure, the endoscopist can select one distinct setting and its degree of enhancement (1–8, where a higher number represents stronger enhancement). These settings, as illustrated in Fig. 1 s in the online-only Supplementary material, are typically preconfigured by the hospital’s maintenance or technical services team and remain unchanged throughout different procedures.
Conversion software
In this study, we used a proprietary software tool that was specifically developed to exactly replicate original EXERA III CV190 images with different enhancement settings. By inputting an image and its original enhancement setting, the software can generate equivalent images at other settings. The main differences between these settings are achieved by amplifying or reducing image sharpness and contrast. Comparative examples of original and converted images are provided in Fig. 2 s . As shown, the converted images are virtually indistinguishable from original images.
Data augmentation
In this study, two different data augmentation methods were used.
Standard data augmentation: this method included geometric transformations (e. g. rotation and flipping), color transformations (e. g. contrast and saturation adjustment), and filtering (e. g. blur and sharpness). These methods are commonly employed to increase the diversity of the training dataset, helping the model to generalize better to new data.
Image enhancement data augmentation: this method comprised a limited selection of standard augmentations such as geometric transformation in combination with image enhancement-based augmentations. For image enhancement data augmentations, we used the proprietary software tool to augment the training set with images across the complete spectrum of available enhancement settings.
Examples of several data augmentations are given in Fig. 3 s .
Development of CADe and CADx systems
For every dataset (CADe and CADx), two separate CAD systems were developed. Each was trained using either standard or image enhancement-based data augmentation. During training, a patient-based random split was executed, allocating 60 %, 20 %, and 20 % for training, validation, and testing, respectively. A ResNet-50 encoder, initialized with ImageNet pre-trained weights, was employed as this is a widely accepted and commonly used architecture. For all CAD systems, the operating threshold was optimized on its corresponding validation set comprising the same enhancement setting as the training set. Further details are given in the Supplementary material (see Algorithm development).
Evaluation of CAD systems
To evaluate these CAD systems, we used the following test sets.
Original test set: this dataset comprised the CAD systems’ original, unaltered test set images, which were acquired using limited variability in enhancement settings.
Simulated test sets: the software tool was then used to convert the original test set to generate new test sets with different enhancement settings. As the tool offers 16 different enhancement settings (i. e. A1-A8, B1-B8), this resulted in 16 simulated test sets per CAD system.
Outcome measures
The study outcomes were:
baseline performance (sensitivity and specificity) of CADe and CADx, trained with standard or image enhancement-based data augmentation, on their original test set;
performance variability (mean and range of sensitivity and specificity) of CADe and CADx, trained with standard or image enhancement-based data augmentation, across simulated test sets.
Post hoc analysis
The original training sets of CADe and CADx were acquired using moderate enhancement settings (A3 and A5). In a post hoc analysis, we investigated more extreme enhancement settings. We analyzed scenarios where a CAD system is trained on one end of the spectrum of enhancement settings (e. g. A1) and applied in clinical practice with settings from the opposite end (e. g. A8).
Statistical analysis
Sensitivity and specificity were chosen as the primary performance metrics in this study. Sensitivity represents the proportion of true positives among all positive cases, while specificity denotes the proportion of true negatives among all negative cases. In the CADe dataset for Barrett’s neoplasia detection, positive cases were defined as images showing neoplastic tissue, while negative cases were nondysplastic Barrett’s images. In the CADx dataset for colorectal polyp characterization, positives included images of neoplastic polyps (adenomas and sessile serrated lesions), and negatives consisted of non-neoplastic polyps (e. g. hyperplastic polyps).
We report sensitivity and specificity for the original test sets to establish baseline performance under the original enhancement settings. Confidence intervals were calculated using the Wilson method. To evaluate variability in performance across different enhancement settings, we tested the systems on an array of simulated test sets generated with varied enhancement settings. For these simulated test sets, we present the median sensitivity and specificity across all settings, alongside the full range of these values to illustrate the extent of performance variability. For a more granular view of this distribution, we provide receiver operating characteristic scatterplots displaying results for each individual simulated test set. To test whether image enhancement-based data augmentation decreased the variability of sensitivity and specificity, we performed a one-sided nonparametric Mood test, using the R package “coin” (R Foundation for Statistical Computing, Vienna, Austria).
Results
Performance of CAD systems using standard data augmentation
For Barrett’s neoplasia detection, the CADe system trained with standard data augmentation displayed a baseline performance of 90 % sensitivity and 89 % specificity on its original test set. On the simulated test sets comprising a wide variety of enhancement settings, the CADe system reached median sensitivity and specificity of 87 % and 89 %, respectively. Performance varied substantially between individual sets. Sensitivity ranged between 83 % and 92 % (Δ 9 %) and specificity ranged between 84 % and 91 % (Δ 7 %).
The CADx system for colorectal polyps, trained using standard data augmentation, reached a baseline performance of 79 % sensitivity and 63 % specificity on its original test set. For the simulated test sets, the CADx system displayed median sensitivity and specificity of 79 % and 59 %, respectively. Similarly to CADe, performance variation was clear. Sensitivity ranged from 78 % to 85 % (Δ 7 %) and specificity ranged from 45 % to 63 % (Δ 18 %).
An example case of substantial performance variability across different enhancement settings is given in [Fig. 4 ].
Fig. 4 Example case of performance variability of the computer-aided detection system trained with standard data augmentation and image enhancement-based data augmentation. Image enhancement-based data augmentation results in substantially more stable predictions.
Performance of CAD systems using image enhancement-based data augmentation
After retraining with image enhancement-based data augmentation, the CADe system showed a baseline performance of 90 % sensitivity and 90 % specificity on the original test set. On the simulated test sets, median sensitivity and specificity were 90 % and 90 %, respectively. Performance variability was significantly lower compared with standard data augmentation. Sensitivity ranged from 89 % to 91 % (Δ 2 %; P < 0.001), and specificity ranged from 90 % to 91 % (Δ 1 %; P = 0.003).
The retrained CADx system displayed a baseline performance of 78 % sensitivity and 63 % specificity on the original test set. On simulated test sets, the retrained CADx system reached median sensitivity and specificity of 79 % and 60 %, respectively. Performance variability was limited, with sensitivity ranging from 78 % to 80 % (Δ 2 %; P = 0.03) and specificity ranging from 55 % to 63 % (Δ 8 %; P = 0.190).
All results are summarized in [Table 1 ] and illustrated in [Fig. 5 ]. Results on individual test sets are given in Table 1 s .
Table 1
Results of computer-aided detection and diagnosis systems trained with standard and image enhancement-based data augmentation.
AI application
Original test set
Simulated test sets
Median
Range (min–max)
P value
CADe
Sensitivity, %
90
87
9 (83–92)
< 0.001
90
90
2 (89–91)
Specificity, %
89
89
7 (84–91)
0.003
90
90
1 (90–91)
CADx
Sensitivity, %
79
79
7 (78–85)
0.03
78
79
2 (78–80)
Specificity, %
63
59
18 (45–63)
0.19
63
60
8 (55–63)
CADe, computer-aided detection; CADx, computer-aided diagnosis.
Fig. 5 Performance variability of both computer-aided detection (CADe) and computer-aided diagnosis (CADx) using standard or image enhancement-based data augmentation. Each dot represents the performance of the respective CAD system on a test set comprising one specific enhancement setting.
Post hoc analysis When the CADe system was trained with standard data augmentation using a different, more extreme enhancement setting (i. e. A1, A8, B1, or B8) and tested on all simulated test sets, performance variability increased even further. For instance, the CADe system trained solely on images with enhancement setting A8 displayed sensitivity scores between 66 % and 89 % (Δ 23 %). For CADx, the system trained on images with setting B1 reached specificity scores ranging between 31 % and 61 % (Δ 30 %). All post hoc results are displayed in Table 2 s , Table 3 s , and Fig. 4 s .
Discussion
Current endoscopy systems offer a wide range of post-processing enhancement settings that endoscopist may use to adjust image characteristics such as color, texture, and contrast. We investigated whether these settings, while often subtle to the human eye, affect the performance of endoscopic AI systems. We tested this on two commonly used endoscopic AI applications, detection of early Barrett’s neoplasia and characterization of diminutive colorectal polyps, and found a remarkable impact of enhancement settings on AI performance.
When we trained and tested both CAD systems on their original datasets without changing enhancement settings of the original images, the systems performed as expected. However, when we evaluated the systems’ performances on test sets with different enhancement settings, performance varied substantially. For instance, the CADe system for detection of Barrett’s neoplasia showed sensitivity varying between 83 % and 92 %: a 9 % absolute difference in neoplasia detection. For the CADx system characterizing colorectal polyps, the variability in performance was even more profound, with specificity scores differing up to 18 %. When we trained the systems with the extremes of the enhancement settings, performance variability increased even more. For example, a CADe system trained only on images with setting B1 showed specificity between 55 % and 88 % depending on the enhancement setting of the test sets.
These findings are significant, as all major manufacturers of endoscopy systems offer a variety of company-specific enhancement modes. Often, these settings are preconfigured by the hospital’s technical services team. Most endoscopist do not adjust these settings further and many may not even be aware of them. As there is no clear default mode, these settings may differ widely between hospitals or even between individual endoscopy suites. Meanwhile, AI systems are typically neither trained nor optimized to perform consistently across the full range of these settings, which may contribute to the lack of generalizability observed when AI systems are deployed outside of the centers where they were initially developed [11 ]
[12 ]
[13 ]. For example, if an AI system trained on data with high-contrast settings is deployed in a clinic where low-contrast settings are standard, it may fail to detect subtle lesions due to the lower visual contrast, potentially leading to missed diagnoses. Conversely, other enhancement settings might amplify the AI system’s sensitivity, causing it to often flag non-neoplastic mucosa as suspicious. This could lead to a high false-positive rate, creating “alert fatigue” in clinicians.
Fortunately, there may be an effective solution to this issue. As the main problem is the lack of heterogeneity of enhancement settings in the training data, an intuitive solution is to introduce the complete spectrum of enhancement settings into the training data. Enhancement settings are based on post-processing transformations. Therefore, this does not require collection of additional real-world clinical data. Rather, the solution may be found with image enhancement-based data augmentation. Data augmentation is a standard step in training AI algorithms in endoscopy. It generally involves making slight modifications to the original images in the training set – such as rotations, color adjustments, or scaling – to artificially expand the dataset and introduce a wider variety of training examples. Image enhancement-based data augmentation involves restricting these augmentations to endoscopy-specific settings, in this case, the variability induced by enhancement settings. Using a proprietary software tool, we duplicated all images in the CAD systems’ training sets to create new images comprising all different enhancement settings. This approach proved to be successful, as both the CADe and CADx systems displayed significantly more stable performance: CADe sensitivity and specificity differences were limited to 2 % and 1 %, while CADx differences were 2 % and 8 %, respectively.
We recognize that this study has some limitations. First, this study is limited by its ex vivo design using simulated still images. Images were converted to different enhancement settings after initial acquisition and storage, using a proprietary software tool that exactly replicates these settings. Although a prospective collection of matched images with different enhancement settings would be ideal, it is impractical, if not impossible, to capture matched pairs across all enhancement settings for each image. Additionally, varying capture devices and compression techniques could introduce further variability, potentially distorting results. In contrast, our ex vivo approach allowed for a standardized, paired analysis across 16 enhancement settings. Moreover, as shown in Fig. 2 s , the converted images were nearly indistinguishable from the original data, supporting the validity of this approach. Second, despite the fact that the evaluated datasets were aimed at different endoscopic applications, the datasets originated from the same manufacturer and endoscope series. Other endoscopy systems were not investigated. Yet it is conceivable that AI systems on these platforms will suffer from similar issues. Third, the current proposed method for image enhancement-based data augmentation remains a quick fix and may not be sustainable long term. Ideally, a more generic method would be developed that is applicable to all current post-processing settings and future settings yet to be developed. Fourth, this study is specifically aimed at post-processing image enhancement. There may be other sources of domain shift that can form a similar threat to model generalizability, such as other virtual chromoendoscopy techniques, differing generations of endoscopes, and quality of the endoscopic imaging procedure.
In conclusion, this study highlights the impact of different enhancement settings on performance variability of endoscopic AI systems. This should be taken into account during future development and implementation of AI in endoscopy. Collecting more diverse data during the development phase may offer the most direct solution, but specific data augmentation techniques to simulate enhancement setting variability or generative AI solutions could be pragmatic alternatives and deserve further investigation.
