Deep Learning for Detecting and Subtyping Renal Cell Carcinoma on Contrast-Enhanced CT Scans Using 2D Neural Network with Feature Consistency Techniques

Abstract

Objective The aim of this study was to explore an innovative approach for developing deep learning (DL) algorithm for renal cell carcinoma (RCC) detection and subtyping on computed tomography (CT): clear cell RCC (ccRCC) versus non-ccRCC using two-dimensional (2D) neural network architecture and feature consistency modules.

Materials and Methods This retrospective study included baseline CT scans from 196 histopathologically proven RCC patients: 143 ccRCCs and 53 non-ccRCCs. Manual tumor annotations were performed on axial slices of corticomedullary phase images, serving as ground truth. After image preprocessing, the dataset was divided into training, validation, and testing subsets. The study tested multiple 2D DL architectures, with the FocalNet-DINO demonstrating highest effectiveness in detecting and classifying RCC. The study further incorporated spatial and class consistency modules to enhance prediction accuracy. Models' performance was evaluated using free-response receiver operating characteristic curves, recall rates, specificity, accuracy, F1 scores, and area under the curve (AUC) scores.

Results The FocalNet-DINO architecture achieved the highest recall rate of 0.823 at 0.025 false positives per image (FPI) for RCC detection. The integration of spatial and class consistency modules into the architecture led to 0.2% increase in recall rate at 0.025 FPI, along with improvements of 0.1% in both accuracy and AUC scores for RCC classification. These enhancements allowed detection of cancer in an additional 21 slices and reduced false positives in 126 slices.

Conclusion This study demonstrates high performance for RCC detection and classification using DL algorithm leveraging 2D neural networks and spatial and class consistency modules, to offer a novel, computationally simpler, and accurate DL approach to RCC characterization.

Keywords

renal cell carcinoma - deep learning - classification - detection - F1 score

Introduction

Renal cell carcinoma (RCC) represents a significant portion of cancer in adults, manifesting in a variety of histological patterns.[1] Of these, the most prevalent and aggressive kind is clear cell RCC (ccRCC).[2] It is of paramount importance to distinguish among RCC subtypes and determine their grades for patient prognosis and to customize treatment employing the latest therapeutic options, including inhibitors of tyrosine kinase and vascular endothelial growth factor.[3] [4] [5] Even in nonmetastatic RCC cases where surgery is the standard irrespective of the type of RCC, preoperative subtype classification still holds clinical relevance as it can guide surgical planning, perioperative care, and risk stratification, especially for aggressive subtypes like ccRCC. However, current diagnostic methods, including renal mass biopsies, are invasive and can lead to erratic results.[6] [7] The use of contrast enhancement patterns in multiphasic computed tomography (CT) and magnetic resonance imaging (MRI) for RCC characterization is also vulnerable to subjectivity and overlapping imaging results.[8] [9] Consequently, there is a critical demand for dependable, noninvasive imaging biomarkers that can accurately classify and grade RCC, thereby improving diagnostic precision and patient management.

Recent advances in deep neural networks (DNN) offer promising avenues for enhancing RCC detection and classification through cross-sectional imaging.[10] [11] [12] [13] Previous research has shown that DNN models can accurately detect and subtype RCC, highlighting their potential role in complex medical image processing and patient prognostication.[12] [14] [15] [16] However, the landscape of RCC characterization and its management implications continues to evolve, necessitating further research. Challenges such as interobserver variability in radiological interpretations and the overlapping image features of RCC subtypes call for more refined and reliable diagnostic tools with feasibility for easy integration into radiological practice.

With this background, our study aimed to investigate a novel method of training and constructing a deep learning (DL) algorithm for RCC detection and subtyping (ccRCC vs. non-ccRCC) utilizing two-dimensional (2D) neural network architectures and feature consistency modules.

Materials and Methods

CT Dataset Preparation

The Institute Ethics Committee approved this retrospective study. We reviewed the pathology report database of our tertiary care hospital for patients with histopathologically proven RCC in nephrectomy specimens, between January 2016 and November 2021. Using these pathology reports as the gold standard, RCC patients were categorized into ccRCC and non-ccRCC (including papillary and chromophobe RCCs). Abdominal CT scans corresponding to these patients were obtained from the hospital's Picture Archiving and Communication System (PACS) using their unique hospital identification numbers. Patients without a preoperative baseline scan in the PACS were excluded from the study. Thus, 196 RCC patients who had preoperative abdominal CT at our facility and thereafter surgery were included in the analysis. In this cohort, 143 were ccRCCs and 53 were non-ccRCCs (42 papillary RCCs and 11 chromophobe RCCs). The authors confirm the availability of, and access to, all original data reported in this study

Computed Tomography Protocol

A dual-source dual-energy 2 × 128 section multidetector CT scanner (Somatom Definition Flash, Siemens Healthineers, Germany) was used for multiphase CT imaging. A noncontrast CT (NCCT) scan is the first step in the standardized renal mass protocol used in our department. Corticomedullary (CM) and nephrographic phase images are then obtained at 25 to 30 and 60 to 70 seconds, respectively, following intravenous injection of 100 mL of iodinated contrast (Omnipaque 350, GE Healthcare) through a peripheral line at 3 to 5 mL/s. Delayed excretory phase images are obtained after 4 to 5 minutes in patients with suspicion of renal collecting system involvement. The retrieved CT scans were made anonymous by removing the Digital Imaging and Communications in Medicine (DICOM) metadata and reassigning them with a new unique ID number for the study. The CM phase images were selected for model training and testing due to their high sensitivity for RCC detection and characterization per established CT protocols, with other contrast phases less consistently available due to protocol variability in the retrospective data.

Manual Tumor Annotation

The CM phase images from the retrieved CT abdomen scans were converted into axial slices with a uniform resolution of 0.7 frames/mm and saved as individual Joint Photographic Experts Group (JPEG) files for each patient. One radiologist (with 9 years of body imaging experience) manually annotated the tumors on these images. Each axial slice was individually examined by the radiologist for the presence of renal tumor. Using the freely accessible open-source “LabelMe” software, manual tumor marking was done by the radiologist using rectangular bounding box placement. The manual labels were subsequently reviewed for correctness by another radiologist (with 20 years of experience in genitourinary imaging). This manual bounding box placement by the radiologist was used as the ground truth for RCC detection during the training and testing of the networks.

Image Preprocessing and Data Stratification

The collected image dataset underwent preprocessing to ensure consistency and quality. This included standardizing image dimensions by rescaling it to 224 × 224, normalizing pixel intensities between 0 and 1 by employing the mean-max normalization algorithm, and applying standardized augmentation techniques such as random rotations, flips, translations, changes in brightness, and contrast for enhanced diversity. Following preprocessing, the data were partitioned into three cohorts: training, validation, and testing. The training cohort, comprising 80% of the data, was utilized to train the model. A separate validation cohort, constituting 10% of the data, facilitated fine-tuning the model parameters and guarded against overfitting. The remaining 10% formed the testing cohort, reserved for assessing the model's performance on unseen data. Care was taken to maintain a balanced distribution of tumor subtypes and to ensure that the data pertaining to one patient was kept in a single cohort to avoid label leaking. Thus, 156 scans (115 ccRCCs and 41 non-ccRCCs) were used for training, while validation and testing included 20 scans each (14 ccRCCs and 6 non-ccRCCs per cohort).

Working of Deep Neural Network Architecture

In this study, we approached detecting and classifying RCC on CT as an image-based object detection problem. We experimented with several 2D deep neural network–based object detection methods for this method. The FocalNet-DINO design was found to be the most efficient of these. [Fig. 1] depicts the architecture of the FocalNet-DINO-based object detection system. This intricate architecture involved dividing the input images into smaller patches, which were then processed through a sophisticated ResNet-based backbone to generate multiscale feature maps. Each feature map was refined using a series of encoder blocks equipped with focal-modulation-based self-attention mechanisms. This enabled efficient and detailed interaction between the features of each patch, resulting in improved refinement. The enhanced multiscale feature map generated by the encoder layer was subjected to a query selection framework that employed a gated aggregation mechanism. This framework was useful in creating prospective tumor locations, which were then refined during the decoder step. The decoder incorporated a deformable cross-attention module, a key component that focused on the feature maps corresponding to every query, facilitating accurate detection and classification of the tumor regions. The classification for each axial section of the CT scan was determined based on the class label of the most confident bounding box in that slice.

Fig. 1 Architecture of the FocalNet-DINO-based object detection algorithm. CNN, convolutional neural network.

Incorporating Spatial and Class Consistency Modules in the DL Algorithm

A novel component of our study was the 2D network's independent inference on each axial section of the CT scan. To refine the predictions, we took advantage of the intrinsic spatial consistency across axial sections. To ensure spatial coherence, we altered the confidence levels of our predictions based on the spatial overlaps of three subsequent frames. Furthermore, the mode of class labels across these frames was used to calculate each prediction's class label. Finally, the axial slices with annotated bounding boxes and classification labels were combined into DICOM files for the final result. A schematic of the suggested RCC detection and classification model's pipeline is presented in [Fig. 2].

Fig. 2 Overall pipeline of the proposed renal cell carcinoma (RCC) detection and classification model. CT, computed tomography.

Testing of the Networks and Data Analysis

We used the free-response receiver operating characteristic (FROC) curve and recall rates at varying false-positive tolerances to evaluate the performance of various network architectures for RCC detection. The FROC curve offers a detailed graphical representation of sensitivity/recall values against false-positives per image (FPI). Our criterion for a true positive prediction was based on the positioning of the predicted bounding box's center within the ground truth bounding box, aligning with the medico-vision community standards. For the classification task, we assessed the performance of the DL models using key metrics such as accuracy, F1 scores, precision, recall, and area under the curve (AUC) scores.

Results

Patient Demography and Tumor Characteristics

The mean age of the 143 patients who had ccRCCs was 67.5 years (range: 51–72 years), and the average size of their tumor was 6.5 cm (standard deviation: ± 2.4 cm). There were 81 men and 62 women in the group. The average age of patient with non-ccRCCs (n = 53) was 64.7 years (range: 48–70 years), with a male-to-female ratio of 31:22 and an average tumor size of 6.4 cm (standard deviation: ± 2.3 cm).

Renal Cell Carcinoma Detection

We compared the performance of the proposed algorithm against the ground truth for its detection performance. A predicted bounding box was considered to be valid if the prediction confidence was greater than 0.5. We measured the recall rate of the proposed algorithm against various tolerance levels of false positives per image (FPI) and the results of the same can be observed in [Table 1]. Similarly, the corresponding FROC curves can be found in [Fig. 3]. The FocalNet-DINO architecture achieved a recall rate of 0.823 with 0.025 FPI, outperforming the next best-performing network, DN-DEF, by 0.9%. The addition of spatial and class consistency modules to our proposed design improved the performance even more, with a 0.2% gain in recall rate at 0.025 FPI. These results, particularly the recall rate of 0.825 at 0.025 FPI, provide strong evidence of the model's high accuracy in RCC detection, underscored by the increased sensitivity achieved through spatial and class consistency modules. [Fig. 4] shows visualizations of the ground truth labels and predictions by various networks for different patients. The proposed feature consistency strategies enabled the detection of cancer in an extra 21 slices of the test dataset, increasing the model's sensitivity. In addition, the spatial and class consistency module demonstrated its effectiveness in improving prediction accuracy by reducing the number of false positives in 126 slices.

Fig. 3 Free-response receiver operating characteristic (FROC) curve comparing various state-of-the-art object detection architectures for renal cell carcinoma detection with the proposed model.

Fig. 4 Visualizations of the ground truth (GT) bounding boxes (blue) and predictions (green) for various tested algorithms for different patients.

Table 1 Comparison of performance of various deep neural network–based object detection architectures for RCC detection and classification

Model name	Venue	Detection results							Classification results
		R@0.025	R@0.05	R@0.1	R@0.3	R@0.5	R@1	R@5	Acc	F1 score	AUC score
DAB-DETR	ICLR'22	0.763	0.845	0.884	0.915	0.927	0.937	0.969	0.942	0.874	0.976
DAB DEF.	ICLR'22	0.786	0.849	0.878	0.927	0.942	0.956	0.977	0.942	0.875	0.977
DN-DETR	CVPR'22	0.81	0.867	0.906	0.93	0.935	0.942	0.968	0.945	0.883	0.981
DN DEF.	CVPR'22	0.815	0.867	0.898	0.933	0.944	0.964	0.990	0.950	0.892	0.98
FocalNet-DINO	NeurIPS'22	0.823	0.893	0.929	0.956	0.964	0.979	0.992	0.950	0.893	0.985
Proposed		0.825	0.895	0.930	0.958	0.964	0.979	0.992	0.951	0.893	0.986

Abbreviations: AUC, area under the curve; R, recall; RCC, renal cell carcinoma.

Renal Cell Carcinoma Classification

The RCC class (ccRCC vs. non-ccRCC) of a CT slice was determined according to the label of the valid bounding box with the highest confidence. A bounding box was considered valid if its confidence was above a threshold of 0.5. If no valid bounding boxes were detected in a slice, that slice was considered normal. The results of the classification approach (sensitivity, specificity, F1 score, accuracy, and AUC score) are included in [Table 1]. The precision recall curve (PR curve) and the AUC curve are shown in [Figs. 5] and [6], respectively. The FocalNet-DINO demonstrated an accuracy of 0.950, an F1 score of 0.893, and an AUC score of 0.985 for classification of RCC (ccRCC vs. non-ccRCC). The addition of the spatial and class consistency modules led to an improvement of 0.1% in the AUC score.

Fig. 5 Precision recall (PR) curve comparing performance of various state-of-the-art object detection architectures for renal cell carcinoma classification with the proposed model.

Fig. 6 Area under the curve (AUC) comparing classification performance of various state-of-the-art object detection architectures with the proposed model.

Discussion

In our study, we showed high accuracy of computationally efficient 2D neural networks for detection and classification of RCC on CT images by approaching it as an image-based object detection task (best performance for FocalNet-DINO) showing further improvement of the performance by incorporation of spatial and class consistency modules in our proposed algorithm.

We used 2D neural networks for RCC diagnosis on CT images. While three-dimensional (3D) networks have the potential to offer a more comprehensive analysis by considering the volumetric context and spatial relationships within the entire tumor, these networks treat the entire CT scan of a patient as a single training example, resulting in requirement of a substantially large dataset for training.[17] [18] In scenarios with limited data, as observed in our work, 3D convolutional neural networks (CNNs) are prone to overfitting, rendering them practically unusable.[19] In contrast, 2D networks use individual slices of the CT scan as training samples. This approach allows for training with a limited number of patient data. Additionally, due to the considerable diversity among CT slices, 2D CNNs exhibit a reduced likelihood of overfitting.[19] Furthermore, 2D CNNs are more computationally efficient, resulting in quicker processing times. This efficiency is particularly pertinent in environments with limited computational resources. Notably, 2D neural networks demonstrate an enhanced ability to handle data variability, require less data for training and validation, and offer faster inference times compared to their 3D counterparts.[18]

Since the proposed 2D neural network analyses each slice of the CT scan independently, there is a high risk of spurious misclassification, false-positive box prediction, or deviated box prediction. This error resembles the salt and pepper noise of the images.[20] To mitigate the above-mentioned effects, we incorporated spatial consistency and class consistency modules. We analyze the spatial overlap of the predicted bounding boxes across the slices and replace the bounding box of any slice with the median of the bounding boxes in its N-5 neighborhood (5 previous and next frames). This removes the false positives and adjusts the deviated bounding boxes. Similarly, we incorporate the class consistency module to replace the class of the detected bounding box as a median of the classes predicted in the N-5 neighborhood. This reduces the risk of misclassification and establishes consistency in the prediction throughout the patient's CT scan. Incorporating these modules ensures the model is practically usable in the radiology workflow.

Previous research has investigated the use of DL models for image analysis of renal tumors, including tumor detection, segmentation, and subclassification, with promising results.[21] [22] [23] [24] However, the majority of prior investigations have focused on distinguishing between benign and malignant renal tumors and cysts.[25] Moreover, in these previous studies, the DL methods for RCC classification required manual tumor localization using tumor regions marked by a radiologist.[24] [26] [27] To the best of our knowledge, two studies report the end-to-end use of DL for both RCC detection and its subtyping into histopathological variants.[11] [13] Both these studies used 3D CNN–based models and found high accuracy of these models for RCC classification, comparable to the radiologists. Our study further contributes to the literature by demonstrating similar high performance for RCC detection and classification by an ingenious approach using 2D neural networks and feature consistency modules. Furthermore, this study adds to the expanding body of evidence supporting the use of DNN in medical imaging, notably for accurately and efficiently classifying complicated malignancies such as RCC. Such differentiation of the more aggressive ccRCC from other non-ccRCC tumor subtypes has the potential to improve prognosis and enable personalized treatments with targeted therapies like tyrosine kinase and vascular endothelial growth factor inhibitors. Given the limitations of the current diagnostic methods in reliably achieving this differentiation, the use of DL models offers a promising noninvasive alternative for accurate RCC subtyping and grading, potentially improving patient outcomes and treatment precision.

Our study had certain limitations. Our study relied on postoperative pathology reports, which limited our analysis to nonmetastatic cases, potentially affecting the generalizability of our findings. Metastatic RCCs, especially non-ccRCC subtypes, can exhibit heterogeneity, necrosis, and other features distinct from nonmetastatic cases, potentially affecting the DL features extracted. In the cases where the model failed to detect or classify RCC accurately, contributing factors likely included low tumor-to-tissue contrast, heterogeneous or necrotic tumor regions, and reduced spatial coherence in certain slices. A comprehensive failure analysis could help identify specific patterns in misclassifications and inform future model improvements. Due to the small individual sample sizes for chromophobe and papillary RCC subgroups, a meaningful analysis of subgroup differences within the non-ccRCC class could not be performed. The long-term impact of integrating our method into clinical practice remains to be seen. Future research should look into the practical usefulness of this method in clinical settings. Integrating our method with other diagnostic modalities, such as MRI or ultrasound, could improve the accuracy and usefulness of RCC classification.

Conclusion

In conclusion, in this study, we leverage 2D neural networks and spatial and class consistency modules to offer a novel, computationally simpler and accurate DL approach to RCC detection and classification on cross-sectional imaging.

Conflict of Interest

None declared.

• References

• 1 Moch H, Cubilla AL, Humphrey PA, Reuter VE, Ulbright TM. The 2016 WHO classification of tumours of the urinary system and male genital organs-part A: renal, penile, and testicular tumours. Eur Urol 2016; 70 (01) 93-105
• 2 Capitanio U, Cloutier V, Zini L. et al. A critical assessment of the prognostic value of clear cell, papillary and chromophobe histological subtypes in renal cell carcinoma: a population-based study. BJU Int 2009; 103 (11) 1496-1500
• 3 Patard JJ, Leray E, Rioux-Leclercq N. et al. Prognostic value of histologic subtypes in renal cell carcinoma: a multicenter experience. J Clin Oncol 2005; 23 (12) 2763-2771
• 4 Lee CH, Motzer R. Combination VEGFR/immune checkpoint inhibitor therapy: a promising new treatment for renal cell carcinoma. Br J Cancer 2018; 119 (08) 911-912
• 5 Atkins MB, Tannir NM. Current and emerging therapies for first-line treatment of metastatic clear cell renal cell carcinoma. Cancer Treat Rev 2018; 70: 127-137
• 6 Abel EJ, Carrasco A, Culp SH. et al. Limitations of preoperative biopsy in patients with metastatic renal cell carcinoma: comparison to surgical pathology in 405 cases. BJU Int 2012; 110 (11) 1742-1746
• 7 Abel EJ, Culp SH, Matin SF. et al. Percutaneous biopsy of primary tumor in metastatic renal cell carcinoma to predict high risk pathological features: comparison with nephrectomy assessment. J Urol 2010; 184 (05) 1877-1881
• 8 Karlo CA, Di Paolo PL, Chaim J. et al. Radiogenomics of clear cell renal cell carcinoma: associations between CT imaging features and mutations. Radiology 2014; 270 (02) 464-471
• 9 Young JR, Margolis D, Sauk S, Pantuck AJ, Sayre J, Raman SS. Clear cell renal cell carcinoma: discrimination from other renal cell carcinoma subtypes and oncocytoma at multiphasic multidetector CT. Radiology 2013; 267 (02) 444-453
• 10 Xu L, Yang C, Zhang F. et al. Deep learning using CT images to grade clear cell renal cell carcinoma: development and validation of a prediction model. Cancers (Basel) 2022; 14 (11) 2574
• 11 Yao N, Hu H, Chen K. et al. A robust deep learning method with uncertainty estimation for the pathological classification of renal cell carcinoma based on CT images. J Imaging Inform Med 2024 (e-pub ahead of print). https://doi.org/10.1007/s10278-024-01276-7
• 12 Wang Z, Zhang X, Wang X. et al. Deep learning techniques for imaging diagnosis of renal cell carcinoma: current and emerging trends. Front Oncol 2023; 13: 1152622
• 13 Uhm KH, Jung SW, Choi MH. et al. Deep learning for end-to-end kidney cancer diagnosis on multi-phase abdominal computed tomography. NPJ Precis Oncol 2021; 5 (01) 54
• 14 Amador S, Beuschlein F, Chauhan V. et al. Deep learning approaches applied to image classification of renal tumors: a systematic review. Arch Comput Methods Eng 2024; 31 (02) 615-622
• 15 Dai C, Xiong Y, Zhu P. et al. Deep learning assessment of small renal masses at contrast-enhanced multiphase CT. Radiology 2024; 311 (02) e232178
• 16 Mahootiha M, Qadir HA, Bergsland J, Balasingham I. Multimodal deep learning for personalized renal cell carcinoma prognosis: integrating CT imaging and clinical data. Comput Methods Programs Biomed 2024; 244: 107978
• 17 Yang YQ, Guo YX, Xiong JY. et al. Swin3d: a pretrained transformer backbone for 3d indoor scene understanding. arXiv preprint arXiv:2304.06906. 2023
• 18 Zhang Y, Shi L, Wu Y, Cheng K, Cheng J, Lu H. Gesture recognition based on deep deformable 3D convolutional neural networks. Pattern Recognit 2020; 107: 107416
• 19 Ying X. An overview of overfitting and its solutions. J Phys Conf Ser 2019; 1168 (02) 022022
• 20 Zheng Q, Delingette H, Duchateau N, Ayache N. 3-D consistent and robust segmentation of cardiac images by deep learning with spatial propagation. IEEE Trans Med Imaging 2018; 37 (09) 2137-2148
• 21 Tanaka T, Huang Y, Marukawa Y. et al. Differentiation of small (≤ 4 cm) renal masses on multiphase contrast-enhanced CT by deep learning. AJR Am J Roentgenol 2020; 214 (03) 605-612
• 22 Oberai A, Varghese B, Cen S. et al. Deep learning based classification of solid lipid-poor contrast enhancing renal masses using contrast enhanced CT. Br J Radiol 2020; 93 (1111) 20200002
• 23 Zabihollahy F, Schieda N, Krishna S, Ukwatta E. Automated classification of solid renal masses on contrast-enhanced computed tomography images using convolutional neural network with decision fusion. Eur Radiol 2020; 30 (09) 5183-5190
• 24 Han S, Hwang SI, Lee HJ. The classification of renal cancer in 3-phase CT images using a deep learning method. J Digit Imaging 2019; 32 (04) 638-643
• 25 Amador S, Beuschlein F, Chauhan V. et al. Deep learning approaches applied to image classification of renal tumors. a systematic review. Arch Computat Methods Eng 2024; 31: 615-622
• 26 Zheng Y, Wang S, Chen Y, Du HQ. Deep learning with a convolutional neural network model to differentiate renal parenchymal tumors: a preliminary study. Abdom Radiol (NY) 2021; 46 (07) 3260-3268
• 27 Zuo T, Zheng Y, He L. et al. Automated classification of papillary renal cell carcinoma and chromophobe renal cell carcinoma based on a small computed tomography imaging dataset using deep learning. Front Oncol 2021; 11: 746750

Publication History

Article published online:
11 December 2024