Performance of AI Approaches for COVID-19 Diagnosis Using Chest CT Scans: The Impact of Architecture and Dataset

Abstract

Purpose

AI is emerging as a promising tool for diagnosing COVID-19 based on chest CT scans. The aim of this study was the comparison of AI models for COVID-19 diagnosis. Therefore, we: (1) trained three distinct AI models for classifying COVID-19 and non-COVID-19 pneumonia (nCP) using a large, clinically relevant CT dataset, (2) evaluated the models’ performance using an independent test set, and (3) compared the models both algorithmically and experimentally.

Materials and Methods

In this multicenter multi-vendor study, we collected n=1591 chest CT scans of COVID-19 (n=762) and nCP (n=829) patients from China and Germany. In Germany, the data was collected from three RACOON sites. We trained and validated three COVID-19 AI models with different architectures: COVNet based on 2D-CNN, DeCoVnet based on 3D-CNN, and AD3D-MIL based on 3D-CNN with attention module. 991 CT scans were used for training the AI models using 5-fold cross-validation. 600 CT scans from 6 different centers were used for independent testing. The models’ performance was evaluated using accuracy (Acc), sensitivity (Se), and specificity (Sp).

Results

The average validation accuracy of the COVNet, DeCoVnet, and AD3D-MIL models over the 5 folds was 80.9%, 82.0%, and 84.3%, respectively. On the independent test set with n=600 CT scans, COVNet yielded Acc=76.6%, Se=67.8%, Sp=85.7%; DeCoVnet provided Acc=75.1%, Se=61.2%, Sp=89.7%; and AD3D-MIL achieved Acc=73.9%, Se=57.7%, Sp=90.8%.

Conclusion

The classification performance of the evaluated AI models is highly dependent on the training data rather than the architecture itself. Our results demonstrate a high specificity and moderate sensitivity. The AI classification models should not be used unsupervised but could potentially assist radiologists in COVID-19 and nCP identification.

Key Points

• This study compares AI approaches for diagnosing COVID-19 in chest CT scans, which is essential for further optimizing the delivery of healthcare and for pandemic preparedness.

• Our experiments using a multicenter, multi-vendor, diverse dataset show that the training data is the key factor in determining the diagnostic performance.

• The AI models should not be used unsupervised but as a tool to assist radiologists.

Citation Format

• Jaiswal A, Fervers P, Meng F et al. Performance of AI Approaches for COVID-19 Diagnosis Using Chest CT Scans: The Impact of Architecture and Dataset. Rofo 2026; 198: 185–198

Zusammenfassung

Ziel

Aktuell existieren verschiedenste Künstliche Intelligenz (KI)-Modelle zur Detektion und Klassifikation von Pneumonien in Thorax-CTs, aber unabhängige Vergleiche fehlen meist. In dieser Studie haben wir (1) drei verschiedene KI-Modelle zur Klassifizierung von COVID-19- und Nicht-COVID-19-Pneumonien (nCP) anhand eines klinisch relevanten CT-Datensatzes trainiert, (2) die Leistung der Modelle anhand eines unabhängigen Testsatzes bewertet und (3) die Modelle sowohl algorithmisch als auch experimentell verglichen.

Materialien und Methoden

In dieser multizentrischen, retrospektiven Studie haben wir insgesamt 1591 Thorax-CTs von COVID-19- (n=762) und nCP (n=829)-Patienten aus China und Deutschland zusammengestellt; in Deutschland wurden die CT-Daten von 3 RACOON-Standorten eingeschlossen. Es wurden 3 open-source KI-Modelle mit unterschiedlichen Architekturen trainiert und validiert: COVNet basierend auf 2D-CNN, DeCoVnet basierend auf 3D-CNN, und AD3D-MIL basierend auf 3D-CNN mit Attention-Modul. Die Performance der Modelle wurde anhand von Genauigkeit (Acc), Sensitivität (Se) und Spezifität (Sp) bewertet.

Ergebnisse

Die durchschnittliche Validierungsgenauigkeit der Modelle COVNet, DeCoVnet und AD3D-MIL über die 5-Fach-Validierung im Training mit n=991 CTs betrug 80,9%, 82,0% bzw. 84,3%. Auf dem unabhängigen Testsatz mit n=600 CTs lieferte COVNet: Acc=76,6%, Se=67,8%, Sp=85,7%; DeCoVnet: Acc=75,1%, Se=61,2%, Sp=89,7%; und AD3D-MIL: Acc=73,9%, Se=57,7%, Sp=90,8%.

Schlussfolgerung

Die Klassifizierungsleistung der evaluierten KI-Modelle hängt in hohem Maße von den Trainingsdaten und weniger von der Architektur selbst ab. Unsere Ergebnisse zeigen eine hohe Spezifität und eine moderate Sensitivität bei der Differenzierung von COVID-19- und Nicht-COVID-19-Pneumonien. Die KI-Klassifikationsmodelle sollten aber nicht unkritisch verwendet werden, könnten aber Radiologen unterstützen.

Kernaussagen

• Vorliegende Studie vergleicht KI-Ansätze zur bildbasierten Diagnose von COVID-19 in Thorax-CTs, was relevant für die weitere Optimierung der Gesundheitsversorgung und die Vorbereitung auf etwaig kommende Pandemien ist.

• Unser multizentrischer, herstellerübergreifender Datensatz zeigt, dass die Trainingsdaten der entscheidende Faktor für die diagnostische Leistungsfähigkeit sind.

• KI-Modelle sollten nicht autonom eingesetzt werden, sondern als unterstützendes Werkzeug in die radiologische Befundung integriert werden, um die diagnostische Entscheidungsfindung zu ergänzen – nicht zu ersetzen.

Keywords

Artificial Intelligence - Pandemic Preparedness - CT Scan

Publication History

Received: 02 August 2024

Accepted after revision: 17 February 2025

Article published online:
29 April 2025

• References

• 1 Inui S, Gonoi W, Kurokawa R. et al. The role of chest imaging in the diagnosis, management, and monitoring of coronavirus disease 2019 (COVID-19). Insights Imaging 2021; 12: 155
• 2 Kwee TC, Kwee RM. Chest ct in covid-19: What the radiologist needs to know. Radiographics 2020; 40: 1848-1865
• 3 Sethuraman N, Jeremiah SS, Ryo A. Interpreting Diagnostic Tests for SARS-CoV-2. JAMA – Journal of the American Medical Association 2020; 323: 2249-2251
• 4 Rubin GD, Ryerson CJ, Haramati LB. et al. The role of chest imaging in patient management during the covid-19 pandemic: A multinational consensus statement from the fleischner society. Radiology 2020; 296: 172-180
• 5 Fervers P, Fervers F, Jaiswal A. et al. Assessment of COVID-19 lung involvement on computed tomography by deep-learning-, threshold-, and human reader-based approaches—an international, multi-center comparative study. Quant Imaging Med Surg 2022; 12: 5156-5170
• 6 Xiong Z, Wang R, Bai HX. et al. Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT. Radiology 2020; 296: E156-E165
• 7 Song Y, Zheng S, Li L. et al. Deep Learning Enables Accurate Diagnosis of Novel Coronavirus (COVID-19) with CT Images. IEEE/ACM Trans Comput Biol Bioinform 2021; 18: 2775-2780
• 8 Rahimzadeh M, Attar A, Sakhaei SM. A fully automated deep learning-based network for detecting COVID-19 from a new and large lung CT scan dataset. Biomed Signal Process Control 2021; 68
• 9 Jin C, Chen W, Cao Y. et al. Development and evaluation of an artificial intelligence system for COVID-19 diagnosis. Nat Commun 2020; 11: 5088
• 10 Li L, Qin L, Xu Z. et al. Using Artificial Intelligence to Detect COVID-19 and Community-acquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy. Radiology 2020; 296: E65-E71
• 11 Qian X, Fu H, Shi W. et al. M3Lung-Sys: A Deep Learning System for Multi-Class Lung Pneumonia Screening from CT Imaging. IEEE J Biomed Health Inform 2020; 24: 3539-3550
• 12 Wang SH, Nayak DR, Guttery DS. et al. COVID-19 classification by CCSHNet with deep fusion using transfer learning and discriminant correlation analysis. Information Fusion 2021; 68: 131-148
• 13 Wang SH, Govindaraj VV, Górriz JM. et al. Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network. Information Fusion 2021; 67: 208-229
• 14 Wang X, Deng X, Fu Q. et al. A Weakly-Supervised Framework for COVID-19 Classification and Lesion Localization from Chest CT. IEEE Trans Med Imaging 2020; 39: 2615-2625
• 15 Han Z, Wei B, Hong Y. et al. Accurate Screening of COVID-19 Using Attention-Based Deep 3D Multiple Instance Learning. IEEE Trans Med Imaging 2020; 39: 2584-2594
• 16 Lee EH, Zheng J, Colak E. et al. Deep COVID DeteCT: an international experience on COVID-19 lung detection and prognosis using chest CT. NPJ Digit Med 2021; 4: 11
• 17 Javaheri T, Homayounfar M, Amoozgar Z. et al. CovidCTNet: an open-source deep learning approach to diagnose covid-19 using small cohort of CT images. NPJ Digit Med 2021; 4: 29
• 18 Wang S, Zha Y, Li W. et al. A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis. European Respiratory Journal 2020; 56
• 19 Zhang K, Liu X, Shen J. et al. Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography. Cell 2020; 181: 1423-1433.e11
• 20 Gunraj H, Wang L, Wong A. COVIDNet-CT: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases From Chest CT Images. Front Med (Lausanne) 2020; 7
• 21 Gunraj H, Sabri A, Koff D. et al. COVID-Net CT-2: Enhanced Deep Neural Networks for Detection of COVID-19 From Chest CT Images Through Bigger, More Diverse Learning. Front Med (Lausanne) 2022; 8
• 22 Qi S, Xu C, Li C. et al. DR-MIL: deep represented multiple instance learning distinguishes COVID-19 from community-acquired pneumonia in CT images. Comput Methods Programs Biomed 2021; 211
• 23 Wu YH, Gao SH, Mei J. et al. JCS: An Explainable COVID-19 Diagnosis System by Joint Classification and Segmentation. IEEE Transactions on Image Processing 2021; 30: 3113-3126
• 24 Amyar A, Modzelewski R, Li H. et al. Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: Classification and segmentation. Comput Biol Med 2020; 126
• 25 Wang B, Jin S, Yan Q. et al. AI-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system. Appl Soft Comput 2021; 98
• 26 Gao K, Su J, Jiang Z. et al. Dual-branch combination network (DCN): Towards accurate diagnosis and lesion segmentation of COVID-19 using CT images. Med Image Anal 2021; 67
• 27 Afshar P, Rafiee MJ, Naderkhani F. et al. Human-level COVID-19 diagnosis from low-dose CT scans using a two-stage time-distributed capsule network. Sci Rep 2022; 12: 4827
• 28 Qi Q, Qi S, Wu Y. et al. Fully automatic pipeline of convolutional neural networks and capsule networks to distinguish COVID-19 from community-acquired pneumonia via CT images. Comput Biol Med 2022; 141
• 29 Mei X, Lee HC, yue Diao K. et al. Artificial intelligence–enabled rapid diagnosis of patients with COVID-19. Nat Med 2020; 26: 1224-1228
• 30 Hou J, Xu J, Jiang L. et al. Periphery-aware COVID-19 diagnosis with contrastive representation enhancement. Pattern Recognit 2021; 118
• 31 Jin YH, Cai L, Cheng ZS. et al. A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019-nCoV) infected pneumonia (standard version). Mil Med Res 2020; 7: 4
• 32 Meng F, Kottlors J, Shahzad R. et al. AI support for accurate and fast radiological diagnosis of COVID-19: an international multicenter, multivendor CT study. Eur Radiol 2022;
• 33 Badrinarayanan V, Kendall A, Cipolla R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans Pattern Anal Mach Intell 2017; 39: 2481-2495
• 34 Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation, in: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). In: . Springer Verlag; 2015: 234-241
• 35 Hofmanninger J, Prayer F, Pan J. et al. Langs, Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem. Eur Radiol Exp 2020; 4: 50
• 36 He K, Zhang X, Ren S. et al. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). 2016
• 37 Deng J, Dong W, Socher R. et al. ImageNet: A large-scale hierarchical image database. Institute of Electrical and Electronics Engineers (IEEE) 2010; 248-255
• 38 He K, Zhang X, Ren S. et al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (pp. 1026–1034). 2015
• 39 Virtanen P, Gommers R, Oliphant TE. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods 2020; 17: 261-272
• 40 Pedregosa F, Varoquaux G, Gramfort A. et al. Scikit-learn: Machine learning in Python. The Journal of machine Learning research 2011; 12: 2825-2830
• 41 Robin X, Turck N, Hainard A. et al. pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011; 12: 77
• 42 Hunter JD. Matplotlib: A 2D graphics environment. Computing in science & engineering 2007; 9 (03) 90-95
• 43 Garg A, Salehi S, la Rocca M. et al. Efficient and visualizable convolutional neural networks for COVID-19 classification using Chest CT. Expert Syst Appl 2022; 195
• 44 Ardakani AA, Kanafi AR, Acharya UR. et al. Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: Results of 10 convolutional neural networks. Comput Biol Med 2020; 121
• 45 Bai HX, Hsieh B, Xiong Z. et al. Performance of Radiologists in Differentiating COVID-19 from Non-COVID-19 Viral Pneumonia at Chest CT. Radiology 2020; 296: E46-E54
• 46 Hochhegger B, Zanon M, Altmayer S. et al. COVID-19 mimics on chest CT: a pictorial review and radiologic guide. Br J Radiol 2021; 94