Artificial Intelligence and Deep Learning for Advancing PET Image Reconstruction: State-of-the-Art and Future Directions

Abstract

Positron emission tomography (PET) is vital for diagnosing diseases and monitoring treatments. Conventional image reconstruction (IR) techniques like filtered backprojection and iterative algorithms are powerful but face limitations. PET IR can be seen as an image-to-image translation. Artificial intelligence (AI) and deep learning (DL) using multilayer neural networks enable a new approach to this computer vision task. This review aims to provide mutual understanding for nuclear medicine professionals and AI researchers. We outline fundamentals of PET imaging as well as state-of-the-art in AI-based PET IR with its typical algorithms and DL architectures. Advances improve resolution and contrast recovery, reduce noise, and remove artifacts via inferred attenuation and scatter correction, sinogram inpainting, denoising, and super-resolution refinement. Kernel-priors support list-mode reconstruction, motion correction, and parametric imaging. Hybrid approaches combine AI with conventional IR. Challenges of AI-assisted PET IR include availability of training data, cross-scanner compatibility, and the risk of hallucinated lesions. The need for rigorous evaluations, including quantitative phantom validation and visual comparison of diagnostic accuracy against conventional IR, is highlighted along with regulatory issues. First approved AI-based applications are clinically available, and its impact is foreseeable. Emerging trends, such as the integration of multimodal imaging and the use of data from previous imaging visits, highlight future potentials. Continued collaborative research promises significant improvements in image quality, quantitative accuracy, and diagnostic performance, ultimately leading to the integration of AI-based IR into routine PET imaging protocols.

Keywords

PET image reconstruction - Artificial intelligence - Deep learning - Medical imaging - Quantitative accuracy

Publication History

Received: 28 August 2023

Accepted after revision: 12 October 2023

Article published online:
23 November 2023

• Literature

• 1 Hoeschen C, Rafecas M, Aspelmeier T. Algorithms for Image Reconstruction. In: Cantone MC, Hoeschen C. Radiation Physics for Nuclear Medicine. Berlin, Heidelberg: Springer Berlin Heidelberg; 2011: 211-232
• 2 Radon J. Über die Bestimmung von Funktionen durch ihre Integralwerte längs gewisser Mannigfaltigkeiten. Berichte Über Verhandlungen K-Sächs Ges Wiss Zu Leipz Math-Phys Kl 1917; 69: 262-277
• 3 Bracewell R. Strip Integration in Radio Astronomy. Aust J Phys 1956; 9: 198
• 4 López-Montes A, Galve P, Udias JM. et al. Real-Time 3D PET Image with Pseudoinverse Reconstruction. Appl Sci 2020; 10: 2829
• 5 Bares R, Klever P, Hellwig D. et al. Pancreatic cancer detected by positron emission tomography with 18F-labelled deoxyglucose: method and first results. Nucl Med Commun 1993; 14: 596-601
• 6 Hudson HM, Larkin RS. Accelerated image reconstruction using ordered subsets of projection data. IEEE Trans Med Imaging 1994; 13: 601-609
• 7 Levitan E, Herman GT. A Maximum a Posteriori Probability Expectation Maximization Algorithm for Image Reconstruction in Emission Tomography. IEEE Trans Med Imaging 1987; 6: 185-192
• 8 Rogasch JM, Hofheinz F, Lougovski A. et al. The influence of different signal-to-background ratios on spatial resolution and F18-FDG-PET quantification using point spread function and time-of-flight reconstruction. EJNMMI Phys 2014; 1: 12
• 9 Lange K, Bahn M, Little R. A Theoretical Study of Some Maximum Likelihood Algorithms for Emission and Transmission Tomography. IEEE Trans Med Imaging 1987; 6: 106-114
• 10 Lipinski B, Herzog H, Rota Kops E. et al. Expectation maximization reconstruction of positron emission tomography images using anatomical magnetic resonance information. IEEE Trans Med Imaging 1997; 16: 129-136
• 11 Bland J, Mehranian A, Belzunce MA. et al. MR-Guided Kernel EM Reconstruction for Reduced Dose PET Imaging. IEEE Trans Radiat Plasma Med Sci 2018; 2: 235-243
• 12 Bishop CM. Pattern Recognition and Machine Learning. Softcover reprint of the original 1st edition 2006 (corrected at 8th printing 2009). New York, NY: Springer New York; 2016
• 13 McCulloch WS, Pitts W. A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 1943; 5: 115-133
• 14 Hornik K, Stinchcombe M, White H. Multilayer feedforward networks are universal approximators. Neural Netw 1989; 2: 359-366
• 15 Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. 2017.
• 16 LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 521: 436-444
• 17 He K, Zhang X, Ren S. et al. Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). In: . Las Vegas, NV, USA: IEEE; 2016: 770-778
• 18 Tieleman T, Hinton G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA Neural Netw Mach Learn 2012; 4: 26-31
• 19 Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM 2017; 60: 84-90
• 20 Boser BE, Guyon IM, Vapnik VN. A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on Computational learning theory. In: . Pittsburgh Pennsylvania USA: ACM; 1992: 144-152
• 21 De A, Horwitz GD. Spatial receptive field structure of double-opponent cells in macaque V1. J Neurophysiol 2021; 125: 843-857
• 22 Turner MR. Texture discrimination by Gabor functions. Biol Cybern 1986; 55: 71-82
• 23 Ronneberger O, Fischer P, Brox T. et al. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N, Hornegger J, Wells WM. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer International Publishing; 2015: 234-241
• 24 Goodfellow I, Pouget-Abadie J, Mirza M. et al. Generative Adversarial Nets. Ghahramani Z, Welling M, Cortes C. Curran Associates, Inc.; 2014.
• 25 Zhu B, Liu JZ, Cauley SF. et al. Image reconstruction by domain-transform manifold learning. Nature 2018; 555: 487-492
• 26 Häggström I, Schmidtlein CR, Campanella G. et al. DeepPET: A deep encoder–decoder network for directly solving the PET image reconstruction inverse problem. Med Image Anal 2019; 54: 253-262
• 27 Liu Z, Chen H, Liu H. et al. Deep Learning Based Framework for Direct Reconstruction of PET Images. In: Shen D, Liu T, Peters TM. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Cham: Springer International Publishing; 2019: 48-56
• 28 Whiteley W, Luk WK, Gregor J. DirectPET: full-size neural network PET reconstruction from sinogram data. J Med Imaging 2020; 7: 1
• 29 ComtatBataille F, Michel C. et al. OSEM-3D Reconstruction Strategies for the ECAT HRRT. In: IEEE Symposium Conference Record Nuclear Science 2004. Rome, Italy: IEEE; 2004: 3492-3496
• 30 Hu Z, Xue H, Zhang Q. et al. DPIR-Net: Direct PET Image Reconstruction Based on the Wasserstein Generative Adversarial Network. IEEE Trans Radiat Plasma Med Sci 2021; 5: 35-43
• 31 Ma R, Hu J, Sari H. et al. An encoder-decoder network for direct image reconstruction on sinograms of a long axial field of view PET. Eur J Nucl Med Mol Imaging 2022; 49: 4464-4477
• 32 Gong K, Guan J, Kim K. et al. Iterative PET Image Reconstruction Using Convolutional Neural Network Representation. IEEE Trans Med Imaging 2019; 38: 675-685
• 33 Ulyanov D, Vedaldi A, Lempitsky V. Deep Image Prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018
• 34 Gong K, Catana C, Qi J. et al. Direct Reconstruction of Linear Parametric Images From Dynamic PET Using Nonlocal Deep Image Prior. IEEE Trans Med Imaging 2022; 41: 680-689
• 35 Yokota T, Kawai K, Sakata M. et al. Dynamic PET Image Reconstruction Using Nonnegative Matrix Factorization Incorporated With Deep Image Prior. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2019
• 36 Ote K, Hashimoto F, Onishi Y. et al. List-Mode PET Image Reconstruction Using Deep Image Prior. IEEE Trans Med Imaging 2023; 42: 1822-1834
• 37 Fessler JA. Optimization Methods for Magnetic Resonance Image Reconstruction: Key Models and Optimization Algorithms. IEEE Signal Process Mag 2020; 37: 33-40
• 38 Gong K, Catana C, Qi J. et al. PET Image Reconstruction Using Deep Image Prior. IEEE Trans Med Imaging 2019; 38: 1655-1665
• 39 Chun Y, Fessler JA. Deep BCD-Net Using Identical Encoding-Decoding CNN Structures for Iterative Image Recovery. In: 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP). Zagorochoria: IEEE; 2018: 1-5
• 40 Gong K, Wu D, Kim K. et al. MAPEM-Net: an unrolled neural network for Fully 3D PET image reconstruction. In: Matej S, Metzler SD. 15th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine. Philadelphia, United States: SPIE; 2019: 102
• 41 Mehranian A, Reader AJ. Model-Based Deep Learning PET Image Reconstruction Using Forward–Backward Splitting Expectation–Maximization. IEEE Trans Radiat Plasma Med Sci 2021; 5: 54-64
• 42 Corda-D’Incan G, Schnabel JA, Hammers A. et al. Single-Modality Supervised Joint PET-MR Image Reconstruction. IEEE Trans Radiat Plasma Med Sci 2023; 1-1
• 43 Whiteley W, Gregor J. CNN-based PET sinogram repair to mitigate defective block detectors. Phys Med Biol 2019; 64: 235017
• 44 Kaplan S, Zhu YM. Full-Dose PET Image Estimation from Low-Dose PET Image Using Deep Learning: a Pilot Study. J Digit Imaging 2019; 32: 773-778
• 45 Wang Y, Zhou L, Yu B. et al. 3D Auto-Context-Based Locality Adaptive Multi-Modality GANs for PET Synthesis. IEEE Trans Med Imaging 2019; 38: 1328-1339
• 46 Lu W, Onofrey JA, Lu Y. et al. An investigation of quantitative accuracy for deep learning based denoising in oncological PET. Phys Med Biol 2019; 64: 165019
• 47 Zhou L, Schaefferkoetter JD, Tham IWK. et al. Supervised learning with cyclegan for low-dose FDG PET image denoising. Med Image Anal 2020; 65: 101770
• 48 Matej S, Surti S, Jayanthi S. et al. Efficient 3-D TOF PET Reconstruction Using View-Grouped Histo-Images: DIRECT—Direct Image Reconstruction for TOF. IEEE Trans Med Imaging 2009; 28: 739-751
• 49 Whiteley W, Panin V, Zhou C. et al. FastPET: Near Real-Time Reconstruction of PET Histo-Image Data Using a Neural Network. IEEE Trans Radiat Plasma Med Sci 2021; 5: 65-77
• 50 Shiri I, Arabi H, Geramifar P. et al. Deep-JASC: joint attenuation and scatter correction in whole-body 18F-FDG PET using a deep residual network. Eur J Nucl Med Mol Imaging 2020; 47: 2533-2548
• 51 Li T, Zhang M, Qi W. et al. Motion correction of respiratory-gated PET images using deep learning based image registration framework. Phys Med Biol 2020; 65: 155003
• 52 Ledig C, Theis L, Huszar F. et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI: IEEE; 2017: 105-114
• 53 YangJianchaoWright J, Huang TS. et al. Image Super-Resolution Via Sparse Representation. IEEE Trans Image Process 2010; 19: 2861-2873
• 54 Jin W, Li X, Fatehi M. et al. Guidelines and evaluation of clinical explainable AI in medical image analysis. Med Image Anal 2023; 84: 102684
• 55 Ribeiro MT, Singh S, Guestrin C. „Why Should I Trust You?“: Explaining the Predictions of Any Classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Francisco California USA: ACM; 2016
• 56 Herington J, McCradden MD, Creel K. et al. Ethical Considerations for Artificial Intelligence in Medical Imaging: Deployment and Governance. J Nucl Med 2023;
• 57 Langlotz CP, Allen B, Erickson BJ. et al. A Roadmap for Foundational Research on Artificial Intelligence in Medical Imaging: From the 2018 NIH/RSNA/ACR/The Academy Workshop. Radiology 2019; 291: 781-791