Bringing Explainability to Deep Learning–Based Clinical Support Decision Systems

Background: Different deep learning models have been developed to support clinical decision making. However, despite promising results, because of the black box nature of neural networks in general, it is difficult for humans to understand how the various parameters within the models are combined to produce an appropriate classification result. Therefore, reluctance to artificial intelligence based clinical decision support models remains high on patient as well as on physician side. Hence, explainability may become a critical feature for clinical decision support systems to increase user acceptance and may become a legal necessity in near feature.

Method: Even though explanability of deep learning methods and black box models in general is a current research area, promising model-agnostic interpretation methods such as LIME and SHAP already exist but are not regularly applied to clinical decision support models. We applied both methods to a Feed Forward Neural Network (FNN) and a Long Short-Term Memory (LSTM) model we trained on the MIMIC-III dataset for the prediction of cardiovascular ICU readmission and analyzed the results.

Results: For the analysis of the explanation results, we first checked automatically whether the clinical parameter values fit the classification results of the respective model. In the second step, we visually processed particularly interesting results and had them reviewed by clinical experts. The analysis of the explanation results shows that the explainability of FNN and the LSTM-based model using LIME and SHAP differ from each other in terms of performance. Also, the importance of features is partly different for the same input and output in both deep learning methods. In addition, based on the results, we found out, that some of the predictions may be made based on wrong parameters.

Conclusion: Our results show that LIME and SHAP are basically useful tools to explain the classification results of our deep learning models and therefore to help physicians to understand the decision made by the system. However, for these methods to be used in a practical clinical context, their performance and their level of detail must be improved. Nonetheless, model-agnostic interpretation methods combined with deep learning models may help physicians to analyze patient's condition faster and more systematically, not only increasing quality of patient care but also increasing user acceptance of clinical AI decisions systems.

Publication History

Article published online:
03 February 2022