Effectiveness of Clinical Decision Support Systems on the Appropriate Use of Imaging for Central Nervous System Injuries: A Systematic Review

Abstract

Background One of the best practices for timely and efficient diagnoses of central nervous system (CNS) trauma and complex diseases is imaging. However, rates of imaging for CNS are high and impose a lot of costs to health care facilities in addition to exposing patients with negative impact of ionizing radiation.

Objectives This study aimed to systematically review the effects and features of clinical decision support systems (CDSSs) for the appropriate use of imaging for CNS injuries.

Method We searched MEDLINE, SCOPUS, Web of Science, and Cochrane without time period restriction. We included experimental and quasiexperimental studies that assessed the effectiveness of CDSSs designed for the appropriate use of imaging for CNS injuries in any clinical setting, including primary, emergency, and specialist care. The outcomes were categorized based on imaging-related, physician-related, and patient-related groups.

Result A total of 3,223 records were identified through the online literature search. Of the 55 potential papers for the full-text review, 11 eligible studies were included. Reduction of CNS imaging proportion varied from 2.6 to 40% among the included studies. Physician-related outcomes, including guideline adherence, diagnostic yield, and knowledge, were reported in five studies, and all demonstrated positive impact of CDSSs. Four studies had addressed patient-related outcomes, including missed or delayed diagnosis, as well as length of stay. These studies reported a very low rate of missed diagnosis due to the cancellation of computed tomography (CT) examine according to the CDSS recommendations.

Conclusion This systematic review reports that CDSSs decrease the utilization of CNS CT scan, while increasing physicians' adherence to the rules. However, the possible harm of CDSSs to patients was not well addressed by the included studies and needs additional investigation. The actual effect of CDSSs on appropriate imaging would be realized when the saved cost of examinations is compared with the cost of missed diagnosis.

Keywords

clinical decision support system - computer-assisted decision-making - appropriateness reviews - hospitals - radiology - imaging - central nervous system - evaluation of impact

Background and Significance

In many countries, a considerable proportion of health care budgets are spent on diagnostic tools such as radiology and laboratory tests. Despite the emphasis on controlling health care costs, using imaging procedures is increasing.[1] According to the evidence, the number of computed tomography (CT) scan orders in the United States raised from 3 million in 1980 to 80 million in 2014[2]; however, a large proportion of this increase is due to the technical advances and improvement in the quality and potential usefulness of CT scans in this period.

Central nervous system (CNS) injuries are a major burden of morbidity and mortality worldwide[3] and neuroimaging plays an important role in the diagnosis of CNS injuries. The best practice for initial evaluation of patients with head injuries is CT scan.[4] Imaging of CNS injuries like minor head injuries has increased dramatically.[5] However, one out of three head CT scans are unnecessary and can be avoided.[6] This led the American Board of Internal Medicine to establish a campaign in 2012 to prevent unnecessary and inappropriate medical imaging, testing, treatment, and procedures. Although since the establishment of this campaign, there has been a reduction in resource utilization, the imaging utilization is still high.[7]

In addition to imposing a lot of cost to health care facilities, unnecessary imaging exposes patients to the negative impact of ionizing radiation.[8] It is estimated that approximately 1.5 to 2% of the cancers are the result of imaging procedures.[9] It also increases the workload of radiology departments and may increase the risk of error as a result.[10] Meanwhile, evidence shows that approximately 20 to 50% of the radiology and imaging procedures are unnecessary.[11] The cost of the examinations and the potential impact of the radiation must be weighed against the potentially improved clinical outcomes, reduced diagnostic delays, and reduced distress of the patient, as well as relatives. Several evidence-based clinical guidelines are proposed for appropriate CT use. These guidelines suggest some criteria for what is “clinically reasonable” while ordering an imaging.

Several interventions have been suggested for improving “clinically reasonable” imaging including but not limited to using clinical practice guidelines,[12] clinical decision support systems (CDSS),[13] audit, and feedback,[14] and electronic health record (EHR).[1] [15] In comparison to other information technologies, CDSSs may have more potential to support physicians in deciding about image ordering, reducing unnecessary imaging and radiation explosion while improving quality of care. CDSS can provide physicians with relevant knowledge through evidence-based practices presented at a suitable time to improve decision-making.[16] Moreover, CDSSs have the potential to improve physicians' communication with radiologists through presenting “structured coded indications.”[17] CDSSs interventions, implemented for appropriate use of imaging, were mostly in the form of duplicate imaging warnings, or the introduction of predetermined appropriate criteria that provide knowledge about when it is appropriate to order a diagnostic examination, as well as matching patients' characteristics with rule-based algorithms. These rules and criteria sets have been assessed and approved in previous studies,[18] [19] [20] but the impact of these technologies, like CDSSs, are not well investigated. Hynes et al[21] implemented CDSS through incorporating the National Emergency X-Radiography Utilization Study (NEXUS) criteria and Canadian C-Spine Rule (CCSR) as “checkbox” items into the computerized order entry system for cervical spine radiographs; if the radiograph was for a patient with trauma, the checkboxes were activated to be completed by physicians and show if the order was indicated by the criteria. However, the literature indicates inconsistent findings of the decision rule effects on physicians' performance and patients' outcomes.[22] [23] [24] Thus, there is a need for a systematic review on the impact of the technologies like CDSSs on ordering appropriate imaging procedures.

There are some related systematic review studies that mostly have been conducted to determine the effectiveness of the decision rules rather than investigating the effectiveness of CDSSs. Desai et al[25] examined the effectiveness of interventions designed to decrease cervical-spine radiography ordering for adults with neck injury referring to emergency departments (EDs). The investigated interventions included evidence-based decision rules like NEXUS or Canadian C-spine rule rather than CDSSs. They concluded that the effectiveness of the strategies was moderate and was not frequently reported. Two other similar systematic reviews by Liu et al[26] and Jenkins et al[27] also aimed at determining the effectiveness of interventions designed to reduce imaging in patients with lower back pain. In addition, Harnan et al[23] systematically reviewed the literature to identify decision rules for minor head injury and compare them according to accuracy. Another similar systematic review is conducted by Goldzweig et al[1] which investigated the impact of EHR-based interventions on appropriate image ordering in ambulatory, hospital, and emergency department settings. EHR-based interventions include computerized provider order entry systems, computerized display of charges, and computerized CDSSs for any kind of imaging.

However, to the best of our knowledge, there is no systematic review with the aim to determine the effectiveness of computerized CDSSs for appropriate image ordering for CNS injuries. Thus, the goal of this study was to systematically review the effectiveness and features of CDSSs designed for the appropriate use of imaging for CNS injuries.

Method

The research question was “Do CDSSs improve appropriate use of imaging in patients with central nervous system injuries?”

Search Strategy and Study Selection

A search strategy was developed using keywords and the Medical Subject Headings (MeSH) terms to identify papers in the literature and adaptations were made for each database. Four databases were searched: Medline (through PubMed), SCOPUS, Web of Science, and Cochrane. We considered studies published till August 11, 2020, without any time limitation. The search strategy consisted of a combination of keywords and MeSH terms associated with diagnostic imaging (laboratory test utilization), wounds and injuries, central nervous system, CDSSs, and utilization review. The search strategy is presented in [Supplementary Appendix A] (available in the online version).

After removing duplicates, two reviewers, working independently, selected the papers based on eligibility criteria. Titles and abstracts were investigated for inclusion. The full text of potentially relevant papers was screened based on the inclusion and exclusion criteria. The reference lists of the identified papers were also searched to find any other relevant paper missed during the databases searches. The researchers resolved disagreements by discussion and consensus, and any remaining disagreements were resolved by the third reviewer.

Study Selection Criteria

Inclusion Criteria

Type of studies: Experimental and quasiexperimental study designs were included randomized controlled trials (RCT), quasiexperimental, nonrandomized controlled clinical trials (NRCT), prospective observational studies, cohort, and interrupted time series (ITS).

Type of population: the study populations in the included studies were imaging procedures of CNS, physicians ordering CNS imaging, or the patients for whom CNS imaging procedures were ordered in any clinical setting, including primary, emergency, and specialist care.

Types of interventions studies using CDSSs as an intervention to improve appropriate image ordering for CNS injuries were included. Any electronic decision rule provided to physicians either standalone or integrated into electronic health record (EHR) or computerized physician order entry (CPOE) was considered as a CDSS.

Type of outcomes: outcome measures were: diagnostic yield and diagnostic detection rate, the number and cost of imaging ordered, guideline adherence for imaging ordering, physicians knowledge and attitude toward imaging, and also patient outcomes. The outcomes were categorized based on imaging-related, physician-related, and patient-related groups. Imaging-related outcomes were proportion and number of imaging, and cost of imaging. Physician-related outcomes were diagnostic yield and diagnostic detection rate, adherence or order cancellation after the reminders (or overriding the reminders), and physicians' knowledge and attitude. Patient-related outcomes were length of stay (LOS), patients' complications or undetected fractures, readmission, patients' disposition, and mortality rate.

Exclusion Criteria

Studies were excluded if they were (1) published in any language rather than English; (2) examined feasibility, validity, accuracy, and usability; (3) described a CDSS; (4) used interventions rather than computerized CDSS; (5) conducted based on a scenario or in an unreal clinical environment (in a simulated setting i.e., for the test of a system); (6) descriptive studies; and/or (7) presented as a congress abstract.

Quality Assessment

The Effective Public Health Practice Project (EPHPP) quality assessment tool was used to assess the methodological quality of the included studies.[28] EPHPP is a suitable tool for assessing a variety of study designs like RCTs, NRCTs, and ITSs. The Cochrane tool[29] was not used for assessing the risk of bias because there was no RCT design among the included studies. Furthermore, the EPHPP tool is recommended by the Cochrane Public Health as an appropriate tool for systematic reviews of effectiveness.[30] EPHPP tool assesses studies based on six criteria including selection bias, study design, confounders, blinding, data collection methods, and withdrawals and dropouts. According to this tool, each criterion is categorized as good, fair, or poor, and then the total rating is determined. Studies with no poor criteria are considered as strong, studies with one poor criterion as medium, and studies with two or more poor criteria are considered as poor. The included studies were independently evaluated by two reviewers and any disagreement over scoring was resolved through consensus (the results are presented as a supplementary).

Quality and features of the CDSSs, presented in the included studies, were assessed using a checklist derived from the study by Goldzweig et al.[1] The checklist consists of three domains: (1) CDSS design, (2) data entry source, and (3) implementation source. The included studies were investigated by two independent reviewers in terms of study design and the degree of information reporting about the CDSSs and characteristics of implementation; any disagreement was resolved through consensus.

Data Extraction

We designed a form to extract data from each of the included studies. For each study, the following data were extracted: study design, sample size, intervention description, and results. One reviewer extracted data which were subsequently reviewed and confirmed by another reviewer.

Data Analysis

A narrative synthesis was used to describe and compare the designs and the results of included studies. We categorized studies based on different features of CDSSs, outcome category, and effects of CDSSs. The effect of interventions were reported based on statistically significant positive, positive without statistical argument, no effect (not statistically significant), negative without statistical argument, or statistically significant negative.[31] The variety of outcomes and results reporting did not allow performing a meta-analysis. However, a forest plot was presented for five studies which had reported odd ratio for the main finding (i.e., proportion of CT utilization). The forest plot was designed using Review Manager (RevMan) Version 5.3.[32]

Results

Study Selection

The literature search identified 3,223 records through online search 26 of which were duplicated ([Fig. 1]). In addition, two additional papers[33] [34] were identified through other sources (snowball search). The papers were screened for eligibility by title and abstract, resulting in 55 potential papers for the full-text review. During the full-text reviewing, 44 more papers were excluded. Finally, 11 eligible studies were included.

Fig. 1 PRISMA flow diagram of the study selection. CNS, central nervous system; ISI, Institute of Scientific Information; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Characteristics of the Included Studies

A substantial number of the included studies were conducted during the last 5 years (n = 8, 72.7%). Most of the included studies were conducted in the United States (n = 8, 72.7%) and one was conducted in each of the following countries: Australia, Canada, and Ireland ([Table 1]).

Table 1 Main characteristics of the included studies

Study (year)/country	Study objective	Setting	Study population	Study design level, duration	Intervention description	Control group/comparison	Imaging modality	Guideline/criteria	Outcome(s)	Finding(s)	Conclusion
Hynes et al (2020)/Ireland[22]	To rationalize the ordering of trauma cervical spine radiographs via the institution of electronic clinical decision support criteria	ED	Preintervention: 182 trauma X-rays Postintervention: 126 trauma X-rays	Before–after  2 months	•CDSS	Usual care (without the intervention)	Cervical spine radiograph	•NEXUS •CCSR	1. Total number of cervical spine trauma radiographs ordered 2. The proportion of requests clinically indicated as per NEXUS/CCSR guidelines 3. The detection of clinically significant C-Spine injury	1. The total number of cervical spine trauma radiographs decreased from 182 in 2016 to 126 in 2017 (a 30.7% reduction, χ2 = 10.20, p < 0.001) 2. The proportion of requests meeting either the NEXUS criteria or CCSR increased from 76.7 to 99.2% (a 22.5% increase, χ2 = 30.78, p < 0.0001) 3. Preintervention in 2016, 10.4% of patients went on to undergo cross-sectional imaging (18 to CT and 1 to MRI). In the comparative period in 2017, this figure was 23.8% (30 CT) and no fractures occurred in the 2016 group. There were 2 fractures in the 2017 group (1.6%), both of these were detected on initial plain radiographs	Introduction of clinical indication criteria to the electronic ordering system for cervical spine radiographs in trauma patients reduced the total number of requests by 30.7% while increasing the compliance proportion, which was indicated to be 99.2%
Zarchi (2020)/the United States[37]	To improve the knowledge of health care providers in the management of minor pediatric head injuries with the goal of reducing the rate of unnecessary CT scans and associated radiation exposure	Urgent care facility	Preintervention: Postintervention: 10 health care providers	Before–after 10 weeks	•Educational workshops •CDS	No control group	CT scan	PECARN	1. Knowledge measurement 2. Protocol adoption measurement 3. CT scan rates	1. The mean scores of knowledge was 30% pretests vs. 76.67% posttests 2. High likelihood of intent to adopt the PECARN protocol 3. Rates of ordered head CT scans was 54.76% pre project vs. 17.39% postproject	Using the PECARN head injury protocol was associated with consistency of care, reduced unnecessary health care resource utilization, and health care costs
Ballard et al et al (2019)/the United States[38]	To determine the effect of providing risk estimates of ciTBI and management recommendations on ED outcomes for children with isolated intermediate PECARN ciTBI risk factors	13 EDs (5 pediatric and 8 general EDs)	Preintervention: 2,618 providers Postintervention: 2,551 providers	•NRCT •31 months	•Clinicians were prompted in the electronic health record to complete a head trauma template that was designed for the study to collect data on all PECARN risk factors for the 2 PECARN age groups (<2 years and 2 to 17 years) •CDSS	Usual care (without the intervention)	Cranial CT	PECARN	1. Proportion of ED cranial CTs 2. Hospital admission 3. Proportion the ED length of stay 4. Proportion of patients with missed clinically important traumatic brain injury during the index ED visit	1. The pooled CT proportion decreased from 24.2% before clinical decision support to 21.6% after it (post–clinical decision support odds ratio = 0.86; 95% CI: 0.73 to 1.01) 2. No change in hospital admission 3. Non-significant Increase in length of stay 4. Clinically important traumatic brain injuries were identified at the ED index visit in 37 of 37 (100%) before and 32 of 33 (97.0%) after clinical decision support implementation	Providing specific risks of clinically important traumatic brain injury through electronic clinical decision support was associated with the modest and safe decrease in ED CT use for children at nonnegligible risk of clinically important traumatic brain injuries
Engineer et al (2018)/The United States[41]	To develop a Pediatric Mild Head Injury Care Path to reduce inappropriate CT utilization with support of a clinical decision support tool (CDST) and a structured parent discussion tool	2 nontrauma center sites	Preintervention: 3,770 patients Postintervention: 227 patients	Before–after 9 weeks	•Engagement of leadership  •Provider education  •Incorporation of a parent discussion tool to guide discussion during the emergency department (ED) visit between the parent and the provider  •CDST embedded in the electronic medical record  •Importation of data into the note to drive compliance	No control group	Head CT	PECARN	1. The rate of head CT utilization	1. Head CT utilization was reduced from 62.7 to 22% (odds ratio = 0.17; 95% CI: 0.12–0.24)	A Pediatric Mild Head Injury Care Path can be implemented in a pediatric and freestanding ED, resulting in reduced head CT utilization and high levels of adherence to CDST recommendations
Sharp et al (2018)/the United States[42]	To evaluate the association of implementation of the Canadian CT Head Rule on head CT imaging in community EDs	13 community EDs	Preintervention: 26,740 patients and 1,751 attending physicians Postintervention: 15,394 patients and 1,576 attending physicians	Prospective, observational, interrupted time-series  24 months	•Clinical leadership endorsement of the Canadian CT Head Rule •Physician education •CDSS	Usual care (without the intervention)	Head CT	CCTHR	1. The proportion of noncontrast head CT scans  2. The proportion of brain injury diagnoses	1. 5.3% (95% CI: 2.5–8.1%) absolute and 15.8% relative reduction was observed in the proportion of encounters resulting in noncontrast head CT scans 2. There was 2.3% (95% CI: 1.5–3.1%) increase in intracranial injuries	A multicomponent implementation of the Canadian CT Head Rule was associated with a modest reduction in CT use and an increased diagnostic yield of head CTs for adult trauma encounters in community EDs
Min et al (2017)/Canada[43]	To determine whether point-of-care CDSS can effectively reduce inappropriate medical imaging of patients who present to the ED with LBP	An academic hospital ED	Preintervention: 46 physicians for 4,562 patient records postintervention: 46 physicians for 4,562 patient records	Before/after  29 months	•CDSS •Patient and physician education	Usual care (without the intervention)	LBP diagnostic imaging	Working group consisting of emergency physicians, radiologists, and family physicians confirm appropriateness criteria and red flags informed by literature	1. LBP visits as a proportion of all ED visits 2. The proportion of LBP visits with a medical imaging test ordered 3. Physician variation in ordering medical imaging for LBP patients	1. No significant change was observed in the proportion of ED visits 2. due to LBP (p = 0.265) 3. The proportion of LBP patients who received a medical image decreased significantly, from 22 to 17% (p = 0.0002) 4. The mean imaging rate among individual physicians decreased from 24 to 20% (p-value was not reported)	CDSS integrated in electronic order entry forms can safely and effectively reduce imaging orders for LBP patients in the ED
Dayan et al (2017)/the United States[39]	To determine whether implementing TBI prediction rules and providing risks of ciTBIs with CDSS reduces CT use	13 EDs (5 PEDs and 8 GEDs)	Intervention group: 16,635 patients Control group: 2,394 patients	Nonrandomized multicenter clinical trial 31 months	•Real-time EHR-based CDSS within site-specific workflows •Specific designation of and facilitation by local opinion leaders to train and encourage staff •Physician education	Usual care (without the intervention)	CT for children with MTBI	PECARN	1. CT rate  2. The number /percentage of patients with ciTBIs not identified on the initial ED visit  3. LOS in the ED for discharged patients	1. Adjusted for time trends, CT rates decreased significantly (p < 0.05) but modestly (2.3–3.7%) at 2 of 4 intervention PEDs for children at very low risk. The other 2 PEDs had small (0.8–1.5%) nonsignificant decreases. CT rates did not decrease consistently at the intervention GEDs, with low baseline CT rates (2.1–4.0%) in those at very low risk. The control PED had little change in CT use in similar children (from 1.6 to 2.9%); the control GED showed a decrease in the CT rate (from 7.1 to 2.6%). For all children with minor head trauma, intervention sites had small decreases in CT rates (1.7–6.2%).   2. ciTBI was identified on the day of ED enrollment in 58/58 (100%) patients before and 55/56 patients (98.2%) after CDS implementation. The 1 patient not identified at the index ED visit after CDS implementation did not meet the PECARN very low risk criteria.  3. The median LOS after CDS implementation increased at 7 of the 8 intervention EDs by 7 to 15 minutes (1 reaching statistical significance, P < 0.001)	The implementation of TBI prediction rules and provision of risks of ciTBIs by using CDSS was associated with modest, safe, but variable decreases in CT use
Bookman et al (2017)/the United States[44]	To evaluate the impact of evidence-based CDS tools integrated into provider workflow in the EHR on utilization of CT brain, C-spine, and pulmonary embolism	Five EDs in a healthcare system with a common EHR	Preintervention: 163 attending providers for 235,858 patient visits postintervention:163 attending providers for 235,858 patient visits	Before/after 17 months	Three CDS tools embedded into the EHR with specific intent to be minimally disruptive to provider workflow	Usual care (without the intervention)	•CT brain •CT C-spine •CT pulmonary embolism	•CCTHR •NEXUS •The Pulmonary Embolism Rule-out Criteria •Wells score	1. Proportion of CTs (i.e., CT brain, CT C-spine, and CT pulmonary embolism) ordered by providers	1. There was greater than 6% decrease in utilization of CT brain and CT C-spine (–10%, 95% CI = − 13% to −7%, p < 0.001; and –6%, 95% CI =–11% to –1%, p = 0.03, respectively). The use of CT pulmonary embolism also decreased but was not significant (−2%, 95% CI: −9 to +5%, p = 0.42).	Embedded CDSS into the EHR is associated with decreased overall utilization of high-cost imaging, especially among higher utilizers
Ip et al (2015)/the United States[45]	To examine the impact of CDSS on head CT utilization in MTBI ED visits	An academic trauma center	Intervention group: 116,009 patients Control group: 53,477 patients	Cohort study  24 months	Real-time CDSS embedded into institutional imaging CPOE system	Usual care (without the intervention)	Head CT	•The New Orleans Criteria •CCTHR •The CT in Head Injury Patients Prediction Rule	1. Intensity of head CT use in MTBI ED visits 2. Rates of delayed imaging and delays in diagnosing radiological significant findings	1. The utilization rate of head CTs among patients with MTBI decreased after implementing CDSS with an absolute difference of 7.8% and a relative decrease of 13.4% (p = 0.005) 2. No significant change was observed in the rate of delayed imaging between the pre-intervention and postintervention periods in the study cohort (6.7 vs. 9.4%, p =.231)	Implementation of a CDSS based on high-quality evidence was associated with a modest but significant decrease in head CT use in patients with MTBI with no evidence of significant increase in follow-up imaging or delayed diagnosis of intracranial injury within 7 days of their index ED visit
Gupta et al (2014)/the United States[46]	To determine the impact of a CPOE-integrated CDSS on adherence to evidence-based guidelines to guide emergency clinician decision making for use of head CT	ED of an academic trauma center	Preintervention: 200 patients Postintervention: 200 patients	Before after  26 months	CDSS	Usual care (without the intervention)	Head CT	•The New Orleans Criteria •CCTHR •The CT in Head Injury Patients Prediction Rule	1. Adherence to the evidence-based guidelines 2. Concordance of adherence documentation between the CDS tool and the clinical note in the EMR	1. Documented guideline adherence improved significantly (p < 0.001) with 27.5% absolute and 56.1% relative effect sizes 2. Concordance for documented guideline adherence between manual chart review and electronic CDS data entry was 70%	Implementing CDSS significantly increased documented adherence to published evidence for imaging in ED patients with MTBI
Goergen et al (2006)/Australia[40]	To determine an unvalidated imaging guideline can reduce the use of imaging in patients with cervical spine trauma	The ED of a tertiary referral hospital	Preintervention: 353 patients Postintervention: 403 patients	Nonrandomized clinical trial  11 months	•Guidelines introduced in small groups •Distribution of pocket sized reminder cards containing guidelines •Poster with guidelines placed in diagnostic imaging department and ED •Guidelines converted to CDSS	Before (without the intervention)	Cervical spine imaging	NEXUS	1. The proportion of patients for whom the CDSS was used 2. The proportion of patients managed with CDSS for whom the guideline advised no imaging and who had no imaging 3. The number of patients with delayed diagnosis of cervical spine injury	1. Forty percent of patients were managed with the assistance of the CDSS. A statistically significant reduction in the use of any cervical spine imaging occurred in the study group compared with controls, with 68.8% of study patients receiving cervical spine imaging compared with 78.5% of controls (9.7% reduction; p = 0.03; 95% CI: 3–16%). 1. 51 out of 141 (36%) were identified as being appropriate for no cervical spine imaging. Of these 51, 43 (86%) received no imaging 2. None of patients identified by the guideline as being at very low risk and thus suitable for no cervical imaging had delayed diagnosis of cervical spine injury at our hospital or elsewhere	It is feasible to disseminate and implement an evidence based imaging guideline for patients with cervical spine trauma. The use of a CDSS can facilitate this and is associated with a safe reduction in the proportion of patients imaged

Abbreviations: ACR, American College of Radiology Appropriateness Criteria; CCSR, Canadian C-Spine Rule; CCTHR, The Canadian CT Head Rule; CDSS, clinical/computerized decision support system; CI, confidence interval; ciTBIs, clinically important traumatic brain injuries; CPOE, computerized physician order entry; CT, computerized tomography; ED, emergency department; GED, general emergency department; GED, general emergency department; LBP, low-back pain; MRI, magnetic resonance imaging; MTBI, mild traumatic brain injury; NEXUS, The National Emergency X-Radiography Utilization Study; PECARN, the Pediatric Emergency Care Applied Research Network; PED, pediatric emergency department; PED, pediatric emergency room; TBI, traumatic brain injury.

Quality Assessment

Designs of the included studies were as follow: five quasiexperimental, four case controls, one cohort, and one interrupted time series. According to the EPHPP quality assessment tool, there was no strong study. There were four studies with moderate quality[35] [36] [37] [38] and seven studies with poor quality.[21] [39] [40] [41] [42] [43] [44] The main limitation of the included studies was not being blinded (90.9% had not blinded assessors or blinding was not mentioned). The other limitation was in data collection methods in which validity and reliability of the used tool were not described in some studies (54.54%); studies in which data were collected using EHR reports were considered poor in terms of reliability and validity due to the lack of description about the validity and reliability, as well as the evidence reporting variable and often limited quality of EHR reports.[45] [46] [47] The results are presented in an [Supplementary Appendix B] (available in the online version).

Quality assessments of the CDSSs are presented in [Table 2]. The vast majority of CDSSs was integrated into CPOEs or EHRs (90.9%) and providing real-time feedback (90.9%) with recommending not to order a specific imaging modality (100%). Most CDSS classifications of the included studies (72.7%) were in B category which could present information on appropriateness or guidelines specifically tailored to the individual patient but physicians could override the recommendation easily ([Table 2]). CDSSs, used in four studies (36.3%), were integrated into and automated through EHR. Eight studies (72.7%) needed the clinicians to enter data like patient indications specifically into the CDSS. Most studies (72.7%) had not reported if they had pilot tested the CDSS before implementation. Six studies (54.5%) reported user training about the intervention or the targeted imaging indication or similar things. Other characteristics, barriers, and facilitators affecting implementation of CDSS were mentioned in [Table 2].

Table 2 Quality assessment of the CDSS

Study	CDSS design				Data entry source		Implementation characteristic
	Is it integrated with CPOE?	Does it give real time feedback at point of care?	Does the CDS suggest a recommended course of action?	CDSS Classification[a]	Is it automated through EHR?	Does clinical staff enter data specifically for intervention?	Was it pilot tested or used an iterative process of development/implementation?	Was there any user training/clinician education?	Are the authors also the developers and part of the user group for the CDS?	Was there use of audit and- feedback (or other internal incentive)?	Are there any other implementation components not already discussed?
Hynes et al[22]	Yes	Yes	Yes[b]	B	No	Yes	NM[c]	NM	Yes	No	The authors emphasized on the “gatekeeper effect” (making the clinician more accountable for the imaging request) and the “educational effect” (increasing the number of physicians who are educated with respect to current guidelines)
Zarchi[37]	Yes	No	Yes	B	No	Yes	Yes	Yes	NM	No	No
Ballard et al[38]	Yes	Yes	Yes	B	NM	Yes	NM	NM	Yes	No	No
Engineer et al[41]	Yes	Yes	Yes	C	Yes	Yes	Yes	Yes	Yes	No	Adherence to the CDSS was almost 100% since: a standardized parent discussion tool helped guide parental expectations and the discussion between parents and the provider, and provider concerns were likely lessened by the utilization of a highly sensitive rule (PECARN) that has been externally validated and included a very large study population as well as role of leadership.
Sharp et al[42]	Yes	Yes	Yes	B	Yes	No	No	NM	Yes	No	Using clinical decision support may be critical to the success in reducing CT use, but the more nebulous effect of leadership, education, and other cultural factors requires further investigation.
Min et al[43]	Yes	Yes	Yes	C	Yes	Yes	NM	Yes	Yes	No	This CDSS constituted a “medium stop” intervention because they neither denied imaging for orders that did not meet appropriateness criteria (a “hard stop”) nor allowed imaging without an explanation (a “soft stop”). The authors emphasized on “gatekeeper effect” and the “education effect”
Dayan et al[39]	Yes	Yes	Yes	B	Yes	No	NM	Yes	Yes	No	the 3 patients with ciTBI in our study who were missed by the rule either had PECARN TBI rule factors that were inaccurately documented in the head trauma template or had histories that were concerning for child abuse (for whom the PECARN rules were not intended)
Bookman et al[44]	Yes	Yes	Yes	C	Yes	Yes	NM	Yes	Yes	No	Use of alerts should be used judiciously and in the appropriate environment
Ip et al	Yes	Yes	Yes	B	Yes	No	NM	NM	Yes	No	No
Gupta et al	Yes	Yes	Yes	B	Yes	Yes	NM	NM	No	No	No
Goergen et al	No	Yes	Yes	B	No	Yes	NM	Yes	NM	No
Sum: Yes No NM	10 1 0	10 1 0	0 11 0	A: 0 B: 8 C: 3 D: 0	4 3 9	8 3 0	2 1 8	6 0 5	8 1 2	0 11 0

Abbreviations: CDSS, clinical/computerized decision support system; CI, confidence interval; EHR, electronic health record; CPOE, computerized physician order entry; NM, not mentioned; PECARN, the Pediatric Emergency Care Applied Research Network.

^a Intervention Classification: “A” interventions provided information only; “B” interventions presented information on appropriateness or guidelines specifically tailored to the individual patient, often as a pop-up or alert. Some of these interventions also recommended alternative interventions, but did not include any barrier for the clinician to order the test; “C” interventions in general were similar to “B” interventions, but required the ordering clinician to justify with free text why they were overriding the decision support recommendation that a study was inappropriate (i.e., a “soft stop”). “D” interventions included a “hard stop,” meaning the intervention prevented the clinician from ordering a test contrary to the CDS determination of inappropriateness, until additional discussion with or permission obtained from another clinician or pathologist.

^b All studies recommended not to order a specific kind of imaging modality.

^c Not mentioned.

CDSS interventions were mostly in the form of an evidence-based rule providing knowledge about when it is appropriate to order the specified imaging, or predefined appropriateness criteria that physicians had to determine which criteria the patient met before ordering the imaging. The guidelines used as the knowledge base of the CDSSs were the CCSR,[21] NEXUS,[21] [42] New Orleans Criteria,[14] Pediatric Emergency Care Applied Research Network (PECARN),[35] [36] Canadian CT Head Rule (CCTHR),[40] and CT in Head Injury Patients Prediction Rule.[14] These interventions support physicians' informed decision-making in the first step of ordering process when they are deciding about ordering an imaging based on patients' indications.

Effects of Clinical Decision Support Systems on Outcomes

The included studies had mostly investigated “proportion of imaging” and “guideline adherence” outcomes. Generally, CDSS interventions showed positive effects on all outcomes ([Table 3]).

Table 3 Effects of CDSS interventions on laboratory testing outcomes

Outcome		Positive		No effect	Negative
Category	subcategory	Statistically significant	Demonstrated		Statistically Significant	Demonstrated
Imaging related	Proportion of imaging	[22] [38] [39] [40] [41] [42] [43] [44] [45]	[37]	[39] [a]		[42] [b]
Physician related	Guideline adherence	[22] [37] [40] [41] [42] [43] [44] [46]
	Knowledge	[37]
	Diagnostic yield	[22] [42]
Patient related	Patient complication/fractured detection			[40] [41] [45]		[38] [39]
	LOS			[38]		[39]

Abbreviations: CDSS, clinical/computerized decision support system; LOS, length of stay.

^a This study used CDSS at different emergency departments (EDs), some of which showed decrease in the computed tomography (CT) use and some showed no difference; however, the total CT rate was decreased

^b This study used CDSS at 13 different EDs, one of which showed increase in the CT use.

Imaging-Related Outcomes

All but one of the included studies have investigated the effects of CDSSs on the proportion of imaging. In general, studies showed positive impact on proportion of imaging. The reported proportion of imaging reduction varied from 2.6[37] to 40%[39] among the included studies. The study by Dayan et al[37] assessed as a moderate quality, investigated the impact of prediction rules through CDSS on traumatic brain injury CT rate in some pediatric and general emergency departments (EDs). Their finding showed small but inconsistent decreases in the EDs, as two pediatric EDs showed a significant but the modest decrease after CDSS implementation, but the other two pediatric EDs did not show significant change in CT rate. There was also little change in general EDs' CT rate. An explanation provided by the authors was that the general ED had low baseline CT use before CDSS implementation. Bookman et al[42] studied the impact of CDSS on the utilization of CT brain, C-spine, and pulmonary embolism. The results indicated significant decrease in CT brain and C-spine but no significant change in pulmonary embolism CT. They believed that previously implementing paper-based CDS for pulmonary embolism CT has resulted in no change after electronic CDS. Sharp et al[40] addressed effect of CDSS on head CT imaging. The results indicated an average decrease of 5% overall at 12 out of 13 EDs, one out of 13 EDs showed 0.4% increase in head CT use and the authors did not mention any special reason.[40] The impact of CDSS on CT cost was not reported in any of the included studies.

Across the five papers that provided analyzable data (odd ratio) from 26,791 patients in intervention and 43,440 patients in control groups ([Fig. 2]), CDSSs produced an average absolute improvement of 0.82% (95% confidence interval: 0.79–0.85%) in the proportion of CT scan utilization.

Fig. 2 Absolute improvements in the proportion of CT scan utilization. Results from five included studies that reported odd ratio are shown. The black diamond in the last row shows the summary overall absolute improvement and 95% confidence interval across the proportion of CT scan utilization. The squares with lines represent estimates and their 95% confidence intervals for each study. CT, computed tomography.

Physician-Related Outcomes

Physician-related outcomes, including guideline adherence, diagnostic yield, and knowledge, were reported in five included studies and all demonstrated positive impact of CDSSs. In the study by Goergen, et al,[38] guideline adherence, defined as the proportion of patients for whom the CDSS recommended no imaging and had no imaging, was 86%. In the study by Hynes et al,[21] the intervention indicated a 22.5% increase, in the proportion of request meeting the NEXUS or CCSR guideline. This study could also improve the detection of clinically significant C-spine injuries through plain radiograph. Zarchi[35] demonstrated that CDSS implementation can increase health care providers' knowledge regarding management of minor head trauma. Health care providers were also more likely to adopt the guideline after implementing CDSS.[35] In the study by Sharp et al,[40] the intervention could increase diagnostic yield of brain injuries. Diagnostic yield is defined as “the proportion of CT studies that identified radiographically significant findings,” e.g., a brain injury.[40] Gupta et al[44] showed 27% absolute and 56% relative adherence to the guidelines after implementing CDSS.

Patient-Related Outcomes

Four studies had addressed patient-related outcomes including missed or delayed diagnosis and LOS. In the study by Ballard et al,[36] a very low rate of missed diagnosis was observed due to the cancellation of CT examine according to the CDSS recommendation. The results showed one missed diagnosis of clinically important traumatic brain injury among 33 patients whose brain injuries were recognized. However, before implementing CDSS, all 37 patients with important injuries were detected by health care providers.[36] This study indicated a small nonsignificant increase in LOS; however, the authors stated that the analysis on LOS was limited due to the lack of data on other variables that could affect LOS suggesting no significant influence of the CDSS on the duration of ED evaluations.[36] Another study by Dayan et al[37] also indicated one missed traumatic brain injury diagnosis out of 56 children with minor blunt head trauma. Similar to the study by Ballard et al,[36] the one had a history of loss of consciousness and did not meet the PECARN very-low-risk criteria.[37] The finding showed increased LOS in seven of the eight intervention emergency departments by 7 to 15 minutes; the increase at only one of the EDs was statistically significant.[37] Opposite to these two studies, Ip et al[43] indicated that the rate of delayed diagnosis remained unchanged after the intervention. Likewise, Goergen et al[37] also revealed no delayed diagnosis of cervical spine injury.

Discussion

The majority of the included studies had investigated the CDSSs effect on imaging-related outcomes. Generally, the results showed improvement in imaging-related, physician-related, and patient-related outcomes. Most of the included studies were conducted after 2017, indicating a new research agenda in health information technology. It also indicates that attention to reducing patients' radiation exposure, as well as resource utilization for appropriate utilizing imaging have been increased recently. Most of the included studies were conducted in emergency departments indicating an opportunity to promote the emergency care value through reducing patients' length of stay, eliminating unnecessary imaging, and allocating limited time and resources of EDs to patients who may benefit more from it.[48] [49] The results of this review showed that CDSSs have the ability to improve imaging utilization of CNS in emergency departments.

Imaging-Related Outcomes

Appropriate imaging based on patient indication was positively affected by the CDSS rules. It is consistent with a systematic review by Main et al[50] who addressed the effectiveness of CDSS integrated in order communication systems on test/image ordering process. They found 9 out of that 13 studies which showed statistically significant improvement in imaging appropriateness, two reported an improvement without statistically significance, and two indicated no effect.[50] However, in Main and colleagues[50] review, most of the included studies had assessed the impact of illustrating tests charges or previous test results or using reminder, and only two studies providing recommendation like the included studies in our systematic review. Our results are also consistent with Goldzweig et al.[1] They examined the effects of EHR-based interventions on appropriate image ordering. Their findings showed that EHR-based interventions can moderately decrease inappropriate image ordering and decrease overall utilization of imaging by a small amount. Although, previous reviews[1] [51] found that hard-stop CDSSs is more effective than other interventions, most of the CDSSs in our study provided recommendation based on patient indication, and just in a few cases, physicians had to justify overriding the recommendation. Hard-stop CDSSs prevent physicians from ordering an imaging until a confirmation is reached from an external member like a radiologist. However, Min et al[41] suggested providing a “medium-stop” CDSS since it neither denies imaging for requests that did not meet appropriateness indications (a “hard stop”) nor allows imaging without a justification (a “soft stop”). Our results indicate that utilization reduction is more among high utilizers.[42] Therefore, it is suggested that before implementing the intervention, high imaging utilizers be recognized and considered as target group for intervening.

Physician-Related Outcomes

The studies reporting physician-related outcomes showed positive effects of CDSSs on adherence to the CDSS rules, physicians' diagnostic yield, and physicians' knowledge.[21] [35] [38] [40] [44] Similarly Main and colleagues[50] found that CDSSs can significantly improve health care provider performance. A systematic review by Roshanov and colleagues[52] showed positive impact of CDSSs on health care providers' diagnostic test ordering behavior. However, they believed that the contributing factors resulting in success or failure are unclear. Some effective factors for greater reductions in avoidable imaging use are mentioned in studies including “gatekeeper effect” which means “making the clinician more accountable for the imaging request” and the “educational effect” which means “increasing the number of physicians who are educated with respect to current guidelines,” as well as education, cultural factors, and role of leadership before and after pilot launch.[20] [39] [40] [41] A 100% compliance rate was reached in the study by Engineer and colleagues[39] with the aim to decrease inappropriate imaging for children. They found that patients/parents may consider CDSS as a restriction of what they perceive as clinically necessary. Therefore, designing a “standardized parent discussion tool” helped guide parental expectations, improve the discussion between parents and the practitioner, and lessen concerns as a result. Moreover, it is reported that a “codesigned” CDSS, considering physicians' insights during the development phase, can significantly improve compliance to the appropriate criteria.[53]

Patient-Related Outcomes

The results also indicated that CDSSs may make little or no difference to patient outcomes including patient complications, delayed diagnosis, or LOS.[14] [36] [37] CDSSs might lead to a small nonsignificant increase in LOS; however, the analysis on LOS was limited due to the lack of data on other variables that could affect LOS including indicators of ED crowding and throughput.[36] [37] Thus, the impact of CDSSs on LOS needs more investigation in future studies. However, the results need to be taken into account with caution; since in two out of four studies,[36] [37] there were few missed/delayed traumatic brain injuries. Only four studies examined patient-related outcomes and two of these showed the very little potential of CDSSs for missing clinically important injuries. Previous systematic reviews also reported limited evidence about the potential harm of CDSSs.[1] [26] [27] Similarly, the systematic review by Hunt et al,[24] on the CDSS impact on physician performance and patients' outcome, indicated that the CDSSs effects on patients' outcomes were not sufficiently studied. Thus, future studies need to investigate the patients' safety and possible harms of CDSSs.

The CCTHR[41] [42] [43] [44] and the PECARN[35] [36] [37] [39] have been the most extensively used criteria in the included studies. In a systematic review by Harnan et al,[23] the CCTHR was also the most widely used decision rule in the included studies. Harnan et al's findings also reported that CCTHR had a sensitivity of 99 to 100% for determining adults with a head injury. Although CCTHR is a highly sensitive rule for detecting injuries requiring neurosurgical intervention,[23] it is reported in the study by Stiell et al[22] that its paper-based implementation into clinical practice had led to an increase in head CT scan in the EDs. The probable reasons listed for this result included the use of simple, inexpensive, educational intervention, suboptimal compliance, and crowded emergency departments. As shown in our review, implementing CDSS can increase compliance by electronic intervening at the point of ordering a CT scan. To include most traumatic patients, CCTHR was used in combination with other appropriateness criteria in two of the included studies[42] [43]; for instance, CCTHR excludes patients with no loss of consciousness, whereas lack of consciousness is not an exclusion criterion in the CT in the Head Injury Patients Prediction Rule. In studies where multiple rules were used as appropriateness criteria, the rules were reviewed for overlap and were merged to maximize sensitivity. For instance, the CCTHR considers age above 65 years as a risk factor, whereas the CT in the Head Injury Patients Prediction Rule and the New Orleans criteria consider age above 60 years as a risk factor; in this case, utilizing a head CT for patients above 60 years old was considered in the CDSS implementation.[44] Engineer et al[39] and Zarchi et al[35] chose the PECARN because evidences report that PECARN has also achieved 100% sensitivity which is better than the Canadian Assessment of Tomography for Childhood Head injury and the Children's Head injury Algorithm for the Prediction of Important Clinical Events.[18] [54] Physicians' adherence to the PECARN is also high and medical staff expresses satisfaction in terms of PECARN usefulness and ease of use.[39] [55]

Strengths and Limitations

A comprehensive search strategy, without any time period restriction, was performed to find the maximum number of relevant studies. To avoid missing any important findings, a variety of interventional study designs were included. We assessed the effects of CDSSs not only on the imaging rate but also on physician- and patient-related clinical outcomes.

A limitation of this review is that due to exclusion of non-English language papers and conference proceedings, some relevant studies might have been missed. Another limitation is the exclusive focus on studies on reducing inappropriate imaging for CNS as the main outcome. Most studies conducted in this field had poor-to-moderate study design which may make the conclusion about the effects difficult due to possible biases. Moreover, it is important to note that most of the included studies were conducted in the United States where imaging is an examination that a clinician order; whereas, some countries may have a different approach for initiating imaging where the clinician presents the diagnostic problem and the radiologist decides whether there is an indication for imaging at all, and what modality and what protocol to use. This issue may influence the results of implementing CDSS, therefore having a rule set that is not constantly modified to accommodate new local protocols can imperil rather than improve good use of diagnostic imaging. Implementing CDSS where imaging is initiated from clinician to radiologist might lead to better results.

Future Research Directions

Since most of the included studies were conducted after 2017, indicating a new research direction, there is a need for more studies investigating effectiveness of CDSSs on the appropriate use of imaging. Moreover, considering the majority of the included studies had poor study design, there is an essential need for more robust study designs. According to the limited evidence on the possible harm of CDSSs and their influence on patients' safety, future research should evaluate these effects. The included papers did not investigate the economic impact of the CDSSs. Future studies are required to compare the cost of an examination to the cost of a missed diagnosis, because the cost of a missed diagnosis might exceed the saved cost of examinations; in addition, considering cost of developing, introducing, and maintaining the CDSSs would help realize the actual impact of CDSSs. Considering most of the CDSSs required manual data entry or providing the clarity necessary to assess each imaging request's adherence to guidelines, it can increase the workflow burden (manual data entry, additional screens, and mouse clicks were required to submit a head CT order) on the ordering physicians, these additional burnouts and time requirements or clinician satisfaction have not been measured in the included studies, suggesting a research direction. Moreover, details about implementing the CDSS interventions including the use of audit and feedback, user training, developers of CDSSs, and engagement of leader physicians were not reported. Reporting more details about implementing the CDSS interventions in the future studies may help produce a greater impact. As stated by Bowen and colleagues,[56] physicians perceived CDSS as a “nuisance,” qualitative researches regarding practitioners' attitudes toward CDSS design, and implementation may help more adoption of CDSS.

Conclusion

This systematic review reports that CDSSs decrease the utilization of CNS CT scan while increasing physicians' adherence to the rules. However, the possible harm of CDSSs to patients needs additional investigation. The actual effect of CDSSs on appropriate imaging would be realized when the saved cost of examinations is compared with the cost of missed diagnosis. As a suggestion, there is an essential need for further studies with more robust methodological designs like randomized controlled trials in this research area.

Clinical Relevance Statement

• Utilization reduction is more among high utilizers; therefore, identifying high imaging utilizers as target group for intervening, before implementing the CDSS intervention, can help improve outcomes.

• “Gatekeeper effect” and “educational effect” are two effective factors for greater reductions in avoidable imaging use in addition to cultural factors, and role of leadership before and after pilot launch.

• Although, hard-stop CDSSs is more effective than other interventions, most of the CDSSs in our study provided recommendation based on patient indication, and just in a few cases physicians had to justify overriding the recommendation.

Multiple Choice Questions

1. Which one of the appropriate imaging criteria or guidelines is not used in the included studies?

   • PECARN

   • OTTAWA

   • New Orleans

   • NEXUS

   Correct Answer: The correct answer is option b. The guidelines used as the knowledge base of the CDSSs were Canadian C-Spine Rule (CCSR)13, the National Emergency X-Radiography Utilization Study (NEXUS)13,36, the New Orleans Criteria 14, the Pediatric Emergency Care Applied Research Network (PECARN) 28,33, the Canadian CT Head Rule 29, and the CT in Head Injury Patients Prediction Rule 14.

2. What kind of the following considerations probably results in less reduction in imaging utilization?

   • Identifying high utilizers

   • Designing soft-stop CDSS

   • Considering gatekeeper effect

   • Educating physicians

   Correct Answer: The correct answer is option b. “Gatekeeper effect” and “educational effect” are two effective factors for greater reductions in avoidable imaging use in addition to designing hard-stop CDSS which make physicians justify overriding the recommendation. Moreover, it is reported that reduction of imaging utilization is more among high utilizers.

Conflict of interest

None declared.

Protection of Human and Animal Subjects

The study is approved by the ethics review board of the Vice-Chancellorship for Research Affairs of Kashan University of Medical Sciences which confirmed the study by the ethical code: IR.KAUMS.MEDNT.Rec.1396.095. Consent to participations is not applicable.

• References

• 1 Goldzweig CL, Orshansky G, Paige NM. et al. Electronic Health Record-Based Interventions for Reducing Inappropriate Imaging in the Clinical Setting: A Systematic Review of the Evidence. Washington (DC): Department of Veterans Affairs (U.S.); 2015
• 2 Friedman DP, Smith NS. Impact of a collaborative radiology utilization management program: does the specialty of the referring provider matter?. Am J Roentgenol 2016; 207 (01) 121-125
• 3 Global GBD. GBD 2016 Traumatic Brain Injury and Spinal Cord Injury Collaborators. Global, regional, and national burden of traumatic brain injury and spinal cord injury, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Neurol 2019; 18 (01) 56-87
• 4 Laughlin S, Montanera W. Central nervous system imaging. When is CT more appropriate than MRI?. Postgrad Med 1998; 104 (05) 73-76
• 5 Pitts SR, Niska RW, Xu J, Burt CW. National Hospital Ambulatory Medical Care Survey: 2006 emergency department summary. Natl Health Stat Rep 2008; (07) 1-38
• 6 Sharp AL, Nagaraj G, Rippberger EJ. et al. Computed tomography use for adults with head injury: describing likely avoidable emergency department imaging based on the Canadian CT head rule. Acad Emerg Med 2017; 24 (01) 22-30
• 7 Harms P. Automated usability evaluation of virtual reality applications. ACM Trans Comput Hum Interact 2019; 26 (03)
• 8 Sarma A, Heilbrun ME, Conner KE, Stevens SM, Woller SC, Elliott CG. Radiation and chest CT scan examinations: what do we know?. Chest 2012; 142 (03) 750-760
• 9 Lehnert BE, Bree RL. Analysis of appropriateness of outpatient CT and MRI referred from primary care clinics at an academic medical center: how critical is the need for improved decision support?. J Am Coll Radiol 2010; 7 (03) 192-197
• 10 Fitzgerald R. Error in radiology. Clin Radiol 2001; 56 (12) 938-946
• 11 Sheng AY, Castro A, Lewiss RE. Awareness, utilization, and education of the ACR appropriateness criteria: a review and future directions. J Am Coll Radiol 2016; 13 (02) 131-136
• 12 Graves JM, Fulton-Kehoe D, Jarvik JG, Franklin GM. Impact of an advanced imaging utilization review program on downstream health care utilization and costs for low back pain. Med Care 2018; 56 (06) 520-528
• 13 Tham E, Swietlik M, Deakyne S. et al; Pediatric Emergency Care Applied Research Network (PECARN). Clinical decision support for a multicenter trial of pediatric head trauma: development, implementation, and lessons learned. Appl Clin Inform 2016; 7 (02) 534-542
• 14 Ip IK, Gershanik EF, Schneider LI. et al. Impact of IT-enabled intervention on MRI use for back pain. Am J Med 2014; 127 (06) 512-8.e1
• 15 Deakyne SJ, Bajaj L, Hoffman J. et al; Pediatric Emergency Care Applied Research Network (PECARN). Development, evaluation and implementation of chief complaint groupings to activate data collection: a multi-center study of clinical decision support for children with head trauma. Appl Clin Inform 2015; 6 (03) 521-535
• 16 Rawson JV, Cronin P. Decision support: the super highway between health services research and change in clinical practice. Acad Radiol 2014; 21 (09) 1081-1082
• 17 Rousseau JF, Ip IK, Raja AS. et al. Can automated retrieval of data from emergency department physician notes enhance the imaging order entry process?. Appl Clin Inform 2019; 10 (02) 189-198
• 18 Easter JS, Bakes K, Dhaliwal J, Miller M, Caruso E, Haukoos JS. Comparison of PECARN, CATCH, and CHALICE rules for children with minor head injury: a prospective cohort study. Ann Emerg Med 2014; 64 (02) 145-152
• 19 Stiell IG, Clement CM, Rowe BH. et al. Comparison of the Canadian CT Head Rule and the New Orleans Criteria in patients with minor head injury. JAMA 2005; 294 (12) 1511-1518
• 20 Hoffman JR, Mower WR, Wolfson AB, Todd KH, Zucker MI. National Emergency X-Radiography Utilization Study Group. Validity of a set of clinical criteria to rule out injury to the cervical spine in patients with blunt trauma. N Engl J Med 2000; 343 (02) 94-99
• 21 Hynes JP, Hunter K, Rochford M. Utilization and appropriateness in cervical spine trauma imaging: implementation of clinical decision support criteria. Ir J Med Sci 2020; 189 (01) 333-336
• 22 Stiell IG, Clement CM, Grimshaw JM. et al. A prospective cluster-randomized trial to implement the Canadian CT Head Rule in emergency departments. CMAJ 2010; 182 (14) 1527-1532
• 23 Harnan SE, Pickering A, Pandor A, Goodacre SW. Clinical decision rules for adults with minor head injury: a systematic review. J Trauma 2011; 71 (01) 245-251
• 24 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346
• 25 Desai S, Liu C, Kirkland SW, Krebs LD, Keto-Lambert D, Rowe BH. Effectiveness of implementing evidence-based interventions to reduce C-spine image ordering in the emergency department: a systematic review. Acad Emerg Med 2018; 25 (06) 672-683
• 26 Liu C, Desai S, Krebs LD, Kirkland SW, Keto-Lambert D, Rowe BH. PRIHS-2 Choosing Wisely Team. Effectiveness of interventions to decrease image ordering for low back pain presentations in the emergency department: a systematic review. Acad Emerg Med 2018; 25 (06) 614-626
• 27 Jenkins HJ, Hancock MJ, French SD, Maher CG, Engel RM, Magnussen JS. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ 2015; 187 (06) 401-408
• 28 The Effective Public Health Practice Project (EPHPP). Quality assessment tool for quantitative. Accessed November 29, 2021: https://link.springer.com/content/pdf/bbm%253A978-3-319-17284-2/1.pdf
• 29 Higgins J, Altman D, Sterne J. Eds. Assessing risk of bias in included studies. In: Higgins JPT, Green S. eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. Accessed November 29, 2021: https://handbook-5-1.cochrane.org/
• 30 Jackson N, Waters E. Guidelines for Systematic Reviews in Health Promotion and Public Health Taskforce. Criteria for the systematic review of health promotion and public health interventions. Health Promot Int 2005; 20 (04) 367-374
• 31 Nabovati E, Vakili-Arki H, Taherzadeh Z. et al. Information technology-based interventions to improve drug-drug interaction outcomes: a systematic review on features and effects. J Med Syst 2017; 41 (01) 12
• 32 Collaboration TC. Review Manager (RevMan) [Computer program]. Version 5.3. Copenhagen: The Nordic Cochrane Centre. Accessed at September 12, 2019 at: https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman
• 33 Blackmore CC, Mecklenburg RS, Kaplan GS. Effectiveness of clinical decision support in controlling inappropriate imaging. J Am Coll Radiol 2011; 8 (01) 19-25
• 34 Doyle J, Abraham S, Feeney L, Reimer S, Finkelstein A. Clinical decision support for high-cost imaging: a randomized clinical trial. PLoS One 2019; 14 (03) e0213373
• 35 Zarchi TB. Intent of health care providers to adopt a clinical decision support tool in the management of minor pediatric head injuries. J Am Assoc Nurse Pract 2020; 32 (02) 168-175
• 36 Ballard DW, Kuppermann N, Vinson DR. et al; Pediatric Emergency Care Applied Research Network (PECARN), Clinical Research on Emergency Services and Treatment (CREST) Network, Partners HealthCare. Implementation of a clinical decision support system for children with minor blunt head trauma who are at nonnegligible risk for traumatic brain injuries. Ann Emerg Med 2019; 73 (05) 440-451
• 37 Dayan PS, Ballard DW, Tham E. et al; Pediatric Emergency Care Applied Research Network (PECARN), Clinical Research on Emergency Services and Treatment (CREST) Network, and Partners Healthcare; Traumatic Brain Injury-Knowledge Translation Study Group. Use of traumatic brain injury prediction rules with clinical decision support. Pediatrics 2017; 139 (04) e20162709
• 38 Goergen SK, Fong C, Dalziel K, Fennessy G. Can an evidence-based guideline reduce unnecessary imaging of road trauma patients with cervical spine injury in the emergency department?. Australas Radiol 2006; 50 (06) 563-569
• 39 Engineer RS, Podolsky SR, Fertel BS. et al. A pilot study to reduce computed tomography utilization for pediatric mild head injury in the emergency department using a clinical decision support tool and a structured parent discussion tool. Pediatr Emerg Care 2018;
• 40 Sharp AL, Huang BZ, Tang T. et al. Implementation of the Canadian CT Head Rule and its association with use of computed tomography among patients with head injury. Ann Emerg Med 2018; 71 (01) 54-63
• 41 Min A, Chan VWY, Aristizabal R. et al. Clinical decision support decreases volume of imaging for low back pain in an urban emergency department. J Am Coll Radiol 2017; 14 (07) 889-899
• 42 Bookman K, West D, Ginde A. et al. Embedded Clinical decision support in electronic health record decreases use of high-cost imaging in the emergency department: EmbED study. Acad Emerg Med 2017; 24 (07) 839-845
• 43 Ip IK, Raja AS, Gupta A, Andruchow J, Sodickson A, Khorasani R. Impact of clinical decision support on head computed tomography use in patients with mild traumatic brain injury in the ED. Am J Emerg Med 2015; 33 (03) 320-325
• 44 Gupta A, Ip IK, Raja AS, Andruchow JE, Sodickson A, Khorasani R. Effect of clinical decision support on documented guideline adherence for head CT in emergency department patients with mild traumatic brain injury. J Am Med Inform Assoc 2014; 21 (e2): e347-e351
• 45 Altman MR, Colorafi K, Daratha KB. The reliability of electronic health record data used for obstetrical research. Appl Clin Inform 2018; 9 (01) 156-162
• 46 Homco J, Carabin H, Nagykaldi Z. et al. Validity of medical record abstraction and electronic health record-generated reports to assess performance on cardiovascular quality measures in primary care. JAMA Netw Open 2020; 3 (07) e209411-e209411
• 47 Parsons A, McCullough C, Wang J, Shih S. Validity of electronic health record-derived quality measurement for performance monitoring. J Am Med Inform Assoc 2012; 19 (04) 604-609
• 48 Sierzenski PR, Linton OW, Amis Jr. ES. et al. Applications of justification and optimization in medical imaging: examples of clinical guidance for computed tomography use in emergency medicine. J Am Coll Radiol 2014; 11 (01) 36-44
• 49 Sharp AL, Cobb EM, Dresden SM. et al. Understanding the value of emergency care: a framework incorporating stakeholder perspectives. J Emerg Med 2014; 47 (03) 333-342
• 50 Main C, Moxham T, Wyatt J, Kay J, Anderson R, Stein K. Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems. In: NIHR Health Technology Assessment programme: Executive Summaries. Southampton, United Kingdom: NIHR Journals Library; 2010
• 51 Zare S, Meidani Z, Shirdeli M, Nabovati E. Laboratory test ordering in inpatient hospitals: a scoping review on the effects and features of clinical decision support systems. BMC Med Inform Decis Mak 2021; 21 (01) 20
• 52 Roshanov PS, You JJ, Dhaliwal J. et al; CCDSS Systematic Review Team. Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review. Implement Sci 2011; 6: 88
• 53 Brunner MC, Sheehan SE, Yanke EM. et al. Joint design with providers of clinical decision support for value-based advanced shoulder imaging. Appl Clin Inform 2020; 11 (01) 142-152
• 54 Babl FE, Borland ML, Phillips N. et al; Paediatric Research in Emergency Departments International Collaborative (PREDICT). Accuracy of PECARN, CATCH, and CHALICE head injury decision rules in children: a prospective cohort study. Lancet 2017; 389 (10087): 2393-2402
• 55 Bressan S, Romanato S, Mion T, Zanconato S, Da Dalt L. Implementation of adapted PECARN decision rule for children with minor head injury in the pediatric emergency department. Acad Emerg Med 2012; 19 (07) 801-807
• 56 Bowen S, Johnson K, Reed MH, Zhang L, Curry L. The effect of incorporating guidelines into a computerized order entry system for diagnostic imaging. J Am Coll Radiol 2011; 8 (04) 251-258

Publication History

Received: 26 July 2021

Accepted: 08 November 2021

Article published online:
12 January 2022