Keywords clinical decision support alerting - order entry - quality improvement - clinical practice guidelines - ambulatory care
Background and Significance
Background and Significance
Public interest in testing for and treating vitamin D deficiency has increased.[1 ]
[2 ] While severe vitamin D deficiency affects bone health and has been studied in other disease processes (e.g., malignancy),[3 ] the benefit of screening the general population for vitamin D deficiency remains to be established.[4 ] Medical guidelines currently do not recommend screening average-risk individuals for vitamin D deficiency.[5 ]
Overuse of diagnostic laboratory testing contributes to the expense in modern medicine.[6 ] Health care systems increasingly utilize clinical decision support (CDS) to modify provider-ordering behavior and to assist in reducing the use of expensive treatment and diagnostic choices.[7 ] CDS has been shown to successfully lower inappropriate Clostridium difficile testing,[8 ] erythrocyte sedimentation rate assessments,[9 ] unnecessary imaging studies,[10 ] and duplicate laboratory testing.[11 ] CDS can reduce errors in order placement[12 ] and increase compliance with laboratory testing recommendations.[6 ]
Researchers lowered inappropriate vitamin D testing through reminding providers of vitamin D testing guidelines,[13 ]
[14 ] by requiring providers to select an approved indication for testing to proceed,[15 ]
[16 ] and by alerting providers if vitamin D testing had recently occurred.[17 ] We hypothesized that a sophisticated CDS that suppresses vitamin D testing alerts based on the diagnoses from the patient's chart would reduce inappropriate vitamin D testing.
Methods and Objectives
The Vanderbilt University Medical Center (VUMC) is an academic, tertiary care center in middle Tennessee, with clinics at the main campus and in several locations in and around Nashville, with approximately 2 million adults' ambulatory annual visit. A single–electronic health record system (Epic Systems, Verona, Wisconsin, United States) is used throughout the medical center.
To assess the efficacy of a clinical alert on inappropriate vitamin D testing, when ordering a 25-hydroxyvitamin D level, providers (physicians and midlevel providers) were assigned to receive (intervention) or not receive (control) an alert based on their unique VUMC provider-identification number. The decision support was triggered whenever a 25-hydroxyvitamin D assay was ordered in the outpatient setting. Upon entering the order, if the patient was at least 18 years of age and the provider was in the intervention group, the decision support compared the International Statistical Classification of Diseases and Related Health Problems (ICD-10) code(s) listed in the visit encounter to a list of conditions for which vitamin D testing is appropriate as per the Endocrine Society ([Table 1 ]).[5 ] If the ICD-10 code was matched to a diagnosis, no alert was triggered. If not, providers in the intervention group received a “pop up” alert suggesting that testing may not be appropriate. The alert was silent (not visible) for providers in the control group, which allowed for data collection for a comparison of how often providers in the control and intervention groups ordered 25-hydroxyvitamin D assays. [Fig. 1 ] contains a depiction of the CDS algorithm and the wording of the alert.
Table 1
Indications for vitamin D testing
Osteoporosis/osteopenia
Fracture (hip, wrist, or vertebral)
Osteomalacia/rickets/vitamin D deficiency
Chronic kidney disease/end-stage renal disease
Malabsorption (including gastric-bypass surgery, inflammatory bowel disease, celiac disease, and other disorders)
Hyperparathyroidism
Hypo- and hypercalcemia
Hypophosphatemia
Cirrhosis
Granulomatous disease (including sarcoidosis, lymphoma, tuberculosis, and other disorders)
Epilepsy
Human immunodeficiency virus infection
Obesity
Fig. 1 25-OHvitamin D = 25-hydroxyvitamin D. ICD-10, International Statistical Classification of Diseases and Related Health Problems.
During an 80-day period (October 24, 2018–January 12, 2019), collected data on all 25-hydroxyvitamin D assays ordered by providers included the presence or absence of a visible alert; completed orders; provider information such as role (attending, resident, midlevel provider, etc.), specialty (primary care or subspecialty), department, and location (clinic type); and basic patient demographic data. All 25-hydroxyvitamin D assays obtained throughout VUMC are processed at the central chemistry facility on the main campus. The direct, nonlabor expense to the laboratory for each 25-hydroxyvitamin D assay was $4.20 during the study period. The charge to the patient/payer was $179.00 per assay.
Encounters, during which a provider initiated the process of ordering a 25-hydroxyvitamin D assay, were the unit of analysis. Multiple order attempts during the same encounter (e.g., an order was started but then cancelled and then later reordered) were treated as a single event with the final outcome (order or no order during the encounter) recorded. The primary means of analysis was a Chi-square test utilizing 2 × 2 tables. The comparison of patients' ages utilized the Mann–Whitney U -test.
Results
Six hundred and sixteen outpatient providers initiated orders for a 25-hydroxyvitamin D assay during the 80 days, the decision support was active, with the total number of such orders per provider ranging from 1 to 154. Providers in the study group and the control group were similar, as were the patients whose providers were in the study or control groups ([Table 2 ]).
Table 2
Comparison of providers and patients by group
Intervention group
Control group
p -Value
Providers[a ]
322
294
Attending physicians (%)
150 (46.6)
116 (39.5)
0.074
Other providers (%)
172 (53.4)
178 (60.5)
Primary care providers (%)
118 (36.8)
102 (34.7)
0.593
Specialists (%)
203 (63.2)
192 (65.3)
Patients[a ]
2181
2032
White (%)
1597 (77.4)
1536 (79.1)
0.208
Non-White (%)
466 (22.6)
407 (20.9)
Male (%)
787 (36.1)
690 (34.0)
0.148
Female (%)
1394 (63.9)
1342 (66.0)
Mean age (y)
49.0
49.4
0.453
Note: % = percentage of group (intervention or control). Other providers = residents, fellows, midlevels providers, etc.
a Not all providers had data on specialty available and not all patients had data on race available.
During the 80 days, providers initiated 12,368 orders for 25-hydroxyvitamin D assays. Of those, 4,213 (2,181 for the intervention group and 2,032 for the control group) adult patients were not linked to an appropriate diagnosis in the patient's chart and considered potentially inappropriate. Providers in the intervention group receiving the alert completed the vitamin D order 89.9% of the time, compared with 98.1% for providers in the control group, who did not receive the alert (p < 0.001; [Table 3 ]).
Table 3
Results
Intervention group
Control group
p -Value
Patients
2,181
2,032
Order continued (%)
1,961 (89.9)
1,993 (98.1)
<0.001
Order discontinued (%)
220 (10.1)
39 (1.9)
Note: % = percent of effect on ordering (continued or discontinued).
In the intervention group, primary care providers who received the alert were more likely to discontinue vitamin D orders than subspecialists (order completed 88.0 vs. 90.9%, p = 0.031). In the intervention group, attending physicians discontinued the order equally as often as other providers such as resident physicians and midlevel providers (order completed 89.6 vs. 90.5%, p = 0.553; [Table 4 ]).
Table 4
Comparison of providers within the intervention subgroup
Order continued
Order discontinued
p -Value
Patients[a ]
1,961
220
Primary care providers (%)
692 (88.0)
94 (12.0)
0.031
Specialists (%)
1,264 (90.9)
126 (9.1)
Attending physicians (%)
1,296 (89.6)
150 (10.4)
0.533
Other providers (%)
665 (90.5)
70 (9.5)
Note: % = percent of effect on ordering (continued or discontinued). Other providers = residents, fellows, midlevels providers, etc.
a Not all providers had data on specialty available.
For the 1,961 overrides of the alert, the providers in the intervention group selected “does not meet criteria” in 1,016 cases (51.8%) or “treating separate illness” in 750 cases (38.2%), with the additional selection of “see comments” in 193 cases (9.8%). Two overrides had no reasons recorded. For the 591 written comments provided (30.1% of overrides), the majority of the comments related to acceptable reasons for testing not well captured by visit diagnoses (e.g., ethnicities at increased risk for deficiency) or acceptable reasons that had not yet been added as a visit diagnosis.
Assuming the control order completion rate (98.1%), 2,140 orders would be expected in the intervention group, instead of the recorded 1,961. Thus, during the 80 days, the alert reduced vitamin D assay orders by 179 in the intervention group. At a cost of $4.20 per assay to the chemistry laboratory, this intervention reduced laboratory expenditure by $752. Extrapolating the results, expanding this intervention to all VUMC providers would lead to an estimated annual laboratory cost reduction of $6,620. Ignoring the cost of pain and inconvenience caused by blood draws to the patients and inconvenience and delays to the providers, based on a charge of $179 per assay to patients/payers (though actual payment is frequently less), the intervention reduced patient/payer spending by $32,000. Expanding this intervention to all providers would result in an estimated savings to patients/payers of $282,000 annually.
Discussion
Implementation of an alert reminding providers that a vitamin D assay may be inappropriate unless the patient had a relevant diagnosis led to a reduction in vitamin D testing, with more than 8% of intended tests never ordered. However, nearly 90% of the time the alert was displayed, the providers ordered the test anyway. Based on calculating the savings/alert ratio (savings in patient/payer spending divided by the number of alerts to achieve the savings), on average each alert saved $14.70. However, looking at the actual laboratory costs of running the assay, the savings were $0.34 per alert.
Testing vitamin D levels has increased worldwide,[18 ] with one study reporting a 94-fold increase in testing over a 4-year period in one location.[19 ] The costs associated with such testing are substantial,[18 ] with recent efforts made to develop a methodology for assessing the degree of inappropriate vitamin D testing.[20 ] In the United Kingdom, approximately 70% of vitamin D testing in one instance was inappropriate based on indication,[21 ] with another such study in Canada identifying a similar frequency of approximately 65%.[22 ] Vitamin D testing occurs despite a lack of guidelines supporting testing for general-risk individuals[4 ]
[5 ] and the existence of scoring systems to identify high-risk individuals.[23 ]
Previous studies of CDS demonstrated a reduction in inappropriate vitamin D testing using a variety of strategies. Two studies reminded providers of conditions for which vitamin D testing is recommended at the time when a vitamin D level was ordered, reducing inappropriate testing from 43.8 to 30.3%[13 ] in one report and from 31 to 23% in another.[14 ] Several reports of an intervention in Canada that required providers to select a type of diagnosis from a list of conditions to proceed with testing both showed over a 90% reduction in the number of vitamin D tests ordered.[15 ]
[16 ] An inpatient alert in one report increased appropriate retesting of vitamin D from 40 to 64% by alerting providers if the vitamin D level had been checked recently.[17 ] Our study used a different method, with providers, receiving an alert only if the patient did not have an appropriate diagnosis for the ordering of vitamin D testing.
The provider response to our intervention was weak. Providers typically overrode the alert and continued with ordering the vitamin D assays. This pattern commonly occurs, with a recent review reporting the “positive predictive value” of such alerts ranging from 8 to 83%.[24 ] We considered the following five potential explanations in our setting:
We worded the alert as an educational message, potentially reducing the force of the reminder.
We placed our alert early in the ordering process, at the time when the vitamin D order was placed. As our system allows providers to add diagnoses after ordering a laboratory, many providers may have supplied an appropriate diagnosis after the order was placed. Most providers selecting “treating separate illness” and some of the written comments may support this explanation. Additionally, relevant diagnoses from previous visits are not always maintained in the problem list, potentially contributing to visit diagnoses being added late in the work flow.
Our study did not utilize a hard stop, which has been shown to be effective in reducing inappropriate vitamin D testing.[13 ]
[15 ]
[16 ]
[17 ] By simply alerting a provider that a test may be inappropriate, our alert served as a soft stop, likely accounting for the limited efficacy despite using a more specific trigger for displaying the alert than has been reported previously.
VUMC implemented its current electronic health record that started less than a year prior to this study. Not only did users have limited experience with this electronic health record, but multiple other electronic alerts remained in place from the time of its implementation, with outpatient providers receiving an average of 4.9 interruptive alerts per day.
Disagreement regarding appropriate vitamin D testing remains,[25 ] and VUMC providers may have disagreed with the provided indications for appropriateness.
In our analysis, comparing the number of times the alert was displayed and the overall financial savings, we created a savings/alert ratio that allows for a meaningful understanding of the financial impact of each display of an alert. While the actual cost saved to VUMC was negligible at $0.34 per alert, a displayed alert saved patients on average of $14.70. While $0.34 may be too low of a savings to warrant the interruption of a clinician by an alert, given the associated cognitive load and potential workflow disruptions,[26 ] a reduction of billed charges of $14.70 to patients for every alert displayed may warrant alerts. Of note, efforts to create and display the alert were shouldered by the organization, while patients were the beneficiary in a fee-for-service model. In an accountable, care model, VUMC and the patients' interests would have been better aligned. While our alert was not as effective as hoped based on the savings/alert ratio, we plan to improve savings by reformulating the message to be more directive and to reduce alerts by changing the firing of the alert to a point in the workflow where additional diagnoses are already recorded.
Strengths and Limitations
Strengths and Limitations
This study has several strengths worth noting. One is the large number of times the alert was displayed, allowing for confidence in the effect of the alert. Assigning providers and not patients to groups allowed us to not bias providers, who would have seen the alert in some patients and may have remembered it in others when not alerted. The control group allowed us to determine the effect of the alert. The diversity of providers included allowed for a comparison of the impact of an alert between primary care providers and specialists, as well as attending physicians, and other providers. By employing a soft-stop approach, provider autonomy was not eliminated in the care of patients.
This study has several additional limitations that must be detailed. By introducing an alert into a system already heavy with other alerts, this alert may have contributed to “alert fatigue.” By displaying the alert at the time the order was placed, the ordering provider may have been informed that an order might be inappropriate simply because the diagnosis had not yet been listed, potentially preventing appropriate vitamin D testing and also limiting our ability to determine the overall frequency of appropriate ordering of vitamin D. Adjusting the timing of this alert could address this problem.
Conclusion
In conclusion, we deployed a simple alert and demonstrated a measurable reduction in inappropriate 25-hydroxyvitamin D testing, though the timing of the alert likely limited its effectiveness. By quantifying the impact per alert (the savings/alert ratio), the financial impact of each display of an alert can be understood, and such a metric could be used as a point of comparison for the financial impact of disparate alerts and for decision making by governing bodies on retiring or maintaining alerts.
Clinical Relevance Statement
Clinical Relevance Statement
Inappropriate testing of vitamin D can be lowered through clinical decision support. To understand the impact of clinical decision support, the savings/alert ratio can be utilized.
Multiple Choice Questions
Multiple Choice Questions
When designing interruptive alerts for clinicians, one way to compare the overall effectiveness may be to
Determine how often the alert is overridden.
Measure the length of time the alert is displayed.
Measure institutional alert fatigue.
Determine cost savings per alert.
Correct Answer: The correct answer is option d. In this study, we were able to demonstrate the average savings per alert displayed providing a measure for its effectiveness.
Which of the following will NOT reduce ordering of unnecessary laboratory tests?
Removing the test from the orderable inventory catalogue.
Hard-stop alerts reminding the clinician that the test is not needed.
Soft-stop alerts reminding the clinician that the test is not needed.
Increasing the cost of the test.
Provider education.
Correct Answer: The correct answer is option d. While option “a” is self-explanatory (if it doesn't exist, it cannot be ordered) both hard- and soft-stop alerts have been shown to reduce ordering of tests. Education will have a short-term effect on ordering of tests.
