Joint Design with Providers of Clinical Decision Support for Value-Based Advanced Shoulder Imaging

Abstract

Background Provider orders for inappropriate advanced imaging, while rarely altering patient management, contribute enough to the strain on available health care resources, and therefore the United States Congress established the Appropriate Use Criteria Program.

Objectives To examine whether co-designing clinical decision support (CDS) with referring providers will reduce barriers to adoption and facilitate more appropriate shoulder ultrasound (US) over magnetic resonance imaging (MRI) in diagnosing Veteran shoulder pain, given similar efficacies and only 5% MRI follow-up rate after shoulder US.

Methods We used a theory-driven, convergent parallel mixed-methods approach to prospectively (1) determine medical providers' reasons for selecting MRI over US in diagnosing shoulder pain and identify barriers to ordering US, (2) co-design CDS, informed by provider interviews, to prompt appropriate US use, and (3) assess CDS impact on shoulder imaging use. CDS effectiveness in guiding appropriate shoulder imaging was evaluated through monthly monitoring of ordering data at our quaternary care Veterans Hospital. Key outcome measures were appropriate MRI/US use rates and transition to ordering US by both musculoskeletal specialist and generalist providers. We assessed differences in ordering using a generalized estimating equations logistic regression model. We compared continuous measures using mixed effects analysis of variance with log-transformed data.

Results During December 2016 to March 2018, 569 (395 MRI, 174 US) shoulder advanced imaging examinations were ordered by 111 providers. CDS “co-designed” in collaboration with providers increased US from 17% (58/335) to 50% (116/234) of all orders (p < 0.001), with concomitant decrease in MRI. Ordering appropriateness more than doubled from 31% (105/335) to 67% (157/234) following CDS (p < 0.001). Interviews confirmed that generalist providers want help in appropriately ordering advanced imaging.

Conclusion Partnering with medical providers to co-design CDS reduced barriers and prompted appropriate transition to US from MRI for shoulder pain diagnosis, promoting evidence-based practice. This approach can inform the development and implementation of other forms of CDS.

Keywords

clinical decision support - medical informatics - diagnostics - informatics

Background and Significance

Ordering advanced imaging with inappropriate indications does not yield more effective medical diagnosis.[1] [2] The Veterans Administration (VA) health care system oversees care to more than 4.5 million veterans age 65 or older[3] and typically evaluates shoulder pain—the second most common site of chronic joint pain[4]—using magnetic resonance imaging (MRI), putting a strain on limited MRI capacity.[5] [6] The costs of this modality are substantial; the VA spent nearly US$20 million on upper extremity joint MRI examinations in 2013.[5] For the most common injuries in this population, such as rotator cuff tears, studies have failed to find significant differences between the diagnostic capabilities of MRI and those of US, concluding that the two modalities are clinically comparable and that their use should be based on a range of factors including cost.[7] [8] [9] Shoulder MRI costs more than twice as much as shoulder US in Medicare patients, and substituting musculoskeletal US for MRI would save an estimated US$146.32 per shoulder examination.[10]

Despite the diagnostic efficacy and lower cost of US, altering provider ordering behavior is challenging, particularly given provider and/or patient preferences for higher-tech diagnostics and wider availability of MRI compared with musculoskeletal US.

Clinical decision support (CDS) alerts improve diagnostic ordering appropriateness.[11] Responding to this reality, Congress passed the Protecting Access to Medicare Act (PAMA) of 2014, which will fully implement on January 1, 2021, following 1 year of testing.[12] PAMA establishes a program whereby providers ordering advanced imaging such as MRI, computed tomography (CT), or positron emission tomography for Medicare beneficiaries must first consult a qualified vendor CDS delineating appropriate use criteria for this imaging.[12] Obtaining preliminary radiographs to exclude superfluous MRI in cases where severe arthritis is an obvious cause of pain, and appropriately substituting US for MRI as specified by a CDS tool, could reduce the 45% of shoulder MRI examinations ordered unnecessarily and still accurately diagnose 85% of all the discovered shoulder pathologies in this population.[5] Unfortunately, using CDS to limit unnecessary imaging has been variably effective, and it has proven difficult to define attributes associated with success.[1] [13] [14] [15] [16] [17] Providers tend to resist CDS, often ignore it, and sometimes game the system.[18]

“Co-designing” CDS tools with referring providers—incorporating their insights during the development phase–may improve compliance with evidence-based ordering over standard CDS creation while mitigating unintended consequences and/or barriers to implementation.[16] [17] [19] [20] [21] [22] [23] A recent systematic review with gap analysis recommended that models reflect the complex adaptive sociotechnical system of health informatics to mitigate implementation barriers.[24] [25] Therefore, rather than solely responding to provider perceived workflow barriers after implementation of a computerized CDS system,[2] we co-designed with referring providers a CDS tool to better guide both providers and their patients toward evidence-based, cost-effective practice.

Objectives

Here we outline a strategy to co-design CDS, primarily guided by the sociotechnical model.[26] Encouraging shoulder US over MRI as a clinically equivalent, efficacious, and less costly alternative,[5] [26] [27] we provide a case where co-designing CDS with referring providers reduced barriers to adoption and helped transition to more efficient shoulder imaging alternatives.

Methods

This study was approved by the University of Wisconsin Institutional Review Board. Verbal informed consent was obtained from providers electing to participate in the study interview process. The Institutional Review Board waived the need for patient informed consent.

Study Setting and Participants

The single site for this study was a 131-bed quaternary care VA facility with five associated community-based outpatient care centers that serves 130,000 veterans and performs 350–400 advanced shoulder imaging studies ordered by VA providers annually, almost exclusively outpatient MRIs. From December 2016 to March 2018, all patients with one or more advanced shoulder imaging orders placed at our institution were included in the study. Provider enrollment for study participation was voluntary. Providers were recruited through purposive sampling of those ordering the highest numbers of shoulder MRIs during the study period, with an attempt to sample diverse specialties. Diagnostic shoulder US could not be ordered through the electronic health record (EHR) prior to the initiation of this study. Two sonographers and one MRI radiologist were trained prior to and throughout the duration of the study (with others in both groups trained subsequently) by a musculoskeletal radiologist (S. S.) with 3 years of experience performing and interpreting diagnostic shoulder US.

Study Design

Our prospective, convergent parallel mixed methods approach[28] used two waves of semistructured qualitative interviews organized using the sociotechnical model[26] and considered alongside monthly, longitudinal quantitative ordering data aggregated from EHRs throughout the 16-month study period ([Fig. 1]).

Fig. 1 Mixed methods study design timeline. CDS, clinical decision support; MRI, magnetic resonance imaging.

Pre-CDS development: the study's preliminary phase (phase 1A) corresponded with the introduction of shoulder US as an available imaging modality. During this initial period, departmental policy requiring radiography prior to scheduling advanced shoulder imaging was enforced, and providers were educated about the appropriate indications for ordering shoulder US over MRI.

The initial interview phase (phase 1B) included the collection of qualitative provider interview data on their perceptions of CDS, exposure to our educational efforts, preferred educational outreach methods, and perceptions of the relative value of shoulder MRI versus US ([Supplementary Appendix A], available in the online version).

Twenty providers who had ordered three or more shoulder MRIs since initiation of phase 1 were e-mailed by a musculoskeletal radiologist requesting an interview. Of the 15 providers who responded—a 75% response rate—all agreed to interview. One-to-one interviews lasting 30 to 45 minutes of 15 providers with the highest monthly history of ordering MRIs to assess shoulder pain were conducted by an interviewer experienced in public health qualitative research. With consent, the interviews were audio-recorded for accuracy and transcribed. Several members of the project team independently read and coded the transcripts (according to a scheme developed by the team prior to interviews) to ensure faithfulness to the data for thematic analysis and resolved discrepancies by consensus of coinvestigators creating the CDS template. Because this was a formative evaluation, it was a descriptive rather than hypothesis-testing assessment.

Development of CDS alert: using a modified Delphi method,[29] a consensus panel including two primary care providers, an operational expert on CDS, two radiologists, a shoulder surgeon, and an interviewer reviewed thematic summaries and potential successful CDS characteristics that emerged from the interviews and proposed implications through short surveys. The panel's responses were summarized and then shared back with the group in another anonymous survey to finalize the CDS alert ([Fig. 2]). Once implemented, this alert window popped up whenever advanced shoulder imaging was ordered.

Fig. 2 Shoulder imaging clinical decision support electronic ordering interface.

Assess resistance to CDS: phase 2 began with CDS implementation 10 months after initiating the study. Five months into phase 2, we interviewed five providers (two of whom had participated in the initial interviews) to identify barriers to CDS adoption, explore providers' perceptions of MRI versus US, and gather suggestions for modifications to the CDS and/or educational program ([Supplementary Appendix B], available in the online version). Saturation occurred in that no additional relevant information came forth by the fourth and fifth interviews.

Quantitative outcome measures: ordering data of MRI and US for shoulder pain was monitored monthly to evaluate the effectiveness of the CDS. Endpoints were appropriate MRI/US use rates (primary) and study wait times (secondary). Appropriateness was determined retrospectively by the musculoskeletal radiologist for all advanced imaging orders during the study period through EHR review of symptoms and clinical concerns in conjunction with best literature recommendations and local orthopedic practice.[5] MRI was deemed appropriate for clinical concern of labrum injury/instability, occult fracture, neoplastic, or infectious etiologies. US was considered the appropriate first advanced imaging modality for all other indications. Use rates were compared for 10 months of provider education alone (phase 1) versus 6 months of education in combination with CDS (phase 2).

Statistical analysis: a priori assessment indicated adequate power to detect statistically significant differences in ordering behavior. We compared provider ordering rates at each time point using generalized estimating equations logistic regression models to account for a provider cluster effect. For continuous measures (waiting times), we computed summary statistics and compared them across time-points using mixed effects analysis of variance with log-transformed data, again accounting for a physician cluster effect. p-Value < 0.05 indicated a significant difference. All analyses were performed (by R. Z.) using statistical software R (version 3.41, R Foundation for Statistical Computing, Vienna, Austria).

Integration of the qualitative and quantitative data: the Delphi consensus panel was reconvened after the completion of phase 2, where results were discussed for the purposes of assessing efficacy of the CDS intervention and exploring unexpected findings.

Results

Qualitative Results

Multiple key thematic issues emerged across all phases: during educational outreach, both provider interview phases, and during the CDS development and implementation process ([Table 1]). Based in part on provider interview feedback, the Delphi panel determined that the CDS intervention should: (1) provide a clear explanation for radiography as the initial modality, (2) identify ultrasound's (US) strengths and limitations, (3) identify situations for which MRI is preferred as a first or second modality, (4) minimize the number of required keystrokes, and (5) suggest radiology e-consults (already existent within the institution to answer provider queries related to medical imaging) for further guidance. Application of these principles is shown in [Fig. 2], where the CDS interface concretely emphasized to the provider (and potentially the patient) the need for plain film evaluation prior to further imaging and the benefits of shoulder US over MRI (the number following the quote attributes it to an individual provider).

Table 1 Issues noted in CDS development with proposed response strategies using the sociotechnical model domain

Sociotechnical model domains	CDS development concerns	Response strategies
Hardware/software	CPRS would not allow for customization or decision-tree workflow requested by providers Difficult to place the link to the new CDS ordering menu to an intuitive location in CPRS	A presentation on shoulder imaging and a reference for shoulder physical examination were attached to the CDS announcement e-mail Education was embedded in CDS
Clinical content	Orthopedic Surgery and Rehabilitative Medicine Care Coordination Agreement initially required preconsultation MRI Unrealistic to expect to train primary care providers through short educational offering, VA educational TMS, or part of CDS on how to evaluate for labral tear or instability Several questions in the interview guide were difficult for interviewees to comprehend	It was modified to MRI or US in the shoulder pain evaluation decision tree This was deemed not clinically problematic in that it can be evaluated by a musculoskeletal specialist if symptoms persist Interviewer provided context
Human–computer interface	Identifying where to order a shoulder US in CPRS was unclear CDS unfamiliar term to interviewees Several providers suggested they do not want more e-mails, imposed outside education, embedded or available links to resources, or anything that removes them from or prolongs examination ordering	Operational expert on CDS updated the ordering menus to clarify Needed to define and provide an example of CDS for interviewees and emphasize that it does not represent preauthorization Radiology e-consult[a] process offered within CDS to assist in advanced imaging guidance
People	Difficulty getting providers together for education was problematic The clinical algorithm requiring radiographs assessing shoulder degenerative changes present in the existing Orthopedic Surgery and Rehabilitative Medicine Care Coordination Agreement was vague and therefore rarely enforced Orthopedic surgeons were more comfortable looking at MRI than USs Reluctant to travel extensively, the veterans were frustrated with their providers about having to travel once for shoulder X-ray and another time for advanced imaging	Education was embedded in CDS Radiologists had to come to a consensus and be educated on grading the various levels of degenerative change on plain shoulder X-rays to make use of the clinical algorithms Shoulder surgeon team member championed appropriate US CDS modified to demonstrate the importance for veteran's care
Workflow and communication	Education to majority community-based outpatient care providers logistically challenging Trainee education also sporadic due to frequent rotations Radiologist limited ”free” time concomitant with scheduled shoulder USs slowed their education CDS requirements: brief, requiring few extra keystrokes, easy to follow, embedded education Many providers would prefer on-site education by subject-matter experts	Education was embedded in CDS We added some shoulder US education to monthly VA radiology resident-internal medicine resident conferences Once the available USs increased after CDS, scheduling was adjusted to “batch” schedule them when radiologists could be proctored These attributes employed wherever possible Difficult to provide with current staffing model
Internal organizational policies, procedures, culture, environment	Orthopedic Surgery and Rehabilitative Medicine Care Coordination Agreement initially required MRI prior to consult–a huge potential barrier Radiology started enforcing at the start of the study (months 1–2) the need for shoulder radiographs available in the system prior to advanced shoulder imaging, which was part of the Orthopedic Surgery and Rehabilitative Medicine Care Coordination Agreement	Project delayed to have agreement modified and signed by various disciplines Provider-to-provider and CDS-embedded education on the need to enforce this existing requirement and reasoning behind it
External rules, regulations, pressures	Logistically difficult for veterans without local access to VA shoulder plain radiographs unless they are willing to travel or pay out of pocket (where available) Because of pressure from veterans desiring only a solitary visit to minimize copays, providers frequently failed to order radiographs in advance of the advanced imaging	CDS-embedded education on the need to enforce this existing requirement and reasoning behind it “Hard stop” instituted to enforce this existing requirement during phase 1A
Measurement and monitoring	Inability to reliably capture externally performed advanced shoulder imaging If the number of outside referrals changed significantly, it could have impacted our data analysis Providers tracked to have ordered multiple advanced shoulder imaging studies were assigned random numbers; their contact information was provided to a musculoskeletal radiologist for request to participate NVivo software has the capability to capture both social and technical elements through input “nodes” in addition to similar learned insights found during the interview process	While every attempt is made to import discovered prior imaging into radiology archives, this could be a confounder Similarly, while the VA tries to reduce fragmented care and keep most imaging internal, this could be a confounder These approaches maximized interviewee safety and minimized coercion Data tracking facilitated while maximizing interviewee safety and minimizing coercion

Abbreviations: CDS, clinical decision support; CPRS, computerized patient record system; MRI, magnetic resonance imaging; TMS, talent management system for ongoing education; US, ultrasound; VA, Veterans Administration.

^a Electronic virtual consult.

…it would be really great if when they want us to do something like that they could make something quick, easy and embedded right where we're using it … I can show my patient so I'm not fighting… (01)

Several providers suggested that a “flow chart” (07) or “algorithms” (08),(10) for shoulder imaging be embedded in the CDS,

It would really be helpful if there are ordering recipes, step-wise tools. That would be very helpful to me. Like a decision tree. (06)

particularly if it could expand to encompass inter-disciplinary collaboration with physical therapy.

I do an x-ray first and then send them to physical therapy and then, if symptoms persist, if they're still having significant symptoms I'll send them for an MRI. (13)

…Physical therapy, actually I use the physical therapist as a diagnostician in some respects or physical therapy assessment for more detailed exam. (14)

While some primary care providers, as is the case with all musculoskeletal specialists, were confident in their ability to triage shoulder pain patients based on their history and physical, this was by no means universal.

Honestly, I think a lot of providers of primary care are not confident of their physical exams…They may over-order because they just don't want to miss something. (09)

Although musculoskeletal specialists appeared more confident in their clinical evaluation of shoulder pain, they felt more comfortable independently reviewing MRI than US advanced shoulder imaging.

…they do not have as much education in reading ultrasounds so that is kind of a limiting factor for us (09)

Quantitative Results

A total of 465 unique patients (mean age: 59 years ± 14 [standard deviation]; age range: 23–93 years) had one or more shoulder MRI or US examinations ordered at our quaternary care Veterans Administration Hospital between December 2016 and March 2018. Of these patients, 429 (92.3%) were male, with a mean age of 60 years ± 14 (range: 23–93 years), and 36 (7.7%) were female, with a mean age of 50 years ± 13 (range: 31–73 years). Imaging orders were placed by 111 individual providers during the study period, 25 (22.5%) of which were musculoskeletal disease specialists such as orthopedic surgeons, rheumatologists, and sports medicine providers (orders per provider—mean: 7; standard deviation: 11.22; range: 1–44). The remaining 86 (77.5%) were primary care providers (orders per provider—mean: 4.5; standard deviation: 5.28; range: 1–27).

Co-designed CDS effectively changed provider ordering habits ([Table 2]), as US orders increased from 17% (58/335) to 50% (116/234) of all orders (p < 0.001) with a concomitant decrease in MRI ordering ([Fig. 3]), whereas overall adherence to appropriate ordering criteria more than doubled from 31% to 67% (p < 0.001). Providers who were not musculoskeletal specialists tended to alter their ordering habits toward US more than specialty providers following CDS introduction ([Table 2]), though this difference was not significant (p = 0.187). There was no significant change to the overall monthly rate of advanced imaging ordering following the introduction of CDS (p = 0.144).

Table 2 Results of quantitative analysis (n = 569 shoulder images ordered)

Variable	Pre-CDS phase 1 (0–10 mo)	Post-CDS phase 2 (10–16 mo)	Statistical analysis	p-Value
MRI ordered	82.7% (277)	50.4% (118)
US ordered	17.3% (58)	49.6% (116)	OR = 4.48 (95% CI: 3.00–6.69)[a]	<0.0001
Total	100% (335)	100% (234)
Appropriate order[b]	31.3% (105)	67.1% (157)	Chi-square (1 df) = 70.8[c]	<0.001
%US orders by providers
MSK specialists	17.2% (16/93)	40.2% (35/87)
Nonspecialists	17.4% (42/242)	55.1% (81/147)	OR = 1.47 (95% CI: 0.829–2.62)[d]	0.187
Average monthly orders	33.5 (335/10)	39.0 (234/6)	OR = 1.14 (95% CI: 0.955–1.37)[e]	0.144
Mean time difference–days elapsed from date desired to date completed (wait time)
MRI	28.2 (SD = 18.6)	29.4 (SD = 15.9)
US	28.1 (SD = 15.2)	33.6 (SD = 21.4)
Total	28.2 (SD = 18.0)	31.6 (SD = 19.1)	F (1,461 df) = 3.96[f]	<0.05

Abbreviations: CDS, clinical decision support; CI, confidence interval; MRI, magnetic resonance imaging; MSK, musculoskeletal; OR, odds ratio; SD, standard deviation; US, ultrasound.

^a Pre- to post-OR for odds of ordering US over MRI calculated using logistic regression (generalized estimating equations [GEE]) controlling for a provider cluster effect.

^b Based on established criteria (8).

^c Chi-squares based on all orders for shoulder images documented during study period.

^d Odds ratio for nonspecialty providers relative to MSK specialty providers calculated using logistic regression (GEE) controlling for a provider cluster effect. Overall pre- to post-OR for odds of ordering US over MRI remains significant, as shown in the first panel.

^e Pre- to post-OR of postrelative to pre-CDS for total monthly orders of advanced imaging calculated using logistic regression.

^f F ratio for time to completion based on total orders fulfilled, not withdrawn or modified (n = 463).

Fig. 3 Monthly trend of advanced shoulder imaging orders by modality.

The mean time from date desired to study completion (wait time) increased by 3.4 days or 12% from pre- to post-CDS (p < 0.05) ([Table 2]). Average wait times for MRI increased from 28.2 (±18.6) to 29.4 days (±15.9) and for US increased from 28.1 (±15.2) to 33.6 (±21.4). The proportion of cases for which MRIs were ordered after US was 7% (n = 20), with the majority ordered by musculoskeletal specialists for “operative planning.”

Discussion

In this study, a user-centered[30] interdisciplinary process was effectively used to develop, test, and implement a CDS alert. The co-designed CDS significantly improved adherence to appropriate use criteria for advanced shoulder imaging—particularly for nonmusculoskeletal specialists—even though the overall rate of advanced imaging did not drop. The latter occurred despite enforcing a requirement for shoulder radiographs prior to ordering advanced shoulder imaging at the initiation of this study. This more appropriate, but stable, volume closely mirrors a recent prospective, randomized evaluation characterizing the impact of CDS on outpatient advanced imaging orders.[31]

Although wait times for advanced shoulder imaging services increased during the study period, immediate access was available 2 months after the study conclusion. We believe that this initial delay was due to more rigid enforcement of antecedent shoulder plain films to reduce unnecessary studies, an unmet demand for appropriate studies and transitory start-up complications with training and routine use of US imaging. The transition to confident, independent practice of shoulder US for both technologists and radiologists required a significant threshold of proctored studies. Resolution of the delay for shoulder US and MRI coincided with this transition, and the improvement in MRI paralleled the shift from shoulder MRI to more timely US examinations. Educating providers on appropriate shoulder imaging and launching the co-designed CDS was challenging for multiple reasons ([Table 1]). In-person presentation to an assembly of ordering providers was hampered by geographical dispersion at six regional sites of service, as well as their varied educational preferences and limited virtual meeting time away from patient care. Point-of-care, one-on-one provider education was suggested as the best alternative rather than another “e-mail blast,” “educational link,” or anything that would significantly impede their clinical workflow. Furthermore, numerous logistical scheduling and administrative hurdles made virtual education more difficult. Essential education was therefore incorporated into the CDS, allowing for successful implementation. It was also challenging for the musculoskeletal radiologist to teach the US technologists and interpreting radiologists due to the duration of training required (2–3 months), shoulder US's steep learning curve, and the extra examination time it took to train in an environment favoring more rapid patient throughput. This contributed to a similar appointment time delay for US as MRI at approximately 30 days, eliminating the access time as a potential confounding factor influencing ordering.

Guiding providers to order an appropriate shoulder MRI was hampered by the disparity in the ability of musculoskeletal specialist and generalist skills to detect shoulder instability on physical examination. The clinical members of our research team, led by a shoulder orthopedic surgeon (J. O.), ultimately decided that missed signs and symptoms of shoulder instability would persist and eventually prompt referral to a musculoskeletal specialist with little clinical impact for those patients who had already undergone shoulder US. During interviews, providers expressed that a collaboration with physical therapy could serve a dual role—supplementing generalist musculoskeletal evaluation skills to inform more appropriate advanced imaging selection and initiating therapy without prerequisite advanced imaging.

A similar approach is typical in the United Kingdom's capitated system where physiotherapists are integral to shoulder physical diagnosis/triage, US for diagnosis, and injections.[32] Compared with expensive advanced medical imaging evaluation, early physical therapy initiation for musculoskeletal disorders appears to be a cost-effective alternative, but further study is certainly warranted, particularly for nonspinal musculoskeletal pain.[33]

A limitation of this study is the older Veteran patient population (mean age of 60), a group more suitable for US diagnosis of shoulder pain due to the high prevalence of rotator cuff tears. Thus, generalizability to younger populations more likely to have posttraumatic glenoid labral tears may be limited. Also, there is still some controversy regarding the depth of evidence supporting widespread change in practice to US diagnosis over MRI for surgical candidates with suspected rotator cuff tear.[7] As musculoskeletal US availability widens, further evidence will accrue defining US diagnostic accuracy beyond a single prospective cohort study.[8] Another limitation is this study's pre–post design without control, which has a lower degree of internal validity than a randomized controlled study. The latter will be more feasible once the VA completes its several-year universal conversion to a new EHR, concomitantly allowing time for shoulder US radiology education to become more widely disseminated throughout the VA. Similarly, replacing computerized patient record system (CPRS), the current EHR, will afford flexibility to better create actionable alerts that prefill appropriate study indications and facilitate prerequisite radiograph ordering. One cannot mitigate changes that occurred in the practice setting over the same time period, such as the prolonged training duration associated with US, or that substantial US adoption may relate to a novelty effect. However, given the magnitude of change over a short time period and the extensive qualitative information collected, the changed ordering patterns were attributed to the co-designed CDS. Longer-term data will discern whether provider education and radiology shoulder US training can be more effectively provided, whether these effects will persist, and whether this approach can be more widely disseminated throughout the VA and generalizable beyond this clinical setting.

Priorities for future research in this domain include (1) longer-term studies of imaging timeliness, (2) maintenance of the response to the CDS, and (3) expansion of the CDS to incorporate not only value-based shoulder imaging, but also timely physical therapy consultation and/or referral. Fundamental to this work will be the need to continually monitor the CDS to assure improved usability after successful migration into a more contemporary EHR, ongoing clinical applicability, and continued provider acceptance,[34] [35] while maintaining the user-centered interdisciplinary design approach that can benefit the goals of both targeted providers and the health care systems.[30] [36] Ultimately, if successful, this approach will be expanded into CDS imaging guidance for the diagnosis and nonsurgical treatments of other musculoskeletal disorders.

Conclusion

CDS co-designed with referring providers was an effective strategy to improve adherence to appropriate use criteria, leading to decreased use of MRI in favor of US evaluation of shoulder pain, with potential resultant cost savings.

Clinical Relevance Statement

Co-designing CDS with referring providers can effectively improve ordering behaviors while concisely educating providers and patients at the point of care. Implementing co-designed CDS increased appropriate ordering from 31% to 67% (p < 0.001). Generalist providers desire interdisciplinary assistance in appropriately navigating the advanced imaging ordering process.

Multiple Choice Questions

1. One important benefit of point-of-care education embedded within CDS is:

   • It allows the patient to read advanced imaging requirements.

   • It can be tailored for both generalist and specialist provider needs.

   • It eliminates interference with provider workflow.

   • It offers the provider advanced physical diagnosis skills tips.

   Correct Answer: The correct answer is option a. For patients desiring advanced imaging, the requirement for less sophisticated imaging can seem burdensome and provoke frustration with their ordering provider. Including requirements in the CDS tool shifts the target of frustration to the “rules” rather than the provider, particularly if the tool emphasizes the rationale and benefits for patient care. Answers b and d are incorrect because it is quite difficult to teach physical diagnosis skills through the EHR and provide a level of teaching that meets the needs of those with disparate levels of relevant clinical knowledge. Answer c is incorrect because inclusion within CDS might reduce educational time spent away from clinical encounters or minimize workflow impediments, but it cannot eliminate the latter.

2. One barrier to musculoskeletal specialists appropriately ordering shoulder US over MRI is:

   • MRI is more appropriate for evaluating shoulder pain in an older patient.

   • MRI alters clinical management frequently when it follows an US.

   • Musculoskeletal specialists are better at independent review of MRI than US.

   • US is usually more uncomfortable for the patient than MRI with similar costs.

   Correct Answer: The correct answer is option c. Musculoskeletal specialists have a long history of independently reviewing shoulder and other musculoskeletal MRI studies. US is more operator-dependent, with limited subsequent review after the patient encounter. Answer a is incorrect because US is actually more appropriate in an older population where labral pathology is uncommon and rotator cuff injury is common. The reverse occurs in a younger population where shoulder dislocation is often the traumatic cause. Answer b is incorrect because MRI after appropriate shoulder US is rarely indicated. In this study, the rate was 7% (most often for surgical planning), which is similar to the 5% rate frequently quoted in the literature. This older patient population rarely undergoes shoulder surgery compared with a younger cohort. Answer d is incorrect because it can be uncomfortable for a patient with significant shoulder pathology to maintain the required position long enough to obtain a diagnostic quality shoulder MRI. Shoulder US, while not pain-free, can be much more comfortable for the patient to undergo. In addition, MRI costs more than double shoulder US according to the Medicare Physician Fee Schedule.

Conflict of Interest

The authors declare that they have no conflicts of interest in the research. D.P.M. reports grants from Veteran's Administration through the University of Wisconsin-Madison during the conduct of the study. K.S.L. reports grants from National Basketball Association-General Electric and Mitek, personal fees from Elsevier, and nonfinancial support from Supersonic Imagine outside the submitted work. M.C.B. reports grants from the Department of Veterans Affairs HSR&D (PPO 15–178) and Exact Sciences, other from Elucent Medical during the conduct of the study. R.Z. reports personal fees from CUNA Mutual Group during the conduct of the study. D.J.V. reports personal fees from National Marrow Donor Program, CHEORS Inc., Novartis, BMS, Medical Decision Modeling Inc., and Evidera outside the submitted work.

Acknowledgments

The authors sincerely thank Abby Duane, Samantha Murphy, Hannah Ortiz, Lois Murphy, and James Combs Jr for their assistance in data collection and study operations. Julian Brunner's contributions to the conception and framework of the study are much appreciated. We thank the Research Office (and particularly Jenna Quinto) at the William S. Middleton Memorial Veterans Hospital for editorial support and proofreading.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was approved by the University of Wisconsin Institutional Review Board.

• References

• 1 Goldzweig CL, Orshansky G, Paige NM. , et al. Electronic health record-based interventions for improving appropriate diagnostic imaging: a systematic review and meta-analysis. Ann Intern Med 2015; 162 (08) 557-565
• 2 Chou R, Fu R, Carrino JA, Deyo RA. Imaging strategies for low-back pain: systematic review and meta-analysis. Lancet 2009; 373 (9662): 463-472
• 3 Office of the Assistant Deputy Under Secretary for Health for Policy and Planning. Washington, DC: VHA National Data Page FY 2018. Available at: https://vaww.va.gov/VHAOPP/enroll01/VitalSignsNational/enrolvsnat18.asp . Accessed July 27, 2018
• 4 United States Bone and Joint Initiative. The Burden of Musculoskeletal Diseases in the United States. Available at: http://www.boneandjointburden.org . Accessed July 27, 2018
• 5 Sheehan SE, Coburn JA, Singh H. , et al. Reducing unnecessary shoulder MRI examinations within a capitated health care system: a potential role for shoulder ultrasound. J Am Coll Radiol 2016; 13 (07) 780-787
• 6 Tashjian RZ. Epidemiology, natural history, and indications for treatment of rotator cuff tears. Clin Sports Med 2012; 31 (04) 589-604
• 7 Lenza M, Buchbinder R, Takwoingi Y, Johnston RV, Hanchard NC, Faloppa F. Magnetic resonance imaging, magnetic resonance arthrography and ultrasonography for assessing rotator cuff tears in people with shoulder pain for whom surgery is being considered. Cochrane Database Syst Rev 2013; 24 (09) CD009020
• 8 Teefey SA, Rubin DA, Middleton WD, Hildebolt CF, Leibold RA, Yamaguchi K. Detection and quantification of rotator cuff tears. Comparison of ultrasonographic, magnetic resonance imaging, and arthroscopic findings in seventy-one consecutive cases. J Bone Joint Surg Am 2004; 86 (04) 708-716
• 9 Rutten MJ, Spaargaren GJ, van Loon T, de Waal Malefijt MC, Kiemeney LA, Jager GJ. Detection of rotator cuff tears: the value of MRI following ultrasound. Eur Radiol 2010; 20 (02) 450-457
• 10 Centers for Medicare & Medicaid Services. Medicare Physician Fee Schedule. Available at: https://www.cms.gov/apps/physician-fee-schedule/ . Accessed June 21, 2019
• 11 Roshanov PS, You JJ, Dhaliwal J. , et al; CCDSS Systematic Review Team. Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review. Implement Sci 2011; 6: 88
• 12 Centers for Medicare & Medicaid Services. Appropriate Use Criteria Program. Available at: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Appropriate-Use-Criteria-Program/index.html . Accessed July 7, 2019
• 13 Georgiou A, Prgomet M, Markewycz A, Adams E, Westbrook JI. The impact of computerized provider order entry systems on medical-imaging services: a systematic review. J Am Med Inform Assoc 2011; 18 (03) 335-340
• 14 Ip IK, Lacson R, Hentel K. , et al. JOURNAL CLUB: predictors of provider response to clinical decision support: lessons learned from the medicare imaging demonstration. Am J Roentgenol 2017; 208 (02) 351-357
• 15 Huber TC, Krishnaraj A, Patrie J, Gaskin CM. Impact of a commercially available clinical decision support program on provider ordering habits. J Am Coll Radiol 2018; 15 (07) 951-957
• 16 Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Inform Assoc 2005; 12 (04) 438-447
• 17 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 18 Goehler A, Moore C, Manne-Goehler JM. , et al. Clinical decision support for ordering CTA-PE studies in the emergency department-a pilot on feasibility and clinical impact in a tertiary medical center. Acad Radiol 2019; 26 (08) 1077-1083
• 19 Sim I, Gorman P, Greenes RA. , et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001; 8 (06) 527-534
• 20 Miller K, Mosby D, Capan M. , et al. Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. J Am Med Inform Assoc 2018; 25 (05) 585-592
• 21 Bettmann MA, Oikarinen H, Rehani M. , et al. Clinical imaging guidelines part 4: challenges in identifying, engaging and collaborating with stakeholders. J Am Coll Radiol 2015; 12 (04) 370-375
• 22 Geeting GK, Beck M, Bruno MA. , et al. Mandatory assignment of modified wells score before CT angiography for pulmonary embolism fails to improve utilization or percentage of positive cases. AJR Am J Roentgenol 2016; 207 (02) 442-449
• 23 Bates DW, Kuperman GJ, Wang S. , et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 24 Kilsdonk E, Peute LW, Jaspers MWM. Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. Int J Med Inform 2017; 98: 56-64
• 25 Sittig DF, Ash JS. On the importance of using a multidimensional sociotechnical model to study health information technology. Ann Fam Med 2011; 9 (05) 390-391
• 26 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74
• 27 Freeman R, Khanna S, Ricketts D. Inappropriate requests for magnetic resonance scans of the shoulder. Int Orthop 2013; 37 (11) 2181-2184
• 28 Creswell JW. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. 4th ed. Thousand Oaks, CA: Sage Publications; 2014: 220-223
• 29 Delbecq A, Van de Ven A, Gustafson D. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. 1st ed. Glenview, IL: Scott, Foresman and Company; 1975: 83-107
• 30 Brunner J, Chuang E, Goldzweig C, Cain CL, Sugar C, Yano EM. User-centered design to improve clinical decision support in primary care. Int J Med Inform 2017; 104: 56-64
• 31 Palin TE, Sharpe RE, Shetterly SM, Steiner JF. Randomized clinical trial of a clinical decision support tool for improving the appropriateness scores for ordering imaging studies in primary and specialty care ambulatory Clinics. Am J Roentgonol 2019; 213: 1-6
• 32 Lumsden G, Lucas-Garner K, Sutherland S, Dodenhoff R. Physiotherapists utilizing diagnostic ultrasound in shoulder clinics. How useful do patients find immediate feedback from the scan as part of the management of their problem?. Musculoskelet Care 2018; 16 (01) 209-213
• 33 Ojha HA, Wyrsta NJ, Davenport TE, Egan WE, Gellhorn AC. Timing of physical therapy initiation for nonsurgical management of musculoskeletal disorders and effects on patient outcomes: a systematic review. J Orthop Sports Phys Ther 2016; 46 (02) 56-70
• 34 Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The value of monitoring clinical decision support interventions. Appl Clin Inform 2018; 9 (01) 163-173
• 35 Wright A, Ai A, Ash J. , et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2018; 25 (05) 496-506
• 36 Jones W, Drake C, Mack D, Reeder B, Trautner B, Wald H. Developing mobile clinical decision support for nursing home staff assessment of urinary tract infection using goal-directed design. Appl Clin Inform 2017; 8 (02) 632-650