To Share is Human! Advancing Evidence into Practice through a National Repository of Interoperable Clinical Decision Support

Abstract

Background Healthcare systems devote substantial resources to the development of clinical decision support (CDS) largely independently. The process of translating evidence-based practice into useful and effective CDS may be more efficient and less duplicative if healthcare systems shared knowledge about the translation, including workflow considerations, key assumptions made during the translation process, and technical details.

Objective Describe how a national repository of CDS can serve as a public resource for healthcare systems, academic researchers, and informaticists seeking to share and reuse CDS knowledge resources or “artifacts.”

Methods In 2016, the Agency for Healthcare Research and Quality (AHRQ) launched CDS Connect as a public, web-based platform for authoring and sharing CDS knowledge artifacts. Researchers evaluated early use and impact of the platform by collecting user experiences of AHRQ-sponsored and community-led dissemination efforts and through quantitative/qualitative analysis of site metrics. Efforts are ongoing to quantify efficiencies gained by healthcare systems that leverage shared, interoperable CDS artifacts rather than developing similar CDS de novo and in isolation.

Results Federal agencies, academic institutions, and others have contributed over 50 entries to CDS Connect for sharing and dissemination. Analysis indicates shareable CDS resources reduce team sizes and the number of tasks and time required to design, develop, and deploy CDS. However, the platform needs further optimization to address sociotechnical challenges. Benefits of sharing include inspiring others to undertake similar CDS projects, identifying external collaborators, and improving CDS artifacts as a result of feedback. Organizations are adapting content available through the platform for continued research, innovation, and local implementations.

Conclusion CDS Connect has provided a functional platform where CDS developers are actively sharing their work. CDS sharing may lead to improved implementation efficiency through numerous pathways, and further research is ongoing to quantify efficiencies gained.

Keywords

clinical decision support - interoperability - computable guidelines - knowledge management - shared repository

Background and Significance

Translation of evidence-based practice into usable and effective clinical decision support (CDS) is time consuming and expensive.[1] Clinical practice guidelines and other evidence-based sources need to be carefully scrutinized and compared with one another, agreed upon by a clinical community, translated into highly specific and executable logic, and thoughtfully implemented within workflow and the local instance of an electronic health record (EHR).[2] [3] However, healthcare systems continue to develop CDS in isolation. It has been estimated that the duplicative effort of translating evidence-based practice into CDS for all recommended care across all healthcare systems in the United States collectively costs our healthcare system $25 billion.[4]

Depending on the audience, CDS as a concept often carries different meanings. CDS is not simply the end product of a translation or even implementation process. The Five Rights framework has been effective in communicating the overall purpose of CDS to a wide spectrum of audiences, including to patients and caregivers.[5] By describing CDS as a process that provides the right information to the right audiences using the right channels and formats at the right times during work flow, this framework emphasizes that CDS is not actually an app or any one “thing.” Rather, CDS is a process of many things, weaving together people, technology, and organizations toward a common goal of improved health and healthcare.

Effectively sharing CDS to eliminate duplicative effort across health care systems, therefore, requires sharing knowledge about the entire process. Recently, the Patient-Centered CDS Learning Network (PCCDS-LN) developed an Analytic Framework for Action that emphasizes learning and puts patients and families at the center of a cycle divided into prioritizing, authoring, implementing, and measuring CDS.[6] Each of these subprocesses is supported by tools and resources—reusable building blocks of CDS—that are shareable among healthcare systems.

CDS Connect (https://cds.ahrq.gov/cdsconnect) was conceived, developed, and launched as a publicly available platform to support such sharing. CDS Connect is part of a larger AHRQ initiative focused on CDS that began in 2016.[7]

Objectives

The goals of the AHRQ CDS initiative are to (1) advance evidence into practice through CDS and (2) make CDS more shareable, standards based, and publicly available. One component of the initiative, CDS Connect, functions as a national repository of CDS and serves as a public resource for healthcare systems, academic researchers, and informaticists seeking to share and reuse CDS knowledge resources or “artifacts.” The purpose of this article is to describe the early experience of CDS Connect, including the content of its repository, standards-based tools for CDS authoring, and use of the platform by a variety of users. The content presented here is based on a panel presentation given at AMIA's 2019 Clinical Informatics Conference (CIC) after which panelists were invited to submit this article. Other components of the AHRQ CDS initiative, including the PCCDS-LN, grant-funded activity, and evaluation plans, are described more fully at https://cds.ahrq.gov/.

Methods

CDS Connect is both a platform as well as a community of contributors and users supported by open-source tools for authoring, testing, and sharing interoperable CDS ([Fig. 1]). AHRQ contracted with the Centers for Medicare and Medicaid Services (CMS) Health Federally Funded Research and Development Center (Health FFRDC) operated by the MITRE Corporation to build and maintain CDS Connect. Central to the platform is a repository of CDS knowledge artifacts, which are discoverable on the website as entries with descriptive metadata. The term “artifact” refers to tools and resources (e.g., CDS software, implementation guides, reports on pilot testing, and other accompanying material) that serve as building blocks when evidence-based practice recommendations are translated into interoperable CDS. The repository was developed using Drupal, a free and open source content-management framework with strong support for authoring, searching, browsing, and engaging community stakeholders. Each artifact is tagged using the appropriate Medical Subject Headings (MeSH) terms, allowing users to easily explore artifacts.[8] CDS artifacts comprise entries of the repository and represent a range of resources—the building blocks of the CDS process previously described—that enable healthcare systems to translate evidence-based practice into CDS regardless of EHR developer or any other commercially available health information technology (IT) system. The platform leverages international standards for codifying the CDS wherever possible and disseminates information about the entire CDS process for adaptation in local healthcare systems.

Fig. 1 Clinical Decision Support (CDS) Connect concept of operations.

Central to this CDS process is the translation of evidence-based practice into interoperable CDS components, which requires careful consideration and successively more precise specification. The CDS Connect project followed a translation process that clinical informaticists often divide into four phases or levels[9] ([Fig. 2]). Each level corresponds to a potential set of CDS artifacts, for example, a structured representation of a guideline recommendation in the form of Health Level 7 (HL7) Clinical Quality Language (CQL) (Level 3). However, how a CDS developer arrived at each level is equally important. In migrating through the levels of translation and arriving at progressively more precise specifications, CDS developers—including the CDS Connect project—make a variety of assumptions, inferences, and other decisions. An account of this decision-making is critical for trustworthy sharing of CDS where healthcare systems must decide whether the same assumptions and inferences—both technical and clinical—are appropriate for their local environments.

Fig. 2 Levels of translation when developing interoperable clinical decision support.[9]

To demonstrate use of the platform and to seed it with CDS artifacts for public use, the project team undertook use cases in each of its first 3 years ([Table 1]). Use cases were designed to demonstrate the potential of different types of CDS technologies that are shareable across systems. For example, the project team developed a Substitutable Medical Applications, Reusable Technologies (SMART) on Fast Healthcare Interoperability Resources (FHIR) application for clinicians in year 2 but developed a CDS Hooks/CQL service that was integrated into a platform for patients in year 3. The project generated an array of artifacts at the conclusion of each use case demonstration that comprised an entry on the CDS Connect repository. Artifacts included Level 2 and Level 3 representations of the CDS, highly detailed user implementation guides that documented decision-making during CDS development, reports on pilot experiences, downloadable technical files (e.g., CQL), and links to primary sources of evidence and other external resources.

Table 1 Use cases adopted by the Clinical Decision Support (CDS) Connect project

Project year	1	2	3
Clinical domain(s)	Cardiovascular disease	Chronic pain management	Cardiovascular disease, diabetes, healthful diet, and physical activity
Guideline developer	US Preventive Services Task Force	CDC	US Preventive Services Task Force
Pilot partner	AllianceChicago	OCHIN	b.well Connected Health
EHR	GE Centricity	Epic	Multiple
CDS technology	Custom CQL-based CDS engine	SMART on FHIR	CQL Hooks
Provider environment	Community health center	Community health center	Multiple
Intended CDS target audience	Clinician	Clinician	Patient

Abbreviations: CQL, clinical quality language; EHR, electronic health record; FHIR, Fast Healthcare Interoperability Resource; GE Centricity, General Electric Centricity; SMART, Substitutable Medical Applications, Reusable Technologies.

In addition to the repository, the project team developed several pieces of software designed to help CDS developers and implementers create, test, share, integrate, and implement evidence-based, interoperable CDS into health IT systems. The CDS Authoring Tool was designed to help nonsoftware engineers build their own standards-based CDS logic using the HL7 CQL standard. The tool allows users to construct CDS logic statements, select and validate value sets and clinical codes through integration with the National Library of Medicine's Value Set Authority Center, and save artifacts for reuse or modification in their own user account. While the initial versions of the Authoring Tool advanced the concept of user-friendly CDS authoring, the tool continues to evolve to support the needs of the CDS community. Other software includes the CDS Connect application programming interface, a custom Drupal module that facilitates the upload of artifacts to the repository, as well as the CQL testing framework, which allows users to test the CQL they have developed to ensure the logic statements are constructed properly and return expected results when run against user-created test patients. [Table 2] lists additional software developed through the course of the project. All software is open source and freely available.[10]

Table 2 Open-source software tools developed by the Clinical Decision Support (CDS) Connect project for authoring, testing, sharing, and implementing interoperable clinical decision support

	CDS Authoring Tool	CQL Services	Application programming interface	Pain Management Summary	CQL Testing Framework
Purpose	A user-friendly web application for creating CDS logic using simple forms and exporting it as CQL that uses FHIR as its data model	A service framework that supports remote execution of CQL over a custom RESTful API or a HL7 CDS Hooks API	A Drupal 8 module that provides a RESTful API for uploading draft artifacts to the CDS Connect repository	A SMART on FHIR app that promotes informed pain management decision-making and can be integrated with EHRs that support the SMART platform	A testing framework that allows CQL authors to develop and run test cases for validating CQL-based CDS logic
Health standards used	HL7 CQL, HL7 FHIR, standard terminologies, integration with VSAC	HL7 CQL, HL7 CDS hooks, HL7 FHIR, standard terminologies, integration with VSAC	HL7 FHIR PlanDefinition	SMART on FHIR, HL7 FHIR, standard terminologies	HL7 CQL, HL7 FHIR, standard terminologies, integration with VSAC
Targeted user	Authors who are not familiar with CDS standards but are interested in developing standards-based CDS logic	Software developers and health IT system integrators	CDS developers	Clinicians	CQL developers
Open source license	Apache 2.0 https://github.com/AHRQ-CDS/AHRQ-CDS-Connect-Authoring-Tool	Apache 2.0 https://github.com/AHRQ-CDS/AHRQ-CDS-Connect-CQL-SERVICES	GPL 2.0 https://github.com/AHRQ-CDS/AHRQ-CDS-Connect-API	Apache 2.0 https://github.com/AHRQ-CDS/AHRQ-CDS-PAIN-MANAGEMENT-SUMMARY	Apache 2.0 https://github.com/AHRQ-CDS/CQL-Testing-Framework

Abbreviations: AHRQ, Agency for Healthcare Research and Quality; API, application programming interface; CDS, clinical decision support; CQL, clinical quality language; FHIR, Fast Healthcare Interoperability Resource; SMART, Substitutable Medical Applications, Reusable Technologies; VSAC, value set authority center.

To support the continued development of the platform and its associated tools, a public workgroup was established that provides input on features, functionality, prioritization, and future direction. The workgroup consists of over 140 volunteers representing more than 80 distinct organizations. Importantly, the workgroup represents multiple stakeholder perspectives, including EHR developers, CDS developers, patients, clinicians, payers, federal agencies, academic researchers, and others. The workgroup provides feedback that drives improvements to the site and sharing process.

Separate from the CDS Connect project, there has been an ongoing effort to quantify potential efficiencies gained from using shared, interoperable CDS. To effectively evaluate the contributions of shareable CDS resources, multiple factors need to be considered. Under contract with AHRQ, MedStar Health (MedStar) has been studying the difference between developing CDS using a healthcare system's own resources and tools versus developing CDS supplemented with artifacts and tools available on CDS Connect. The MedStar project has been framed to evaluate the CDS lifecycle in its current state (referred to as “isolated CDS build”) and future state (referred to as “shareable CDS resources build”) through the application of artifacts and the Authoring Tool available on the platform.

Four different healthcare systems participated in this comparison. The approach allowed for systematic comparisons and a rigorous evaluation to demonstrate within-site and between-site analyses to test the efficiencies of shareable CDS resources in multiple healthcare settings. Methods included stakeholder interviews, participatory data collection, process mapping, and usability testing to better describe and quantify efficiencies. While research suggests there is an ideal CDS process, there are many complications due to organizational differences.[11] Collectively, these analyses have been used as the strategic evidence necessary to build a business case that considers key aspects of an organization's clinical and business strategies.

Results

CDS Connect currently has over 50 entries corresponding to CDS developed and shared by academic institutions, private for-profit and private nonprofit organizations, and federal agencies ([Table 3]). The CDS artifacts cover a wide spectrum of clinical domains from highly specialized areas such as ophthalmology and neurosurgery to more generalized areas such as preventive medicine. L2 and L3 representations are the most common. The platform website has had over 75,000-page views since December 2017. CDS artifact documents have been downloaded from the repository over 5,000 times within the same time period. The CDS Authoring Tool has 175 registered users. Because the platform does not currently support tracking of artifacts following download (e.g., whether the artifacts have been operationalized elsewhere), knowledge about use of the artifacts come from individual contributors, explained further below.

Table 3 Overview of content on clinical decision support Connect repository

Contributors	Number of entries (as of November 2019)	Clinical domains	Types of clinical decision support
Agency for Healthcare Research and Quality	12	Cardiovascular disease, diabetes, pain management, preventive care	Data summary, event-condition-action rule, risk assessment
Centers for Disease Control and Prevention	7	Infectious disease, pain management	Event-condition-action rule, multimodal
Children's Hospital of Philadelphia	2	Preventive care, refugee health	Multimodal, order set
HLN Consulting, LLC	1	Preventive care	Multimodal
NORC at the University of Chicago/Yale University	1	Infectious disease	Calculator
RTI-UNC Evidence-Based Practice Center	2	Substance use	Alert, smart documentation form
University of Pennsylvania Health System	1	Infectious disease	Reference information
Veterans Health Administration	31	Endocrinology, mental health, neurology, neurosurgery, ophthalmology, preventive care, rheumatology	Event-condition-action rule, multimodal, order set, smart documentation form

Contributor Experience: Case Studies

One nongovernment author uploaded two artifacts to the repository soon after the site was first launched. These artifacts have been downloaded 174 times. One of these artifacts was published with the intent to disseminate, while the other was published as an experimental model with the hope of receiving feedback and identifying collaborators. Two sites (both academic sites with connections to large hospital systems and strong ties to refugee communities) have finished implementing the first artifact (The Refugee Health Module) with another two sites currently in progress. Numerous other sites are considering implementation and have provided feedback to make the artifact faster to implement and to match better with generic workflows while allowing for local customizations. One other site (another academic hospital based system) that began implementation was unable to complete the process largely due to organizational changes, namely, merging with another site. As part of the dissemination process, feedback was obtained from each implementation site, whether successful or unsuccessful. This feedback was used to modify the CDS artifact and the implementation guidance provided through CDS Connect. Feedback was felt to be generalizable and included: incorporation of standard clinical terminologies, development of a preimplementation checklist, and identification of points for local customization. Full details of this dissemination effort have been published elsewhere.[12]

The second artifact published by this author (The Healthy Weight Care Assistant) relied heavily on custom code and site-specific functions. It was published as a means to seek feedback and collaboration. Publishing this artifact on the repository resulted in two collaborative agreements with organizations and researchers looking to adapt or develop similar CDS products through commercial and academic pathways.

Another set of CDS authors has found success with disseminating through CDS Connect. Their artifact has been downloaded 153 times and has been successfully implemented at one outside site as an L2 CDS artifact. Feedback provided on this artifact led to a revised version of the CDS that was implemented locally and subsequently uploaded to the repository. The feedback received from the remote implementing institution included supporting alternative medications and medication allergies. Recognizing the potential of the site, this CDS artifact has also been encoded in a more mature state (Level 3) using CQL developed through the CDS Authoring Tool.

Evaluations to understand the role of shareable CDS illustrated changes between the current state and the shareable state of CDS resources. By aggregating the responses to compare within and between sites, the research team recognized many challenges in the evaluation of CDS builds—namely, significant differences based on the level of artifact structure, the type of CDS build, and the institutional priorities. Efficiencies were noted from stakeholders in the areas of reduced time for tasks, reduced team sizes, and fewer tasks to be completed as the outputs were already provided in the publicly available artifacts. However, not all the research sites noted the same gains in efficiency as indicated by stakeholder discussions and participatory data collection. Overall reduced time depends on the specific materials available in individual artifacts. Feedback from the stakeholders regarding overall impressions of the platform and the build process using the artifacts was positive, with suggestions provided for improved efficiency. All sites reported that the more executable, structured, and thorough the information contained in the artifact, the more helpful it was.

Discussion

CDS Connect is a national repository of publicly available, standards-based CDS knowledge resources designed to foster sharing among CDS developers and implementers. In its first 3 years, the platform has grown to support four different federal agencies and multiple external organizations seeking to share and reuse CDS.

Moving toward Shared, Interoperable CDS

The notion of sharing CDS is not new, and previous attempts have been met with varying success. AHRQ funded two large contracts with a common aim of demonstrating how CDS can become more systematic and replicable across sites and systems. The Clinical Decision Support Consortium (CDSC) developed a service-oriented approach to sharing several hundred CDS rules in a cloud-based system that served eight clinical sites using five different EHRs.[13] [14] The GuideLines Into DEcision Support (GLIDES) project created and demonstrated tools that were used to translate evidence-based guidelines into CDS in four different healthcare systems and that were used by four different medical societies during guideline development.[15] However, both the CDSC and GLIDES projects identified significant challenges from social and legal networks that are needed to facilitate sharing among healthcare systems to gaps in standards that formalize guidelines for implementation as CDS.

One area that has seen growth and adoption is the set of standards to represent CDS, its underlying logic and its exchange across different IT systems. HL7 FHIR has gained significant traction as a means of exchanging data to drive CDS systems, including data models to support that exchange.[16] HL7 CQL aims to harmonize logic that supports both quality measurement and CDS and that has been adopted by CMS in its quality reporting programs.[17] HL7 CDS Hooks and SMART on FHIR are increasingly being used to launch CDS as apps that are external to the host environment.[18] [19]

CDS Connect provides a glimpse of the computable guidelines of the future. Another federal agency, the Centers for Disease Control and Prevention (CDC), has an ongoing effort, Adapting Clinical Guidelines for the Digital Age, which aims to make it easy for clinicians to do the right thing by applying guidelines in practice more easily, quickly, accurately, and consistently.[20] Long-lag time, inconsistencies, and inaccuracies in translation from guideline development through implementation lead to a problem analogous to the “telephone game,” where multiple translations of guidelines add complexity, opportunity for error, and variation across sites or providers ([Fig. 3]). A multistakeholder group of experts from the entire guideline development and implementation continuum determined that by redesigning the process to include all relevant perspectives from the outset, the interoperable and shareable tools may be developed and disseminated alongside the guidelines. CDS Connect is a critical component of the strategy to provide reusable, vetted tools for implementers to be able to apply the scientific evidence in the recommendations more easily, quickly, accurately, and consistently. This strategy has the potential to improve health outcomes as well as improve efficiencies in the time and resources used across healthcare organizations to implement guidelines.

Fig. 3 Redesigning guideline development and implementation.

Sharing CDS through a national repository should be a beneficial experience for both the developer and the implementer. The developer of content is able to identify collaborators working on similar content and potentially create partnerships that would enhance their own CDS. There is also the opportunity to leverage implementer stories and refine the CDS locally. A national repository provides a publicly accessible and stable location for developers to use as a reference point for CDS, limiting the time they would need to spend on personal site maintenance activities. Finally, by preparing a submission for a national repository, the developer has an opportunity to consider all the lessons learned during the CDS development process and apply these internally to other projects.

CDS projects of interest are not unique among organizations. Implementers are often focused on the same or similar health conditions. This overlap has led to duplicative efforts across multiple sites with a waste of limited talent being asked to perform the same tasks. Artifacts published on CDS Connect aim to decrease these duplicative tasks and allow developers to focus on the true unique needs of their health system and/or innovative CDS. As requests for additional CDS within organizations continue to increase and development queues grow longer, a limiting factor in getting evidence into practice via CDS is a skilled workforce. While education and recruitment efforts are needed, reduction in duplicative tasks may help to leverage available workforce and hasten the incorporation of evidence-based CDS into EHRs. A vetted and unbiased repository, supported by standards that advance interoperability, may decrease organizational resistance to adoption of CDS designed externally. Additionally, as adoption of externally developed CDS becomes more of a common practice, it is hoped that organizational barriers to incorporation will also diminish.

Other options for sharing CDS exist, each with their own benefits and limitations and will continue to evolve. Many CDS developers use vendor repositories. This practice may be intentional or automatic as organizations are able to auto-populate these repositories from the clinical environment. These systems allow for direct importation of some CDS artifacts into an EHR system, but the vendor-supplied CDS often provide limited guidance, source attribution, search support, pathways for feedback, or lessons learned from implementation. Cloud-based CDS options also exist and are useful in numerous settings. However, these may be less customizable for site-specific workflows, and organizations have been traditionally averse to transmitting personal health information to cloud-based CDS servers and allowing these to directly trigger actions within the EHR, which has limited their reach. Some CDS developers have also had success with commercialization of CDS artifacts, such as order sets or documentation templates. CDS Connect allows for developers of content to assert ownership rights and therefore does not negate the rights of commercial CDS developers.

Creating a Business Case for using Shared CDS

The effort to quantify efficiencies using interoperable CDS can help healthcare systems construct a business case for using shareable CDS. MedStar's evaluation of the CDS lifecycle from its current state (isolated CDS build) to future state (shareable CDS resources build) through the application of CDS Connect artifacts demonstrated potential efficiencies gained in time and resources. The research team created metrics to characterize and evaluate processes and resources used during the CDS lifecycle. Qualitative metrics were linked to processes (e.g., testing, iterations) and quantitative metrics were linked to time (e.g., individual calendar time and team time) and resources (e.g., total personnel). The research team developed unique tools to collect data through stakeholder discussion and surveys for CDS design, development, and deployment. This resulted in a total of six tools that captured metrics related to the CDS lifecycle. A primary finding is the CDS builds vary significantly based on site, EHR vendor platform, and complexity of clinical condition being addressed. Therefore, comparisons between CDS build both within healthcare systems and between healthcare systems are challenging and represent an area that deserves further exploration. Results at the time of presentation at AMIA CIC indicated efficiencies vary based on the level of maturity and will likely affect its degree of sharing. CDS resources that are accompanied by value sets, implementation guides, and computable logic are more likely to increase efficiency and thus sharing between sites. Despite the presence of evidence-based guidelines, all sites spent a considerable amount of time verifying the credibility of the clinical evidence. Verification of the evidence occurred for several reasons: (1) participants wanted confirmation that the evidence originated from sources perceived as trustworthy and reputable, (2) verification was essential to ensure that the CDS reflected updated evidence, (3) verification confirmed important information such as patient population and clinical users, (4) verification confirmed that information matched local site workflows, and (5) verification ensured that nomenclature for CDS specifications matched the site-specific EHR and recommendations matched the practices of the local site.

Shareable resources also led to efficiencies during the development stage including a reduction in the number of iterations and team time spent on iterations. The deployment stage was not associated with noticeable efficiencies. However, not all the research sites noted the same gains in efficiency as indicated by stakeholder discussions and participatory data collection. Published reports indicate time and effort to develop shareable CDS are substantial but have resulted in improved efficiency during implementation at subsequent sites.[12]

Overall reduced time depends on the specific materials available in individual artifacts. It is noteworthy that efficiencies only came from the use of mature artifacts with structured and computer-interpretable code. Mature artifacts with comprehensive clinical and technical information, recent updates, and code with the reasoning behind it also produced greater trust. With regards to team makeup, at all sites, the roles were similar for all CDS builds, with all sites reporting some combination of clinical team members and technical team members to accomplish the CDS build. Regarding feedback from the development stakeholders, while team makeup for the shareable build was the same as their standard team makeup, the scale of the teams was reported to be smaller at some sites which may reflect the CDS artifact under consideration.

Feedback from the stakeholders regarding overall impressions of CDS Connect and the build process using the artifacts was positive, with suggestions provided for improved efficiency. All sites reported that the more executable, structured, and thorough the information contained in the artifact, the more helpful it would be. Although some CDS Connect artifacts provide detailed guides for the design and development phases in the form of evidence reviews and technical codes and wireframes, additional resources may be useful during deployment. Some of these resources include guides on technical and user testing, metrics to capture during testing, identifying the population of end users to target for training, training content and measuring training effectiveness, and tracking tool usage postdeployment.

Supporting a Trustworthy CDS Ecosystem

The concept of “trust” was identified by stakeholders in reference to the credibility of the information provided in the artifacts on CDS Connect. The more thorough and up to date an artifact was, the more credibility the evidence had, and as such, the more likely it was to be “trusted” by the clinical team members. Similarly, artifacts with comprehensive specification information and documentation were more likely to be “trusted” by technical team members.

It is important to acknowledge the importance of trust in a shared repository of interoperable CDS. There are multiple layers of trustworthy interactions among contributors, users, developers, implementers, and evaluators of shared, interoperable CDS, as well as the need to incorporate the perspectives of patients, families, and caregivers. Developed through a Trusted Use Framework Working Group, the PCCDS-LN recently released “recommendations for building and maintaining trust in clinical decision support knowledge artifacts.”[21] The framework identified 12 key factors for trustworthy interactions, nine areas of focus or “trust attributes,” and 33 recommendations for each trust attribute. The CDS Connect project has incorporated many of the recommendations and will continue to address others.

Conclusion

A new era of shared, interoperable CDS has begun thanks to advances in health IT standards and publicly available tools and resources. CDS Connect can be a central player in the dissemination of actionable knowledge, as identified by the CDC-led multistakeholder initiative on Adapting Clinical Guidelines for the Digital Age. We have highlighted the experiences of two healthcare systems who are already reaping the benefits of contributing to a repository of CDS knowledge resources, and we are quantifying efficiencies gained in CDS development for healthcare systems that reuse and adapt CDS developed elsewhere.

Clinical Relevance Statement

Clinical informaticists working in healthcare systems, technology companies, academic settings, and other environments will find a repository of publicly available CDS and associated open source tools at https://cds.ahrq.gov/cdsconnect. To facilitate trustworthy sharing of CDS knowledge and to build on each other's experiences, a community of contributors and users of the CDS repository and its tools exist. All are invited to participate.

Multiple Choice Questions

1. Sharing of CDS between sites has the potential to decrease variability of care between organizations. By allowing organizations to start CDS development by leveraging shared source material, publishing shareable CDS addresses has which of the following benefits (complete the following sentence): because shareable CDS is shareable,

   • Implementing organizations do not need to consider intellectual property restrictions since these are handled by the original CDS developer.

   • It will work the same for most organizations so there is no need to account for differences in workflow.

   • People working on similar projects can draw upon each other's work and focus on the innovative aspects of CDS development.

   • When you leave your organization you can take it with you and continue to optimal health care right away.

   Correct Answer: The correct answer is option is c. Publishing shareable CDS has many benefits including being an avenue for dissemination, reduction in waste during development, and potentially directly linking to executable source code. However, it is not the answer to every problem with CDS development. Shareable CDS can lead to collaborations and building upon each other's work would allow for optimization and innovation of the shared material (answer c, correct answer). CDS developers always need to be aware of intellectual property considerations, both of the original CDS and of the publishers of the evidence source (answer a). Shareable CDS may give implementation teams a leg up but do not address the local workflows or local variable definitions (answer b). Shareable CDS is shared between organizations and is not tied directly to your personal EHR profile, so it would not give you the opportunity to take it with you—however fun this would be (answer d).

2. An important reason for the limited availability of CDS capabilities is the application-specific and institution-specific nature of most CDS implementations. Which evaluation leverages a sociotechnical approach to understand complications due to organizational differences?

   • Evaluate healthcare system policies and procedures including flowcharts.

   • Use an ethnographic and economic approach evaluating personnel, resources, workflow, and usability considerations.

   • Evaluate the final CDS products to understand technical specifications across CDS modules.

   • Use an analytic approach to identify errors in CDS logic and implementation.

   Correct Answer: The correct answer is option b. Despite its potential, CDS implementation and actualization remain nascent due to the many barriers to realizing the full benefits of CDS-facilitated value improvement. Contributing factors include lack of shareable CDS content, absence of systematic means to validate content, technical difficulties of sharing CDS across institutions and EHR systems, and suboptimal user interfaces, implementation choices, and workflows. An ethnographic and economic approach evaluating personnel, resources, workflow, and usability considerations provides a well-rounded evaluation that considers sociotechnical elements of a system. Methods including stakeholder interviews, task analysis and process mapping, qualitative data collection from CDS designers and developers, heuristic evaluation, and think-aloud protocol can help identify structure and process challenges developing CDS in the absence of externally developed resources.

Conflict of Interest

E.A.L., G.M., and M.M. have nothing to disclose. K.M. reports funding from AHRQ. J.J.M. is the primary author of the Healthy Weight Care Assistant. This CDS artifact is noncommercial and he receives no financial benefits from this software. Funding for this software was provided through an unrestricted grant from the American Beverage Foundation. J.J.M. is the co-owner/author of the ADHD care assistant. This CDS artifact is noncommercial and he receives no financial benefits from this software. J.J.M. is the primary author/owner of NICUtrition.com, a neonatal nutrition calculator, and GEM Lite, a guideline translation support application. These software applications are noncommercial and he receives no financial benefits from ownership of this software.

Acknowledgment

The views expressed are those of the authors and do not necessarily reflect official positions of AHRQ or CDC. The authors would like to thank all members of the CDS Connect workgroup for their continued contribution and feedback on the project. The authors would also like to thank all team members for artifacts uploaded to CDS Connect for their contributions and efforts related to this project.

Protection of Human and Animal Subjects

The work described was performed in compliance with US Code of Federal Regulations 45 CFR 46 and was reviewed by Institutional Review Boards at MITRE and MedStar Health.

• References

• 1 Jacob V, Thota AB, Chattopadhyay SK. , et al. Cost and economic benefit of clinical decision support systems for cardiovascular disease prevention: a community guide systematic review. J Am Med Inform Assoc 2017; 24 (03) 669-676
• 2 Tcheng JE, Bakken S, Bates DW. , et al., Eds. Optimizing strategies for clinical decision support. Washington, D.C.: National Academy of Medicine; 2017
• 3 Masterson Creber RM, Dayan PS, Kuppermann N. , et al; Pediatric Emergency Care Applied Research Network (PECARN) and the Clinical Research on Emergency Services and Treatments (CREST) Network. Applying the RE-AIM framework for the evaluation of a clinical decision support tool for pediatric head trauma: a mixed-methods study. Appl Clin Inform 2018; 9 (03) 693-703
• 4 Johnston D, Pan E, Walker J, Bates DW, Middleton B. The value of computerized provider order entry in ambulatory settings. Boston: Center for Information Technology Leadership; Available at: https://www.researchgate.net/publication/7649275_The_value_of_CPOE_in_ambulatory_settings . Accessed January 8, 2020
• 5 Osheroff JA. , Ed. Improving medication use and outcomes with clinical decision support: a step-by-step guide. 1st ed. Chicago: HIMSS Publishing; 2009
• 6 Patient-Centered Clinical Decision Support Learning Network. Analytic framework for action. Available at: https://pccds-ln.org/analytic-framework . Accessed July 8, 2019
• 7 Agency for Healthcare Research and Quality. Patient-centered outcomes research clinical decision support: accelerating evidence into practice through CDS. Available at: https://cds.ahrq.gov/ . Accessed July 8, 2019
• 8 U.S. National Library of Medicine. Welcome to Medical Subject Headings. Available at: https://www.nlm.nih.gov/mesh/meshhome.html . Accessed July 8, 2019
• 9 Boxwala AA, Rocha BH, Maviglia S. , et al. A multi-layered framework for disseminating knowledge for computer-based decision support. J Am Med Inform Assoc 2011; 18 (Suppl. 01) i132-i139
• 10 Agency for Healthcare Research and Quality. Clinical Decision Support Repositories. Available at: https://github.com/AHRQ-CDS?tab=repositories . Accessed July 8, 2019
• 11 Bates DW, Kuperman GJ, Wang S. , et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 12 Orenstein EW, Yun K, Warden C. , et al. Development and dissemination of clinical decision support across institutions: standardization and sharing of refugee health screening modules. J Am Med Inform Assoc 2019; 26 (12) 1515-1524
• 13 Agency for Healthcare Research and Quality. Health Information Technology: Clinical Decision Support Consortium (Massachusetts). Available at: https://healthit.ahrq.gov/ahrq-funded-projects/clinical-decision-support-consortium . Accessed July 8, 2019
• 14 Middleton B. The clinical decision support consortium. Stud Health Technol Inform 2009; 150: 26-30
• 15 Agency for Healthcare Research and Quality. Health Information Technology: Guidelines Into Decision Support (Connecticut). Available at: https://healthit.ahrq.gov/ahrq-funded-projects/guidelines-decision-support-glides#h=glides . Accessed July 8, 2019
• 16 Health Level 7. Fast Healthcare Interoperability Resources (FHIR) Overview. Available at: https://www.hl7.org/fhir/overview.html . Accessed July 8, 2019
• 17 Centers for Medicare & Medicaid Services. Electronic Clinical Quality Improvement Resource Center: CQL – Clinical Quality Language. Available at: https://ecqi.healthit.gov/cql-clinical-quality-language#quicktabs-tabs_cql1 . Accessed July 8, 2019
• 18 Health Level 7. CDS Hooks. Available at: https://cds-hooks.hl7.org/ . Accessed July 8, 2019
• 19 Boston Children's Hospital Computational Health Informatics Program and Harvard Medical School Department of Biomedical Informatics. SMART: An App Platform for Healthcare. Available at: https://smarthealthit.org/ . Accessed July 8, 2019
• 20 Centers for Disease Control and Prevention. Deputy Director for public health science and surveillance: adapting clinical guidelines for the digital age. Available at: https://www.cdc.gov/ddphss/clinical-guidelines/index.html . Accessed July 8, 2019
• 21 Middleton B, Platt JE, Richardson J, Blumenfeld BH. Recommendations for building and maintaining trust in clinical decision support knowledge artifacts. Research Triangle Park, NC: Patient-Centered Clinical Decision Support Learning Network; 2018