Who Watches the Watchers

Summary

The rise in the use of electronic health records (EHRs) and associated resources over the last decade is leading to the end of the paper medical record and all the risks associated with the use of a paper chart. However, there has not been a concomitant creation of a systematic oversight body that is specifically charged with ensuring the public’s safety through the use of EHR knowledge resource tools or EHRs themselves. We recommend the formation a Health Information Technology Safety Center. Such a center could collect error reports, review EHRs and the knowledge resources incorporated within them, and investigate particularly challenging EHR-related safety issues at participating health care delivery organizations. Safety issues could be identified, corrected, and the solutions widely disseminated.

Citation: Labkoff SE, Sittig DF. Who watches the watchers: working towards safety for EHR knowledge resources. Appl Clin Inform 2017; 8: 680–685 https://doi.org/10.4338/ACI-2017-02-IE-0032

The last century saw the growth of many complex industries such as air transportation, energy production and development of medicines. The public’s use and interaction with those industries had the potential to result in great harm through airline crashes, chemical leaks and explosions, and adverse drug events. In an attempt to ensure safety and protect the public, various governmental oversight agencies were formed at the federal level in the United States. For example, the Federal Aviation Administration (FAA) was formed in 1958 to ensure the safety of the flying public. The Federal Power Commission was formed in 1920 and its successor, the Federal Energy Regulatory Commission was formed in 1977, while the modern regulatory functions of the Food and Drug Administration (FDA) date to the 1938 passage of the Food, Drug & Cosmetic Act requiring that drugs be labeled with adequate directions for safe use.

Recently, the Office of the National Coordinator for Health Information Technology (ONC) clarified regulations that gave it oversight over the certification of EHRs [[1]]. Briefly, this new regulation “creates a regulatory framework for ONC’s direct review of health information technology (health IT) certified under the Program, including, when necessary, requiring the correction of nonconformities found in health IT certified under the Program and suspending and terminating certifications issued to Complete EHRs and Health IT Modules”. It is currently not clear how these regulations will be affected by the new presidential executive order on “Reducing Regulation and Controlling Regulatory Costs” [[2]]. Nevertheless, this new regulatory authority provides another tool for the Federal government to oversee corrections to certified health IT products, such as CDS-enabled EHRs.

With federal incentives of over $34 billion over the last 7 years, adoption of electronic health record (EHR) systems in the United States has soared to above 96 percent of US hospitals and 70 percent of office-based physicians [[3]]. To support this transition, ancillary clinical decision support (CDS) tools such as online medical compendia, medical calculators and real-time, interruptive clinical alerts (collectively CDS) have been developed and added to clinical work flows. CDS tools now provide both passive information and active alerts to improve adherence to clinical guidelines and inform clinicians of dangerous situations that might not have otherwise been caught – such as in-the-moment drug-drug interactions [[4]] or out-of-bounds critical laboratory values [[5]].

The utility of these tools is generally unquestioned [[6]]. However, EHR systems are commonly configured and often customized to fit the local needs of end users. Such flexibility adds to the diversity and complexity of these systems. When additional complexity is layered on systems that have such dependencies, the complexity increases. Testing each situation that arises based upon a user-configured option and/or a new system that depends on a different part of the EHR presents a tremendous software quality assurance challenge. Compounding the challenge is the dependence on external CDS knowledge resources that are updated on an irregular basis, resulting in a potentially monumental task to repeatedly and comprehensively test these systems and assure patient safety.

At the national level in the USA, as well as across several other countries around the world, there is relatively little oversight, governance or regulation concerning the veracity of CDS knowledge resources that provide data or recommendations used in EHR systems, or on EHRs themselves [[7]]. This issue was addressed in the 2014 FDASIA Health IT Report: Proposed Strategy and Recommendations for a Risk-Based Framework [[8]]. That report was prepared in response to Section 618 of the Food and Drug Administration Safety and Innovation Act (FDASIA), which required FDA, the Office of the National Coordinator for Health Information Technology (ONC) and the Federal Communications Commission (FCC) to develop a proposed strategy and recommendations on a risk-based regulatory framework for health information technology, including EHRs [[9]]. The report includes recommendations for change, but in general focuses mainly on not stifling innovation by introducing additional regulatory burdens on this growing field. However, there has been a slow, but steady stream of reports of safety issues within the field [[10]–[15]].

With regards to systems that feed data into EHRs, the task of testing and ensuring that data are accurate and up-to-date is left to the CDS content publishers, vendors of the EHR systems, and the health care delivery systems. Two recent studies exploring the accuracy of online CDS medical compendia demonstrated that errors often exist in these compendia [[16], [17]]. The degree of errors varies from the potentially insubstantial (e.g., errors in Patient Education material) to the potentially dangerous (e.g., mistakes in boxed warnings). An example of a CDS medical calculator producing erroneous risk scores was found recently embedded in an EHR system in England which affected thousands of patients [[18]]. At a meeting of the Pew Charitable Trust in December 2016, issues related to the creation of a Health Information Technology Safety Center were again discussed. And in January of 2017, the Centers for Disease Control and Prevention began to investigate various issues related to the use and safety of clinical decision support in a Request for Information (RFI) [[19]].

Issues surrounding CDS-enabled EHR use, user interfaces, system interoperability, and how the field will advance were the focus of the American Medical Informatics Association’s (AMIA) EHR 2020 report [[20]]. It resulted in a series of recommendations relating to user interface design, analysis of data, EHR certification, and how to continue to foster innovation. In line with this effort, we propose that the issue of the veracity of CDS knowledge resources for EHRs be added to the debate at the national level at forums convened by both medical and informatics organizations such as, for example, the American Medical Association (AMA), American College of Physicians (ACP), American Medical Informatics Association (AMIA), and Healthcare Information Management Systems Society (HIMSS).

The rapid rise in the use of EHRs and CDS knowledge resources over the last decade is leading to the end of the paper medical record and all the risks associated with the use of a paper chart. However, there has not been a concomitant creation of a systematic oversight body that is specifically charged with ensuring the public’s safety through the use of CDS knowledge resource tools or EHRs themselves. Given the complexity of these systems and their impact on patients, it is incumbent upon the industry to institute a means of proactive, self-oversight to ensure that these CDS-enabled EHR systems do not cause harm. The adoption of EHR-related safety recommendations for system testing [[21]], CDS veracity, technical product testing, and on-going monitoring of CDS reliability to detect malfunctions is long overdue [[22], [23]]. Because of the complexity and the speed of updates and changes to these computer-based tools in this 21^st century industry, it is not clear that a regulatory approach like those already in place at the FDA and other federal agencies is appropriate for this task.

Therefore, we support efforts to form a Health Information Technology Safety Center [[24], [25]]. Such a center could collect error reports, review EHRs and the CDS knowledge resources incorporated within them, and investigate particularly challenging EHR-related safety issues at participating health care delivery organizations. Safety issues could be identified, corrected, and the solutions widely disseminated. In addition, the Safety Center can become a center of excellence for helping to promote the use of standards for knowledge representation for CDS and other tools. Identifying an appropriate business model for such a Center is beyond the scope of this editorial. However, we strongly believe that the creation of such a dynamic, public-private partnership could help ensure that EHRs are both safe and effective in managing patient information and the clinical decision support knowledge resources that are embedded within them in the 21^st century and beyond.

Multiple Choice Question

The Office of the National Coordinator for Health Information Technology (ONC) has the authority to:

• Review and require corrective action on the part of EHR vendors in the event that they discover unsafe conditions.

• Change the way healthcare organizations implement and test their CDS-enabled EHRs in the event that they discover unsafe conditions.

• Change the way clinicians utilize their CDS-enabled EHRs to care for patients in the event that they discover unsafe conditions.

• Review and restrict usage of specific EHR screen design templates in the event that they discover unsafe conditions

Correct answer A) Recently released regulations clarify the role of the ONC in providing oversight of the EHR certification program which includes the ability to require corrective actions on the part of EHR vendors in the event that they discover unsafe conditions surrounding use of a particular brand of EHR. They do not have the ability to change the way healthcare organization use their EHRs although they have issued guidance in the form of the SAFER guides that can be used by healthcare organizations to improve their processes. The ONC cannot currently restrict usage of specific EHR design templates even if they are shown to be dangerous without further authority from the department of Health and Human Services (HHS) or Congress. Furthermore, they cannot change the way either healthcare organizations or clinicians use their EHRs.

Before implementing a new clinical decision support intervention, organizations should:

• Call reference sites to find out how much it really costs.

• Thoroughly test the CDS intervention with as many different combinations of patient data as possible.

• Create a communication strategy to inform the local news media of your accomplishment.

• Make a plan to review the appropriateness of the CDS recommendations after three years.

Correct answer B) Testing CDS interventions before implementation and after every change to the application software is critical to ensure safe and effective use of the system. While it is important to have a plan to reassess the clinical validity of every CDS intervention, this should be done on an annual basis not on a triennial basis. Similarly, while a communication strategy is important to inform clinical staff of the new CDS intervention, notifying the local media is not necessary or advised. Finally, while finding out the true cost of implementation for the intervention may be important, the cost pales in comparison to the potential negative impact of an erroneous CDS recommendation due to inadequate testing of the system.

Opinions offered by Drs. Labkoff and Sittig are not necessarily those of Purdue Pharma L.P. or the University of Texas Health Science Center at Houston.

Conflict of Interest

Dr. Labkoff is employed by Purdue Pharma, L.P., Dr. Sittig is employed by the University of Texas Health Science Center at Houston, TX. The authors have no other conflicts to report.

Human Subjects Protections

This editorial contains no patient data, therefore, it is not subject to Human Subject Research Approval

• References

• 1 Department of Health and Human Services. 45 CFR Part 170 RIN 0955-AA00 ONC Health IT Certification Program: Enhanced Oversight and Accountability. 19 October 2016 Available at: https://www.fed eralregister.gov/documents/2016/10/19/2016–24908/onc-health-it-certification-program-enhanced-oversight-and-accountability
• 2 The White House Office of the Press Secretary. Presidential Executive Order on Reducing Regulation and Controlling Regulatory Costs. January 30, 2017 Available at: https://www.whitehouse.gov/the-press-of fice/2017/01/30/presidential-executive-order-reducing-regulation-and-controlling
• 3 Office of the National Coordinator for Health Information Technology Website. Available at: https://dash-board.healthit.gov/evaluations/data-briefs/non-federal-acute-care-hospital-ehr-adoption-2008–2015.php Accessed 21 Nov, 2016
• 4 McEvoy DS, Sittig DF, Hickman TT, Aaron S, Ai A, Amato M, Bauer DW, Fraser GM, Harper J, Kennemer A, Krall MA, Lehmann CU, Malhotra S, Murphy DR, O’Kelley B, Samal L, Schreiber R, Singh H, Thomas EJ, Vartian CV, Westmorland J, McCoy AB, Wright A. Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc. 2016 Aug 28. pii: ocw114. doi: 10.1093/jamia/ocw114.
• 5 Kizzier-Carnahan V, Artis KA, Mohan V, Gold JA. Frequency of Passive EHR Alerts in the ICU: Another Form of Alert Fatigue?. J Patient Saf. 2016 Jun 22.
• 6 Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, Wing L, Kendrick AS, Sanders GD, Lobach D. Effect of clinical decision-support systems: a systematic review. Annals of internal medicine 2012; 157 (01) 29-43.
• 7 Magrabi F, Aarts J, Nohr C, Baker M, Harrison S, Pelayo S, Talmon J, Sittig DF, Coiera E. A comparative review of patient safety initiatives for national health information technology. Int J Med Inform. 2012 Dec 18. doi: pii: S1386–5056(12)00238–9. 10.1016/j.ijmedinf.2012.11.014.
• 8 FDASIA Health IT Report: Proposed Strategy and Recommendations for a Risk-Based Framework. 2014 ONC, FDA, FCC. Available at: https://www.healthit.gov/sites/default/files/fdasia_healthitreport_final.pdf Accessed 21 Nov, 2016.
• 9 United States. 112th Cong. 2cd sess. S. 3187 Washington: GPO, 2012. 3 January 2012. Available at: https://www.gpo.gov/fdsys/pkg/BILLS-112s3187enr/pdf/BILLS-112s3187enr.pdf
• 10 Health Information Technology Policy Comm.. Certification/Adoption Workgroup, Testimony Of Jeffrey Shuren, Director Of FDA’s Center For Devices And Radiological Health (25 February 2010). Available at: http://www.cchfreedom.org/pdfs/Health%20IT%20Deaths%20-%20FDA%20jeffrey%20Shuren.pdf Accessed 5 January 2014
• 11 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006; 13: 547-556.
• 12 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14: 415-423.
• 13 Meeks DW, Smith MW, Taylor L, Sittig DF, Scott JM, Singh H. An analysis of electronic health record-related patient safety concerns. J Am Med Inform Assoc 2014; 21 (06) 1053-1059.
• 14 Myers RB, Jones SL, Sittig DF. Review of reported clinical information system adverse events in US Food and Drug Administration databases. Appl Clin Inform 2011; 2: 63-74.
• 15 Menon S, Singh H, Giardina T, Rayburn WL, Davis BP, Russo EM, Sittig DF. Safety Huddles to Proactively Identify and Address Electronic Health Record Safety. J Amer Med Inform Assoc. 2017 (in press).
• 16 Randhawa AS, Babalola O, Henney Z, Miller M, Nelson T, Oza M, Patel C, Randhawa AS, Riley J, Snyder S. So SA collaborative assessment among 11 pharmaceutical companies of misinformation in commonly used online drug information compendia. Ann Pharmacother 2016; 50 (Suppl. 05) 352-359.
• 17 Talwar S, Randhawa A, Dankiewicz E, Crudele N, Haddox DJ. Caveat emptor: Erroneous safety information about opioids in online drug-information compendia. J Opioid Manag 2016; 12 (04) 281-288.
• 18 Matthews-King A. GPs told to review patients at risk as IT error miscalculates CV score in thousands. Available at: http://www.pulsetoday.co.uk/your-practice/practice-topics/it/gps-told-to-review-patients-at-risk-as-it-error-miscalculates-cv-score-in-thousands/20031807.fullarticle Accessed 10 January, 2017
• 19 Request for Information-Clinical Decision Support. Available at: https://www.fbo.gov/utils/view?id=82b0a8b1777f3c11599787d28bc544e7 Accessed 13 January, 2017
• 20 Payne T, Corley S, Cullen T, Gandhi T, Harrington L, Kuperman G, Mattison J, McCallie DP, McDonald CJ, Tang P, Tierney W, Weaver C, Weir C, Zaroukian M. Report of the AMIA EHR 2020 Task Force on the Status and Future Direction of EHRs. Available at: http://dx.doi.org/10.1093/jamia/ocv066ocv066 First published online: 28 May 2015
• 21 Wright A, Aaron S, Sittig DF. Testing electronic health records in the ,,production“ environment: an essential step in the journey to a safe and effective health care system. J Am Med Inform Assoc 2017; 24 (01) 188-192.
• 22 Wright A, Hickman TT, McEvoy D, Aaron S, Ai A, Andersen JM, Hussain S, Ramoni R, Fiskio J, Sittig DF, Bates DW. Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016; 23 (06) 1068-1076.
• 23 Sittig DF, Ash JS, Singh H. The SAFER guides: empowering organizations to improve the safety and effectiveness of electronic health records. Am J Manag Care 2014; 20 (05) 418-423.
• 24 “Health IT Safety Center Road Map.”. Available at: http://www.healthitsafety.org Accessed 11 December, 2016
• 25 Singh H, Classen DC, Sittig DF. Creating an oversight infrastructure for electronic health record-related patient safety hazards. J Patient Saf 2011; 7 (04) 169-174.