System Configuration Evaluation for a Province-Wide Clinical Information System Using the eSafety Checklist

Abstract

Background According to Digital Health Canada 2013 eSafety Guidelines, an estimated one-third of patient safety incidents following implementation of clinical information systems (CISs) are technology-related. An eSafety checklist was previously developed to improve CIS safety by providing a comprehensive listing of system-agnostic, evidence-based configuration recommendations.

Objectives We sought to use the checklist to support safe initial configuration of a provincial system-wide CIS (Alberta, Canada), referred to as Connect Care.

Methods The checklist was applied to 13 Connect Care modules in three successive phases. First, the checklist was adapted to an abbreviated high-priority version. Second, demonstrations of each module were recorded. Finally, independent evaluation of each recording was conducted by two eSafety evaluators using the abbreviated eSafety checklist.

Results All modules achieved greater than 72% compliance, with an average of 84%. Overall, 273 opportunities for improvement were identified, with four major areas or themes emerging: (1) inconsistent date and time, (2) unclear patient identification, (3) ineffective alert system, and (4) insufficient decision support. These opportunities were forwarded to the appropriate build teams for review and implementation.

Conclusion This work is the first to utilize the eSafety checklist in a real-world CIS, which will become one of the largest in Canada. The checklist has shown clinical applicability in identifying gaps in CIS configuration and should be considered for use in future and pre-existing CISs.

Keywords

electronic health records - patient safety - informatics - information systems

Background and Significance

Clinical information systems (CISs) are imperative to delivering safe, quality care in the 21st century.[1] As patients engage with their local health care systems at various touch-points across the care continuum, CISs connect their health care journey, affording clinicians a holistic view of their treatment. Despite the numerous benefits of CISs, they can also introduce unintended patient care consequences if not configured and used safely. It is estimated that one-third of patient safety incidents following a CIS implementation are caused by its configuration and use.[2] [3]

For example, a recent issue was encountered at our organization where medication dosing bands were configured for a medication that was intended to be dosed by body weight. This caused several patients to receive a higher than recommended dose until it was caught and corrected. In another case, an intravenous medication was ordered without a duration or end date and was not configured to have a maximum safe dosing alert. This resulted in the patient receiving an unsafe excess amount of drug. Of course, many other failed checks in the care process contributed, but the system, if configured correctly, could have easily prevented these events.

These examples illustrate the importance of building safety and risk management into the specification and design of CISs.[4] The eSafety checklist developed by Dhillon-Chattha and colleagues provides a detailed listing of system-agnostic, evidence-based configuration recommendations that improve CIS safety.[5] [6] It contains 642 items organized into 10 common CIS system capabilities including: global settings; patient identification; clinical documentation; order management; clinical decision support; medication management; referral management; results management; clinical communication; and, patient portal ([Fig. 1]). Items are intended to identify system strengths and areas for improvement, which can then be prioritized and actioned by organizations according to their own patient safety policies. The tool can be used by organizations to support configuration of new CIS or to optimize existing systems. This article describes how the eSafety checklist was adapted and used by Alberta Health Services (AHS) to support safe configuration of their new system-wide CIS, Connect Care (core application developed by Epic Systems, Verona, Wisconsin, United States).

Fig. 1 eSafety checklist: an evidence based, system agnostic tool.

As a result of Connect Care, AHS will be Canada's largest province-wide, fully integrated health system responsible for delivering publicly funded health services to over 4.3 million residents. Connect Care is currently being implemented across AHS in nine phases (“waves”) with the first having launched in November 2019. Upon complete implementation, Connect Care will replace almost 1,300 disparate legacy software systems. A province-wide CIS will significantly improve continuity of patient care and provide a more holistic patient record for improved decision-making. Due to the size, scope, and complexity of this implementation, safety was identified as a key priority by AHS throughout the life cycle of this project. Thus, the eSafety checklist was used to identify high-priority safe configuration practices for consistent application across the various Connect Care modules, to be recommended to leadership and implemented prior to launch.

Objective

The objective of this project was to evaluate each of the 13 modules in the Connect Care application and identify misaligned system configurations according to the eSafety checklist, which could then be recommended for remediation. A secondary objective was to measure compliance of each module with the eSafety checklist and quantify patient safety risk of the identified misaligned system configurations.

The project consisted of three phases: (1) adaptation of the eSafety checklist to an abbreviated, high-priority listing, (2) a recorded video demonstration of the workflows and components in each Connect Care module, and (3) independent evaluation of each recording by two eSafety evaluators using the abbreviated eSafety checklist. An overview of the process is depicted in [Fig. 2].

Fig. 2 Flowchart of the process for applying the eSafety checklist to Connect Care configuration including three phases: (1) abbreviated checklist development, (2) module demonstration, and (3) module evaluation.

Methods

Development of an Abbreviated Checklist

Due to project timelines, a comprehensive evaluation of Connect Care configurations using the entire eSafety checklist was not a viable option prior to implementation. Therefore, an abbreviated version of the eSafety checklist was developed and used for evaluation prior to launching the first wave of Connect Care. Abbreviated checklist items were selected by AHS Human Factors specialists by prioritizing items that could be assessed through a general video demonstration and had high potential for human error and/or patient harm. The abbreviated eSafety checklist focuses on 169 items while still providing evaluators the ability to assess additional items, if deemed relevant to a given module. Evaluators were trained on use of the abbreviated checklist and were asked to familiarize themselves with it and the entire eSafety checklist prior to module demonstrations.

For reference, the abbreviated checklist is included in [Supplementary Appendix A]> (available in the online version; abbreviated eSafety checklist).

Module Demonstrations

Module demonstrations were provided by each of the module configuration leads to 13 eSafety evaluators, including 5 eQuality and eSafety specialists and 8 Human Factors specialists. Due to system security and access policy limitations during prelaunch of Connect Care, eSafety evaluators did not have access to a sandbox environment. Two evaluators were assigned to each Connect Care module and demonstrations were provided for each of the following modules: ambulatory (out-patient); anesthesia; cardiology; diagnostic imaging; emergency; in-patient documentation; in-patient orders; laboratory; obstetrics; oncology; patient portal; pharmacy; and surgery.

During each demonstration, evaluators were shown commonly performed clinical activities by a member of the build team and were provided opportunity to ask questions. Demonstrations were recorded via Skype so that further detailed review could be conducted by evaluators. The abbreviated eSafety checklist was completed primarily in reference to the recorded demonstration videos. Build teams remained accessible for questions and clarifications throughout the evaluation process.

Module Evaluations

Module evaluations were conducted in April 2019. The abbreviated eSafety checklist was applied independently by at least two evaluators per Connect Care module, from which configuration compliance and patient safety risks were identified. The recommended checklist configurations were first rated based on compliance. A technical description of the compliance ratings can be found in the “Instructions & Scoring” tab of the eSafety checklist ([Supplementary Appendix A]>, available in the online version). However, for the purposes of this evaluation, the following compliance ratings were used:

• Yes: the recommended practice was demonstrated and correctly configured.

• No: the recommended practice was demonstrated but was incorrectly configured.

• Partial: the recommended practice was demonstrated but was inconsistently configured.

• Not applicable: the recommended practice was not demonstrated.

Based on the above definitions, items were assigned a configuration compliance score of 1 for “yes,” 0 for “no,” and 0.5 for “partial” compliance. Items which could not be rated based on the available information were excluded to not affect the percent compliance calculations.

The likelihood of occurrence and impact on patient safety was also considered. Patient safety risk ratings were as follows:

• Minor: consequences are trivial. Alternatives may depend on personal preferences.

• Medium: consequences with significant user frustration and operational impact.

• Major: consequences with high patient safety risk or potential to impact more than one module.

• Red flag: relates to high frequency and/or high consequence patient safety risk which cannot be mitigated through design, training, or a workaround of some kind.

Completed checklists were compared across raters for each module. Any identified discrepancies in ratings were noted and re-evaluated in order to gain consensus agreement amongst evaluators. All partial and noncompliant items, along with evaluator rational, ratings, and notable examples were compiled into a table of Connect Care Application Opportunities (provided in [Supplementary Appendix B]>, available in the online version). Lastly, to further support build efforts, the identified opportunities were prioritized for implementation using the following definitions:

• Urgent: item of high frequency and/or high consequence patient safety risk; or impacts multiple modules requiring immediate action.

• Quick-win: patient safety risk item expected to be easily resolved.

• Postlaunch: unrelated to patient safety; to be addressed in future waves.

• Vendor review: issues requiring additional build review, content support, and/or training.

Results

The number of checklist items that were applicable and could be reviewed varied from one module to the next. Of 169 items, Ambulatory offered the most items, with over 80% of the abbreviated checklist items able to be assessed for compliance ([Table 1]). Some checklist items were not applicable during the review due to the current state of the module's build. However, this limitation did not reduce the validity of the eSafety checklist.

Table 1 Number of checklist items reviewed and resulting classification per module

Module	No. of items reviewed and assessed for compliance	No. of items compliant	No. of items partially compliant	No. of items noncompliant	No. of items reviewed and deemed not applicable (excluded)
Outpatient (Ambulatory)	145	125	16	4	24
Obstetrics (Stork)	142	119	15	8	27
Emergency (ASAP)	134	112	14	8	35
Oncology (Beacon)	126	94	20	12	43
Inpatient documentation (ClinDoc)	115	100	9	6	54
Inpatient orders (IP Orders)	110	88	16	6	59
Pharmacy (Willow)	102	74	14	14	67
Diagnostic imaging (Radiant)	95	73	13	9	74
Surgery (OpTime)	91	73	10	8	78
Patient portal (MyChart)	86	57	10	19	83
Cardiology (Cupid)	85	65	8	12	84
Anesthesia	79	64	6	9	90
Laboratory (Beaker)	69	51	10	8	100

Overall Compliance

Of the modules reviewed, all were more than 72% compliant. On average, modules achieved 84% configuration compliance. Ambulatory (92%), Obstetrics (89%), and Emergency (89%) received the highest configuration compliance while, Patient Portal (72%), Pharmacy (79%), and Oncology (81%) were identified as having the lowest configuration compliance ([Fig. 3]).

Fig. 3 Connect Care configuration compliance with the eSafety checklist.

Identified Opportunities

Across the 13 modules demonstrated, 273 opportunities for improvement were identified. Given the overall compliance scores, it is unsurprising that Oncology, Patient Portal, and Pharmacy also had the greatest proportion of configuration opportunities ([Fig. 3]).

Major Areas of Inconsistency

An analysis of the results of the eSafety checklist identified four major areas of inconsistency. These areas cover noncompliant items which were viewed across multiple modules and were identified as a “major” patient safety risk. The four areas include (1) inconsistent date and time, (2) unclear patient identification, (3) ineffective alert system, and (4) insufficient decision support.

These areas are discussed in detail below, with supporting examples. Where applicable, checklist items are noted in parentheses.

Inconsistent Date and Time

Date and time were not expressed in a consistent format and placement (eSafety checklist items 1.1.7 and 1.1.8). Additionally, the month was often not spelled out or abbreviated, as it is recommended to be 2-digit day, alphabetical abbreviated month, and 4-digit year (1.1.9). For example, patient date of birth was displayed with 1 digit month/day, 4-digit year, while orders were dated 2-digit month/day and 2-digit year ([Supplementary Fig. S1], available in the online version). There were also inconsistencies in use of 12 hours (hh:mm) as opposed to 24-hour (hhmm) times ([Supplementary Fig. S2], available in the online version).

[Supplementary Fig. S2] (available in the online version) shows an example of the correct alphanumeric dating format (green box); however, in other places in the system it was displayed inconsistently with single-digit day or nonabbreviated month ([Supplementary Fig. S3], available in the online version).

Unclear Patient Identification

Patient names appeared truncated (2.1.11) and information required to accurately identify a patient was not always clearly displayed or consistently placed on screens where patient care documentation occurs (2.3.1). An example of name truncation is provided in [Supplementary Fig. S4] (available in the online version). In another example, color contrast (grey on black) resulted in poor legibility of relevant patient information ([Supplementary Fig. S5], available in the online version).

The location of patient information was also inconsistent across modules. Some had a “patient storyboard” (sidebar), while others did not. Additionally, minimal alerting was provided when accessing the records of individuals with similar or sound-alike names (2.6.2). An alert does present for patients with similar and sound-alike names, however only as a mouse over label, and there is a lack of visual distinctive features consistent with warnings ([Supplementary Fig. S6], available in the online version).

Ineffective Alert Systems

Within the module, standard rules are not used for colors of severity warnings and alerts (1.1.26). Additionally, color, shape, and size are not always properly manipulated to make visual alerts distinct from one another (5.2.7), nor do they always offer potentially appropriate actions in response (5.3.10). For example, there was noted inconsistent use of color to indicate high severity warning and alerts within the same module; sometimes red was used, other times orange ([Supplementary Fig. S7], available in the online version).

System-generated alerts are triggered at the time of order submission, not at order entry (4.5.1). An example of this is shown in [Supplementary Fig. S8] (available in the online version) where a drug–drug interaction alert appears at the time of signing as opposed to the time of entry.

Medication warnings were also found to be embedded in text with no visual hierarchy, making them unapparent and easily missed ([Supplementary Fig. S9], available in the online version). Some alerts did not have any provided options to quickly resolve issues, requiring the user to back track their actions ([Supplementary Fig. S10], available in the online version).

Insufficient Decision Support

Clinical decision support was highly variable across modules. In some instances, users were required to remember important information from one page to use on another page (1.6.2). As shown in [Supplementary Fig. S11] (available in the online version), the user was required to manually enter information from a previous point within the workflow ([Supplementary Fig. S11], available in the online version).

The system also failed to generate normal reference ranges (3.7.5), alert the user of critical laboratory values (5.7.1), or consistently emphasize the medication name to stand out from the rest of the medication order (6.1.5). While the system may provide an indication/alert of abnormal values, it does not indicate the normal range ([Supplementary Fig. S12], available in the online version). In another example, urinalysis was flagged as abnormal result, but the system does not specify which of many individual component values are abnormal, requiring the user to manually identify ([Supplementary Fig. S13], available in the online version).

Additionally, automated systems were not used to detect typographical errors (1.12.8), as shown in [Supplementary Fig. S14] (available in the online version).

Discussion

The eSafety checklist was used to assess the Connect Care application and identify several opportunities for better achievement of CIS best practices. To our knowledge, this is the first time the eSafety checklist has been used in clinical system evaluation, and the largest formal eSafety evaluation of a CIS of this size and scope. An ambulatory CIS evaluation tool was piloted by Co et al, but this tool mostly concerns medication decision support and was tested on several smaller ambulatory clinics.[7] On the other hand, individual U.S. hospitals are now required to do yearly safety assessments using the SAFER Guides, a 146-item checklist-based risk assessment tool.[8] The SAFER Guides were considered in the development of the eSafety checklist and are designed to be used at the program level, whereas the eSafety checklist is more comprehensive with 642 items and is designed for configuration interventions for front-line informatics professionals. Although the recommended process for applying the SAFER Guides is similar to how we applied the eSafety checklist, there were several key recommendations that should be noted, namely the multidisciplinary nature of the assessment team and the need for annual re-assessment.[9]

Overall compliance with eSafety practices was good with 84% average, but still leaves ample room for improvement as Connect Care continues to be configured, implemented, and improved. Accordingly, all opportunities identified for each module were provided to the relevant build team for further consideration and action.

The four main areas of inconsistency (inconsistent date/time, unclear patient identification, ineffective alert system, and insufficient decision support) identified are a key takeaway from this project that should be considered in existing and future CIS builds. Inconsistencies in date and time conventions have the potential to create confusion, introduce miscommunication amongst clinicians, and enable inaccurate documentation, which has the potential to result in poor and/or unsafe patient care.[10] Accordingly, these types of misconfigurations were rated as major risks with urgent implementation priority ([Supplementary Appendix B]>: application opportunities; available in the online version).

Accurate patient identification is key to ensuring the correct patient receives appropriate and timely care. Use of truncated patient names is seemingly minor since column width can be adjusted by the user. However, this does not imply that user will do so, and at best presents an additional fatigue/workload on the user. User fatigue is a key factor in patient safety errors.[11]

With respect to ineffective alert system, things like unintuitive alerts and inconsistent color and coding conventions enhance the likelihood and continuation of potentially harmful practices. A lack of visually distinctive features consistent with warnings is detrimental as it may result in users bypassing or missing the notification. Red is the most natural severe alert color, and orange likely implies a lower level of severity,[12] [13] but this was not consistent in the system. Several instances of inconsistent color of severity warnings (example: “high” being orange in some instances and red in others) were flagged in the system and rated as major risks. Alerting systems which do not provide meaning or guidance will over time become a source of annoyance and disruption. Users may learn to ignore alerts or develop alert fatigue. Unacknowledged alerts can have significant impacts on patient care.[14] [15]

Another key area for improvement was insufficient or missing decision support. Reference ranges should always be provided in alerts, and this presents a major risk as results could be misinterpreted. When presenting multi-component alerts, the critical or abnormal component should be identified.[16] Additionally, automated systems were not used to detect typographical errors, which could result in miscommunication, or add additional workload on staff to fix mistakes.

Several limitations of this work should be noted. The eSafety checklist was applied by two reviewers per module and, while necessary from a resource standpoint, this does create potential for inconsistency in application. However, the checklist was designed to be objective and has a high agreement between reviewers.[5] Additionally, in this exercise, one reviewer was consistently present at all observation sessions to introduce and set the stage for each observation session.

The Connect Care configuration was incomplete when the review occurred at 6 months before the first launch of the application. The review was intended to occur before launch to give time to act on identified opportunities for improvement. However, some of the deficiencies may have been remedied through the natural course of continued configuration. Nonetheless, utilization of the checklist at this point ensured key deficiencies were clearly highlighted for attention.

Due to the strict build timelines, module analysts facilitating the demonstrations were limited to a 1- to 2-hour session with the evaluators. As a result, it is possible that not all key areas of each module were revealed to the evaluators to assess and rate for adherence to the checklist.

Although the full checklist could not be applied due to project timelines, the abbreviated checklist acted as a prompt for evaluators to attend to the items in the shortened list. Evaluators were familiar with the full checklist and could reference it when considering observations aligned with items not in the subset. Alternatively, familiarity with human factors and eSafety best practice components had assisted evaluators to search for a checklist item in a certain category to consider for noncompliance.

Some limitations of the checklist could also carry over into its application. For example, in the original checklist, all relevant configuration practices are included irrespective of the evidence strength. As a result, both evidence strength and the potential patient impact would ideally be included to help builders prioritize potential system changes when providing recommendations.

Conclusion and Future Directions

This study provides further validation of the eSafety checklist as a tool for configuration of electronic health records. In applying the tool to one of the largest CIS implementations in Canada, it has shown clinical applicability in identifying gaps in CIS configuration. After improvements are made by build teams, this assessment will be replicated to assess and confirm improvements to the system.

We encourage utilization and adaptation of the eSafety checklist at institutions across Canada and North America for future CIS implementations. An important future development for the checklist is the establishment of a formal process for keeping it updated with current evidence and best practice, which will evolve over time.

Clinical Relevance Statement

The eSafety checklist provides early identification of system configuration that is inconsistent with best practices. Inconsistent configuration with best practices can lead to implementation of a system prone to user errors. The eSafety approach in using this tool aims to prevent patient harm in the design, implementation, and use of clinical information systems.

Multiple-Choice Questions

1. What does CIS stand for?

   • Clinical Informatics Software

   • Clinical Informatics System

   • Clinical Information Software

   • Clinical Information System

   Correct Answer: The correct answer is option d.

2. Which is a correct example of the best practice for expression of date in an electronic health record?

   • 23-May-1993

   • 23-05-1993

   • 05-23-1993

   • May 23, 1993

   Correct Answer: The correct answer is option a. Alphanumeric month and four digit year ensures there can be no confusion between day, month, and year.

3. Which is not a major area of inconsistency identified by the eSafety checklist in this research?

• Inconsistent date and time

• Unclear patient identification

• Ineffective alert system

• Insufficient decision support

Correct Answer: The correct answer is option d.

Conflict of Interest

The authors have no conflicts of interest to declare with respect to the content of this manuscript.

Acknowledgements

We recognize the following Alberta Health Services teams for their work as evaluators in performing and recording system configuration observations:

• eQuality and eSafety, Provincial Patient Safety, Quality Healthcare Improvement.

• Human Factors Team, Provincial Patient Safety, Quality Healthcare Improvement.

We also recognize the efforts of Tiffany Pitamber and Sara Czerwonka in conducting data analysis and evaluation of results. We recognize the efforts of Katarina Fenske in conducting follow-up meetings with stakeholder groups to address findings and identify opportunities for improvement.

Protection of Human and Animal Subjects

According to the policy activities that constitute research at Alberta Health Services, this work met criteria for operational improvement activities exempt from ethics review.

• References

• 1 Islam MM, Poly TN, Li YJ. Recent advancement of clinical information systems: opportunities and challenges. Yearb Med Inform 2018; 27 (01) 83-90
• 2 Westbrook JI, Reckmann M, Li L. et al. Effects of two commercial electronic prescribing systems on prescribing error rates in hospital in-patients: a before and after study. PLoS Med 2012; 9 (01) e1001164
• 3 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; 74 (7–8): 519-526
• 4 Walker JM, Carayon P, Leveson N. et al. EHR safety: the way forward to safe and effective systems. J Am Med Inform Assoc 2008; 15 (03) 272-277
• 5 Dhillon-Chattha P, McCorkle R, Borycki E. An evidence-based tool for safe configuration of electronic health records: the eSafety checklist. Appl Clin Inform 2018; 9 (04) 817-830
• 6 Koppel R. Response to: an evidence-based tool for safe configuration of electronic health records: the eSafety checklist. Appl Clin Inform 2018; 9 (04) 831-832
• 7 Co Z, Holmgren AJ, Classen DC. et al. The development and piloting of the ambulatory electronic health record evaluation tool: lessons learned. Appl Clin Inform 2021; 12 (01) 153-163
• 8 Sittig DF, Ash JS, Singh H. The SAFER guides: empowering organizations to improve the safety and effectiveness of electronic health records. Am J Manag Care 2014; 20 (05) 418-423
• 9 Sittig DF, Sengstack P, Singh H. Guidelines for US hospitals and clinicians on assessment of electronic health record safety using SAFER guides. JAMA 2022; 327 (08) 719-720
• 10 Institute of Medicine Committee on Data Standards for Patient. Aspden P, Corrigan JM, Wolcott J, Erickson SM. eds. Patient Safety: Achieving a New Standard for Care. Washington, DC: National Academies Press; 2004
• 11 Mao X, Jia P, Zhang L, Zhao P, Chen Y, Zhang M. An evaluation of the effects of human factors and ergonomics on health care and patient safety practices: a systematic review. PLoS One 2015; 10 (06) e0129948
• 12 Braun CC, Silver NC. Interaction of signal word and colour on warning labels: differences in perceived hazard and behavioural compliance. Ergonomics 1995; 38 (11) 2207-2220
• 13 Phansalkar S, Edworthy J, Hellier E. et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010; 17 (05) 493-501
• 14 Rush JL, Ibrahim J, Saul K, Brodell RT. Improving patient safety by combating alert fatigue. J Grad Med Educ 2016; 8 (04) 620-621
• 15 Khalifa M, Zabani I. Improving utilization of clinical decision support systems by reducing alert fatigue: Strategies and recommendations. Stud Health Technol Inform 2016; 226: 51-54
• 16 Sittig DF, Murphy DR, Smith MW, Russo E, Wright A, Singh H. Graphical display of diagnostic test results in electronic health records: a comparison of 8 systems. J Am Med Inform Assoc 2015; 22 (04) 900-904

Publication History

Received: 14 December 2022

Accepted: 05 May 2023

Article published online:
13 September 2023