Keywords audit and feedback - electronic health records and systems - care transition - provider-provider communication - pediatrics
Background and Significance
Background and Significance
Clinicians develop expertise by learning from past performance.[1 ] Clinicians who receive information about their performance can become better calibrated and thus make better diagnosis and treatment decisions.[2 ] Calibration is the alignment between clinicians' confidence in their accuracy and their actual accuracy[3 ] and can be achieved by receiving consistent feedback about one's patient outcomes.[3 ]
[4 ]
[5 ]
[6 ]
[7 ]
A total of 40% critically ill children with acute illness and injury are initially diagnosed and treated at community hospitals.[8 ]
[9 ] Front-line clinicians have variable training in pediatrics and may infrequently encounter very ill children which can result in suboptimal care, most commonly undertreatment.[10 ] Timely interventions based on accurate assessments by community physicians have been shown to improve outcomes in critically ill children[11 ]
[12 ]; thus, it is essential for referring clinicians to learn about their transferred patients' outcomes to improve future performance. However, in our highly compartmentalized model of health care, referring clinicians do not receive consistent feedback on individual patient outcomes to continuously improve their practice. In a statewide survey, we conducted in Iowa, referring clinicians received patient outcome feedback on only 40% of patients they transferred to pediatric intensive care units (PICUs) over a year. Among clinicians who obtained feedback,13% received information they did not expect; 40% of these clinicians indicated that this experience changed their practice.[13 ]
Objectives
Given this significant gap in feedback and its potential to improve the emergency care of critically ill children, our objective was to develop, implement, and evaluate a semiautomated electronic health record (EHR)-supported system at a single institution to deliver timely and relevant PICU patient outcome feedback to referring emergency department (ED) physicians. In this case study, we report the feasibility, usability, and relevance of the feedback system to both ED and PICU physicians.
Methods
Using a human factors engineering approach informed by the Health Information Technology (IT) Safety Framework,[14 ] we iteratively designed, implemented, and evaluated a semiautomated electronic feedback system leveraging the Epic EHR at a single institution. This project was reviewed and determined to be exempt from human subjects research oversight by the University of Iowa Institutional Review Board. In the next sections, we describe the clinical settings for the project and how we determined the need for feedback, designed the feedback system/report, and evaluated its performance.
Clinical Settings
The EHR-supported feedback system was developed for implementation at the University of Iowa Hospitals and Clinics (UIHC) ED and the UI Stead Family Children's Hospital PICU. Both serve mostly rural populations within a large catchment area. The UIHC ED is a tertiary referral academic ED with 46 beds accommodating approximately 15,000 pediatric visits per year and is staffed by 30 emergency medicine (EM) attending physicians (six pediatric EM trained). The UI PICU is the only academic tertiary referral PICU in Iowa, admitting approximately 1,200 patients per year and is staffed by 11 attending intensivists.
Determining the Need for Patient Outcome Feedback
We conducted two focus groups of ED physicians (six UI ED physicians and six non-UI-affiliated referring physicians) to substantiate our earlier survey indicating the gap in feedback[13 ] and explore physicians' perceptions of the relevance of receiving consistent patient outcome information. Referring physicians noted the various unsystematic ways in which they find out what happened to their patients. This included their own efforts to reaccess patients' medical records or call the PICU to obtain an update on patients. Sometimes, they received patient outcome information through third parties, such as colleagues, patients' family members, or via social media. Referring physicians underscored that knowing their patients' outcomes can help them evaluate and improve their practice. They also indicated that open communication between medical professionals who share patients across practice settings can improve transitions of care for critically ill children.
Development and Evaluation of the Feedback System Using a Human Factors Engineering Approach
Our team used the UIHC's existing Epic EHR platform to develop an electronic feedback system. Commonly used EHR platforms, such as Epic, have existing functionality, such as the ability to insert patient data into templated letters[15 ]
[16 ] that can automate steps in the feedback process and can be readily adapted for implementation.[17 ]
We used a human factors engineering approach as applied to biomedical informatics projects to develop and test the EHR-supported feedback process. This approach uses a functional model of a human factors engineering lifecycle adapted for health care work situations and incorporates usability engineering methods to evaluate health care products and workflows.[18 ] Throughout development, we were guided by the Health Information Technology Safety Framework[14 ]
[17 ] which provides a multidimensional sociotechnical approach for evaluating and ensuring safety of health IT in complex adaptive health care settings. This approach accounts for important interactions among clinicians, tasks/workflows, and technology that may present as facilitators or barriers to implementation. [Fig. 1 ] shows each phase of the project with its corresponding goals and methods.
Fig. 1 Overview of development and evaluation of an EHR-supported patient outcome feedback system using a human factors engineering approach. ED, emergency department; EHR, electronic health record; IT, information technology, PICU, pediatric intensive care unit; UI, University of Iowa.
Phase I: Analysis of the Sociotechnical System
To develop a deep understanding of the health care work system where the feedback process will be integrated, we performed semistructured interviews and focus groups with various stakeholders. During the same focus groups, we conducted with referring physicians above, in addition to determining the need for feedback, we also gathered information on physicians' clinical workflows and environments and elicited their perspectives on how an electronic feedback system may be best implemented. We also interviewed support staff (e.g., UI telephone operators), health IT/EHR platform specialists, and administrators (e.g., information officers and legal counsel), who we anticipated will play a role in the feedback process or have knowledge of resources relevant to delivering feedback. Finally, we reviewed organizational and state/federal data transfer regulations to be considered.
Phase II: Cooperative Design
We collaboratively designed the EHR-supported feedback system by translating stakeholder, organizational, and usability objectives (from Phase I) into feedback process and feedback report requirements. We designed a workflow for how feedback reports could be generated, delivered, and received. For each step, we identified tasks that need to be performed, who/what will perform the tasks (staff/clinicians versus EHR), and when/where tasks will be performed ([Fig. 2 ]). Stakeholders also designed the feedback report by considering clinical data relevant to ED clinicians and the EHR's capabilities to pull/push and summarize clinical data ([Fig. 3 ]).
Fig. 2 Feedback process systems design showing (A ) overall feedback process requirements and (B ) feedback workflow with functions of each step and information technology/human requirements. Epic applications and functions are shown since this was the specific EHR platform used for the project. ATC, admission and transfer center; ED, emergency department; EHR, electronic health record; IT, information technology; PICU, pediatric intensive care unit; UI, University of Iowa; UIHC, UI Hospitals and Clinics.
Fig. 3 Feedback report design showing (A ) report requirements and (B ) a prototype report generated in the EHR and sources of the report content. Epic was the specific EHR platform used for the project. EHR, electronic health record; PICU, pediatric intensive care unit.
Phase III: Implementation and Iterative Evaluation
Over 6 months, we completed three cycles of implementation, evaluation, and redesign. We generated feedback reports for all patients who were transferred to the PICU within 24 hours of a UIHC ED visit. As illustrated in [Fig. 2 ], patients eligible for feedback and their referring physicians were identified on the third PICU day and the initial feedback reports were manually generated by our team. Reports were then routed to PICU physicians who reviewed, edited, and sent them to ED physicians.
We also developed a set of evaluation metrics that would best reflect the feedback system's performance with regard to stakeholder goals while also considering the feasibility of data collection. These metrics and data sources are summarized in [Table 1 ]. We evaluated the feedback system's feasibility and timeliness of feedback delivery and receipt by reviewing EHR access logs to determine feedback report throughput. We analyzed feedback reports sent to determine the sources of feedback report content. We conducted usability testing[19 ] of the EHR interface with six PICU physicians, asking them to review, edit, and send actual feedback reports while thinking aloud.[19 ]
[20 ] We audio-recorded physicians' comments and two research team members directly observed how they interacted with the electronic interface, paying particular attention to the functionality of relevant EHR applications (Epic In Basket and Epic Quick Letter) and problems in human–computer interaction. We also asked each physician to assess the process using the System Usability Scale (SUS), a validated 10-item survey to evaluate an individual's assessment of a system's usability. SUS scoring yields a composite measure between 0 to 100 which represents overall usability. Products with SUS scores >85 are generally considered highly usable.[21 ] Finally, we conducted semistructured interviews with PICU physicians and ED-referring physicians to determine how well the feedback process is integrated into physician workflows and to gather physicians' perceptions of the relevance of feedback reports to their clinical practice. Descriptive statistics were used to analyze and report quantitative data. Qualitative data from interviews were transcribed, coded, and synthesized into common themes.
Table 1
Evaluation metrics and data sources
Evaluation metrics
Data sources
Feasibility, timeliness, and impact on workload
Feedback report throughput[a ]
EHR reports and access logs
Sources of feedback report content
Feedback report
Impact on workload
Direct observation, physician interviews
Usability of electronic interface
Time for PICU physician to complete editing and sending report
Direct observation
System Usability Scale score[b ]
Rating by PICU physician
Navigation problems in electronic interface
Direct observation, physician interviews
Actions taken to confirm information automatically populated by the EHR or to seek additional information
Direct observation, physician interviews
Changes made to information automatically populated by the EHR
Direct observation, physician interviews
Relevance to clinical practice
Physician interviews
Abbreviations: EHR, electronic health record; PICU, pediatric intensive care unit.
a Feedback report throughput includes the number of reports generated, routed to PICU physicians, sent to emergency department (ED) physicians, and reviewed by ED physicians and the length of time between these events.
b The System Usability Scale (SUS) is a validated 10-item survey used to evaluate an individual's assessment of a system's usability. Scoring of the SUS yields a composite measure between 0 and 100 that represents overall usability. Products with SUS scores >85 are generally considered highly usable.[21 ]
Results
Feasibility and Timeliness of Feedback Delivery
Over 6 months, 119 feedback reports were generated and routed to PICU physicians. PICU physicians edited and sent 98 (82%) feedback reports. Referring ED clinicians received and reviewed 86 (72%) reports. Overall, feedback reports were sent by 10 unique PICU physicians and reviewed by 25 unique ED clinicians.
During the first two cycles of implementation, 21 (18%) feedback reports were inappropriately deleted before PICU physicians were able to review them. Our IT team identified that this was due to a preexisting custom Epic communication management workflow, wherein pended feedback reports were deleted when other users created and sent new unrelated reports using the same application (Epic Quick Letter). This was addressed by revising the programmed Epic workflow, preventing further loss of reports.
Otherwise, we found that the EHR-supported feedback process performed technically well. Majority of the feedback report content (86%) was automatically pulled in from the EHR, with only 12% of the content typed in by PICU physicians (most commonly to clarify/add diagnoses and include information on clinical course and status). Feedback reports were generated and routed to PICU physicians a mean of 3.4 days from PICU admission. PICU physicians completed and sent the reports a mean of 0.7 days after they were routed to them. Reports were reviewed by ED clinicians a mean of 1.8 days after being sent and a mean of 6.8 days from their encounter with the patient in the ED ([Table 2 ]).
Table 2
Technical characteristics of EHR-enabled patient outcome feedback system and update report
Technical characteristics
Mean or proportion
Update report content, total word count, mean (SD)
277 (59)
Information automatically pulled from EHR, word count, mean (SD, % of total)
239 (47, 86%)
Information selected by clinician from dropdown choices, word count, mean (SD, % of total)
7 (5, 3%)
Information manually added by clinician, word count, mean (SD, % of total)
31 (26, 12%)
Feedback process delivery metrics[a ]
Time to routing of initiated update report to PICU physician from PICU admission, days, mean (SD)
3.4 (0.7)
Time to completion of report once received by PICU physician, days, mean (SD), days, mean (SD)
0.7 (1.9)
Time to report viewed by ED clinician once sent by PICU clinician, days, mean (SD)
1.8 (3.8)
Time to report viewed by ED clinician from ED visit, days, mean (SD)
6.8 (4.3)
Update report received by the correct ED clinician, n (%)
90 (92)
Abbreviations: ED, emergency department; EHR, electronic health record; PICU, pediatric intensive care unit; SD, standard deviation.
a Of 98 letters sent to ED clinicians by PICU physicians.
Usability
Direct observation of six PICU physicians (two physicians per iterative cycle), four of which were novice users (less than 10 feedback reports previously sent), showed an overall mean of 4.7 minutes to review, edit, and send feedback reports. However, nonnovice users performed faster (mean of 2.5 minutes). Usability testing with the same six physicians yielded a mean SUS score of 88.3, suggesting an “excellent” usability.[22 ] Key observations included difficulties in navigation and physicians confirming information by reviewing other sections of the EHR (both unique to novice users). Half of physicians did not edit diagnoses automatically pulled from EHR problem lists, while all physicians typed in additional information regarding the patient's clinical course and intermediate outcomes ([Table 3 ]).
Table 3
Usability characteristics of EHR-enabled patient outcome feedback system and update report
Usability characteristics[a ]
Time for PICU physician to complete reviewing, editing, and sending update report (min), minutes, mean (SD)
4.7 (2.9)
System Usability Scale score,[b ] mean (SD)
88.3 (7.9)
Key observations from usability testing with PICU physicians, n
Frequency (n =6)
Had difficulties with navigating from EHR messaging application to letter editing/sending application (only observed among novice users)
3
Needed to confirm information by reviewing other sections of EHR
3
Did not edit diagnoses automatically pulled from EHR problem list (i.e., did not include a narrative summary of diagnoses)
3
Added more information describing the patient's clinical course and outcomes
6
Abbreviations: EHR, electronic health record; PICU, pediatric intensive care unit; SD, standard deviation.
a Usability testing was performed with six PICU physicians with varying levels of experience in sending feedback reports prior to testing.
b The System Usability Scale (SUS) is a validated 10-item survey to evaluate an individual's assessment of a system's usability. SUS scoring yields a composite measure between 0 and 100, which represents overall usability. Products with SUS scores >85 are generally considered highly usable.[21 ]
Impact on Workload and Relevance of Feedback Reports to Clinical Practice
Interviews with six PICU physicians and four ED physicians revealed that the feedback process was well-integrated into their respective clinical workflows. Because reports were routed or sent to physicians' EHR in-baskets, sending and receiving reports closely aligned with the physicians' existing identical workflows for signing clinical notes. ED physicians also appreciated that receiving feedback reports within the EHR provided them easy access to additional information in the patient's chart if needed. Both PICU and ED physicians indicated that they sent/reviewed feedback reports at the same time that they signed their notes and that the overall time burden for completing/reviewing reports was acceptable. ED physicians noted that they received reports within an acceptable timeframe as well.
PICU physicians indicated that sending patient outcome feedback to their ED colleagues is a worthwhile endeavor and will overall help improve the emergency care of children prior to PICU transfer. ED physicians indicated that receiving patient outcome feedback is relevant to their clinical practice, since it helps them identify whether they have appropriately triaged patients (i.e., to the PICU vs. general wards) and provides them with a venue to reflect on potential improvements in their diagnostic reasoning and clinical management.
Iterative Changes
After the first cycle, we disseminated a navigation guide to PICU physicians, this supported the learnability of the process and served as a quick reference while the task was performed. We also sent e-mails to PICU physicians at the beginning of each clinical service week to remind them to review and send reports. This was phased out after 5 weeks when the process was well-integrated into the workflow. After the second cycle, we responded to questions from PICU physicians regarding specific clinical information that ED physicians found helpful (additional diagnoses since the ED visit, evolution of the clinical course, and patient disposition) by reporting back results of our interviews with ED physicians. We also revised the EHR access log reports we created to track feedback report throughput, so we can more clearly delineate time intervals between tasks. Our IT team also identified and addressed the software error which caused feedback reports to be inappropriately deleted, as described earlier. After the third cycle, we added ED resident physicians as feedback recipients at the request of the pediatric ED medical director. We also noted that significant time and effort was needed to manually identify eligible patients and their respective referring physicians and PICU physicians to ensure correct routing and delivery of feedback reports, suggesting the need for more automation in this process.
Discussion
In this case study, we showed that a semiautomated EHR-supported system to deliver timely and relevant PICU patient outcome feedback to referring physicians is technically feasible, usable, and relevant to both ED and PICU physicians at a single institution, sharing a common EHR platform. Although our system is distinct from prior published work, deploying electronic interventions to improve subspecialty referrals have been used in the past with varying success.[23 ]
[24 ]
The main challenges we encountered included the need for PICU physician review of feedback reports (due to limitations in the EHR's ability to accurately summarize clinical data[25 ]), PICU physicians' clinical workload affecting timely completion of feedback reports, and the significant effort in manually identifying eligible patients and their corresponding ED and PICU physicians for the purpose of generating feedback reports for routing to PICU physicians. Expanding the electronic process to deliver feedback beyond our institution has also been challenging due to the limited interoperability of EHR platforms, making it difficult to send feedback reports between institutions with different EHR systems and even between those with the same but differently configured EHR system. Pending progress on this issue, delivering feedback outside of our health care organization will necessitate a hybrid electronic-paper (faxed reports) feedback process.
Limitations and Strengths
Limitations and Strengths
Our case study has limitations. The feedback system was tested in only one institution and only between two groups of physicians, although efforts are underway in our institution to expand the process outside of our health care organization in different clinical settings. We also only determined the technical performance and relevance of the feedback system and did not measure impact on clinical practice and patient outcomes which will be essential next steps.
Despite these limitations, we expect that the system we have developed can be adapted to other institutions and/or other clinical settings across the continuum of patient care. Although we used the Epic EHR platform to develop and test the feedback process, we anticipate that other EHR platforms have similar functions that can be used to build a similar feedback system. We recommend that institutions follow a similar step-wise human factors approach to development to ensure that the system is feasible in their setting and fulfills their specific feedback goals. The following are the key lessons we learned:
Create a sense of serious urgency (a “burning platform”) to highlight the importance and need for feedback between clinicians in the target settings.
Involve all stakeholders from the beginning, especially referring clinicians, to achieve consensus and clarity around feedback goals and the processes to be implemented to attain these goals.
Understand the context of the health care system and the environment where the feedback intervention will be embedded in both clinical settings sending and receiving feedback, not just the IT or EHR platform to be used.
Collaboratively design the feedback process involving all stakeholders.
Identify important and feasible metrics to evaluate the performance and impact of the feedback process.
Conclusion
In conclusion, an EHR-supported system to deliver timely and relevant PICU patient outcome feedback to referring ED physicians was technically feasible, usable, and important to physicians. Next steps include wider implementation to referring clinicians from other institutions, evaluation of the feedback system's impact on clinical practice and patient outcomes, and investigation of potential applicability to other clinical settings involved in similar care transitions.
Clinical Relevance Statement
Clinical Relevance Statement
Patient outcome feedback is an important way by which clinicians can improve their performance and the care they provide to patients. Leveraging the EHR to support an electronic semiautomated feedback system that can deliver timely and relevant patient outcome information to clinicians is a feasible way to operationalize consistent clinician feedback. The feedback system we developed can potentially be adapted for implementation in other clinical settings involved in similar care transitions.
Multiple Choice Questions
Multiple Choice Questions
Why is it important to conduct stakeholder interviews and focus groups before implementing a new information technology-supported process?
To determine perceived facilitators and barriers to the process to inform development
To inform stakeholders of the new workflow and specific tasks that they will now need to do
To convince stakeholders that implementation of the new process will be of benefit to them
To emphasize the importance of adhering to the new process
Correct Answer: The correct answer is option a. It is important to determine stakeholders' perspectives regarding how proposed new processes can be facilitated or hindered within their specific health care environments to proactively leverage facilitators and develop solutions to minimize barriers for more successful implementation.
What data can usability testing provide that are useful to inform the development and evaluation of a new electronic process?
Functionality of electronic applications
Problems in the human–computer interface
Impact on clinicians' workload
All of the above
Correct Answer: The correct answer is option d. Usability testing is important to (1) determine how well electronic programs perform their specified functions, (2) identify problems in the program interface that may interfere with human operation, and (3) measure the workload conferred by the new process (e.g., time to completion of tasks, effort required to complete tasks, and others).
