Keywords analytics - documentation burden - EHR system - clinical informatics
Background and Significance
Background and Significance
Usage analytics is an emerging source of data for measuring burden related to use of the electronic health record (EHR),[1 ] investigating the impacts of EHR use on patient care[2 ] and evaluating the impact of interventions[1 ]
[3 ] or events (e.g., COVID-19)[4 ] on EHR use. Analytics data from non-EHR care delivery tools such as nurse call systems and communication devices have also been used to measure and predict strain amongst clinicians.[5 ] For EHR burden, measures have traditionally focused on subjective measures through surveys,[6 ]
[7 ]
[8 ] including validated instruments for concepts like “burnout” (e.g., Mini-Z survey[9 ]). While these approaches allow for characterizing perceived burden and EHR experience,[10 ] discrepancies between what is estimated by end-users and what is observed in the system have been identified.[8 ] Moreover, it remains difficult to pinpoint the specific challenges that lead to the burden.[11 ]
[12 ] To identify the root causes of EHR-related burden and evaluate the impact of the solutions, usage log data will be a critical component toward enabling data-informed training[13 ] and optimization of the system.[14 ]
[15 ]
[16 ] Over the past decade, EHR usage logs have been increasingly used to gather metrics for documentation burden concepts[1 ] such as effort (e.g., after-hours work) and time (e.g., average time spent),[17 ]
[18 ] with a recent scoping review[1 ] reporting the use of this method of data collection within 80% of studies across a variety of hospital settings (e.g., general surgery,[19 ] primary care, and internal medicine[20 ]
[21 ]
[22 ]).
Previous research has demonstrated the need for validity and reliability within EHR-based metrics and reporting.[23 ]
[24 ] Therefore, before employing these tools to measure efficiency, understanding the reliability and validity of these tools is important. To ensure administrators and physicians are confident in using EHR usage log data, the validation process described in this case report is necessary. Given that each organization has unique workflow and documentation practices on the EHR system, it is important that the metrics are able to accurately provide a complete picture of the usage patterns of clinicians. A systematic review of EHR logs[25 ] used for measuring clinical activity demonstrated that only 22% of the 85 studies validated the accuracy of the findings. Currently, two approaches are used for validation: (1) action–activity mappings (by consulting EHR vendors, consensus between researchers, or direct observation) or (2) activity patterns or duration (using direct observation or using self-reported data). These “time–motion” studies[25 ] involve real-time observation and comparison of how long a function has been performed by a clinician (e.g., using a stopwatch or TimeCat[26 ]) against the usage log readings. Use of this method has been fairly successful in ophthalmology[27 ] and primary care[28 ] settings, among others.[29 ] However, time–motion studies require significant human resources and time to conduct the validation, and can be difficult to scale in organizations with many departments and unique workflows. The presence of an observer may also hinder the comfort levels of patients and providers within the clinical environment, particularly in a mental health setting. Moreover, given the current pandemic and the rise of telemedicine or distancing requirements for in-person care, it is not feasible nor appropriate to introduce an observer within the clinical environment. From a privacy perspective, recording physician's screens can also introduce a challenge for privacy preservation as identifying information of patient names, images, and diagnoses are unintentionally captured.[30 ] Thus, a safe, remote, and less invasive approach for validation of the usage log data of EHR systems is needed.
Objectives
In this case report, we introduce an approach using clinical test cases that was implemented to validate the usage log data for an integrated EHR system in use at a large academic teaching mental health hospital. While other organizations may have used this approach in their validation, to our knowledge, this is the first study that discusses this approach within the academic literature. This approach overcomes the limitations identified above in current validation approaches and provides an effective way to test out a large number of workflows with limited resources, and in a manner that is noninvasive to the clinical environment. We highlight the utility of this approach, including key considerations for applying this methodology and using EHR log data in practice.
Materials and Methods
The approach for validation of usage log data is composed of three phases ([Fig. 1 ]). The validation was conducted in an academic teaching mental health hospital located in Toronto, Ontario and approval was obtained from the organizational Quality Project Ethics Review Board.
Fig. 1 Overview of approach for usage log validation.
Phase 1: Creating Test Cases Based on Real-World Clinical Workflows
Given that the ultimate goal of EHR usage data is to accurately capture the usage patterns of clinicians using the EHR, a guiding principle of this stage was to develop use cases (hereafter called “test runs”) that mimicked as closely to real-world clinical workflows used by physicians at the organization. To create these test runs, we began by consulting the organization's EHR training modules that are available to all physicians on our intranet, and provide detailed line-by-line physician workflows of how to carry out common tasks within the EHR. Through consultation from senior clinical leadership including the Chief Medical Informatics Officer (T.T., who is also a practicing physician within the organization), physician super users of our EHR system, and our clinical informatics nurse, we identified common inpatient and outpatient physician workflows and reconciled any differences between training documents and real-world EHR usage. The clinical informatics nurse consulted is solely responsible for EHR education and training for all 400 physicians at our organization and hence well versed in physicians' use of the EHR. Using an agile, iterative approach, consultation with these aforementioned individuals helped develop test runs that closely resembled real-world physician workflows (see [Table 1 ] for example test runs; see the [Supplementary Material ] for all test runs [available in the online version]). Since organizations have unique clinical workflows, we wanted to ensure that these test cases were specific to our organization, and therefore did not consult the literature for this step.
Table 1
Example of a test run used for validation of the usage log data
Example inpatient test run: preparing for inpatient discharge
Example outpatient test run: regular follow-up at clinic
• Search and open MRN
• Add discharge diagnosis (bipolar disorder)
• Initiate a discharge order for patient, including the location of discharge
• Complete discharge summary note for the patient
• Search and open MRN
• Order “lorazepam”
• Complete an outpatient progress note
• Search and open another MRN
• Review laboratory results
• Complete discharge documentation for the patient
Abbreviations: MRN, medical record number.
Phase 2: Execution: Conducting the Clinical Workflows in the Test Environment
This involved setting up a dedicated mock physician account in a nonproduction version of the EHR system, so that all activity within this account was recorded. The only difference between the test environment used within this study and our organization's production environment is the absence of real patient data. EHR functionality and modules were identical between the two environments, and our vendor provided confirmation that the methods used to measure usage metrics were identical between the two environments. Mock patients within the test environment were developed by our clinical informatics nurses that mimic real patients with representative diagnoses, care plans, etc. The Research Assistant (RA) performed the test runs that were developed in phase 1, using pauses and variations in documentation time to resemble real-world interruptions and workflows.
For data collection, two complementary techniques were used: (1) a spreadsheet with automatic time-log capture[31 ] was developed: when each task within a test run was complete, the RA manually indicated completion in the respective cell, which then automatically logged the time of completion; (2) a screen recording tool was used: this method was used to re-measure the execution of the test runs through measuring the times by a stopwatch. The spreadsheet was used to easily calculate the time spent across each task of a test run (as well as the entire test run) and the screen recording allowed for retrospective review and confirmation of time spent.
We gathered EHR-usage data from the back-end analytics platform from the EHR vendor. By logging the exact date and time of the interactions in the test account, the EHR vendor extracted the most granular level of detail during that time period to allow for comparison. The following data were sent through a spreadsheet:
Phase 3: Data Analysis: Comparison between Usage Logs and Test Case Observations
To determine if the time extracted from the EHR back-end analytics platform was comparable to the data we collected through the spreadsheet and screen recording, we converted all time measures to seconds. We compared (1) time spent per patient for the following metrics: total time in EHR, documentation time, order time, chart review time, allergies time, problem and diagnosis time, and (2) counts for the following metrics: patients seen, notes written.
Utility was defined by the absolute differences and percent differences observed between the two methods. To explore the discrepancies between recorded values and those found within the analytics platform, the RA replicated the tasks and consulted with our vendor to identify the root causes.
Results
A total of 10 test runs were conducted by the RA (A.K.) across 3 days, with one system interruption reported in Run 8. Differences in measurements found between the two methods of data collection (i.e., RA recorded values and usage analytics platform) for time-based metrics averaged across 3 days are summarized in [Fig. 2 ] and count-based metrics are summarized in [Fig 3 ]. Results of independent t -tests performed for all eight metrics are highlighted in [Supplementary Table S1 ]; however, it should be noted that these results are based on one user with 3 days of data. A summary of measurements extracted from the analytics platform compared with recorded values is outlined in [Supplementary Table S2 ] (available in the online version). The percent difference between measurements recorded by the RA and the usage analytics platform ranged from 9 to 60%. The discrepancies observed in time in EHR and order time in EHR were relatively small (<20%). Of the 3 concurrent days of data collection of documentation time in EHR and chart review time per EHR metrics, findings from one of these days yielded large % differences (57–60%) between time captured by the usage analytics log and our spreadsheet.
Fig. 2 Measurements recorded by usage analytics platform and test case observations for time-based metrics (averaged across 3 days).
Discussion
Validation of metrics is often considered a barrier toward the full uptake and use of usage log data to support characterization and mitigation of the EHR burden. This is mainly due to the resource and time required to conduct robust time validation studies, hindering its practical execution. This work outlines a feasible and resource-efficient approach for validating usage log data for use in practice. Previous validation studies using screen recordings of EHR sessions have demonstrated that metrics such as total time spent time within the EHR correlated strongly with observed metrics (r = 0.98, p < 0.001), where each minute of audit-log-based time corresponded to 0.95 minutes of observed time[32 ] across a variety of provider roles. Other research within ophthalmology clinics validating EHR log data using time–motion observations have demonstrated that 65 to 69% of the total EHR interaction estimates were ≤3 minutes from the observed timing, and 17 to 24% of the EHR interaction estimates >5 minutes from the observed timing.[33 ]
Lessons Learned
We learned the following lessons from our technique:
Ensure partnership with EHR vendor : this approach allowed for a collaborative review of discrepancies by both organization and vendor. We identified some discrepancies amongst our metrics after splitting it by the reported number of patients seen. Upon review with our vendor developers, the number of patients seen is only recorded if they only complete a certain action (e.g., completing a form as opposed to an actual documentation). Additionally, when looking at our after-hours metric, the hours were predetermined by our vendor (6 p.m.–6 a.m.), and this time frame might be different from physician's work hours. This helped us explain why some values were overinflated against our own measurements, and also helped us brainstorm other situations (e.g., physicians signing on resident's notes) that might impact the time spent.
Iteratively test and maximize data transparency : the resource-efficient approach of this method allows us to adjust the level of depth of our workflows and repeat the runs as necessary (e.g., after EHR upgrades). We began our validation with less complex test runs that repeated the essential tasks in a very controlled setting (results not shown here) and gradually increased to more sophisticated test runs as it aligns to our metrics. Thus, this method allows for a step-wise, controlled validation approach that can be embedded as part of the implementation lifecycle. The screen-recording helped us explore the rationales for the discrepancies. These recordings also allow for transparency and replicability of the results.
Interpret results appropriately
: our findings provide support of the accuracy of the usage log data to our clinical workflows. Previous studies have reported variations of overestimations of 43% (4.3 vs. 3.0 minutes) to underestimations by 33% (2.4 ± 1.7 vs. 1.6 ± 1.2 minutes).[25 ] For most of our metrics, the discrepancy between the usage analytics platform and our observation was fairly consistent across the 3 days of test runs, which suggests that the metrics are consistently calculated on a day-to-day basis. For large differences within our data (e.g., 124% underestimation of Allergies time), it is important to note that these differences can be amplified when small amounts of time are spent carrying out a particular task (i.e., <30 seconds/patient). Moreover, there is a slight variation in the back-end calculation of certain metrics where it might appear that the user is “documenting.” In these cases, back-end timers may have counted this task within a different area of the chart since the documentation sat within a larger section (e.g., a workflow page). While it is very difficult to obtain very high accuracy as per the nature of log data (e.g., when mouse stops moving),[34 ] the presence of these results provided confidence on the level of accuracy (i.e., range of error) we can expect in practice. When differences between observed and measured time are vast, organizations can still make use of usage analytics tools to evaluate the impact of initiatives; however, in these cases, organizations will need to consider the % change instead of absolute value of time spent in EHR activities pre- and post-initiative.
Limitations
Several limitations should be considered for these results and methodology. Foremost, while the test runs created for use in our validation closely resemble physician behavior in the EHR, we did not use real-world observational data of multiple physicians, which highlights the variety in practice and use of the EHR. The human-factors bias of using a single RA for conducting the tests is an added limitation. We used test runs inclusive of pauses and interruptions to EHR use to resemble real-world interruptions and workflows; however, we recognize that there could be longer workflow interruptions caused on a clinical unit.
Moreover, we used a test environment (or “instance”) of the EHR instead of a production environment. However, since back-end metrics calculation within the test environment was confirmed (by our vendor) to be identical with how metrics are calculated in the production environment, the impact of difference in environments is negligible. Moreover, the current validation was only conducted at one mental health organization. Despite the context being a large academic mental health hospital, we caution generalizing these results as the workflows likely differ across organizations.
Future Directions
The findings from the use of this methodology identify a few areas for future consideration. Foremost, based on our information gathering,[8 ] we only validated a small number of usage log metrics that were considered a priority for our organization. While we anticipate that this method should suffice for other metrics, future clinician metrics considered important for reducing EHR burden[3 ] (e.g., medication reconciliation) should also be validated using a similar approach mentioned in this report. Moreover, it would be useful to explore the application of this approach to other EHR systems at different organizations. Lastly, we continue to see a paucity of data validation studies being published. In an effort to promote transparency and understanding of the utility of usage log data, we encourage other mental health and nonmental health organizations to share their validation results. Once validation of usage log data has been conducted, organizations can use appropriate metrics to measure EHR-related burden prior and after implementing initiatives aimed at reducing burden.
Conclusion
This case report introduces a novel minimally invasive approach toward validating usage log data. By applying it to usage data at a Canadian mental health hospital, we demonstrated the flexibility and utility of the approach in comparison to conventional time–motion studies. Future studies should aim to explore and optimize this methodology for validation of usage data across various EHR systems and practice settings.
Clinical Relevance Statement
Clinical Relevance Statement
This initiative will provide a feasible and low-burden approach to validating EHR usage data for further optimization and quality-improvement initiatives.
Multiple Choice Questions
Multiple Choice Questions
When studying EHR burden, what are the some of the EHR analytics metrics to consider?
Correct Answer: The correct answer is option d, all of the above. All the three metrics listed above are valuable in measuring the amount of burden caused by the EHR, and could be helpful in measuring the impact of interventions.
When interpreting EHR analytics data, what is a good metric to use for measuring change due to an intervention?
% difference
Mean difference
Median difference
Standard deviation
Correct Answer: The correct answer is option a, % difference. While the mean or median difference is an absolute measure of measuring change, using the percent (%) difference measure takes into account minor discrepancies between actual time spent and time recorded by analytics systems.
Fig. 3 Measurements recorded by usage analytics platform and test case observations for count-based metrics (averaged across 3 days).
