Keywords
emergency service, hospital - social vulnerability index - patient portals - after visit summary - electronic health record
Background and Significance
Background and Significance
The Office of the National Coordinator for Health Information Technology's (ONC) Final Rule implementing the 21st Century Cures Act and its provision on information blocking went into effect on April 5, 2021, expediting the release of test results to patients via patient portals across the United States.[1]
[2]
[3]
[4] While the Cures Act defines the concept of information blocking in practice, the Final Rule created the corollary practice of information sharing, consisting of the immediate release of electronic health information contained within the annually updated US Core Data for Interoperability standard data set.[5] This practice is colloquially known as open notes and open results for clinical notes and test results, respectively. A robust literature supports open notes, though open results are more controversial.[6]
[7]
[8]
The information blocking provisions apply uniformly across different care settings. As a result, test results may be released to patients while receiving care in acute settings such as emergency departments (EDs), inpatient units, or intensive care units. This information release may create challenges related to patient engagement, expectation management, and the need for pre-counseling regarding impactful test results, which is especially difficult in time-limited environments such as the ED. Foster and Krasowski found that nearly 9% of patients who arrived to the ED with active portal accounts reviewed ED results within 72 hours.[9] However, the effect of open results on ED patients during their acute visit remains unexplored.
As a first step to evaluate how patients use portals in real-time while in the ED, we performed a retrospective electronic health record (EHR)-based study to identify the proportion of ED patients who viewed results while still actively receiving care in the ED and describe these patients' characteristics.
Objectives
The primary objective of this study was to determine the proportion of patients who viewed test results while receiving care in the ED and explore temporal trends since the 21st Century Cures Act Final Rule went into effect. Further, we sought to compare characteristics of the patient populations who viewed and did not view results during their ED stay and identify factors associated with real-time ED-based enrollment and use for patients who were not enrolled prior to ED arrival.
Methods
Setting
This study was performed at a single ED in an academic quaternary care center in Dallas, Texas, United States. Our site is a certified stroke and comprehensive cancer center with an active cardiac catheterization lab. The site is not a trauma center.
Our health system adopted an opt-in model for patient portal enrollment, which requires a clinician, nurse, or member of our registration or patient access team to provide an access token to new patients for enrollment in the portal. This contrasts with an opt-out model, where new patients are automatically provided access upon registration at their first appointment, usually via text message, email, or QR code. Opt-out models have been shown to increase portal enrollment but may increase costs due to EHR licensing fees, which are usually based on the number of enrolled users.[10]
Despite our site's opt-in model, clinicians, nurses, and registration staff can easily send patients portal enrollment tokens via text message, email, or the after visit summary (AVS). The EHR-based workflows for sending tokens are accessible with a single click from Epic's Storyboard – an at-a-glance summarization tool visible from most Epic workflows. Patients can use this token to link their account to the EHR through a web interface or mobile application. During the study period, our department did not directly advertise the availability of immediately released results to patients. However, information about the portal was readily available on our institution's website and on AVSs (ED, inpatient, and ambulatory). Additionally, other health system inpatient services and clinics have varied practices surrounding portal enrollment that might have influenced enrollment patterns. During the study period, there was not a designated or mandated workflow by which patients were routinely given tokens to enroll in the portal. Enrollment was facilitated using the techniques above by a combination of registration staff, physicians, advanced practice providers, and nurses at the request of either the patient or treatment team.
Study Design
We conducted a retrospective observational study using EHR audit log data to determine the proportion of patients who viewed test results and to describe the differences among populations who did and did not view their test results during their ED encounter. We included adult (≥18 years old) patients who presented to the ED between April 5, 2021 and April 4, 2022 and had at least one resulted order during their stay, representing the full year after the 21st Century Cures Act Final Rule took effect. Prior to April 5, 2021, our department did not release results until the end of the ED encounter.
Data Extraction
Our medical center uses the Epic EHR (Epic Systems Corporation, Verona, Wisconsin, United States). We extracted study data using Epic's Clarity reporting database, which maintains detailed audit logs for patient access to the MyChart patient portal. For each ED patient included, we extracted information about whether patients viewed laboratory, imaging, electrocardiography, and surgical/anatomical pathology tests results between ED arrival and departure. We also extracted demographic and basic clinical data.
We categorized our primary outcome as result(s) viewed (vs. result(s) not viewed) if a patient viewed at least one (vs. did not view any) test result(s) between ED arrival and departure. We defined ED departure using either admission or discharge timestamps. Demographic data included age, sex, race, ethnicity, preferred language, insurance status, emergency severity index (ESI), and social vulnerability index (SVI). We also extracted disposition status and ESI. The ESI is a widely recognized acuity score typically performed by ED nurses in triage.[11] For each encounter, we captured whether the patient had already been enrolled in MyChart prior to ED arrival.
Non-binary sex data were inconsistently available at the time of data extraction. Primary insurance status associated with each encounter was classified as commercial (including commercial, agency, exchange, and managed care), Medicaid, Medicare, self-pay (including Medicaid-pending), or other (including worker's compensation or other non-classified payors).
Our ED registration workflow and question structure follows the United States Office of Management and Budget (OMB) two-question approach for capturing ethnicity and race.[12] Specifically, patients are asked first about ethnicity (options: Hispanic or Latino, Non-Hispanic/Latino, Unknown, Declined) then race (options: American Indian or Alaska Native, Asian, Black or African American, Native Hawaiian or Other Pacific Islander, White, Some other race, Decline, Unavailable/Unknown). Patients can report multiple races with which they identify (i.e., multiple select). In line with the Institute of Medicine (now National Academy of Medicine)'s proposed approach[12]
[13] and the American Community Survey tables B02001 and B03002 filtered for Texas, we included the following concepts as dichotomous, potentially co-occurring variables representing race and ethnicity: Hispanic or Latino (Hispanic), American Indian or Alaska Native (AIAN), Asian, Black or African American (Black), Native Hawaiian or Other Pacific Islander (NHOPI), White, and Other. All categories are reported in demographic tables. For modeling purposes, race and ethnicity groups with <5% prevalence were combined into Other to allow models to compute (Other, AIAN, and NHOPI in our dataset).
SVI scores were determined using a county level index maintained by the Centers for Disease Control and Prevention's (CDC).[14] The CDC's 2018 SVI uses 15 census variables related to four themes to create theme level and overall percentile scores (with 0–100% represented as 0–1) representing a county's social vulnerability. The themes and corresponding variables are:
-
Socioeconomic status (below poverty, unemployed, income, no high school diploma).
-
Household composition and disability (aged 65 or older, aged 17 or younger, older than age 5 with a disability, single-parent households).
-
Minority status and language (minority, speak English “less than well”).
-
Housing type and transportation (multiunit structures, mobile homes, crowding, no vehicle, group quarters).
We mapped ZIP code and State (to account for zip codes including more than one State) from each patient's home address to their county's SVI score.
ED dispositions were classified as admission (including observation), discharge, and other (including patients who left without being seen, left prior to completion of care, left against medical advice, or immediately went to labor and delivery).
Analysis
We first calculated descriptive statistics for the populations who viewed and did not view results during their ED stay. We then measured the weekly proportion of patients who did and did not view test results during their ED stay. We used weeks to represent time instead of days to smooth weekday-weekend variation, minimizing cyclical trends. We evaluated the association between time (study week) and proportion of patients who viewed results using Kendall's tau (τ) correlation coefficient.
To account for potential confounding, we used binary logistic regression to evaluate result(s) viewed status as a function of time accounting for the following covariate predictors: time (study week), age, SVI, sex, combined race and ethnicity, language, insurance status, and disposition. Continuous variables were modeled using restricted cubic splines (RCSs) – piecewise polynomials constrained to linear relationships in the tails. RCSs allow models to include dynamic non-linear relationships between variables without erratic behavior at the extremes. We modeled age, study week, and SVI as RCSs using five knots (transition points) located at equally spaced quantiles, which have been shown to be adequate for most situations where clearer choices for knots (e.g., known dates of changes or known variable cutoff values) are not available.[15]
Since the unit of evaluation was a patient encounter, some patients accounted for multiple visits. To account for within-patient clustering, we used Huber-White covariance analysis with clustering, where each patient represents a cluster of their encounters.[15] Of our population, 1.7% had missing SVI scores. These scores are likely missing not at random since we care for undomiciled patients without home addresses, which prevents linking patients to SVI scores. To account for missing values, we performed random forest imputation using the missRanger package.[16]
We additionally hypothesized that some patients not enrolled in MyChart at ED arrival would enroll in and use the portal during their ED stay. To account for this, we created an additional binary logistic regression model that included only the patients who were not enrolled in the portal at arrival. The outcome variable was result(s) viewed status (i.e., whether the patient viewed a test result during their stay and by proxy enrolled in MyChart during their stay). The inputs and structure of the model were otherwise identical to the prior model with the following exception: only 0.8% (n = 152) of patients who were not enrolled at ED presentation had Other category insurance. To allow the model to compute, these cases were removed from this model.
Overall model performance was assessed using the concordance index and likelihood ratio χ
2 test compared to the null model. Individual covariates were assessed using Wald statistics. For categorical variables, we calculated odds ratios (ORs). For continuous variables modeled using splines, we calculated ORs comparing interquartile values (high-to-low).
Analytics and visualizations were performed using R version 4.1.2.[17] Two-sided p-values <0.05 were considered statistically significant.
Results
During the study period, there were 60,314 ED encounters representing 31,164 unique patients. In 21,158 (35.2%) encounters, the patients were not enrolled in MyChart at admission. In 7,377 (12.2%) encounters, the patient viewed their results between ED arrival and departure. Of the patients who were not enrolled in MyChart at admission, 927 (4.4%) ultimately viewed results and enrolled in MyChart during their visit. Baseline characteristics for the overall cohort comparing the result(s) viewed and result(s) not viewed groups are shown in [Table 1]. Baseline characteristics for patients who ultimately enrolled after arrival – comparing result(s) viewed (and enrollment by proxy) and result(s) not viewed – are displayed in [Table 2].
Table 1
Baseline characteristics of all ED patients regardless of enrollment status at arrival
|
Not-viewed (N = 52,937)
|
Viewed (N = 7,377)
|
|
Age
|
|
|
|
Median (IQR)
|
54 (37, 68)
|
52 (36, 67)
|
|
Sex
|
|
|
|
Female
|
30,414 (57.5%)
|
4,431 (60.1%)
|
|
Male
|
22,521 (42.5%)
|
2,946 (39.9%)
|
|
Race/Ethnicity (multiple select)
|
|
|
|
Hispanic
|
9,510 (18.0%)
|
1,140 (15.5%)
|
|
American Indian or Alaska Native
|
251 (0.5%)
|
29 (0.4%)
|
|
Asian
|
1,352 (2.6%)
|
428 (5.8%)
|
|
Black
|
17,471 (33.0%)
|
1,332 (18.1%)
|
|
Native Hawaiian or Other Pacific Islander
|
55 (0.1%)
|
13 (0.2%)
|
|
White
|
28,740 (54.3%)
|
5,020 (68.0%)
|
|
Other Race
|
2,422 (4.6%)
|
331 (4.5%)
|
|
Language
|
|
|
|
English
|
50,312 (95.0%)
|
7,108 (96.4%)
|
|
Spanish
|
2,133 (4.0%)
|
169 (2.3%)
|
|
Other
|
492 (0.9%)
|
100 (1.4%)
|
|
Social vulnerability index (SVI)
|
|
|
|
Median (IQR)
|
0.75 (0.55, 0.75)
|
0.69 (0.39, 0.75)
|
|
Insurance
|
|
|
|
Commercial
|
22,488 (42.5%)
|
4,142 (56.1%)
|
|
Medicaid
|
3,795 (7.2%)
|
268 (3.6%)
|
|
Medicare
|
21,584 (40.8%)
|
2,742 (37.2%)
|
|
Other
|
292 (0.6%)
|
8 (0.1%)
|
|
Self-Pay
|
4,778 (9.0%)
|
217 (2.9%)
|
|
Disposition
|
|
|
|
Admit
|
22,981 (43.4%)
|
3,879 (52.6%)
|
|
Discharge
|
27,862 (52.6%)
|
3,265 (44.3%)
|
|
Other
|
2,094 (4.0%)
|
233 (3.2%)
|
|
ESI
|
|
|
|
1
|
362 (0.7%)
|
22 (0.3%)
|
|
2
|
23,314 (44.0%)
|
3,790 (51.4%)
|
|
3
|
25,352 (47.9%)
|
3,316 (45.0%)
|
|
4
|
3,695 (7.0%)
|
220 (3.0%)
|
|
5
|
72 (0.1%)
|
7 (0.1%)
|
|
MyChart status at arrival
|
|
|
|
Activated
|
32,706 (61.8%)
|
6,450 (87.4%)
|
|
Not-activated
|
20,231 (38.2%)
|
927 (12.6%)
|
Note: Proportion of missing values not included.
Table 2
Baseline characteristics of ED patients who were unenrolled at arrival, stratified by those in the not viewed and viewed groups
|
Not-viewed (N = 20,231)
|
Viewed (N = 927)
|
|
Age
|
|
|
|
Median (IQR)
|
48 (32, 63)
|
38 (27, 54)
|
|
Sex
|
|
|
|
Female
|
11,025 (54.5%)
|
539 (58.1%)
|
|
Male
|
9,205 (45.5%)
|
388 (41.9%)
|
|
Race/Ethnicity (multiple select)
|
|
|
|
Hispanic
|
4,534 (22.4%)
|
207 (22.3%)
|
|
American Indian or Alaska Native
|
57 (0.3%)
|
3 (0.3%)
|
|
Asian
|
362 (1.8%)
|
37 (4.0%)
|
|
Black
|
7,900 (39.0%)
|
246 (26.5%)
|
|
Native Hawaiian or Other Pacific Islander
|
32 (0.2%)
|
3 (0.3%)
|
|
White
|
9,567 (47.3%)
|
541 (58.4%)
|
|
Other Race
|
997 (4.9%)
|
37 (4.0%)
|
|
Language
|
|
|
|
English
|
18,677 (92.3%)
|
884 (95.4%)
|
|
Other
|
251 (1.2%)
|
10 (1.1%)
|
|
Spanish
|
1,303 (6.4%)
|
33 (3.6%)
|
|
Social vulnerability index (SVI)
|
|
|
|
Median (IQR)
|
0.75 (0.57, 0.75)
|
0.75 (0.55, 0.75)
|
|
Insurance
|
|
|
|
Commercial
|
8,317 (41.1%)
|
594 (64.1%)
|
|
Medicaid
|
2,289 (11.3%)
|
77 (8.3%)
|
|
Medicare
|
5,686 (28.1%)
|
145 (15.6%)
|
|
Other
|
161 (0.8%)
|
0 (0%)
|
|
Self-pay
|
3,778 (18.7%)
|
111 (12.0%)
|
|
Disposition
|
|
|
|
Admit
|
7,086 (35.0%)
|
304 (32.8%)
|
|
Discharge
|
12,136 (60.0%)
|
594 (64.1%)
|
|
Other
|
1,009 (5.0%)
|
29 (3.1%)
|
|
ESI
|
|
|
|
1
|
168 (0.8%)
|
2 (0.2%)
|
|
2
|
7,384 (36.5%)
|
295 (31.8%)
|
|
3
|
10,520 (52.0%)
|
554 (59.8%)
|
|
4
|
2,046 (10.1%)
|
69 (7.4%)
|
|
5
|
42 (0.2%)
|
6 (0.6%)
|
Note: Proportion of missing values not included.
The weekly proportion of patients who viewed their results is shown in [Fig. 1]. There was an overall trend toward more result(s) viewed based on visual review consistent with Kendall's τ = 0.61 (p <0.0001). For reference, τ = −1 represents a perfect negative monotonic relation between variables, τ = 0 represents no correlation, and τ = 1 represents perfect positive monotonic correlation. We observed an anomalous increase in the proportion of patients who viewed results between May 14 and 17, 2021 with resumption of the prior trend thereafter.
Fig. 1 Weekly proportion of ED patients viewing one or more test results during their ED encounter. ED, emergency department.
The first binary logistic regression model for covariate analysis of all patients yielded a concordance index of 0.680 and statistically significant likelihood ratio χ2 for the overall model (p <0.0001). ORs and 95% confidence intervals (CIs) are shown in [Fig. 2]. The second binary logistic regression model for covariate analysis of patients who were not enrolled at arrival yielded a concordance index of 0.716 and statistically significant likelihood ratio χ
2 for the overall model (p <0.0001). ORs and 95% CIs are shown in [Fig. 3]. Consistent with the correlation analysis, passing time (week of study), the primary outcome, was associated with increased result viewing overall and among patients who were not enrolled at admission when also accounting for additional covariates.
Fig. 2 Overall model (all patients): Odds ratios (ORs) of result(s) viewed vs. result(s) not viewed for all model features with 95% confidence intervals.
Fig. 3 Model of patients not enrolled at arrival: odds ratios (ORs) of result(s) viewed vs. result(s) not viewed for all model features with 95% confidence intervals.
Among all patients, older age, increased SVI (representing increased vulnerability), male sex, Hispanic ethnic group, Black race, Spanish primary language, and public insurance or self-pay status were factors associated with lower odds of viewing patient portal results during the ED encounter. “Admission” disposition, Asian race, and Other race were associated with increased odds of viewing results. Among patients who were not enrolled at arrival, older age, increased SVI, male sex, Black race, public insurance or self-pay status, and Other dispositions were factors associated with lower odds of viewing results. In this group, Asian, Hawaiian, or Pacific Islander race/ethnicity was associated with viewing results (and thus enrolling while in the ED).
Discussion
In our retrospective study using single site EHR data, we observed that patients increasingly reviewed ED test results in real-time since implementation of the ONC's 21st Century Cures Act Final Rule. This association remained significant even after adjusting for covariates among all ED patients and those who were not enrolled in the patient portal at arrival.
The observation that ED patients used their encounter to enroll in the portal and view results during their ED stay highlights the ED as a potential hub for engaging patients in their care and connecting patients and health systems. Unfortunately, we witnessed similar disparities related to age, race and ethnicity, social vulnerability, and insurance status as have been observed in more general patient portal research. These observed disparities highlight the possibility of ED-based portal enrollment and education, given the ED's status as the primary health care contact point for many patients from underrepresented groups. However, given the broad range of disparities, individualized interventions may be needed to improve equitable use of portals.
Given sometimes lengthy waiting and boarding times in EDs across the country, there may be idle periods from the patient's perspective that could be harnessed for portal-related patient outreach and education. The increasing uptake in portal usage during ED visits may facilitate other opportunities for patient engagement in the ED, such as patient-reported outcome measures, patient questionnaires, real-time customer service feedback, and educational opportunities related to health, literacy, financial assistance, or language. Patient engagement may also enable work sharing among patients and providers. For example, patients could use the portal to describe their symptoms, complete reviews of systems using structured data, and provide actionable information to optimize ED triage, flow, and care provision. Furthermore, network analysis of how ED portal enrollment affects downstream patient interactions with health systems may inform future efforts to optimize both scheduled and unscheduled care in ways that are convenient and safe for both patients and health systems.
An interesting observation about portal enrollment during our study comes from the fact that there was no active advertising to patients about the availability of the patient portal in the ED beyond a small note in the AVS, which did not change as part of the deployment of open notes and open results in the ED. This might suggest that patients are inherently interested in viewing their clinical data and actively seek it during clinical encounters, though this notion would be better assessed using qualitative methods.
Anomalous Enrollment Spike
We observed a transient, but substantial, increase in results viewing shortly after open results deployment in mid-May. We sought clarification from our site's MyChart team about possible causes. Inpatient order results were initially released in batches several times per day after the deployment of open results at our hospital. Starting May 13, individual inpatient order results were converted to immediate release. This resulted in an increased number of real-time email and text-message-based push notifications sent to patients in the inpatient setting (including the ED). This default notification setting was disabled for inpatients a few days later due to clinician concerns about having inadequate time to discuss results with patients in the acute setting. However, patients could still access immediately released results and customize their notification preferences as desired. The observed spike (May 14–May 17, 2021) aligned with these release and notification changes (May 13 and May 18, 2021, respectively). The rate of results viewing subsequently returned to its prior trajectory.
While temporarily disabling this feature was appropriate at the time given the need to provide clinicians and patients sufficient time to adjust to open results, this experience highlights an opportunity to facilitate engagement via subtle adjustments to patient portal policies and default settings. We hope to revisit this as a potential strategy for engaging patients during future QI and research efforts.
Strengths and Limitations
Strengths of our study include the use of covariate analysis to account for potential confounders with respect to our primary observation that real-time patient portal use is increasing in the ED, the relatively large number of encounters included, and the yearlong observation period.
Limitations include the single site nature of the study and data-related limitations innate to using EHR data. Furthermore, given the dynamic nature of the clinical environment and EHR, it is possible that other changes to the system that went unaccounted for may have been implemented during the year of observation. Additional unaccounted cofounders might include dynamic influences from waves of COVID-19 infections, influence of critical illness, and influence of particular result types. There are also barriers to portal enrollment and use that may contribute to our findings – especially those indicating inequitable access – that are not well-explored by our quantitative analysis. Future qualitative studies with patients and care teams would provide a helpful perspective on our observed findings. Similarly, we did not evaluate patient behavior related to viewing ED results after the ED encounter, which warrants additional study. Finally, the limitation of this study to an academic quaternary care center limits the generalizability of these findings to community hospitals, county hospitals, or rural centers. To address these limitations, we are engaging in a multisite study with EDs across the country.
Future Directions
There are several features of this analysis that were underexplored and would benefit from future study. Geospatial analysis could serve to identify patients at risk for under-enrollment and use of patient portals. Our early attempt to include county-level SVI data generated from ZIP and State fields as a covariate in our models provides a signal suggesting that this is a worthy avenue to explore. More granular methods that map full addresses to census tracts will be more accurate for identifying patients or pursuing community-level investment in digital literacy training. Another unexplored question relates to whether patients logged into the portal but were subsequently unable to successfully locate test results. Our data extraction technique did not allow us to address this question, but we will attempt to do so in further studies. There are numerous ongoing efforts nationally to evaluate the impact of open results on many different groups. We hope to continue to contribute to the understanding of how open results impact patients and clinicians in acute care settings.
We also note that while it is important for health systems to evaluate overall trends in ED-based portal use, it is also important to characterize the extent of care disparities in ED portal access and use. Prior studies examining portal access and use in general have revealed disparities affecting patients who are older, from underrepresented racial or ethnic groups, non-English-speaking, and lacking digital and overall literacy.[6]
[18]
[19]
[20]
[21]
[22]
[23] Given the national role of EDs as safety nets for underserved patients, ED-based portal enrollment and education may provide critical opportunities for narrowing this “digital divide”[18] and we hope to pursue further study in this domain.
Conclusion
In this retrospective observational study using EHR data from a single academic quaternary care center, real-time ED patient portal use increased among ED patients since implementation of the 21st Century Cures Act. Patients from racial, ethnic, language, and socioeconomic minorities were underrepresented among ED portal users overall and the group of patients who enrolled and used the portal during their ED stay. Further mixed methods research in multiple EDs and among underrepresented patients is necessary for more generalizable application of our observations.
Clinical Relevance Statement
Clinical Relevance Statement
Since the implementation of 21st Century Cures Act ONC Final Rule, patients have increasingly used patient portals to manage their health care. However, real-time use of patient portals in the ED has not been previously studied. This study provides an initial view into which patients use portals in real-time during ED encounters, which has far-reaching implications for local and population level patient outreach and addressing care disparities.
Multiple Choice Questions
Multiple Choice Questions
-
What did the Office of the National Coordinator for Health Information Technology (ONC)'s Final Rule prohibit providers, vendors, exchanges, and others from doing?
Correct Answer: The correct answer is b. This prohibition effectively created the concept of information sharing (including open results) across the country, drastically increasing the viability of patient access to their test results.
-
In this study, no intervention was made to inform patients of the availability of their data. Patient viewing of test results via patient portal in the ED __________?
-
Increased.
-
Decreased.
-
Stay the same.
Correct Answer: The correct answer is a. Despite the absence of an intervention informing patients of access to their information beyond simply releasing results, there was a continuous increase in the proportion of patients who viewed their results during their ED stay.
