Examining the Relationship between Health Literacy, Health Numeracy, and Patient Portal Use

Keywords

patient portals - health literacy - health numeracy - patient portal engagement

Background and Significance

Patient portals are a health information technology tool that can help patients engage in their health care by providing access to personal health information. Originally implemented in outpatient settings, patient portals allow patients to manage appointments, retrieve test results, pay medical bills, and communicate with providers through secure messaging. Inpatient portals offer functions tailored to the hospital environment, such as checking vital signs, viewing a daily care plan, or ordering meals. While patient portals have been shown to positively impact patient outcomes, such as improving knowledge, self-efficacy, and engagement,[1] [2] barriers to portal use may limit these benefits.

In practice, portal use may be affected by both the health literacy and numeracy (HLN) of patients themselves.[3] Health literacy is defined as the ability to obtain, understand, and apply information to inform decisions about one's health,[4] while health numeracy specifically concerns how a patient obtains, understands, and applies quantitative information.[5] In the outpatient setting, individuals with limited health literacy are less likely to register for patient portals, and those who do register then use portals less compared with individuals with proficient health literacy.[6] Furthermore, individuals with limited HLN have more difficulty using outpatient portals.[7] [8]

The present study assesses the relationship between HLN and portal use, considering the context of inpatient settings, where patient portal implementation has been initiated more recently. Specifically, we seek to test whether people with higher HLN use portal technology at a higher rate. Additionally, patient portals made available in the inpatient setting support access to the health information on hospital-provided devices and broadband,[9] enabling technology use and experience viewing health information. Our study, thus, seeks to evaluate whether interactions with portal technology itself might subsequently affect future assessments of patients' literacy or numeracy skills. Answering these questions will help clarify whether offering patient portals in inpatient settings can improve HLN skills and whether the inpatient setting can support interventions that enhance the use of this patient-facing health technology.

Methods

Study Setting and Design

Our study was conducted at The Ohio State Wexner Medical Center, a large academic medical center (AMC), that offers its hospitalized patients access to an inpatient portal (MyChart Bedside; Epic Systems: Verona, WI, United States) via an AMC-provided tablet, as well as an outpatient portal outside of hospital stays (MyChart; Epic Systems: Verona, WI, United States). The AMC-provided tablet was offered to all patients admitted to an AMC hospital that meet eligibility criteria for the organizational implementation of the tablets: over the age of 18 years, not a prisoner, and English documented as their preferred language. The tablet provisioning process involved unit staff (e.g., nurses and patient care associates) approaching patients and asking them if they would like to have a tablet. For patients who responded affirmatively, the unit staff would assign the patient their tablet by inputting the tablet bar code into the patient's electronic medical record (EMR). Then, the unit staff would guide the patient to the inpatient portal “Terms and Conditions” page. There, the patient would complete both the enrollment of their tablet and activate their inpatient portal account. Unit staff could also document if a patient did not have the cognitive capacity to accept the terms of use for the tablet.

The inpatient portal on the provisioned tablets included a variety of functions such as enabling patients to communicate with their care team or check laboratory results. After discharge, patients could use the outpatient portal to continue to engage in their health care. Details about the list of functions available to the patients via the inpatient and outpatient portals can be found in the supplementary material (see [Supplementary Tables A.1] and [B.1], available in the online version).

Table 1 Demographic characteristics of patients at the time of their hospital admission, summary statistics of their inpatient portal use, their outpatient portal use after being discharged, and the dichotomized survey items representing patient health literacy and numeracy skills

Characteristic	n = 654[a]
Age
18–29	106 (16.2%)
30–39	141 (21.6%)
40–49	126 (19.3%)
50–59	138 (21.1%)
60–69	112 (17.1%)
≥70	31 (4.7%)
Gender
Male	251 (38.4%)
Female	403 (61.6%)
Race
White	516 (78.9%)
Black	113 (17.3%)
Other	25 (3.8%)
Educational attainment
Less than high school	34 (5.2%)
High school	166 (25.4%)
Some college	207 (31.7%)
College graduate	172 (26.3%)
Graduate or beyond	75 (11.5%)
Marital status
Married	327 (50.0%)
Living as married	36 (5.5%)
Widowed	88 (13.5%)
Divorced	24 (3.7%)
Separated	17 (2.6%)
Single, never married	162 (24.8%)
Employment
Employed	320 (48.9%)
Unemployed	69 (10.6%)
Homemaker	18 (2.8%)
Student	9 (1.4%)
Retired	75 (11.5%)
Disabled	141 (21.6%)
Other	22 (3.4%)
Household income
$0–$19,999	223 (34.1%)
$20,000–$49,999	160 (24.5%)
$50,000–$99,999	158 (24.2%)
$100,000–$199,999	94 (14.4%)
$200,000 or more	19 (2.9%)
General health
Excellent	20 (3.1%)
Very good	95 (14.5%)
Good	244 (37.3%)
Fair	219 (33.5%)
Poor	76 (11.6%)
Have you ever used a patient portal?
Yes	481 (73.5%)
No	137 (20.9%)
I don't know	36 (5.5%)
Charlson score	1 (0, 2)
Length of tablet provision (days)	4 (3, 7)
Count of IPP sessions during admission	16 (9, 29)
Comprehensive IPP use	173 (26.5%)
Count of OPP sessions 6 month postdischarge	9 (0, 45)
I feel uncomfortable with health information that has a lot of numbers
Strongly agree to neutral	373 (57.0%)
Somewhat to strongly disagree	281 (43.0%)
I depend on numbers and statistics to help me make decisions about my health
Strongly disagree to neutral	375 (57.3%)
Somewhat to strongly agree	279 (42.7%)
I know how to take my medications
Strongly disagree to neutral	29 (4.4%)
Somewhat to strongly agree	625 (95.6%)
I can fill out medical forms
Strongly disagree to neutral	60 (9.2%)
Somewhat to strongly agree	594 (90.8%)
I frequently have someone help me read hospital material
Strongly agree to neutral	209 (32.0%)
Somewhat to strongly disagree	445 (68.0%)
I frequently have difficulty using written health information
Strongly agree to neutral	122 (18.7%)
Somewhat to strongly disagree	532 (81.3%)

Abbreviations: IPP, inpatient portal; OPP, outpatient portal.

^a n (%); median (Q1, Q3).

This study uses data collected as part of a pragmatic randomized controlled trial (RCT) designed to examine patient portal use in the inpatient setting.[10] The parent RCT study used a 2 × 2 factorial design to test the effects of access to advanced patient portal functions (versus limited functions), and of in-person training (versus video training), on patient portal use. Only patients randomized into the study arm with full access to advanced patient portal functions and video training were considered in the present study (n = 718). The video tutorial provided a brief introduction to the inpatient portal and its functionality. This analysis focused specifically on the impact of HLN on patient portal use rather than on the impacts of the study interventions on outcomes.

Study enrollment occurred between December 2016 and August 2019 and included patients who were at least 18 years of age, spoke and read English, were not involuntarily detained, and who had agreed to accept the tablet. Participants completed an admission survey on the tablet shortly after provisioning, and a telephone survey 6 months after discharge.

Outcome Measures

Outcome measures for portal use relative to the frequency and diversity of functions used via both the inpatient and outpatient portals were computed from audit log files obtained through the AMC's Information Warehouse (IW). Log files contain records of all actions performed by patients while using patient portals. We processed log files to quantify how often patients used portals and, in the case of the inpatient portal, how many different functions they used. Three previously validated outcome measures[11] [12] were considered: (1) inpatient portal frequency of use, defined as the count of inpatient portal sessions during the patient's hospital stay; (2) comprehensiveness of inpatient portal use, a binary outcome for the patients who used eight or more of the ten MyChart Bedside functions available; (3) outpatient portal frequency of use, defined as the count of outpatient portal sessions in 6 months between the administration of the admission and postdischarge surveys. Portal sessions were calculated as the coherent, limited, and uninterrupted use between login and logout of one or more of the patient portal functions as recorded by the IW.

Demographic Characteristics

Patients' age, gender, and race were acquired from the EMR through the AMC's IW. Other patient sociodemographic characteristics (e.g., marital status, level of education, household income, and employment) were acquired from survey questions.

Survey Measures

Enrolled patients were administered surveys that included questions related to HLN. HLN was measured on a 6-item, 5-point Likert scale (1 = “strongly disagree” and 5 = “strongly agree”). Two items focused on quantitative information (e.g., “I depend on numbers and statistics to help me make decisions about my health”); two items focused on medications and medical forms (e.g., “I can fill out medical forms”); the last two items focused on the use of written information (e.g., “I frequently have someone help me read hospital material”). The questions were adapted from validated measures and included three items from a health literacy screening questionnaire,[13] based on Chew et al's work,[14] which is widely used to identify individuals with inadequate health literacy; one item was adapted from the patient activation measure[15] and two analogous items that focused on quantitative information from the Health Information National Trends Survey[16] were included to capture patient health numeracy skills.

Questions were asked at two time points: upon enrollment into the RCT during hospital admission (admission survey) and then at 6 months after discharge (postdischarge survey). Only data from patients who answered all six HLN-related items from the admission survey and had complete demographic data were included in this analysis. Ultimately, the estimates generated from this study were based on a sample composed of 654 patients. However, multiple imputation for missing demographic and HLN items at admission time was performed as a sensitivity analysis for the primary outcomes and patient portal use and can be found in the supplementary material (see section D). Patients' household income was the most underreported variable; it was found to not be correlated with patients' HLN skill levels (see [Supplementary Table D.2], available in the online version) and was imputed as a function of patient educational attainment, employment status, and other demographic characteristics; the few HLN item responses missing were imputed as a function of the patient demographics. Estimates from the multiple imputation by chained equations did not ultimately differ from the main findings presented below (see [Supplementary Table D.3], available in the online version).

Table 2 Inpatient portal use metrics (frequency of use and comprehensiveness of use) as a function of patient health literacy and numeracy

	IPP frequency of use[a]		IPP comprehensiveness of use[a]
Variable	IRR	95% CI	OR	95% CI
I feel uncomfortable with health information that has a lot of numbers
Strongly agree to neutral	–	–	–	–
Somewhat to strongly disagree	1.13[b]	1.01, 1.28	1.27	0.83, 1.95
I depend on numbers and statistics to help me make decisions about my health
Strongly disagree to neutral	–	–	–	–
Somewhat to strongly agree	1.01	0.89, 1.13	1.26	0.83, 1.91
I know how to take my medications
Strongly disagree to neutral	–	–	–	–
Somewhat to strongly agree	0.88	0.65, 1.16	1.20	0.44, 3.59
I can fill out medical forms
Strongly disagree to neutral	–	–	–	–
Somewhat to strongly agree	0.98	0.79, 1.21	1.12	0.52, 2.51
I frequently have someone help me read hospital material
Strongly agree to neutral	–	–	–	–
Somewhat to strongly disagree	0.94	0.82, 1.08	0.88	0.55, 1.43
I frequently have difficulty using written health information
Strongly agree to neutral	–	–	–	–
Somewhat to strongly disagree	1.02	0.86, 1.21	0.94	0.53, 1.71
No. Obs.	654		654

Abbreviations: CI, confidence interval; IPP, inpatient portal; IRR, incidence rate ratio; OR, odds ratio.

^a Coefficients adjusted for length of tablet provision and patient demographic characteristics.

^b p < 0.05.

Table 3 Negative binomial regression model for the count of outpatient portal sessions in the 6 months following discharge from the medical center as a function of patient health literacy and numeracy

	OPP frequency of use[a]
Variable	IRR	95% CI
I feel uncomfortable with health information that has a lot of numbers
Strongly agree to neutral	–	–
Somewhat to strongly disagree	0.96	0.71, 1.30
I depend on numbers and statistics to help me make decisions about my health
Strongly disagree to neutral	–	–
Somewhat to strongly agree	0.79	0.58, 1.08
I know how to take my medications
Strongly disagree to neutral	–	–
Somewhat to strongly agree	1.13	0.54, 2.20
I can fill out medical forms
Strongly disagree to neutral	–	–
Somewhat to strongly agree	0.56[b]	0.31, 0.98
I frequently have someone help me read hospital material
Strongly agree to neutral	–	–
Somewhat to strongly disagree	0.94	0.67, 1.32
I frequently have difficulty using written health information
Strongly agree to neutral	–	–
Somewhat to strongly disagree	1.02	0.67, 1.52
No. Obs.	654

Abbreviations: CI, confidence interval; IRR, incidence rate ratio; OPP, outpatient portal.

^a Coefficients adjusted for previous OPP frequency of use and patient demographics characteristics.

^b p < 0.05.

Statistical Analysis

To test whether HLN skills predict patient portal engagement, we used generalized linear models with dichotomized survey items as predictors while controlling for participants' sociodemographic characteristics. Items from the admission survey were dichotomized to balance response frequency distributions: “strongly agree” to “somewhat agree” were considered positive; all other responses were considered negative, including the neutral answer, to minimize imbalance; negatively worded items were reverse coded. To account for overdispersion, negative binomial regression models were used to estimate the association between health literacy and frequency of inpatient and outpatient portal use; a logistic regression model was used to study comprehensive inpatient portal use. In both models related to inpatient portal use, the tablet's length of provision, measured in days, was included as an offset. In the model related to outpatient portal use, the count of sessions in 6 months prior to hospital admission was included as a control variable.

In the absence of guidelines about meaningful change in the metrics of patient portal engagement adopted by the present study, odds ratios and incidence rate ratios (IRRs) were considered as a proxy for effect size. We hypothesized that individuals with high HLN skills would engage with patient portals at twice the frequency of individuals with low HLN skills or would be twice as likely to be a comprehensive inpatient portal user, an effect qualified as small.[17] Using the characteristics of our sample, and on the basis of a z-test for Poisson distributed count data, we calculated that a sample of 654 subjects, allocated with a 2:1 ratio to the high HLN skill level versus the low HLN skill level, with an average baseline count of four inpatient portal sessions per day, would be able to detect an increase in incidence rate as low as 1.14 with a power of 90% and an α level of 0.05. Similarly, on the basis of a z-test for logistic regression, we calculated that with a 2:1 ratio of high HLN skill level to low HLN skill level individuals and a base proportion of comprehensive inpatient portal use of 0.26, our sample size would be able to detect an increase in the likelihood of being a comprehensive inpatient portal user of 1.80 with a power of 90% and an α level of 0.05, if the effect was indeed present in the data. The calculation of detectable effects based on the sample size for our secondary data analysis was performed using G*Power[18] and details can be found in section C of the [Supplementary Material] (available in the online version).

We additionally performed a per-function analysis of the inpatient and outpatient portal sessions using the same model specifications described above to disaggregate the effect of interacting with the portals for reasons not related to the management of their health-related issues. The per-function results did not alter the conclusions of our main analysis and can be found in the [Supplementary Material] (see sections A and B, available in the online version).

Finally, the longitudinal effect of portal use on HLN was tested using numerical values of survey items from both the admission and postdischarge surveys as repeated measures in a linear mixed-effects model, controlling for the participants' demographic characteristics and the outpatient portal frequency of use in 6 months between the two surveys. All statistical analyses were performed using R 4.1.2,[19] and the imputation of missing values relied on the mice package.[20]

Results

Participant characteristics are presented in [Table 1]. On average, participants appeared to be neutral or moderately confident with their HLN skills, felt strongly that they knew how to take their medications, and felt they could typically complete medical forms by themselves.

Analysis of inpatient portal use showed no significant associations between HLN and frequency with which patients accessed the inpatient portal in all predictors except one: those who declared that they were more comfortable with health information containing numbers used the portal more frequently (IRR 1.13, 95% confidence interval [CI] 1.01 to 1.28). There was no difference in the comprehensiveness of inpatient portal functions used between the levels of patients' HLN while they were admitted at the AMC ([Table 2]).

The analysis of the use of the outpatient portal ([Table 3]) showed one significant association between a survey item and the number of portal sessions (“I can fill out medical forms,” IRR 0.56, 95% CI 0.31 to 0.98), but no consistent trends emerged.

A longitudinal analysis comparing responses provided on the survey administered 6 months postdischarge to the original responses on the admission survey items was performed. The response rate to the postdischarge survey was 40%, and nonresponders were more likely to have lower baseline HLN self-reported skills (see [Supplementary Table E.1], available in the online version); 15 patients were reported deceased at 6 months and were excluded from the analysis. The analysis did not reveal significant differences between the levels of self-reported health numeracy; however, the health literacy items related to written information showed a modest increase (see [Table 4]). At 6 months postdischarge, compared with during admission, patients required help reading hospital materials less frequently (disagreement with the statement “I frequently have someone help me read hospital material” increased by 0.17 on the 5-point Likert scale, 95% CI 0.03 to 0.30) and used written health information with more ease (disagreement with the statement “I frequently have difficulty using written health information” increased by 0.14 points, 95% CI 0.02 to 0.26).

Table 4 Changes in levels of health literacy and numeracy skills reported by patients 6 months after hospital discharge[a]

	6 month postdischarge[b]
Outcome	Difference	95% CI
I feel uncomfortable with health information that has a lot of numbers[c]	−0.11	−0.29, 0.07
I depend on numbers and statistics to help me make decisions about my health	−0.01	−0.16, 0.14
I know how to take my medications	0.05	−0.02, 0.12
I can fill out medical forms	0.07	−0.01, 0.15
I frequently have someone help me read hospital material[c]	0.17[d]	0.03, 0.30
I frequently have difficulty using written health information[c]	0.14[d]	0.02, 0.26
n	639

Abbreviation: CI, confidence interval.

^a Estimates and confidence intervals are computed via mixed-effects adjusted linear regression models, using the answers to the admission survey and 6 months postdischarge survey and controlling for outpatient portal frequency of use and demographic characteristics.

^b Coefficients adjusted for outpatient portal frequency of use and patient demographics characteristics.

^c Item was reverse coded.

^d p < 0.05.

Discussion

Our analysis assessed the relationship between HLN and portal use. Unlike studies that have examined health literacy or numeracy separately, using a single survey item for each,[7] [21] [22] our analysis used multiple items which allowed us to examine and compare distributions of these items as well as identify which might be related to portal use. We expected to find HLN to be related to patient portal use, mirroring results that have linked limited health literacy with lower portal use in the outpatient setting.[6] However, our results do not corroborate that finding for outpatient portal use. And in terms of inpatient portal use, despite concerns that low HLN might act as a barrier to portal adoption,[23] our findings align with the limited number of available studies that also report that health literacy did not impact patient portal use during hospitalizations.[24] [25]

Plausible explanations for the lack of association found between patient HLN and portal use can be found in the differences between the present study and extant research, particularly the one in the outpatient setting. Available findings come from research focused on individuals managing a single, often long-term illness[6] [26] [27]; our analytical sample included patients with a wide array of health issues, not just chronic conditions. Additionally, our primary outcomes were derived from the log file analysis. This approach differs from prior studies of patient portals which relied on survey analysis of self-reported measures concerning participants' health literacy and their measures of technology use. This discrepancy might indicate that, given the readily available technology (or, in the case of the inpatient portal, of technology supplied by the healthcare provider), behavioral data derived from log files might provide a picture that differs from individuals' perceptions about their use of technology. Nonetheless, the introduction of an inpatient portal via hospital-provided tablets may have practical implications, including a cascading positive effect that could contribute to further adoption of patient-facing health information technology tools, found for example in the case of telehealth visits.[28]

Our findings further show a small positive effect of outpatient portal use after discharge on perceived HLN 6 months postdischarge. This finding could result from the exposure to the inpatient portal in the hospital setting or from the video-based tutorial. Prior work examining the impact of either an in-person tutorial or a video-based training on patient portal use showed improvements in eHealth literacy after 6 months—a concept related to HLN but focused on the electronic context.[29] Other related work suggests that communication between the care team and the patient during hospitalization, as well as clarity of discharge instructions, can impact health literacy.[30] Ultimately, these findings suggest that the early postdischarge period might represent an opportune time to intervene on patients' HLN.

There are limitations to our findings. First, we did not evaluate influences of clinical- and provider-level factors on patient portal adoption, use, and use over time. Second, our findings are based on a single patient portal platform, which includes both MyChart and MyChart Bedside, and may not be generalizable to other patient portals. Third, our study followed the AMC's protocol for offering tablets in which nursing staff assessed a patient's ability to utilize a portal as part of the decision to offer the portal; this approach could have biased our sample toward patients who had higher perceived HLN.[31] As prior work has highlighted bias in perceptions of patients' ability to use the portal and provisioning of portals,[32] [33] [34] future work should take this potential bias into account and focus on the importance of asking all patients if they are interested in using a portal as well as consider how to systematically assess patients' capacities to use portals when evaluating overall use and impacts. This effort should extend to include non-English speakers and members of underrepresented groups.[35]

Conclusion

The present study tested the relationship between patient HLN and the use of patient portals, yet our findings did not show evidence that these attributes were predictors of inpatient portal use. At the same time, outpatient portal use showed a small positive effect on self-reported health literacy, but a corresponding effect on health numeracy was not found. In an era when access to patient-facing health information technologies continues to expand, it will be important to continue to consider how to best promote and support patients' adoption and use of patient portals to ensure these tools can be leveraged to help patients engage in their own health care.

Clinical Relevance Statement

AMCs should design and deploy targeted initiatives to encourage adoption and continued use of patient portals across their patient populations. Patients' hospital encounters may provide opportunities to deliver interventions to patients and provide them with the training required to improve their HLN skills that can ultimately help patients better manage their own health care.

Multiple Choice Questions

1. When assessing the likelihood of patient portal use during hospitalization, which baseline characteristic(s) should be considered?

   • a. Health literacy skills

   • b. Numeracy skills

   • c. Prior electronic portal use

   • d. All of the above

   Correct Answer: The correct answer is option d. Patients' baseline HLN skills and prior electronic portal use are all important considerations when exploring the likelihood of portal use during hospitalization.

2. Which of the following strategies should be implemented to promote inpatient portal adoption and use?

   • a. Digital skills assessment to evaluate whether patients will be able to successfully use the portal independently

   • b. Systematically querying all patients to gauge their interest in portal use

   • c. HLN assessment to evaluate whether patients will be able to successfully use the portal independently

   • d. None of the above

   Correct Answer: The correct answer is option b. Systematically querying all patients to gauge their interest in portal use is important from not only an individual patient perspective but also as a means to promote population-based health information technology equity.

• References

• 1 Dendere R, Slade C, Burton-Jones A, Sullivan C, Staib A, Janda M. Patient portals facilitating engagement with inpatient electronic medical records: a systematic review. J Med Internet Res 2019; 21 (04) e12779
• 2 Han HR, Gleason KT, Sun CA. et al. Using patient portals to improve patient outcomes: systematic review. JMIR Human Factors 2019; 6 (04) e15038
• 3 Smith SG, O'Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc 2015; 22 (04) 888-895
• 4 Ratzan SC. Health literacy: communication for the public good. Health Promot Int 2001; 16 (02) 207-214
• 5 Rudd RE. Improving Americans' health literacy. N Engl J Med 2010; 363 (24) 2283-2285
• 6 Sarkar U, Karter AJ, Liu JY. et al. The literacy divide: health literacy and the use of an internet-based patient portal in an integrated health system-results from the diabetes study of northern California (DISTANCE). J Health Commun 2010; 15 (2, Suppl 2): 183-196
• 7 Mackert M, Mabry-Flynn A, Champlin S, Donovan EE, Pounders K. Health literacy and health information technology adoption: the potential for a new digital divide. J Med Internet Res 2016; 18 (10) e264
• 8 Taha J, Sharit J, Czaja SJ. The impact of numeracy ability and technology skills on older adults' performance of health management tasks using a patient portal. J Appl Gerontol 2014; 33 (04) 416-436
• 9 Kelly MM, Hoonakker PLT, Coller RJ. Inpatients sign on: an opportunity to engage hospitalized patients and caregivers using inpatient portals. Med Care 2019; 57 (02) 98-100
• 10 McAlearney AS, Sieck CJ, Hefner JL. et al. High Touch and High Tech (HT2) Proposal: transforming patient engagement throughout the continuum of care by engaging patients with portal technology at the bedside. JMIR Res Protoc 2016; 5 (04) e221
• 11 Huerta T, Fareed N, Hefner JL. et al. Patient engagement as measured by inpatient portal use: methodology for log file analysis. J Med Internet Res 2019; 21 (03) e10957-e10957
• 12 Di Tosto G, McAlearney AS, Fareed N, Huerta TR. Metrics for outpatient portal use based on log file analysis: algorithm development. J Med Internet Res 2020; 22 (06) e16849
• 13 Wallace LS, Rogers ES, Roskos SE, Holiday DB, Weiss BD. Brief report: screening items to identify patients with limited health literacy skills. J Gen Intern Med 2006; 21 (08) 874-877
• 14 Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med 2004; 36 (08) 588-594
• 15 Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the patient activation measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004; 39 (4 Pt 1): 1005-1026
• 16 National Cancer Institute. About HINTS. Accessed June 06, 2022 at: https://hints.cancer.gov/about-hints/learn-more-about-hints.aspx
• 17 Chen H, Cohen P, Chen S. How big is a big odds ratio? interpreting the magnitudes of odds ratios in epidemiological studies. Commun Stat Simul Comput 2010; 39 (04) 860-864
• 18 Faul F, Erdfelder E, Buchner A, Lang AG. Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods 2009; 41 (04) 1149-1160
• 19 R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2021. Accessed June 06, 2022 at: http://www.R-project.org
• 20 van Buuren S, Groothuis-Oudshoorn K. Mice: multivariate imputation by chained equations in R. J Stat Softw 2011; 45: 1-67
• 21 Altin SV, Finke I, Kautz-Freimuth S, Stock S. The evolution of health literacy assessment tools: a systematic review. BMC Public Health 2014; 14: 1207-1207
• 22 Tieu L, Schillinger D, Sarkar U. et al. Online patient websites for electronic health record access among vulnerable populations: portals to nowhere?. J Am Med Inform Assoc 2017; 24 (e1): e47-e54
• 23 Grossman LV, Masterson Creber RM, Ancker JS. et al. Technology access, technical assistance, and disparities in inpatient portal use. Appl Clin Inform 2019; 10 (01) 40-50
• 24 Walker DM, Hefner JL, Fareed N, Huerta TR, McAlearney AS. Exploring the digital divide: age and race disparities in use of an inpatient portal. Telemed J E Health 2020; 26 (05) 603-613
• 25 Davis SE, Osborn CY, Kripalani S, Goggins KM, Jackson GP. Health literacy, education levels, and patient portal usage during hospitalizations. AMIA Annu Symp Proc 2015; 2015: 1871-1880
• 26 Lor M, Koleck TA, Bakken S, Yoon S, Dunn Navarra AM. Association between health literacy and medication adherence among hispanics with hypertension. J Racial Ethn Health Disparities 2019; 6 (03) 517-524
• 27 Martinez W, Hackstadt AJ, Hickson GB. et al. The my diabetes care patient portal intervention: usability and pre-post assessment. Appl Clin Inform 2021; 12 (03) 539-550
• 28 Hsiao V, Chandereng T, Lankton RL. et al. Disparities in telemedicine access: a cross-sectional study of a newly established infrastructure during the COVID-19 pandemic. Appl Clin Inform 2021; 12 (03) 445-458
• 29 Lyles CR, Tieu L, Sarkar U. et al. A randomized trial to train vulnerable primary care patients to use a patient portal. J Am Board Fam Med 2019; 32 (02) 248-258
• 30 Jessup RL, Osborne RH, Buchbinder R, Beauchamp A. Using co-design to develop interventions to address health literacy needs in a hospitalised population. BMC Health Serv Res 2018; 18 (01) 989
• 31 Gaughan AA, Walker DM, Sova LN, Vink S, Moffatt-Bruce SD, McAlearney AS. Improving provisioning of an inpatient portal: perspectives from nursing staff. Appl Clin Inform 2022; 13 (02) 355-362
• 32 Arcury TA, Quandt SA, Sandberg JC. et al. Patient portal utilization among ethnically diverse low income older adults: observational study. JMIR Med Inform 2017; 5 (04) e47-e47
• 33 Irizarry T, Shoemake J, Nilsen ML, Czaja S, Beach S, DeVito Dabbs A. Patient portals as a tool for health care engagement: a mixed-method study of older adults with varying levels of health literacy and prior patient portal use. J Med Internet Res 2017; 19 (03) e99
• 34 Sadasivaiah S, Lyles CR, Kiyoi S, Wong P, Ratanawongsa N. Disparities in patient-reported interest in web-based patient portals: survey at an urban academic safety-net hospital. J Med Internet Res 2019; 21 (03) e11421
• 35 Lyles CR, Fruchterman J, Youdelman M, Schillinger D. Legal, practical, and ethical considerations for making online patient portals accessible for all. Am J Public Health 2017; 107 (10) 1608-1611