Assessing Electronic Health Literacy at an Urban Academic Hospital

Abstract

Background Residents of the Bronx suffer marked health disparities due to socioeconomic and other factors. The coronavirus disease 2019 pandemic worsened these health outcome disparities and health care access disparities, especially with the abrupt transition to online care.

Objectives This study classified electronic health literacy (EHL) among patients at an urban, academic hospital in the Bronx, and assessed for associations between EHL levels and various demographic characteristics.

Methods We designed a cross-sectional, observational study in adults 18 years or older presenting to the Montefiore Einstein Center for Cancer Care (MECCC) Department of Radiation Oncology or the Montefiore Department of Medicine in the Bronx. We assessed EHL using the eHealth Literacy Scale (eHEALS) survey, a previously validated tool, and our newly developed eHealth Literacy Objective Scale-Scenario Based (eHeLiOS-SB) tool.

Results A total of 97 patients recruited from the MECCC and Department of Medicine participated in this study. There was a statistically significant association between age and EHL as assessed by both eHEALS and eHeLiOS-SB, with older adults having lower EHL scores. Additionally, a question designed to assess general attitudes toward digital health technologies found that most participants had a positive attitude toward such applications.

Conclusion Many patients, especially older adults, may require additional support to effectively navigate telehealth. Further research is warranted to optimize telemedicine strategies in this potentially-marginalized population and ultimately to create telehealth practices accessible to patients of all ages and demographics.

Keywords

digital health literacy - electronic health literacy - eHEALS - eHeLiOS-SB

Background and Significance

Marked health disparities have long existed in the New York City borough of the Bronx, which is home to a racially and ethnically diverse population and has the lowest per capita income of any county in New York State.[1] Along these lines, Bronx residents face higher rates of chronic conditions than the other New York City boroughs,[2] and during the coronavirus disease 2019 (COVID-19) pandemic, the Bronx had the highest rates of COVID-related hospitalizations and deaths.[3] Health disparities and access to care were further worsened for Bronx households; 38% of whom do not have home access to broadband internet,[4] when many health care systems abruptly transitioned to providing care online in response to COVID-19.[5] [6]

The use of digital health technologies such as online patient portals and remote patient monitoring apps has also increased in recent years.[7] [8] These technologies have been vital in managing chronic diseases during the pandemic without requiring in-person clinic visits. Bronx residents may lack the reliable internet access required for telemedicine and other online health services or the skills to effectively navigate these tools. Additionally, the rate of higher education among Bronx residents (defined by the attainment of a Bachelor's degree or higher) is estimated to be nearly half that of the national average (20[9] vs. 38%[10]), which may further hinder Bronx residents from effectively using online health resources. Therefore, it is essential to be able to assess electronic health literacy (EHL; also sometimes called Digital Health Literacy), which is the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained,[11] in this population to better tailor digital health interventions. Questionnaires, such as the eHealth Literacy Scale (eHEALS) have been developed and validated previously to study EHL.[12] [13] [14] [15] [16] [17]

The eHEALS questionnaire is designed to evaluate patients' subjective, self-assessed comfort with technology and EHL, but does not objectively assess patients' familiarity with or ability to complete common internet-based tasks. To address this gap, our study team designed a 10-question objective test, the eHealth Literacy Objective Scale-Scenario Based (eHeLiOS-SB), to evaluate participants' ability to manage common EHL tasks and to navigate digital health information. eHeLiOS-SB was designed to evaluate nine different skills deemed relevant to health-related use of the internet: (1) operational skills, (2) navigation skills, (3) seeking/ soliciting health information on search engines, (4) ability to comprehend and understand the information, (5) ability to appraise health information, (6) ability to identify misinformation on the web and social media platforms based on widely known basic scientific knowledge, (7) ability to protect users' own privacy and respect others' privacy on the web, (8) ability to engage with digital health platforms trustfully and with confidence, and (9) ability to reach out to health care providers using digital tools. eHeLiOS-SB, as opposed to eHEALS, asks participants to read a scenario about a situation that could arise while using the internet and indicate how they would proceed from a list of several options. Thus, eHeLiOS-SB is more of an objective measure of EHL than eHEALS, which assesses a patient's subjective self-assessment of their EHL.

This study's objectives were to (1) calculate the correlation of EHL scores between the well-established measure of subjective EHL called eHEALS and our newer measure of objective EHL (called eHeLiOS-SB) in an adult population of underserved patients in the Bronx, (2) to classify patient EHL in this population as “good,” “moderate,” or “low,” and (3) assess for associations between various demographic characteristics and patients' EHL.

Methods

Study Design and Participants

We conducted a cross-sectional, observational study in adults presenting to either the Montefiore Einstein Center for Cancer Care (MECCC) Department of Radiation Oncology or Department of Medicine outpatient clinics (either in-person or via telemedicine) for an initial consultation, follow-up, or radiotherapy treatment. Patients were eligible if they had an Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 3 and were able to complete questionnaires in either English or Spanish. The ECOG performance scale score is a measurement of how a “disease impacts a patient's daily living abilities.”[18] An ECOG performance score of 0 indicates that the disease has no impact on a patient's ability to perform tasks of daily living, and a score of 4 indicates that the patient is completely disabled due to their disease. The research team screened the electronic medical records of patients scheduled in the clinic for eligibility and the study was explained to eligible patients. Voluntary informed consent was obtained from interested patients. For the patients who completed the study via telehealth, there was an option to either complete an e-consent form through our project's REDCap database via e-mail or provide verbal consent after being read the consent form. The Albert Einstein College of Medicine's Institutional Review Board (IRB) approved the study, and the study follows the Strengthening the Reporting of Observational Studies in Epidemiology reporting guidelines.

All participants were asked to complete the eHEALS questionnaire. Patients who presented in-person were also asked to complete the eHeLiOS-SB objective test of EHL skills. Patients who received telemedicine visits were not asked to complete eHeLiOS-SB as in-person interaction was required due to the visual nature of the eHeLiOS-SB questions. Those who completed eHEALS through telehealth appointments were read the survey questions by the study coordinator, who then also recorded the responses. Participants who presented in-person completed all questionnaires on paper and provided consent for study entry, and those who completed the study during a telehealth visit were consented verbally. Study coordinators administered the surveys to the participants; however, responses to each questionnaire item needed to be given by the patient themselves, whether verbally or in writing. Participants were not given a time limit for filling out both survey tools.

Questionnaires

The primary measure for subjective, self-reported EHL in this study was the participants' score on the eHEALS questionnaire,[19] which is widely used and has been previously validated as a reliable measure of EHL.[12] [17] [20] [21] It contains 10 questions assessing how adults use the internet and other electronic means to access health information and their comfort level. The first two questions are descriptive and ask patients about their perceived usefulness and importance of using the internet to make health decisions. The remaining 8 questions quantify a respondent's EHL.[22] Each question is scored using a 1 to 5-point Likert scale ranging from “strongly disagree” (1 point) to “strongly agree” (5 points). When added together, these questions provide a total EHL score ranging from 8 to 40 (a higher score indicates better EHL).

To objectively assess whether patients can effectively navigate health information online, we developed a multiple choice study tool (eHeLiOS-SB) that presents study participants with nine common situations that may arise while using the internet to gather health information (see [Fig. 1] for an example of a question on the eHeLiOS-SB tool). For each scenario, a picture of a website is shown along with a question stem describing a scenario in which a patient might seek health information online. Four potential answers are marked on the page and the participant chooses the most appropriate answer. The last question (#10) is subjective and asks participants to indicate their overall trust for and general attitudes toward digital health technologies. By combining data from the eHEALS and eHeLiOS-SB questionnaires, we collected data on patients' self-perceived levels of EHL and their objective competency with using digital health tools.

Fig. 1 An example of a question on the eHeLiOS-SB tool. eHeLiOS-SB, eHealth Literacy Objective Scale-Scenario Based.

Data Collection and Evaluation

Data were collected from patient's electronic medical records, including age, gender, zip code, ethnicity, and cancer diagnosis. Patients were then administered questions from the eHEALS tool and, if they presented in-person, our newly designed eHeLiOS-SB tool. There was no previously defined “cut-off” score on the eHEALS questionnaire, which determines a patient as “eHealth literate” or “not eHealth literate.” For the sake of analysis for this study, we defined subjective EHL as follows for the eHEALS questionnaire: a score of 32 or higher (out of a 40-point maximum score) indicated “good EHL,” a score of 24 to 31 indicated “moderate EHL,” and a score below 24 indicated “low EHL.” Similarly, using eHeLiOS-SB, “good EHL” was defined as a score from 8 to 9 (out of a maximum of 9, a score of 5–7 indicated “moderate EHL,” and a score below 5 indicated “low EHL.” These cutoff points for different EHL levels on eHeLiOS-SB were defined arbitrarily and a priori. More accurate and valid cutoff values will be defined throughout our upcoming studies.

Statistical Analysis

The average eHEALS, eHeLiOS-SB scores, and other continuous scale variables were summarized using mean and standard deviation (SD) or median and interquartile range, as appropriate. Categorical variables were presented as frequency counts and percentages. We also categorized subjective EHL as “good,” “moderate,” or “low” based on the previously defined cut points; the prevalence of “good” EHL and eHeLiOS-SB were presented as proportions along with its 95% Clopper–Pearson 95% confidence interval (CI). The study aimed to assess the agreement between the eHEALS and eHeLiOS-SB scales, which we considered as both continuous and categorical scales. Lin's[23] Concordance Correlation Coefficient (CCC) was used to examine agreement between two scales, and EHEALS was compared with both raw and scales eHeLiOS-SB scores. eHeLIOS-SB scores were scaled by multiplying 4.44 to have a comparable mean and dispersion between the eHeLiOS-SB and eHEALS scores (given that the eHEALS questionnaire has 40 questions and eHeLiOS-SB has 9). In addition, we fitted unit difference Bland–Altman plots, which measure the agreement between two quantitative measurements by constructing limits of agreement. We also assessed agreement between categorical scores using Cohen's kappa and Fleiss–Cohen (quadratic) weighted kappa statistics.

As appropriate, the comparison between participant characteristics by eHEALS and eHeLiOS-SB categorical groups was assessed using analysis of variance or Pearson chi-square statistics. We evaluated the association between participant characteristics and categorical scores using proportional odds models. The proportional odds assumption was assessed using a score test. All analyses were performed using SAS software version 9.4.

Results

Participant Characteristics

There were 97 participants in the study cohort who presented to either the MECCC Department of Radiation Oncology or the Department of Medicine clinics. Characteristics of the study cohort are summarized in [Table 1]. The average age overall was 55 years, with the average age of the radiation oncology participants being 66 and the average age of the department of medicine participants being 49. The participants were predominantly male (77%), Black (59%), and the majority of participants either had Medicaid insurance coverage (39%) or private health insurance (41%).

Table 1 Distribution of sociodemographic characteristics by cohort

Parameter	All (n = 97)	Radiation oncology cohort (n = 31)	Department of medicine cohort (n = 66)
Age, mean (SD)	55 (14)	65.9 (10)	49 (13)
Gender, n (%)
Male	75 (77)	21 (68)	54 (82)
Female	22 (23)	10 (32)	12 (18)
Race/ethnicity, n (%)[a]
Black	59 (64)	17 (63)	42 (64)
White	2 (2)	2 (7)	1 (1)
Hispanic	21 (23)	0 (0)	21 (32)
Other[b]	10 (11)	8 (30)	2 (3)
Insurance type, n (%)
Medicare	19 (20)	13 (42)	6 (9)
Medicaid	38 (39)	4 (13)	34 (52)
Private	40 (41)	14 (45)	26 (39)

Abbreviation: SD, standard deviation.

^a Information on race/ethnicity was unavailable for two male and two female participants.

^b Participants' race/ethnicity were characterized as other if they were in the radiation/oncology cohort and were not considered non-Hispanic White, or Black, or if they were in the internal medicine cohort and were not considered non-Hispanic White, Black, or Hispanic.

Agreement Between the eHEALS and eHeLiOS-SB Tools

All 97 participants complete the eHEALS survey, whereas 96 completed the eHeLiOS-SB survey (one participant from the radiation oncology cohort did not complete the eHeLiOS-SB questionnaire due to completing the study via a telehealth appointment). The estimate of the internal consistency (reliability) of the overall eHeLiOS-SB scale (objective questions: Q1–Q9) as measured by Cronbach's α is 0.73, which is within acceptable limits for all scale variables (i.e., α ≥ 0.70). The mean eHeLiOS-SB score among participants was 6.6 (SD = 1.95), and the mean eHEALS score was 28.4 (SD = 6.87), both of which indicate “moderate EHL” on their respective scales. The prevalence of “good” eHeLiOS-SB scores (defined as a score of 8 or more questions correct out of 9) was 42% (95% CI: 31–52%), whereas the prevalence of “good” eHEALS scores (defined as a score of 32 or more out of 40) was 36% (95% CI: 26–46%).

The agreement between unscaled and scaled eHeLiOS-SB versus eHEALS based on Lin's CCC was 0.01 and 0.23, respectively. The difference between the scaled eHeLiOS-SB and eHEALS scores is graphed against the average score in [Fig. 2]. A negative bias score indicates the average eHeLiOS-SB score measures 1.03 (95% CI: −20.06 to −17.99) units more than the eHEALS score. When analyzed as a categorical variable, the weighted kappa produced a similar agreement of 0.24 (95% CI: 0.06–0.41). Both the CCC and kappa measures indicate that the agreement between eHEALS and eHeLiOS-SB scores was poor and that a specific score on one tool does not indicate performance on the other. The agreement of eHeLiOS-SB and eHEALS is graphed according to the cumulative frequency of “low,” “moderate,” and “good” EHL on the two tools in [Fig. 3].

Fig. 2 Bland–Altman plot of agreement between scaled eHeLiOS-SB scores and eHEALS scores demonstrating average bias (solid black line) from zero difference (blue dotted line) and 95% limits of agreement (red dotted lines). eHEALS, eHealth Literacy Scale; eHeLiOS-SB, eHealth Literacy Objective Scale-Scenario Based.

Fig. 3 Agreement chart of assessing electronic literacy between eHeLiOS-SB and eHEALS scales. eHEALS, eHealth Literacy Scale; eHeLiOS-SB, eHealth Literacy Objective Scale-Scenario Based; EHL, electronic health literacy.

Associations Between Demographic Characteristics and EHL Scores

The distribution of demographic variables, including age, gender, race, and type of insurance, was compared against the categorical eHeLiOS-SB and eHEALS groups ([Table 2]). There was a statistically significant difference in age between the EHL score according to both eHeLiOS-SB and eHEALS, with a lower age being positively associated with “good” EHL scores (p < 0.0001 for eHeLiOS-SB and p = 0.001 for eHEALS).

Table 2 Distribution of sociodemographic characteristics by eHeLiOS-SB and eHEALS literacy categories (“good,” “moderate,” “low”)

	eHEALS				eHeLiOS-SB
Parameter	Low EHL (n = 21)	Moderate EHL (n = 41)	Good EHL (n = 35)	p [b]	Low EHL (n = 15)	Moderate EHL (n = 41)	Good EHL (n = 40)	p [b]
Age, mean (SD)	63 (12)	56 (12)	49 (16)	0.001	63 (10)	59 (12)	47 (15)	<0.001
Gender, n (%)
Female	14 (67)	29 (71)	21 (60)	0.61	11 (73)	25 (61)	27 (67)	0.65
Male	7 (33)	12 (29)	14 (40)		4 (27)	16 (39)	13 (33)
Race/ethnicity, n (%)[a]
Hispanic	5 (24)	7 (17)	9 (26)	0.71	2 (13)	9 (22)	10 (25)	0.89
Black	11 (52)	28 (68)	20 (57)		11 (74)	24 (58)	24 (60)
Other	5 (24)	6 (15)	6 (17)		2 (13)	8 (20)	6 (15)
Type of insurance, n (%)
Medicaid	8 (38)	18 (44)	12 (34)	0.68	8 (53)	14 (34)	16 (40)	0.15
Medicare	6 (29)	6 (15)	7 (20)		4 (27)	11 (27)	4 (10)
Private	7 (33)	17 (41)	16 (46)		3 (20)	16 (39)	20 (50)

Abbreviations: eHEALS, eHealth Literacy Scale; eHeLiOS-SB, eHealth Literacy Objective Scale-Scenario Based; EHL, electronic health literacy; SD, standard deviation.

^a Information on race/ethnicity was unavailable for two male and two female participants.

^b A two-tailed t-test was used to compare male versus female age, and a Pearson's chi-square test was used to compare categorical variables between the male and female participants.

The proportional odds models were fitted to investigate this association further. We modeled the odds of good literacy versus moderate or poor literacy and the odds of good and moderate literacy versus poor literacy simultaneously. The results are presented in [Table 3]. The eHeLiOS-SB model suggests that the odds decreased by 8% per 1 unit increase in age, adjusting for other covariates, both for good versus moderate or poor literacy and for good or moderate to poor literacy. The eHEALS score model suggests similar results with a statistically significant decreased odds of 6% per 1-year increase in age, adjusting for other covariates such as gender, race, and insurance status.

Table 3 Adjusted odds ratio, 95% confidence interval, and p-value from proportional odds model for eHEALS and eHeLiOS-SB

	eHEALS			eHeLiOS-SB
Parameter	Odds ratio[a]	95% CI	p	Odds ratio	95% CI	p
Age	0.92	(0.89–0.96)	<0.0001	0.94	(0.90–0.97)	0.0002
Gender
Male (ref)	1			1
Female	0.9	(0.34–2.34)	0.83	0.74	(0.30–1.81)	0.5
Race/ethnicity
Black (ref)	1			1
Hispanic	0.68	(0.22–2.13)	0.51	0.57	(0.19–1.67)	0.31
Other	1.32	(0.40–4.33)	0.65	0.7	(0.23–2.09)	0.52
Type of insurance
Private (ref)	1			1
Medicaid	0.38	(0.14–1.02)	0.05	0.64	(0.26–1.61)	0.35
Medicare	0.56	(0.17–1.91)	0.36	1.26	(0.39–4.02)	0.7

Abbreviations: CI, confidence interval; eHEALS, eHealth Literacy Scale; eHeLiOS-SB, eHealth Literacy Objective Scale-Scenario Based; EHL, electronic health literacy.

^a Cumulative logits fitted modeled “Good EHL” versus “Moderate EHL or Low EHL” and “Good EHL or Moderate EHL versus Low EHL.”

General Attitudes Toward Digital Health Technologies

Question #10 on the eHeLiOS-SB tool assessed skepticism and attitudes toward mobile health technologies used to manage the disease better. [Table 4] shows the frequency of each chosen answer. Answer choice A, which indicated an interest in using and trust in such technologies, was most frequently chosen at 72%, followed by answer choice D (which indicated unwillingness to use such technologies due to hidden fees, privacy breaches, or other concerns) at 14%, and answer choice B (which indicated an unwillingness to use such technologies due to distrust of health information from the internet) at 10%. Answer choice C (which indicated an unwillingness to use such technologies due to difficulty in navigating and understanding online health information) was chosen least frequently by participants, at 4%.

Table 4 The frequency of each answer to eHeLiOS-SB question #10 as chosen by study participants

Answer choice	Total, frequency (%)
A “Yes, I will be very interested in using such an app. I also trust the information on this app as the app was developed at a hospital. However, I might talk to my doctor about it first,” frequency (%)	69 (72)
B “No, because I do not trust anyone other than my doctor for my health, especially not the information I see on the internet,” frequency (%)	10 (10)
C “No, I find it unable or difficult to understand medical information on a computer/laptop/internet. Also, I found the ones I used before cumbersome requiring me to enter a lot of data every time,” frequency (%)	4 (4)
D “No, I am concerned about hidden fees, privacy breach, etc. with apps,” frequency (%)	13 (14)
Total, frequency (%)	96 (100)

Abbreviation: eHeLiOS-SB, eHealth Literacy Objective Scale-Scenario Based.

Discussion

This study assessed EHL among a potentially marginalized patient population using both subjective (through the eHEALS questionnaire) and objective measures (using the questionnaire designed by the study authors, called eHeLiOS-SB) of EHL. The eHEALS and eHeLiOS-SB scores did not show strong agreement, suggesting that patients' subjective assessments of their EHL may not reliably translate to actual EHL skills. We also found that younger age is associated with better EHL as measured by both eHEALS and eHeLiOS-SB and that other demographic characteristics such as race/ethnicity and type of insurance coverage were not associated with EHL scores.

The prevalence of “good” eHeLiOS-SB scores (42%) versus the prevalence of “good” eHEALS scores (36%) may suggest that patients' objective EHL is higher than their self-assessed EHL in this study. This discrepancy is consistent with the literature regarding perceived and performed eHealth literacy.[24] [25] A study by van der Vaart et al had participants complete the eHEALS questionnaire and then complete an operational internet skills test. The study found that the correlation between perceived EHL (as measured by eHEALS) and actual performance on the skills test was weak and nonsignificant.[24] The Pearson correlation coefficient was 0.18 (p = 0.09) for the association between eHEALS scores and successful completion of the performance tasks, indicating that a high perceived level of EHL as measured by eHEALS was not strongly indicative of performance on a task-based performance test. The Pearson correlation coefficients for eHEALS scores and the four types of performance tasks on the performance test (operational, formal, informational, and strategic) were all under 0.2, similarly suggesting that eHEALS scores were not predictive of scores on the performance test, an objective measure of EHL.[24] Previous findings also suggest that an individual's perception of their EHL is correlated with performed literacy, though the assessment of actual skills is not entirely accurate.[25] For example, in a study by Neter and Brainin, participants' perceived EHL was measured with eHEALS and objective EHL was assessed with completion of 15 computerized tasks. The correlation factor between overall perceived EHL and performed EHL was 0.34 (p < 0.01), indicating that there is an association between the two.[25] This moderate association between eHEALS scores and completion of performance-based tasks suggests that people are able to somewhat accurately assess their own EHL, though this assessment may not be entirely in line with scores on a performance-based test. The inconsistencies in the literature regarding the association between subjective EHL and scores on performance-based tasks are in agreement with the results of our study, which found that there was a very weak association between scores on eHEALS and performance on eHeLiOS-SB (our objective measure of EHL), given by a kappa value of 0.24 (95% CI 0.06–0.41). Together, the results of our study seem to agree with existing literature that a person's self-assessed level of EHL as measured by eHEALS is not strongly predictive of their actual, objectively measured EHL (using tools like eHeLiOS-SB or other performance-based tests). This may suggest that both subjective and objective tests should be used when evaluating EHL in a population, as only using eHEALS may not accurately predict a population's eHealth literacy levels. An accurate assessment of EHL is needed to reform telemedicine practices to ensure patients can competently use suggested and required online health resources.

Our study found that age was correlated with EHL, as measured by both eHEALS and eHeLiOS-SB. This finding agrees with several previously published papers on the association between demographic variables such as age and eHealth literacy levels as measured by eHEALS,[26] [27] [28] [29] [30] such as one study of 262 20 to 65 year olds in Taiwan, which found that EHL levels were negatively associated with age.[26] Another study conducted in older adults by Wang et al similarly found that EHL levels as measured by eHEALS were lower on average in the older adult population than they were in university students.[30] We believe that younger people have better EHL due to exposure to the internet and various types of online portals at a younger age in more recent generations. Cognitive decline in older populations may also be a factor, but we believe that increased exposure to the internet and technology among younger patients is likely the more significant contributor.[31] However, some studies have found that higher age may be associated with higher eHEALS literacy scores,[32] suggesting that the association of age and eHealth literacy scores may be affected by other variables (such as educational resources available to older adults and the proportion of older adults with advanced degrees in the study population). For example, a study of 320 adults in South Korea found that age was positively associated with EHL. This finding may be because all participants had received a high school education, 77% had received an advanced degree, and the majority of participants used the internet on a daily basis.[32] This may explain why our study, which was conducted in a population with lower levels of education on average, found that age was inversely associated with EHL.

The last question on the eHeLiOS-SB scale asks about patients' willingness to trust information provided on a hospital-developed mobile health app. Answer choice A, which indicates a willingness to use and trust such information, was chosen 69% of the time. However, some patients were concerned about relying on health information from sources other than their physicians or had other concerns about digital health technologies in general. Existing literature suggests that older adults have mixed feelings about using technology to help manage their diseases.[33] [34] [35] [36] [37] [38] [39] [40] For example, in one study of 83 older Norwegian adults (ages 79 and older) who were asked about their perspective on assistive technology, 77% felt that such technologies were helpful in everyday life, but 60% had concerns they were “too old” to use such technologies and 17% were concerned about a breach of privacy.[39] While other studies have generally suggested that older adults are open to using new health technology, especially once they have been educated on how technology can benefit their health,[33] [34] [35] [36] [37] several concerns have been raised by patients that may pose barriers to incorporating this technology. These concerns include a lack of instructions, a feeling that the technology was too complex,[33] [38] the expense of such technology,[33] and fear of privacy breach.[39] Patients also feel that using apps would not be useful to help them manage their chronic illness.[40] Our results were similar, with most patients generally trusting of mobile health technologies, but a sizeable minority of patients had reservations of various types. These results suggest that digital health technologies should continue to be offered and implemented in the care of patients of all ages, but that there should be further discussion about safe and effective use of these applications with patients. More extensive education and support may be required for patients in underserved communities such as the Bronx given their lower level of education and their more infrequent use of the internet compared with other populations.

A major strength of our study is the ability to compare patients' subjective reports of their EHL with an objective measure of their EHL skills. We believe we have demonstrated that such a study is feasible, even among a population that may have lower EHL than other groups. Further studies comparing eHEALS and eHeLiOS-SB can better characterize the relationship between subjective and objective EHL, associations between various demographic characteristics and EHL score, and patient attitudes regarding digital health technologies, in general. Further validation of eHeLiOS-SB, as a newly designed tool, is also warranted; psychometric validation is underway. Another strength is the high proportion of patients who are members of linguistic/racial/ethnic minority groups and those coming from lower socioeconomic (SES) circumstances. Our study will not only help inform telemedicine strategies but can also help improve practices and design interventions to evaluate EHL among underserved populations and overcome barriers to optimal health care delivery. Given the limited access many underserved communities have to reliable internet service, it is essential to gather data on internet use and EHL to optimally design telemedicine services moving forward. Such initiatives could include creating educational modules on using online health care portals and finding trusted health sources online for older adults to complete before their telemedicine appointment or creating a posttelehealth visit survey for patients to complete that assesses their understanding of the visit.

One limitation of this study is that patients were recruited from only one health care facility. Because of this, our cohort may not be representative of the broader population. For example, factors such as education level, access (or lack thereof) to updated technological equipment, and age of first use of technology could influence subjective and/or objective EHL. We did not collect information regarding the patients' background familiarity with digital instruments, and this is a limitation to our study that will be considered for future studies. Additionally, our study population was relatively small (97 people in total), which could lead some of our observations to be underpowered. Though the study was offered to patients through both in-person and virtual modalities, most study participants who agreed to participate were those who had in-person appointments. This may be related to low electronic literacy or convenience; we do not anticipate any sampling bias as we reached all patients. Another limitation of our study was that the proportion of patients offered participation in the study versus how many actually agreed to participate was not recorded. We recognize that our sample may be biased since participant EHL levels may have affected participation rate. Also, in regard to our study's methods, there was one participant who completed the study via a virtual appointment. We recognize that this deviation from in-person data collection may impact the results of this participant, and this will be taken into consideration for future studies. Finally, validation analyses for the eHeLiOS-SB tool are underway and will be reported separately in a future manuscript. Because of this, we were not able to discuss the validation results of eHeLiOS-SB in this manuscript. Further study with larger cohorts coming from a variety of geographical, linguistic, and SES backgrounds would allow for more robust observations and more generalizable conclusions.

Conclusion

Both subjective and objective EHL (as measured by the eHEALS and eHeLiOS-SB tools, respectively) indicate the age is a major determining factor for a patients EHL level, suggesting that older patients may require additional counseling when attempting to implement these technologies. The existing eHEALS and the newer eHeLiOS-SB tool showed a low level of concordance, potentially because they measure different aspects of digital health literacy. Most patients say they are open to and trusting of mobile health technologies, but a substantial minority report concerns about incorporating technology into their health care. Further study is needed to better define predictors of EHL and develop optimal strategies to incorporate digital health technology for care of patients from all backgrounds.

Clinical Relevance Statement

Our study will help shape clinical practices surrounding telemedicine. Our study indicates that patients, especially older patients, may require additional guidance when accessing and using telemedicine practices as EHL may decrease with age.

Multiple-Choice Questions

1. Which EHL questionnaire(s) elicited general attitudes toward electronic health resources from patients?

   • eHEALS only

   • eHeLiOS-SB only

   • Both eHEALS and eHeLiOS-SB

   • Neither eHEALS or eHeLiOS-SB

   Correct answer: The correct answer is option b. Questions 1 to 9 of eHeLiOS-SB asked participants to complete a task based on a scenario that may arise when using online health information/portals. For these questions, there was only one correct answer. However, the last question on eHeLiOS-SB asked patients about their general attitudes and thoughts regarding these online health resources, for which there was no one correct answer.

2. Which demographic factor was significantly associated with EHL level as measured by both eHEALS and eHeLiOS-SB?

   • Gender

   • Race/ethnicity

   • Insurance status

   • Age

   Correct answer: The correct answer is option D. When various demographic variables such as gender, race/ethnicity, insurance status, and age were compared against measured EHL (by both eHEALS and eHeLiOS-SB), age was the only statistically significant association for both questionnaires.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This study was approved by the Albert Einstein College of Medicine's IRB. Patients were fully informed of the purpose, potential risks, and other critical issues regarding this study. The consent form used to consent patients included all elements as required by CFR 21 Part 50.25 and the local IRB. Information about the subjects was kept confidential and managed according to the requirements of the Health Insurance Portability Accountability Act of 1996 (HIPAA).

^* These authors contributed equally to this study.

• References

• 1 SAIPE Datasets. United States Census Bureau. Accessed September 5, 2022 2021 https://www.census.gov/programs-surveys/saipe/data/datasets.All.List_1743592724.html
• 2 Belanger MJ, Hill MA, Angelidi AM, Dalamaga M, Sowers JR, Mantzoros CS. Covid-19 and disparities in nutrition and obesity. N Engl J Med 2020; 383 (11) e69
• 3 Wadhera RK, Wadhera P, Gaba P. et al. Variation in COVID-19 hospitalizations and deaths across New York City boroughs. JAMA 2020; 323 (21) 2192-2195
• 4 The New York City Internet Master Plan. Mayor's Office of the Chief Technology Officer. Accessed August 15, 2022 at: https://www1.nyc.gov/assets/cto/downloads/internet-master-plan/NYC_IMP_1.7.20_FINAL-2.pdf
• 5 Keesara S, Jonas A, Schulman K. Covid-19 and health care's digital revolution. N Engl J Med 2020; 382 (23) e82
• 6 Mann DM, Chen J, Chunara R, Testa PA, Nov O. COVID-19 transforms health care through telemedicine: evidence from the field. J Am Med Inform Assoc 2020; 27 (07) 1132-1135
• 7 Portz JD, Brungardt A, Shanbhag P. et al. Advance care planning among users of a patient portal during the COVID-19 pandemic: retrospective observational study. J Med Internet Res 2020; 22 (08) e21385
• 8 Perkins S, Cohen JM, Nelson CA, Bunick CG. Teledermatology in the era of COVID-19: experience of an academic department of dermatology. J Am Acad Dermatol 2020; 83 (01) e43-e44
• 9 Bronx County, New York. United States Census Bureau. Accessed September 16, 2022 https://www.census.gov/quickfacts/fact/table/bronxcountynewyork/PST045221
• 10 Census Bureau Releases New Educational Attainment Data. United States Census Bureau. Accessed October 27, 2022 https://www.census.gov/newsroom/press-releases/2022/educational-attainment.html#:~:text=10.9%25%20in%202011.-,Sex,women%20and%2046.9%25%20were%20men
• 11 Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res 2006; 8 (02) e9
• 12 Nguyen J, Moorhouse M, Curbow B, Christie J, Walsh-Childers K, Islam S. Construct validity of the eHealth Literacy Scale (eHEALS) among two adult populations: a Rasch analysis. JMIR Public Health Surveill 2016; 2 (01) e24
• 13 Paige SR, Miller MD, Krieger JL, Stellefson M, Cheong J. Electronic health literacy across the lifespan: measurement invariance study. J Med Internet Res 2018; 20 (07) e10434
• 14 Paige SR, Krieger JL, Stellefson M, Alber JM. eHealth literacy in chronic disease patients: an item response theory analysis of the eHealth literacy scale (eHEALS). Patient Educ Couns 2017; 100 (02) 320-326
• 15 Robinson C, Graham J. Perceived internet health literacy of HIV-positive people through the provision of a computer and Internet health education intervention. Health Info Libr J 2010; 27 (04) 295-303
• 16 Efthymiou A, Middleton N, Charalambous A, Papastavrou E. Adapting the eHealth Literacy Scale for carers of people with chronic diseases (eHeals-Carer) in a sample of Greek and Cypriot carers of people with dementia: reliability and validation study. J Med Internet Res 2019; 21 (11) e12504
• 17 Milne RA, Puts MTE, Papadakos J. et al. Predictors of high eHealth literacy in primary lung cancer survivors. J Cancer Educ 2015; 30 (04) 685-692
• 18 ECOG Performance Status Scale. ECOG-ACRIN Cancer Research Group. Accessed January 13, 2023 at: https://ecog-acrin.org/resources/ecog-performance-status/
• 19 Norman CD, Skinner HA. eHEALS: the eHealth Literacy Scale. J Med Internet Res 2006; 8 (04) e27
• 20 Heiman H, Keinki C, Huebner J. Working Group Prevention and Integrative Oncology of the German Cancer Society. EHealth literacy in patients with cancer and their usage of web-based information. J Cancer Res Clin Oncol 2018; 144 (09) 1843-1850
• 21 Richtering SS, Morris R, Soh SE. et al. Examination of an eHealth literacy scale and a health literacy scale in a population with moderate to high cardiovascular risk: Rasch analyses. PLoS One 2017; 12 (04) e0175372
• 22 Aponte J, Nokes KM. Validating an electronic health literacy scale in an older Hispanic population. J Clin Nurs 2017; 26 (17-18): 2703-2711
• 23 Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989; 45 (01) 255-268
• 24 van der Vaart R, van Deursen AJ, Drossaert CH, Taal E, van Dijk JA, van de Laar MA. Does the eHealth Literacy Scale (eHEALS) measure what it intends to measure? Validation of a Dutch version of the eHEALS in two adult populations. J Med Internet Res 2011; 13 (04) e86
• 25 Neter E, Brainin E. Perceived and Performed eHealth literacy: survey and simulated performance test. JMIR Human Factors 2017; 4 (01) e2
• 26 Guo SH, Hsing HC, Lin JL, Lee CC. Relationships between mobile eHealth literacy, diabetes self-care, and glycemic outcomes in Taiwanese patients with type 2 diabetes: cross-sectional study. JMIR Mhealth Uhealth 2021; 9 (02) e18404
• 27 Tennant B, Stellefson M, Dodd V. et al. eHealth literacy and Web 2.0 health information seeking behaviors among baby boomers and older adults. J Med Internet Res 2015; 17 (03) e70
• 28 Hoogland AI, Mansfield J, Lafranchise EA, Bulls HW, Johnstone PA, Jim HSL. eHealth literacy in older adults with cancer. J Geriatr Oncol 2020; 11 (06) 1020-1022
• 29 Shiferaw KB, Mehari EA. Internet use and eHealth literacy among health-care professionals in a resource limited setting: a cross-sectional survey. Adv Med Educ Pract 2019; 10: 563-570
• 30 Wang Y, Song Y, Zhu Y, Ji H, Wang A. Association of eHealth literacy with health promotion behaviors of community-dwelling older people: the chain mediating role of self-efficacy and self-care ability. Int J Environ Res Public Health 2022; 19 (10) 6092
• 31 Murman DL. The impact of age on cognition. Semin Hear 2015; 36 (03) 111-121
• 32 Lee J, Tak SH. Factors associated with eHealth literacy focusing on digital literacy components: a cross-sectional study of middle-aged adults in South Korea. Digit Health 2022;8:20552076221102765
• 33 Vaportzis E, Clausen MG, Gow AJ. Older adults perceptions of technology and barriers to interacting with tablet computers: a focus group study. Front Psychol 2017; 8: 1687
• 34 Greenwald P, Stern ME, Clark S, Sharma R. Older adults and technology: in telehealth, they may not be who you think they are. Int J Emerg Med 2018; 11 (01) 2
• 35 Campelo AM, Katz L. Older adults' perceptions of the usefulness of technologies for engaging in physical activity: using focus groups to explore physical literacy. Int J Environ Res Public Health 2020; 17 (04) 1144
• 36 Kim S, Choudhury A. Comparison of older and younger adults' attitudes toward the adoption and use of activity trackers. JMIR Mhealth Uhealth 2020; 8 (10) e18312
• 37 Puri A, Kim B, Nguyen O, Stolee P, Tung J, Lee J. User acceptance of wrist-worn activity trackers among community-dwelling older adults: mixed method study. JMIR Mhealth Uhealth 2017; 5 (11) e173
• 38 Snyder CF, Blackford AL, Wolff AC, Carducci MA, Herman JM, Wu AW. PatientViewpoint Scientific Advisory Board. Feasibility and value of PatientViewpoint: a web system for patient-reported outcomes assessment in clinical practice. Psychooncology 2013; 22 (04) 895-901
• 39 Halvorsrud L, Holthe T, Karterud D, Thorstensen E, Lund A. Perspectives on assistive technology among older Norwegian adults receiving community health services. Disabil Rehabil Assist Technol 2021; 0 (00) 1-8
• 40 Trawley S, Baptista S, Browne JL, Pouwer F, Speight J. The use of mobile applications among adults with type 1 and type 2 diabetes: results from the second MILES-Australia (MILES-2) study. Diabetes Technol Ther 2017; 19 (12) 730-738

Publication History

Received: 02 December 2022

Accepted: 21 February 2023

Accepted Manuscript online:
23 February 2023

Article published online:
17 May 2023