Consumer Health Informatics Adoption among Underserved Populations: Thinking beyond the Digital Divide

Summary

Objectives: Underserved populations can benefit from consumer health informatics (CHI) that promotes self-management at a lower cost. However, prior literature suggested that the digital divide and low motivation constituted barriers to CHI adoption. Despite increased Internet use, underserved populations continue to show slow CHI uptake. The aim of the paper is to revisit barriers and facilitators that may impact CHI adoption among underserved populations.

Methods: We surveyed the past five years of literature. We searched PubMed for articles published between 2012 and 2017 that describe empirical evaluations involving CHI use by underserved populations. We abstracted and summarized data about facilitators and barriers impacting CHI adoption.

Results: From 645 search results, after abstract and full-text screening, 13 publications met the inclusion criteria of identifying barriers to and facilitators of underserved populations' CHI adoption. Contrary to earlier literature, the studies suggested that the motivation to improve health literacy and adopt technology was high among studied populations. Beyond the digital divide, barriers included: low health and computer literacy, challenges in accepting the presented information, poor usability, and unclear content. Factors associated with increased use were: user needs for information, user-access mediated by a proxy person, and early user engagement in system design.

Conclusions: While the digital divide remains a barrier, newer studies show that high motivation for CHI use exists. However, simply gaining access to technology is not sufficient to improve adoption unless CHI technology is tailored to address user needs. Future interventions should consider building larger empirical evidence on identifying CHI barriers and facilitators.

 

MissingLinkText

Keywords

Medical informatics applications - consumer health information - ethnic groups - socioeconomic factors - minority groups - health disparities

Introduction

The digital divide describes “the gap between those who have and do not have access to computers and the Internet”[1]. Systematic reviews of consumer health informatics (CHI) adoption have placed the digital divide as one of the core barriers to securing equal participation in technology-based health management solutions, specifically among underserved populations—groups that disproportionately experience difficulty accessing care due to social, economic, geographic, racial, or ethnic status[2] [3] [4] [5].

Internet adoption around the globe, however, has rapidly increased over the past 10 years, including in developing countries[6], groups with low socioeconomic status, and racial and ethnic minority groups in developed countries[7] [8]. Between 2010 and 2012, Hispanic and African-American populations in the U.S. represented the populations with the highest smartphone ownership rates, at 61% and 59%, respectively[9]. Between 2013 and 2015, adults in developing countries who use the Internet at least occasionally or report owning a smartphone increased significantly, from 6% in India up to 31% in Turkey[6].

Tasks previously only possible on desktop computers, such as Internet and e-Health access, are now widely available through mobile phones and tablets. The global trend of increased Internet access and mobile phone ownership offers low-cost, scalable opportunities for CHI to empower individuals. A randomized controlled trial of MyHealthKeeper, a personal health record system from South Korea that allowed for sharing between patients and healthcare providers, resulted in significant improvements in weight-loss and triglyceride levels among users[10]. The Finnish National Archive of Health Information (KanTa), the national health data repository, was developed with the goal for citizens to access their own health information electronically[11]. In Sub-Saharan Africa, mobile and Internet technology penetration has resulted in increased female economic participation[12]. Mobile health applications in developing countries have shown effectiveness in many areas of medical care: improvement in patient follow-up[13], uptake of counseling and testing[14], and improved patient adherence and response to treatment[10] [15].

Given the increased penetration of Internet and mobile technologies across the globe, continuing to assume that basic technology access is the main contributor to health disparity vis-à-vis the digital divide may be in sufficient. This lack of context may potentially increase health disparities over time if left unchecked. Improving CHI adoption requires users to remain highly committed and motivated[2] [16] [17] [18]. Sustained engagement, necessary for the adoption of any technology, has proved challenging in other scenarios[19] [20] [21].

In this review, we survey articles that were published between 2012 and 2017 about CHI use. Given the changing atmosphere of underserved populations’ technology use, we concentrate on identifying barriers and facilitators. We suggest facilitators for developing future CHI systems that are sensitive to diverse user populations.

 

MissingLinkText

Methods

Our goal is to review the past five years of literature to follow up on persistent qualitative barriers and facilitators to CHI use among underserved populations. The goal is not to systematically review all possible literature, but rather update whether more recent literature continues to view the digital divide and motivation to use as the major barriers. We aim to enrich the discussion regarding what facilitators we can employ for future work in developing and evaluating CHI and overcome barriers to this end.

Operationalizing the Terms CHI and Underserved Population

We adapted the definition of CHI from the most recently updated publication on CHI ontologies[22], which describes CHI as a technology that is: (1) consumer facing, where consumers refer to patients, caregiv-ers, or healthy individuals with prevention needs; (2) interactive for the consumer, including features such as buttons or links that enable retrieval of further information initiated by the consumer; and (3) providing tailored information, where the tailoring should happen for each consumer (e.g., providing personal health records, rather than general health information from Internet search) or the consumer group (e.g., tailored for cancer survivors group).

Defining underserved populations varies by country-specific political, cultural, and socioeconomic factors. Our review of a Cochrane study and national and international agencies publications settled around a common definition of underserved, or medically underserved, as those groups experiencing barriers to basic health needs due to social (including racial/ethnic minorities), economic, and geographic factors[23] [24] [25] [26]. In the U.S., some agencies like the National Institute for Minority Health and Health Disparities (NIMHD), and the Health Resources and Services Administration's Medically Underserved Areas (HRSA-MUA), which defines underserved as those experiencing a lack of access to basic health care, have operationalized these key characteristics linked to social, economic, and geographic vulnerabilities that are in-step with these global definitions for purposes of research and funding[27] [28] [29].

For the present review, we define under-served populations as racial/ethnic minorities in the context of country (e.g., Turkish in Germany, Hispanics and African-Americans in the U.S.), social (e.g., education, literacy, language), economic (e.g., employment, poverty, insurance), or geographic (e.g., rural) barriers.

 

MissingLinkText

Data Sources and Searches

We searched PubMed from October 2012 to October 2017 for full-text studies published in the English language regarding barriers and facilitators to CHI use in underserved populations. Staying within the scope of a survey, we restricted keywords to only Medical Subject Headings (MeSH) terms representative of “barriers and facilitators of CHI use among underserved populations” as operationalized above and we avoided custom keywords that might potentially bias the search results if not systematically chosen. We summarize the search strategy in [Table 1].

Table 1 Medical Subject Headings terms used for searching abstracts on PubMed

Consumer health informatics technology	▪ Consumer facing ▪ Interactive, such as buttons or links that enable retrieval of further information initiated by the consumer ▪ Providing tailored information, where the tailoring should happen for each consumer (e.g., providing personal health information) or the consumer group (e.g., tailored for Spanish speaking groups)
Consumer	Consumers could refer to patients, caregivers, or healthy individuals with prevention needs
Underserved population	▪ Racial/ethnic minorities in the study context of the country (e.g., Turkish in Germany, Hispanics and African-Americans in the U.S.) ▪ Social barriers (e.g., education, literacy) ▪ Economic barriers (e.g., employment, poverty, insurance) ▪ Geographic barriers (e.g., rural)
Barriers and facilitators	Keywords related to access, tailoring, or user-centered design
Constraints	▪ Full text ▪ Published in the past 5 years ▪ Written in English

 

MissingLinkText

Data Extraction and Synthesis

After a reviewer training phase to ensure inter-annotator agreement, we performed a single review for title and abstract screening. A full review of the selected text was performed by MS and JH, and data abstraction was performed by JH.

From each study selected for full-text review, we abstracted the following: population characteristics, setting, number of subjects, health conditions studied, study type, and barriers/facilitators to CHI use. To identify the paper's definition for un-derserved population we abstracted race/ ethnicity, income and insurance indicators, education level, and geography. We followed the thematic analysis commonly used in qualitative research methods[30].

We searched for themes that “emerge as being important to the description of the phenomenon”[31] such as, in our case, facilitators and barriers to CHI adoption among the target population. We identified the themes through “careful reading and re-reading of the data”[32]. We recognized patterns within the data, allowing emerging themes to become the categories for analysis. We adapted PRISMA guidelines for our protocol specification, data abstraction, and synthesis[33].

 

MissingLinkText

 

MissingLinkText

Results

We obtained 639 abstracts after removing missing abstracts from 645 search results. There were no duplicates. We removed 586 abstracts because there was no CHI involved in the study—either because the study did not examine implemented CHI (e.g., interviews and questionnaires about future CHI) or because the CHI being studied did not involve direct interaction with users (e.g., educational videos, one-way text message alert system). We then removed an additional 22 abstracts because the studies reported lacked an underserved population perspective on CHI even if the study did involve CHI.

As a result, after the abstract screening, 31 articles remained, from which we excluded 18 articles through the full-text screening (see [Figure 1]). Reasons for exclusion during full-text review included one or more of the following reasons: no data on the underserved (n=5) or on facilitators and barriers to technology adoption (n=5), the article was only about the study protocol and was lacking outcome data (n=3), the technology was not CHI (n=4), or the study did not empirically test technology on human subjects (n=2). As a result, 13 articles finally remained for full-text analysis.

Fig. 1 Selecting articles related to the use of CHI by underserved populations. Diagram adapted from PRISMA guidelines.

The CHI solutions evaluated in these 13 articles included patient education tools (n=7)[34] [35] [36] [37] [38] [39] [40], patient portals (n=4)[41] [42] [43] [44], and technology-based illness intervention (n=2)[45] [46]. Nine articles presented mixed methods[35] [36] [37] [38] [39] [42] [43] [44] [46], nine included interviews and focus groups[34] [35] [36] [37] [38] [39] [43] [44] [46], five included surveys[2] [36] [39] [42] [44], three included trials[37] [45] [46], and three included a cohort analysis[41] [42] [46]. The total sample size ranged from 21 patients to more than 200,000 patients. The studies were from two countries: U.S. (n=12) and Netherlands (n=1)[34]. Six health areas were covered: cancer (n=4)[35] [36] [38] [39], cardiometabolic risk and nutrition (n=2)[34] [40], HIV/AIDS (n=1)[44], environmental health for prenatal patients (n=1)[37], medication adherence (n=1)[46], and dementia (n=1)[45]. Two articles covered the use of general patient portals, not specific to an illness[41] [42] and two specified “chronic conditions” without further detail[42] [43]. Eligibility criteria included older adults in five studies[40] [42] [43] [45] [46]. Four articles focused only on Hispanic populations[35] [37] [44] [46], four only on African-Americans[36] [38] [39] [43], two on Hispanic and African-Americans[40] [45], and the rest on one or more ethnic minorities[34] [41] [42]. Nine focused on low-income patients[36] [37] [38] [40] [41] [42] [43] [44] [46], and 11 on low health literacy or low education populations[34] [35] [36] [37] [38] [39] [40] [42] [43] [44] [45]. [Tables 2] and [3] display a summary of the inal articles.

Table 2 Characteristics of CHI and population in the selected studies

Author, year	Type of CHI applications	Total number of participants	Reported age of participants	Indicators for underserved populations
				Ethnicities of interest	Income indicators	Reported education level	Reported insurance status
Ancker, 2017[41]	Patient portal: Medline connected links in medical records	12,877	18-24: 18.5% 25-44: 43.5% 45-64: 30% 65+: 8%	Latinos by language preference, Black, White, Other, Unknown	Patients at Federally Qualified Health Centers (FQHC)	n/a	Private, Medicaid, uninsured, Medicare
Gordon, 2016[42]	Patient portal: Kaiser North California	231,082	65-79	English speaking non-Hispanic White, Black, Hispanic, Filipino, and Chinese	30.3% had low income	[Survey] 22% Latino seniors and 4% others did not graduate from high school	Kaiser HMO
Damman, 2016[34]	Patient education tool: Web-based cardiometabolic disease risk calculator and information	23	40-66	n/a	n/a	Low health literacy
Kukafka, 2015[35]	Patient education tool: Web-based decision aid for breast cancer prevention	34	Mean age: 53.4 (SD=n/a)	Hispanic	n/a	41% had low numeracy	n/a
Owens, 2015[36]	Patient education tool: Computer-based decision aid	21	37-66	African-American	More than half had income lower than $39,999/yr	14% finished high school	20% Medicaid or no coverage or other insurance
Smith, 2015[43]	Patient portal: Registration and utilization of a patient portal	534	55-74	African-American	FQHC and ambulatory care clinic	17.4% had low health literacy, 14.8% graduated from high school or less	n/a
Rosas, 2014[37]	Patient education tool: Kiosk, interactive game for prenatal and environmental health	152	n/a	Hispanic	Low income, FQHC	Low literacy	n/a
Odlum, 2014[44]	Patient portal: Internet-based electronic personal health, management tools	42 [Survey], 15 [Focus groups]	24-63	Hispanic	80.9% earned less than $10,000/yr	83% graduated from high school or General Educational Development (GED)	Medicaid Special Needs Plan
Cogbill, 2014[38]	Patient education tool: Online colorectal cancer education website	18 [Focus groups], 60 [Trial]	45-75	African-American	[Focus groups] 16.7% earned $10,000/yr or less, [Trial] 33.3% earned $10,000/yr or less	[Focus groups] 33.3% graduated from high school or less, [Trial] 31.7% graduated from high school or less	n/a
Czaja, 2013[45]	Technology-based intervention: In-home and videophone technology, multi-component psychosocial intervention	110	Mean age: 60.9 (SD: 12.8)	Hispanic, African-American	n/a	40% of the intervention group and 36.5% of the control group had less than high school education	n/a
Bass, 2013[39]	Patient education tool: Low-literacy computer touch-screen colonoscopy decision aid	102	50-74	African-American	n/a	Low literacy; less than 6th grade REALM score[87]	n/a
Lapane, 2012[46]	Technology-based intervention: Tailored DVDs on medication adherence	326 [Telephone survey], 106 [First focus group], 16 [Second focus group]	Eligibility: At least 65 years old; n/a for actual participants	Hispanic	Low-income	n/a	n/a
Neuen-schwander, 2012[40]	Patient education tool: Web-based nutrition education program	123	18-30: 48% 31-50: 39.8% 51-70: 10.6% 71+: 1.6%	Hispanic, African-American	SNAP-Ed eligible	43% graduated from high school or less	n/a

Table 3 Barriers, facilitators, and study context of the reviewed articles

Author, year	Health condition of interest	Study or instrument type	Barriers	Facilitators
Ancker, 2017[41]	n/a	Cohort study	Low health and computer literacy	Needs for more information
Gordon, 2016[42]	Chronic disease	Cohort study, survey	Low health and computer literacy	Help from proxy users
Damman, 2016[34]	Type 2 diabetes, cardiovascular disease, chronic kidney disease	Interviews	Low health and computer literacy, challenges accepting the presented information
Kukafka, 2015[35]	Breast cancer	Focus groups, survey	Challenges accepting the presented information
Owens, 2015[36]	Prostate cancer	Focus groups, survey		Early user engagement In design
Smith, 2015[43]	One or more chronic conditions	Interviews	Low health and computer literacy, challenges accepting the presented information, poor usability and clarity	Early user engagement in design
Rosas, 2014[37]	Pregnancy, environmental health	Pre/post test, open-ended interviews		Early user engagement in design
Odlum, 2014[44]	HIV/AIDS	Focus groups, survey	Challenges accepting the presented information, poor usability and clarity	Needs for more information
Cogbill, 2014[38]	Colorectal cancer	Focus groups, a 3-week feasibility trial	Low health and computer literacy, challenges accepting the presented information, poor usability and clarity	Early user engagement in design, needs for more information
Czaja, 2013[45]	Caregivers of patients with dementia	A 5-month randomized clinical trial		Early user engagement in design, needs for more information, proxy users’ help
Bass, 2013[39]	Colorectal cancer	Focus groups, survey, segmentation analysis		Early user engagement in design
Lapane, 2012[46]	Medication adherence	Cohort study, focus groups, survey	Challenges accepting the presented information, poor usability and clarity	Early user engagement in design
Neuenschwander, 2012[40]	Nutrition	1-month randomized, block equivalence trial	Low health and computer literacy

Barriers

We found three main barriers to CHI adoption among underserved populations: (1) low health literacy[34] [40] [41] [42] [43] and lack of experience with information technology use[38] [42] [43]; (2) challenges in accepting the presented information[34] [35] [38] [43] [44] [46]; and (3) poor usability and clarity of content[38] [43] [44] [46].

• Low health literacy and lack of experience with information technology use

Many study participants did not have experience in using the Internet from school or work, lacked cellphones with Internet access, and had little contact with Internet technology[38] [42] [43]. This inexperience hindered their ability to adopt and use CHI solutions without appropriate training. In a trial involving nutrition education websites in a low-income community in the U.S. Midwest[40], understanding medical language, or health literacy, was a barrier to using the tool. Conversely, an in-person meeting was perceived as being more useful than the CHI intervention. Furthermore, participants did not think they could use a CHI application[38] [42]. This result was supported by a patient portal use study among older adults in Northern California[42] and a feasibility trial of an online colorectal cancer education program with African-American older adults[38]. Focus groups conducted in the latter study showed that texting might not be feasible for the population because the participants either did not own cell phones or, when they owned one, did not know how to use it or feared that texting would constitute a financial burden[38].

• Challenges in accepting the presented information

In some studies, even when participants had access to technology, some of them did not find the presented information useful[38] [44]. In another study[34], the presented information contradicted what participants believed about their own health, or they misinterpreted the materials. For instance, in a Dutch study involving low health literacy individuals, participants were provided their cardiovascular risk using technology-based educational materials[34]. The participants either did not believe or misconstrued their risk based on how the information was visualized. Similarly, a study displayed breast cancer risk using a web-based decision aid tool where the majority of participants were Hispanic women with low numeracy. The participants, who felt uncertain about the models presented to them, attributed the reason of their distrust with their healthcare providers to their past interaction with the providers[35]. Odlum et al.[44] studied the use of Internet-based electronic personal health management tools among a mostly minority, low-income HIV/AIDS urban clinic population. The participants preferred to enter their own health history rather than accepting the data generated by the clinic, which they found confusing.

• Poor usability and clarity of content

Lastly, usability problems and a lack of message clarity hindered CHI adoption. For instance, losing an access code after registration deterred older adults from using a patient portal[43]. Confusing user interfaces made it difficult to use patient portals or to benefit from patient education materials[38] [43] [44]. Tailored educational materials (DVD) for medication adherence aimed at older adults were mainly critiqued for their background color rather than their content[46]. In a study of a colorectal cancer screening education tool with African-American men, the participants felt that the messages were vague and should be further tailored—otherwise they would not motivate behavior change[38].

 

MissingLinkText

Facilitators

We found three main CHI facilitators: (1) early user engagement through iterative user-centered design[36] [37] [38] [39] [43] [45] [46]; (2) engaging users early in the design development process and identifying their health information needs[38] [41] [44] [45]; and (3) proxies, such as caregivers or family members, who are more familiar with technology, and use CHI on behalf of the users[42] [45].

• Early user engagement through iterative user-centered design

Participants were more willing to use CHI when the system was usable, engaging, trusted, and tailored[36] [37] [38] [39] [43] [45] [46]. To meet these requirements, one aspect frequently discussed was that CHI should allow for customized communication modes. In a study evaluating an environmental health education intervention for pregnant Hispanic women via a kiosk, researchers communicated information through both audio and text on the screen. Some participants preferred voice to text whereas others preferred reading the information on-screen[37]. Conversely, participants in a colorectal cancer screening study that used text messages versus emails to assess an educational tool had contrasting preferences for receiving reminders and learning materials[38]. Factors influencing their decisions included perceived cost of texting, ease of use, annoyance, and likelihood to grab attention. Bass et al. developed a colonoscopy decision aid for African-American men[39]; survey and focus group results showed that photographs were preferred over graphics in depicting educational materials. As a result, the aid included photographs coupled with testament videos from the actual clinic patients. While web-based nutrition education was as effective as in-person counseling for low-income participants in Neuenschwander et al.'s study[40], some topics (e.g., nutrition facts labeling) benefited from a combination of web-based and in-person approaches.

Three studies used varied methodological contexts and study scales (sample size ranging from 21 to 534)[36] [43] [46]. These studies emphasized the importance of early engagement of end-users into the design process through user-testing and improvement of functionality[36] [43]. This process helped add tailored information that met user needs. For instance, a mixed methods study presented the development of a touch-screen decision aid for low health literate African-Americans with colorectal cancer[39]. The study revealed that psychosocial issues related to the colonoscopy rather than medical information on colorectal cancer were the more critical factors in decision-making. In the health-education study with prenatal Hispanic women, adding games helped children, partners, and all family members engage in learning about environmental exposures using a kiosk[37]. Further examples of tailoring included adding actors for a video intervention, who were relatable to the user population[46].

• Intrinsic needs for more information

Consumer health informatics use was facilitated when participants had an intrinsic need for more information[44]. The works by Ancker et al.[41] and Odlum et al.[44] demonstrated that participants found that practical tips for provider engagement and health management were most useful. In a study with Medicaid users, participants felt that information on facilitating provider visits was useful as a personal health management tool[44]. African-American participants from a colorectal cancer screening tool study expressed that they wanted tips on free or low-cost screening[38]. A videophone-based intervention for dementia patients and their caregivers showed that having access to a support group was helpful and that some participants wanted more information on accessing support groups as part of these interventions[45].

• Proxy users’ help

For those with low computer literacy, having a delegated person who could help use the CHI had an impact on CHI adoption[42] [45]. As evidenced by the videophone study involving dementia patients and their caregivers, caregivers felt motivated to use the technology when it allowed them to better understand their patients’ illness. CHI could contribute to an increase in caregivers’ abilities to help patients take care of the illness[45].

 

MissingLinkText

 

MissingLinkText

Discussion

The barriers and facilitators discussed in the present review—low health literacy, tailoring, and the digital divide —have all been considered at length in prior literature[3] [46] [47] [48] [49]. Studies published in the past 5 years shows a lag in CHI adoption among the underserved when compared to the general public. Additionally, our results further show that a digital divide persists[50] [51] [52] [53]. At the same time, increased mobile technology adoption by underserved populations has slowly changed the state and nature of the digital divide[9] [54]. Furthermore, newer studies suggest an increased willingness to engage with CHI tools among underserved populations[41] [55]. Our findings lead us to discuss how the digital divide, literacy, and user-centered design of CHI should be approached.

Re-thinking the Digital Divide, Motivation, and Perceived Usefulness

A 2008 systematic review report to the Agency for Healthcare Research and Quality (AHRQ)[3] concluded that users found the majority of the evaluated CHI tools to be usable. A 2010 report to the Office of the National Coordinator for Health Information Technology (ONC)[4] showed that health and technological literacy, culture and language, level of comfort in interacting with the health care system, and digital divide added to the evidence of these factors as barriers to CHI adoption. A more recent systematic review in 2011[5] further confirmed these findings, identifying perceived benefits of health information technology, and conversely highlighting a lack of trust, technical problems, limited access to computers or hardware, technology fears, and cognitive and physical disabilities as persistent barriers to CHI adoption among underserved populations. These reports emphasize a lack of user-motivation and barriers to technology access rather than the design of CHI systems as the main drivers of the low adoption rate.

However, 10 years later, the digital divide does not appear to persist due to a lack of technology adoption, especially given the increased mobile technology use and Internet adoption among underserved populations globally[9] [54] [56]. Instead, the digital divide is driven by more complex, multi-dimensional factors. Ancker et al. showed from their 2017 study on patient portals at Federally Qualified Health Centers (FQHCs)[41], that ethnic minorities, such as Hispanics and African Americans, were more likely to use hyperlinked patient education materials in patient portals than were Caucasian users. Ancker et al. posited two possible causes: first, users’ motivation to understand the medical jargon, and second, a strong association between low health literacy and the need for further explanations. In either case, the study shows intrinsic motivation to learn by those often labeled a ‘disengaged’ population[57] [58] [59]. All people, regardless of their privilege status, were highly motivated, had intrinsic needs for information[41], wanted to manage their own health information and to share it with their providers[44], wanted tips on free or low cost ways of accessing care[38], or wanted to learn more on how to use CHI[41] [44] [60].

Our review also found that motivated use was not limited to the target user of CHI. If the patients themselves could not use the CHI tool, caregivers became highly motivated, assuming CHI user roles[42] [45]. This concept of a proxy user aligns with past findings that refer to caregivers’ effective use of technology either for themselves[60] or as helpers for patients[61] [62]. A systematic review of older adults’ use of patient portals identified technical assistance and family and provider advices as the main facilitators for patient portal use[51]. In an interview study about patient portal use in safety net hospitals, caregivers expressed interest in using patient portals to interpret health information, advocate for quality care, or manage health behaviors and medical care of patients[63].

Engaged and motivated users should receive sustainable and culturally appropriate support to help improve computer and health literacy. A recent study by the University of Kansas[64] found that low-income African-Americans wanted to learn how to use computers but study participants felt that the educators were condescending and hence they lost their motivation to return to the education sessions. A similar report was published in 1991, where African-American women engaged in a literacy improvement program were discouraged from going back to the classes because “the instructors were too mean”[65]. Such breach of trust between innovation disseminators and end-users must be repaired, and assumptions around technology adoption in underserved populations should be reconsidered and addressed in design and dissemination.

Institutional response to health disparities, or variation in access, quality, and care, has been incremental and piecemeal with respect to underserved populations. For example, language services are mandated for all hospitals that receive federal support like Medicare reimbursement, which is nearly all hospitals in the U.S.; however, uptake is less than 70%[66]. Thus, CHI interventions must not only consider barriers related to technology adoption, but also culturally competent care delivery, health equity, and significant institutional barriers. Understanding the values and beliefs of underserved populations must be a priority given the growing racial, ethnic, and linguistic diversity of many countries.

Internationally, similar principles apply. For instance, McBride et al. used SMS to help with maternal health among ethnic minorities in Vietnam[67]. A U.K. based systematic review reported, when developing ethnic-specific dietary assessment tools, using customized portion sizes by sex and age, household utensil usage, and literacy levels are critical[48]. The growing focus on patient-centered care serves as an opportunity to secure institutional buy-in to tailor healthcare[68] [69] [70]. Culturally and linguistically appropriate care could aid CHI diffusion for underserved populations across the globe given universal health care, along with rapid Internet for low-resource and rural areas[66].

Although the digital divide still exists, barriers to technology access will likely diminish, but general computer literacy will likely continue to impede progress globally. Research groups such as the 2G Lab at the University of Michigan[71] are responding to emerging digital literacy gaps by redesigning and repurposing older technologies (e.g., non-smart phones). Such endeavors consider tailoring needs to the individual a technology innovation in and of itself. This focus on the individual may be the key to CHI development for underserved populations.

It is no longer sufficient to state that the core barriers to CHI adoption by under-served populations are the lack of access to technology, or the lack of motivation or perceived usefulness of CHI applications by the end users as past studies suggested[72] [73] [74]. Rather, it is currently more important to determine how CHI can be tailored to support culturally relevant, intrinsic, and personalized information needs.

 

MissingLinkText

Re-thinking CHI Usability Evaluations for Underserved Populations

Usability and design problems can discourage even highly motivated users. In some of the studies reviewed[38] [43] [44] [46], participants did not find information useful because the display was confusing or they could not relate to the content or the actors communicating the information. These findings contradict the general consensus of the AHRQ report[3], which found the reviewed CHI systems’ usability to be high.

This contradiction may be explained by differing variables and tasks chosen for the usability evaluation. For instance, Greenberg and Buxton described this phenomenon in their seminal article, “Usability Evaluation Considered Harmful (Some of the Time)”[75]. They discussed the importance of choosing appropriate evaluation techniques to the problem and the stage of the design cycle. Otherwise, the results can be meaningless. Depending on the user groups tested and the tasks chosen, the results may not reflect how the technology would actually evolve for its intended audience and actual use. Aspects of user-centered design other than usability, such as understanding requirements, considering cultural aspects, and developing and showing stakeholders design alternatives, should be taken into account when evaluating technology use.

Many “quick and dirty” usability evaluation solutions exist that have proven to be as equally reliable and powerful as more comprehensive measurements[76] [77] [78] [79]. However, these methods should be carefully chosen when involving populations who may have linguistic, cultural, and literacy challenges. For instance, Bangor et al. discussed how simpler usability measurements, such as the System Usability Scale Survey (SUS)[80], should be accompanied by other measurements[81]. The language used for the survey items in the SUS, because of its terseness, can cause comprehension problems for non-native English speakers[82] [83] [84]. Peres et al. warned that shorter surveys meant to be designed for non-usability specialists can in fact hinder a correct interpretation of the results if a facilitator was not present for clarifications[85]. These studies demonstrated the need for careful consideration while interpreting scores in evaluating a system.

Lessons learned for improving CHI adoption among the underserved include assuring user-centered design has been deployed before dissemination and evaluation. In their 2012 systematic review[47], Montague and Perchonok suggested providing tailored, relevant, and contextu-ally situated health technology to enable behavior change among underserved populations. Simply translating English to Spanish, for instance, has been shown to be an ineffective solution to increasing technology adoption[86]. Personalizing the tool for each individual and understanding intrinsic needs of users and utilizing proxy users, such as caregivers or younger family members, should help motivate CHI adoption by underserved populations.

 

MissingLinkText

Limitations and Future Directions

We confined our search to PubMed indexed publications and our search strategy was very specific. We adopted a definition of underserved population that, though promulgated by U.S. health agencies[27] [28] [29], is race/ethnicity neutral. However, we acknowledge that this perspective might not agree with other definitions developed outside the U.S. While we attempted to be as inclusive as possible of the international context, several studies were excluded during the abstract or full text screening process. We restricted publications to English only, which may have eliminated studies of CHI in underserved populations. We did not employ pre-specified procedures to assess the risk of bias in individual studies. Nevertheless, we have referenced selected studies within this review outside of the context of our data synthesis. Future research may consider using inclusion and exclusion criteria that are specifically designed to address non-U.S. contexts to complement what might have been lost in this review. We need more empirical research reporting facilitators and barriers that can apply to a broader international context and address the ‘research divide’ shown from the results of our screening process.

On the 31 screened articles, many did not discuss factors that impacted CHI adoption. However, studies that incorporated qualitative methods, such as conducting focus groups after a trial or coupling surveys with interviews provided insights into what might have been barriers or facilitators to CHI adoption but were not generalizable. These studies did not empirically confirm factors around barriers and facilitators for generalization, perhaps because they were derived from qualitative feedbacks from a small number of individuals. Future studies should consider testing the effectiveness of facilitators and barriers in CHI adoption in a larger, confirmatory study setting to understand scalability and generalizability issues that are predominant among under-served population groups. Furthermore, user-centered design techniques that result in reliable methods for tailoring, such as expected scenarios of use, reflections, case studies, and participatory critique should be considered in addition to usability methods.

 

MissingLinkText

 

MissingLinkText

Conclusion

The digital divide and few perceived benefits of CHI use were previously considered as the dominant barriers to CHI adoption among underserved populations. The narrowing digital divide, due to increasing technology access, will not by itself solve the problem of low adoption rates. Digital divide can come from a variety of factors, including lack of net neutrality and geographic constraints that require resolution before asserting improved technology access as a solution. Contrary to misleading assumptions that underserved populations, who may suffer from low health and computer literacy, are largely disinterested in engaging with technology, studies published in the past five years indicate high motivation to adopt technology and improve literacy. CHI development should benefit from varied user-centered design techniques that address context and individualized needs of each user. At the same time, there is still much to be learned about underserved populations’ CHI use. Future studies should develop systematic methods of evaluating effective user-centered design and adoptability of CHI use among underserved populations.

 

MissingLinkText

 

MissingLinkText

Acknowledgement

We thank the Faculty-Student Mentoring Program (F-SMP) at San Diego State University for providing opportunities t o undergraduate students to gain research experience. We also thank Shraddha S. Navandar who helped with managing the undergraduate research volunteers during the review process.

• References

• 1 van Dijk J. Digital divide research, achievements and shortcomings. Poetics 2006; Aug 1; 34 (04) 221-235
• 2 Chang BL, Bakken S, Brown SS, Houston TK, Kreps GL, Kukafka R. , et al. Bridging the digital divide: reaching vulnerable populations. J Am Med Inform Assoc 2004; Nov 11 (06) 448-457
• 3 Jimison H, Gorman P, Woods S, Nygren P, Walker M, Norris S. , et al. Barriers and drivers of health information technology use for the elderly, chronically ill, and underserved. Evid Rep Technol Assess 2008; Nov 175: 1-1422
• 4 NORC at the University of Chicago. Understanding the impact of health IT in underserved communities and those with health disparities. The United States Department of Health and Human Services; 2010 Oct. Report No.: HHSP23320095635WC.
• 5 Christopher Gibbons M. Use of health information technology among racial and ethnic under-served communities. Perspect Health Inf Manag 2011; Jan 1; 8: 1f
• 6 Poushter J. Smartphone ownership and internet usage continues to climb in emerging economies. Pew Research Center [Internet]. 2016;22. Available from: http://s1.pulso.cl/wp-content/uploads/2016/02/2258581.pdf
• 7 Clark SJ, Butchart A, Kennedy A, Dombkowski KJ. Parents’ experiences with and preferences for immunization reminder/recall technologies. Pediatrics 2011; Nov 128 (05) e1100-e1105
• 8 Malvey D, Slovensky DJ. mHealth: Transforming Healthcare. Springer; 2014
• 9 Lopez MH, Gonzalez-Barrera A, Patten E. Closing the digital divide: Latinos and technology adoption [Internet]. Pew Research Center; 2013. Available from: http://assets.pewresearch.org/wp-content/uploads/sites/7/2013/03/Latinos_Social_Media_and_Mobile_Tech_03-2013_final.pdf
• 10 Ryu B, Kim N, Heo E, Yoo S, Lee K, Hwang H. , et al. Impact of an Electronic Health Record-Integrated Personal Health Record on Patient Participation in Health Care: Development and Randomized Controlled Trial of MyHealthKeeper. J Med Internet Res 2017; Dec 7; 19 (12) e401
• 11 Suna T. Finnish national archive of health information (KanTa): General concepts and information model. Fujitsu Sci Tech J 2011; 47 (01) 49-57
• 12 Efobi U, Tanankem B, Asongu S. Female Economic Participation With Information and Communication Technology (ICT) Advancement: Evidence From Sub-Saharan Africa. South African Journal of Economics [Internet] 2018 Jan 3 [cited 2018 May 3]; Available from: https://papers.ssrn.com/abstract=3170739
• 13 Chang AY, Ghose S, Littman-Quinn R, Anolik RB, Kyer A, Mazhani L. , et al. Use of mobile learning by resident physicians in Botswana. Telemed J E Health 2012; Jan 18 (01) 11-13
• 14 Lester RT, Ritvo P, Mills EJ, Kariri A, Karanja S, Chung MH. , et al. Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet 2010; Nov 27; 376 (9755): 1838-1845
• 15 Adedeji AA, Sanusi B, Tella A, Akinsanya M, Ojo O, Akinwunmi MO. , et al. Exposure to anti-malarial drugs and monitoring of adverse drug reactions using toll-free mobile phone calls in private retail sector in Sagamu, Nigeria: implications for phar-macovigilance. Malar J 2011; Aug 9; 10 (01) 230
• 16 Laxman K, Krishnan SB, Dhillon JS. Barriers to adoption of consumer health informatics applications for health self management. Health Science Journal 2015; 9 (05) 1
• 17 Hung M, Conrad J, Hon SD, Cheng C, Franklin JD, Tang P. Uncovering patterns of technology use in consumer health informatics: Uncovering patterns of technology use. WIREs Comput Stat 2013; Nov 31; 5 (06) 432-447
• 18 Beard L, Schein R, Morra D, Wilson K, Keelan J. The challenges in making electronic health records accessible to patients. J Am Med Inform Assoc 2012; Jan 1; 19 (01) 116-120
• 19 Venkatesh V, Morris MG, Davis GB, Davis FD, DeLone WH, McLean ER. , et al. User acceptance of information technology: Toward a unified view. MIS Q 2003; 27 (03) 425-478
• 20 Venkatesh V, Brown SA. A longitudinal investigation of personal computers in homes: adoption determinants and emerging challenges. MIS Q 2001; 71-102
• 21 Rogers E. Diffusion of innovations. NewYork, NY: Simon and Schuster; 2010
• 22 Ramaprasad A, Syn T. An Ontology of Consumer Health Informatics. In: Contemporary Consumer Health Informatics. Springer, Cham 2016; . p. 333-46 . (Healthcare Delivery in the Information Age)
• 23 Grobler L, Marais BJ, Mabunda S. Interventions for increasing the proportion of health professionals practising in rural and other underserved areas. Cochrane Database Syst Rev 2015; Jun 30; (06) CD005314
• 24 Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care, Board on Health Sciences Policy, Institute of Medicine. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care (with CD). National Academies Press; 2009
• 25 Committee on Oral Health Access to Services, Board on Health Care Services, Board on Children, Youth and Families, Institute of Medicine, Division of Behavioral and Social Sciences and Education, National Research Council. Improving Access to Oral Health Care for Vulnerable and Underserved Populations. National Academies Press; 2012
• 26 Medically Underserved Areas and Populations (MUA/Ps) | Bureau of Health Workforce [Internet]. [cited 2018 May 7]. Available from: https://bhw.hrsa.gov/shortage-designation/muap
• 27 Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care, Board on Health Sciences Policy, Institute of Medicine. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care (with CD). National Academies Press; 2009
• 28 Committee on Oral Health Access to Services, Board on Health Care Services, Board on Children, Youth and Families, Institute of Medicine, Division of Behavioral and Social Sciences and Education, National Research Council. Improving Access to Oral Health Care for Vulnerable and Underserved Populations. National Academies Press; 2012
• 29 Medically Underserved Areas and Populations (MUA/Ps) | Bureau of Health Workforce [Internet]. [cited 2018 May 7]. Available from: https://bhw.hrsa.gov/shortage-designation/muap
• 30 Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 2009; Jun 26 (02) 91-108
• 31 Daly J, Kellehear A, Gliksman M. The public health researcher: A methodological approach. Melbourne, Australia: Oxford University Press; 1997
• 32 Rice PL, Ezzy D. Qualitative research methods: A health focus. Melbourne, Australia [Internet]. 1999; Available from: https://pdfs.semanticscholar.org/6455/3eeb64d62e2ff1bb501bcf360c1b-024fa140.pdf#page=47
• 33 Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA. , et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009; Jul 21; 339: b2700
• 34 Damman OC, Bogaerts NMM, van Dongen D, Timmermans DRM. Barriers in using cardiomet-abolic risk information among consumers with low health literacy. Br J Health Psychol 2016; Feb 21 (01) 135-156
• 35 Kukafka R, Yi H, Xiao T, Thomas P, Aguirre A, Smalletz C. , et al. Why Breast Cancer Risk by the Numbers Is Not Enough: Evaluation of a Decision Aid in Multi-Ethnic, Low-Numerate Women. J Med Internet Res 2015; Jul 14; 17 (07) e165
• 36 Owens OL, Friedman DB, Brandt HM, Bernhardt JM, Hébert JR. An Iterative Process for Developing and Evaluating a Computer-Based Prostate Cancer Decision Aid for African American Men. Health Promot Pract 2015; Sep 16 (05) 642-655
• 37 Rosas LG, Trujillo C, Camacho J, Madrigal D, Bradman A, Eskenazi B. Acceptability of health information technology aimed at environmental health education in a prenatal clinic. Patient Educ Couns 2014; Nov 97 (02) 244-247
• 38 Cogbill S, Francis B, Sanders Thompson VL. Factors affecting African American men's use of online colorectal cancer education. J Cancer Educ 2014; Mar 29 (01) 25-29
• 39 Bass SB, Gordon TF, Ruzek SB, Wolak C, Ruggieri D, Mora G. , et al. Developing a computer touchscreen interactive colorectal screening decision aid for a low-literacy African American population: lessons learned. Health Promot Pract 2013; Jul 14 (04) 589-598
• 40 Neuenschwander LM, Abbott A, Mobley AR. Comparison of a web-based vs in-person nutrition education program for low-income adults. J Acad Nutr Diet 2013; Jan 113 (01) 120-126
• 41 Ancker JS, Mauer E, Hauser D, Calman N. Expanding access to high-quality plain-language patient education information through context-specific hyperlinks. AMIA Annu Symp Proc 2016; 2016: 277-84
• 42 Gordon NP, Hornbrook MC. Differences in Access to and Preferences for Using Patient Portals and Other eHealth Technologies Based on Race, Ethnicity, and Age: A Database and Survey Study of Seniors in a Large Health Plan. J Med Internet Res 2016; Mar 4; 18 (03) e50
• 43 Smith SG, O'Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc 2015; Jul 22 (04) 888-895
• 44 Odlum M, Gordon P, Camhi E, Valdez E, Bakken S. Exploring factors related to the adoption and acceptance of an internet-based electronic personal health management tool (EPHMT) in a low income, special needs population of people living with HIV and AIDS in New York City. Stud Health Technol Inform 2014; 201: 145-52
• 45 Czaja SJ, Loewenstein D, Schulz R, Nair SN, Perdomo D. A videophone psychosocial intervention for dementia caregivers. Am J Geriatr Psychiatry 2013; Nov 21 (11) 1071-1081
• 46 Lapane KL, Goldman RE, Quilliam BJ, Hume AL, Eaton CB. Tailored DVDs: a novel strategy for educating racially and ethnically diverse older adults about their medicines. Int J Med Inform 2012; Dec 81 (12) 852-860
• 47 Montague E, Perchonok J. Health and wellness technology use by historically underserved health consumers: systematic review. J Med Internet Res 2012; May 31; 14 (03) e78
• 48 Almiron-Roig E, Aitken A, Galloway C, Ellahi B. Dietary assessment in minority ethnic groups: a systematic review of instruments for portion-size estimation in the United Kingdom. Nutr Rev 2017; Mar 75 (03) 188
• 49 Sakaguchi-Tang DK, Bosold AL, Choi YK, Turner AM. Patient Portal Use and Experience Among Older Adults: Systematic Review. JMIR Med Inform 2017; Oct 16; 5 (04) e38
• 50 Nambisan P. Patient Portal Readiness (PPR) among the Underserved: Impact of PHIM Activities, Health Information Seeking, and Patient Attitudes towards Record Keeping. In: 2015 48th Hawaifi International Conference on System Sciences; 2015; . p. 2985-91
• 51 Sakaguchi-Tang DK, Bosold AL, Choi YK, Turner AM. Patient Portal Use and Experience Among Older Adults: Systematic Review. JMIR Med Inform 2017; Oct 16; 5 (04) e38
• 52 Nambisan P. Factors that impact Patient Web Portal Readiness (PWPR) among the underserved. Int J Med Inform 2017; Jun; 102: 62-70
• 53 Ohno-Machado L. Focusing on the patient: mHealth, social media, electronic health records, and decision support systems. J Am Med Inform Assoc 2014; Jan 1; 21 (06) 953
• 54 Neogi PK, Brocca J. Broadband Adoption and Use in Canada and the US: Is the Digital Divide Closing?. SSRN Electronic Journal [Internet]. 2011 ; Available from: http://dx.doi.org/10.2139/ssrn.1983590
• 55 Krebs P, Duncan DT. Health App Use Among US Mobile Phone Owners: A National Survey. JMIR Mhealth Uhealth 2015; Nov 4; 3 (04) e101
• 56 Donner J, Escobari MX. A review of evidence on mobile use by micro and small enterprises in developing countries. J Int Dev 2010; Jul 1; 22 (05) 641-658
• 57 Durand M-A, Carpenter L, Dolan H, Bravo P, Mann M, Bunn F. , et al. Do interventions designed to support shared decision-making reduce health inequalities?. A systematic review and meta-analysis. PLoS One 2014; Apr 15; 9 (04) e94670
• 58 Boag-Munroe G, Evangelou M. From hard to reach to how to reach: A systematic review of the literature on hard-to-reach families. Research Papers in Education 2012; 27 (02) 209-39
• 59 Raja S, Hasnain M, Vadakumchery T, Hamad J, Shah R, Hoersch M. Identifying elements of patient-centered care in underserved populations: a qualitative study of patient perspectives. PLoS One 2015; May 19; 10 (05) e0126708
• 60 Becker SA, Webbe F. Use of handheld technology by older adult caregivers as part of a virtual support network. In: Pervasive Health Conference and Workshops 2006; . p. 1-10
• 61 Bevan JL, Pecchioni LL. Understanding the impact of family caregiver cancer literacy on patient health outcomes. Patient Educ Couns 2008; Jun 71 (03) 356-364
• 62 Schaepe KS. Bad news and first impressions: patient and family caregiver accounts of learning the cancer diagnosis. Soc Sci Med 2011; Sep 73 (06) 912-921
• 63 Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R. , et al. Barriers and Facilitators to Online Portal Use Among Patients and Caregivers in a Safety Net Health Care System: A Qualitative Study. J Med Internet Res 2015; Dec 3; 17 (12) e275
• 64 Seo H, Erba J, Geana M, Lumpkins C. Calling Doctor Google? Technology Adoption and Health Information Seeking among Low-income African-American Older Adults. Journal of Public Interest Communications
• 65 Fitzsimmons KA. African-American women who persist in literacy programs: An exploratory study. Urban Rev 1991; Dec 1; 23 (04) 231-50
• 66 Schiaffino MK, Nara A, Mao L. Language Services In Hospitals Vary By Ownership And Location. Health Aff 2016; Aug 1; 35 (08) 1399-403
• 67 McBride B, Nguyen LT, Wiljer D, Vu NC, Nguyen CK, O'Neil J. Development of a Maternal, Newborn and Child mHealth Intervention in Thai Nguyen Province, Vietnam: Protocol for the mMom Project. JMIR Res Protoc [Internet] 2018 Jan [cited 2018 May 4]; 7(01). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5785686/
• 68 Rathert C, Wyrwich MD, Boren SA. Patient-centered care and outcomes: a systematic review of the literature. Med Care Res Rev 2013; Aug; 70 (04) 351-379
• 69 Davis K, Schoenbaum SC, Audet A-M. A 2020 vision of patient-centered primary care. J Gen Intern Med 2005; 20 (10) 953-7
• 70 Finkelstein J, Knight A, Marinopoulos S, Gibbons MC, Berger Z, Aboumatar H. , et al. Enabling patient-centered care through health information technology. Evid Rep Technol Assess 2012; Jun; (206) 1-1531
• 71 IT leader Edward Happ to head two new information centers at UMSI [Internet]. . The University Record. [cited 2017 Nov 28]. Available from: https://record.umich.edu/articles/it-leader-edward-happ-head-two-new-information-centers-umsi
• 72 Jones LM, Veinot TC, Pressler SJ. Cell Phone Information Seeking Explains Blood Pressure in African American Women. West J Nurs Res 2018; May; 40 (05) 617-632
• 73 Stowell E, Lyson MC, Saksono H, Wurth RC, Jimison H, Pavel M. , et al. Designing and Evaluating mHealth Interventions for Vulnerable Populations. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ‘18 [Internet]. 2018. Available from: http://dx.doi.org/10.1145/3173574.3173589
• 74 Jimison HB, Gordon CM. Decision Support for Patients. . In: Health Informatics 2016; . p. 163-179
• 75 Greenberg S, Buxton B. Usability Evaluation Considered Harmful (Some of the Time). In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM; 2008; . p. 111-120 . (CHI ‘08).
• 76 Karwowski W, Soares MM, Stanton NA. Human Factors and Ergonomics in Consumer Product Design: Uses and Applications. CRC Press; 2011
• 77 Jordan PW, Thomas B, McClelland IL, Weerdmeester B. Usability Evaluation In Industry. CRC Press; 1996
• 78 Buley L. The User Experience Team of One: A Research and Design Survival Guide. Rosenfeld Media; 2013
• 79 Summers S, Watt A. Quick and dirty usability testing in the technical communication classroom. In: 2015 IEEE International Professional Communication Conference (IPCC) [Internet]. 2015 . Available from: http://dx.doi.org/10.1109/ipcc.2015.7235831
• 80 Sauro J. A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices. . CreateSpace; 2011
• 81 Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. Int J Hum Comput Interact 2008; 24 (06) 574-94
• 82 Lewis JR, Sauro J. The Factor Structure of the System Usability Scale. In: Lecture Notes in Computer Science 2009; . p. 94-103
• 83 Keh Ann Yunmi, , Keh, Yunmi A. A Comparative Study of Argumentation Structure Between Native and Non-native Speakers of English. English Teaching 2010; 65 (03) 69-95
• 84 Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; May 1; 4 (03) 114-23
• 85 Peres SC, Camille Peres S, Pham T, Phillips R. Validation of the System Usability Scale (SUS). Proc Hum Fact Ergon Soc Annu Meet 2013; 57 (01) 192-196
• 86 Gallant LM, Irizarry C, Boone GM, Ruiz-Gordon B. Spanish Content on Hospital Websites: An Analysis of US Hospitals’ in Concentrated Latino Communities. J Comput Mediat Commun 2010; 15 (04) 552-74
• 87 Murphy PW, Davis TC, Long SW, Jackson RH, Decker BC. Rapid Estimate of Adult Literacy in Medicine (REALM): A Quick Reading Test for Patients. Journal of Reading 1993; 37 (02) 124-130