A Systematic Review of Patient-Facing Visualizations of Personal Health Data

Keywords

data visualization - comprehension - patient engagement - health literacy - consumer health information

Background and Significance

Initiatives to return personal health data to patients are on the rise. Driven largely by meaningful use stage-2,[1] patient portals have allowed patients and caregivers direct electronic access to medical results that were previously only released to the patient in paper format after an onerous request process.[2] Other initiatives have pushed for increased transparency of all personal medical information, such as OpenNotes, and have been met with positive responses from patients.[3] At the same time, patients are increasingly capable of generating their own health data through mobile health technologies and patient-reported outcomes (PROs) surveys.[4] [5] Patients can immediately visualize their PROs and other patient-generated health data (PGHD) easily and quickly through the same electronic systems, including smartphone applications and wearables.[6] [7]

The potential benefits of returning health data to the patient include increased comprehension of health status, engagement in care, and adoption of positive health behaviors.[8] [9] [10] As such, it represents an opportunity for the patient to become a more active participant in their health and wellbeing. Nonetheless, there is reticence to allow patients direct access to their health data without clinician interpretation due to concerns about poor comprehension, risk perception, and possibly dangerous or unhealthy behaviors in response to the data.[7] [11] Personal health information can be complex, especially when including numerous data points and medical jargon. It may also require contextualization based on age, gender, baseline status, and other personal characteristics. Many clinicians prefer to deliver medical results and information to patients so they can provide necessary interpretation and contextualization.[12] [13] [14] However, the sheer volume of patient data capable of being generated and returned to patients in today's electronic age makes this infeasible in many circumstances.

Information visualizations are a promising solution to aid patients in interpreting and contextualizing their health information.[15] [16] [17] They represent data and convey information by leveraging the powerful ability of humans to visually perceive differences in the sizes, shapes, colors, and spatial positions of objects.[18] Information visualizations are especially appealing because they do not rely on the patient having high literacy or numeracy. For example, specific design components, such as colors, may facilitate interpretation as effectively as explanatory text.[19] Moreover, information visualizations can be delivered to patients easily through the technologies they are already widely using, such as smartphone applications and web pages.[15]

Despite this promise, those developing patient-facing visualizations must overcome several obstacles to successfully convey meaning from personal health data. Patients vary widely with regards to health literacy, cultural context, and cognitive status, all of which influence comprehension.[20] Prior reviews have investigated health-related information visualizations and evaluation methods more broadly,[21] [22] [23] [24] but none have examined visualizations and methods pertaining specifically to patients' personal health data. The cognitive processes employed by patients interpreting personal health data differ from those employed by researchers and clinicians with advanced statistical and medical knowledge.[25] [26] Therefore, a focused investigation on the unique needs and preferences of patients viewing their personal health information is warranted to develop tools that display personal health data for use in clinical practice that align with patients' cognitive processes. Those developing, implementing, and evaluating visualizations for patients require the latest evidence to ensure that patients are able to understand and, when appropriate, act upon health data in a safe and effective manner.

Objectives

The overall objective of this systematic review was to summarize the state of the science of patient-facing information visualizations of personal health data. Specifically, we aimed to investigate (1) the types and components of patient-facing visualizations, and (2) methods and findings related to the development and evaluation of patient-facing visualizations currently reported in the literature. We aimed to examine published literature across several scientific fields in which tools that display personal health data are being developed and deployed, including but not limited to informatics, medicine, nursing, computer science, and engineering. By describing and synthesizing these findings, we aimed to generate an initial set of recommendations for those seeking to develop patient-facing visualizations that promote understanding and interpretation of personal health data.

Methods

Information Sources and Search Strategy

We searched five scholarly databases (PubMed, Embase, Scopus, ACM Digital Library [Association for Computing Machinery Digital Library], and IEEE computation index [Institute of Electrical and Electronics Engineers Computational Index]) in consultation with a biomedical librarian in December 2018. These databases were selected with the goal of capturing relevant literature from a variety of fields, including medicine, nursing, biomedical informatics, computer science, and engineering. Our search strategy included the following terms: (patient OR patients OR consumer* OR user*) AND (“Computer Graphics” [Mesh] OR visualiz* OR graph*) AND (“Health Records, Personal”[Mesh] OR “Electronic Health Records”[Mesh] OR “Telemedicine”[Mesh] OR “electronic health record” OR “test results”). Search terms were also determined in consultation with a biomedical librarian and content experts and iteratively by examining keywords in retrieved articles. No filters or additional search criteria were applied. Scopus was searched for literature that was not formally published in peer-reviewed journals (gray literature)[27] using the same terms. An inspection of reference lists from retrieved articles and of the authors' personal libraries identified any relevant articles not obtained through the database search.

Eligibility Criteria and Screening

Eligibility criteria are presented in [Table 1]. We excluded visualizations that contained only population-level data, such as prevalence of certain health conditions, because our primary interest was in visual tools that return a patients' own data to them. In addition, we excluded articles that lacked figures showing the visualizations as these were necessary to evaluate and compare visualizations between articles; text-based descriptions alone do not provide adequate detail and names for certain visualizations vary between disciplines. Three reviewers (M.R.T., S.I., and A.M.) used Covidence, a Cochrane's technology platform, to select eligible articles from the pool of retrieved articles.[28] The reviewers screened titles and abstracts against the eligibility criteria. Full texts of the articles included were rescreened using the same criteria. Any discrepancies between the reviewers were discussed and resolved.

Table 1 Article eligibility criteria

Inclusion	Exclusion
• Article written in English	• Article written in non-English language without translation
• Intended audience of visualization includes patients	• Intended audience of visualization does not include patients (for example, intended for clinicians or researchers)
• Visualization displays personal health data, including laboratory values, mobility and activity data, and patient-reported outcomes	• Visualization displays population level or non-health related data
• Visualization is included as a figure in the article	• Visualization is not included as a figure in the article

Methodological Quality Assessment of Articles

We appraised risk of bias in the included articles with the Mixed Methods Appraisal Tool (MMAT).[29] Appraising risk of bias uncovers limitations of methodological quality for consideration when gathering evidence. The MMAT is specifically designed for concomitantly appraising studies with different designs, such as quantitative, qualitative, and mixed-methods researches. It produces comparable scores across study designs[30] with highly reliable interclass correlations (ICC) ranging from 0.84 to 0.94.[31] [32] [33]

The MMAT consists of two initial screening questions that identify articles for which further appraisal may not be feasible or appropriate, as follows: (1) Are there clear qualitative and quantitative research questions/objectives, or a clear mixed methods question/objective? and (2) Do the collected data address the research question/objective? Articles failing either or both screening questions do not proceed to domain-specific appraisal. Subsequent question sets are specific to the study design. Domain-specific questions number four for qualitative articles and four questions for each of the three quantitative study designs (randomized controlled, nonrandomized, or descriptive). Mixed-method articles are evaluated using both the qualitative and appropriate quantitative study questions. There are three additional questions specific to mixed-method articles. The quality appraisal score is determined by dividing n criteria met by N total criteria in each applicable domain and converted to percentages for comparison across articles. Following this protocol, three reviewers (M.R.T., A.M., and D.B.) independently appraised each study. Interrater reliability was calculated and discrepancies were discussed until resolved. Scores are reported according to a star-rating system as suggested by the authors of the MMAT[34] and in alignment with a recent review that also used this tool to appraise quality.[35]

Data Extraction and Synthesis

The data extraction methods for this systematic review followed preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines.[36] [37] We conducted several activities to extract and synthesize findings in formats that aligned with the two research objectives. To address the first objective, two authors (M.R.T. and A.M.) independently extracted information about characteristics of the visualizations included in each manuscript, including type (e.g., line graph, bar graph, etc.), aesthetic components (e.g., use of color), and informational components (e.g., contextualizing information). The two authors worked from the same definitions of visualization types during this phase, depicted in [Fig. 1], for standardization in the data extraction process. In addition, the two authors discussed visualization types or components in the included articles that were unclear until consensus was reached. To address the second objective, the two authors independently extracted relevant characteristics from each study including field of publication, research aims, study design and methods, sample characteristics, clinical focus, outcome measures, and major findings. The first author (M.R.T.) initially synthesized extracted information by comparing articles within the data extraction tables and creating visualizations. All authors participated in further synthesis and final reporting of results.

Fig. 1 Visualization types guiding data extraction.

Results

Search Results

A total of 2,362 articles were retrieved from five databases ([Fig. 2]). After 224 (10%) duplicate articles were excluded, 2,138 articles were screened based on title and abstract. During title/abstract screening, 2,032 articles were excluded. The most common reasons were articles described visualizations of nonhealth data (n = 870; 41%) and were not patient-facing (n = 834; 39%). During full-text screening of the remaining articles (n = 106), 67 articles were also excluded. The most common reason was that no visualization was discussed or included in the article (n = 47; 44%). A total of 39 articles were ultimately included.

Fig. 2 Flow diagram of article screening. ACM, Association for Computing Machinery; IEEE, Institute of Electrical and Electronics Engineers.

Results Pertaining to Visualizations

Description of Visualizations in Included Articles

[Table 2] addresses the first research objective by describing the visualizations in the included articles (n = 39). Most patient-facing visualizations focused on monitoring and management of various chronic conditions (n = 21, 54%). Of these, almost half (n = 12) were specifically for diabetes. Visualizations for wellness or prevention (n = 15, 38%) mostly displayed laboratory values that are routinely monitored for preventive purposes, such as cholesterol. In most articles, visualizations were embedded in an existing mobile or web-based application (n = 22, 56%), such as patient portals and consumer-facing applications. Half of these applications (n = 11) were integrated into clinical care. The remaining articles (n = 17, 44%) displayed visualizations in paper or web-based mockups. Overall, the data displayed in the visualizations came from patients (e.g., self-reported symptoms) and providers (e.g., laboratory values) nearly equally. Continuous data were by far the most common type of data displayed (83%), and it was most frequently presented in line graphs and number lines. Icons were the most common types of visualizations displaying categorical data, including both nominal and ordinal data.

Table 2 Characteristics of visualizations in all included studies (n = 39)

First author (year)	Tested with human patients[b]	Clinical focus	Clinical purpose	Display medium			Health data being displayed and visualization information
				Paper	Electronic (web or mobile)	Embedded in existing tool	Data	Source		Visualization type	Data type
								User	Clinician		Continuous	Nominal	Ordinal
Ankerson, (2017)[76]		Chronic condition (irritable bowel syndrome)	Monitoring and management; decision support			Constant Care application[c]	Total inflammatory burden score (TIBS) calculated using colitis/Crohn's indices and fecal calprotectin level	✓		Line graph[d]	✓
Arcia et al, (2015)[a] [19]	✓	General	Wellness/prevention	✓			Stress, exercise, sleep, blood pressure, diet	✓	✓	Icon		✓	✓
										Number line	✓
Arcia et al, (2016)[a] [58]	✓	General	Wellness/prevention	✓			Stress, exercise, sleep, blood pressure, diet	✓	✓	Icon		✓	✓
										Number line	✓
Beaudin, (2006)[77]	✓	General	Wellness/prevention	✓	✓		Weight, step count, diet, groceries, symptoms, skin changes, general behaviors (e.g., time spent watching television)	✓		Line graph[d]	✓
										Bar graph	✓
										Body map			✓
Brewer et al, (2012)	✓	General	Wellness/prevention		✓		BMI, BP, cholesterol		✓	Bar graph	✓
Britto et al, (2009)	✓	Chronic condition (pediatric cystic fibrosis, diabetes, and arthritis)	Monitoring and management			Patient portal MyCare Connection[c]	Laboratory values		✓	Bar graph	✓
Cox et al, (2015)	✓	Chronic condition (cystic fibrosis)	Increasing physical activity			Web-based application: ActivOnline	Passively collected physical activity data (steps, exercise duration)	✓		Line graph[d]	✓
Desai, (2018)[78]	✓	Chronic condition (type-2 diabetes)	Monitoring and management			Mobile application	Current and forecasted blood glucose levels	✓		Icon	✓
										Number line	✓
										Line graph	✓
Elder and Barney, (2012)	✓	General	Wellness/prevention	✓			Cholesterol		✓	Number line	✓
										Line graph	✓
										Radar graph	✓
Farmer, (2005)[79]		Chronic condition (type-1 diabetes)	Monitoring and management		✓		Blood glucose readings; blood glucose data at set time points (pre-meal and bedtime), mean blood glucose, insulin dosage	✓		Bar graph[d]	✓
Filipi, (2013)[80]		Chronic condition (migraine)	Symptom monitoring and management		✓		Headache severity	✓		Bar graph[d]	✓
Foraker, (2014)[81]		Cardiovascular disease in older women	Wellness/prevention			Patient portal (Epic MyChart)[c]	American Heart Association's Life's Simple 7: smoking status, body mass index, blood pressure, cholesterol, fasting glucose, physical activity, and diet.		✓	Icon	✓
Fraccaro et al, (2018)[63]	✓	Chronic condition (chronic kidney disease)	Monitoring and management			Web-based application: PatientView[c]	Laboratory values		✓	Number line	✓
										Line graph[d]	✓
Frost and Massagli, (2008)[52]	✓	Chronic condition (amyotrophic lateral sclerosis/ALS)	Shared decision-making			Web-based application: PatientsLikeMe	Progression rates, prescriptions, symptoms	✓		Line graph[d]	✓
										Bar graph	✓
Harris et al, (2010)[50]	✓	Chronic condition (diabetes)	Monitoring and management			Mobile application HealthReach Mobile[c]	Blood glucose levels	✓		Line graph[d]	✓
Hohenstein et al, (2018)[62]	✓	General	Wellness/prevention			Mobile application: Nutriphone	Laboratory values		✓	Visual paragraph	✓		✓
										Number line	✓
Hosseini, (2017)[82]		Chronic condition (pediatric asthma)	Monitoring and management		✓		Risk Level, Peak Expiratory Flow, and Forced Expiratory Volume	✓		Icon			✓
										Line graph[d]	✓
Ilic, (2009)[83]		Cognitive status	Wellness/prevention		✓		Results of cognitive testing such as attention and working memory	✓		Bar graph	✓
										Line graph[d]	✓
										Icon	✓
Kallen, (2012)[84]	✓	Hospice care	Symptom monitoring and management; patient-provider communication		✓		Symptom trends, pain assessment, laboratory values, vital signs	✓	✓	Bar graph			✓
										Body map		✓
										Line graph[d]	✓
Kopanitsa, (2015)[a]	✓	General	Wellness/prevention			Patient portal[c]	Laboratory values		✓	Line graph[d]	✓
Kopanitsa, (2016)[a] [85]	✓	Older adult health	Wellness/prevention; patient-provider communication			Patient portal[c]	Laboratory values		✓	Line graph[d]	✓
Lee, (2005)[86]		Chronic condition (asthma)	Monitoring and management		✓		Medication usage and symptoms	✓		Line graph[d]	✓
Lowe-Strong, (2005)[87]		Chronic condition (multiple sclerosis)	Symptom monitoring and management		✓		Pain/incontinence/weakness severity	✓		Bar graph[d]	✓
										Body map		✓
Martinez et al, (2018)[57]	✓	Chronic condition (type-2 diabetes)	Monitoring and management			Patient portal	Hemoglobin A1c, blood pressure, cholesterol, microalbumin, flu vaccine status		✓	Icon	✓
										Number line	✓
Mena, (2013)[88]	✓	Cardiovascular disease	Wellness/prevention			Mobile application: ARV mobile[c]	Ambulatory blood pressure (systolic and diastolic) and heart rate	✓		Line graph[d]	✓
Morrow et al, (2017)[48]	✓	Older adult health	Wellness/prevention			Patient portal[c]	Cholesterol		✓	Number line	✓
Robu, (2016)[89]		Pregnancy	Monitoring and management			Mobile application: ObGynCare [c]	Blood pressure, heart rate, weight, blood sugar	✓		Line graph[d]	✓
Rudin, (2017)[90]	✓	Chronic condition (asthma)	Monitoring and management			Patient portal[c]	Asthma symptoms	✓		Line graph[d]	✓
Schenk, (2011)[91]		Chronic condition (diabetes)	Monitoring and management	✓			Blood glucose levels, medication adherence	✓		Line graph[d]	✓
Scherer et al, (2018)[a] [45]	✓	Chronic condition (type-2 diabetes)	Monitoring and management			Patient portal	Hemoglobin A1c and laboratory values		✓	Number line	✓
Skrovseth, (2012)[92]		Chronic condition (diabetes)	Monitoring and management			Mobile application FewTouch	Glucose measurements	✓		Line graph[d]	✓
										Scatterplot			✓
Smith et al, (2006)[53]	✓	Chronic condition (type 1 diabetes)	Monitoring and management; patient-provider communication		✓		Blood glucose, health behaviors	✓		Scatterplot[d]	✓
Snyder, (2018)[93]	✓	Cancer	Monitoring and management		✓		Physical and emotional function and symptoms	✓		Line graph[d]			✓
Solomon et al, (2016)[61]	✓	Chronic condition (diabetes)	Monitoring and management			Patient portal	Hemoglobin A1c and laboratory values		✓	Number line	✓
Tao et al, (2018)[60]	✓	General	Wellness/prevention		✓		Blood glucose levels	✓		Number line[d]	✓
Voils, (2012)[94]		Chronic condition (diabetes)	Genetic testing for diabetes self-management	✓			Calculated lifetime risk of diabetes		✓	Number line			✓
Watson et al, (2009)[64]	✓	Chronic condition (diabetes)	Monitoring and management		✓		Blood glucose levels	✓		Line graph[d]	✓
Zikmund-Fisher et al, (2017)[a] [47]	✓	General	Wellness/prevention			Patient portal	Laboratory values		✓	Number line	✓
Zikmund-Fisher et al, (2018)[a] [46]	✓	General	Wellness/prevention			Patient portal	Laboratory values		✓	Number line	✓

Abbreviations: ALS, amyotrophic lateral sclerosis; ARV, average real variability; BMI, body mass index; BP, blood pressure.

^a The following articles originate from the same parent study: Arcia et al 2015 and 2016; Kopanitsa 2015 and 2016; Zikmund-Fisher et al 2017 and 2018.

^b Characteristics and quality appraisal for the 27 studies that do contain human patients testing are reported in [Table 3].

^c Integrated into clinical care.

^d Longitudinal data display.

[Fig. 3] displays the prevalence and relationship between visualization types and components. The most common visualization types in these articles were line graphs (35%), followed by number lines (25%), bar graphs (16%), and icons (12%). Line graphs were the most common types of visualizations employed when longitudinal data were displayed. Other visualizations were used in three or fewer articles, and included body maps, scatterplots, radar graphs, and visual paragraphs. The most common visualization component overall was numbers, included in 88% of visualizations, and in all types of visualizations we reviewed except body maps. Color was also frequently included (77%) and was found in all types of visualizations we reviewed except visual paragraphs and radar graphs. The “traffic light” color scheme (red–yellow–green) was most frequently employed among the visualizations that included color. Data labels (which specify units and other details about the data within the visualization), longitudinal data displays, contextual information, such as descriptive text (“your blood pressure improved”) and references ranges were included in nearly half of visualizations (40–47%). Legends for colors or symbols and advanced data analytics were included in relatively few visualizations. The visualizations that did feature data analytics included algorithms that calculated risk scores and identified significant trends or patterns in the data. All other visualizations displayed raw scores or values, or simple descriptive statistics, such as counts and means.

Fig. 3 Prevalence and relationships between types and components of visualizations in included studies (n = 39). Visualization components are not mutually exclusive.

Frequencies of Patient-Facing Visualizations in Published Literature

[Fig. 4] maps characteristics of the visualizations in the articles included in this review during the publication timeframe, from 2005 to 2018 ([Fig. 4]). While chronic conditions remained a major clinical focus, visualizations pertaining to general wellness, and prevention (e.g., exercise data) became more widely used in the articles we examined. Informatics journals were the initial fields publishing this work, and for the past few years have been leading the number of articles in this area once again. Visualizations embedded in existing tools (e.g., mobile applications and patient portals) increased in the included articles, as did visualizations displaying laboratory values. Publications reporting visualizations of PGHD and PROs have remained common. Overall, line graphs were the most common of all visualizations throughout the examined publication timeframe. Number lines and icons increased in the past few years, with six articles publishing number line visualizations in 2018. Bar graphs and other visualization types became less common in the articles we examined since 2013.

Fig. 4 Frequencies of visualization characteristics in included studies over time (n = 39). PGHD, patient-generated health data; PRO, patient-reported outcome.

Results Pertaining to Human Patients Research on Visualizations

The characteristics and quality appraisal results of the 27 articles of the 39 articles that reported on human patients, research are reported in [Table 3], addressing the second research objective. The articles that did not include human patients research (n = 12) reported instead on the development of a patient-focused technical system and technical feasibility.

Table 3 Characteristics and quality appraisal of studies reporting human patients research (n = 27 out of 39)

First author (Year), MMAT score	Field of journal	Purpose	Sample	Mean age in years (SD/range)	Gender (% female)	Race/ethnicity (% White non-Hispanic)	Study design	Methodologies	Outcome measures	Study findings
Arcia et al, 2015a	Informatics	To provide detailed descriptions of an iterative methodological approach to the development of visualizations and the resulting types of visualizations	The 14 academic health care researchers and students from nursing, biomedical informatics, and public health				Qualitative	Iterative, consensus-based design process in working group format	N/A	N/A—article was methods-oriented and did not report results.
Arcia et al, 2016b	Informatics	To develop tailored infographics that support comprehension of health information, engage the viewer, and may have the potential to motivate health-promoting behaviors	The 102 English and Spanish speaking adults	53.2 (16.5)	85	5	Qualitative	User-centered design sessions: focus groups	N/A	Visualizations that contained more detail, context, and used familiar colors and symbols were most understood. Some participants unexpectedly interpreted metaphorical icons in a rigid literal fashion (e.g., bowls of fruit to represent fruit servings), resulting in lower comprehension.
Beaudin, 2006b	Informatics	To elicit specific feedback from health professionals and patients about how they might use longitudinal health monitoring data for proactive health and well-being	Eight health care providers and 26 patients				Qualitative	User-centered design sessions: semistructured interviews, sorting activities	N/A	Patients' primary motivations for visualizing health-related data were to solve a specific health-related problem. They were willing to do so even if it invited negative self-evaluation.
Brewer et al, 2012a	Informatics	To compare the relative usability of tables and horizontal bar graphs for consumers viewing medical information online	The 106 community-dwelling adults and administrative staff members from academic medical center	46 (30–83)	84	82	Quantitative	Usability testing of different visualization formats; varied formats and normality between and within patients	Viewing time, recall, understanding, perceived ease of use	Participants required less viewing time when using bar graphs and preferred them to tables, but performed equally well in terms of recall accuracy and understanding on both formats. Bar graphs were most preferred when viewing borderline test results.
Britto et al, 2009b	Informatics	To evaluate the usability of portals for parents of children with cystic fibrosis, diabetes or arthritis	The 16 parents of pediatric patients at academic medical center with specific medical conditions	39 (5.4)	81	94	Mixed-methods	Scenario-based testing with think-aloud protocols	Time to complete tasks, successful task completion, satisfaction	Participants found graphing laboratory results with the interface and interpreting the data challenging. Mean task completion time for graphing laboratory results was 431 (±286) seconds.
Cox et al, 2015b	Medicine/Healthcare	To assess the feasibility and acceptability of an Internet-based program specifically designed to encourage participation in physical activity by adults with CF	The 10 adults with confirmed diagnosis CF	30 (8)	60		Mixed-methods	Feasibility and acceptability testing: semistructured interviews, survey collection, evaluation of usage logs	System usability and perceived benefit (acceptability); frequency of Web site access and number of physical activity sessions recorded (feasibility)	Participants described a positive reaction to receiving graphical representation of their activity participation but would have preferred a mobile interface, such as an application.
Desai, 2018b	Computer science/engineering	To identify the key visual elements that help individuals internalize the predicted impact of their meals and inform their nutritional decision-making	The 13 adults with type-2 diabetes	51 (20–61)	100		Qualitative	User-centered design sessions: focus groups	N/A	Effective visualizations utilized simple and explicit, yet information-rich design. Common metaphors alongside words, numbers, and colors, conveyed a sense of authority and encourage action and learning.
Elder and Barney, 2012b	Medicine/Healthcare	To better understand patients' experiences with, and preferences for, results notification	The 12 adults with and without a chronic disease requiring regular testing				Qualitative	Semistructured interviews with think-aloud protocol	Satisfaction, self-reported comprehension	Participants preferred and understood bar graphs and results with contextual information (reference ranges, explanations) compared with simpler (text interpretation only) and more complex (line and radar graphs) visualizations.
Fraccaro et al, 2018b	Informatics	To evaluate whether visual cues improve patients' abilities to correctly interpret laboratory test results presented through patient portals	The 20 kidney transplant patients	51.8 (10.3)	20		Quantitative	Comprehension testing of different visualization formats using objective comprehensive measures and eye tracking methods; varied formats and visual cues between subjects	Accuracy of risk interpretation, visual search behavior	Misinterpretation (most often underestimation) of risk was common, particularly for medium risk clinical scenarios. Higher visual search efficiency was associated with higher risk interpretation accuracy, suggesting effective coping with information overload.
Frost and Massagli, 2008b	Informatics	To identify and analyze how users of PatientsLikeMe reference personal health information within patient-to-patient dialogues	The 95 users of PatientsLikeMe with ALS				Qualitative	Usage and content analyses of patient posts on PatientsLikeMe Web site	N/A	Visualizations prompted participants to ask more sophisticated clinical questions, identify relationships over time, and engage with other users on the Web site.
Harris et al, 2010b	Informatics	To describe end-user design evaluations for wireless glucose meter uploads, automated self-management support messages, and blood glucose feedback displays for mobile phones	The 14 adults with type-1 or -2 diabetes	18–70			Qualitative	Feasibility and acceptability testing; design sessions with think-aloud protocol	N/A	Participants found value in data visualizations as one of the most desirable system features.
Hohenstein et al, 2018a	Computer science/engineering	To explore how people interpret medical test results, examined in the context of a mobile blood testing system developed to enable self-care and health management	The 303 general population, recruited from college campus	46.0 (16.3)	51	67	Mixed-methods	Comprehension testing of different visualization formats; randomly assigned to view one of three different laboratory tests, order of formats not varied	Objective accuracy and subjective confidence in interpreting laboratory results, behavioral intention	Most participants accurately interpreted their data but nearly half were not confident in their interpretations. Demographics interacted with interface design to impact interpretation accuracy and false confidence.
Kallen, 2012b	Medicine/Healthcare	To help palliative and hospice care practices improve patient care quality and operational efficiency	The 27 health care providers, patients, and caregivers				Mixed-methods	User-centered design with iterative cycles of feedback and development	Usability and usefulness of prototype (including visualizations)	Participants reported that the prototype was usable and perceived it would facilitate communication, shared decision making, patient self-management, and identification of relationships between care events and outcomes.
Kopanitsa, 2015a	Informatics	To assess a method for standards-based medical data visualization for laboratory results using ISO 13606 Archetypes	The 30 general medicine patients at an outpatient clinic		47		Quantitative	Usability testing: survey collection, evaluation of usage logs	System functionality, accessibility, efficiency, usability	Patients and doctors reported high acceptability of the visualizations displayed in the system.
Kopanitsa, 2016b	Informatics	To implement a web portal for diabetes patients to present medical data for better doctor-patient communication and patient involvement with diagnostic and treatment processes	The 36 home health care recipients	63.7			Qualitative	Usability testing: semistructured interviews	N/A	Participants reported high usability and acceptability of the visualizations and the web portal in which they were integrated.
Martinez et al, 2018a	Informatics	To design and evaluate an innovative, patient-facing diabetes dashboard embedded in an existing patient portal and integrated into an EHR	The 14 adults with type-2 diabetes	63.4 (11.0)	57	50	Mixed-methods	Usability testing: objective task performance, usability survey, semistructured interviews	Task performance, usability	Confusion over an icon visualization resolved when “hover-over” information icon providing a nontechnical description of the measure (e.g., HbA1c) and links to literacy level-sensitive educational materials were added.
Mena, 2013a	Medicine/health care	To describe a mobile personal health monitor(PHM) application for ABP monitoring.	The 21 volunteer subjects without history of cardiovascular disease and hypertension age 51 years	58.9 (6.1)	62		Quantitative	Feasibility testing: surveys	N/A	Most participants found the visualizations and application in which they were embedded easy to use and considered the time spent learning how to use it reasonable.
Morrow et al, 2017a	Informatics	To describe a multidisciplinary approach to designing and evaluating portal-based messages that convey clinical test results so as to support patient-centered care	The 24 older adults	76	67		Mixed-methods	Comprehension testing: randomly assigned participants to one of four visualization formats; semistructured interviews	Satisfaction, gist comprehension, risk perception, behavioral intention	Audio- and video-enhanced visualizations were tested and compared, but visualizations alone were not. Participants were satisfied with and accurately remembered and responded to risk information.
Rudin, 2017a	Informatics	To efficiently identify core components for an mHealth-based asthma symptom–monitoring intervention using PROs	Nine asthma patients	21–74	67	67	Qualitative	User-centered design sessions with iterative cycles of feedback and development	N/A	One of the four core components identified was visualizations to allow patients to review their symptom history. The ability to customize viewing timeframes of historical data was added to accommodate patients' differing perspectives about the amount of data they preferred to see. Data points were color coded by severity to facilitate interpretation.
Scherer et al, 2018b	Informatics	To test the impact of including clinically appropriate goal ranges outside the standard range in the visual displays of laboratory test results	The 6,776 demographically diverse web-based panel, oversampled individuals with diabetes	49.1 (15.8)	50	68	Quantitative	Comprehension testing and preferences of different visualization formats; randomly assigned to view one of three different formats	Accuracy of risk interpretation, behavioral intention	Visualizations including a goal range produced higher levels of comprehension and decreased negative reactions to borderline laboratory results.
Smith et al, 2006a	Medicine/health care	To provide an intervention where individuals with T1 diabetes can collect, visualize, and describe behavioral and biomedical data	Seven young adults with T1 diabetes	19–21	43	100	Qualitative	Semistructured interviews	N/A	Visualizations helped participants articulate concerns about stress, peer relations, and unhealthy routines.
Snyder, 2018a	Medicine/health care	To develop stakeholder-engaged, evidence-based recommendations for PRO data display to promote understanding and use	The 28, not mutually exclusive (15 doctor or nurse clinicians, 10 patients or caregiver advocates, 12 participants with PhDs, six journal editorial board members)		79		Qualitative	Modified Delphi's methodology	N/A	The use of consistent visualizations throughout an interface, line graphs for data over time, and clearly conveying concerning values are considered optimal.
Solomon et al, 2016b	Computer science/engineering	To explore ways to design meaningful representations of test results using an iterative user-centered design process	The 18 diabetes patients and friends/relatives of diabetes patient	Median 50–59	47		Qualitative	User-centered design sessions with think-aloud protocol	N/A	Color was effective at communicating information about risk, particularly the stoplight theme (e.g., red is danger zone).
Tao et al, 2018a	Informatics	To examine the effects of graphical formats and age on consumers' comprehension and perceptions of the use of self-monitoring test results	The 72 healthy adults recruited from university and local community	Younger adult group: 22.3 (2.6); older adult group: 65.8 (3.6)	56		Quantitative	Comprehension testing of different visualization formats (format and normality varied); eye tracking methods	Task performance, comprehension, visual search behavior, preferences of the graphical formats	All formats yielded comparable task performance, but those containing text, color, and personalized information were associated with increased comprehension, risk perception, confidence in interpretation.
Watson et al, 2009a	Medicine/health care	To show how ongoing shared access to blood glucose readings provided by this program would improve communication between patients and providers and enhance diabetes care	Seven patients with T2 diabetes who self-monitor blood glucose	51 (35–65)	57	100	Mixed-methods	Feasibility testing: surveys, focus groups	Patient and provider satisfaction, frequency of use, and changes in glucose levels over a period of 3 months	Graphs facilitated rapid understanding of data. Participants wanted the ability to customize the graph view by time period and type of blood glucose readings (e.g., bedtime).
Zikmund-Fisher et al, 2017b	Informatics	To test whether presenting laboratory test results in visual displays (number lines) could improve understanding	The 1,620 adults in U.S. population (random sample, recruited online)	48.9 (15.7)	52	78	Quantitative	Comprehension testing (randomized order of four visualizations); survey completion	Subjective sense of urgency, behavioral intention, preferences	Visual displays of borderline test values, but not extreme values, reduced participants' perceived urgency and desire to contact health care providers immediately. Controlling for health literacy, numeracy, and graphical literacy, gradient line displays resulted in the greatest sensitivity to changes in test results.
Zikmund-Fisher et al, 2018b	Informatics	To test the effect of including an additional harm anchor reference point in visual displays of laboratory test results	1,618 adults in U.S. population (random sample, recruited online)	48.8 (19–89)	52	78	Quantitative	Comprehension testing (randomized order of four visualizations); survey completion	Subjective sense of urgency, behavioral intention, preferences	Presenting patients with evaluative cues regarding when test results become clinically concerning can reduce the perceived urgency of out-of-range results that do not require immediate clinical action.

Abbreviations: ABP, ambulatory blood pressure; CF, cystic fibrosis; EHR, electronic health record; ISO, the International Organization for Standardization; MMAT, mixed-methods appraisal tool PRO, patient-reported outcome.

Note: MMAT scores denoted by star rating system as follows: ^a51–75% and ^b76–100%.

Risk of Bias

Of the 27 articles, qualitative articles (n = 12) scored between 50 and 100%, indicating low-to-high quality. They primarily lost points for inconsistencies between identified qualitative methodology and description of procedures, and for failing to consider the researchers' influence on findings through interactions with participants. Quantitative articles (n = 8) scored between 50 and 100% (low-to-high quality), losing points for sampling issues (nonrepresentative samples and biased sampling strategies), and failing to use validated scales when measuring outcomes. Mixed-method articles (n = 7) scored between 54 and 85% (low-to-high quality). In addition to losing points for the same reasons as qualitative and quantitative articles, the mixed-method articles also lost points for failing to cohesively integrate the quantitative and qualitative findings. Interrater reliability between the three reviewers was acceptable (0.77–0.88).

Characteristics of Articles with Human Patients Research

The vast majority of the articles (n = 17, 63%) were published in informatics journals, while seven (26%) were published in medical and health care journals, and three (11%) were published in computer science or engineering journals. Sample sizes varied from seven to over 6,700 participants (mean = 369 ± 1,255). Among the articles reporting these characteristics, participants were predominantly middle aged (mean = 51 ± 11), female (60% on average), and White non-Hispanic (70% on average).

Educational attainment was reported in eight articles (30%) and was generally high with 77% of participants in these articles having a college degree or higher on average. Health literacy was measured in five articles (19%), most commonly using the brief three-item screener.[38] Three articles measured numeracy, all using the Subjective Numeracy Scale,[39] and two measured graph literacy using the scale published by Galesic and Garcia-Retamero.[40] Levels of health literacy, numeracy, and graph literacy vary widely between these articles, but the format for reporting (e.g., raw scores, percentages, or descriptions such as “inadequate”) was inconsistent.

The most prevalent research purposes stated by authors were feasibility testing (n = 8), comprehension testing (n = 7), user-centered design sessions (n = 6), and usability testing (n = 4). Common methods for assessing cognitive processes in response to visualizations included “think-aloud protocols,” a usability method in which end-users think aloud as they are performing a set of specified, representative tasks,[41] [42] and eye tracking in which eye movements are unobtrusively measured as the user views content of interest.[43] [44] In some articles (n = 4, 15%), randomization was employed to vary either the order in which visualizations were presented or conditions that participants viewed.[45] [46] [47] [48]

Outcomes pertinent to the visualizations included comprehension (including accuracy and perceived confidence in interpretation), risk perception, behavioral intention (i.e., intention to act in response to the visualization), satisfaction, and preferences. The level of detail in study findings related to visualizations versus other outcomes, for example, usability of an application in which visualizations are embedded, varied widely. Of the articles reporting visualization-specific findings, many reported that visualizations were generally well liked by patients and considered valuable.[49] [50] [51] Visualizations helped patients to ask more sophisticated clinical questions, identify relationships over time, and articulate specific health concerns.[52] [53]

The level of support for patient comprehension of different visualization types varied between studies. For example, two articles reported that line graphs were difficult for patients to interpret.[54] [55] Conversely, a panel of experts in a third article recommended line graphs for longitudinal data.[56] Metaphorical icons increased comprehension in some cases[57] but not when patients interpreted them too literally. For example, patients in one study interpreted icons of fruit baskets, intended to represent the number of fruit servings they had consumed, to mean they consumed each of the illustrated fruits.[58] Nonetheless, bar graphs and number lines were liked and understood by most patients,[55] [59] and they also improved understanding of borderline test results.[47] [59]

Some articles reported that coloring particular regions of visualizations helps patients to understand when values have reached levels that are considered high risk and also increases confidence in interpretation.[58] [60] [61] Importantly, confidence in interpretation did not always correlate with objective comprehension.[62] Several articles showed that inclusion of contextual information in visualizations (e.g., reference ranges and explanatory text), especially when personalized, also improves objective comprehension.[45] [46] [55] [56] [57] [58] [60] Although these components are also helpful for patients interpreting medium risk clinical scenarios and borderline test results, these scenarios remained the most challenging for patients to interpret.[47] [63]

The only study measuring a clinical outcome reported a trend of decreasing blood glucose and hemoglobin A1c levels over 3 months among participants who completed a web-based self-monitoring intervention that included line graphs of blood glucose over time.[64] However, the small sample size (n = 7) prevented the statistical significance of these trends from being assessed.

Discussion

In this systematic review, we reviewed 39 articles to evaluate the types and components of patient-facing visualizations, and research methods and findings related to the development and evaluation of visualizations with patients currently reported in the literature. The visualization types and components in the 39 articles differed, but most included color and number components, and line graphs were the most common type of visualization. The evaluation methods used in the 27 articles that conducted human patients' research on the visualizations varied enormously, with little standardization across the articles with regards to instruments, methodologies, or measurement of patient characteristics that may influence interpretation, such as health literacy or numeracy. As a result, findings cannot easily be compared, and best practices are difficult to identify.

In fact, our review suggests that there is a need for greater attention to developing patient-facing visualizations in general. Our search identified a significantly larger number of articles that included health-care provider or researcher-facing visualizations (n = 834) compared with patient-facing visualizations. This suggests that systems for patients to visualize their health data are not being developed at the same pace as provider-facing systems, despite the fact that more than half of individuals nationwide currently have the patient-portal access.[65] Below we highlight three key opportunities that may advance future research on patient-facing visualizations and allow best practices to more readily be identified.

First, in research on patient-facing visualizations, there are opportunities for more robust data collection and reporting. Nearly half of the articles included in this review either did not conduct any human patients' research on the technology system they developed or did not report findings, specifically related to the visualizations. We also excluded a large number of articles that appeared otherwise eligible for inclusion because they did not include actual images of the visualizations. In addition, many articles reported either no characteristics of the patient sample or a very limited number (e.g., age and gender only). Patient characteristics that were reported indicate nongeneralizable samples, with most participants being middle aged, female, White, and having high-educational attainment. This suggests that findings may be biased to a specific set of patients, and less applicable to underserved, very young, and very old individuals. Importantly, health literacy, graph literacy, and numeracy of the patients were rarely measured or reported. These patient characteristics influence comprehension and risk perception and are therefore important to measure, so that visualizations can be appropriately tailored. Color blindness also influences comprehension of certain colored visual elements, such as the “traffic-light” color scheme that was popular in the included articles. Given that this condition affects an estimated 8% of the male population,[66] color-blindness screening should also be included in future research. In addition, colors and symbols take on different meanings depending on the patient's cultural background[19] [67] which highlights the importance of investigating how the target patient population interprets color when evaluating visualizations which use color encodings. In sum, efforts to collect and report more detailed sample data will facilitate identification of optimal visualizations for specific patient populations.

Second, this review shows there are no consistent approaches in the literature for selecting, developing, or evaluating visualizations according to a particular data type or goal. Nuances in visualization types and components varied widely between articles, as did the methods and measures used to evaluate them. For example, some articles reported general qualitative preferences while some conducted quantitative comprehension and risk perception testing using randomization methods. Moreover, in several articles, the purposes of the visualizations were not explicitly stated. The range of possible purposes is wide, from interpreting a single value, to interpreting values in the context of a goal value or range, to identifying important trends over time. Ultimately, this makes comparison and identification of visualizations that are successful in achieving a particular communication goal challenging. Related reviews examining evaluation methods of health-related visualizations, albeit more broadly, also found a lack of standardized tools and methods for evaluating these visualizations,[21] [22] which hinders progress toward identifying optimal visualizations for an intended audience.

Finally, there exist synergistic opportunities to enhance interpretation and guide behaviors by leveraging data science and using clinically meaningful boundaries, otherwise known as minimally important differences (MIDs).[68] Less than one quarter of the visualizations in the reviewed articles used advanced analytics to generate deeper insights in the data, such as risk scores and significant trends. There have been countless advancements in health data science such that algorithms can now identify meaningful patterns and generate predictions from personal health data. As such, there exists a major opportunity to pair advanced analytics with visualization to convey clinically meaningful and actionable information to patients. It will also be important to extend this work beyond visualizations of data with clear boundaries for action, such as laboratory values, which have well-established normal ranges. The prevalence of visualizations displaying laboratory values among articles in this review is aligned with a recent report showing that laboratory test results are the most common type of information currently offered in patient portals.[65] Nonetheless, MIDs for many other types of personal health data (symptoms and health behaviors) are equally important but more challenging to identify and convey to patients. Currently, research identifying MIDs in PROs and other PGHD is gaining momentum.[68] Visualization-focused research can move this research forward by identifying optimal formats for conveying the MIDs to patients.

Future work should always consider the evolving clinical infrastructure in which these visualizations, and the digital tools in which they are embedded, are being deployed. Clinical practice is changing in the wake of the Affordable Care Act and associated legislation (HITECH, MACRA, 21st Century Cures), which led to a cascade of policies and initiatives encouraging patients to directly exchange health data with providers.[69] [70] [71] [72] This legislation also led to regulatory changes, including a more defined role of the Food and Drug Administration (FDA) in regulating digital health.[69] Whereas low-risk general wellness products (e.g., weight management and physical fitness tracking) are generally not regulated, some tools may be considered Software as a Medical Device (SaMD) and subject to FDA regulation.[73] While compliance with the FDA's evolving regulations will be important, the safety implications of releasing technologies for use by patients that are not regulated will be even more important to address. Without thoughtful design and evaluation, patients without high levels of health literacy and numeracy may draw inaccurate conclusions from personal health data and take inappropriate, possibly unsafe actions as a result of these conclusions.[74] [75] Thus, future work should address the potential role that information visualizations may play in mitigating unintended negative impacts of these policy changes on clinical care, patient safety, health behaviors, and the patient-provider relationship, while bolstering positive impacts.

Conclusion

This systematic review evaluating the state of the science of patient-facing visualizations reveals current trends and research findings that suggest certain visualization types and components that may foster comprehension and interpretation. However, more attention on developing and evaluating patient-facing visualizations is needed. We find opportunities for such research to engage in more robust data collection and reporting and more systematic methods for evaluation; identify and communicate clinically actionable boundaries (MIDs) for diverse data types; and use data science to bolster the patients' ability to interpret and act upon the data. This research will be critically important as patient access of their personal health data through digital health tools continues to rise.

Clinical Relevance Statement

Direct patient access of personal health information through digital and electronic platforms is becoming increasingly prevalent. Patient-facing visualizations may be a powerful a tool for increasing understanding and interpretation of these data. Recommendations generated from the literature base are important for those seeking to develop or implement patient-facing visualizations in clinical settings.

Multiple Choice Questions

1. Which of the following was the most popular type of visualization for displaying longitudinal data among the articles included in this review?

   • Bar graphs.

   • Scatterplots.

   • Line graphs.

   • Number lines.

   Correct Answer: The correct answer is option c. In addition to being the most common visualization in the publication timeframe examined (2005–2018), in general, line graphs were the most common visualization type for displaying longitudinal data; 80% of longitudinal data visualizations used line graphs.

2. Which type of personal health data are most common in both patient portals and in the visualizations in this review that were specifically focused on wellness and prevention?

   • Laboratory values.

   • Blood pressure readings.

   • Symptoms.

   • List of current medications.

   Correct Answer: The correct answer is option a. A recent report from the Office of the National Coordinator for Health Information Technology (ONC) showed that laboratory test results are the most common type of information currently offered in patient portals.[65] In our review, over half of the visualizations focused on wellness and prevention displayed laboratory values, more than any other type of health data.

• References

• 1 CMS. 2016 Program Requirements. Available at: https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/2016ProgramRequirements.html . Accessed August, 26, 2019
• 2 Oh H, Rizo C, Enkin M, Jadad A. What is eHealth (3): a systematic review of published definitions. J Med Internet Res 2005; 7 (01) e1
• 3 Mishra VK, Hoyt RE, Wolver SE, Yoshihashi A, Banas C. Qualitative and quantitative analysis of patients' perceptions of the patient portal experience with OpenNotes. Appl Clin Inform 2019; 10 (01) 10-18
• 4 Health IT. Patient-generated health data. Available at: https://www.healthit.gov/policy-researchers-implementers/patient-generated-health-data . Accessed August, 26, 2019
• 5 Baumhauer JF. Patient-reported outcomes - are they living up to their potential?. N Engl J Med 2017; 377 (01) 6-9
• 6 The Office of the National Coordinator for Health Information Technology. Conceptualizing a data infrastructure for the capture, use, and sharing of patient-generated health data in care delivery and research through 2024. Available at: http://healthit.gov/sites/default/files/onc_pghd_practical_guide.pdf . Accessed August, 26, 2019
• 7 Lai AM, Hsueh PS, Choi YK, Austin RR. Present and future trends in consumer health informatics and patient-generated health data. Yearb Med Inform 2017; 26 (01) 152-159
• 8 Baker L, Rideout J, Gertler P, Raube K. Effect of an internet-based system for doctor-patient communication on health care spending. J Am Med Inform Assoc 2005; 12 (05) 530-536
• 9 Zhou YY, Kanter MH, Wang JJ, Garrido T. Improved quality at Kaiser Permanente through e-mail between physicians and patients. Health Aff (Millwood) 2010; 29 (07) 1370-1375
• 10 McInnes DK, Shimada SL, Midboe AM. , et al. Patient use of electronic prescription refill and secure messaging and its association with undetectable HIV viral load: a retrospective cohort study. J Med Internet Res 2017; 19 (02) e34
• 11 Reading MJ, Merrill JA. Converging and diverging needs between patients and providers who are collecting and using patient-generated health data: an integrative review. J Am Med Inform Assoc 2018; 25 (06) 759-771
• 12 Sanger PC, Hartzler A, Lordon RJ. , et al. A patient-centered system in a provider-centered world: challenges of incorporating post-discharge wound data into practice. J Am Med Inform Assoc 2016; 23 (03) 514-525
• 13 Cohen DJ, Keller SR, Hayes GR, Dorr DA, Ash JS, Sittig DF. Integrating patient-generated health data into clinical care settings or clinical decision-making: lessons learned from project HealthDesign. JMIR Human Factors 2016; 3 (02) e26
• 14 Cheng K, Hayes G, Hirano S, Nagel M, Baker D. Challenges of integrating patient-centered data into clinical workflow for care of high-risk infants. Pers Ubiquitous Comput 2015; 19 (01) 45-57
• 15 Grossman LV, Feiner SK, Mitchell EG, Masterson Creber RM. Leveraging patient-reported outcomes using data visualization. Appl Clin Inform 2018; 9 (03) 565-575
• 16 Woods SS, Evans NC, Frisbee KL. Integrating patient voices into health information for self-care and patient-clinician partnerships: veterans affairs design recommendations for patient-generated data applications. J Am Med Inform Assoc 2016; 23 (03) 491-495
• 17 Few S. Data visualization for human perception. In: Soegaard M, Friis Dam R. , eds. The Encyclopedia of Human-Computer Interaction, 2nd ed. Aarhus, Denmark: Interaction Design Foundation; 2013
• 18 Chen HM. An overview of information visualization. Libr Technol Rep 2017 53. (03):
• 19 Arcia A, Velez M, Bakken S. Style guide: an interdisciplinary communication tool to support the process of generating tailored infographics from electronic health data using EnTICE3. EGEMS (Wash DC) 2015; 3 (01) 1120
• 20 Ancker JS. Delivering patient data to patients themselves. EGEMS (Wash DC) 2018; 6 (01) 16
• 21 Wu DTY, Chen AT, Manning JD. , et al. Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation. J Am Med Inform Assoc 2019; 26 (04) 314-323
• 22 Isenberg T, Isenberg P, Chen J, Sedlmair M, Möller T. A systematic review on the practice of evaluating visualization. IEEE Trans Vis Comput Graph 2013; 19 (12) 2818-2827
• 23 Faisal S, Blandford A, Potts HW. Making sense of personal health information: challenges for information visualization. Health Informatics J 2013; 19 (03) 198-217
• 24 Backonja U, Chi N-C, Choi Y. , et al. Visualization approaches to support healthy aging: A systematic review. J Innov Health Inform 2016; 23 (03) 860
• 25 Mamykina L, Heitkemper EM, Smaldone AM. , et al. Structured scaffolding for reflection and problem solving in diabetes self-management: qualitative study of mobile diabetes detective. J Am Med Inform Assoc 2016; 23 (01) 129-136
• 26 Garcia-Retamero R, Cokely ET. Designing visual aids that promote risk literacy: a systematic review of health research and evidence-based design heuristics. Hum Factors 2017; 59 (04) 582-627
• 27 Cochrane Handbook for Systematic Reviews of Interventions. Part 2: general methods for Cochrane reviews. Available at: https://handbook-5-1.cochrane.org/chapter_6/6_2_1_8_grey_literature_databases.htm
• 28 Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation; 2016. . Available at: www.covidence.org . Accessed September 29, 2019
• 29 Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. Int J Nurs Stud 2009; 46 (04) 529-546
• 30 Pace R, Pluye P, Bartlett G. , et al. Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int J Nurs Stud 2012; 49 (01) 47-53
• 31 Johnston A, Abraham L, Greenslade J. , et al. Review article: Staff perception of the emergency department working environment: Integrative review of the literature. Emerg Med Australas 2016; 28 (01) 7-26
• 32 Mey A, Plummer D, Dukie S, Rogers GD, O'Sullivan M, Domberelli A. Motivations and barriers to treatment uptake and adherence among people living with hiv in australia: a mixed-methods systematic review. AIDS Behav 2017; 21 (02) 352-385
• 33 Tretteteig S, Vatne S, Rokstad AM. The influence of day care centres for people with dementia on family caregivers: an integrative review of the literature. Aging Ment Health 2016; 20 (05) 450-462
• 34 Pluye P, Robert E, Cargo M. , et al. Welcome to the public wiki 'Mixed Methods Appraisal Tool.' Available at: http://mixedmethodsappraisaltoolpublic.pbworks.com . Accessed August 26, 2019
• 35 Lekoane KMB, Kuupiel D, Mashamba-Thompson TP, Ginindza TG. Evidence on the prevalence, incidence, mortality and trends of human papilloma virus-associated cancers in sub-Saharan Africa: systematic scoping review. BMC Cancer 2019; 19 (01) 563
• 36 Liberati A, Altman DG, Tetzlaff J. , et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 2009; 62 (10) e1-e34
• 37 PRISMA. Transparent Reporting of Systematic Reviews and Meta-Analyses. Available at: http://www.prisma-statement.org/ . Accessed August 26, 2019
• 38 Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med 2004; 36 (08) 588-594
• 39 Zikmund-Fisher BJ, Smith DM, Ubel PA, Fagerlin A. Validation of the Subjective Numeracy Scale: effects of low numeracy on comprehension of risk communications and utility elicitations. Med Decis Making 2007; 27 (05) 663-671
• 40 Galesic M, Garcia-Retamero R. Graph literacy: a cross-cultural comparison. Med Decis Making 2011; 31 (03) 444-457
• 41 Phansalkar S, Patel VL, Hoffman JM, Hurdle JF. Use of verbal protocol analysis for identification of ADE signals. AMIA Annu Symp Proc 2006; 1063
• 42 Currie LM, Graham M, Allen M, Bakken S, Patel V, Cimino JJ. Clinical information needs in context: an observational study of clinicians while using a clinical information system. AMIA Annu Symp Proc 2003; 2003: 190-194
• 43 Ehmke C, Wilson S. Identifying web usability problems from eye-tracking data. Proceedings. Available at: https://www.bcs.org/upload/pdf/ewic_hc07_lppaper12.pdf . Accessed August 26, 2019
• 44 Rayner K. Eye movements and attention in reading, scene perception, and visual search. Q J Exp Psychol (Hove) 2009; 62 (08) 1457-1506
• 45 Scherer AM, Witteman HO, Solomon J, Exe NL, Fagerlin A, Zikmund-Fisher BJ. Improving the understanding of test results by substituting (not adding) goal ranges: web-based between-subjects experiment. J Med Internet Res 2018; 20 (10) e11027
• 46 Zikmund-Fisher BJ, Scherer AM, Witteman HO, Solomon JB, Exe NL, Fagerlin A. Effect of harm anchors in visual displays of test results on patient perceptions of urgency about near-normal values: experimental study. J Med Internet Res 2018; 20 (03) e98
• 47 Zikmund-Fisher BJ, Scherer AM, Witteman HO. , et al. Graphics help patients distinguish between urgent and non-urgent deviations in laboratory test results. J Am Med Inform Assoc 2017; 24 (03) 520-528
• 48 Morrow D, Hasegawa-Johnson M, Huang T. , et al. A multidisciplinary approach to designing and evaluating Electronic Medical Record portal messages that support patient self-care. J Biomed Inform 2017; 69: 63-74
• 49 Cox NS, Alison JA, Button BM, Wilson JW, Holland AE. Feasibility and acceptability of an internet-based program to promote physical activity in adults with cystic fibrosis. Respir Care 2015; 60 (03) 422-429
• 50 Harris LT, Tufano J, Le T. , et al. Designing mobile support for glycemic control in patients with diabetes. J Biomed Inform 2010; 43 (5, Suppl): S37-S40
• 51 Kopanitsa G. Evaluation study for an ISO 13606 archetype based medical data visualization method. J Med Syst 2015; 39 (08) 82
• 52 Frost JH, Massagli MP. Social uses of personal health information within PatientsLikeMe, an online patient community: what can happen when patients have access to one another's data. J Med Internet Res 2008; 10 (03) e15
• 53 Smith BK, Frost J, Albayrak M, Sudhakar R. Facilitating narrative medical discussions of type 1 diabetes with computer visualizations and photography. Patient Educ Couns 2006; 64 (1,3): 313-321
• 54 Britto MT, Jimison HB, Munafo JK, Wissman J, Rogers ML, Hersh W. Usability testing finds problems for novice users of pediatric portals. J Am Med Inform Assoc 2009; 16 (05) 660-669
• 55 Elder NC, Barney K. “But what does it mean for me?” Primary care patients' communication preferences for test results notification. Jt Comm J Qual Patient Saf 2012; 38 (04) 168-176
• 56 Snyder C, Smith K, Holzner B, Rivera YM, Bantug E, Brundage M. ; PRO Data Presentation Delphi Panel. Making a picture worth a thousand numbers: recommendations for graphically displaying patient-reported outcomes data. Qual Life Res 2019; 28 (02) 345-356
• 57 Martinez W, Threatt AL, Rosenbloom ST, Wallston KA, Hickson GB, Elasy TA. A Patient-facing diabetes dashboard embedded in a patient web portal: design sprint and usability testing. JMIR Human Factors 2018; 5 (03) e26
• 58 Arcia A, Suero-Tejeda N, Bales ME. , et al. Sometimes more is more: iterative participatory design of infographics for engagement of community members with varying levels of health literacy. J Am Med Inform Assoc 2016; 23 (01) 174-183
• 59 Brewer NT, Gilkey MB, Lillie SE, Hesse BW, Sheridan SL. Tables or bar graphs? Presenting test results in electronic medical records. Med Decis Making 2012; 32 (04) 545-553
• 60 Tao D, Yuan J, Qu X. Presenting self-monitoring test results for consumers: the effects of graphical formats and age. J Am Med Inform Assoc 2018; 25 (08) 1036-1046
• 61 Solomon J, Scherer AM, Exe NL, Witteman HO, Fagerlin A, Zikmund-Fisher BJ. Is This Good or Bad?: Redesigning Visual Displays of Medical Test Results in Patient Portals to Provide Context and Meaning. Proceedings of the 2016 CHI Conference Extended Abstracts 2016 ;doi:10.1145/2851581.2892523
• 62 Hohenstein JC, Baumer EP, Reynolds L. , et al. Supporting accurate interpretation of self-administered medical test results for mobile health: assessment of design, demographics, and health condition. JMIR Human Factors 2018; 5 (01) e9
• 63 Fraccaro P, Vigo M, Balatsoukas P. , et al. Presentation of laboratory test results in patient portals: influence of interface design on risk interpretation and visual search behaviour. BMC Med Inform Decis Mak 2018; 18 (01) 11
• 64 Watson AJ, Kvedar JC, Rahman B, Pelletier AC, Salber G, Grant RW. Diabetes connected health: a pilot study of a patient- and provider-shared glucose monitoring web application. J Diabetes Sci Technol 2009; 3 (02) 345-352
• 65 Patel V, Johnson C. ONC Data Brief. Individuals' use of online medical records and technology for health needs. Available at: https://www.healthit.gov/sites/default/files/page/2018-03/HINTS-2017-Consumer-Data-Brief-3.21.18.pdf . Accessed August 26, 2019
• 66 Facts About Color Blindness. Color Blindness. 2015 https://nei.nih.gov/health/color_blindness/facts_about
• 67 Lor M. Color-encoding visualizations as a tool to assist a nonliterate population in completing health survey responses. Inform Health Soc Care 2018 ; (e-pub ahead of print) Doi: 10.1080/17538157.2018.1540422
• 68 Lapin B, Thompson NR, Schuster A, Katzan IL. Clinical utility of patient-reported outcome measurement information system domain scales. Circ Cardiovasc Qual Outcomes 2019; 12 (01) e004753
• 69 21st Century Cures Act (Cures Act) 2016. Available at: https://www.fda.gov/regulatory-information/selected-amendments-fdc-act/21st-century-cures-act . Accessed September 29, 2019
• 70 The Health Information Technology for Economic and Clinical Health (HITECH) Act 2009. Available at: https://www.healthit.gov/sites/default/files/hitech_act_excerpt_from_arra_with_index.pdf . Accessed September 29, 2019
• 71 Medicare Access and CHIP Reauthorization Act of 2015. Available at: https://www.govtrack.us/congress/bills/114/hr2 . Accessed September 29, 2019
• 72 Patient Protection and Affordable Care Act 2010
• 73 FDA. Guidances with digital health content. Available at: https://www.fda.gov/MedicalDevices/DigitalHealth/ucm562577.htm . Accessed August 26, 2019
• 74 Howie L, Hirsch B, Locklear T, Abernethy AP. Assessing the value of patient-generated data to comparative effectiveness research. Health Aff (Millwood) 2014; 33 (07) 1220-1228
• 75 Veinot TC, Mitchell H, Ancker JS. Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc 2018; 25 (08) 1080-1088
• 76 Ankersen DV, Carlsen K, Marker D, Munkholm P, Burisch J. Using eHealth strategies in delivering dietary and other therapies in patients with irritable bowel syndrome and inflammatory bowel disease. J Gastroenterol Hepatol 2017; 32 (Suppl 1): 27-31
• 77 Beaudin JS, Intille SS, Morris ME. To track or not to track: user reactions to concepts in longitudinal health monitoring. J Med Internet Res 2006; 8 (04) e29
• 78 Desai PM, Levine ME, Albers DJ, Mamykina L. Pictures Worth a Thousand Words: Reflections on Visualizing Personal Blood Glucose Forecasts for Individuals with Type 2 Diabetes, in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada: ACM; 2018: 1-13
• 79 Farmer A, Gibson O, Hayton P, Bryden K, Dudley C, Neil A, Tarassenko L. A real-time, mobile phone-based telemedicine system to support young adults with type 1 diabetes. Inform Prim Care 2005; 13 (03) 171-177
• 80 Filipi JM, Khairat S. Tracking and visualizing headache trends on a mobile or desktop website. Stud Health Technol Inform 2013; 192: 1199
• 81 Foraker RE, Shoben AB, Lopetegui MA. , et al. Assessment of life’s Simple 7 in the primary care setting: the Stroke Prevention in Healthcare Delivery EnviRonmEnts (SPHERE) study. Contemp Clin Trials 2014; 38 (02) 182-189
• 82 Hosseini A, Buonocore CM, Hashemzadeh S. Feasibility of a secure wireless sensing smartwatch application for the self-management of pediatric asthma. Sensors (Basel) 2017; 17 (08) pii:E1780
• 83 Ilic V. Web environment for psychometric diagnostics and psychotherapy. International Review on Computers and Software (IRECOS) 2009; 4: 278-287
• 84 Kallen MA, Yang D, Haas N. A technical solution to improving palliative and hospice care. Support Care Cancer 2012; 20 (01) 167-174
• 85 Kopanitsa G, Karpov A, Lakovenko G, Laskovenko A. Implementation of a web portal for diabetes patients using open source data visualization libraries. Stud Health Technol Inform 2016; 224: 189-194
• 86 Lee HR, Yoo SK, Jung SM, Kwon NY, Hong CS. A Web-based mobile asthma management system. J Telemed Telecare 2005; 11 (Suppl 1): 56-59
• 87 Lowe-Strong A, McCullagh PJ. Monitoring of symptoms and interventions associated with multiple sclerosis. Stud Health Technol Inform 2005; 117: 223-228
• 88 Mena LJ, Felix VG, Ostos R. Mobile personal health system for ambulatory blood pressure monitoring. Comput Math Methods Med 2013; 2013: 598196
• 89 Robu A, Gauca B, Crisan-Vida M, Stoicu-Tivadar L. Integrated system for monitoring and prevention in obstetrics-gynaecology. Stud Health Technol Inform 2016; 221: 8-12
• 90 Rudin RS, Fanta CH, Predmore Z. Core components for a clinically integrated mHealth app for asthma symptom monitoring. Appl Clin Inform 2017; 8 (04) 1031-1043
• 91 Schenk Jr RJ, Schenk J. Integration of remote blood glucose meter upload technology into a clinical pharmacist medication therapy management service. J Diabetes Sci Technol 2011; 5 (01) 188-191
• 92 Skrøvseth SO, Årsand E, Godtliebsen F, Hartvigsen G. Mobile phone-based pattern recognition and data analysis for patients with type 1 diabetes. Diabetes Technol Ther 2012; 14 (12) 1098-1104
• 93 Snyder C, Smith K, Holzner B, Rivera YM, Bantug E, Brundage M. ; PRO Data Presentation Delphi Panel. Making a picture worth a thousand numbers: recommendations for graphically displaying patient-reported outcomes data. Qual Life Res 2019; 28 (02) 345-356
• 94 Voils CI, Coffman CJ, Edelman D. , et al. Examining the impact of genetic testing for type 2 diabetes on health behaviors: study protocol for a randomized controlled trial. Trials 2012; 13: 121