A RE-AIM Evaluation of a Visualization-Based Electronic Patient-Reported Outcome System

Abstract

Objectives Health care systems are primarily collecting patient-reported outcomes (PROs) for research and clinical care using proprietary, institution- and disease-specific tools for remote assessment. The purpose of this study was to conduct a Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation of a scalable electronic PRO (ePRO) reporting and visualization system in a single-arm study.

Methods The “mi.symptoms” ePRO system was designed using gerontechnological design principles to ensure high usability among older adults. The system enables longitudinal reporting of disease-agnostic ePROs and includes patient-facing PRO visualizations. We conducted an evaluation of the implementation of the system guided by the RE-AIM framework. Quantitative data were analyzed using basic descriptive statistics, and qualitative data were analyzed using directed content analysis.

Results Reach—the total reach of the study was 70 participants (median age: 69, 31% female, 17% Black or African American, 27% reported not having enough financial resources). Effectiveness—half (51%) of participants completed the 2-week follow-up survey and 36% completed all follow-up surveys. Adoption—the desire for increased self-knowledge, the value of tracking symptoms, and altruism motivated participants to adopt the tool. Implementation—the predisposing factor was access to, and comfort with, computers. Three enabling factors were incorporation into routines, multimodal nudges, and ease of use. Maintenance—reinforcing factors were perceived usefulness of viewing symptom reports with the tool and understanding the value of sustained symptom tracking in general.

Conclusion Challenges in ePRO reporting, particularly sustained patient engagement, remain. Nonetheless, freely available, scalable, disease-agnostic systems may pave the road toward inclusion of a more diverse range of health systems and patients in ePRO collection and use.

Keywords

patient-reported outcomes - patient engagement - data visualizations - remote assessment

Protection of Human and Animal Subjects

This study was reviewed and approved by the Weill Cornell Medicine Institutional Review Board.

Publication History

Received: 08 August 2022

Accepted: 04 January 2023

Accepted Manuscript online:
05 January 2023

Article published online:
22 March 2023

• References

• 1 Lavallee DC, Chenok KE, Love RM. et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood) 2016; 35 (04) 575-582
• 2 National Quality Forum. Patient-reported outcomes. Accessed January 22, 2023 at: https://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx
• 3 Warsame R, D'Souza A. Patient reported outcomes have arrived: a practical overview for clinicians in using patient reported outcomes in oncology. Mayo Clin Proc 2019; 94 (11) 2291-2301
• 4 Basch E. Patient-reported outcomes - harnessing patients' voices to improve clinical care. N Engl J Med 2017; 376 (02) 105-108
• 5 Burns DJP, Arora J, Okunade O. et al. International Consortium for Health Outcomes Measurement (ICHOM): standardized patient-centered outcomes measurement set for heart failure patients. JACC Heart Fail 2020; 8 (03) 212-222
• 6 Field J, Holmes MM, Newell D. PROMs data: can it be used to make decisions for individual patients? A narrative review. Patient Relat Outcome Meas 2019; 10: 233-241
• 7 Jonkman NH, Westland H, Groenwold RHH. et al. Do self-management interventions work in patients with heart failure? An individual patient data meta-analysis. Circulation 2016; 133 (12) 1189-1198
• 8 Vodicka E, Kim K, Devine EB, Gnanasakthy A, Scoggins JF, Patrick DL. Inclusion of patient-reported outcome measures in registered clinical trials: evidence from ClinicalTrials.gov (2007-2013). Contemp Clin Trials 2015; 43: 1-9
• 9 Scoggins JF, Patrick DL. The use of patient-reported outcomes instruments in registered clinical trials: evidence from ClinicalTrials.gov. Contemp Clin Trials 2009; 30 (04) 289-292
• 10 Schwartzberg L. Electronic patient-reported outcomes: the time is ripe for integration into patient care and clinical research. Am Soc Clin Oncol Educ Book 2016; 35: e89-e96
• 11 Aiyegbusi OL, Nair D, Peipert JD, Schick-Makaroff K, Mucsi I. A narrative review of current evidence supporting the implementation of electronic patient-reported outcome measures in the management of chronic diseases. Ther Adv Chronic Dis 2021; 12: 20406 223211015958
• 12 Hassett MJ, Cronin C, Tsou TC. et al. eSyM: an electronic health record-integrated patient-reported outcomes-based cancer symptom management program used by six diverse health systems. JCO Clin Cancer Inform 2022; 6 (06) e2100137
• 13 Zylla DM, Gilmore GE, Steele GL. et al. Collection of electronic patient-reported symptoms in patients with advanced cancer using Epic MyChart surveys. Support Care Cancer 2020; 28 (07) 3153-3163
• 14 Galesic M, Garcia-Retamero R. Graph literacy: a cross-cultural comparison. Med Decis Making 2011; 31 (03) 444-457
• 15 Reading Turchioe M, Grossman LV, Myers AC, Baik D, Goyal P, Masterson Creber RM. Visual analogies, not graphs, increase patients' comprehension of changes in their health status. J Am Med Inform Assoc 2020; 27 (05) 677-689
• 16 Hawley ST, Zikmund-Fisher B, Ubel P, Jancovic A, Lucas T, Fagerlin A. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns 2008; 73 (03) 448-455
• 17 Zikmund-Fisher BJ, Scherer AM, Witteman HO. et al. Graphics help patients distinguish between urgent and non-urgent deviations in laboratory test results. J Am Med Inform Assoc 2017; 24 (03) 520-528
• 18 REDCap (Research Electronic Data Capture). Accessed January 22, 2023 at: https://www.project-redcap.org/
• 19 Masterson Creber RM, Hickey KT, Maurer MS. Gerontechnologies for older patients with heart failure: what is the role of smartphones, tablets, and remote monitoring devices in improving symptom monitoring and self-care management?. Curr Cardiovasc Risk Rep 2016; 10 (10) 30
• 20 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1–2): 23-30
• 21 E-mail usage in the United States - statistics & facts. Statista. Accessed November 18, 2022 at: https://www.statista.com/topics/4295/e-mail-usage-in-the-united-states/
• 22 Sheet MF. Pew Research Center: internet, science & tech. Published April 7, 2021. Accessed November 18, 2022 at: https://www.pewresearch.org/internet/fact-sheet/mobile/
• 23 Baldwin JL, Singh H, Sittig DF, Giardina TD. Patient portals and health apps: pitfalls, promises, and what one might learn from the other. Healthc (Amst) 2017; 5 (03) 81-85
• 24 Glasgow RE, Harden SM, Gaglio B. et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019; 7: 64
• 25 Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med 2004; 36 (08) 588-594
• 26 Assarroudi A, Heshmati Nabavi F, Armat MR, Ebadi A, Vaismoradi M. Directed qualitative content analysis: the description and elaboration of its underpinning methods and data analysis process. J Res Nurs 2018; 23 (01) 42-55
• 27 Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15 (09) 1277-1288
• 28 Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today 2004; 24 (02) 105-112
• 29 Rosett HA, Herring K, Ratliff W, Koontz BF, Zafar Y, LeBlanc TW. Integration of electronic patient-reported outcomes into clinical workflows within the Epic electronic medical record. J Clin Orthod 2019; 37 (31, suppl): 102
• 30 Reading Turchioe M, Burgermaster M, Mitchell EG, Desai PM, Mamykina L. Adapting the stage-based model of personal informatics for low-resource communities in the context of type 2 diabetes. J Biomed Inform 2020; 110: 103572
• 31 Reading M, Baik D, Beauchemin M, Hickey KT, Merrill JA. Factors influencing sustained engagement with ECG self-monitoring: perspectives from patients and health care providers. Appl Clin Inform 2018; 9 (04) 772-781
• 32 Miyamoto SW, Henderson S, Young HM, Pande A, Han JJ. Tracking health data is not enough: a qualitative exploration of the role of healthcare partnerships and mhealth technology to promote physical activity and to sustain behavior change. JMIR Mhealth Uhealth 2016; 4 (01) e5
• 33 Snyder LE, Phan DF, Williams KC. et al. Comprehension, utility, and preferences of prostate cancer survivors for visual timelines of patient-reported outcomes co-designed for limited graph literacy: meters and emojis over comics. J Am Med Inform Assoc 2022; 29 (11) 1838-1846
• 34 Stonbraker S, Porras T, Schnall R. Patient preferences for visualization of longitudinal patient-reported outcomes data. J Am Med Inform Assoc 2020; 27 (02) 212-224
• 35 Benda NC, Montague E, Valdez RS. Chapter 15 - Design for inclusivity. In: Sethumadhavan A, Sasangohar F, eds. Design for Health. Cambridge, MA: Academic Press; 2020: 305-322
• 36 Platt J, Spector-Bagdady K, Platt T. et al. Ethical, legal, and social implications of learning health systems. Learn Health Syst 2018; 2 (01) e10051
• 37 Thorpe JH. Cartwright-Smith, L Gray, E Mongeon, M. Legal and ethical architecture for PCOR data. George Washington University; 2017. Accessed January 22, 2023 at: https://www.healthit.gov/sites/default/files/page/2018-06/PCOR%20Architecture%20%28MERGE%29%20updated%20Appendix%20B.pdf
• 38 Gensheimer SG, Wu AW, Snyder CF. PRO-EHR Users' Guide Steering Group, PRO-EHR Users' Guide Working Group. Oh, the places we'll go: patient-reported outcomes and electronic health records. Patient 2018; 11 (06) 591-598
• 39 Brundage MD, Wu AW, Rivera YM, Snyder C. Promoting effective use of patient-reported outcomes in clinical practice: themes from a “Methods Tool kit” paper series. J Clin Epidemiol 2020; 122: 153-159
• 40 Skovlund PC, Ravn S, Seibaek L, Thaysen HV, Lomborg K, Nielsen BK. The development of PROmunication: a training-tool for clinicians using patient-reported outcomes to promote patient-centred communication in clinical cancer settings. J Patient Rep Outcomes 2020; 4 (01) 10
• 41 Christiansen MN, Køber L, Weeke P. et al. Age-specific trends in incidence, mortality, and comorbidities of heart failure in Denmark, 1995 to 2012. Circulation 2017; 135 (13) 1214-1223