Download PDF
Appl Clin Inform 2023; 14(02): 227-237
DOI: 10.1055/a-2008-4036
Special Section on Patient Engagement in Informatics

A RE-AIM Evaluation of a Visualization-Based Electronic Patient-Reported Outcome System

Meghan Reading Turchioe
1   Columbia University School of Nursing, New York, New York, United States
,
Sabrina Mangal
2   University of Washington School of Nursing, Seattle, Washington, United States
3   Division of General Internal Medicine, Department of Medicine, Weill Cornell Medicine, New York, New York, United States
,
Parag Goyal
3   Division of General Internal Medicine, Department of Medicine, Weill Cornell Medicine, New York, New York, United States
,
Kelly Axsom
4   Division of Cardiology, Center for Advanced Cardiac Care, Columbia University Medical Center, New York, New York, United States
,
Annie Myers
1   Columbia University School of Nursing, New York, New York, United States
,
Lisa G. Liu
5   Department of Pediatrics, University of California San Francisco, San Francisco, California, United States
,
Jessie Lee
5   Department of Pediatrics, University of California San Francisco, San Francisco, California, United States
,
Thomas R. Campion Jr.
2   University of Washington School of Nursing, Seattle, Washington, United States
6   Clinical and Translational Science Center, Weill Cornell Medicine, New York, New York, United States
,
Ruth Masterson Creber
1   Columbia University School of Nursing, New York, New York, United States
› Author Affiliations Funding This study was supported by a National Institute of Nursing Research grant (R00NR016275; PI: RMC) and training grant (K99NR019124; PI: MRT). This work was also supported by grant number UL1 TR 002384 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH).
› Further Information
Also available at
    Permissions and Reprints

Abstract

Objectives Health care systems are primarily collecting patient-reported outcomes (PROs) for research and clinical care using proprietary, institution- and disease-specific tools for remote assessment. The purpose of this study was to conduct a Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation of a scalable electronic PRO (ePRO) reporting and visualization system in a single-arm study.

Methods The “mi.symptoms” ePRO system was designed using gerontechnological design principles to ensure high usability among older adults. The system enables longitudinal reporting of disease-agnostic ePROs and includes patient-facing PRO visualizations. We conducted an evaluation of the implementation of the system guided by the RE-AIM framework. Quantitative data were analyzed using basic descriptive statistics, and qualitative data were analyzed using directed content analysis.

Results Reach—the total reach of the study was 70 participants (median age: 69, 31% female, 17% Black or African American, 27% reported not having enough financial resources). Effectiveness—half (51%) of participants completed the 2-week follow-up survey and 36% completed all follow-up surveys. Adoption—the desire for increased self-knowledge, the value of tracking symptoms, and altruism motivated participants to adopt the tool. Implementation—the predisposing factor was access to, and comfort with, computers. Three enabling factors were incorporation into routines, multimodal nudges, and ease of use. Maintenance—reinforcing factors were perceived usefulness of viewing symptom reports with the tool and understanding the value of sustained symptom tracking in general.

Conclusion Challenges in ePRO reporting, particularly sustained patient engagement, remain. Nonetheless, freely available, scalable, disease-agnostic systems may pave the road toward inclusion of a more diverse range of health systems and patients in ePRO collection and use.

Keywords

patient-reported outcomes - patient engagement - data visualizations - remote assessment

Protection of Human and Animal Subjects

This study was reviewed and approved by the Weill Cornell Medicine Institutional Review Board.


Supplementary Material



Publication History

Received: 08 August 2022

Accepted: 04 January 2023

Accepted Manuscript online:
05 January 2023

Article published online:
22 March 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Lavallee DC, Chenok KE, Love RM. et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood) 2016; 35 (04) 575-582
    CrossrefPubMedGoogle Scholar
  • 2 National Quality Forum. Patient-reported outcomes. Accessed January 22, 2023 at: https://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx
    PubMed
  • 3 Warsame R, D'Souza A. Patient reported outcomes have arrived: a practical overview for clinicians in using patient reported outcomes in oncology. Mayo Clin Proc 2019; 94 (11) 2291-2301
    CrossrefPubMedGoogle Scholar
  • 4 Basch E. Patient-reported outcomes - harnessing patients' voices to improve clinical care. N Engl J Med 2017; 376 (02) 105-108
    CrossrefPubMedGoogle Scholar
  • 5 Burns DJP, Arora J, Okunade O. et al. International Consortium for Health Outcomes Measurement (ICHOM): standardized patient-centered outcomes measurement set for heart failure patients. JACC Heart Fail 2020; 8 (03) 212-222
    CrossrefPubMedGoogle Scholar
  • 6 Field J, Holmes MM, Newell D. PROMs data: can it be used to make decisions for individual patients? A narrative review. Patient Relat Outcome Meas 2019; 10: 233-241
    CrossrefPubMedGoogle Scholar
  • 7 Jonkman NH, Westland H, Groenwold RHH. et al. Do self-management interventions work in patients with heart failure? An individual patient data meta-analysis. Circulation 2016; 133 (12) 1189-1198
    CrossrefPubMedGoogle Scholar
  • 8 Vodicka E, Kim K, Devine EB, Gnanasakthy A, Scoggins JF, Patrick DL. Inclusion of patient-reported outcome measures in registered clinical trials: evidence from ClinicalTrials.gov (2007-2013). Contemp Clin Trials 2015; 43: 1-9
    CrossrefPubMedGoogle Scholar
  • 9 Scoggins JF, Patrick DL. The use of patient-reported outcomes instruments in registered clinical trials: evidence from ClinicalTrials.gov. Contemp Clin Trials 2009; 30 (04) 289-292
    CrossrefPubMedGoogle Scholar
  • 10 Schwartzberg L. Electronic patient-reported outcomes: the time is ripe for integration into patient care and clinical research. Am Soc Clin Oncol Educ Book 2016; 35: e89-e96
    CrossrefPubMedGoogle Scholar
  • 11 Aiyegbusi OL, Nair D, Peipert JD, Schick-Makaroff K, Mucsi I. A narrative review of current evidence supporting the implementation of electronic patient-reported outcome measures in the management of chronic diseases. Ther Adv Chronic Dis 2021; 12: 20406 223211015958
    CrossrefPubMedGoogle Scholar
  • 12 Hassett MJ, Cronin C, Tsou TC. et al. eSyM: an electronic health record-integrated patient-reported outcomes-based cancer symptom management program used by six diverse health systems. JCO Clin Cancer Inform 2022; 6 (06) e2100137
    CrossrefPubMedGoogle Scholar
  • 13 Zylla DM, Gilmore GE, Steele GL. et al. Collection of electronic patient-reported symptoms in patients with advanced cancer using Epic MyChart surveys. Support Care Cancer 2020; 28 (07) 3153-3163
    CrossrefPubMedGoogle Scholar
  • 14 Galesic M, Garcia-Retamero R. Graph literacy: a cross-cultural comparison. Med Decis Making 2011; 31 (03) 444-457
    CrossrefPubMedGoogle Scholar
  • 15 Reading Turchioe M, Grossman LV, Myers AC, Baik D, Goyal P, Masterson Creber RM. Visual analogies, not graphs, increase patients' comprehension of changes in their health status. J Am Med Inform Assoc 2020; 27 (05) 677-689
    CrossrefPubMedGoogle Scholar
  • 16 Hawley ST, Zikmund-Fisher B, Ubel P, Jancovic A, Lucas T, Fagerlin A. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns 2008; 73 (03) 448-455
    CrossrefPubMedGoogle Scholar
  • 17 Zikmund-Fisher BJ, Scherer AM, Witteman HO. et al. Graphics help patients distinguish between urgent and non-urgent deviations in laboratory test results. J Am Med Inform Assoc 2017; 24 (03) 520-528
    CrossrefPubMedGoogle Scholar
  • 18 REDCap (Research Electronic Data Capture). Accessed January 22, 2023 at: https://www.project-redcap.org/
    PubMed
  • 19 Masterson Creber RM, Hickey KT, Maurer MS. Gerontechnologies for older patients with heart failure: what is the role of smartphones, tablets, and remote monitoring devices in improving symptom monitoring and self-care management?. Curr Cardiovasc Risk Rep 2016; 10 (10) 30
    CrossrefPubMedGoogle Scholar
  • 20 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1–2): 23-30
    CrossrefPubMedGoogle Scholar
  • 21 E-mail usage in the United States - statistics & facts. Statista. Accessed November 18, 2022 at: https://www.statista.com/topics/4295/e-mail-usage-in-the-united-states/
    PubMed
  • 22 Sheet MF. Pew Research Center: internet, science & tech. Published April 7, 2021. Accessed November 18, 2022 at: https://www.pewresearch.org/internet/fact-sheet/mobile/
    PubMed
  • 23 Baldwin JL, Singh H, Sittig DF, Giardina TD. Patient portals and health apps: pitfalls, promises, and what one might learn from the other. Healthc (Amst) 2017; 5 (03) 81-85
    CrossrefPubMedGoogle Scholar
  • 24 Glasgow RE, Harden SM, Gaglio B. et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019; 7: 64
    CrossrefPubMedGoogle Scholar
  • 25 Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med 2004; 36 (08) 588-594
    PubMedGoogle Scholar
  • 26 Assarroudi A, Heshmati Nabavi F, Armat MR, Ebadi A, Vaismoradi M. Directed qualitative content analysis: the description and elaboration of its underpinning methods and data analysis process. J Res Nurs 2018; 23 (01) 42-55
    CrossrefPubMedGoogle Scholar
  • 27 Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15 (09) 1277-1288
    Google Scholar
  • 28 Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today 2004; 24 (02) 105-112
    CrossrefPubMedGoogle Scholar
  • 29 Rosett HA, Herring K, Ratliff W, Koontz BF, Zafar Y, LeBlanc TW. Integration of electronic patient-reported outcomes into clinical workflows within the Epic electronic medical record. J Clin Orthod 2019; 37 (31, suppl): 102
    PubMedGoogle Scholar
  • 30 Reading Turchioe M, Burgermaster M, Mitchell EG, Desai PM, Mamykina L. Adapting the stage-based model of personal informatics for low-resource communities in the context of type 2 diabetes. J Biomed Inform 2020; 110: 103572
    CrossrefPubMedGoogle Scholar
  • 31 Reading M, Baik D, Beauchemin M, Hickey KT, Merrill JA. Factors influencing sustained engagement with ECG self-monitoring: perspectives from patients and health care providers. Appl Clin Inform 2018; 9 (04) 772-781
    Article in Thieme ConnectPubMedGoogle Scholar
  • 32 Miyamoto SW, Henderson S, Young HM, Pande A, Han JJ. Tracking health data is not enough: a qualitative exploration of the role of healthcare partnerships and mhealth technology to promote physical activity and to sustain behavior change. JMIR Mhealth Uhealth 2016; 4 (01) e5
    CrossrefPubMedGoogle Scholar
  • 33 Snyder LE, Phan DF, Williams KC. et al. Comprehension, utility, and preferences of prostate cancer survivors for visual timelines of patient-reported outcomes co-designed for limited graph literacy: meters and emojis over comics. J Am Med Inform Assoc 2022; 29 (11) 1838-1846
    CrossrefPubMedGoogle Scholar
  • 34 Stonbraker S, Porras T, Schnall R. Patient preferences for visualization of longitudinal patient-reported outcomes data. J Am Med Inform Assoc 2020; 27 (02) 212-224
    CrossrefPubMedGoogle Scholar
  • 35 Benda NC, Montague E, Valdez RS. Chapter 15 - Design for inclusivity. In: Sethumadhavan A, Sasangohar F, eds. Design for Health. Cambridge, MA: Academic Press; 2020: 305-322
    Google Scholar
  • 36 Platt J, Spector-Bagdady K, Platt T. et al. Ethical, legal, and social implications of learning health systems. Learn Health Syst 2018; 2 (01) e10051
    CrossrefPubMedGoogle Scholar
  • 37 Thorpe JH. Cartwright-Smith, L Gray, E Mongeon, M. Legal and ethical architecture for PCOR data. George Washington University; 2017. Accessed January 22, 2023 at: https://www.healthit.gov/sites/default/files/page/2018-06/PCOR%20Architecture%20%28MERGE%29%20updated%20Appendix%20B.pdf
    PubMed
  • 38 Gensheimer SG, Wu AW, Snyder CF. PRO-EHR Users' Guide Steering Group, PRO-EHR Users' Guide Working Group. Oh, the places we'll go: patient-reported outcomes and electronic health records. Patient 2018; 11 (06) 591-598
    CrossrefPubMedGoogle Scholar
  • 39 Brundage MD, Wu AW, Rivera YM, Snyder C. Promoting effective use of patient-reported outcomes in clinical practice: themes from a “Methods Tool kit” paper series. J Clin Epidemiol 2020; 122: 153-159
    CrossrefPubMedGoogle Scholar
  • 40 Skovlund PC, Ravn S, Seibaek L, Thaysen HV, Lomborg K, Nielsen BK. The development of PROmunication: a training-tool for clinicians using patient-reported outcomes to promote patient-centred communication in clinical cancer settings. J Patient Rep Outcomes 2020; 4 (01) 10
    CrossrefPubMedGoogle Scholar
  • 41 Christiansen MN, Køber L, Weeke P. et al. Age-specific trends in incidence, mortality, and comorbidities of heart failure in Denmark, 1995 to 2012. Circulation 2017; 135 (13) 1214-1223
    CrossrefPubMedGoogle Scholar