Usability and Workflow Evaluation of “RhEumAtic Disease activitY” (READY)

Summary

Background RhEumAtic Disease activitY (READY) is a mobile health (mHealth) application that aims to create a shared platform integrating data from both patients and physicians, with a particular emphasis on arthritis disease activity.

Methods We made READY available on an iPad and pilot implemented it at a rheumatology outpatient clinic. We conducted 1) a usability evaluation study to explore patients’ and physicians’ interactions with READY, and 2) a time motion study (TMS) to observe the clinical workflow before and after the implementation.

Results A total of 33 patients and 15 physicians participated in the usability evaluation. We found usability problems in navigation, data entry, pain assessment, documentation, and instructions along with error messages. Despite these issues, 25 (75,76%) patients reported they liked READY. Physicians provided mixed feedback because they were concerned about the impact of READY on clinical workflow. Six physicians participated in the TMS. We observed 47 patient visits (44.72 hours) in the pre-implementation phase, and 42 patient visits (37.82 hours) in the post-implementation phase. We found that patients spent more time on READY than paper (4.39mins vs. 2.26mins), but overall, READY did not delay the workflow (pre = 52.08 mins vs. post = 45.46 mins). This time difference may be compensated with READY eliminating a workflow step for the staff.

Conclusion Patients preferred READY to paper documents. Many found it easier to input information because of the larger font size and the ease of ‘tapping’ rather than writing-out or circling answers. Even though patients spent more time on READY than using paper documents, the longer usage of READY was mainly due to when troubleshooting was needed. Most patients did not have problems after receiving initial support from the staff. This study not only enabled improvements to the software but also serves as good reference for other researchers or institutional decision makers who are interested in implementing such a technology.

Citation: Yen P, Lara B, Lopetegui M, Bharat A, Ardoin S, Johnson B, Mathur P, Embi P, Curtis J. Usability and workflow evaluation of “RhEumAtic Disease activitY” (READY).

Keywords

Rheumatoid arthritis - mobile health - disease activity - time motion study - mobile application - mHealth - workflows and human interactions - mobile computing and communication - usability

• References

• 1 Hersh WR, Cimino J, Payne PR, Embi P, Logan J, Weiner M, Bernstam EV, Lehmann H, Hripcsak G, Hartzog T, Saltz J. Recommendations for the use of operational electronic health record data in comparative effectiveness research. EGEMS (Wash DC) 2013; 01 (01) 1018.
• 2 Lopetegui MA, Bai S, Yen PY, Lai A, Embi P, Payne PR. Inter-observer reliability assessments in time motion studies: the foundation for meaningful clinical workflow analysis. AMIA Annu Symp Proc 2013; 2013: 889-96.
• 3 Singh JA, Saag KG, Bridges Jr SL, Akl EA, Bannuru RR, Sullivan MC, Vaysbrot E, McNaughton C, Osani M, Shmerling RH, Curtis JR, Furst DE, Parks D, Kavanaugh A, O’Dell J, King C, Leong A, Matteson EL, Schousboe JT, Drevlow B, Ginsberg S, Grober J, St Clair EW, Tindall E, Miller AS, McAlindon T. 2015 American College of Rheumatology Guideline for the Treatment of Rheumatoid Arthritis. Arthritis Rheumatol 2016; 68 (01) 1-26.
• 4 Pincus T, Swearingen CJ, Bergman M, Yazici Y. RAPID3 (Routine Assessment of Patient Index Data 3), a rheumatoid arthritis index without formal joint counts for routine care: proposed severity categories compared to disease activity score and clinical disease activity index categories. J Rheumatol 2008; 35 (11) 2136-47.
• 5 Yazici Y, Bergman M, Pincus T. Time to score quantitative rheumatoid arthritis measures: 28-Joint Count, Disease Activity Score, Health Assessment Questionnaire (HAQ), Multidimensional HAQ (MDHAQ), and Routine Assessment of Patient Index Data (RAPID) scores. J Rheumatol 2008; 35 (04) 603-9.
• 6 Aletaha D, Nell VP, Stamm T, Uffmann M, Pflugbeil S, Machold K, Smolen JS. Acute phase reactants add little to composite disease activity indices for rheumatoid arthritis: validation of a clinical activity score. Arthritis Res Ther 2005; 07 (04) R796-806.
• 7 Blake H. Innovation in practice: mobile phone technology in patient care. Br J Community Nurs 2008; 13 (04) 160 2-5..
• 8 Patrick K, Griswold WG, Raab F, Intille SS. Health and the mobile phone. Am J Prev Med 2008; 35 (02) 177-81.
• 9 Plaza I, Demarzo MM, Herrera-Mercadal P, Garcia-Campayo J. Mindfulness-based mobile applications: literature review and analysis of current features. JMIR Mhealth Uhealth 2013; 01 (02) e24.
• 10 Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res 2012; 14 (05) e128.
• 11 Mathur P, Southern L, Wang S, Heckler C, Lele O, Embi P, Curtis JR. High Level Architecture and Evaluation of Patient Linkages for READY - An Electronic Measurement Tool for Rheumatoid Arthritis. Proceedings of the American Medical Informatics Association Annual Symposium. 2015
• 12 Curtis JR. RhEumAtic Disease activitY. 2014 Available from: https://itunes.apple.com/us/app/rheumaticdisease-activity/id657411562?mt=8
• 13 Gilbert A, Sebag-Montefiore D, Davidson S, Velikova G. Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic. Gynecol Oncol 2015; 136 (03) 429-39.
• 14 Jensen RE, Rothrock NE, DeWitt EM, Spiegel B, Tucker CA, Crane HM, Forrest CB, Patrick DL, Fredericksen R, Shulman LM, Cella D, Crane PK. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care 2015; 53 (02) 153-9.
• 15 Schick-Makaroff K, Molzahn A. Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Outcomes 2015; 13: 2.
• 16 Zheng K, Haftel HM, Hirschl RB, O’Reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc 2010; 17 (04) 454-61.
• 17 Bruland P, Forster C, Breil B, Stander S, Dugas M, Fritz F. Does single-source create an added value? Evaluating the impact of introducing x4T into the clinical routine on workflow modifications, data quality and cost-benefit. Int J Med Inform 2014; 83 (12) 915-28.
• 18 Koldby S, Schou IJensen. Clinical simulation and workflow by use of two clinical information systems, the electronic health record and digital dictation. Stud Health Technol Inform 2013; 192: 402-6.
• 19 Mensah N, Sukums F, Awine T, Meid A, Williams J, Akweongo P, Kaltschmidt J, Haefeli WE, Blank A. Impact of an electronic clinical decision support system on workflow in antenatal care: the QUALMAT eCDSS in rural health care facilities in Ghana and Tanzania. Glob Health Action 2015; 08: 25756.
• 20 Militello LG, Arbuckle NB, Saleem JJ, Patterson E, Flanagan M, Haggstrom D, Doebbeling BN. Sources of variation in primary care clinical workflow: implications for the design of cognitive support. Health Informatics J 2014; 20 (01) 35-49.
• 21 Lewis C. Using the “think aloud” method in cognitive interface design. New York: IBM; 1982
• 22 Holzinger A. Usability engineering methods for software developers. Commun ACM 2005; 48 (01) 71-4.
• 23 Cohen T, Kaufman D, White T, Segal G, Staub AB, Patel V, Finnerty M. Cognitive evaluation of an innovative psychiatric clinical knowledge enhancement system. Studies in Health Technology & Informatics 2004; 107 (Pt 2): 1295-9.
• 24 Peute LWP, Jaspers MWM. The significance of a usability evaluation of an emerging laboratory order entry system. International Journal of Medical Informatics 2007; 76 2-3 157-68.
• 25 Elhadad N, McKeown K, Kaufman D, Jordan D. Facilitating physicians’ access to information via tailored text summarization. AMIA Annu Symp Proc. 2005. Annual Symposium Proceedings/AMIA Symposium; 226-30.
• 26 Yu H, Lee M, Kaufman D, Ely J, Osheroff JA, Hripcsak G, Cimino J. Development, implementation, and a cognitive evaluation of a definitional question answering system for physicians. Journal of Biomedical Informatics 2007; 40 (03) 236-51.
• 27 Horsky J, Kaufman DR, Patel VL. The cognitive complexity of a provider order entry interface. AMIA Annu Symp Proc. 2003. Annual Symposium Proceedings/AMIA Symposium; 294-8.
• 28 Lopetegui M, Yen PY, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking about?. J Biomed Inform 2014; 49: 292-9.
• 29 Medicine NLo. MeSH: Time and Motion Studies. 1991 Available from: http://www.ncbi.nlm.nih.gov/mesh?term=Time+and+Motion+Studies
• 30 Lopetegui M, Yen PY, Lai AM, Embi PJ, Payne PR. Time Capture Tool (TimeCaT): development of a comprehensive application to support data capture for Time Motion Studies. AMIA Annu Symp Proc 2012; 2012: 596-605.
• 31 Chen JW. Developing a process for reducing functional discrepancies. Doctoral dissertation: School of Health Information Sciences, University of Texas Health Sciene Center at Houston. 2008
• 32 van den Brink RH, Troquete NA, Beintema H, Mulder T, van Os TW, Schoevers RA, Wiersma D. Risk assessment by client and case manager for shared decision making in outpatient forensic psychiatry. BMC Psychiatry 2015; 15: 120.
• 33 Klingaman EA, Medoff DR, Park SG, Brown CH, Fang L, Dixon LB, Hack SM, Tapscott SL, Walsh MB, Kreyenbuhl JA. Consumer satisfaction with psychiatric services: The role of shared decision making and the therapeutic relationship. Psychiatr Rehabil J 2015; 38 (03) 242-8.
• 34 Flynn D, Knoedler MA, Hess EP, Murad MH, Erwin PJ, Montori VM, Thomson RG. Engaging patients in health care decisions in the emergency department through shared decision-making: a systematic review. Acad Emerg Med 2012; 19 (08) 959-67.
• 35 Asan OP DS, Montague E. More screen time, less face time - implications for EHR design. J Eval Clin Pract 2014; 20 (06) 896-901.
• 36 Patel V, Hale TM, Palakodeti S, Kvedar JC, Jethwani K. Prescription Tablets in the Digital Age: A Cross-Sectional Study Exploring Patient and Physician Attitudes Toward the Use of Tablets for Clinic-Based Personalized Health Care Information Exchange. JMIR Res Protoc 2015; 04 (04) e116.
• 37 Asan O, Carayon P, Beasley JW, Montague E. Work system factors influencing physicians’ screen sharing behaviors in primary care encounters. Int J Med Inform 2015; 84 (10) 791-8.
• 38 Singman EL, Haberman CV, Appelbaum J, Tian J, Shafer K, Toerper M, Katz S, Kelsay M, Boland MV, Greenbaum M, Adelman R, Thomas RC, Vakili S. Electronic Tracking of Patients in an Outpatient Ophthalmology Clinic to Improve Efficient Flow: A Feasibility Analysis and Benchmarking Study. Qual Manag Health Care 2015; 24 (04) 190-9.
• 39 eHealth Observatory and School of Health Information Science University of Victoria. Canada Health Infoway Benefits Evaluation Indicators, technical report. 2012 Available from: https://www.infoway-inforoute.ca/index.php/programs-services/benefits-evaluation
• 40 Agency for Healthcare Research and Quality. Health IT Evaluation Measures. 2014 Available from: http://healthit.ahrq.gov/health-it-tools-and-resources/health-it-evaluation-toolkit-and-evaluation-measures-quick-reference
• 41 Spetz J, Burgess J, Phibbs C. The effect of health information technology implementation in Veterans Health Administration hospitals on patient outcomes. Healthcare 2014; 02: 40-7.
• 42 PROMIS: Dynamic Tools to Measure Health Outcomes from the Patient Perspective. Available from: http://www.nihpromis.org/