Download PDF
Appl Clin Inform 2023; 14(02): 212-226
DOI: 10.1055/a-2006-4936
Research Article

Usability Testing of an Interoperable Computerized Clinical Decision Support Tool for Fall Risk Management in Primary Care

Kristen Shear
1   Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, Florida, United States
,
Hannah Rice
2   Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
,
Pamela M. Garabedian
3   Department of Information Systems, Mass General Brigham, Somerville, Massachusetts, United States
,
Ragnhildur Bjarnadottir
1   Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, Florida, United States
,
Nancy Lathum
2   Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
,
Ann L. Horgas
4   Department of Biobehavioral Nursing Science, College of Nursing, University of Florida, Gainesville, Florida, United States
,
Christopher A. Harle
5   Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, United States
,
Patricia C. Dykes
2   Department of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
,
Robert Lucero
6   UCLA School of Nursing, University of California Los Angeles, Los Angeles, California, United States
› Author Affiliations Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: this work was supported by the Agency for Healthcare Research and Quality [grant number: U18HS027557].
› Further Information
Also available at
    Permissions and Reprints

Abstract

Background Falls are a widespread and persistent problem for community-dwelling older adults. Use of fall prevention guidelines in the primary care setting has been suboptimal. Interoperable computerized clinical decision support systems have the potential to increase engagement with fall risk management at scale. To support fall risk management across organizations, our team developed the ASPIRE tool for use in differing primary care clinics using interoperable standards.

Objectives Usability testing of ASPIRE was conducted to measure ease of access, overall usability, learnability, and acceptability prior to pilot .

Methods Participants were recruited using purposive sampling from two sites with different electronic health records and different clinical organizations. Formative testing rooted in user-centered design was followed by summative testing using a simulation approach. During summative testing participants used ASPIRE across two clinical scenarios and were randomized to determine which scenario they saw first. Single Ease Question and System Usability Scale were used in addition to analysis of recorded sessions in NVivo.

Results All 14 participants rated the usability of ASPIRE as above average based on usability benchmarks for the System Usability Scale metric. Time on task decreased significantly between the first and second scenarios indicating good learnability. However, acceptability data were more mixed with some recommendations being consistently accepted while others were adopted less frequently.

Conclusion This study described the usability testing of the ASPIRE system within two different organizations using different electronic health records. Overall, the system was rated well, and further pilot testing should be done to validate that these positive results translate into clinical practice. Due to its interoperable design, ASPIRE could be integrated into diverse organizations allowing a tailored implementation without the need to build a new system for each organization. This distinction makes ASPIRE well positioned to impact the challenge of falls at scale.

Keywords

usability testing - clinical decision support - falls - primary care

Protection of Human and Animal Subjects

IRB approval was received for protocol number: Site 1: 2020P002075; Site 2: CED000000426_.


Note

P.C.D. and R.L. contributed equally to the development of the manuscript as co–senior investigators.

The views expressed herein are those of the author(s) and do not reflect the official policy or position of Brooke Army Medical Center, the Department of Defense, or any agencies under the U.S. Government.




Publication History

Received: 07 July 2022

Accepted: 02 January 2023

Accepted Manuscript online:
04 January 2023

Article published online:
15 March 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Florence CS, Bergen G, Atherly A, Burns E, Stevens J, Drake C. Medical costs of fatal and nonfatal falls in older adults. J Am Geriatr Soc 2018; 66 (04) 693-698
    CrossrefPubMedGoogle Scholar
  • 2 Moreland B, Kakara R, Henry A. Trends in nonfatal falls and fall-related injuries among adults aged ≥65 years - United States, 2012-2018. MMWR Morb Mortal Wkly Rep 2020; 69 (27) 875-881
    CrossrefPubMedGoogle Scholar
  • 3 Clinical Decision Support. The Office of the National Coordinator for Health Information Technology. Accessed March 7, 2021 at: https://www.healthit.gov/topic/safety/clinical-decision-support
    PubMedGoogle Scholar
  • 4 Bryan C, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inform Prim Care 2008; 16 (02) 79-91
    PubMedGoogle Scholar
  • 5 Trinkley KE, Kroehl ME, Kahn MG. et al. Applying clinical decision support design best practices with the practical robust implementation and sustainability model versus reliance on commercially available clinical decision support tools: randomized controlled trial. JMIR Med Inform 2021; 9 (03) e24359
    CrossrefPubMedGoogle Scholar
  • 6 Taylor SF, Coogle CL, Cotter JJ, Welleford EA, Copolillo A. Community-dwelling older adults' adherence to environmental fall prevention recommendations. J Appl Gerontol 2019; 38 (06) 755-774
    CrossrefPubMedGoogle Scholar
  • 7 Assurance NCfQ. Fall risk management. National Committee for Quality Assurance. Accessed April 25, 2022 at: https://www.ncqa.org/hedis/measures/fall-risk-management/
    PubMed
  • 8 Howland J, Hackman H, Taylor A, O'Hara K, Liu J, Brusch J. Older adult fall prevention practices among primary care providers at accountable care organizations: a pilot study. PLoS One 2018; 13 (10) e0205279
    CrossrefPubMedGoogle Scholar
  • 9 Spears GV, Roth CP, Miake-Lye IM, Saliba D, Shekelle PG, Ganz DA. Redesign of an electronic clinical reminder to prevent falls in older adults. Med Care 2013; 51 (03) (3, suppl 1): S37-S43
    CrossrefPubMedGoogle Scholar
  • 10 Zheng MY, Suneja A, Chou AL, Arya M. Physician barriers to successful implementation of US Preventive Services Task Force routine HIV testing recommendations. J Int Assoc Provid AIDS Care 2014; 13 (03) 200-205
    CrossrefPubMedGoogle Scholar
  • 11 Hoskins KF, Tejeda S, Vijayasiri G. et al. A feasibility study of breast cancer genetic risk assessment in a federally qualified health center. Cancer 2018; 124 (18) 3733-3741
    CrossrefPubMedGoogle Scholar
  • 12 Kurth AE, Krist AH, Borsky AE. et al. U.S. Preventive Services Task Force methods to communicate and disseminate clinical preventive services recommendations. Am J Prev Med 2018; 54 (1S1): S81-S87
    CrossrefPubMedGoogle Scholar
  • 13 Braunstein ML. Health Informatics on FHIR: How HL7's New API is Transforming Healthcare. Cham: Springer; 2018: 292
    CrossrefGoogle Scholar
  • 14 Barnett ML, Bitton A, Souza J, Landon BE. Trends in outpatient care for medicare beneficiaries and implications for primary care, 2000 to 2019. Ann Intern Med 2021; 174 (12) 1658-1665
    CrossrefPubMedGoogle Scholar
  • 15 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17
    CrossrefPubMedGoogle Scholar
  • 16 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
    Google Scholar
  • 17 Nielsen J. Chapter 2 - What is usability? In: Nielsen J, ed. Usability Engineering. Burlington, MA: Morgan Kaufmann; 1993: 23-48
  • 18 Peute LW, Spithoven R, Bakker PJ, Jaspers MW. Usability studies on interactive health information systems; where do we stand?. Stud Health Technol Inform 2008; 136: 327-332
    PubMedGoogle Scholar
  • 19 Nielsen J, Mack RL. Usability Inspection Methods. Morristown, NJ: Wiley; 1994
    CrossrefGoogle Scholar
  • 20 Li AC, Kannry JL, Kushniruk A. et al. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform 2012; 81 (11) 761-772
    CrossrefPubMedGoogle Scholar
  • 21 Rice H, Garabedian PM, Shear K. et al. Clinical decision support for fall prevention: defining end-user needs. Appl Clin Inform 2022; 13 (03) 647-655
    Article in Thieme ConnectPubMedGoogle Scholar
  • 22 Rice HG, Garabedian P, Shear K. et al. Computerized clinical decision support for fall prevention: defining end-user requirements for primary care staff and patients. Paper presented at: 2022 AMIA Informatics Summit; March 21–24, 2022; Chicago, IL, United States. Accessed January 18, 2023 at: https://knowledge.amia.org/75287-amia-1.4633888/t005-1.4635399/t005-1.4635400/2082-1.4635455/2082-1.4635456
    PubMedGoogle Scholar
  • 23 Grossman DC, Curry SJ, Owens DK. et al; US Preventive Services Task Force. Interventions to prevent falls in community-dwelling older adults: US Preventive Services Task Force recommendation statement. JAMA 2018; 319 (16) 1696-1704
    CrossrefPubMedGoogle Scholar
  • 24 Shubert TE, Smith ML, Jiang L, Ory MG. Disseminating the Otago Exercise Program in the United States: perceived and actual physical performance improvements from participants. J Appl Gerontol 2018; 37 (01) 79-98
    CrossrefPubMedGoogle Scholar
  • 25 De Vito Dabbs A, Myers BA, Mc Curry KR. et al. User-centered design and interactive health technologies for patients. Comput Inform Nurs 2009; 27 (03) 175-183
    CrossrefPubMedGoogle Scholar
  • 26 Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. 2nd ed. Amsterdam: Elsevier; 2016: 350
    Google Scholar
  • 27 Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35 (03) 379-383
    CrossrefPubMedGoogle Scholar
  • 28 Albert W, Tullis TS. Chapter 5 - Self-reported metrics. In: Albert W, Tullis TS, eds. Measuring the User Experience. 3rd ed. Burlington, MA: Morgan Kaufmann; 2023: 109-151
    Google Scholar
  • 29 Lewis JR, Sauro J. The Factor Structure of the System Usability Scale. Berlin, Heidelberg: Springer; 2009: 94-103
    Google Scholar
  • 30 Borsci S, Federici S, Bacci S, Gnaldi M, Bartolucci F. Assessing user satisfaction in the era of user experience: comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience. Int J Hum Comput Interact 2015; 31 (08) 484-495
    CrossrefPubMedGoogle Scholar
  • 31 Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017; 17 (01) 88
    CrossrefPubMedGoogle Scholar
  • 32 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
    CrossrefPubMedGoogle Scholar
  • 33 Connect Repository CDS. Agency for Healthcare Research and Quality. Accessed April 29, 2022 at: https://cds.ahrq.gov/cdsconnect/repository
    PubMed
  • 34 Camacho PM, Petak SM, Binkley N. et al. American Association of Clinical Endocrinologists/American College of Endocrinology Clinical Practice Guidelines for the Diagnosis and Treatment of Postmenopausal Osteoporosis-2020 Update. Endocr Pract 2020; 26 (suppl 1): 1-46
    Google Scholar
  • 35 Weinfeld JM, Gorman PN. Primary care physician designation and response to clinical decision support reminders: a cross-sectional study. Appl Clin Inform 2016; 7 (02) 248-259
    Article in Thieme ConnectPubMedGoogle Scholar