Download PDF
CC BY 4.0 · ACI open 2025; 09(01): e18-e28
DOI: 10.1055/a-2437-9977
Research Article

Human-Centered Design and Iterative Refinement of Tools and Methods to Implement a Surveillance and Risk Prediction System for Clinical Deterioration in Ambulatory Cancer Care

Daniel J. France
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
2   Vanderbilt University School of Nursing, Nashville, Tennessee, United States
,
Paromita Nath
3   Department of Mechanical Engineering, Henry M. Rowan College of Engineering, Glassboro, New Jersey, United States
,
Jason Slagle
4   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Shilo Anders
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Megan Salwei
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Timothy Vogus
5   Owen Graduate School of Management, Vanderbilt University, Nashville, Tennessee, United States
,
Hannah Slater
4   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Carrie Reale
6   Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Laurie Novak
4   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Lori-Anne Parker-Danley
7   Department of Patient and Family Engagement, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Zachary Kohutek
8   Department of Radiation, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Rajiv Agarwal
9   Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Joyce Harris
4   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Barbara Yudiskas
6   Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Ralph Conwill
6   Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Terrell Smith
10   Department of Patient Care, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Evan Rhodes
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Emma Schremp
11   Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Erin A. Gillaspie
12   Department of Thoracic Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Adam Wright
4   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Robert E. Freundlich
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Ryan Myles
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Janelle Faiman
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Sankaran Mahadevan
13   Department of Civil Engineering, Vanderbilt University, Nashville, Tennessee, United States
,
Matthew Weinger
1   Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, United States
› Author Affiliations Funding This study was supported by grant 5R18HS026616 to Drs. Weinger and France from the Agency for Healthcare Research and Quality (AHRQ, Rockville, MD). The use of REDCap and MyCap were supported by a grant from the National Center for Advancing Translational Sciences (NCATS) Clinical Translational Science Award (CTSA) Program (5UL1TR002243) via the Vanderbilt Institute for Clinical and Translational Research (VICTR) center.
› Further Information
Also available at
   Permissions and Reprints

Abstract

Background A common cause of preventable harm is the failure to detect and appropriately respond to clinical deterioration. Timely intervention is needed, particularly in medically complex patients, to mitigate the effects of adverse events, disease progression, and medical error. This challenging problem requires clinical surveillance, early recognition, timely notification of the appropriate clinicians, and effective intervention.

Objectives We determined the feasibility of designing, developing, and implementing the tools and processes to create a surveillance-and-risk prediction system to detect clinical deterioration in cancer outpatients.

Methods We used systems engineering and iterative human-centered design to develop a functional prototype of a surveillance-and-risk prediction system. The system includes passive surveillance involving wearable sensors, active surveillance involving patient event and symptom reporting as well as extraction of selected patient data from the electronic health record (EHR), a predictive model, and communication of estimated risk to clinicians. System usability was evaluated using patient and clinician interviews and clinician ratings using the System Usability Scale (SUS).

Results Fifty of 71 recruited patients enrolled in the feasibility study. Patient-reported outcome measures and clinical data extracted from the EHR were the best predictors of a patient's 7-day risk of experiencing unplanned treatment events (UTEs, i.e., emergency room visits, hospital admissions, or major treatment changes). Deep learning neural network models using these predictors demonstrated modest performance in predicting 7-day UTE risk (PROMS, F-measure: 0.900, area under the receiver operating characteristic curve [AUC-ROC]: 0.983; clinical data from EHR F-measure: 0.625, AUC-ROC: 0.983). Patient risk scores were communicated to clinicians using a risk communication prototype rated favorably by clinicians with a SUS score of 76 out of 100 (median = 80; range: 60–85).

Conclusion We demonstrate the feasibility of a surveillance-and-risk prediction system for detecting and reporting clinical deterioration in cancer outpatients. Future research is needed to fully implement and evaluate system adoption and effectiveness under different clinical situations.

Keywords

surveillance-and-risk prediction - ambulatory cancer - wearables - patient-reported outcome measures - non-routine events - patient-centered care

Protection of Human and Animal Subjects

The study was approved by the The Vanderbilt University Institutional Review Board Institutional Review Board. Patient recruitment and enrollment occurred between September 2019 and July 2023.


Supplementary Material



Publication History

Received: 18 March 2024

Accepted: 10 September 2024

Article published online:
21 February 2025

© 2025. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, and reproduction so long as the original work is properly cited. (https://creativecommons.org/licenses/by/4.0/)

Georg Thieme Verlag KG
Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany

 

  • References

  • 1 Committee on Improving the Quality of Cancer Care: Addressing the Challenges of an Aging Population. Board on Health Care Services; Institute of Medicine; Levit LA, Balogh E, Nass SJ. et al. (editors). Delivering High-Quality Cancer Care: charting a New Course for a system in crisis. Washington, D.C.: National Academies Press; 2013
    Search in Google Scholar
  • 2 Jeong A, Shin DW, Kim SY. et al. The effects on caregivers of cancer patients' needs and family hardiness. Psychooncology 2016; 25 (01) 84-90
    CrossrefPubMedSearch in Google Scholar
  • 3 Lapid MI, Atherton PJ, Kung S. et al. Cancer caregiver quality of life: need for targeted intervention. Psychooncology 2016; 25 (12) 1400-1407
    CrossrefPubMedSearch in Google Scholar
  • 4 Hurria A, Togawa K, Mohile SG. et al. Predicting chemotherapy toxicity in older adults with cancer: a prospective multicenter study. J Clin Oncol 2011; 29 (25) 3457-3465
    CrossrefPubMedSearch in Google Scholar
  • 5 Lustig A. Medication error prevention by pharmacists–an Israeli solution. Pharm World Sci 2000; 22 (01) 21-25
    CrossrefPubMedSearch in Google Scholar
  • 6 Schwappach DL, Wernli M. Medication errors in chemotherapy: incidence, types and involvement of patients in prevention. A review of the literature. Eur J Cancer Care (Engl) 2010; 19 (03) 285-292
    CrossrefPubMedSearch in Google Scholar
  • 7 Gandhi TK, Bartel SB, Shulman LN. et al. Medication safety in the ambulatory chemotherapy setting. Cancer 2005; 104 (11) 2477-2483
    CrossrefPubMedSearch in Google Scholar
  • 8 Markert A, Thierry V, Kleber M, Behrens M, Engelhardt M. Chemotherapy safety and severe adverse events in cancer patients: strategies to efficiently avoid chemotherapy errors in in- and outpatient treatment. Int J Cancer 2009; 124 (03) 722-728
    CrossrefPubMedSearch in Google Scholar
  • 9 Chera BS, Mazur L, Buchanan I. et al. Improving patient safety in clinical oncology: applying lessons from normal accident theory. JAMA Oncol 2015; 1 (07) 958-964
    CrossrefPubMedSearch in Google Scholar
  • 10 Christiansen AH, Lipczak H, Knudsen JL. Attention to cancer patients' safety after primary treatment is needed. Dan Med J 2015; 62 (06) A5090
    PubMedSearch in Google Scholar
  • 11 Walsh KE, Dodd KS, Seetharaman K. et al. Medication errors among adults and children with cancer in the outpatient setting. J Clin Oncol 2009; 27 (06) 891-896
    CrossrefPubMedSearch in Google Scholar
  • 12 Shafiq J, Barton M, Noble D, Lemer C, Donaldson LJ. An international review of patient safety measures in radiotherapy practice. Radiother Oncol 2009; 92 (01) 15-21
    CrossrefPubMedSearch in Google Scholar
  • 13 MacKenzie AR, Parker I. Introduction to quality issues in vulnerable populations. J Oncol Pract 2015; 11 (03) 185-186
    CrossrefPubMedSearch in Google Scholar
  • 14 Gansler T, Henley SJ, Stein K, Nehl EJ, Smigal C, Slaughter E. Sociodemographic determinants of cancer treatment health literacy. Cancer 2005; 104 (03) 653-660
    CrossrefPubMedSearch in Google Scholar
  • 15 Ohri N, Kabarriti R, Bodner WR. et al. Continuous activity monitoring during concurrent chemoradiotherapy. Int J Radiat Oncol Biol Phys 2017; 97 (05) 1061-1065
    CrossrefPubMedSearch in Google Scholar
  • 16 Meisenberg BR, Hahn E, Binner M. et al. ReCAP: Insights Into the Potential Preventability of Oncology Readmissions. J Oncol Pract 2016; 12 (02) 153-154 ; e49–56
    CrossrefPubMedSearch in Google Scholar
  • 17 Schmidt PE, Meredith P, Prytherch DR. et al. Impact of introducing an electronic physiological surveillance system on hospital mortality. BMJ Qual Saf 2015; 24 (01) 10-20
    CrossrefPubMedSearch in Google Scholar
  • 18 Hosein FS, Bobrovitz N, Berthelot S, Zygun D, Ghali WA, Stelfox HT. A systematic review of tools for predicting severe adverse events following patient discharge from intensive care units. Crit Care 2013; 17 (03) R102
    CrossrefPubMedSearch in Google Scholar
  • 19 McNeill G, Bryden D. Do either early warning systems or emergency response teams improve hospital patient survival? A systematic review. Resuscitation 2013; 84 (12) 1652-1667
    CrossrefPubMedSearch in Google Scholar
  • 20 McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 2007; 18 (03) CD005529
    PubMedSearch in Google Scholar
  • 21 Li D, Lyons P, Klaus J, Gage B, Kollef M, Lu C. Integrating Static and Time-Series Data in Deep Recurrent Models for Oncology Early Warning Systems. Paper presented at: 30th ACM International Conference on Information and Knowledge Management; 2021; Gold Coast, Queensland, Australia
    PubMed
  • 22 Baker E, Fatoye F. Clinical and cost effectiveness of nurse-led self-management interventions for patients with copd in primary care: a systematic review. Int J Nurs Stud 2017; 71: 125-138
    CrossrefPubMedSearch in Google Scholar
  • 23 Jensen L, Troster SM, Cai K. et al. Improving heart failure outcomes in ambulatory and community care: a scoping study. Med Care Res Rev 2017; 74 (05) 551-581
    CrossrefPubMedSearch in Google Scholar
  • 24 Wigg AJ, McCormick R, Wundke R. et al. Efficacy of a chronic disease management model for patients with chronic liver failure. Clin Gastroenterol Hepatol 2013; 11 (07) 850-858.e1–4
    CrossrefPubMedSearch in Google Scholar
  • 25 Jones DA, DeVita MA, Bellomo R. Rapid-response teams. N Engl J Med 2011; 365 (02) 139-146
    CrossrefPubMedSearch in Google Scholar
  • 26 Chan PS, Jain R, Nallmothu BK, Berg RA, Sasson C. Rapid response teams: a systematic review and meta-analysis. Arch Intern Med 2010; 170 (01) 18-26
    CrossrefPubMedSearch in Google Scholar
  • 27 Low CA, Dey AK, Ferreira D. et al. Estimation of symptom severity during chemotherapy from passively sensed data: Exploratory study. J Med Internet Res 2017; 19 (12) e420
    CrossrefPubMedSearch in Google Scholar
  • 28 Weinger MB, Slagle J. Human factors research in anesthesia patient safety. Proc AMIA Symp 2001; 756-760
    PubMedSearch in Google Scholar
  • 29 Weinger MB, Wiklund M, Gardner-Bonneau D. Eds. Human Factors in Medical Device Design: A Handbook for Designers. Boca Raton, FL: CRC Press/Taylor & Francis; 2011
    Search in Google Scholar
  • 30 Norman D, Draper SW. eds. User Centered System Design. Hillsdale, NJ: Lawrence Erlbaum & Assoc; 1986
    CrossrefSearch in Google Scholar
  • 31 Weinger MB, Slagle J, Jain S, Ordonez N. Retrospective data collection and analytical techniques for patient safety studies. J Biomed Inform 2003; 36 (1–2): 106-119
    CrossrefPubMedSearch in Google Scholar
  • 32 Harris PA, Swafford J, Serdoz ES. et al. MyCap: a flexible and configurable platform for mobilizing the participant voice. JAMIA Open 2022; 5 (02) ooac047
    CrossrefPubMedSearch in Google Scholar
  • 33 Oken A, Rasmussen MD, Slagle JM. et al. A facilitated survey instrument captures significantly more anesthesia events than does traditional voluntary event reporting. Anesthesiology 2007; 107 (06) 909-922
    CrossrefPubMedSearch in Google Scholar
  • 34 Ownby KK. Use of the distress thermometer in clinical practice. J Adv Pract Oncol 2019; 10 (02) 175-179
    PubMedSearch in Google Scholar
  • 35 Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Qual Life Res 2009; 18 (07) 873-880
    CrossrefPubMedSearch in Google Scholar
  • 36 Hong JC, Niedzwiecki D, Palta M, Tenenbaum JD. Predicting emergency visits and hospital admissions during radiation and chemoradiation: an internally validated pretreatment machine learning algorithm. JCO Clin Cancer Inform 2018; 2: 1-11
    CrossrefPubMedSearch in Google Scholar
  • 37 Dedoose Version 9.0.107, cloud application for managing, analyzing, and presenting qualitative and mixed method research data 2023. Los Angeles, CA: SocioCultural Research Consultants, LLC; . Available at: www.dedoose.com
  • 38 Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs 2008; 62 (01) 107-115
    CrossrefPubMedSearch in Google Scholar
  • 39 Vogus TJ, Sutcliffe KM. The Safety Organizing Scale: development and validation of a behavioral measure of safety culture in hospital nursing units. Med Care 2007; 45 (01) 46-54
    CrossrefPubMedSearch in Google Scholar
  • 40 Hundt AS, Carayon P, Yang Y. et al. Role network analysis of team interactions and individual activities: Application to VTE prophylaxis. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2017; Vol. 61, pp. 896–900
    PubMed
  • 41 Salwei ME, Carayon P, Hundt AS. et al. Role network measures to assess healthcare team adaptation to complex situations: the case of venous thromboembolism prophylaxis. Ergonomics 2019; 62 (07) 864-879
    CrossrefPubMedSearch in Google Scholar
  • 42 International Organization for Standardization. ISO 9241-210: Ergonomics of Human-System interaction. Part 210: Human-centred design for interactive systems. Geneva, Switzerland: ISO; 2010
    Search in Google Scholar
  • 43 Darwish M. Mural: Verso Books; 2017
    PubMed
  • 44 Brooke J. SUS: A “quick and dirty” usability scale. In: Jordan PW, Werdmeester BA, McClelland AL. eds. Usability Evaluation in Industry. London, UK: Taylor & Francis; 1996
    Search in Google Scholar
  • 45 Bangor AW, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 114-123
    PubMedSearch in Google Scholar
  • 46 Pavic M, Klaas V, Theile G, Kraft J, Tröster G, Guckenberger M. Feasibility and usability aspects of continuous remote monitoring of health status in palliative cancer patients using wearables. Oncology 2020; 98 (06) 386-395
    CrossrefPubMedSearch in Google Scholar
  • 47 Koumakis L, Schera F, Parker H. et al. Fostering palliative care through digital intervention: a platform for adult patients with hematologic malignancies. Front Digit Health 2021; 3: 730722
    CrossrefPubMedSearch in Google Scholar
  • 48 Owusuaa C, van der Padt-Pruijsten A, Drooger JC. et al. Development of a clinical prediction model for 1-year mortality in patients with advanced cancer. JAMA Netw Open 2022; 5 (11) e2244350
    CrossrefPubMedSearch in Google Scholar
  • 49 Li D, Vaidya J, Wang M. et al. Predicting clinical deterioration of outpatients using multimodal data collected by wearables. Proc ACM Interact Mob Wearable Ubiquitous Technol 2018; 1-22
    PubMedSearch in Google Scholar
  • 50 Lizée T, Basch E, Trémolières P. et al. Cost-effectiveness of web-based patient-reported outcome surveillance in patients with lung cancer. J Thorac Oncol 2019; 14 (06) 1012-1020
    CrossrefPubMedSearch in Google Scholar
  • 51 Chung AE, Basch EM. Incorporating the patient's voice into electronic health records through patient-reported outcomes as the “review of systems”. J Am Med Inform Assoc 2015; 22 (04) 914-916
    CrossrefPubMedSearch in Google Scholar
  • 52 Chung AE, Shoenbill K, Mitchell SA. et al. Patient free text reporting of symptomatic adverse events in cancer clinical research using the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). J Am Med Inform Assoc 2019; 26 (04) 276-285
    CrossrefPubMedSearch in Google Scholar
  • 53 Basch E, Mody GN, Dueck AC. Electronic patient-reported outcomes as digital therapeutics to improve cancer outcomes. JCO Oncol Pract 2020; 16 (09) 541-542
    CrossrefPubMedSearch in Google Scholar
  • 54 Basch E, Pugh SL, Dueck AC. et al. Feasibility of patient reporting of symptomatic adverse events via the Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) in a chemoradiotherapy cooperative group multicenter clinical trial. Int J Radiat Oncol Biol Phys 2017; 98 (02) 409-418
    CrossrefPubMedSearch in Google Scholar
  • 55 Basch E, Schrag D, Henson S. et al. Effect of electronic symptom monitoring on patient-reported outcomes among patients with metastatic cancer: a randomized clinical trial. JAMA 2022; 327 (24) 2413-2422
    CrossrefPubMedSearch in Google Scholar
  • 56 Basch E, Snyder C. Overcoming barriers to integrating patient-reported outcomes in clinical practice and electronic health records. Ann Oncol 2017; 28 (10) 2332-2333
    CrossrefPubMedSearch in Google Scholar
  • 57 Basch EM, Reeve BB, Mitchell SA. et al. Electronic toxicity monitoring and patient-reported outcomes. Cancer J 2011; 17 (04) 231-234
    CrossrefPubMedSearch in Google Scholar
  • 58 Morawski K, Dvorkis Y, Monsen CB. Predicting hospitalizations from electronic health record data. Am J Manag Care 2020; 26 (01) e7-e13
    CrossrefPubMedSearch in Google Scholar