Provider Response to a Venous Thromboembolism Risk Assessment and Prophylaxis Ordering Tool: Observational Study

Abstract

Objectives Our health system launched an initiative to regulate venous thromboembolism (VTE) risk assessment and prophylaxis with electronically embedded risk assessment models based on validated clinical prediction rules. Prior to system-wide implementation, usability testing was conducted on the VTE clinical decision support system (CDSS) to assess provider perceptions, facilitate adoption, and usage of the tool. The objective of this study was to conduct usability testing with end users on the CDSS' risk assessment model and prophylaxis ordering components.

Methods This laboratory usability testing study was conducted with 24 health care providers. Participants were given two case scenarios that mirrored real-world scenarios to assess likelihood of use and adoption. During each case scenario, participants engaged in a think-aloud session, verbalizing their decision-making process while interacting with the tool. Following each case scenario, participants completed the System Usability Scale (SUS) and a posttask interview. Participants' comments and interactions with the VTE CDSS were placed into coding categories and analyzed for generalizable themes by three independent coders.

Results Of the 24 participants, 50% were female and the mean age of all participants was 32.76 years. The average SUS across the different services lines was 72.39 (C grade). Each participant's comments were grouped into three overarching themes: functionality, visibility/navigation, and content. Comments included personalizing workflow for each service line, minimizing the number of clicks, clearly defining risk models, including background on risk scores, and providing treatment guidelines for order sets.

Conclusion An important step toward providing quality health care to patients at risk of developing a VTE event is providing user-friendly tools to providers. Following usability testing, our study revealed opportunities to positively impact provider behavior and acceptance. The rigor and breadth of this usability testing study and adoption of the optimizations should increase provider adoption and retention of the VTE CDSS.

Keywords

health information technology - electronic health record - clinical decision support systems - venous thromboembolism - usability testing

Human Subject Research Approval

All research activities were commenced after approval from the Institutional Review Board at Northwell Health. Written informed consent was obtained from all participants. All methods were performed in accordance with relevant guidelines and regulations.

Publication History

Received: 29 June 2022

Accepted: 26 October 2022

Article published online:
28 December 2022

• References

• 1 Anderson Jr. FA, Zayaruzny M, Heit JA, Fidan D, Cohen AT. Estimated annual numbers of US acute-care hospital patients at risk for venous thromboembolism. Am J Hematol 2007; 82 (09) 777-782
• 2 Cohen AT, Agnelli G, Anderson FA. et al; VTE Impact Assessment Group in Europe (VITAE). Venous thromboembolism (VTE) in Europe. The number of VTE events and associated morbidity and mortality. Thromb Haemost 2007; 98 (04) 756-764
• 3 Raskob GE, Angchaisuksiri P, Blanco AN. et al; ISTH Steering Committee for World Thrombosis Day. Venous thromboembolism: a call for risk assessment in all hospitalised patients. Thromb Haemost 2016; 116 (05) 777-779
• 4 Wells PS, Anderson DR, Rodger M. et al. Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis. N Engl J Med 2003; 349 (13) 1227-1235
• 5 Spyropoulos AC, McGinn T, Khorana AA. The use of weighted and scored risk assessment models for venous thromboembolism. Thromb Haemost 2012; 108 (06) 1072-1076
• 6 Kakkar AK, Davidson BL, Haas SK. Investigators Against Thromboembolism (INATE) Core Group. Compliance with recommended prophylaxis for venous thromboembolism: improving the use and rate of uptake of clinical practice guidelines. J Thromb Haemost 2004; 2 (02) 221-227
• 7 Kahn SR, Lim W, Dunn AS. et al. Prevention of VTE in nonsurgical patients. Antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 2012; 141 (2 suppl)
• 8 Preston H, Swan I, Davies L. et al. Improving VTE risk assessment and prophylaxis prescribing rate in medical patients: integrating risk assessment tool into the workflow. BMJ Open Qual 2020; 9 (02) e000903
• 9 Bahl V, Hu HM, Henke PK, Wakefield TW, Campbell Jr. DA, Caprini JA. A validation study of a retrospective venous thromboembolism risk scoring method. Ann Surg 2010; 251 (02) 344-350
• 10 Spyropoulos AC, Anderson Jr. FA, FitzGerald G. et al; IMPROVE Investigators. Predictive and associative models to identify hospitalized medical patients at risk for VTE. Chest 2011; 140 (03) 706-714
• 11 Darzi AJ, Repp AB, Spencer FA. et al. Risk-assessment models for VTE and bleeding in hospitalized medical patients: an overview of systematic reviews. Blood Adv 2020; 4 (19) 4929-4944
• 12 Blondon M, Spirk D, Kucher N. et al. Comparative performance of clinical risk assessment models for hospital-acquired venous thromboembolism in medical patients. Thromb Haemost 2018; 118 (01) 82-89
• 13 Bullock-Palmer RP, Weiss S, Hyman C. Innovative approaches to increase deep vein thrombosis prophylaxis rate resulting in a decrease in hospital-acquired deep vein thrombosis at a tertiary-care teaching hospital. J Hosp Med 2008; 3 (02) 148-155
• 14 Decousus H, Tapson VF, Bergmann JF. et al; IMPROVE Investigators. Factors at admission associated with bleeding risk in medical patients: findings from the IMPROVE investigators. Chest 2011; 139 (01) 69-79
• 15 Rosenberg DJ, Press A, Fishbein J. et al. External validation of the IMPROVE bleeding risk assessment model in medical patients. Thromb Haemost 2016; 116 (03) 530-536
• 16 Rosenberg D, Eichorn A, Alarcon M, McCullagh L, McGinn T, Spyropoulos AC. External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system. J Am Heart Assoc 2014; 3 (06) e001152
• 17 Hostler DC, Marx ES, Moores LK. et al. Validation of the IMPROVE bleeding risk score. Chest 2015;149(02):
• 18 Hostler DC, Marx ES, Moores LK. et al. Validation of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score. Chest 2016; 149 (02) 372-379
• 19 McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Evidence-Based Medicine Working Group. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. JAMA 2000; 284 (01) 79-84
• 20 Shuman AG, Hu HM, Pannucci CJ, Jackson CR, Bradford CR, Bahl V. Stratifying the risk of venous thromboembolism in otolaryngology. Otolaryngol Head Neck Surg 2012; 146 (05) 719-724
• 21 Pannucci CJ, Bailey SH, Dreszer G. et al. Validation of the Caprini risk assessment model in plastic and reconstructive surgery patients. J Am Coll Surg 2011; 212 (01) 105-112
• 22 Zhou HX, Peng LQ, Yan Y. et al. Validation of the Caprini risk assessment model in Chinese hospitalized patients with venous thromboembolism. Thromb Res 2012; 130 (05) 735-740
• 23 Obi AT, Pannucci CJ, Nackashi A. et al. Validation of the Caprini venous thromboembolism risk assessment model in critically ill surgical patients. JAMA Surg 2015; 150 (10) 941-948
• 24 Caprini JA. Thrombosis risk assessment as a guide to quality patient care. Dis Mon 2005; 51 (2-3): 70-78
• 25 Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough?. Hum Factors 1992; 34 (04)
• 26 Nielsen J. How Many Test Users in a Usability Study? Nielsen Norman Group. Accessed November 24, 2022, at: https://www.nngroup.com/articles/how-many-test-users/
• 27 Ortel TL, Arnold K, Beckman M. et al. Design and implementation of a comprehensive surveillance system for venous thromboembolism in a defined region using electronic and manual approaches. Appl Clin Inform 2019; 10 (03) 552-562
• 28 Pavon JM, Sloane RJ, Pieper CF. et al. Automated versus manual data extraction of the Padua prediction score for venous thromboembolism risk in hospitalized older adults. Appl Clin Inform 2018; 9 (03) 743-751
• 29 Brooke J. SUS -a quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL. eds. Usability Evaluation in Industry. United Kingdom: Taylor & Francis; 1996: 189-194
• 30 Vlachogianni P, Tselios N. Perceived usability evaluation of educational technology using the System Usability Scale (SUS): a systematic review. J Res Technol Educ 2022; 54 (03)
• 31 Measuring Usability with the System Usability Scale (SUS) – MeasuringU. Accessed September 22, 2022, at: https://measuringu.com/sus/
• 32 Kahn SR, Morrison DR, Cohen JM. et al. Interventions for implementation of thromboprophylaxis in hospitalized medical and surgical patients at risk for venous thromboembolism. Cochrane Database Syst Rev 2013; 2013 (07) CD008201
• 33 Borab ZM, Lanni MA, Tecce MG, Pannucci CJ, Fischer JP. Use of computerized clinical decision support systems to prevent venous thromboembolism in surgical patients a systematic review and meta-analysis. JAMA Surg 2017; 152 (07) 638-645
• 34 Haut ER, Lau BD, Kraenzlin FS. et al. Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch Surg 2012; 147 (10) 901-907
• 35 Nimeri AA, Bautista J, Ibrahim M. et al. Mandatory risk assessment reduces venous thromboembolism in bariatric surgery patients. Obes Surg 2018; 28 (02) 541-547
• 36 Marín-Peña O, Parvizi J, Restrepo C, Castel-Oñate A. International Consensus Meeting on Venous Thromboembolism (ICM-VTE) after orthopedic procedures, any change in our clinical practice? Rev Esp Cir Ortop Traumatol2022 (e-pub ahead of print)
• 37 Lau BD, Streiff MB, Pronovost PJ, Haider AH, Efron DT, Haut ER. Attending physician performance measure scores and resident physicians' ordering practices. JAMA Surg 2015; 150 (08) 813-814