Provider Response to a Venous Thromboembolism Risk Assessment and Prophylaxis Ordering Tool: Observational Study

Keywords

health information technology - electronic health record - clinical decision support systems - venous thromboembolism - usability testing

Background and Significance

Venous thromboembolism (VTE) presents with two types of blood clots: deep vein thrombosis, which is usually located in the legs, and pulmonary embolism, which is in the lungs. Studies show that annually up to 8 million hospitalized and acutely ill patients are at risk for developing VTE.[1] VTE, the third leading vascular diagnosis after heart attack and stroke, is considered the number one preventable cause of death in hospitalized individuals because of its origin in stasis and immobility. It is responsible for approximately 300,000 deaths a year in the United States.[2] At our health care facility, rates of VTE are comparable or higher than other hospital-acquired events including urinary catheter-associated infections, central line blood stream infections, Clostridioides difficile-associated diarrhea, pressure ulcers, and falls (based on internal quality metrics).

VTE identification has become an international concern, sparking concerted efforts to develop and implement guidelines to assess its risk in hospitalized individuals.[3] Appropriate diagnosis of VTE can minimize risk of thromboembolic complications while ensuring that patients without VTE avoid unnecessary treatment.[4] A risk assessment model (RAM) for VTE must identify individuals who meet the minimum risk threshold of developing VTE in the absence of prophylaxis; predict level of risk (including surgical or disease-specific) to allow for more tailored strategies in prophylaxis treatment; exclude patients without a positive risk/benefit ratio; and be evidence-based and methodologically transparent.[5] Issues of adoptability for VTE RAMs include lack of physician familiarity or agreement with guidelines, underestimation of VTE risk, concern over risk of bleeding, and the perception that the guidelines are resource intensive or difficult to implement in a practical fashion.[6]

Current guidelines are aimed at reducing VTE events using individual VTE risk assessment tools.[3] [7] These tools have been identified by the Centers for Medicare and Medicaid Services as essential components of VTE prevention. A previous study demonstrated that a paper-based VTE risk assessment tool improved the rate of appropriate patient assessment and prophylaxis implementation.[8] Northwell Health's Coalition of Leadership on Thrombosis Council, working in conjuncture with its clinical and system leadership, has targeted a system-wide implementation of two highly validated RAMs: Caprini risk score for surgical patients[9] and the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) risk score for medical patients.[10] Among several VTE RAMs, the IMPROVE and Caprini have been identified in systematic reviews as being externally validated,[11] further increasing reliability of both models. One study highlighted the IMPROVE RAM classified more patients as low risk (two-thirds of patients) compared with the Geneva RAM (one-third of patients).[12] Another study reported increased VTE prophylaxis rates and a reduction in hospital-acquired VTE rates with the use of the Caprini RAM.[13] The risk scores from these validated tools will be used to assist providers in ordering appropriate VTE prophylaxis for each patient's level of risk.

The IMPROVE risk score calculator has been developed into multiplatform applications for use at the patient bedside. Risk factors for IMPROVE include age, previous VTE event, history of thrombophilia, paralysis, malignancy, complete immobilization for one or more days, and intensive care unit/cardiac care unit stay.[14] [15] Having been validated in multiple settings,[15] [16] [17] [18] the model is graded[19] as Level II evidence—broadly validated. In addition, it is endorsed by the International Society on Thrombosis and Hemostasis.[3]

The Caprini RAM is the most widely used and well-validated risk prediction for postsurgical patients.[9] [20] [21] [22] Most recently, the tool showed a high degree of validation when used to assess VTE risk in critically ill surgical patients.[23] The RAM consists of risk factors with various scores ranging from 1 to 5. Risk factors with the highest degree of concern for assessing VTE include elective lower extremity arthroplasty, hip, pelvis or leg fracture, stroke (in less than 1 month), multiple trauma (in less than 1 month), and acute spinal cord injury (in less than 1 month). Additional risk factors include age, sex, type of surgery, recent events including major surgery, presence of venous disease or clotting disorder, and past medical history of inflammatory bowel disease, obesity, malignancy, acute myocardial infarction, or chronic obstructive pulmonary disease.[24]

Our study included a system-wide initiative to develop logic for implementation of the clinical decision support system (CDSS) based on the IMPROVE and Caprini risk assessment tools into the Allscripts Sunrise Clinical Manager electronic health record (EHR). A prototype was built which included the RAM and prophylaxis ordering tool within a playground EHR environment.

Objectives

The objective of this study was to conduct usability testing with end users on the CDSS' RAM and prophylaxis ordering components. Feedback was obtained to optimize the VTE CDSS prior to system-wide integration.

Methods

Laboratory usability testing was performed to assess ease of use and usefulness of the VTE RAM based on the IMPROVE ([Fig. 1]) and Caprini ([Fig. 2]). The VTE RAM presented either the IMPROVE or Caprini risk assessment tools based on if the patient was a medical or surgical admission. The prophylaxis ordering tool consisted of the patient type, score integrated from the assessment tool, prophylaxis ordering options, and patient care orders (pharmacological or nonpharmacological; [Fig. 1]). Prior to commencement of research activities, approval was obtained from Northwell Health's Institutional Review Board (IRB). The type of usability testing conducted was “think-aloud” testing. “Think-aloud” testing involves study participants interacting with the tool by completing case scenarios by verbalizing their thought processes while making clinical decisions. While using the tool, physicians were asked to ascertain critical patient risk factors for VTE risk and complete the appropriate order set for each case scenario. During usability testing all human–computer interactions, including audio and continuous screen capture, were recorded.

Fig. 1 International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) risk assessment model and prophylaxis ordering tool. VTE, venous thromboembolism.

Fig. 2 Caprini's risk assessment model.

Participants in the “think-aloud” testing phase included 24 different health care providers (i.e., attendings, residents, physician assistants, and nurse practitioners). The sample size of 24 participants was sufficient to elicit usability themes and is typical for usability studies.[25] [26] Participants were recruited from many clinical departments (obstetrics, gynecology, surgery, neurology, orthopedics, and medicine). Leadership in each hospital (medical directors, chief medical information officers, residency program directors) helped to identify volunteer participants from the respective groups. In addition, the research team presented the study during faculty meetings and resident conferences. Also, a list of participants was obtained through department administrators and an e-mail invitation was sent with study information to seek interest for study participation.

The usability testing sessions took place at two academic tertiary care hospitals in Northwell Health. The setting of the testing was a conference room within each tertiary care hospital. Three moderators led a total of 24 usability testing sessions over the course of 3 days. The first 2 days of testing were conducted at one academic tertiary care hospital, and the third day of testing was conducted at another academic tertiary care hospital. A conference room was reserved at each of the academic tertiary care hospitals to complete the usability testing sessions. Each participant was given two real-world case scenarios lasting 20 minutes each. During the allotted time the participants would review and assess the cases using the VTE CDSS. Each case scenario detailed either a medical or surgical admission. Case scenarios included descriptions for various patient characteristics: weight (normal or obese), renal function (normal, moderate, or severely reduced), and VTE risk (low, intermediate, or high; [Fig. 3]). See [Supplementary Appendix A] (available in the online version) for example of case scenario. The scenarios mirrored real-world cases to accurately assess the providers' likelihood to adopt and use the VTE CDSS. Following each scenario was a 10-minute debriefing session facilitated by one of the study moderators. The total time allotment for each session was 60 minutes.

Fig. 3 Case scenarios. Case scenarios were created for various patient cases based on real-life clinical examples. Starting from the left of the figure to the right, case scenarios reflected venous thromboembolism risk (low, intermediate, or high), weight (within normal limits [WNL] or obese), and renal function (normal, moderately reduced, or severely reduced).

At the start of each usability session, participants were asked to complete a prequestionnaire which included demographic information and a self-reported assessment of their EHR use ([Table 1]). After reading through the case scenarios pertinent to their respective service line, participants were directed to click the prototype which opened the VTE risk assessment module. Participants reviewed the auto-filled risk factors in the module, clicked or unclicked risk factors, clicked on the calculate button to generate a score, and selected the attestation button. After attesting to the score generated, the participant was directed to the order set which displayed VTE management according to risk level based on the score received. Reviewing the auto-filled components of the EHR is critical to ensure tool accuracy as evidenced by previous literature where careful review of the data collected electronically was required.[27] [28]

Table 1 Participant characteristics

Service lines	Participants	Number of cases
Gynecology	3	6
Obstetrics	4	8
Orthopaedics	4	8
Medicine	10	20
Neurology	2	4
Surgery	1	2
Total	24	48
Current level of clinical training	Percentage of participants
Attending/faculty	17 (4/24)
NP	8 (2/24)
PA	25 (6/24)
Residents	50 (12/24)
Variable	Mean	Standard deviation
Age	32.76	8.24
Gender	50% (female)
Years of practice	5.75	6.43
Years of EHR use	4.69	3.32
Likert scale (1 = not comfortable at all, 5 = extremely comfortable)
Comfort with EHR use	3.92	1.10
Comfort with best practice alert	3.00	1.32
Comfort with order sets	3.88	1.23

Abbreviations: EHR, electronic health record; NP, nurse practitioner; PA, physician assistant.

Participants' comments and interactions with the VTE CDSS during “think-aloud” sessions and posttask interviews were audio recorded and transcribed with annotations. Hypercam screen capture software was used to record all video and audio interactions with the VTE CDSS interface during the “think-aloud” usability testing phase. These files were saved to the protected health information hard drive as per IRB approved protocol. The qualitative content was analyzed through an inductive approach. There was a total of 1,080 minutes of videos that were analyzed; the average number of minutes per session was 45 minutes and there were 24 sessions. Three independent coders reviewed the transcripts, screen capture videos, and audio recordings. Usability themes were identified, and transcripts were coded based on the themes. Recommendations were drawn after all comments were reviewed. Discrepancies between coders were discussed until an agreement was reached.

Upon completion of each case scenario, participants also completed a System Usability Scale (SUS)[29] questionnaire developed by Brooke. It consists of 10 statements for which a respondent gives a subjective evaluation of a system's ease of use and acceptability. It is a reliable scale used worldwide with high validity and reliability.[30] The different grades are defined by 80 to 100 as A, 71 to 79 as B, 62 to 70 as C, 52 to 61 as D, and below 51 as F.[31] Scores were tabulated for each service line as well as a combined overall score representing the VTE CDSS's overall usability.

Results

The quantitative analysis of the usability testing included analyzing the individual SUS scores across service lines. The overall average score given by all the participants was a 72.39, which is equivalent to a C grade (good) on the usability scale. In [Table 2], SUS scores according to each service line is displayed with gynecology having the highest score and surgery having the lowest score.

Table 2 System Usability Scale (SUS) scores across service lines

Service line	Average SUS	Grade	Adjective ratings
Gynecology	82.92	B	Good
Obstetrics	76.25	C	Good
Orthopaedics	75.63	C	Good
Medicine	70.87	C	Ok
Neurology	61.25	D	Ok
Surgery	57.50	F	Ok
Overall	70.74	C	N/A

Qualitative analyses of the usability results from the transcripts were developed by first categorizing the comments and feedback made by participants while interacting with the VTE CDSS during the “think-aloud” usability testing, followed by the posttask interview. To facilitate qualitative analyses of providers' commentary, thematic analysis was conducted by three independent coders resulting in three broad themes being identified—functionality, visibility/navigation, and content (see [Table 3] for themes and definitions). To establish interrater reliability, one sample scenario was coded and analyzed with a single discrepancy noted within coding.

Table 3 Usability themes and definitions

Theme: functionality
Definition	Comments	Recommendations
• Commentary on the perceived effectiveness, efficiency and “ease- or lack-of-ease of use of the tool • Commentary on the extent the tool (and information provided by it) is perceived as helpful during clinical decision-making and care delivery • Commentary on the extent to which the text within the tool is comprehendible	• “Having the order set integrated into the tool is beneficial and acts a reminder. I do think it's good because some people forget to put in the heparin order after the C-section.” (Service Line: Obstetrics, Frequency of Similar Comments: 5) • “I think that if patients are flagged to having a deviation of prophylaxis, then I think it would be more helpful; it would make the clinician more confident and be aware of complex heparin dosages. Or if there's a contraindication” (Service Line: Gynecology, Frequency of Similar Comments: 6)	• Ensuring good workflow for each individual service line will enhance provider use of the tool • Include dialog boxes to explain the appropriate treatments offered within the tool
Theme: visibility/navigation
Definition	Comments	Recommendations
• Commentary on the extent an image, text, or message • is noticed or attended to in the tool • Commentary on the content of information provided by the tool • Commentary on the provider's ability to move through the system	• “In the order set, there were just a lot of boxes there. They were grayed out, but it makes it look a little cumbersome because there are a lot of things on the screen. Just by being there, it makes things a little more crowded” (Service Line: Medicine, Frequency of Similar Comments: 5) • “In the past, I've seen like if you click antepartum, it will get bold or louder and everything for it would follow. I think that's something to think about” (Service Line: Obstetrics, Frequency of Similar Comments: 7) • “Forgot about the agreement thing since it was out of the way. I had to click that I attest” (Service Line: Obstetrics, Frequency of Similar Comments: 3)	• Eliminating the choices as opposed to graying out choices would lessen distraction for clinicians • The risk assessment models need to be more clearly defined based on the service line • Attestation button could be placed closer to the submit button to ensure continuous flow
Theme: content
Definition	Comments	Recommendations
• Commentary on the content of the information provided by the tool	• “I'm glad that they put in the BMI and creatinine clearance populated” (Service Line: Medicine, Frequency of Similar Comments: 7)  • “For the dosage, it's a drop down menu. I didn't play around with the dose per say, but I would imagine it's a free text option. If not 5,000, maybe sometimes we use 7,500 for someone who is really obese, and you're worried about under-dosing” (Service Line: Medicine, Frequency of Similar Comments: 5)	• Auto-populating patient information and providing explanations on risk scores would improve interest and use of tool  • Providing appropriate evidence-based dosage recommendations will help clinicians make confident treatment decisions

In the functionality theme, participants frequently noted the need to personalize the workflow for each specific service line. In the visibility/navigation theme, participants remarked on the need to minimize boxes and clicks. In the content theme, participants expressed the need to clearly define the risk models, include background and explanation on risk scores, and to provide treatment guideline recommendations for each of the order sets. Following this analysis, the study team sought to identify theme-specific recommendations based on the qualitative feedback.

In addition to these different themes, participant comments were sorted into positive and negative comments. Examples of the positive comments were: “Helps to reduce errors in clinical decision making by providing appropriate recommendations based on risk score,” “Decreases inappropriate ordering of medications,” and “Provides reductions in dosage errors and potential harm to patients.” Negative comments about the tool included “found the tool design unnecessarily complex and the workflow cumbersome,” “far too many clicks,” “text is difficult to read,” and “tool is crowded, recommend reducing the number of boxes.”

Discussion

Previously, Kahn et al[32] reviewed the effects of interventions designed to increase implementation of thromboprophylaxis; statistically significant improvements in prophylaxis prescription were noted with alerts and multifaceted interventions. Similarly, a systematic review found computerized CDSSs increased the proportion of surgical patients who were prescribed VTE prophylaxis.[33] However, with the additional process of usability testing we identified significant barriers in the VTE risk assessment and prophylaxis ordering CDSS that can be overcome through implementation of improvements to functionality, visibility, navigation, and content.

Average SUS scores were higher in the following services lines: obstetrics, gynecology, orthopedics, and medicine while average SUS scores were lower in the following services lines: neurology and surgery ([Table 2]). This can be attributed to usability comments (under the functionality theme) such as “pretty simple tool and self-explanatory” made by service lines with higher SUS. Whereas usability comments such as “too much clicking” and “if it [tool] makes you not want to click things, then they should go away instead of them being grayed out” were noted in the service lines with lower SUS. Different usability characteristics were important for the services lines to utilize the tool efficiently.

The first set of recommendations based on the reactions of the providers was to improve functionality. This can be achieved by implementing specific workflows for each individual service line and increasing provider use of the VTE CDSS by minimizing the number of clicks. Also, ensuring the order set and risk assessment reference for one another, and including dialogue boxes to explain the appropriate treatment recommendations programmed as part of the workflows within the tool would help to improve the functionality overall.

The second set of recommendations was based on visibility and navigation of the VTE CDSS. Improvements included explaining the meaning and significance of icons (e.g., red star) and eliminating choices as opposed to graying out clinically inappropriate options based on the risk assessment scores ([Fig. 4]). An important improvement would be to clearly define the RAMs for each specific service line. For example, CDS implementation significantly improved compliance with VTE prophylaxis guidelines in hospitalized adult trauma patients.[34] In another study, decreased rates of VTE in bariatric surgery patients were noted when VTE management included completion of the Caprini RAM.[35]

Fig. 4 Areas identified for improvement in the venous thromboembolism (VTE) risk assessment and prophylaxis ordering tool. The red box toward the top of the figure highlights where the risk assessment score displays and the red box toward the bottom highlights the recommended orders pertaining to that risk assessment score. The recommendation elicited from testing was to remove choices as opposed to graying out choices which would lessen distraction for clinicians.

Additionally, the content must be elaborated by providing appropriate evidence-based dosage recommendations to help clinicians make confident clinical decisions. Presenting additional background and explanation on risk scores would improve interest in and use of the tool. Customizing the order set to reflect medications pertinent to specific service lines was a common theme as well as including additional patient types in the order set. After the patient has been assessed for VTE risk and no contraindication exists, an adaptive design may be helpful to allow providers to prescribe medications in no risk and low risk patients that pertain to certain procedures. Recent literature has cited aspirin and mechanical measures as the safest and most effective combination in many cases of orthopaedic surgery.[36]

Lesson learned included giving detailed training sessions to clinical providers, prior to tool implementation, with the following: (1) emphasis on utilizing the CDSS in both simple and complex cases of VTE management, (2) a workflow shift where orders were previously typed to instead clicking orders, and (3) guidance on appropriate prophylaxis treatment options based on risk assessment tools (Caprini and IMPROVE), as well as explanation of contraindications. We hope the CDSS for VTE assessment will serve as a successful example for future studies geared toward integrating evidenced-based assessment into the EHR. The next step will be to revise the CDSS based on the providers' feedback received and administer a postimplementation survey after the tool has been launched in the live EHR environment to assess for barriers in real-time use of the CDSS.

Limitations

The sample size in the study is typical for usability studies and a majority (80%) of the usability problems are detected with 4 to 5 participants and severe problems are noticed by the first participants.[25] For service lines such as gynecology, neurology, and surgery, there were less than 4 to 5 participants and results collected may not be generalizable but major concerns were elicited for iterative revisions. In addition, resident physicians at academic hospitals may be largely responsible for VTE prophylaxis prescription,[37] unfortunately comparison between prescribers was not analyzed in this study. Lastly, the Caprini RAM is more detailed than the IMPROVE as demonstrated in [Figs. 1] and [2]. This may have attributed to differences in feedback elicited and SUS in participants from the different service lines that completed Caprini versus IMPROVE. We did not collect data points to compare this hypothesis but support the notion that it should be an interesting topic for future research in VTE assessment models.

Conclusion

To increase evidence-based decision making, the VTE risk assessment and prophylaxis ordering CDSS should include integration into the clinical workflow for each service line. In addition, improvements should be made in functionality, visibility, navigation, and content to increase usability. Prior to system-wide integration, a detailed training session will need to include an overview of appropriate VTE medications based on risk assessment tools (Caprini and IMPROVE). Providing evidence-based guidelines as well as explanation of contraindications; emphasizing the importance of utilizing the tool in both simple and complex cases; and highlighting the changes in workflow also will be required for successful implementation. The CDSS design should include a link that directs providers to a document listing further resources on evidence-based guidelines and list of contraindications.

Clinical Relevance Statement

To increase evidence-based decision making, the VTE CDSS must include integration into the clinical workflow for each service line. Providing evidence-based guidelines as well as explanation of contraindications; emphasizing the importance of utilizing the tool in both simple and complex cases; and highlighting the changes in workflow also will be required for successful implementation and patient care for evaluation of VTE.

Multiple-Choice Questions

1. What is the third leading vascular diagnosis?

   • Heart attack

   • Venous thromboembolism

   • Stroke

   • Pulmonary hypertension

   Correct Answer: The correct answer is option b.

   Explanation of answer: in the “Background/Significance” section of the article, it is cited that venous thromboembolism (VTE) is the third leading vascular diagnosis after heart attack and stroke.

2. How can feedback obtained from usability testing be organized?

   • Orders

   • Verdicts

   • Themes

   • Diagnoses

   Correct Answer: The correct answer is option c.

   Explanation of answer: in the “Results” section of the article, it is noted that to facilitate qualitative analyses of providers' commentary, thematic analysis was conducted by three independent coders resulting in three broad themes.

• References

• 1 Anderson Jr. FA, Zayaruzny M, Heit JA, Fidan D, Cohen AT. Estimated annual numbers of US acute-care hospital patients at risk for venous thromboembolism. Am J Hematol 2007; 82 (09) 777-782
• 2 Cohen AT, Agnelli G, Anderson FA. et al; VTE Impact Assessment Group in Europe (VITAE). Venous thromboembolism (VTE) in Europe. The number of VTE events and associated morbidity and mortality. Thromb Haemost 2007; 98 (04) 756-764
• 3 Raskob GE, Angchaisuksiri P, Blanco AN. et al; ISTH Steering Committee for World Thrombosis Day. Venous thromboembolism: a call for risk assessment in all hospitalised patients. Thromb Haemost 2016; 116 (05) 777-779
• 4 Wells PS, Anderson DR, Rodger M. et al. Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis. N Engl J Med 2003; 349 (13) 1227-1235
• 5 Spyropoulos AC, McGinn T, Khorana AA. The use of weighted and scored risk assessment models for venous thromboembolism. Thromb Haemost 2012; 108 (06) 1072-1076
• 6 Kakkar AK, Davidson BL, Haas SK. Investigators Against Thromboembolism (INATE) Core Group. Compliance with recommended prophylaxis for venous thromboembolism: improving the use and rate of uptake of clinical practice guidelines. J Thromb Haemost 2004; 2 (02) 221-227
• 7 Kahn SR, Lim W, Dunn AS. et al. Prevention of VTE in nonsurgical patients. Antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 2012; 141 (2 suppl)
• 8 Preston H, Swan I, Davies L. et al. Improving VTE risk assessment and prophylaxis prescribing rate in medical patients: integrating risk assessment tool into the workflow. BMJ Open Qual 2020; 9 (02) e000903
• 9 Bahl V, Hu HM, Henke PK, Wakefield TW, Campbell Jr. DA, Caprini JA. A validation study of a retrospective venous thromboembolism risk scoring method. Ann Surg 2010; 251 (02) 344-350
• 10 Spyropoulos AC, Anderson Jr. FA, FitzGerald G. et al; IMPROVE Investigators. Predictive and associative models to identify hospitalized medical patients at risk for VTE. Chest 2011; 140 (03) 706-714
• 11 Darzi AJ, Repp AB, Spencer FA. et al. Risk-assessment models for VTE and bleeding in hospitalized medical patients: an overview of systematic reviews. Blood Adv 2020; 4 (19) 4929-4944
• 12 Blondon M, Spirk D, Kucher N. et al. Comparative performance of clinical risk assessment models for hospital-acquired venous thromboembolism in medical patients. Thromb Haemost 2018; 118 (01) 82-89
• 13 Bullock-Palmer RP, Weiss S, Hyman C. Innovative approaches to increase deep vein thrombosis prophylaxis rate resulting in a decrease in hospital-acquired deep vein thrombosis at a tertiary-care teaching hospital. J Hosp Med 2008; 3 (02) 148-155
• 14 Decousus H, Tapson VF, Bergmann JF. et al; IMPROVE Investigators. Factors at admission associated with bleeding risk in medical patients: findings from the IMPROVE investigators. Chest 2011; 139 (01) 69-79
• 15 Rosenberg DJ, Press A, Fishbein J. et al. External validation of the IMPROVE bleeding risk assessment model in medical patients. Thromb Haemost 2016; 116 (03) 530-536
• 16 Rosenberg D, Eichorn A, Alarcon M, McCullagh L, McGinn T, Spyropoulos AC. External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system. J Am Heart Assoc 2014; 3 (06) e001152
• 17 Hostler DC, Marx ES, Moores LK. et al. Validation of the IMPROVE bleeding risk score. Chest 2015;149(02):
• 18 Hostler DC, Marx ES, Moores LK. et al. Validation of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score. Chest 2016; 149 (02) 372-379
• 19 McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Evidence-Based Medicine Working Group. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. JAMA 2000; 284 (01) 79-84
• 20 Shuman AG, Hu HM, Pannucci CJ, Jackson CR, Bradford CR, Bahl V. Stratifying the risk of venous thromboembolism in otolaryngology. Otolaryngol Head Neck Surg 2012; 146 (05) 719-724
• 21 Pannucci CJ, Bailey SH, Dreszer G. et al. Validation of the Caprini risk assessment model in plastic and reconstructive surgery patients. J Am Coll Surg 2011; 212 (01) 105-112
• 22 Zhou HX, Peng LQ, Yan Y. et al. Validation of the Caprini risk assessment model in Chinese hospitalized patients with venous thromboembolism. Thromb Res 2012; 130 (05) 735-740
• 23 Obi AT, Pannucci CJ, Nackashi A. et al. Validation of the Caprini venous thromboembolism risk assessment model in critically ill surgical patients. JAMA Surg 2015; 150 (10) 941-948
• 24 Caprini JA. Thrombosis risk assessment as a guide to quality patient care. Dis Mon 2005; 51 (2-3): 70-78
• 25 Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough?. Hum Factors 1992; 34 (04)
• 26 Nielsen J. How Many Test Users in a Usability Study? Nielsen Norman Group. Accessed November 24, 2022, at: https://www.nngroup.com/articles/how-many-test-users/
• 27 Ortel TL, Arnold K, Beckman M. et al. Design and implementation of a comprehensive surveillance system for venous thromboembolism in a defined region using electronic and manual approaches. Appl Clin Inform 2019; 10 (03) 552-562
• 28 Pavon JM, Sloane RJ, Pieper CF. et al. Automated versus manual data extraction of the Padua prediction score for venous thromboembolism risk in hospitalized older adults. Appl Clin Inform 2018; 9 (03) 743-751
• 29 Brooke J. SUS -a quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL. eds. Usability Evaluation in Industry. United Kingdom: Taylor & Francis; 1996: 189-194
• 30 Vlachogianni P, Tselios N. Perceived usability evaluation of educational technology using the System Usability Scale (SUS): a systematic review. J Res Technol Educ 2022; 54 (03)
• 31 Measuring Usability with the System Usability Scale (SUS) – MeasuringU. Accessed September 22, 2022, at: https://measuringu.com/sus/
• 32 Kahn SR, Morrison DR, Cohen JM. et al. Interventions for implementation of thromboprophylaxis in hospitalized medical and surgical patients at risk for venous thromboembolism. Cochrane Database Syst Rev 2013; 2013 (07) CD008201
• 33 Borab ZM, Lanni MA, Tecce MG, Pannucci CJ, Fischer JP. Use of computerized clinical decision support systems to prevent venous thromboembolism in surgical patients a systematic review and meta-analysis. JAMA Surg 2017; 152 (07) 638-645
• 34 Haut ER, Lau BD, Kraenzlin FS. et al. Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch Surg 2012; 147 (10) 901-907
• 35 Nimeri AA, Bautista J, Ibrahim M. et al. Mandatory risk assessment reduces venous thromboembolism in bariatric surgery patients. Obes Surg 2018; 28 (02) 541-547
• 36 Marín-Peña O, Parvizi J, Restrepo C, Castel-Oñate A. International Consensus Meeting on Venous Thromboembolism (ICM-VTE) after orthopedic procedures, any change in our clinical practice? Rev Esp Cir Ortop Traumatol2022 (e-pub ahead of print)
• 37 Lau BD, Streiff MB, Pronovost PJ, Haider AH, Efron DT, Haut ER. Attending physician performance measure scores and resident physicians' ordering practices. JAMA Surg 2015; 150 (08) 813-814