Usability Evaluation of a Clinical Decision Support System for Geriatric ED Pain Treatment

Summary

Background

Older adults are at risk for inadequate emergency department (ED) pain care. Unrelieved acute pain is associated with poor outcomes. Clinical decision support systems (CDSS) hold promise to improve patient care, but CDSS quality varies widely, particularly when usability evaluation is not employed.

Objective

To conduct an iterative usability and redesign process of a novel geriatric abdominal pain care CDSS. We hypothesized this process would result in the creation of more usable and favorable pain care interventions.

Methods

Thirteen emergency physicians familiar with the Electronic Health Record (EHR) in use at the study site were recruited. Over a 10-week period, 17 1-hour usability test sessions were conducted across 3 rounds of testing. Participants were given 3 patient scenarios and provided simulated clinical care using the EHR, while interacting with the CDSS interventions. Quantitative System Usability Scores (SUS), favorability scores and qualitative narrative feedback were collected for each session. Using a multi-step review process by an interdisciplinary team, positive and negative usability issues in effectiveness, efficiency, and satisfaction were considered, prioritized and incorporated in the iterative redesign process of the CDSS. Video analysis was used to determine the appropriateness of the CDS appearances during simulated clinical care.

Results

Over the 3 rounds of usability evaluations and subsequent redesign processes, mean SUS progressively improved from 74.8 to 81.2 to 88.9; mean favorability scores improved from 3.23 to 4.29 (1 worst, 5 best). Video analysis revealed that, in the course of the iterative redesign processes, rates of physicians’ acknowledgment of CDS interventions increased, however most rates of desired actions by physicians (such as more frequent pain score updates) decreased.

Conclusion

The iterative usability redesign process was instrumental in improving the usability of the CDSS; if implemented in practice, it could improve geriatric pain care. The usability evaluation process led to improved acknowledgement and favorability. Incorporating usability testing when designing CDSS interventions for studies may be effective to enhance clinician use.

Keywords

Clinical decision support systems - decision support techniques - aged - acute pain - medical record systems - computerized - man-machine systems

• References

• 1 Hwang U, Richardson LD, Harris B, Morrison RS. The quality of emergency department pain care for older adult patients. J Am Geriatr Soc 2010; 58: 2122-2128.
• 2 Platts-Mills TF, Esserman DA, Brown L, Bortsov AV, Sloane PD, McLean SA. Older US emergency department patients are less likely to receive pain medication than younger patients: results from a national survey. Ann Emerg Med 2012; 60: 199-206.
• 3 Fosnocht D, Swanson E, Barton E. Changing attitudes about pain and pain control in emergency medicine. Emerg Med Clin N Amer 2005; 23: 297-306.
• 4 Morrison RS, Magaziner J, McLaughlin MA, Orosz G, Silberzweig SB, Koval KJ, Siu AL. The impact of post-operative pain on outcomes following hip fracture. Pain 2003; 103 (03) 303-311.
• 5 Dworkin R. Which individuals with acute pain are most likely to develop a chronic pain syndrome?. Pain Forum 1997; 06: 127-136.
• 6 Duggleby W, Lander J. Cognitive status and postoperative pain: older adults. J Pain Sympt Manage 1994; 09: 19-27.
• 7 Lynch E, Lazor M, Gelis J, Orav J, Goldman l, Marcantonio E. The impact of postoperative pain on the development of postoperative delirium. Anesth Analg 1998; 86: 781-785.
• 8 The AGS guideline on the management of persistent pain in older persons. 2002 http://www.americangeriatrics.org/products/positionpapers/persistent_pain_guide.shtml Accessed January 16, 2007.
• 9 American Geriatrics Society Panel on the Pharmacological Management of Persistent Pain in Older. PersonsPharmacological management of persistent pain in older persons. Pain Med 2009; 10: 1062-1083.
• 10 Jenkins HJ, Hancock MJ, French SD, Maher CG, Engel RM, Magnussen JS. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ 2015; 187: 401-408.
• 11 Gupta A, Ip IK, Raja AS. Effect of clinical decision support on documented guideline adherence for head CT in emergency department patients with mild traumatic brain injury. J Am Med Inform Assoc 21 2014; e347-e351.
• 12 Britton DJ, Bloch RB, Strout TD. Impact of a computerized order set on adherence to Centers for Disease Control guidelines for the treatment of victims of sexual assault. J Emerg Med 2013; 44: 528-535.
• 13 Dashboard.HealthIT.gov [Internet]. Washington, D.C: Health IT Adoption & Use Dashboard from U.S. HHS/ONC; [updated 2015 Apr; cited 2015 Aug 15]. Available from: http://dashboard.healthit.gov/HITAdoption/?view=0
• 14 Institute of Medicine (U.S.), and Committee on Patient Safety and Health Information Technology. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, D.C.: National Academies Press; 2012
• 15 Romano MJ, Stafford RS. Electronic health records and clinical decision support systems: impact on national ambulatory care quality. Arch Intern Med 2015; 171 (10) 897.
• 16 Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med 2014; 160 (01) 48-54.
• 17 Chaudhry B, Wang J, Maglione M, Monica W, Roth E, Morton SC, Shekelle PG. Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752.
• 18 Strom BL, Schinnar R, Bilker W, Hennessy S, Leonard CE, Pifer E. Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID - warfarin co-prescribing as a test case. J Am Med Inform Assoc 2010; 17 (04) 411-415.
• 19 Strom BL, Schinnar R, Aberra F, Bilker W, Hennessy S, Leonard CE, Pifer E. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010; 170 (17) 1578-1583.
• 20 Tamblyn R, Reidel K, Huang A, Taylor L, Winslade N, Bartlett G, Grad R, Jacques A, Dawes M, Larochelle P, Pinsonneault A. Increasing the detection and response to adherence problems with cardiovascular medication in primary care through computerized drug management systems: a randomized controlled trial. Med Decis Making 2010; 30 (02) 176-188.
• 21 Gurwitz JH, Field TS, Rochon P, Judge J, Harold LR, Bell CM, Lee M, White K, LaPrino J, Erramuspe-Mainard J, DeFlorio M, Gavendo L, Baril JL, Reed G, Bates DW. Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc 2008; 56 (12) 2225-2233.
• 22 Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C, Khorassani R, Tanasijevic M, Middletone B. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10: 523-530.
• 23 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330: 765.
• 24 Gold M, Hossain M, Charles DR, Furukawa MF. Evolving vendor market for HITECH-certified ambulatory EHR products. Am J Manag Care 2013; 11: 353-361.
• 25 Nielson J. Usability 101: Introduction to Usability [Internet]. Freemont: Nielsen Norman Group. 2012 Jan [cited 2015 Aug 15]. Available from: http://www.nngroup.com/articles/usability-101-introduction-to-us ability/
• 26 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45: 1202-1216.
• 27 Hwang U, Belland LK, Handel DA, Yadav K, Heard K, Rivera-Reyes L, Eisenberg A, Noble MJ, Mekala S, Valley M, Wingel G, Todd KH, Morrison RS. Is all pain is treated equally? A multicenter evaluation of acute pain care by age. Pain 2014; 155: 2568-2574.
• 28 Techsmith. Morae – usability testing software. http://www.techsmith.com/morae.asp Accessed March 27, 2011.
• 29 Brooke J. SUS: a ‘quick and dirty’ usability scale. In Jordan PW, Thomas B, Weerdmeester BA, McClelland IL. editors Usability Evaluation in Industry. London: Taylor and Francis; 189-194.
• 30 Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean; adding an Aadjective rating scale. J Usability Studies 2009; 04: 114-123.
• 31 McCoy AB, Thomas EH, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 (02) 195-202.
• 32 Ong M-S, Coiera E. Evaluating the effectiveness of clinical alerts: a signal detection approach. AMIA Annu Symp Proc 2011; 1036-1044.
• 33 Lewis JR, Sauro J. The factor structure of the system usability scale. In HCI International 2009: Proceedings of the 1st International Conference on Human Centered Design. 2009. San Diego: Springer-Verlag; 94-103.
• 34 Emani S, Ting DY, Healey M, Lipsitz SR, Karson AS, Einbinder JS, Leinen L, Suric V, Bates DW. Physician beliefs about the impact of meaningful use of the EHR: a cross-sectional study. Apple Clin Infom 2014; 27: 789-801.
• 35 Bukata R. EHRs are Inevitable, Yet Studies Still Pose Serious Questions. Emer Phys Monthly. 2014;10:4. Available from: http://epmonthly.com/article/ehrs-are-inevitable-yet-studies-still-pose-serious-questions/
• 36 Ward MJ, Landman AB, Case K, Berthelot J, Pilgrim RL, Pines JM. The effect of electronic health record implementation on community emergency department operational measures of performance. Ann Emerg Med 2014; 63: 723-730.