Improving Evaluation to Address the Unintended Consequences of Health Information Technology

Summary

Background and objectives: With growing use of IT by healthcare professionals and patients, the opportunity for any unintended effects of technology to disrupt care health processes and outcomes is intensified. The objectives of this position paper by the IMIA Working Group (WG) on Technology Assessment and Quality Development are to highlight how our ongoing initiatives to enhance evaluation are also addressing the unintended consequences of health IT.

Methods: Review of WG initiatives Results: We argue that an evidence-based approach underpinned by rigorous evaluation is fundamental to the safe and effective use of IT, and for detecting and addressing its unintended consequences in a timely manner. We provide an overview of our ongoing initiatives to strengthen study design, execution and reporting by using evaluation frameworks and guidelines which can enable better characterization and monitoring of unintended consequences, including the Good Evaluation Practice Guideline in Health Informatics (GEP-HI) and the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI). Indicators to benchmark the adoption and impact of IT can similarly be used to monitor unintended effects on healthcare structures, processes and outcome. We have also developed EvalDB, a web-based database of evaluation studies to promulgate evidence about unintended effects and are developing the content for courses to improve training in health IT evaluation.

Conclusion: Evaluation is an essential ingredient for the effective use of IT to improve healthcare quality and patient safety. WG resources and skills development initiatives can facilitate a pro-active and evidence-based approach to detecting and addressing the unintended effects of health IT.

Keywords

Medical informatics - evaluation studies - evidence-based medicine - information systems - patient safety

 

• References

• 1 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-23.
• 2 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-12.
• 3 Weiner JP, Kfuri T, Chan K, Fowles JB. “e-Iatro-genesis”: the most critical unintended consequence of CPOE and other HIT. J Am Med Inform Assoc 2007; 14 (03) 387-8.
• 4 Rahadhan P, Poon SK, Land L. Understanding unintended consequences for EMR: a literature review. Stud Health Technol Inform 2012; 178: 192-8.
• 5 Kuperman GJ, McGowan JJ. Potential unintended consequences of health information exchange. J Gen Intern Med 2013; 28 (12) 1663-6.
• 6 Campbell EM, Sittig DF, Guappone KP, Dykstra RH, Ash JS. Overdependence on technology: an unintended adverse consequence of computerized provider order entry. AMIA Annu Symp Proc 2007; 94-8.
• 7 Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc 2011; 18 (01) 82-90.
• 8 Coiera E, Aarts J, Kulikowski C. The dangerous decade. J Am Med Inform Assoc 2012; 19 (01) 2-5.
• 9 ECRI Institute’s 2016 Top 10 Health Technology Hazards. ECRI Institute; USA:
• 10 Magrabi F, Baker M, Sinha I, Ong MS, Harrison S, Kidd MR. et al. Clinical safety of England’s national programme for IT: A retrospective analysis of all reported safety events 2005 to 2011. Int J Med Inform 2015; 84 (03) 198-206.
• 11 Magrabi F, Ong MS, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012; 19 (01) 45-53.
• 12 Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: The National Academies Press: Institute of Medicine; 2012
• 13 Kuziemsky CE. Review of Social and Organizational Issues in Health Information Technology. Healthc Inform Res 2015; 21 (03) 152-60.
• 14 Ammenwerth E, Shaw NT. Bad health informatics can kill--is evaluation the answer?. Methods Inf Med 2005; 44 (01) 1-3.
• 15 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-12.
• 16 HI B. Bad Health Informatics can Kill. [Available from: http://iig.umit.at/efmi.
• 17 Rigby M, Ammenwerth E, Beuscart-Zephir MC, Brender J, Hypponen H, Melia S. et al. Evidence Based Health Informatics: 10 Years of Efforts to Promote the Principle. Joint Contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Medical Inform 2013; 8: 34-46.
• 18 Ammenwerth E, Rigby M. editors. Evidence-Based Health Informatics - Promoting Safety and Efficiency through Scientific Methods and Ethical Policy. Amsterdam: IOS Press; 2016
• 19 Wall R. Computer Rx: more harm than good?. Journal Med Syst 1991; 15 5-6 321-34.
• 20 Nykänen P, Kaipio J. Quality of health IT evaluations. In: Ammerwerth E, Rigby M. editors. Evidence-based health informatics. Stud Health Technol Inform. 2016. 222: 291-303.
• 21 Yusof MM, Papazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform 2008; 77 (06) 377-85.
• 22 Basden A. Philosophical frameworks for understanding information systems. Hershey, PA, USA: IGI Global; 2008
• 23 House ER. Origins of the Ideas in Evaluating with Validity. New Directions for Evaluation. 2014; 2014 (142) 9-15.
• 24 Friedman CF, Wyatt J. Evaluation methods in medical informatics, 2 Ed. New York: Springer Verlag; 2006
• 25 Delone WH, McLean ER. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. J Manage Inf Syst 2003; 19 (04) 9-30.
• 26 Venkatesh V, Morris M, Davis G, Davis F. User acceptance of information technology: toward a unified view. MIS Quarterly 2003; 27 (03) 425-78.
• 27 Ammenwerth E, Iller C, Mahler C. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Med Inform Decis Mak 2006; 6: 3.
• 28 Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007; 76 (01) 21-7.
• 29 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care - an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-9.
• 30 Cusack C, Byrne C, Hook J, McGowan J, Poon E, Zafar A. Health Information Technology Evaluation Toolkit: 2009 Update. Rockville, MD: Agency for Healthcare Research and Quality; 2009. [updated June. Available from: https://healthit.ahrq.gov/sites/default/files/docs/page/health-information-technology-evaluation-tool-kit-2009-update.pdf.
• 31 Pawson R, Manzano-Santaella A. A realist diagnostic workshop. Evaluation 2012; 18 (02) 176-91.
• 32 De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C. et al. Theory of Change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials 2014; 15: 267.
• 33 Taplin DH, Clark H, Collins E, Colby DC. Theory of Change technical papers New York: ActKnowl-edge. 2013 [updated 31 July 2015. Available from: http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/ToC-Tech-Papers.pdf.
• 34 Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC. et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform 2011; 80 (12) 815-27.
• 35 Brender J. Handbook of evaluation methods for health informatics. USA: Elsevier Academic Press; 2006
• 36 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-31.
• 37 Hyppönen H, Kangas M, Reponen J, Nøhr C, Villumsen S, Koch S. et al. Nordic eHealth Benchmarking: Status 2014: Nordisk Ministerråd. 2015
• 38 Lääveri T, Vainiomäki S, Kaipio J, Reponen J, Vänskä J, Lehtovirta M. et al. Yksityissektorin potilastietojärjestelmät arvioitu 2014 [Evaluation of private sector patient information systems 2014]. Finnish Medical Journal 2015; 23: 1660-7.
• 39 Hyppönen H, Hämäläinen P, Reponen J. editors. E-health and e-welfare of Finland. Check point 2015. Helsinki, Finland: National Institute for Health and Welfare; 2015
• 40 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78 (01) 1-9.
• 41 Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Appl Clin Inform 2013; 4 (03) 331-58.
• 42 de Keizer NF, Talmon J, Ammenwerth E, Brender J, Rigby M, Nykanen P. Systematic prioritization of the STARE-HI reporting items. An application to short conference papers on health informatics evaluation. Methods Inf Med 2012; 51 (02) 104-11.
• 43 EQUATOR.. The Equator Network: Enhancing the Quality and Transparency of Health Research. 2014 [Available from: http://www.equator-network.org.
• 44 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inf Med 2005; 44 (01) 44-56.
• 45 Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W. et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med 2010; 49 (02) 105-20.
• 46 Eval EW. Curricula of health IT evaluation courses. 2014 [Available from:. https://iig.umit.at/efmi/curricula.htm.
• 47 Balogh E, Miller B, Ball J. Improving Diagnosis in Health Care. Institute of Medicine. National Academies Press; 2015
• 48 Poon EG, Cusack CM, McGowan JJ. Evaluating healthcare information technology outside of academia: observations from the national resource center for healthcare information technology at the Agency for Healthcare Research and Quality. J Am Med Inform Assoc 2009; 16 (05) 631-6.
• 49 Rigby M, Georgiou A, Hypponen H, Ammenwerth E, de Keizer N, Magrabi F. et al. Patient Portals as a Means of Information and Communication Technology Support to Patient- Centric Care Coordination - the Missing Evidence and the Challenges of Evaluation. A joint contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Med Inform 2015; 10 (01) 148-59.
• 50 Ijzerman MJ, Steuten LM. Early assessment of medical technologies to inform product development and market access: a review of methods and applications. Appl Health Econ Health Policy 2011; 9 (05) 331-47.