Yearbook of Medical Informatics, Table of Contents Yearb Med Inform 2016; 25(01): 61-69DOI: 10.15265/IY-2016-013 IMIA and Schattauer GmbH Georg Thieme Verlag KG Stuttgart Improving Evaluation to Address the Unintended Consequences of Health Information Technology A Position Paper from the Working Group on Technology Assessment & Quality Development F. Magrabi 1 Australian Institute of Health Innovation, Macquarie University, Sydney, Australia , E. Ammenwerth 2 UMIT, University for Health Sciences, Medical Informatics and Technology, Hall in Tyrol, Austria , H. Hyppönen 3 National Institute for Health and Welfare, Information Department, Helsinki, Finland , N. de Keizer 4 Academic Medical Center, Department of Medical Informatics, Amsterdam, The Netherlands , P. Nykänen 5 University of Tampere, School of Information Sciences, eHealth Research, Finland , M. Rigby 6 Keele University, School of Social Science and Public Policy, Keele, United Kingdom , P. Scott 7 School of Computing, University of Portsmouth, Portsmouth, United Kingdom , J. Talmon 8 Hi-Way, Sint Odiliënberg, The Netherlands , A. Georgiou 1 Australian Institute of Health Innovation, Macquarie University, Sydney, Australia › Author Affiliations Recommend Article Abstract Full Text PDF Download Keywords KeywordsMedical informatics - evaluation studies - evidence-based medicine - information systems - patient safety References References 1 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-23. 2 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-12. 3 Weiner JP, Kfuri T, Chan K, Fowles JB. “e-Iatro-genesis”: the most critical unintended consequence of CPOE and other HIT. J Am Med Inform Assoc 2007; 14 (03) 387-8. 4 Rahadhan P, Poon SK, Land L. Understanding unintended consequences for EMR: a literature review. Stud Health Technol Inform 2012; 178: 192-8. 5 Kuperman GJ, McGowan JJ. Potential unintended consequences of health information exchange. J Gen Intern Med 2013; 28 (12) 1663-6. 6 Campbell EM, Sittig DF, Guappone KP, Dykstra RH, Ash JS. Overdependence on technology: an unintended adverse consequence of computerized provider order entry. AMIA Annu Symp Proc 2007; 94-8. 7 Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc 2011; 18 (01) 82-90. 8 Coiera E, Aarts J, Kulikowski C. The dangerous decade. J Am Med Inform Assoc 2012; 19 (01) 2-5. 9 ECRI Institute’s 2016 Top 10 Health Technology Hazards. ECRI Institute; USA: 10 Magrabi F, Baker M, Sinha I, Ong MS, Harrison S, Kidd MR. et al. Clinical safety of England’s national programme for IT: A retrospective analysis of all reported safety events 2005 to 2011. Int J Med Inform 2015; 84 (03) 198-206. 11 Magrabi F, Ong MS, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012; 19 (01) 45-53. 12 Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: The National Academies Press: Institute of Medicine; 2012 13 Kuziemsky CE. Review of Social and Organizational Issues in Health Information Technology. Healthc Inform Res 2015; 21 (03) 152-60. 14 Ammenwerth E, Shaw NT. Bad health informatics can kill--is evaluation the answer?. Methods Inf Med 2005; 44 (01) 1-3. 15 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-12. 16 HI B. Bad Health Informatics can Kill. [Available from: http://iig.umit.at/efmi. 17 Rigby M, Ammenwerth E, Beuscart-Zephir MC, Brender J, Hypponen H, Melia S. et al. Evidence Based Health Informatics: 10 Years of Efforts to Promote the Principle. Joint Contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Medical Inform 2013; 8: 34-46. 18 Ammenwerth E, Rigby M. editors. Evidence-Based Health Informatics - Promoting Safety and Efficiency through Scientific Methods and Ethical Policy. Amsterdam: IOS Press; 2016 19 Wall R. Computer Rx: more harm than good?. Journal Med Syst 1991; 15 5-6 321-34. 20 Nykänen P, Kaipio J. Quality of health IT evaluations. In: Ammerwerth E, Rigby M. editors. Evidence-based health informatics. Stud Health Technol Inform. 2016. 222: 291-303. 21 Yusof MM, Papazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform 2008; 77 (06) 377-85. 22 Basden A. Philosophical frameworks for understanding information systems. Hershey, PA, USA: IGI Global; 2008 23 House ER. Origins of the Ideas in Evaluating with Validity. New Directions for Evaluation. 2014; 2014 (142) 9-15. 24 Friedman CF, Wyatt J. Evaluation methods in medical informatics, 2 Ed. New York: Springer Verlag; 2006 25 Delone WH, McLean ER. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. J Manage Inf Syst 2003; 19 (04) 9-30. 26 Venkatesh V, Morris M, Davis G, Davis F. User acceptance of information technology: toward a unified view. MIS Quarterly 2003; 27 (03) 425-78. 27 Ammenwerth E, Iller C, Mahler C. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Med Inform Decis Mak 2006; 6: 3. 28 Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007; 76 (01) 21-7. 29 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care - an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-9. 30 Cusack C, Byrne C, Hook J, McGowan J, Poon E, Zafar A. Health Information Technology Evaluation Toolkit: 2009 Update. Rockville, MD: Agency for Healthcare Research and Quality; 2009. [updated June. Available from: https://healthit.ahrq.gov/sites/default/files/docs/page/health-information-technology-evaluation-tool-kit-2009-update.pdf. 31 Pawson R, Manzano-Santaella A. A realist diagnostic workshop. Evaluation 2012; 18 (02) 176-91. 32 De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C. et al. Theory of Change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials 2014; 15: 267. 33 Taplin DH, Clark H, Collins E, Colby DC. Theory of Change technical papers New York: ActKnowl-edge. 2013 [updated 31 July 2015. Available from: http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/ToC-Tech-Papers.pdf. 34 Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC. et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform 2011; 80 (12) 815-27. 35 Brender J. Handbook of evaluation methods for health informatics. USA: Elsevier Academic Press; 2006 36 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-31. 37 Hyppönen H, Kangas M, Reponen J, Nøhr C, Villumsen S, Koch S. et al. Nordic eHealth Benchmarking: Status 2014: Nordisk Ministerråd. 2015 38 Lääveri T, Vainiomäki S, Kaipio J, Reponen J, Vänskä J, Lehtovirta M. et al. Yksityissektorin potilastietojärjestelmät arvioitu 2014 [Evaluation of private sector patient information systems 2014]. Finnish Medical Journal 2015; 23: 1660-7. 39 Hyppönen H, Hämäläinen P, Reponen J. editors. E-health and e-welfare of Finland. Check point 2015. Helsinki, Finland: National Institute for Health and Welfare; 2015 40 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78 (01) 1-9. 41 Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Appl Clin Inform 2013; 4 (03) 331-58. 42 de Keizer NF, Talmon J, Ammenwerth E, Brender J, Rigby M, Nykanen P. Systematic prioritization of the STARE-HI reporting items. An application to short conference papers on health informatics evaluation. Methods Inf Med 2012; 51 (02) 104-11. 43 EQUATOR.. The Equator Network: Enhancing the Quality and Transparency of Health Research. 2014 [Available from: http://www.equator-network.org. 44 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inf Med 2005; 44 (01) 44-56. 45 Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W. et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med 2010; 49 (02) 105-20. 46 Eval EW. Curricula of health IT evaluation courses. 2014 [Available from:. https://iig.umit.at/efmi/curricula.htm. 47 Balogh E, Miller B, Ball J. Improving Diagnosis in Health Care. Institute of Medicine. National Academies Press; 2015 48 Poon EG, Cusack CM, McGowan JJ. Evaluating healthcare information technology outside of academia: observations from the national resource center for healthcare information technology at the Agency for Healthcare Research and Quality. J Am Med Inform Assoc 2009; 16 (05) 631-6. 49 Rigby M, Georgiou A, Hypponen H, Ammenwerth E, de Keizer N, Magrabi F. et al. Patient Portals as a Means of Information and Communication Technology Support to Patient- Centric Care Coordination - the Missing Evidence and the Challenges of Evaluation. A joint contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Med Inform 2015; 10 (01) 148-59. 50 Ijzerman MJ, Steuten LM. Early assessment of medical technologies to inform product development and market access: a review of methods and applications. Appl Health Econ Health Policy 2011; 9 (05) 331-47.
