Evidence Based Health Informatics: 10 Years of Efforts to Promote the Principle

Summary

Objectives: To present the importance of Evidence-based Health Informatics (EBHI) and the ethical imperative of this approach; to highlight the work of the IMIA Working Group on Technology Assessment and Quality Improvement and the EFMI Working Group on Assessment of Health Information Systems; and to introduce the further important evaluation and evidence aspects being addressed.

Methods: Reviews of IMIA, EFMA and other initiatives, together with literature reviews on evaluation methods and on published systematic reviews.

Results: Presentation of the rationale for the health informatics domain to adopt a scientific approach by assessing impact, avoiding harm, and empirically demonstrating benefit and best use; reporting of the origins and rationale of the IMIA- and EQUATOR-endorsed Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) and of the IMIA WG's Guideline for Good Evaluation Practice in Health Informatics (GEP-HI); presentation of other initiatives for objective evaluation; and outlining of further work in hand on usability and indicators; together with the case for development of relevant evaluation methods in newer applications such as telemedicine. The focus is on scientific evaluation as a reliable source of evidence, and on structured presentation of results to enable easy retrieval of evidence.

Conclusions: EBHI is feasible, necessary for efficiency and safety, and ethically essential. Given the significant impact of health informatics on health systems, care delivery and personal health, it is vital that cultures change to insist on evidence-based policies and investment, and that emergent global moves for this are supported.

Keywords

Evidence-based health informatics - evaluation - evidence - health informatics - ethics

• References

• 1 http://www.imia-medinfo.org/new2/GA/2012beijing/1210GA-beijing-president-WHO.pdf (accessed 31 January 2013).
• 2 Rigby M, Ammenwerth E, Talmon J, Nykanen P, Brender J, de Keizer N. Health Informatics 3.0 and other Increasingly Dispersed Technologies Require Even Greater Trust: Promoting Safe Evidence-based Health Informatics. In: Geissbuhler A, Kulikowski CA. editors. IMIA Yearbook of Medical Informatics 2011: Towards Health Informatics 3.0. Stuttgart: Schattauer; 2011. p. 105-11.
• 3 Beauchamp T, Childress JF. Principles of Biomedical Ethics. 6th ed. New York: Oxford University Press; 2009
• 4 European Commission. Communication from the Commission on the Precautionary Principle, COM(2000) 1. Brussels: 2000
• 5 Fisher E, Jones J, von Schomberg R. editors. Implementing the Precautionary Principle: Perspectives and Prospects. Cheltenham, UK and Northampton. MA, US: Edward Elgar; 2006
• 6 Ammenwerth E, Shaw NT. Bad health informatics can kill — is evaluation the answer?. Methods Inf Med 2005; 44 (1) 1-3.
• 7 Web list “Bad Health Informatics can Kill”. (accessible via the Bad Health Informatics link at http:// iig.umit.at/efmi) (accessed 31 January 2013).
• 8 http://business-ethics.com/ (accessed 31 January 2013).
• 9 http://www.ibe.org.uk/ (accessed 31 January 2011).
• 10 Patrick JD. The Validity of Personal Experiences in Evaluating HIT. Appl Clin Inf 2010; 1: 462-5.
• 11 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=C0NSLEG:1993L0042:20071011:en:PDF (accessed 31 January 2013).
• 12 Haynes RB, Hayward RS, Jadad AR, Sebaldt RJ. Evidence based health informatics: an overview of the Health Information Research Unit at Mc-Master University. Leadersh Health Serv 1996; May-Jun 5 (3) 41-4.
• 13 Rigby M. Evaluation: 16 powerful reasons why not to do it-and 6 over-riding imperatives. In: Patel V, Rogers R, Haux R. editors. Proceedings of the 10th World Congress on Medical Informatics (Medinfo 2001). Amsterdam: IOS Press; 2001. p. 1198-202.
• 14 Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. et al. Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inf 2004; Jun 30 73 (6) 479-91.
• 15 Ammenwerth E. Health technology assessment. Findings from the Section on Assessing Information Technologies for Health. Yearb Med Inform 2006; 16-9.
• 16 Ammenwerth E, Schnell-Inderst P, Siebert U. Vision and Challenges of Evidence-Based Health Informatics: A case study of a CPOE meta-analysis. Int J Med Inform 2010; 79: e83-e88.
• 17 Sackett D, Rosenberg W, Gray J, Haynes R, Richardson S. Evidence based medicine: what it is and what it isn't. BMJ 1996; 312 (7023) 71-2.
• 18 Ammenwerth E. Evidence based health informatics. Stud Health Technol Inform 2010; 151: 427-34.
• 19 Wyatt J. Assessing and improving evidence based health informatics research. Stud Health Technol Inform 2010; 151: 435-45.
• 20 Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W. et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. IMIA White Paper. Methods Inf Med 2010; 49 (2) 1-16.
• 21 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI — Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78 (1) 1-9.
• 22 http://www.equator-network.org/resource-centre/library-of-health-research-reporting/reporting-guidelines/other-reporting-guidelines/ (accessed 31 January 2013).
• 23 Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir M-C. et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform 2011; 80: 815-27.
• 24 de Keizer NF, Talmon J, Ammenwerth E, Brender J, Nykanen P, Rigby M. Mini Stare-HI: Guidelines for Reporting Health Informatics Evaluations in Conference Papers. In: Safran C, Reti S, Marin HF. editors. MEDINFO 2010 — Proceedings of the 13th World Congress on Medical Informatics. Amsterdam: IOS Press; 2010. p. 1206-10.
• 25 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 — 2002. Methods Inf Med 2005; 44: 44-56.
• 26 Ammenwerth E, de Keizer N. Evaluation Database — A web-based inventory of evaluation studies in medical informatics. Institute for Health Information Systems, University for Health Sciences, Medical Informatics and Technology; Austria: (on-going) (available at http://evaldb.umit.at) (accessed 31 January 2013).
• 27 Rigby M, Brender J, Beuscart-Zephir M-C, Hypponen H, Nykänen P, Talmon J. et al. Next Steps in Evaluation and Evidence — from Generic to Context-Related. In: Moen A, Andersen SK, Aarts J, Hurlen P. editors. User Centred Networked Health Care — Proceedings of MIE 2011. IOS Press; 2011. p. 208-12.
• 28 Melia S. The challenge of evaluating telehealth interventions — Dissertation in part fulfilment of the award of MBA (Health Executive). Keele University; 2012
• 29 Nykänen P, Kaipio J, Kuusisto A. Evaluation of the Finnish nursing classification system and four nursing documentation systems in Finland — lessons learned and directions or the future. Int J Med Inform 2012; 81 (8) 505-20.
• 30 Kopanitsa G, Tsvetkova Z, Veseli H. Analysis of metrics for usability evaluation of electronic health record systems. In: Blobel B. editor Large-scale projects in eHealth. Amsterdam: IOS Press; 2012. p. 129-33.
• 31 Anttila M. Adoption of information technology based patient education in psychiatric nursing. PhD dissertation. University of Turku; 2012 Accessible at: http://www.doria.fi/handle/10024/74662 (accessed 31 January 2013).
• 32 Oluoch T, Santas X, Kwaro D, Were M, Biondich P, Bailey C. et al. The effect of electronic medical record-based clinical decision support on HIV care in resource-constrained settings: a systematic review. Int J Med Inform 2012; 81 (10) e83-92.
• 33 Scott PJ, Briggs J. STAT-HI: A socio-technical assessment tool for health informatics implementations. Open Med Inform J 2010; 4: 214-20.
• 34 Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; Apr 14 357 (9263) 1191-4.
• 35 Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM. et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. BMJ 2003; 326 (7379) 41-4.
• 36 Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999; Nov 27 354 (9193) 1896-900.
• 37 Talmon J, Ammenwerth E, Geven T. The quality of reporting of health informatics evaluation studies: a pilot study. Stud Health Technol Inform 2007; 129 (Pt 1) 193-7.
• 38 de Keizer NF, Talmon J, Ammenwerth E, Brender J, Rigby M, Nykanen P. Systematic Prioritization of the STARE-HI Reporting Items. An Application to Short Conference Papers on Health Informatics Evaluation. Methods Inf Med. 2011 Mar 4 50. (3).
• 39 Brender J. Handbook of Evaluation Methods for Health Informatics. New York: Academic Press; 2006. p. 361.
• 40 Friedman CP, Wyatt JC. Evaluation Methods in Biomedical Informatics. New York: Springer; 2006
• 41 Friedman CP, Altman RB, Kohane IS, McCormick KA, Miller PL, Ozbolt JG. et al. Training of the next generation of informaticians: the impact of “BISTI” and bioinformatics — a report of the American College of Medical Informatics. J Am Med Inform Assoc 2004; 11 (3) 167-72.
• 42 Gardner RM, Overhage JM, Steen EB, Munger BS, Holmes JH, Williamson JJ. et al. Core Content for the Subspecialty of Clinical Informatics. J Am Med Inform Assoc 2009; 16: 153-7.
• 43 Borycki E, Kushniruk A. Identifying and preventing technology-induced error using simulations: application of usability engineering techniques. Healthc Q 2005; 8 Spec No 99-105.
• 44 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; Aug 74 (7- 8) 519-26.
• 45 International Electrotechnical Commission. Medical Electrical Equipment — General requirements for basic safety and essential performance. Collateral Standard. Usability. Geneva: International Electrotechnical Commission; 2007. Report No.: IEC 60601-1-6).
• 46 International Electrotechnical Commission. Medical Electrical Equipment — General requirements for basic safety and essential performance. Collateral Standard. Usability. Geneva: International Electrotechnical Commission; 2010. Report No.: IEC 60601-1-6.
• 47 Beuscart-Zephir MC, Elkin P, Pelayo S, Beuscart R. The human factors engineering approach to biomedical informatics projects: state of the art, results, benefits and challenges. Yearb Med Inform 2007; 109-27.
• 48 Beuscart-Zephir MC, Pelayo S, Borycki E, Kushniruk A. Human Factors Considerations in Health IT Design and Development. In: Carayon P. editor. Human Factors and Ergonomics in Health Care and Patient Safety 2nd Edition. CRC Press; 2012. p. 649-70.
• 49 Borycki E, Kushniruk A. Where do technology-induced errors come from? Towards a model for conceptuallizing and diagnosing errors caused by technology. In: Kushniruk A, Borycki E. editors. Human, Social and Organizational Aspects of Health Information Systems. Hershey, NY: Idea Group; 2008
• 50 Beuscart-Zephir MC, Borycki E, Carayon P, Jaspers MW, Pelayo S. Evolution of Human factors Research and Studies of Health information technologies: the Role of Patient safety. Yearb Med Inform. 2013 Seitenzahlen.
• 51 Magrabi F, Aarts J, Nohr C, Baker M, Harrison S, Pelayo S. et al. A comparative review of patient safety initiatives for national health information technology. Int J Med Inform 2013; May 82 (5) e139-48.
• 52 Marcilly R, Beuscart-Zephir MC, Pelayo S. Seeking Evidence to support Usability Principles for medication-related CDS functions. 2013
• 53 Peute LW, Spithoven R, Bakker PJ, Jaspers MW. Usability studies on interactive health information systems; where do we stand?. Stud Health Technol Inform 2008; 136: 327-32.
• 54 Lawrence M, Olesen F. Indicators of Quality in Health Care. Eur J Gen Pract 1997; 3 (3) 103-8.
• 55 Hyppönen H, Ammenwerth E, Faxwaag A, Nohr C, Walldius A. eHealth indicators: results of the MIE 2011 workshop. Stud Health Technol Inform 2012; 180: 328-32.
• 56 Hyppönen H, Vuokko R, Mäkelä-Bengs P, Doupi P, Saranto K, Mäkelä M. Impacts of structuring the patient record: a systematic review protocol and results of previous reviews. [Submitted to Int J Med Inform, under review].
• 57 DeLone WH, McLean ER. The DeLone and McLean Model of Information Systems Success: a ten-year update. Journal of Management Information Systems 2003; 19 (4) 9-30.
• 58 Lampe K, Mäkelä M. editors. HTA Core Model for medical and surgical interventions. First Public Draft (Revised) EUnetHTA: Work Package 4, 2007, available at http://www.eunethta.eu/outputs/hta-core-model-medical-and-surgical-interventions-10r (accessed 31 January 2013).
• 59 Hyppönen H, Ammenwerth E, de Keizer N. Exploring a methodology for eHealth indicator development. Stud Health Technol Inform 2012; 180: 338-42.
• 60 Hyppönen H, Faxvaag A, Nohr C, Vimarlund V, Hardardottir GA, Reponen J. et al. Nordic eHealth Indicators — Organisation of research, first results and plan for the future. [in press, Nordic Council of Ministers, Tema Nord-series number TN2013:522].
• 61 Rigby M. Evaluation — the Cinderella Science of ICT in Health. In Kulikowski C, Haux R. editors. IMIA Yearbook of Medical Informatics 2006. Stuttgart: Schattauer; 2006. p. 114-20.
• 62 Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform 2008; Jun 77 (6) 386-98.
• 63 Cusack CM, Byrne C, Hook JM, McGowan J, Poon EG, Zafar A. Health information technology evaluation toolkit: 2009 Update (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 29004-0016.) AHRQ publication No. 09-0083-EF. Rockville, MD: Agency for Healthcare Research and Quality; June 2009.
• 64 Kidholm K, Bowes A, Dyrehauge S, Granstrom Ekeland A, Flottorp SA, Kvistgaard Jensen L. et al. The MAST Manual. MAST-Model for assessment of tele medicine. http://www.epractice.eu/en/library/315610 (accessed 31 January 2013).
• 65 Lorenzi N, Riley R, Blyth A, Southon G, Dixon B. Antecedents of the people and organizational aspects of medical informatics. J Am Med Inform Assoc 1997; 4 (2) 79.
• 66 Berg M. Medical work and the computer-based patient record: a sociological perspective. Methods Inf Med 1998; 7 (3) 294-301.
• 67 Goorman E, Berg M. Modelling nursing activities: electronic patient records and their discontents. Nurs Inq 2000; 7 (1) 3-9.
• 68 Aarts J, Doorewaard H, Berg M. Understanding implementation: the case of a computerized physician order entry system in a large Dutch university medical center. J Am Med Inform Assoc 2004; 11 (3) 207-16.
• 69 Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007; 76 (Suppl 1) 21-7.
• 70 Niazkhani Z, Pirnejad H, Aarts J, Adams S, Bal R. Reporting qualitative research in health informatics: REQ-Hl recommendations. Stud Health Technol Inform 2011; 169: 877-81.
• 71 Eysenbach G. CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res. 2011 Dec 31 13. (4) e126. doi: 10.2196/jmir.1923.
• 72 Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. Conceptual Framework for Development of Comprehensive e-Health Evaluation Tool. Telemed J E Health 2013; 19 (1) 48-53.
• 73 Call to Action on Global eHealth Evaluation — Consensus Statement of the WHO Global eHealth Evaluation Meeting. Bellgio: September 2011. available from http://www.healthunbound.org/content/call-action-global-ehealth-evaluation (accessed 31 January 2013).
• 74 Ham C, Hunter DJ, Robinson R. Evidence-based Policy. BMJ 1995; 310 (6972) 71-2.
• 75 Walshe K, Rundall TG. Evidence-based Management: From Theory to Practice in Health Care. The Milbank Quarterly 2001; 79: 429-57.
• 76 Gray JAM. Evidence-based Healthcare: How to make Health Policy and Management Decisions. 2nd ed. Edinburgh: Churchill Livingstone; 2011. p. 444.
• 77 Rigby M, Ronchi E, Graham S. Evidence for building a smarter health and wellness future — Key messages and collected visions from a Joint OECD and NSF workshop. Int J Med Inform. 2012 Oct 29. pii: S1386-5056(12)00195-5. doi: 10.1016/j. ijmedinf.2012.10.003. [Epub ahead of print].