An Inventory of Evaluation Studies of Information Technology in Health Care

Summary

Objectives: During the last years the significance of evaluation studies as well as the interest in adequate methods and approaches for evaluation has grown in medical informatics. In order to put this discussion into historical perspective of evaluation research, we conducted a systematic review on trends in evaluation research of information technology in health care from 1982 to 2002.

Methods: The inventory is based on a systematic literature search in PubMed. Abstracts were included when they described an evaluation study of a computer-based component in health care. We identified 1035 papers from 1982 to 2002 and indexed them based on a multi-axial classification describing type of information system, study location, evaluation strategy, evaluation methods, evaluation setting, and evaluation focus.

Results and Conclusions: We found interesting developments in evaluation research in the last 20 years. For example, there has been a strong shift from medical journals to medical informatics journals. With regard to methods, explanatory research and quantitative methods have dominated evaluation studies in the last 20 years. Since 1982, the number of lab studies and technical evaluation studies has declined, while the number of studies focusing on the influence of information technology on quality of care processes or outcome of patient care has increased. We interpret this shift as a sign of maturation of evaluation research in medical informatics.

Keywords

Evaluation studies - health and medical informatics - technology assessment - information technology - health care - health information systems - inventory - review

• References

• 1 Beynon-Davies P, Lloyd-Williams M. When health information systems fail. Top Health Inf Manage 1999; 20 (01) 66-79.
• 2 Forsythe DE, Buchanan BG. Broadening our approach to evaluating medical information systems. In. Clayton P. editor. 15th Annual Symposium on Computer Applications in Medical Care. New York: McGraw-Hill; 1992: 8-12.
• 3 Friedman C, Wyatt JC. Evaluation Methods in Medical Informatics. New York: Springer; 1997
• 4 Heathfield H, Peel V, Hudson P, Kay S, Mackay L, Marley T. et al. Evaluating Large Scale Health Information Systems: From Practice Towards Theory. In. Masys D. editor. AMIA Annual Fall Symposium. Philadelphia: Hanley & Belfus; 1997: 116-20.
• 5 Moehr JR. Evaluation: salvation or nemesis of medical informatics?. Comput Biol Med 2002; 32 (03) 113-25.
• 6 Ammenwerth E, Brender J, Nykänen P, Prokosch H-U. Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems – reflections and lessons based on the HISEVAL workshop in Innsbruck. Int J Med Inf 2004; 73 (06) 479-91.
• 7 Anderson JG, Aydin CE, Jay SJ. editors Evaluating Health Care Information Systems – Methods and Applications. London, New Delhi: Sage Publications; 1994
• 8 van Gennip E, Talmon J. editors Assessment and evaluation of information technologies in medicine. Studies in. Health Technology and Informatics vol 17. Amsterdam: IOS Press. 1995
• 9 Whitten P, Mair F, Haycox A, May C, Williams T, Hellmich S. Systematic review of cost effectiveness studies of telemedicine interventions. BMJ 2002; 324: 1434-7.
• 10 Haux R, Ammenwerth E, Herzog W, Knaup P. Health Care in the Information Society: A Prognosis for the Year 2013. Int J Med Inf 2003; 66: 3-12.
• 11 Tierney W, Overhage J, McDonald C. A Plea for Controlled Trials in Medical Informatics. J Am Med Inform Assoc 1994; 1 (04) 353-5.
• 12 Heathfield H, Buchan I. Current evaluations of information technology in health care are often inadequate. BMJ 1996; 313 07063 1008.
• 13 Kaplan B. Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64: 39-56.
• 14 Simborg D, McDonald L, Liebman J, Musco P. Ward information management system – an evaluation. Comput Biomed Res 1972; 5 (05) 484-97.
• 15 Pub Med. National Library of Medicine. http://www.ncbi.nlm.nih.gov.
• 16 Krobock JR. A taxonomy: hospital information systems evaluation methodologies. J Med Syst 1984; 8 (05) 419-29.
• 17 Sawyer S, Chen T. Conceptualizing Information Technology in the Study of Information Systems: Trends and Issues. In. Myers M, Whitley E, Wynn E, DeGross J. editors. Global and Organizational Discourse about Information Technology. London: Kluwer; 2002: 1-23.
• 18 Grémy F, Degoulet P. Assessment of health information technology: which questions for which systems? Proposal for a taxonomy. Med Inform (Lond) 1993; 18 (03) 185-93.
• 19 van der Loo R. Overview of Published Assessment and Evaluation Studies. In. van Gennip EMSJ, Talmon JS. editors. Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995: 261-82.
• 20 Ammenwerth E. Die Bewertung von Informationssystemen des Gesundheitswesens [The evaluation of health information systems]. Publication Series No 1 of UMIT. University for Health Sciences, Medical Informatics and Technology, Hall in Tyrol, Austria. Aachen: Shaker; 2004
• 21 Bortz J, Döring N. Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler. BerlIn: Springer; 2002
• 22 Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med 1996; 5: 271-8.
• 23 Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. Int J Med Inf 2001; 64: 15-37.
• 24 Roine R, Ohinmaa A, Hailey D. Assessing telemedicine: a systematic review of the literature. Canad Med Assoc J 2001; 165 (06) 765-71.
• 25 Friedman C. Is Medical Informatics a mature science?. Int J Med Inf 2003; 69: 261-72.
• 26 Hunt D, Haynes R, Hanna S, Smith K. Effects of Computer-Based Clinical Decision Support Systems on Physician Performance and Patient Outcomes. JAMA 1998; 280 (015) 1339-45.
• 27 Moormann P, van der Lei J. An Inventory of Publications on computer-based medical records: an update. Methods Inf Med 2003; 42: 199-202.
• 28 Fletcher RH, Fletcher SW. Clinical research in general medical journals: a 30-year perspective. N Engl J Med 1979; 301 (04) 180-3.
• 29 Tierney W, McDonald C. Testing Informatics Innovations: The Value of Negative Trials. J Am Med Inform Assoc 1996; 3 (05) 358-9.
• 30 McDonald C. Protocol-based computer reminders, the quality of care and the non-perfectability of man. 1976; 295 (24) 1351-5.
• 31 Rigby M. Health informatics as a tool to improve quality in non-acute care – new opportunities and a matching need for a new evaluation paradigm. Int J Med Inf 1999; 56: 141-50.
• 32 Haux R, Kulikowski C. Yearbook of Medical Informatics (annual edition). Stuttgart: Schattauer; Since 2001. http://iig.umit.at/yearbook.