Integrating Quantitative and Qualitative Methods in Patient Care Information System Evaluation

Summary

Objective: The aim of this paper is twofold. First, we describe two important dimensions of patient care information systems (PCIS) evaluation: the domain of evaluation and the different phases of the PCIS implementation. Second, we claim that, though Randomized Controlled Trials (RCTs) are often still seen as the standard approach, this type of design hardly generates relevant information for the organizational decision maker.

Method: Interpretive study of evaluation literature. Results and Conclusions: The field of evaluation is scattered and the types of questions that can be asked and methods that can be used seem infinite and badly demarcated. Different stakeholders, moreover, often have different priorities in evaluating ICT. The most important reason for the lack of relevance of RCTs is that they are ill suited for investigating why and how a PCIS is being used, or not, and what the (often unplanned) effects and consequences are. Subsequently, our aim is to contribute to the discussion about the viability of qualitative versus quantitative methods in PCIS evaluation, by arguing for a specific integration of quantitative and qualitative research methods. The joint utilization of these methods, we claim, yields the richest results.

Keywords

Evaluation - multi-method approach - randomized controlled trial - patient care information system

• References

• 1 Talmon J, Enning J, Castañeda G, Eurlings F, Hoyer D, Nykänen P. et al. The VATAM guidelines. Int J Med Inf 1999; 56: 107-15.
• 2 Southon G. IT, Change and evaluation: an overview of the role of evaluation in health services. Int J Med Inf 1999; 56: 125-33.
• 3 Heathfield HA, Buchan IA. Current evaluations of information technology in health care are often inadequate. BMJ 1996; 313: 1008.
• 4 Greatbatch D, Murphy E, Dingwall R. Evaluating medical information systems: ethnomethodological and interactionist approaches. Health Services Management Research 2001; 14 (Suppl. 03) 181-91.
• 5 Savage J. Ethnography and health care. BMJ 2000; 321: 1400-2.
• 6 Heathfield H, Hudson P, Kay S, Mackay L, Marley T, Nicholson L. et al. Issues in multi-disciplinary assessment of healthcare information systems. Information Technology & People 1999; 12: 253-75.
• 7 Jones MR. Mission impossible? Pluralism and multi-paradigm IS research. Information Systems Review 2000; 1: 217-32.
• 8 Mingers J. Combining IS research methods: towards a pluralist methodology. Information Systems Research 2001; 12: 240-59.
• 9 Øvretveit J. Evaluating health interventions. Buck-ingham: Open University Press; 1998
• 10 Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer-Verlag; 1997
• 11 Kaplan B, Maxwell J. Qualitative research methods for evaluating computer information systems. In Anderson J, Aydin C, Jay S. editors. Evaluating Health Care Information Systems. California: Sage; 1994. pp. 45-69.
• 12 Van der Loo R. Overview of published assessment and evaluation studies. In: Van Gennip E, Talmon J. editors. Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp. 261-83.
• 13 Kaplan B. Evaluating informatics applications, clinical decision support systems literature review. Int J Med Inf 2001; 64: 15-37.
• 14 Kaplan B, Duchon D. Combining qualitative and quantitative methods in information systems research, a case study. MIS Quarterly 1988; 571-86.
• 15 Friedman CP, Elstein AE, Wolf FM, Murphy GC, Franz TM, Heckerling PS. et al. Enhancement of clinicians’ diagnostic reasoning by computer-based consultation, a multi site study of two systems. JAMA 1999; 282: 19
• 16 Kuperman GJ, Teich JM, Tanasijevic MJ, Luf NM, Rittenberg E, Jha A. et al. Improving response to critical laboratory results with automation, results of a randomized controlled trial. J Am Med Inform Ass 1999; 6: 512-22.
• 17 Berwick DM. Developing and testing changes in delivery of care. Ann Intern Med 1998; 128: 651-56.
• 18 Anderson J, Aydin C, Jay S. editors. Evaluating health care information systems. Thousand Oaks (CA): Sage; 1994
• 19 Kaplan B. Addressing organizational issues into the evaluation of medical systems. J Am Med Inform Ass 1996; 4: 94-101.
• 20 Pope C. Qualitative research, reaching the parts other methods cannot reach, an introduction to qualitative methods in health and health services research. BMJ 1995; 311: 42-5.
• 21 Jørgensen T. Measuring effects. In: Van Gennip E, Talmon J. editors. Assessment and evaluation of information technologies in medicine. Amsterdam: IOS Press; 1994. pp. 99-111.
• 22 Serafeimidis V, Smithson S. Information systems evaluation in practice: a case study of organizational change. J Inf Techn 2000; 15: 93-105.
• 23 Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T. et al. Evaluating large scale health information systems, from practice towards theory. Proc AMIA Annu Fall Symp 1997; 116-20.
• 24 Farbey B, Land F, Targett D. The moving staircase, problems of appraisal and evaluation in a turbulent environment. Information Technology & People 1999; 12: 238-52.
• 25 Berg M. Patient care information systems and health care work: a sociotechnical approach. Int J Med Inf 1999; 55: 87-101.
• 26 Hebert M. Professional and organizational impact of using patient care information systems. Medinfo 1998; 9: 849-53.