Download PDF
Methods Inf Med 2011; 50(04): 299-307
DOI: 10.3414/ME10-01-0042
Original Articles
Schattauer GmbH

Rapid Assessment of Clinical Information Systems in the Healthcare Setting

An Efficient Method for Time-pressed Evaluation
C. K. McMullen
1   The Center for Health Research, Kaiser Permanente Northwest, Portland, OR, USA
,
J. S. Ash
2   Oregon Health & Science University, Portland, OR, USA
,
D. F. Sittig
3   University of Texas School of Biomedical Informatics at Houston, Houston, TX, USA
,
A. Bunce
2   Oregon Health & Science University, Portland, OR, USA
,
K. Guappone
4   Providence Portland Medical Center, Portland, OR, USA
,
R. Dykstra   *
2   Oregon Health & Science University, Portland, OR, USA
,
J. Carpenter
4   Providence Portland Medical Center, Portland, OR, USA
,
J. Richardson
2   Oregon Health & Science University, Portland, OR, USA
,
A. Wright
5   Brigham & Women’s Hospital, Boston, MA, USA
› Author Affiliations
Further Information

Publication History

received: 28 May 2010

accepted: 07 October 2010

Publication Date:
18 January 2018 (online)

    Permissions and Reprints

Summary

Objective: Recent legislation in the United States provides strong incentives for implementation of electronic health records (EHRs). The ensuing transformation in U. S. health care will increase demand for new methods to evaluate clinical informatics interventions. Timeline constraints and a rapidly changing environment will make traditional evaluation techniques burdensome. This paper describes an anthropological approach that provides a fast and flexible way to evaluate clinical information systems.

Methods: Adapting mixed-method evaluation approaches from anthropology, we describe a rapid assessment process (RAP) for assessing clinical informatics interventions in health care that we developed and used during seven site visits to diverse community hospitals and primary care settings in the U. S.

Setting: Our multidisciplinary team used RAP to evaluate factors that either encouraged people to use clinical decision support (CDS) systems or interfered with use of these systems in settings ranging from large urban hospitals to single-practitioner, private family practices in small towns.

Results: Critical elements of the method include: 1) developing a fieldwork guide; 2) carefully selecting observation sites and participants; 3) thoroughly preparing for site visits; 4) partnering with local collaborators; 5) collecting robust data by using multiple researchers and methods; and 6) analyzing and reporting data in a structured manner helpful to the oraganizations being evaluated.

Conclusions: RAP, iteratively developed over the course of visits to seven clinical sites across the U.S., has succeeded in allowing a multidisciplinary team of informatics researchers to plan, gather and analyze data, and report results in a maximally efficient manner.

Keywords

Ethnography - qualitative methods - clinical informatics - healthcare evaluation mechanisms - computerized medical record systems

* deceased


 

  • References

  • 1 Blumenthal D. Stimulating the adoption of health information technology. N Engl J Med 2009; 360 (15) 1477-1479.
    CrossrefPubMedGoogle Scholar
  • 2 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-231.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 3 Ash J, Stavri P, Kuperman G. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc 2003; 10 (03) 229-234.
    CrossrefPubMedGoogle Scholar
  • 4 Kaplan B, Harris-Salamone KD. Health IT success and failure: recommendations from literature and an AMIA workshop. J Am Med Inform Assoc 2009; 16 (03) 291-299.
    CrossrefPubMedGoogle Scholar
  • 5 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346.
    CrossrefPubMedGoogle Scholar
  • 6 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 7494 765.
    CrossrefPubMedGoogle Scholar
  • 7 Garg AX, Adhikari NK, McDonald H, RosasArellano MP, Devereaux PJ, Beyene J. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238.
    CrossrefPubMedGoogle Scholar
  • 8 Kaplan B. Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64 (01) 39-56.
    CrossrefPubMedGoogle Scholar
  • 9 Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316 7149 1959-1961.
    CrossrefPubMedGoogle Scholar
  • 10 McNall M, Foster-Fishman PG. Methods of Rapid Evaluation, Assessment and Appraisal. American Journal of Evaluation 2007; 28 (02) 151-168.
    CrossrefPubMedGoogle Scholar
  • 11 Patton MQ. Utilization-Focused Evaluation. 3rd ed. Thousand Oaks, CA: Sage; 1997
    Google Scholar
  • 12 Guba EG LY. Fourth generation evaluation. Newbury Park, CA: Sage; 1989
    Google Scholar
  • 13 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385.
    CrossrefPubMedGoogle Scholar
  • 14 Butler MO. Translating Evaluation Anthropology. NAPA Bulletin 2005; 24: 17-30.
    PubMedGoogle Scholar
  • 15 Dewalt KM, Dewalt BR, Wyland CB. Participant observation – 1998. In: Bernard HR. editor. Handbook of Methods in Cultural Anthropology. Walnut Creek, CA: Altamira Press; 1998. pp 259-300.
    Google Scholar
  • 16 Creswell JW. Qualitative Inquiry & Research Design: Choosing among Five Approaches. Thousand Oaks, CA: Sage Publications; 2007
    Google Scholar
  • 17 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512.
    CrossrefPubMedGoogle Scholar
  • 18 Sittig DF, Ash JS. Clinical information systems: Overcoming adverse consequences. Sudbury, MA; 2010
    Google Scholar
  • 19 Ash J, Sittig D, McMullen C, Guappone K, Dykstra R, Carpenter J. A rapid assessment process for clinical informatics interventions. AMIA Annu Symp Proc. 2008 pp 26-30.
    PubMedGoogle Scholar
  • 20 Beebe J. Rapid Assessment Process: An Introduction. Walnut Creek, CA: AltaMira; 2001
    Google Scholar
  • 21 Handwerker WP. Quick Ethnography. Walnut Creek, CA: AltaMira Press; 2001
    Google Scholar
  • 22 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI – Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78 (01) 1-9.
    CrossrefPubMedGoogle Scholar
  • 23 Scrimshaw SCM HE. Rapid Assessment Procedures for Nutrition and Primary Health Care: Anthropological Approaches to Improving Programme Effectiveness. Los Angeles, CA: 1987
    Google Scholar
  • 24 Needle RH, Tsukamoto T, Goosby E, Bates C, von Zinkmangel D. Crisis Response Teams and Communities Combet HIV/AIDS in Racial and Ethnic Minority Populations: A guide for conducting community-based Rapid Assessment, Rapid Response and Evaluating. Washington DC: 2000
    Google Scholar
  • 25 Manderson L, Aaby P. Rapid Anthropological Procedures (RAP) and their applicability to tropical diseases. Health Policy Planning 1992; 7: 46-55.
    CrossrefPubMedGoogle Scholar
  • 26 Trotter RT, Goosby E, Needle RH, Bates C, Singer M. A methodological model for rapid assessment, response, and evaluation: The RARE Program in public health. Field Methods 2001; 13 (02) 137-159.
    CrossrefPubMedGoogle Scholar
  • 27 Trotter RT, Needle RH. RARE Field Team Principal Investigator Guide. Washington DC: 2000
    Google Scholar
  • 28 Butler MO, Linstone HA. Decision making for technology executives: Using multiple perspectives to improve performance. Boston: Artech House; 1999
    Google Scholar
  • 29 Manderson L, Aaby P. An epidemic in the field? Rapid assessment procedures and health assessment. Social Science and Medicine 1992; 35 (07) 839-950.
    CrossrefPubMedGoogle Scholar
  • 30 O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ 2010; 341: c4587.
    CrossrefPubMedGoogle Scholar
  • 31 Greene J, McClintock CM. Triangulation in evaluation: design and analysis issues. Evaluation Review 2010; 9 (05) 523-545.
    PubMedGoogle Scholar
  • 32 Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530.
    CrossrefPubMedGoogle Scholar
  • 33 Utarini A, Winkirst A, Pelto GH. Appraising Studies in Health Using Rapid Assessment Procedures (RAP): Eleven Critical Criteria. Human Organization 2001; 60 (04) 390-400.
    CrossrefPubMedGoogle Scholar
  • 34 Thomas MD, Blacksmith J, Reno J. Utilizing insider-outsider research teams in qualitative research. Qual Health Res 2000; 10 (06) 819-828.
    CrossrefPubMedGoogle Scholar
  • 35 Fernald DH, Duclos CW. Enhance your team-based qualitative research. Ann Fam Med 2005; 3 (04) 360-364.
    CrossrefPubMedGoogle Scholar
  • 36 Patton MQ. Qualitative Evaluation and Research Methods. 3rd ed. Thousand Oaks, CA: Sage; 2002
    Google Scholar
  • 37 Bernard HR, Ryan GW. Text analysis: Qualitative and Quantitative methods. Handbook of Methods in Cultural Anthropology. Walnut Creek, CA: AltaMira Press; 1998
    Google Scholar
  • 38 Miles MB, Huberman AM. Qualitative Data Analysis. 2nd ed. Thousand Oaks, CA: Sage; 1994
    Google Scholar
  • 39 Van den Hoonaard WC. Analytical Field Research: Working with Sensitizing Concepts. Thousand Oaks, CA: Sage; 1997
    Google Scholar
  • 40 Bernard HR, Ryan GW. Analyzing Qualitative Data: Systematic Approaches. Thousand Oaks, CA: Sage Publications; 2010
    Google Scholar
  • 41 Spradley JP. The Ethnographic Interview. Fort Worth, TX: Harcourt Brace Jovanovich; 1979
    Google Scholar
  • 42 Khajouei R, Jaspers MW. The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med 2010; 49 (01) 3-19.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 43 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inf Med 2005; 44 (01) 44-56.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 44 Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. Int J Med Inform 2001; 64 (01) 15-37.
    CrossrefPubMedGoogle Scholar
  • 45 Patton MC. The View from Evaluation. NAPA Bulletin 2005; 24: 31-40.
    PubMedGoogle Scholar
  • 46 Ash JS, Sittig DF, Dykstra R, Wright A, McMullen C, Richardson J. et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform 2010; 160: 806-810.
    PubMedGoogle Scholar
  • 47 Schensul JJ, Schensul SL. Ethnographic evaluation of AIDS prevention programs: better date for better programs. New Directions for Program Evaluation 1990; 46: 51-62.
    PubMedGoogle Scholar
  • 48 Checkland P, Holwell S. Action Research: Its Nature and Validity In Systemic Practice and Action Research. Action Research 1998; 11 (01) 9-21.
    PubMedGoogle Scholar
  • 49 Reason P, Bradbury H. Handbook of Action Research. London UK: Sage Publications; 2001
    Google Scholar
  • 50 Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health 1998; 19: 173-202.
    CrossrefPubMedGoogle Scholar