The Human Factors Engineering Approach to Biomedical Informatics Projects: State of the Art, Results, Benefits and Challenges

Summary

Objectives

The objective of this paper is to define a comprehensible overview of the Human Factors approach to biomedical informatics applications for healthcare. The overview starts with a presentation of the necessity of a proper management of Human factors for Healthcare IT projects to avoid unusable products and unsafe work situations. The first section is dedicated to definitions of the Human Factors Engineering (HFE) main concepts. The second section describes a functional model of an HFE lifecycle adapted for healthcare work situations. The third section provides an overview of existing HF and usability methods for healthcare products and presents a selection of interesting results. Thelast section discusses the benefits and limitations of the HF Eapproach.

Methods

Literature review based on Pubmedand conference proceedings in the field of Medical Informatics coupled with a review of other databases and conference proceedings in the field of Ergonomics focused on papers addressing healthcare work and system design.

Results

Usability studies performed on healthcare applications have uncovered unacceptable usability flaws that make the systems error prone, thus endangering the patient safety. Moreover, in many cases, the procurement and the implementation process simply forget about human factors: following only technological considerations, they issue potentially dangerous and always unpleasant work situations. But when properly applied to IT projects, the HFE approach proves efficient when seeking to improve patient safety, users’ satisfaction and adoption of the products.

Conclusions

We recommend that the HFE methodology should be applied to most informatics and systems development projects, and the usability of the products should be systematically checked before permitting their release and implementation. This requires the development of Centers specialized in Human Factors for Healthcare and Patient safety in each Country/Region.

Keywords

Human Factors Engineering - usability - socio-technical approach - healthcare IT applications - computer-controlled medical devices - patient safety - IT

• References

• 1 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; Jun 23; 163 (12) 1409-16.
• 2 Kaushal R, Jha AK, Franz C, Glaser J, Shetty KD, Jaggi T. et al. Return on investment for a computerized physician order entry system. J Am Med Inform Assoc 2006; May; 13 (03) 261-6.
• 3 Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM. et al. Effect of Computerized Physician Order Entry and a team intervention on prevention of serious medication errors. JAMA 1998; 280: 1311-6.
• 4 Bates DW, Teich JM, Lee M, Seger D, Kuperman GJ. Ma’Luf et al. The impact of Computerized Physician Order Entry on medication error prevention. J Am Med Inform Assoc 1999; 06 (04) 313-21.
• 5 Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting Adverse Drug Events using Information Technology. J Am Med Inform Assoc 2003; 10 (02) 115-28.
• 6 Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E. et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; May 16; 144 (10) 742-52.
• 7 Gainer A, Pancheri K, Zhang J. Improving the Human Computer Interface design for a physician order entry system. 2003: 847.
• 8 King WJ, Paice N, Rangrej J, Forestell GJ, Swartz R. The effects of computerized order entry on medication errors and adverse drug events in pediatric inpatients. Pediatrics 2003; 112: 506-9.
• 9 Scanlon M. Computer physician order entry and the real world: we’re only humans. Jt Comm J Qual Saf 2004; Jun; 30 (06) 342-6.
• 10 Horsky J, Kaufman DR, Patel VL. The cognitive complexity of a provider order entry interface. AMIA Annu Symp Proc. 2003: 294-8.
• 11 Horsky J, Kaufman DR, Oppenheim MI, Patel VL. A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. J Biomed Inform 2003; Feb; 36 1-2 4-22.
• 12 Gorman PN, Berg Marc. Modeling nursing activities: electronic records and their discontents. In: Haux R, Kulikowski C. editors. IMIA Yearbook of Medical Informatics. 2002: 358-64.
• 13 Gorman PN, Lavelle MB, Ash JS. Order creation and communication in healthcare. Methods Inf Med 2003; 42 (04) 376-84.
• 14 Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA 2005; Mar 9; 293 (10) 1261-3.
• 15 Cuschieri A. Nature of human error: implications for surgical practice. Ann Surg 2006; Nov; 244 (05) 642-8.
• 16 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; Dec; 116 (06) 1506-12.
• 17 Karsh B-T, Holden RJ. New technology implementation in Health Care. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 393-411.
• 18 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; Mar; 11 (02) 104-12.
• 19 Cedars-Sinai suspends CPOE use. http://www.ihealthbeat.org/index.cfm?action=dspItem&itemid=117159.2007.Ref Type: Internet Communication
• 20 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE. et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; Mar 9; 293 (10) 1197-203.
• 21 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: The relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; Aug; 74 7-8 519-26.
• 22 Graham MJ, Kubose TK, Jordan D, Zhang J, Johnson TR, Patel VL. Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units. Int J Med Inform 2004; Nov; 73 11-12 771-9.
• 23 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; Feb; 36 1-2 23-30.
• 24 Lin L, Vicente KJ, Doyle DJ. Patient Safety, Potential Adverse Drug Events, and Medical Device Design: A Human Factors Engineering Approach. J Biomed Inform 2001; Aug; 34 (04) 274-84.
• 25 Reddy MC, McDonald DW, Pratt W, Shabot MM. Technology, work, and information flows: lessons from the implementation of a wireless alert pager system. J Biomed Inform 2005; Jun; 38 (03) 229-38.
• 26 Beuscart-Zephir MC, Pelayo S, Degoulet P, Anceaux F, Guerlinger S, Meaux JJ. A usability study of CPOE’s medication administration functions: impact on physician-nurse cooperation. Medinfo 2004; 11 (Pt 2): 1018-22.
• 27 Beuscart-Zephir MC, Pelayo S, Anceaux F, Meaux JJ, Degroisse M, Degoulet P. Impact of CPOE on doctor-nurse cooperation for the medication ordering and administration process. Int J Med Inform 2005; Aug; 74 7-8 629-41.
• 28 Beuscart-Zephir MC, Pelayo S, Anceaux F, Maxwell D, Guerlinger S. Cognitive analysis of physicians and nurses cooperation in the medication ordering and administration process. Int J Med Inform 2007; Jun; 76 (Suppl. 01) 65-77.
• 29 Del Beccaro MA, Jeffries HE, Eisenberg MA. Harry Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit. Pediatrics 2006; Jul; 118 (01) 290-5.
• 30 Ammenwerth E, Talmon J, Ash JS, Bates DW, Beuscart-Zephir MC, Duhamel A. et al. Impact of CPOE on Mortality Rates – Contradictory Findings, Important Messages. Methods Inf Med 2006; 45 (06) 586-93.
• 31 Zhang J. Human-centered computing in health information systems Part 1: Analysis and design. J Biomed Inform 2005; Feb; 38 (01) 1-3.
• 32 Rinkus S, Walji M, Johnson-Throop KA, Malin JT, Turley JP, Smith JW. et al. Human-centered design of a distributed knowledge management system. J Biomed Inform 2005; Feb; 38 (01) 4-17.
• 33 Carayon P. Human Factors and Ergonomics in Health Care and Patient Safety. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 3-21.
• 34 Henriksen K. Human Factors and Patient Safety: continuing challenges. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 21-39.
• 35 Wears RL, Perry SJ. Human factors and ergonomics in the emergency department. Ann Emerg Med 2002; Aug; 40 (02) 206-12.
• 36 Wears RL, Perry SJ. Human Factors and ergonomics in the emergency department. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 851-65.
• 37 Gosbee J. Human factors engineering and patient safety. Qual Saf Health Care 2002; Dec 1; 11 (04) 352-4.
• 38 Gosbee J, Anderson T. Human factors engineering design demonstrations can enlighten your RCA team. Qual Saf Health Care 2003; Apr 1; 12 (02) 119-21.
• 39 Gosbee JW. Conclusion: You need human factors engineering expertise to see design hazards that are hiding in “plain sight!”. Jt Comm J Qual Saf 2004; Dec; 30 (12) 696-700.
• 40 Nemeth C, Nunnally M, O’Connor M, Klock PA, Cook R. Getting to the point: developing IT for the sharp end of healthcare. J Biomed Inform 2005; Feb; 38 (01) 18-25.
• 41 Chan W. Increasing the success of physician order entry through human factors engineering. J Healthc Inf Manag 2002; 16 (01) 71-9.
• 42 Vicente K J. From patients to politicians: a cognitive engineering view of patient safety. Qual Saf Health Care 2002; 11: 302-4.
• 43 Woods DD, Patterson ES, Cook RI. Behind human error: taming complexity to improve patient safety. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 459-77.
• 44 International Ergonomics association. http://wwwieacc/ergonomics/2006 Available from: URL: http://www.iea.cc/ergonomics/
• 45 Carayon P. Human factors of complex sociotechnical systems. Applied Ergonomics 2006; Jul; 37 (04) 525-35.
• 46 Clegg CW. Sociotechnical principles fro system design. Applied Ergonomics 2000; 31: 463-77.
• 47 Berg M. Patient care information systems and health care work: a sociotechnical approach. Int J Med Inform 1999; 55: 87-101.
• 48 Kuutti K. Activity theory and its applications in information systems research and design. In: Nissen HE, Klein HK, Hirscheim R. editors. Information systems research arena of the 90’s. Amsterdam: Elsevier North-Holland; 1991: 529-50.
• 49 Vygotski L. Mind in society. The development of higher psychological processes. Cambridge: Cambridge University Press; 1978
• 50 Cacciabue PC. Guide to applying Human Factors methods. London: Springer-Verlag; 2004
• 51 Carayon P, Alvarado CJ, Hundt AS. Reducing workload and increasing patient safety through work and workspace design. CQPI, center fo quality and productivity improvement. 2003
• 52 Abran A, Khelfi A, Suryn W. Usability meanings and interpretations in ISO standards. Software quality journal 2003; 11: 325-38.
• 53 Leplat J, Hoc JM. Tâche et activité dans l’analyse psychologique des situations. Cahiers de Psychologie Cognitive 1983; 03: 49-63.
• 54 Mayhew D. The Usability Engineering Lifecycle: a practitioner’s handbook for user interface design. Morgan Kaufmann Publishers. 1999
• 55 Samaras GM, Horst RL. A systems engineering perspective on the human-centered design of health information systems. J Biomed Inform 2005; Feb; 38 (01) 61-74.
• 56 Pelayo S, Leroy N, Guerlinger S, Degoulet P, Meaux JJ, Beuscart-Zephir MC. Cognitive analysis of physicians’ medication ordering activity. Stud Health Technol Inform 2005; 116: 929-34.
• 57 Bernonville S, Kolski C, Beuscart-Zephir MC. Contribution and limits of UML models for task modelling in a complex organizational context: case study in the healthcare domain. In press. 2005
• 58 Bernonville S, Leroy N, Kolski C, Beuscart-Zephir MC. Explicit combination between Petri Nets and ergonomics criteria: basic principles of the ErgoPNets method. Presses Universitaires de Valenciennes. 2006
• 59 Buckle P, Clarkson PJ, Ward J, Anderson J. Patient safety, systems design and ergonomics. Applied Ergonomics 2006; 37: 491-500.
• 60 Malhotra S, Laxmisan A, Keselman A, Zhang J, Patel VL. Designing the design phase of critical care devices: a cognitive approach. J Biomed Inform 2005; Feb; 38 (01) 34-50.
• 61 Mendonca EA, Chen ES, Stetson PD, McKnight LK, Lei J, Cimino JJ. Approach to mobile information and communication for health care. Int J Med Inform 2004; Aug; 73 7-8 631-8.
• 62 Beuscart-Zephir MC, Anceaux F, Crinquette V, Renard JM. Integrating users’ activity modeling in the design and assessment of hospital electronic patient records: the example of anesthesia. Int J Med Inform 2001; Dec; 64 2-3 157-71.
• 63 Beuscart-Zephir MC, Menu H, Evrard F, Guerlinger S, Watbled L, Anceaux F. Multidimensional evaluation of a Clinical Information System for anaesthesiology: quality management, usability, and performances. Stud Health Technol Inform 2003; 95: 649-54.
• 64 Merry AF, Webster CS, Mathew DJ. A new, safetyoriented, integrated drug administration and automated anesthesia record system. Anesth Analg 2001; Aug; 93 (2): 385-90.
• 65 Blike George, Cravero Joseph, Nelson Eugene. Same patients, same critical events – different systems of care, different outcomes: description of a Human Factors approach aimed at improving the efficacy and safety of sedation/analgesia care. Qual Manag Health Care 2001; 10 (01) 17-36.
• 66 Rose AF, Schnipper JL, Park ER, Poon EG, Li Q, Middleton B. Using qualitative studies to improve the usability of an EMR. J Biomed Inform 2005; Feb; 38 (01) 51-60.
• 67 Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005; Feb; 38 (01) 75-87.
• 68 Grenier AS, Carayon P, Casper GR, Or CKL, Burke LJ, Brennan PF. Usability evaluation of an Internetbased health information/communication system for CHF patients. Proceedings of the IEA 2006 Congress. 2006
• 69 Karsh BT. Beyond usability: designing effective technology implementation systems to promote patient safety. Qual Saf Health Care 2004; Oct; 13 (05) 388-94.
• 70 Ammenwerth E, Mansmann U, Iller C, Eichstadter R. Factors affecting and affected by user acceptance of computer-based nursing documentation: results of a two-year study. J Am Med Inform Assoc 2003; Jan; 10 (01) 69-84.
• 71 Lu YC, Xiao Y, Sears A, Jacko JA. A review and a framework of handheld computer adoption in healthcare. Int J Med Inform 2005; Jun; 74 (05) 409-22.
• 72 Poon EG, Jha AK, Christino M, Honour MM, Fernandopulle R, Middleton B. et al. Assessing the level of healthcare information technology adoption in the United States: a snapshot. BMC Med Inform Decis Mak 2006; 06: 1.
• 73 Wu RC, Abrams H, Baker M, Rossos PG. Implementation of a computerized physician order entry system of medications at the University Health Network—physicians’ perspectives on the critical issues. Healthc Q 2006; 09 (01) 106-9.
• 74 Ginsburg G. Human factors engineering: a tool for medical device evaluation in hospital procurement decision-making. J Biomed Inform 2005; Jun; 38 (03) 213-9.
• 75 Beuscart-Zephir MC, Watbled L, Carpentier AM, Degroisse M, Alao O. A rapid usability assessment methodology to support the choice of clinical information systems: a case study. Proc AMIA Symp. 2002: 46-50.
• 76 Beuscart-Zephir MC, Anceaux F, Menu H, Guerlinger S, Watbled L, Evrard F. User-centred, multidimensional assessment method of Clinical Information Systems: a case-study in anaesthesiology. Int J Med Inform 2005; Mar; 74 2-4 179-89.
• 77 Wetterneck TB, Skibinski KA, Roberts TL, Kleppin SM, Schroeder ME, Enloe M. et al. Using failure mode and effects analysis to plan implementation of smart i.v. pump technology. Am J Health Syst Pharm 2006; Aug 15; 63 (16) 1528-38.
• 78 Hundt AS, Carayon Pascale, Wetterneck BTosha. HRO characteristics as demonstrated through implementation of a smart IV pump. Proceedings of the IEA 2006 Congress. 2006
• 79 Baxter GD, Monk AF, Tan K, Dear PRF, Newell SJ. Using cognitive task analysis to facilitate the integration of decision support systems into the neonatal intensive care unit. Artif Intell Med 2005; Nov; 35 (03) 243-57.
• 80 Weir CR, Hicken BL, Nebeker J, Campo R, Drews F, Lebar B. A Cognitive Task Analysis of Information Management Strategies in a Computerized Provider Order Entry Environment. J Am Med Inform Assoc. 2006 Oct 26
• 81 Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM. Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. J Biomed Inform 2005; Jun; 38 (03) 189-99.
• 82 Zhang J, Norman DA. Representation in distributed cognitive tasks. Cognitive Science 1994; 18 (01) 87-122.
• 83 Patel VL. Distributed and collaborative cognition in healthcare: implications for systems development. Artif Intell Med. 1998 12. (2)
• 84 Xiao Y. Artifacts and collaborative work in healthcare: methodological, theoretical, and technological implications of the tangible. J Biomed Inform 2005; Feb; 38 (01) 26-33.
• 85 Zhang J, Patel VL, Johnson TR, Shortliffe EH. A cognitive taxonomy of medical errors. J Biomed Inform 2004; Jun; 37 (03) 193-204.
• 86 Patterson ES, Rogers ML, Chapman RJ, Render ML. Compliance with intended use of Bar Code Medication Administration in acute and long-term care: an observational study. Hum Factors 2006; 48 (01) 15-22.
• 87 Wolf LD, Potter P, Sledge JA, Boxerman SB, Grayson D, Evanoff B. Describing nurses’ work: combining quantitative and qualitative analysis. Hum Factors 2006; 48 (01) 5-14.
• 88 Pizziferri L, Kittler AF, Volk LA, Honour MM, Gupta S, Wang S. et al. Primary care physician time utilization before and after implementation of an electronic health record: a time-motion study. J Biomed Inform 2005; Jun; 38 (03) 176-88.
• 89 Actogram Kronos [computer program]. 2001
• 90 Carayon P, Gurses AP. A human factors engineering conceptual framework of nursing workload and patient safety in intensive care units. Intensive Crit Care Nurs 2005; Oct; 21 (05) 284-301.
• 91 Carayon Pascale, Wetterneck BTosha, Sears A, Ozkaynak Mustatfa, Ram Prashant, De Silvey Joshua. et al. Observing nurse interaction with medication administration technologies. In: Henriksen K. editor. Rockville, MD: Agency for Healthcare Research and Quality; 2005: 349-64.
• 92 Carayon P, Alvarado CJ, Hundt AS. Work system design in Health Care. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 61-79.
• 93 Mackenzie CF, Xiao Y. Video analysis in Health Care. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 663-79.
• 94 Zheng K, Padman R, Johnson MP, Diamond HS. Understanding technology adoption in clinical care: Clinician adoption behavior of a point-of-care reminder system. Int J Med Inform 2005; Aug; 74 7-8 535-43.
• 95 Gurses PAyse. Carayon Pascale. Performances obstacles and nursing workload in intensive care units: a cluster analysis approach. Proceedings of the IEA 2006 Congress: Elsevier. 2006
• 96 Lane R, Stanton ANeville, Harrison D. Applying hierarchical task analysis to medication administration errors. Applied Ergonomics 2006; 37: 669-79.
• 97 Rinkus SM, Chitwood A. Cognitive analyses of a paper medical record and electronic medical record on the documentation of two nursing tasks: patient education and adherence assessment of insulin administration. Proc AMIA Symp. 2002: 657-61.
• 98 Rogers M, Patterson ES, Woods DD. Cognitive work analysis in Health Care. In: Carayon, editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 601-15.
• 99 Miller RA, Waitman LR, Chen S, Rosenbloom ST. The anatomy of decision support during inpatient care provider order entry (CPOE): empirical observations from a decade of CPOE experience at Vanderbilt. J Biomed Inform 2005; Dec; 38 (06) 469-85.
• 100 Endsley M. Measurement of situation awareness in dynamic systems. Human Factors 1995; 37: 65-84.
• 101 Chi MTH, Feltovich PJ, Glaser R. Categorization and representation of physics problems by experts and novices. Cognitive Science 1981; 05: 121-52.
• 102 Card Sword [computer program]. 2005
• 103 Ericsson KA, Simon HA. Protocol Analysis: verbal reports as data. Cambridge, MA: MIT Press; 1984
• 104 Marshall EC, Duncan KD, Baker SM. Thje role of withheld information in the training of process plant fault diagnosis. Ergonomics 1981; 24 (09) 711-24.
• 105 Graesser AC, Robertson SP, Anderson PA. Incorporating inferences in narratives representations: a study of how and why. Cognit Psychol 1981; 13: 343-70.
• 106 Sébillotte S. Methodology guide to task analysis with the goal of extracting relevant characteristics for human-computer interfaces. Int J Hum Comput Interact 1995; 07: 341-63.
• 107 Carthey J. The role of structured observational research in health care. Qual Saf Health Care 2003; Dec 1; 12 (90002): 13ii-16.
• 108 Rubin J. Handbook of usability testing. New York: John Wiley & Sons; 1194
• 109 Wixon D, Wilson C. The usability engineering framework for product design and evaluation. In: Helander MG, Landauer TK, Prabhu PV. editors. Handbook of Human-Computer Interaction. Amsterdam: Elsevier; 1997: 653-88.
• 110 Nielsen J. Usability Engineering. Boston: Academic Press; 1993
• 111 Dumas JS. User-based evaluations. In: Jacko JA, Sears A. editors. The Human-Computer Interaction Handbook. Mahwah, NJ: Lawrence Erlbaum Associates; 2003: 1093-117.
• 112 Cockton GC, Lavery Darryn, Woolrych Alan. Inspection-based evaluations. In: Jacko JA, Sears A. editors. The Human-Computer Interaction Handbook. Mahwah, NJ: Lawrence Erlbaum Associates; 2003: 1118-38.
• 113 Lewis C, Wharton C. Cognitive walkthroughs. In: Helander MG, Landauer TK, Prabhu PV. editors. Handbook of Human-Computer Interaction. Amsterdam: Elsevier; 1997: 717-32.
• 114 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; Feb; 37 (01) 56-76.
• 115 Borycki E, Kushniruk A. Identifying and preventing technology-induced error using simulations: application of usability engineering techniques. Healthc Q. 2005 08. Spec No:99-105
• 116 Jaspers MW, Steen T, Van den BC, Geenen M. The think aloud method: a guide to user interface design. Int J Med Inform 2004; Nov; 73 11-12 781-95.
• 117 Kaufman DR, Patel VL, Hilliman C, Morin PC, Pevzner J, Weinstock RS. et al. Usability in the real world: assessing medical information technologies in patients’ homes. J Biomed Inform 2003; Feb; 36 1-2 45-60.
• 118 Elkin PL, Sorensen Barb, De Palo Diane, Poland Greg, Bailey Kent, Wood Douglas. et al. Optimization of a Research Web nvironment for Academic Internal Medicine Faculty. J Am Med Inform Assoc 2002; 09: 472-8.
• 119 Kushniruk AW, Borycki EM. Low-cost rapid usability engineering: designing and customizing usable healthcare information systems. Healthc Q 2006; 09 (04) 98-100 102.
• 120 Cimino JJ, Patel VL, Kushniruk AW. Studying the Human – Computer – Terminology Interface. J Am Med Inform Assoc 2001; 08 (02) 163-73.
• 121 Gosbee J, Gosbee LL. Usability evaluation in Health Care. In: Carayon P. editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. New Jersey: Lawrence Erlbaum Associates; 2007: 679-93.
• 122 McKnight LK, Elkin PL, Ogren PV, Chute CG. Barriers to the clinical implementation of compositionality. Proc AMIA Symp 1999; 320-4.
• 123 Allen M, Currie LM, Bakken S, Patel VL, Cimino JJ. Heuristic evaluation of paper-based Web pages: a simplified inspection usability methodology. J Biomed Inform 2006; Aug; 39 (04) 412-23.
• 124 Beuscart-Zephir MC, Leroy N, Alao O, Darmoni S. Usability assessment study of a web site displaying medical resources on line: the CISMeF. Stud Health Technol Inform 2002; 90: 133-7.
• 125 Peute LWP, Jaspers MWM. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Medi Inform. In Press 2007
• 126 Molich Rolf, Ede Meghan, Kaasgaard Klaus, Karyukin Barbara. Comparative usability evaluation. Behaviour and Information Technology 2004; 23 (01) 65-74.
• 127 Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Inform Assoc 2005; Jul; 12 (04) 438-47.
• 128 Patterson ES, Nguyen AD, Halloran JP, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc 2004; Jan; 11 (01) 50-9.
• 129 Elkin PL, Mohr DN, Tuttle MS, Cole WG, Atkin GE, Keck K. et al. Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System. Proc AMIA Annu Fall Symp. 1997: 500-4.
• 130 Agency for Healthcare Research and Quality. http://www ahrq gov/2007 Available from: URL: http://www.ahrq.gov/