Current Challenges and Opportunities for Better Integration of Human Factors Research with Development of Clinical Information Systems

Summary

Objectives Clinical information system (CIS) developers and implementers have begun to look to other scientific disciplines for new methods, tools, and techniques to help them better understand clinicians and their organizational structures, clinical work environments, capabilities of clinical information and communications technology, and the way these structures and processes interact. The goal of this article is to help CIS researchers, developers, implementers, and evaluators better understand the methods, tools, techniques, and literature of the field of human factors.

Methods We developed a framework that explains how six key human factors topics relate to the design, implementation, and evaluation of CISs.

Results Using this framework we discuss the following six topics: 1) informatics and patient safety; 2) user interface design and evaluation; 3) workflow and task analysis; 4) clinical decision making and decision support; 5) distributed cognition; and 6) mental workload and situation awareness.

Conclusions Integrating the methods, tools, and lessons learned from each of these six areas of human factors research early in CIS design and incorporating them iteratively during development can improve user performance, user satisfaction, and integration into clinical workflow. Ultimately, this approach will improve clinical information systems and healthcare delivery.

Keywords

Human factors - clinical information systems - computerized medical record - human information processing - clinical decision support systems

 

• References

• 1 Linder JA, Ma J, Bates DW, Middleton B, Stafford RS. Electronic health record use and the quality of ambulatory care in the United States. Arch Intern Med 2007; 167 (13) 1400-5.
• 2 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE. et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-203.
• 3 Congressional Budget Office: Evidence on the Costs and Benefits of Health Information Technology. May 2008 Available at: http://www.cbo.gov/ftpdocs/ 91xx/doc9168/05-20-HealthIT.pdf
• 4 Carayon P, Schoofs HA, Karsh BT, Gurses AP, Alvarado CJ, Smith M. et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (Suppl. 01) i50-i58.
• 5 Cochran GL, Jones KJ, Brockman J, Skinner A, Hicks RW. Errors prevented by and associated with bar-code medication administration systems. Jt Comm J Qual Patient Saf 2007; May; 33 (05) 293-301 245
• 6 Morriss Jr FH. Effectiveness of a Barcode Medication Administration System in Reducing Preventable Adverse Drug Events in a Neonatal Intensive Care Unit: A Prospective Cohort Study. 2008
• 7 Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The Effect of Electronic Prescribing on Medication Errors andAdverse Drug Events: A Systematic Review. J Am Med Inform Assoc 2008; 15 (05) 585-600.
• 8 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; MarApr; 11 (02) 104-12.
• 9 Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003; 348 (25) 2526-34.
• 10 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006; 13 (05) 547-56.
• 11 Weiner JP, Kfuri T, Chan K, Fowles JB. “eIatrogenesis”: the most critical unintended consequence of CPOE and other HIT. J Am Med Inform Assoc 2007; 14 (03) 387-88.
• 12 Zhang J, Patel VL, Johnson TR, Shortliffe EH. A cognitive taxonomy of medical errors. J Biomed Inform 2004; 37 (03) 193-204.
• 13 Horsky J, Kuperman GJ, Patel VL. Comprehensive analysis of a medication dosing error related to CPOE. J Am Med Inform Assoc 2005; 12 (04) 377-82.
• 14 Hammond KW, Helbig ST, Benson CC, BrathwaiteSketoe BM. Are electronic medical records trustworthy? Observations on copying, pasting and duplication. AMIA Annu Symp Proc 2003; 269-73.
• 15 Hartzband P, Groopman J. Off the record—avoiding the pitfalls of going electronic. N Engl J Med 2008; 358 (16) 1656-8.
• 16 Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct text entry in electronic progress notes. An evaluation of input errors. Methods Inf Med 2003; 42 (01) 61-7.
• 17 Glassman PA, Belperio P, Simon B, Lanto A, Lee M. Exposure to automated drug alerts over time: effects on clinicians’ knowledge and perceptions. of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-47.
• 18 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-47.
• 19 Cook RI, Woods DD. Implications of automation surprises in aviation for the future of total intravenous anesthesia (TIVA). J Clin Anesth 1996; 08 (03) 29S-37S.
• 20 Sarter NB, Woods DD, Billings CE. Automation Surprises. In Handbook of Human Factors and Ergonomics. Edited by Salvendy G. Wiley; 1997
• 21 Karsh BT, Holden RJ, Alper SJ, Or CK. A human factors engineering paradigm for patient safety: designing to support the performance of the healthcare professional. Qual Saf Health Care 2006; 15 (Suppl. 01) i59-i65.
• 22 Amalberti R, Vincent C, Auroy Y, de Saint MG. Violations and migrations in health care: a framework for understanding and management. Qual Saf Health Care 2006; 15 (Suppl. 01) i66-i71.
• 23 Koppel R, Wetterneck T, Telles JL, Karsh BT. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc 2008; 15 (04) 408-23.
• 24 Niazkhani Z. Evaluating the impact of CPOE systems on medical workflow: a mixed method study. Stud Health Technol Inform 2008; 136: 881-2.
• 25 Saleem JJ, Russ AL, Justice CF, Hagg H, Ebright PR, Woodbridge PA. et al. Exploring the persistence of paper with the electronic health record. Int J Med Inform. in press.
• 26 Unertl KM, Weinger MB, Johnson KB. Applying direct observation to model workflow and assess adoption. AMIA Annu Symp Proc 2006; 794-8.
• 27 Reason J. Beyond the organisational accident: the need for “error wisdom” on the frontline. Qual Saf Health Care 2004; 13 (Suppl. 02) ii28-ii33.
• 28 Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. Mahwah, NJ: Lawrence Erlbaum Associates; 2007
• 29 Walker JM, Carayon P, Leveson N, Paulus RA, Tooker J, Chin H. et al. EHR safety: the way forward to safe and effective systems. J Am Med Inform Assoc 2008; 15 (03) 272-7.
• 30 Tucker AL, Edmondson AC. Why Hospitals Don’t Learn from Failures: Organizational and Psychological Dynamics that Inhibit System Change. California Management Review 2003; 45 (02) 55-72.
• 31 Render ML. Research and redesign are safer than warnings and rules. Crit Care Med 2004; 32 (04) 1074-5.
• 32 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76.
• 33 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1-2): 23-30.
• 34 Chung PH, Zhang J, Johnson TR, Patel VL. An extended hierarchical task analysis for error prediction in medical devices. AMIA Annu Symp Proc 2003; 165-9.
• 35 Kirwan B, Ainsworth LK. A Guide to Task Analysis. Bristol, PA: Taylor & Francis, Inc; 1992
• 36 Lane R, Stanton NA, Harrison D. Applying hierarchical task analysis to medication administration errors. Appl Ergon 2006; 37 (05) 669-79.
• 37 DeRosier J, Stalhandske E, Bagian JP, Nudell T. Using health care Failure Mode and Effect Analysis: the VA National Center for Patient Safety’s prospective risk analysis system. Jt Comm J Qual Improv 2002; 28 (05) 248-67 209.
• 38 Ash JS, Sittig DF, McMullen CK, Guappone K, Dykstra R, Carpenter J. A rapid assessment process for clinical informatics interventions. AMIA Annu Symp Proc 2008; 26-30.
• 39 Zhan C, Hicks RW, Blanchette CM, Keyes MA, Cousins DD. Potential benefits and problems with computerized prescriber order entry: analysis of a voluntary medication error-reporting database. Am J Health Syst Pharm 2006; 63 (04) 353-8.
• 40 Gosbee J, Anderson T. Human factors engineering design demonstrations can enlighten your RCA team. Qual Saf Health Care 2003; 12 (02) 119-21.
• 41 Eastwood G. ICT opportunities in healthcare. In: Business Insights; London: 2005: 163.
• 42 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-12.
• 43 Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lessons still to be learnt. BMJ 2003; 326 (7394): 860-3.
• 44 Schulman J, Kuperman GJ, Kharbanda A, Kaushal R. Discovering how to think about a hospital patient information system by struggling to evaluate it: a committee’s journal. J Am Med Inform Assoc 2007; 14 (05) 537-41.
• 45 Myers B. Challenges of HCI design and implementation. Interactions 1994; 01 (01) 73-83.
• 46 Burmester M, Machate J. Creative design of interactive products and use of usability guidelines—A contradiction?. In: Jacko JA, Stephanidis C, Harris D, Mahwah NJ. editors. Human-computer interaction: Theory and practice. NJ: Lawrence Erlbaum Associates; 2003
• 47 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-23.
• 48 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care-an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-9.
• 49 Runyon B, Handler T, Hieb B, Shaffer V, Jonathan E, Lovelock JD. et al. Hype cycle for healthcare provider applications and systems. In: Gartner Industry Report ID Number G00148329 edition. Stamford, CT: Gartner Inc; 2007
• 50 Preece J, Sharp H, Rogers Y. Interaction Design; Beyond human-computer interaction (2nd edition). New York: Wiley; 2007
• 51 Pilemalm S, Timpka T. Third generation participatory design in health informatics—making user participation applicable to large-scale information system projects. J Biomed Inform 2008; 41 (02) 327-39.
• 52 McGowan JJ, Cusack CM, Poon EG. Formative evaluation: a critical component in EHR implementation. J Am Med Inform Assoc 2008; 15 (03) 297-301.
• 53 Westbrook JI, Braithwaite J, Georgiou A, Ampt A, Creswick N, Coiera E. et al. Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory. J Am Med Inform Assoc 2007; 14 (06) 746-55.
• 54 Karsh BT. Beyond usability: designing effective technology implementation systems to promote patient safety. Qual Saf Health Care 2004; 13 (05) 388-94.
• 55 Liljegren E. Usability in a medical technology context assessment of methods for usability evaluation of medical equipment. International Journal of Industrial Ergonomics 2006; 36: 345-52.
• 56 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-53.
• 57 Peute LW, Jaspers MW. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inform 2007; 76 (2-3): 157-68.
• 58 Bernonville S, Kolski C, Leroy N, Beuscart-Zephir MC. Integrating the SE and HCI models in the human factors engineering cycle for re-engineering Computerized Physician Order Entry systems for medications: Basic principles illustrated by a case study. Int J Med Inform. In press. 2008
• 59 Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005; 38 (01) 75-87.
• 60 Benyon D, Turner P, Turner S. Designing interactive systems: People, activities, contexts. Harlow, UK: Addison-Wesley; 2005
• 61 Bias R, Mayhew D. Cost-justifying usability: An update in the internet age. New York: Morgan-Kaufmann; 2005
• 62 Dumas J, Redish J. A practical guide to usability testing. Intellect. 1999
• 63 Reid PP, Compton D, Grossman JH, Fanjiang G. Committee on Engineering and the Health Care System. Building a better delivery system: A new engineering/health care partnership. In: Washington, D.C: National Academies Press; 2005
• 64 Sharp A, McDermott P. Workflow Modeling: Tools for Process Improvement and Application Development. 2nd Edition.. Artech House Publishers; 2008
• 65 Hackos JTP, Redish JC. User and Task Analysis for Interface Design. 1st ed.. Wiley; 1998
• 66 Hazlehurst B, McMullen C, Gorman P, Sittig D. How the ICU follows orders: care delivery as a complex activity system. AMIA Annu Symp Proc 2003; 284-8.
• 67 Sittig DF, Ash JS, Zhang J, Osheroff JA, Shabot MM. Lessons from “Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system”. Pediatrics 2006; 118 (02) 797-801.
• 68 Vicente KJ. Cognitive Work Analysis. Lawrence Erlbaum Associates. 1999
• 69 Booch G, Rumbaugh J, Jacobson I. Unified Modeling Language User Guide, The. 2nd Edition.. Addison-Wesley Professional; 2005
• 70 Card SK, Moran TP, Newell A. The Psychology of Human-Computer Interaction. Hillsdale, NJ: L. Erlbaum Associates, Inc; 1983
• 71 Butler KA, Zhang J, Esposito C, Bahrami A, Hebron R, Kieras D. Work-centered design: a case study of a mixed-initiative scheduler. In: Proceedings of the SIGCHI conference on Human factors in computing systems. San Jose: CA; 2007: 747-56.
• 72 Wickens CD, Lee JD, Liu Y, Gordon SEBecker. An Introduction to Human Factors Engineering. 2nd Ed.. Upper Saddle River, NJ: Pearson Prentice Hall; 2004
• 73 Wickens CD. Engineering Psychology and Human Performance. 2nd Ed.. NewYork, NY: HarperCollins Publishers, Inc; 1992
• 74 Zsambok CE, Klein G. Naturalistic Decision Making. Mahwaw, NJ: Erlbaum; 1997
• 75 Barnett GO, Cimino JJ, Hupp JA, Hoffer EP. DXplain. An evolving diagnostic decision-support system. JAMA 1987; 258 (01) 67-74.
• 76 Graber ML, Mathew A. Performance of a web-based clinical diagnosis support system for internists. J Gen Intern Med 2008; 23 (Suppl. 01) 37-40.
• 77 Friedman CP, Elstein AS, Wolf FM, Murphy GC, Franz TM, Heckerling PS. et al. Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA. 1999; 282 (19) 1851-6.
• 78 Evans RS, Pestotnik SL, Classen DC, Clemmer TP, Weaver LK, Orme Jr JF. et al. A computerassisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998; 338 (04) 232-8.
• 79 Gardner RM, Cannon GH, Morris AH, Olsen KR, Price WG. Computerized Blood Gas Interpretation and Reporting System. Computer 1975; 08 (01) 39-45.
• 80 Sittig DF, Pace NL, Gardner RM, Beck E, Morris AH. Implementation of a computerized patient advice system using the HELP clinical information system. Comput Biomed Res 1989; 22 (05) 474-87.
• 81 Groopman JE. How Doctors Think. Boston: Houghton-Mifflin; 2007
• 82 Bates DW, Pappius E, Kuperman GJ, Sittig D, Burstin H, Fairchild D. et al. Using information systems to measure and improve quality. Int J Med Inform 1999; 53 (2-3): 115-24.
• 83 Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma’Luf N. et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 06 (04) 313-21.
• 84 Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc 2001; 08 (04) 299-308.
• 85 Doebbeling BN, Chou AF, Tierney WM. Priorities and strategies for the implementation of integrated informatics and communications technology to improve evidence-based practice. J Gen Intern Med 2006; 21 (Suppl. 02) S50-S57.
• 86 Agrawal A, Mayo-Smith MF. Adherence to computerized clinical reminders in a large healthcare delivery network. Stud Health Technol Inform 2004; 107 (Pt 1): 111-4.
• 87 Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R. et al. Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA 2000; 284 (11) 1411-6.
• 88 Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-38.
• 89 Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M. et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc 2005; 12 (04) 431-7.
• 90 Shea S, DuMouchel W, Bahamonde L. A metaanalysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996; 03 (06) 399-409.
• 91 Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007; 14 (02) 141-5.
• 92 Snyder-Halpern R. Indicators of organizational readiness for clinical information technology/systems innovation: a Delphi study. Int J Med Inform 2001; 63 (03) 179-204.
• 93 Osheroff JA, Pifer EA, Teich JM, Sittig DA, Jenders RA. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. Chicago: HIMSS; 2005
• 94 Dror IE, Harnad S. Cognition Distributed. Amsterdam: John Benjamins Publishing Company; 2008
• 95 Hollan J, Hutchins E, Kirsh D. Distributed cognition: toward a new foundation for human-computer interaction. ACM Transactions on Computer-Human Interaction 2000; 07 (02) 174-96.
• 96 Hutchins E. Cognition in the wild. Cambridge, MA: MIT Press; 1995
• 97 Hutchins E. How a cockpit remembers its speed. Cognitive Science 1995; 19: 265-88.
• 98 Norman DA. Cognitive artifacts. In Designing interaction: Psychology at the human-computer interface. NewYork: Cambridge University Press; 1991
• 99 Rogers Y, Ellis J. Distributed cognition: an alternative framework for analysing and explaining collaborative working. Journal of Information Technology 1994; 09: 119-28.
• 100 Zhang J, Norman DA. Representations in distributed cognitive tasks. Cognitive Science 1994; 18 (01) 87-122.
• 101 Zhang J. Distributed representation as a principle for the analysis of cockpit information displays. International Journal of Aviation Psychology 1997; 07: 105-21.
• 102 Zhang J. A distributed representation approach to group problem solving. Journal of American Society of Information Science 1998; 49 (09) 801-9.
• 103 Patel VL, Kaufman DR, Allen VG, Shortliffe EH, Cimino JJ, Greenes RA. Toward a framework for computer-mediated collaborative design in medical informatics. Methods Inf Med 1999; 38 (03) 158-76.
• 104 Patel VL, Cytryn KN, Shortliffe EH, Safran C. The collaborative health care team: the role of individual and group expertise. Teach Learn Med 2000; 12 (03) 117-32.
• 105 Suchman L. Plans and situated actions: The problem of human-machine communication. New York: Cambridge University Press; 1987
• 106 Wegner DM. Transactive memory: A contemporary analysis of the group mind. In: Mullen B, Goethals GR. editors. Theories of group behavior. NewYork: Springer-Verlag; 1987
• 107 Zhang J. The nature of external representations in problem solving. Cognitive Science 1997; 21: 179-217.
• 108 Patel VL, Zhang J, Yoskowitz NA, Green R, Sayan OR. Translational cognition for decision support in critical care environments: a review. J Biomed Inform 2008; 41 (03) 413-31.
• 109 Nardi BA. Studying context: a comparison of activity theory, situated action models, and distributed cognition. In: Nardi BA, editor. Context and Consciousness: Activity Theory and HumanComputer Interaction. Cambridge, MA: MIT Press 1996; 69-102.
• 110 Wright PC, Fields RE, Harrison MD. Analyzing human-computer interaction as distributed cognition: the resources model. Human-Computer Interaction 2000; 15: 1-41.
• 111 Cohen T, Blatter B, Almeida C, Shortliffe E, Patel VL. A cognitive blueprint of collaboration in context: distributed cognition in the psychiatric emergency department. Artif Intell Med 2006; 37 (02) 73-83.
• 112 Hazlehurst B, Gorman PN, McMullen CK. Distributed cognition: an alternative model of cognition for medical informatics. Int J Med Inform 2008; 77 (04) 226-34.
• 113 Xiao Y. Artifacts and collaborative work in healthcare: methodological, theoretical, and technological implications of the tangible. J Biomed Inform 2005; 38 (01) 26-33.
• 114 Patel V, Zhang J. Distributed cognition in critical care. 2009 Personal Communication
• 115 Selcon SJ, Taylor RM, Koritsas E. Workload or situation awareness? TLX vs. SART for aerospace systems design evaluation. In: Proceedings of the Human Factors Society 35th Annual Meeting 1991. Santa Monica, CA: 1991: 62-6.
• 116 Endsley MR. Design and evaluation for situation awareness enhancement. In: Proceedings of the Human Factors Society 32nd Annual Meeting 1988. Santa Monica, CA: 1988: 97-101.
• 117 Endsley MR. Situation Awareness. In: Salvendy G, editor. Handbook of Human Factors, 3rd Ed. Hoboken, NJ: John Wiley & Sons, Inc 2006; 528-42.
• 118 Tsang P, Wilson GF. Mental Workload. In: Salvendy G, editor. Handbook of Human Factors, 2nd Ed. New York: John Wiley & Sons 1997; 417-49.
• 119 NASA Task Load Index (TLX) for Windows. Available at: http://www.nrl.navy.mil/aic/ide/ NASATLX.php Accessed on: April 27, 2009.
• 120 Hart S, Staveland L. Development of the NASATLX (Task Load Index): Results of empirical and theoretical research. In: Hancock PA, Meshkati N, editors. Human Mental Workload. North-Holland: Elsevier Science Publishers 1988; 139-83.
• 121 Wachter SB, Johnson K, Albert R, Syroid N, Drews F, Westenskow D. The evaluation of a pulmonary display to detect adverse respiratory events using high resolution human simulator. J Am Med Inform Assoc 2006; 13 (06) 635-42.
• 122 Saleem JJ, Patterson ES, Militello L, Anders S, Falciglia M, Wissman JA. et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc 2007; 14 (05) 632-40.
• 123 Singh H, Petersen LA, Thomas EJ. Understanding diagnostic errors in medicine: a lesson from aviation. Qual Saf Health Care 2006; 15 (03) 159-64.
• 124 Endsley MR, Selcon SJ, Hardiman TD, Croft DG. A comparative analysis of SAGAT and SART for evaluations of situation awareness. In: Proceedings of the Human Factors & Ergonomics Society 42nd Annual Meeting 1998. Santa Monica, CA: 1998: 82-6.
• 125 Hogan MP, Pace DE, Hapgood J, Boone DC. Use of human patient simulation and the situation awareness global assessment technique in practical trauma skills assessment. J Trauma 2006; 61 (05) 1047-52.
• 126 Wright MC, Taekman JM, Endsley MR. Objective measures of situation awareness in a simulated medical environment. Qual Saf Health Care 2004; 13 (Suppl. 01) i65-i71.
• 127 Handbook of Human Factors. 3rd Ed.. Wiley & Sons, Inc; 2006