Human Factors and Organizational Issues

Summary

Objective: To select the best papers that made original and high impact contributions in the area of human factors and organizational issues in biomedical informatics in 2019.

Methods: A rigorous extraction process based on queries from Web of Science® and PubMed/Medline was conducted to identify the scientific contributions published in 2019 that address human factors and organizational issues in biomedical informatics. The screening of papers on titles and abstracts independently by the two editors led to a total of 30 papers. These papers were discussed for a selection of 15 finalist papers, which were then reviewed by the two editors and by three external reviewers from internationally renowned research teams.

Results: The query process resulted in 626 papers that reveal interesting and rigorous methods and important studies in human factors that move the field forward, particularly in clinical informatics and emerging technologies such as brain-computer interfaces. This year three papers were clearly outstanding and help advance the field. They provide examples of applying existing frameworks together in novel and highly illuminating ways, showing the value of theory development in human factors.

Conclusion: The selected papers make important contributions to human factors and organizational issues, expanding and deepening our knowledge of how to apply theory and applications of new technologies in health.

Keywords

Human factors - human-computer interaction - organizational issues - patient safety - health information technology

Introduction

Papers published in 2019 on human factors and organizational issues (HFOI) show the emergence of new technologies, such as the multifaceted app suite for chronic disease self-management, brain-computer interface evaluation, and new combinations or improvement of existing methods and theories, particularly in analyzing workflows in clinical care and the cognitive burdens involved. None of the papers is explicitly focused on ethics, though much of the work has ethical implications in improving both provider and system performance and understanding diagnostic delays and errors. Papers show a certain maturity in the development of theory and applications in highly specific practical contexts.

The selection of papers on HFOI intends to illuminate current progress of HFOI research in biomedical informatics. The selected papers offer a glimpse of the quality and breadth of the work being conducted in the HFOI community.

Methods

We collected papers from 2019 by a standard search in PubMed and Web of Science®. Searches were performed in October and December 2019 to identify peer-reviewed journal articles published in 2019, in the English language, related to HFOI research in biomedical informatics. In addition to the search of electronic databases, manual searches of key themes were performed in major biomedical journals (e.g., Journal of the American Medical Informatics Association, Methods of Information in Medicine, Journal of Medical Internet Research, etc.).

We used free-text and coded keywords. Free-text keywords were listed in with respect to the questions addressed by the section. Relevant MeSH terms were also identified, and PubMed test queries were made iteratively. Two final queries were built, one based on MeSH terms for major topics and one on free-text keywords searched in titles or abstracts via PubMed and Web of science®. This process led to an initial set of 626 papers that the two section editors independently evaluated using papers’ titles and abstracts. Each editor chose the 15 papers considered as the best list of candidate best papers. The two selections overlapped, and the set was finally reduced to 15 papers by discussion. These were evaluated by both section editors and three other external reviewers, with scoring based on the topic’s importance to medical informatics, scientific and/or practical impact, quality of scientific work, originality and innovativeness, coverage of related literature, organization and clarity of thought, and whether or not to include the paper as a best paper. Three papers were considered as being ‘best papers’ on the basis of their substantially higher scores by all reviewers and the agreement that they were excellent examples of HFOI studies that break new ground and add to our tools to advance the field ([Table 1]). A content summary of the selected best papers can be found in the appendix of this synopsis.

Table 1 Best paper selection of articles for the IMIA Yearbook of Medical Informatics 2020 in the section ‘Human Factors and Organizational Issues’. The articles are listed in alphabetical order of the first author’s surname.

Section HumanFactors and Organizational Issues

▪ Lee JY, van Karnebeek CDM, Wasserman WW. Development and user evaluation of a rare disease gene prioritization workflow based on cognitive ergonomics. J Am Med Inform Assoc 2019;26(2):124–33. ▪ Wang J, Liang H, Kang H, Gong Y. Understanding health information technology induced medication safety events by two conceptual frameworks. Appl Clin Inform 2019;10:158–67. ▪ Patterson ES, Su G, Sarkar U. Reducing delays to diagnosis in ambulatory care settings: A macrocognition perspective. Appl Ergon2020;82:102965.

Conclusions

Noteworthy papers published in 2019 provide examples of applying existing frameworks together in novel and highly illuminating ways, showing the value of theory development in human factors.

Wang et al., [1] applied the Sittig and Singh sociotechnical basis of errors framework [2] and Coeira’s information value framework [3] together to analyze US FDA (Food and Drug Administration) MAUDE (Manufacturer and User Facility Device Experience) data on health-IT related errors and near-misses. The dual analysis strategy permitted to understand and classify the main causes of error in this area, then to apply data-related causes to analyze how missing, partial, null, or false data caused downstream errors, calculate the probabilities and help identify the major impacts on outcomes. This analysis provides an excellent method to prioritize which types of errors should be dealt with early, and helps understand overall causation. It also paves the way for a formal reporting template and develops a knowledge base supported by case-based reasoning.

Lee et al., [4] aimed to understand and leverage a combination of cognitive prototype approach with computational approaches, using the best of each. Prototypical thinking is common in clinical diagnosis and genetic counseling. The authors showed that a tool which elicits prototypes before asking the geneticist to identify disease phenotypes had the same accuracy but took less time. Combining tools with both strategies would be optimal, leveraging the cognitive characteristics of experts. The paper is a nice illustration of the understanding and formalization of cognitive processes and how to draw lessons to design computer-assisted gene prioritization to augment clinical reasoning and cognitive characteristics of experts.

The aim of Patterson et al., [5] was to increase system resilience by using macrocognition concepts to analyze ambulatory care work in a partnership between clinicians and human factors experts. This helped identify a set of resilience strategies and ways in which information systems can be modified to increase safety. It augments the existing Systems Engineering Initiative for Patient Safety (SEIPS) theoretical framework [6], bridges disciplines, and suggests design solutions.

Although not selected as best papers, the remaining candidate best papers were just as interesting. They also bring a real added value for the HFOI community.

Sumak et al., [7] present a very interesting study that contributes to the body of research and design for hands-free computer interaction, in particular for motor-impaired users who need innovative solutions for interacting with Information and Communication Technology (ICT). The authors aimed at investigating the benefits and drawbacks of a solution for touchless interaction with a computer using the EPOC+ brain-computer interface at home which allowed to measure and combine EEG and EMG signals with data from facial expressions. The authors studied user perceptions in terms of acceptance and use of the solution with a 5-step protocol that compared the user’s current human computer interaction approach with the EPOC+, with 18 participants (10 non-disabled and 8 disabled). An interesting initial finding is that non-disabled users experienced significantly more difficulty using the EPOC+, although both groups required more time to use EPOC+ than their usual solution.

Chen et al., [8] made use of a multi-function suite of 13 clinical apps to address depression and other mental health issues, by measuring experience and user engagement via usage patterns, mechanisms of change, and usability. They used log data collected during treatment delivery, integrating text and visual analytics in a novel approach aiding analysis of a rich complex dataset. They found four clusters of different types of users based on app usage patterns, and compared them in terms of engagement and associated factors, highlighting mechanisms of change and usability perceptions. The paper has both practical and scientific value: it contributes to richer methods for understanding user engagement more holistically and it contains a detailed description of a log analysis work which may be helpful.

Giordanengo et al., [9] evaluated the design of a diabetes dashboard displaying patient-collected data to clinicians, using iterative participatory design with 14 clinicians. The study permits the assessment of the reliability of self-collected data, calculations, and highlighted conditions requiring investigation. 100% of participants deemed it useful for daily use. The study relies on an iterative participatory process involving workshops with patients, clinicians, and researchers to define which information should be available and how it should be displayed. Several methods are used to address main barriers to the use of self-collected data, such as visualization and scenarios, and the identification of medical events causes. The studied system does not limit the integration of data to specific companies or types of sensors, and includes personal goals. For all these reasons, this paper contributes greatly to HF tools for addressing chronic disease.

Yera et al., [10] characterized medication dashboard use by 35 users over 10 months. They were able to distinguish pharmacist from non-pharmacist use, using features tied to exploration and dwell time via user interface events, with 80% accuracy using multilayer perception giving the highest precision and recall values. Primary users showed distinctive behavior with lower time between clicks on overview screens, and less visual search activity on patient risk screens and visualizations. This is a contribution showing that using machine learning techniques to distinguish between classes of users both helps understand the differences in their behavior and model useful user interface improvements or customizations. The study includes new workflows to group population health monitoring activities and nudges to increase engagement.

Businger et al., [11] described the Patient Safety Learning Lab approach, implemented in 12 units of a large academic medical center over 18 months, impacting 12,000 patient admissions. The paper describes implementation challenges and the approach in designing and implementing a tool suite integrated with a major commercial EHR to improve safety. It contributes to our knowledge of how to change health system culture and provide recommendations for each part of institutional structures. It provides a description of how one center implemented novel tools in a commercial EHR, using APIs and real-time data extraction, and thinking outside current workflows. Not all institutions have the same resources but the paper provides examples of execution in a certain type of system.

Richardson et al., [12] conducted a study of live usability testing which showed how even a small study may reveal more different problems in ‘live’ in situ testing than in usual laboratory think-aloud testing, providing additional areas in which system redesign was needed, particularly for clinical decision support. This calls attention to the need for flexible and widely applicable clinical decision support tools to achieve the fastest way to perform tasks.

Want et al., [13] study concerned work-centered usability methods to evaluate integrated patient-focused status displays for Emergency Department (ED) communication and situational awareness. The study uses multiple design methods to show ED data in a prioritized and highly cognitively supporting way, using focus groups, surveying, and scenario-based study. There are some limitations with the static tests, small numbers, and incomplete functions of the prototype, but the paper provides an interesting insight.

Setiawan et al., [14] provided an example of testing for a very interesting and specialized app architecture, in this case aimed at patients with chronic conditions and disabilities (PwCCDs). This allows studying a highly customizable app suite with clinician- and patient-facing components in a large study via survey, and performing feasibility studies with smaller numbers, in real deployment. It shows means of usability testing of more complex multi-stakeholder applications with shifting components and stakeholder levels of participation, and showed clear relationship of usability with use.

Woolridge et al., [15] provided a fascinating paper with a detailed and groundbreaking large study of pediatric trauma care processes, highlighting and providing techniques to understand the true complexity of processes across the multiple levels found in healthcare. The authors argued for shifting from eliminating to coping with complexity, tools for awareness, reliance and adaptation across individuals, team and organizational levels, with sociotechnical solutions. The paper greatly contributes to the understanding of how to realistically improve care and safety in complex settings.

Cho et al., [16] demonstrated an improvement on usual eye-tracking protocols by devising a retrospective think-aloud, and related findings from eye tracking measures to success of failure in task completion. The new method asked participants to share their inner experience using the app and explain their experience and gaze patterns during errors making it possible to catch instances in which the user had more confidence than warranted, i.e., problems that would be undetected in traditional usability testing.

Kutney-Lee et al., [17] presented a large (~12,000 person) study of nurses in 353 hospitals, using regression models to assess EHR adoption level, work environment, and usability outcomes. The paper illuminates our understanding of how EHR implementation and the work environment affect usability in practice, helping evaluate their effects on care quality. The study found that nurses are still often not included in selection, modification, and other aspects of EHR deployment, with high rates of dissatisfaction, and that advanced functionality of EHRs is needed to achieve desired effects, with satisfaction increasing with the EHR level of sophistication.

Acknowledgement

We would like to acknowledge the support of Adrien Ugon, Martina Hutter, Kate Fultz Hollis, Lina Soualmia, Brigitte Séroussi, and the whole Yearbook Editorial Committee as well as the reviewers for their contribution to the selection process of the Human Factor and Organizational Issues section for the IMIA Yearbook 2020.

• References

• 1 Wang J, Liang H, Kang HY, Gong Y. Understanding Health Information Technology Induced Medication Safety Events by Two Conceptual Frameworks. Appl Clin Inform 2002; 10 (01) 158-67
• 2 Sittig D, Singh H. A New Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems. Qual Saf Health Care 2010; 19 (suppl 3): i68-i74
• 3 Coiera E. A new informatics geography. Yearb Med Inform 2016; ;(01): 251-5
• 4 Lee JJY, van Karnebeek CDM, Wasserman WW. Development and User Evaluation of a Rare Disease Gene Prioritization Workflow Based on Cognitive Ergonomics. J Am Med Inform Assoc 2019; 26 (02) 124-33
• 5 Patterson ES, Su G, Sarkar U. Reducing Delays to Diagnosis in Ambulatory Care Settings: A Macrocognition Perspective. Appl Ergon 2020; 82: 102965
• 6 Carayon P, Hundt AS, Karsh BT, Gurses AP, Smith AM, Brennan PF. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (Suppl 1): i50-8
• 7 Šumak B, Špindler M, Debeljak M, Heriko M, Pušnik M. An empirical evaluation of a hands-free computer interaction for users with motor disabilities. J Biomed Inform 2019; 96: 103249
• 8 Chen AT, Wu S, Tomasino KN, Lattie EG, Mohr DC. A Multi-Faceted Approach to Characterizing User Behavior and Experience in a Digital Mental Health Intervention. J Biomed Inform 2019; 94: 103187
• 9 Giordanengo A, Årsand E, Woldaregay AZ, Bradway M, Grottland A, Hartvigsen G. et al. Design and Prestudy Assessment of a Dashboard for Presenting Self-Collected Health Data of Patients With Diabetes to Clinicians: Iterative Approach and Qualitative Case Study. JMIR Diabetes 2019; 4 (03) e14002
• 10 Year A, Muguerza J, Arbelaitz O, Perona I, Keers RN, Ashcroft DM. et al Modelling the interactive behaviour of users with a medication safety dashboard in a primary care setting. Int J Med Inform 2019; 129: 395-403
• 11 Businger AC, Fuller TE, Schnipper JL, Rossetti SC, Schnock KO, Rozenblum R. et al. Lessons Learned Implementing a Complex and Innovative Patient Safety Learning Laboratory Project in a Large Academic Medical Center. J Am Med Inform Assoc 2020; 27 (02) 301-7
• 12 Richardson S, Feldstein D, McGinn T, Park LS, Khan S, Hess R. et al. Live Usability Testing of Two Complex Clinical Decision Support Tools: Observational Study. JMIR Hum Factors 2019; 6 (02) e12471
• 13 Wang X, Kim TC, Hegde S, Hoffman DJ, Benda NC, Franklin ES. et al. Design and Evaluation of an Integrated, Patient-Focused Electronic Health Record Display for Emergency Medicine. Appl Clin Inform 2019; Aug; 10 (04) 693-706
• 14 Setiawan MA, Zhou L, Alfikri Z, Saptono A, Fairman AD, Dicianno BE. et al. An Adaptive Mobile Health System to Support Self-Management for Persons With Chronic Conditions and Disabilities: Usability and Feasibility Studies. JMIR Form Res 2019; 3 (02) e12982
• 15 Wooldridge AR, Carayon P, Hoonakker P, Hose BZ, Ross J, Kohler JE. et al. Complexity of the pediatric trauma care process: implications for multi-level awareness. Cogn Technol Work 2019; 21 (03) 397-416
• 16 Cho H, Powell D, Pichon A, Kuhns LM, Garofalo R, Schnall R. Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation. Int J Med Inform 2019; 129: 366-73
• 17 Kutney Lee A, Sloane DM, Bowles KH, Burns LR, Aiken LH. Electronic Health Record Adoption and Nurse Reports of Usability and Quality of Care: The Role of Work Environment. Appl Clin Inform 2019; Jan; 10 (01) 129-39