Improving the Effectiveness of Health Information Technology: The Case for Situational Analytics

Abstract

Health information technology has contributed to improvements in quality and safety in clinical settings. However, the implementation of new technologies in health care has also been associated with the introduction of new sociotechnical hazards, produced through a range of complex interactions that vary with social, physical, temporal, and technological context. Other industries have been confronted with this problem and have developed advanced analytics to examine context-specific activities of workers and related outcomes. The skills and data exist in health care to develop similar insights through situational analytics, defined as the application of analytic methods to characterize human activity in situations and identify patterns in activity and outcomes that are influenced by contextual factors. This article describes the approach of situational analytics and potentially useful data sources, including trace data from electronic health record activity, reports from users, qualitative field data, and locational data. Key implementation requirements are discussed, including the need for collaboration among qualitative researchers and data scientists, organizational and federal level infrastructure requirements, and the need to implement a parallel research program in ethics to understand how the data are being used by organizations and policy makers.

Keywords

qualitative - methodologies - evaluation - system improvement - analytics - data access - integration - analysis

Background and Significance

Health information technology (HIT) has contributed to improvements in quality and safety in clinical settings.[1] [2] [3] However, the implementation of new technologies in health care has also been associated with new sociotechnical hazards,[4] which we define as hazards to patients or staff resulting from complex system interactions where the system consists of people, technology, work processes, and the social and physical environment.[5] [6] Seventy-one percent of HIT safety studies in a recent review reported sociotechnical hazards.[7] An example would be violations of protocols for safe HIT use (e.g., in barcode medication administration [BCMA], scanning the patient and the medication out of the proper sequence) that may be more common when a system is poorly designed.[8] Additionally, there may be a poor fit between the HIT's design and the workflow it is intended to support,[9] [10] or organizational policies that inadvertently create conflicting goals for workers (e.g., using inefficient safety technology in the face of production pressure).[11] To date, the response of the biomedical informatics research community has been to recommend: (1) improving HIT usability assessment,[12] (2) measuring HIT-related patient safety problems,[13] [14] (3) identifying best practices of implementation,[15] and (4) engaging users in reporting problems and adverse events related to HIT.[16]

We assert that there is an additional key opportunity for improving HIT safety through learning about how contextual factors contribute to sociotechnical failures. Consider the BCMA example introduced above. The specific situation of use is a nurse scanning a patient's barcoded armband and the medication, checking these against the prescription displayed in the BCMA software, and then administering the medication. The context includes a variety of elements that structure the use of HIT and can be conceptualized in terms of distance from that situation (in terms of space or time), as depicted in [Fig. 1]. The situational context has characteristics that have been extensively explored in the human factors literature,[17] [18] [19] and include myriad “performance shaping factors” such as workload, lighting, distractions, fatigue, etc.[20] There are also temporal characteristics: not only are medications typically prescribed to be given at a specific time but the time of actual administration carries meaning relative to other events, for example, it is the last hour of the nurse's shift, or the last hours before the patient's discharge from the hospital, or other patients that the nurse is caring for have tasks required at the same time. Further removed from the situation of use, and affecting other workers and patients, is the organizational context, where myriad policies and management decisions of multiple organizations shape clinical activity.[11] For example, the organization may have policies for how many minutes after the intended time a medication administration would be considered “late” and what frequent such late administrations would mean for nurses' job evaluations.

Fig. 1 Contextual factors vary based on their proximity to the situation of health information technology (HIT) use.

Finally, the public context, experienced across organizations, shapes practices through regulations and other institutional and social patterns. Existing measures of HIT implementation success provide limited insight into why situational factors can increase the risk of problems. Providing managers with better information on situational factors can enable responses to adapt the organizational context or attempt to influence the public context in a way that reduces future risk.

Other industries have developed advanced analytics to examine very specific, situated actions of workers and outcomes related to those actions.[21] [22] For example, in professional sports, high quality activity data are available during and after games through instant replays and video footage from multiple angles. Such data are possible because all action occurs in a single physical space and the range of possible activities is reasonably constrained and known. Unfortunately, health care does not have these characteristics. However, the emergence of new data science capabilities and new strategies for engaging clinical staff and patients bring new opportunities to address the challenge of sociotechnical risk measurement.

We describe a new perspective to studying HIT safety with a focus on activity in context. Situational analytics is the application of analytic methods to characterize human activity in situations and identify patterns in activity and outcomes that are influenced by contextual factors. We hypothesize that situational analytics will help identify potentially unsafe patterns in human–technology interactions and identify causal factors that may not be visible with existing measurement schemes.

Constraints on HIT Safety Measurement

The identification of robust, meaningful measures that reflect the impact of HIT on patient safety is an urgent and rapidly evolving area of research.[8] [12] [16] [23] [24] Contextual issues present a major challenge. A task force organized by the National Quality Forum observed that, “…given that harm and adverse effects, as well as the benefits of HIT, are often software-specific and context-specific, it is challenging to generalize about the safety of HIT as a whole, or of any specific function.”[13] Useful research-based models of safety in HIT have been developed.[13] [17] However, as Koppel has observed, many elements of existing HIT safety frameworks seem “unknown or unknowable.”[25] Clinical workers are distributed across hospitals, clinics, and home settings, and their activities are complex and nonlinear. Processes can be linked across time and places in ways that are not easily identifiable and are sometimes unexpected, and the introduction of new technology disrupts activity patterns in ways that are difficult to predict.[26] Sociotechnical hazards exist when social and technological features are intertwined, and are widespread in health care.[27]

Prediction and Mitigation of Sociotechnical Hazards

HIT developers cannot readily comprehend what users actually experience when using their products in everyday work. Understanding that a hazard has a sociotechnical source is only the first step; teasing apart the specific problems and identifying solutions is more challenging. Because we rely on complaints and chance observations to identify problems and possible improvements, we may miss many opportunities to improve patient safety and to increase the overall value of HIT to workers and patients. One problem is that developers can only observe a sliver of activities that produce outcomes. For example, improving the safety of medication administration is critical. One way to gain insight would be to observe all medication administrations and associated interactions, HIT screens, and environmental constraints. However, achieving this goal is impossible in practice due to resource constraints. The result is that developers cannot “see” into the situated performances of everyday clinical tasks that, taken together, reveal the conflicts between contextual realities and design or implementation shortcomings that produce potentially hazardous situations.

In her landmark 1987 book Plans and Situated Actions,[28] Lucy Suchman introduced a new way of thinking about human–machine interactions. The central theme of her work is that our actions are strongly influenced by the material and social circumstances in which we act. We might make elaborate plans for what we want to do, but when we start to implement our plans they are frequently changed or discarded because of unforeseen circumstances. She argued that “…however planned, purposeful actions are inevitably situated actions,” which she defined as “actions taken in the context of particular, concrete circumstances.” These concepts have transformed the practices of industrial researchers and human–computer interaction designers by bringing the notions of context and activity to the forefront. This approach contrasts with prior approaches that emphasized user knowledge and cognition. From the perspective of situated action, if we can understand the situation and its potential effect on a user's actions, we can create a much better solution than if we based our design on assumptions about how the typical user will act in the typical situation.

New Contextually Rich Data Sources: Biomedical informatics researchers have begun to move away from sparse, linear characterizations of work activity, resulting in important fundamental insights about the patterned functioning of otherwise invisible contextual variables such as autonomy,[29] paper artifacts,[30] [31] and local temporal patterns.[32] With the emergence of contextual data resources such as locational data for humans and equipment and time-stamped clinical actions throughout the continuum of care, this contextually grounded approach can be used to analyze multisource data to provide new perspectives on system status and events. These data can describe contextual factors for human actors (e.g., workload and complexity of assigned patients), settings (e.g., locations, resources available), and actions (e.g., complexity, sequence, time). Qualitative research can document locally identified problems, potential contextual relationships, and sources of data. This information can be used to construct empirical models in clinical settings that help explain technology use and its effectiveness, efficiency, and potential risks to patient and worker safety. For example, alerts and other notifications used to encourage evidence-based medicine in busy clinical settings are frequently ignored. Understanding the context—when and why it is ignored—can point to possible solutions. Is a specific alert more likely to be ignored when the provider is caring for a large number of patients? In the middle of the night? In certain locations? With certain types of patients? Using HIT data to reconstruct and model situations or activities can shed light on contextual factors that contribute to low or high technology adoption.

Situational Analytics versus Other Approaches: Situational analytics is grounded in the ethnomethodological insights from Suchman's work on situated action. As a general approach, it could be embedded in other frameworks or used standalone in implementation research, quality improvement, design research, management studies, or social science research. Zheng et al[33] described a complementary approach, computational ethnography, that seeks to improve human–computer interaction through leveraging in situ data that reflects users' interactions with systems. While we do not characterize situational analytics as ethnographic, the approach would utilize many of the same data resources, in addition to external sources such as environmental information, workload data, and other contextual information external to HIT. Similarly, situational analytics shares some features with workflow analysis[10] [34] and monitoring initiatives.[35] The difference is again the focus on developing a robust picture of the situation in which a user is acting, rather than a trajectory of HIT interaction or process steps. We are unaware of any initiatives proposing or using situational analytics as described here.

How Situational Analytics can Supplement Traditional Observational Methods for Understanding HIT Use in Context: The Case of BCMA

A variety of information sources can be used to explore situated action. The most robust approach is direct observation and interviews of users. This type of data collection is expensive, requires specialized skills, and is only minimally available to most organizations. Nevertheless, it is instructive to consider the types of information that are produced to identify automated sources that could approach the same level of insight, and even improve on a key limitation of observational data—a lack of access to the internal logic of the system that is inaccessible to users. Our prior work explored the role of informatics rollout teams,[36] [37] who operate at the boundary between technology design and use. Informatics-trained nurses (referred to as mediators) were able to speak the languages of both developers and clinical staff and were thus able to translate those perspectives for each side. Their skills allowed them to understand problems between the two groups during an implementation of BCMA. In one example of a problem identified and resolved, nurses were designating doses as “missed” even if they were given later in an effort to avoid a dose being reported as “late.” In this case, management changed policies to remove the stigma (and improve documentation) of late doses. In another case, there was a difficult-to-detect discrepancy in digit rounding between the pharmacy system and the BCMA system that produced erroneous wrong-dose alerts for nurses. [Fig. 2] illustrates how the process and available information changed after rollout. The rollout period of BCMA and accompanying intense analysis of situated use produced important insights for developers and managers (including unit managers and hospital administrators), whose roles were to adapt the technology and the organizational context, respectively, to optimize overall processes. Screenshots were a key communication tool, enabling the team to make more rapid sense of problems, enabling better communication with managers and developers. Mediators spent 1 to 2 weeks on each patient care unit during rollout, after which bedside nurses and unit managers were expected to identify problems and report those that needed to be addressed.[37]

Fig. 2 Situational analytics is performed by rollout teams during informatics implementations, reducing risks to patient safety.

How could situational analytics address the problem represented by the red bar in [Fig. 2]? After the initial implementation, when human observers are no longer onsite, managers and development teams could use automated data resources on patterns of activity to refine HIT and organizational context, and eventually predict how to deploy technologies and policies that optimize safety. For example, managers could work with the development team and clinical leaders to identify alerts that seem to be ignored or dismissed without an action. With effective visualization of log data along with other organizational information such as the patient census, admission-discharge-transfer, staffing, and employee timekeeping data, a pattern may emerge that demonstrates that action on certain alerts is sensitive to workload (either the unit being short-staffed or the census is especially high). If developers know that certain alerts are more likely to be ignored as individual workload increases, different strategies for alerting may be invoked during peak periods of workload. Similarly, the organization might use the data to better understand the implications of high workload on patient safety and use it to make organizational changes, such as adding staff at peak times and during breaks.

Implications of a Focus on Situated Action

Data resources for situational analytics would include: (1) log data from HIT activity (e.g., orders, communications, and charting actions), (2) reports from users, including help desk reports and data from online reports that include screenshots and images of problematic design or environment, (3) qualitative field data on the use of tools in everyday clinical work, and (4) locational data from tagged people and equipment in (and beyond) the clinical environment. We anticipate that other sources of data would emerge over time. For example, a recent study suggests that telephone call rates between units is associated with unit-level workload.[38] Integrating these data are an exciting challenge for data scientists and qualitative researchers. Methods and tools will be needed to optimize the potential gains. To understand situated action at a large scale, investments and new collaborations will be required, as outlined below.

Collaboration between Qualitative Researchers and Data Scientists

Emerging research challenges are focusing on identifying patterns in everyday activity, whether they are observed by an ethnographer or identified in clinical or operational data. Collaboration among data scientists and qualitative researchers can produce new insights, such as previously undetected pathways to process failure (e.g., when a patient is transferred from hospital X, medication data are missing), ways to make more informed trade-offs between data security and information access, or unexpected sources of information for clinical decision making (e.g., tailored prescribing of medication based on whether the patient has access to refrigerated storage).

Infrastructure to Support Situational Analytics and Implement Recommendations

In health care organizations, new infrastructure will be needed including multidisciplinary teams that include the necessary expertise and new governance expectations to integrate safety and quality objectives with HIT design and implementation. At the federal level, it includes funding to develop methods to integrate qualitative and quantitative data, and new approaches to capture feedback from users. The National Science Foundation's Secure and Trustworthy Cyberspace program sought projects involving both computer scientists and social scientists. Similar programs targeted at HIT safety can produce innovations not only in technology, but in social and organizational aspects of patient and worker safety.

A Parallel Research Program on Ethical and Social Implications

As data resources are increasingly utilized to characterize health care work and health care workers with the goal of improved patient and worker safety, researchers must reflect on the ethical, legal, and social implications of their analyses and how they are being put to practical use at organizational and policy levels.

Conclusion and Clinical Relevance

HIT systems document a vast number of clinical actions, producing data that can be used for analysis. Combining these data with (1) qualitative research that provides focus, and (2) input from clinicians using in-context capture tools (e.g., screenshots, images, and videos) will result in new and valuable insights on how to improve all components of the health care sociotechnical system.

Multiple Choice Questions

1. To understand situated action at a large scale in health care, investments will need to be made in:

   • Hardware.

   • Software.

   • Collaboration between data scientists and qualitative researchers.

   • Blockchain infrastructure.

   Correct Answer: The correct answer is option c. The paper specifically calls for investment in c, but does not mention options a, b, or d.

2. An example of a factor related to public context is:

   • Workload.

   • Medicare payment models.

   • Temperature in the operating room.

   • User interface design.

   Correct Answer: The correct answer is option b. The other factors are situational or technological factors.

Conflict of Interest

L.L.N. reports a grant from National Library of Medicine, during the conduct of the study; a grant from Stead Fund for Innovation outside the submitted work.

Acknowledgments

The authors would like to acknowledge collaborators Daniel Fabbri, Thomas Lasko, Colin Walsh, Bradley Malin, and Jacob VanHouten for inspiring this paper.

Protection of Human and Animal Subjects

This paper does not present new research data for which Human Subjects approval is required. The published research referenced in the paper was approved by the Vanderbilt University Institutional Review Board.

• References

• 1 Bates DW, Teich JM, Lee J. , et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6 (04) 313-321
• 2 Poon EG, Keohane CA, Yoon CS. , et al. Effect of bar-code technology on the safety of medication administration. N Engl J Med 2010; 362 (18) 1698-1707
• 3 Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood) 2011; 30 (03) 464-471
• 4 Patterson ES, Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inform Assoc 2002; 9 (05) 540-553
• 5 Coiera E, Ash J, Berg M. The Unintended Consequences of Health Information Technology Revisited. Stuttgart, Germany: IMIA Yearbook; 2016: 163-169
• 6 Castro GM, Buczkowski L, Hafner JM. The contribution of sociotechnical factors to health information technology–related sentinel events. Jt Comm J Qual Patient Saf 2016; 42 (02) 70-76
• 7 Kim MO, Coiera E, Magrabi F. Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review. J Am Med Inform Assoc 2017; 24 (02) 246-250
• 8 Koppel R, Wetterneck T, Telles JL, Karsh B-T. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc 2008; 15 (04) 408-423
• 9 Holden RJ, Brown RL, Alper SJ, Scanlon MC, Patel NR, Karsh BT. That's nice, but what does IT do? Evaluating the impact of bar coded medication administration by measuring changes in the process of care. Int J Ind Ergon 2011; 41 (04) 370-379
• 10 Unertl KM, Novak LL, Johnson KB, Lorenzi NM. Traversing the many paths of workflow research: developing a conceptual framework of workflow terminology through a systematic literature review. J Am Med Inform Assoc 2010; 17 (03) 265-273
• 11 Novak LL, Holden RJ, Anders SH, Hong JY, Karsh B-T. Using a sociotechnical framework to understand adaptations in health IT implementation. Int J Med Inform 2013; 82 (12) e331-e344
• 12 Middleton B, Bloomrosen M, Dente MA. , et al; American Medical Informatics Association. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc 2013; 20 (e1): e2-e8
• 13 Belmont E, Singh H. Identification and Prioritization of Health IT Patient Safety Measures [Internet]. National Quality Forum. Available at: http://www.qualityforum.org/Publications/2016/02/Identification_and_Prioritization_of_HIT_Patient_Safety_Measures.aspx . Accessed August 25, 2019
• 14 Thayer JG, Miller JM, Fiks AG, Tague L, Grundmeier RW. Assessing the safety of custom web-based clinical decision support systems in electronic health records: a case study. Appl Clin Inform 2019; 10 (02) 237-246
• 15 Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc 2003; 10 (03) 229-234
• 16 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: the Health IT Safety Framework. BMJ Qual Saf 2016; 25 (04) 226-232
• 17 Carayon P. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. Boca Raton: CRC Press; 2011
• 18 Weinger MB, Wiklund ME, Gardner-Bonneau D. , eds. Handbook of Human Factors in Medical Device Design. Boca Raton, FL: CRC Press; 2010: 850
• 19 Salvendy G. Handbook of Human Factors and Ergonomics. Hoboken, NJ: John Wiley & Sons; 2012
• 20 Weinger MB, Gaba DM. Human factors engineering in patient safety. Anesthesiology 2014; 120 (04) 801-806
• 21 Sharma R, Reynolds P, Scheepers R, Seddon PB, Shanks GG. Business Analytics and Competitive Advantage: A Review and a Research Agenda. In: Respício A, Adam F, Phillips-Wren GE, Teixeira C, Telhada J. , eds. Bridging the Socio-technical Gap in Decision Support Systems. Amsterdam: IOS Press; 2010: 187-198
• 22 Morris B. Stephen Curry Is The Revolution. Available at: http://fivethirtyeight.com/features/stephen-curry-is-the-revolution/ . Accessed August 25, 2019
• 23 Meeks DW, Takian A, Sittig DF, Singh H, Barber N. Exploring the sociotechnical intersection of patient safety and electronic health record implementation. J Am Med Inform Assoc 2014; 21 (e1): e28-e34
• 24 Jylhä V, Mikkonen S, Saranto K, Bates DW. The impact of information culture on patient safety outcomes. Development of a structural equation model. Methods Inf Med 2017; 56 (Open): e30-e38
• 25 Koppel R. The health information technology safety framework: building great structures on vast voids. BMJ Qual Saf 2016; 25 (04) 218-220
• 26 Wachter RM. The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine's Computer Age. New York: McGraw-Hill Education; 2015
• 27 Karsh B-T, Weinger MB, Abbott PA, Wears RL. Health information technology: fallacies and sober realities. J Am Med Inform Assoc 2010; 17 (06) 617-623
• 28 Suchman L. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge: Cambridge University Press; 1987
• 29 Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. An unintended consequence of CPOE implementation: shifts in power, control, and autonomy. In AMIA Annual Symposium Proceedings (Vol. 2006, p. 11). American Medical Informatics Association
• 30 Østerlund CS. Documents in place: demarcating places for collaboration in healthcare settings. Comput Support Coop Work CSCW. 2008; 17 (2–3): 195-225
• 31 Berg M, Toussaint P. The mantra of modeling and the forgotten powers of paper: a sociotechnical view on the development of process-oriented ICT in health care. Int J Med Inform 2003; 69 (2-3): 223-234
• 32 Reddy M, Dourish P, Pratt W. Temporality in medical work: time also matters. Comput Support Coop Work 2006; 15: 29-53
• 33 Zheng K, Hanauer DA, Weibel N, Agha Z. Computational Ethnography: Automated and Unobtrusive Means for Collecting Data In Situ for Human–Computer Interaction Evaluation Studies. In: Cognitive Informatics for Biomedicine. Cham: Springer; 2015: 111-140
• 34 Ozkaynak M, Wu DTY, Hannah K, Dayan PS, Mistry RD. Examining workflow in a pediatric emergency department to develop a clinical decision support for an antimicrobial stewardship program. Appl Clin Inform 2018; 9 (02) 248-260
• 35 Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The value of monitoring clinical decision support interventions. Appl Clin Inform 2018; 9 (01) 163-173
• 36 Novak L, Brooks J, Gadd C, Anders S, Lorenzi N. Mediating the intersections of organizational routines during the introduction of a health IT system. Eur J Inf Syst 2012; 21 (05) 552-569
• 37 Novak LL, Anders S, Gadd CS, Lorenzi NM. Mediation of adoption and use: a key strategy for mitigating unintended consequences of health IT implementation. J Am Med Inform Assoc 2012; 19 (06) 1043-1049
• 38 Miller A, Aswani A, Zhou M, Weinger M, Slagle J, France D. Using telephone call rates and nurse-to-patient ratios as measures of resilient performance under high patient flow conditions. Cogn Technol Work 2019; 21 (02) 225-236