Using a Sociotechnical Model to Understand Challenges with Sepsis Recognition among Critically Ill Infants

Abstract

Objective The aim of the study is to apply a sociotechnical model to the requirements phase of implementing a machine learning algorithm-based system to support sepsis recognition in the neonatal intensive care unit.

Methods We incorporated components from the sociotechnical model, Safety in Engineering for Patient Safety 2.0, in three requirements phase activities: (1) semi-structured interviews, (2) user profiles, and (3) system use cases.

Results Thirty-one neonatal intensive care unit clinicians participated in semi-structured interviews (11 nurses, 10 front line ordering clinician, five fellows, and five attending physician). Interview transcripts were coded and then compiled into themes deductively based on components from the sociotechnical model (persons, environment, organization, tasks, tools and technology, collaboration, and outcomes). The interview analysis was used to create four user profiles defining responsibilities in sepsis recognition, team collaboration, and attributes relevant to sepsis recognition. Two user profiles (nurse, front line ordering clinician) included variants based on experience relevant to sepsis recognition. The interview analysis was used to develop three system use cases representing clinical sepsis scenarios. Each use case defines the precondition, actors, and high-level sequence of actions, and includes variants based on sociotechnical works system factors that can complicate sepsis recognition. The interview analysis, user profiles, and use cases serve as the foundation for supporting sociotechnical design to all subsequent human-centered design methods including subject recruitment, formative design, summative user testing, and simulation testing.

Conclusion Integration of the sociotechnical model-guided requirements gathering activities, analysis, and deliverables by framing a range of sociotechnical components and the interconnectedness of these components in the broader work system. Applying the sociotechnical model resulted in discovering work system, process, and outcome requirements that would otherwise be difficult to capture, or missed entirely, using traditional requirements gathering methods or approaches to clinical decision support design.

Keywords

clinical decision support - environment - clinical care - neonatology - pediatrics - domain - technology

Introduction

Sepsis is a leading cause of morbidity and mortality in infants worldwide.[1] [2] Although sepsis affects relatively few healthy full term infants, the incidence is 200-fold higher in those born prematurely who experience the highest mortality (7–28%).[3] [4] Among survivors, 30 to 50% incur major long-term impairments including prolonged hospitalization, chronic lung disease, and neurodevelopmental disabilities.[5] [6] [7] Early sepsis diagnosis and treatment is fundamental to the prevention of infant mortality, however, recognition remains especially difficult due to the unique, heterogeneous clinical presentation of sepsis in infants with complex medical conditions and frequent comorbidities that can confound the diagnosis.[1] [3] [4] Advances in predictive modeling support early recognition of infant sepsis,[8] but research has yet to demonstrate the effective translation of these tools to clinical care.[9]

Sociotechnical models represent interconnected human, technical, environmental, and other factors essential to the evaluation and design of information technology.[10] [11] Sociotechnical models address challenges in complex health care work systems including patient safety and the design of technology.[12] [13] [14] One strategy for promoting sociotechnical design is to integrate sociotechnical components into existing software development and human-centered design methods.[15] [16] [17] The Safety Engineering in Patient Safety (SEIPS) model, version 2, was developed to guide the design and evaluation of health care systems.[14] SEIPS 2.0 provides a comprehensive framework representing the sociotechnical components of the work system including people, tasks, tools, environment, and organization, the processes that occur within this work system, and outcomes at the patient, professional and organizational level.[14]

Researchers at our organization have developed a machine learning prediction model for infant sepsis that utilizes electronic health record (EHR) data.[18] Although the model has similar predictive accuracy to sepsis prediction models in other clinical domains, implemented prediction models have shown mixed effectiveness attributed to sociotechnical factors not accounted for in system design.[19] Prior work has demonstrated the utility of human factors and user-centered methods in addressing neonatal intensive care unit (NICU) sepsis including the critical decision method in assessing nurse decision making and ethnographic approaches, though neither address clinical decision support (CDS) based on machine learning.[20] [21] However, the work demonstrates that developing CDS for NICU sepsis recognition is an ideal setting for incorporating sociotechnical models into human-centered design methods and requirements processes that address sociotechnical factors.[20]

Objectives

The aim of the study is to apply a sociotechnical model to the requirements gathering phase of implementing a machine learning algorithm-based CDS to improve NICU sepsis recognition.

Methods

We applied the SEIPS 2.0[14] model to three requirements gathering activities: (1) user interviews; (2) user profiles; and (3) use cases ([Fig. 1]).

Fig. 1 Sociotechnical requirements process.

Setting

The setting comprises an urban, quaternary-care pediatric hospital with 564 beds, 40% of which are allocated to intensive care including a 105-bed level 4 NICU that provides care for over 1,000 critically ill infants with complex medical and surgical conditions annually.

Participants

We recruited a purposive sample of NICU attending physicians, neonatology fellows, front line ordering clinicians (FLOC) such as residents and advanced practice providers, and nurses.

Interviews

A semi-structured interview guide was developed to elicit open-ended responses on six topics: (1) participant's role; (2) experience in sepsis recognition including difficult example cases; (3) teamwork; (4) technology; (5) sepsis recognition challenges; (6) improving sepsis recognition. Interviews were audio recorded and transcribed. The interview guide was based on the team's prior experience in clinical care and developing CDS. While we did not directly apply SEIPS or other frameworks to the interview guide, the emergence of SEIPS-based themes in the analysis phase influenced question probes for sociotechnical factors in subsequent interviews. Three authors (D.K., G.S., and N.M.) conducted the interviews, representing a combination of human factors experience (D.K. and G.S.) and clinical and informatics training (N.M.).

Qualitative Analysis

Two study team members (D.K. and N.M.) inductively free coded a random selection of 10% of the transcripts using NVivo v12 (QSR International, Melbourne, Australia). A coding guideline document was created and reviewed by team members with neonatology expertise ([Supplementary Appendix A], available in the online version). Three randomly selected transcripts were coded by two team members (D.K. and N.M.) to calculate Cohen's Kappa (k) for each code. Any code with a k <0.6 was reviewed to reach consensus and refine the definition. The team used a process of constant comparison to construct themes. In this process, the team identified emerging themes that could be organized by a sociotechnical framework.[22] The entire study team met and reached consensus that the coded data could be organized deductively based on the SEIPS 2.0 framework.[14]

User Profiles

Interview results were used to develop user profiles describing aggregate characteristics of each clinical role including team collaboration and sepsis recognition responsibilities. We synthesized coded responses to describe characteristics for each role including high level responsibilities and daily routines in general, as well as responsibilities and tasks specific to sepsis recognition. This analysis was iteratively reviewed and refined by all study team members including neonatologists. The nurse and FLOC user profiles were supplemented with profile variations based on experience with sepsis recognition (e.g., “inexperienced” vs. “experienced”).[23] [24]

System Use Cases

Interview analysis informed the development of use cases to represent realistic scenarios of sepsis recognition and user interaction with the system and other team members. Use case development followed a similar process to user profile development via an analysis of interview codes including example cases in sepsis recognition, and challenges to sepsis recognition, and were iteratively reviewed and refined by study team members including neonatologists. Use cases often represent complex environments via alternate scenarios.[25] We applied a similar approach to our use cases by incorporating sociotechnical-based variations derived from the interview analysis of factors that can contribute to delayed sepsis recognition (e.g., environment, organization and other factors).[25] [26]

Results

Thirty-one NICU clinicians participated in the interviews: attending physicians (5), Neonatology fellows (5), FLOCs (10), and nurses (11) ([Table 1]). We applied a SEIPS 2.0 to the analysis of the interview transcripts and developed user profiles and use cases that included sociotechnical factors and variants.

Table 1 Participant demographics

Role	N	Gender		Age		Years in practice		Years in NICU
		Female	Male	Mean	SD	Mean	SD	Mean	SD
Attending	5	3	2	47.6	11.3	22.2	11.3	19	11.6
Fellow	5	2	3	33.0	1.2	5.6	1.7	2.6	0.9
FLOC	10	8	2	37.7	9.1	13.3	9.6	10.4	7
Nurse	11	11	0	29.5	5.5	7.0	5.2	6.5	5.5
Total	31	24	7	35.7	9.6	11.3	9.3	9.1	8.3

Interview Coding and Thematic Analysis

Twenty-three codes were identified from the interview transcripts ([Supplementary Appendix A], available in the online version). A comparison of three randomized interviews achieved a Cohen's kappa (k) of 0.76. In the thematic analysis, the team identified emerging themes that could be organized using the SEIPS 2.0 framework.[27] Results were organized into sociotechnical themes derived from the SEIPS 2.0: (1) persons: patient, (2) persons: clinicians (3) environment, (4) organization, (5) tasks, (6) tools and technology, and (7) collaboration.[14] [Table 2] provides examples of SEIPS-derived themes.

Table 2 Sociotechnical themes

SEIPS Themes	Factors
Persons: Patient	Complexity of NICU patients including multiple clinical conditions, high risk for sepsis, and conditions that mimic sepsis.  NICU patients vital sign baselines are quite variable and often outside of “normal” ranges for older children and adults.  Patients often present subjective signs and symptoms important to sepsis recognition such as lethargy.
Persons: Clinicians	Critical role of nurses in detecting changes by being most familiar with patient.  New nurses lack of experience in recognizing signs of sepsis.  More senior nurses can have concerns based on experience, but are not able to fully describe source of concern (“gut feeling”).  Instances of bias with patients who were initially not a concern for sepsis.
Environment	Atypical among non-academic medical centers at over 100 beds, mix of complex patients including medical, surgical, premature, chronic lung disease.  Highly stressful environment, especially for new nurses and FLOCs.  Inability for education and training, especially nursing, to prepare for NICU overall.
Tools and Technology	Variability in EHR and monitor use, skills, and training.  Limitations of EHR and monitor vital sign trend functionality including NICU patient baselines, difficulty in using during high workload.  Variability in EHR documentation including episodic events and subjective impressions.
Tasks	New nurses focused on patient tasks and not yet experienced in sepsis recognition.  Importance of bedside assessment for sepsis recognition.
Organization	Significant increase in patient coverage for attending, fellow, and FLOC at night shift.  Inconsistency in recording episodic events that may be important to sepsis recognition.  Training for new nurses focuses on sepsis work ups vs. signs and symptoms of sepsis.  Work in progress on sepsis clinical pathway that may include sepsis huddle.
Process-Collaboration	Chain of command: Nurse-FLOC-Fellow-Attending and examples of where reluctance to escalate issues result in delayed recognition.  Challenges in communicating uncertainty, where some feel certainty is essential.  Informal and formal support of nurses by other nurses including bedside colleagues, charge nurses, and expert nurses.  Clinicians in all roles taking the time to respond to concerns as an opportunity for education.
Outcomes	Order sepsis work up - supported by contributing outcomes including escalation of concerns to fellow/attending, bedside assessment, clinical team discussion or “sepsis huddle,” support for new nurses by charge nurse, nurse expert or other nursing colleagues.

As NICU patients are incapable of participating in their care we framed the patient under both the environment and persons component. At the environment level, a unit consisting of over 100 critically ill infants, with a wide range of conditions that increase sepsis risk while simultaneously making sepsis detection difficult, creates an environment repeatedly described as highly stressful and challenging to sepsis recognition. At the person level, the highly variable and unique vital sign baseline for each patient in the NICU was discussed by every participant. Most NICU infant vital signs do not fall into a normal range and sepsis recognition based on standard vital sign thresholds, as with adults and older children, is challenging. Participants stressed the critical importance of becoming familiar with each infant's distinctive baseline, detecting subtle variations, and determining if there is a potential problem. Participants noted that the baseline also includes subjective signs and symptoms important to sepsis recognition including lethargy and level of alertness.

The four clinical roles were also included in the persons component. While all four roles are engaged in sepsis recognition, every participant stressed the critical importance of nurses. Nurses were described as the “eyes and ears” of sepsis recognition by being involved in the care of one to two patients over 12 hour shifts and best positioned to notice subtle changes in patient baselines. However, multiple participants noted that sepsis recognition among nurses was highly dependent on experience, where experienced nurses can explain their concerns in detail, or may have a less precise “gut feeling” that is generally trusted by others. By contrast newer nurses lack experience in sepsis recognition and may demonstrate a range of responses from being overly concerned by any change to missing significant signs and symptoms. Similar opinions were expressed regarding the experience of FLOCs who are responsible for responding to nursing concerns. FLOCs vary by education and training including nurse practitioner, medical assistant, and medical doctor further divided by resident and experience. While limited by our relatively small sample, our interview results do suggest further exploration by background and education. For example, a physician FLOC described opportunities to educate bedside nurses on sepsis detection, while a nurse practitioner FLOC with extensive prior experience as a NICU bedside nurse commented on their ability to recognize challenges facing newer nurses.

The SEIPS-guided analysis revealed additional important work systems components. Regarding tools and technology, participants identified the EHR and bedside monitors as facilitators and barriers to recognition of infant sepsis. Many described variabilities in training and usage of EHR vital sign trending functionality, yet those familiar with the functionality stated it was not tailored to infants and often required time and effort that were not always available with the demanding NICU workload. Some participants reported using more advanced features of bedside monitors while others were unfamiliar with these functions. Variability in EHR documentation practices was also identified as a challenge.

Organizational factors were discovered to impact the environment and collaboration. While nursing coverage remains constant regardless of shift, the night shift increases the number of patients cared for by each FLOC, fellow, and attending. Another organizational and collaboration factor discussed was the “chain of command” in information flow and decision-making between the Nurse-FLOC-Fellow-Attending team. While most comments on collaboration and teamwork were positive, breakdowns in the chain of command information flow were repeatedly cited in example cases where sepsis recognition was delayed. These and other cases were complicated by multiple combinations of night shift workload demands, other workload demands such as admissions and discharges, nurses experience in recognizing and communicating subtle signs and symptoms, patients who were not indicated as a concern for sepsis at shift hand off, and reluctance to raise vague concerns or jump the chain of command when concerns were not addressed.

User Profiles

We applied our interview analysis to develop user profiles relevant to sepsis recognition. While mainly informed by the SEIPS persons component, other work system components informed the results by describing the person within the work system. [Fig. 2] demonstrates an NICU nurse user profile with two variants based on the level of experience. The same experience-based variant was applied to the FLOC profile. User profiles for each role ([Supplementary Appendix B], available in the online version) will be used to guide subsequent phases including the recruitment of participants for formative and summative usability testing, and patient simulations.

Fig. 2 Example user profile with sociotechnical variants.

Use Cases

Interview analysis and user profiles were reviewed with neonatology experts to describe a foundational set of three NICU sepsis clinical scenarios: (1) Infants with low complexity conditions with a change in clinical status suggestive of sepsis. (2) Infants with high complexity conditions whose clinical findings related to their underlying conditions may periodically overlap with those of sepsis. (3) Infants at higher risk for sepsis presenting with only subtle signs of sepsis. These scenarios were further developed into three use cases for the sepsis CDS system ([Supplementary Appendix C], available in the online version).

Each use case can incorporate one or many sociotechnical variants discovered from the interview analysis. [Table 3] demonstrates the use case of an infant at high risk for infection that includes preconditions, actors, and the scenario. The use case includes categorized sociotechnical variants that could be incorporated to the use case. For example, for use in both design and simulation testing, the use case includes the nursing user profile variant of “less experience,” plus additional sociotechnical variants such as challenges in the night shift staffing level and hesitancy to escalate concerns over the chain of command. These sociotechnical variants are incorporated into the three primary clinical use case scenarios to guide subsequent system design and user testing.

Table 3 Use case example with sociotechnical variations

Use case ID	1.0
Use case title	Low complexity infant
Precondition(s)	Patient: Premature infant, 31 wk.  On room air.  Patient baseline heartrate is low.  Patient is usually agitated or “fussy” during diaper changes or other activities.
Primary actor	NICU Nurse, “the user”
Additional actors	FLOC, Fellow, Attending
Primary use case—sequence of actions	Nurse notes patient requires extra heat from isolette.  Heart rate increase.  During a diaper change the patient is lethargic.  The User documents temp instability, heart rate, and patient behavior in her.  The System updates its assessment to support user perception and comprehension of an increase in sepsis risk.  User responds to System and contacts FLOC to discuss.  User refers to system in discussion with FLOC.  FLOC agrees on concern and contacts fellow and attending for bedside assessment.  User and FLOC refer to system in discussion with fellow/attending.  Fellow/attending make decision on sepsis work-up.
Sociotechnical variations
People	Newer nurse with less experience (User Profile variant).  Experienced nurse with “gut feeling” (User Profile variant).  FLOC with bias.
Environment	Familiarity with patient over time.  Workload imposed by shift/coverage.  Workload imposed by patient volume, admit/discharge, or transfers.
Tasks	Workload imposed by competing priorities within a patient.  Workload imposed by rapid vs. slow change in clinical status (rapid may necessitate more tasks—place new IV, place breathing tube, administer new medications…).
Technology and tools	Quality/consistency of EHR documentation.  Training/experience in using monitor or EHR vital sign trending and other tools.  Variability in recording episodic events.
Organization	Hierarchy or “chain of command.”  NICU team (surgical, chronic lung disease…).
Collaboration	Quality/content of handoff.  Confidence in escalating concerns.  Availability of supporting or supervisory staff to consult.

Discussion

We applied a sociotechnical model to requirements gathering activities for sepsis recognition among NICU infants. We analyzed interview transcripts to derive three analytic products to define the requirements for an infant sepsis early recognition system: a qualitative thematic analysis, user profiles, and use cases with sociotechnical variants. These products will be incorporated into all subsequent activities such as informing iterative design, subject recruitment, and defining test scenarios for formative, summative, and simulation-based testing.

Comparing this work to other studies demonstrates the utility of a sociotechnical analysis. In 1993, Crandall and Getchell-Reiter applied the critical decision method to elicit expert knowledge from NICU nurses including sepsis recognition.[20] In 2019, Harte et al described ethnographic approaches combined with interviews and participatory design in developing requirements for NICU sepsis CDS.[21] Our sociotechnical approach produced information that encompassed findings from both studies. For example, interview questions and probes on experience with difficult cases resulted in findings on NICU nurse decision making and challenges to newer NICU nurses inexperienced with sepsis. In terms of system requirements and scenarios, our approach resulted in identifying a range of sociotechnical factors that can contribute to delayed sepsis recognition and should be considered in a broader work system design. We believe our work supports an initial approach of performing a sociotechnical analysis to identify work system components, processes, and outcomes to identify and inform the application of subsequent human-centered design methods.

Applying the SEIPS model produced findings that might not have been identified through a typical requirement gathering process. The most commonly used framework for CDS development focuses on the five rights (information, time, user, format, channel).[28] [29] Our analysis demonstrated that lapses in sepsis recognition may arise not only from the lack of perception of relevant information (e.g., subtle vital sign changes), but also failure to act upon that information within a complex work system. We argue that aspects of the care delivery system captured by the sociotechnical variants (e.g., What happens at night when there's less staffing, and/or when the nurse is inexperienced?) are scenarios that must be accounted for to ensure a system that supports safe outcomes. This suggests traditional CDS frameworks focused on information delivery would not sufficiently meet requirements, and supports a movement toward using sociotechnical design as recommended by recent literature.[15] [17] [28] [30] [31] [32] [33] In addition, this work also suggests that optimal machine learning enabled CDS may need to vary by multiple factors such as role, environment, and levels of clinical experience in recognizing sepsis. To support such CDS, the machine-learning algorithms may also need to advance to not only focus on the prediction of sepsis, but also estimating factors such as the user's workload, level of comfort diagnosing sepsis, etc. Requirements gathering guided and framed by the SEIPS model defined the complex, dynamic, and interconnected sociotechnical components that will likely be critical in the design of not just new technology, but a new work system that will incorporate the technology.

Limitations

Our study was conducted at a single institution in a level 4 NICU, and results may not apply to other hospitals where the medical complexity of infants, clinician roles, workflow, and team communication may be different. Our small sample size did not allow us to fully explore the impact of prior training and experience within specific roles, which emerged as an important sociotechnical component. However, our study provides an approach that if further refined and modified through future research efforts could address sociotechnical considerations in NICU settings with different team structures and levels of medical complexity.

Conclusion

We demonstrated how a sociotechnical model can be integrated into requirements gathering activities and deliverables. Applying the SEIPS model to existing human-centered design methods generated detailed work system requirements represented by user profiles, use cases, and sociotechnical variants that might otherwise have been difficult to discover.

Clinical Relevance Statement

Multiple sociotechnical factors affect the ability of clinicians to recognize the subtle and heterogenous signs of sepsis among medically complex neonates. Efforts to develop health information technologies that address complex clinical problems, such as the recognition of sepsis in the NICU setting, should incorporate sociotechnical models with human-centered design methods.

Conflict of Interest

None declared.

Acknowledgments

The authors wish to acknowledge the following for their critical contributions to the project: Anne O'Hearn for managing participant scheduling; Katherine Kellom for consultation on qualitative analysis; Fran Balamuth, Chris Bonafide, Sarah Coggins, Aaron Masino, and Evelyn Wang for consultation, domain expertise, and study methods.

Protection of Human and Animal Subjects

The study was determined to be exempt from human studies by the Children's Hospital of Philadelphia, Institutional Review Board. This research was funded by the Institute for Biomedical Informatics at the University of Pennsylvania School of Medicine and the Department of Biomedical and Health Informatics at the Children's Hospital of Philadelphia. The funding sources had no role in the conception, study design, data collection, analysis, or decision to publish this manuscript.

Author Contributions

D.J.K. contributions include the conception of the work, acquisition, analysis, and interpretation of data, drafting, and revising the manuscript. L.S., M.C.H., and R.W.G. contributions include the conception of the work, analysis, and interpretation of data, and revising the manuscript. G.P.S. contributions include the conception of the work, acquisition of data, and revising the manuscript. N.M. contributions include the conception of the work, acquisition, analysis, and interpretation of data, and revising the manuscript. All authors approved the final version of the manuscript and agree to be accountable for all aspects of the work.

• References

• 1 Vogel L. Sepsis kills one million newborns a year: WHO. CMAJ 2017; 189 (40) E1272
• 2 Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR. et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet 2020; 395: 200-211
• 3 Liu L, Oza S, Hogan D. et al. Global, regional, and national causes of child mortality in 2000-13, with projections to inform post-2015 priorities: an updated systematic analysis. Lancet 2015; 385 (9966): 430-440
• 4 Giannoni E, Schlapbach LJ. Editorial: sepsis in neonates and children. Front Pediatr 2020; 8: 621663
• 5 Stoll BJ, Hansen NI, Adams-Chapman I. et al; National Institute of Child Health and Human Development Neonatal Research Network. Neurodevelopmental and growth impairment among extremely low-birth-weight infants with neonatal infection. JAMA 2004; 292 (19) 2357-2365
• 6 Stoll BJ, Hansen NI, Bell EF. et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network. Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network. Pediatrics 2010; 126 (03) 443-456
• 7 Cohen-Wolkowiez M, Moran C, Benjamin DK. et al. Early and late onset sepsis in late preterm infants. Pediatr Infect Dis J 2009; 28 (12) 1052-1056
• 8 Ginestra JC, Giannini HM, Schweickert WD. et al. Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock. Crit Care Med 2019; 47 (11) 1477-1484
• 9 A Path for Translation of Machine Learning Products into Healthcare Delivery. EMJ Innov. 2020 January 27. March 11, 2020 Available at: https://www.emjreviews.com/innovations/article/a-path-for-translation-of-machine-learning-products-into-healthcare-delivery/
• 10 Cherns A. The principles of sociotechnical design. Hum Relat 1976; 9 (08) 783-792
• 11 Clegg CW. Sociotechnical principles for system design. Appl Ergon 2000; 31 (05) 463-477
• 12 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74
• 13 Carayon P, Schoofs Hundt A, Karsh BT. et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (Suppl. 01) i50-i58
• 14 Holden RJ, Carayon P, Gurses AP. et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics 2013; 56 (11) 1669-1686
• 15 Baxter G, Sommerville I. Socio-technical systems: from design methods to systems engineering. Interact Comput 2011; 23 (01) 4-17
• 16 Mumford E. The story of socio-technical design: reflections on its successes, failures and potential. Inf Syst J 2006; 16 (04) 317-342
• 17 Berg M, Langenberg C, vd Berg I, Kwakkernaat J. Considerations for sociotechnical design: experiences with an electronic patient record in a clinical context. Int J Med Inform 1998; 52 (1-3): 243-251
• 18 Masino AJ, Harris MC, Forsyth D. et al. Machine learning models for early sepsis recognition in the neonatal intensive care unit using readily available electronic health record data. PLoS One 2019; 14 (02) e0212665
• 19 Teng AK, Wilcox AB. A review of predictive analytics solutions for sepsis patients. Appl Clin Inform 2020; 11 (03) 387-398
• 20 Crandall B, Getchell-Reiter K. Critical decision method: a technique for eliciting concrete assessment indicators from the intuition of NICU nurses. ANS Adv Nurs Sci 1993; 16 (01) 42-51
• 21 Harte R, Quinlan LR, Andrade E. et al. Defining user needs for a new sepsis risk decision support system in neonatal ICU settings through ethnography: user interviews and participatory design. In: Ahram T, Karwowski W, Taiar R, eds. Human Systems Engineering and Design. Cham: Springer International Publishing; 2019: 221-227
• 22 Cohen D, Crabtree BF, Damschroder L, Hamilton AB, Heurtin-Roberts S, Leeman J. et al. Qualitative Methods in Implementation Science. Bethesda, MD: National Cancer Institute, National Institutes of Health;
• 23 LeRouge C, Ma J, Sneha S, Tolle K. User profiles and personas in the design and development of consumer health technologies. Int J Med Inform 2013; 82 (11) e251-e268
• 24 Holden RJ, Kulanthaivel A, Purkayastha S, Goggins KM, Kripalani S. Know thy eHealth user: Development of biopsychosocial personas from a study of older adults with heart failure. Int J Med Inform 2017; 108: 158-167
• 25 Cockburn A. Writing Effective Use Cases. Addison-Wesley Reading; 2001
• 26 Hughes HPN, Clegg CW, Bolton LE, Machon LC. Systems scenarios: a tool for facilitating the socio-technical design of work systems. Ergonomics 2017; 60 (10) 1319-1335
• 27 Assarroudi A, Heshmati Nabavi F, Armat MR, Ebadi A, Vaismoradi M. Directed qualitative content analysis: the description and elaboration of its underpinning methods and data analysis process. J Res Nurs 2018; 23 (01) 42-55
• 28 Osheroff J, Teich J, Levick D. et al. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. HIMSS; 2012: 350
• 29 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 30 Ackerman MS, Goggins SP, Herrmann T, Prilla M, Stary C. eds. Designing Healthcare That Works: A Socio-Technical Approach. London, United Kingdom: San Diego, CA: Academic Press; 2018: 207
• 31 Papoutsi C, Wherton J, Shaw S, Morrison C, Greenhalgh T. Putting the social back into sociotechnical: case studies of co-design in digital health. J Am Med Inform Assoc 2021; 28 (02) 284-293
• 32 Hasvold PE, Scholl J. Flexibility in interaction: sociotechnical design of an operating room scheduler. Int J Med Inform 2011; 80 (09) 631-645
• 33 Wooldridge AR, Carayon P, Hundt AS, Hoonakker PLT. SEIPS-based process modeling in primary care. Appl Ergon 2017; 60: 240-254

Publication History

Received: 15 September 2021

Accepted: 14 April 2022

Article published online:
29 July 2022