The Informatics Stack: A Heuristic Tool for Informatics Teaching^[*]

Summary

Objective: To develop a heuristic framework for students to organize and apply the many concepts of informatics for rapid use.

Method: Organization of curriculum material and recurrent refinement by student feedback. An Informatics Stack was developed based on several existing informatics and software-development frameworks comprising several levels of abstraction, from what a system is supposed to accomplish (4 levels) to how it accomplishes it (5 levels). At each level, there are specific concerns, types of interoperability, ethical and legal issues, testing and evaluation approaches and methods, and relevant scientific disciplines, and privacy (upper 5 levels), confidentiality (middle 3 levels), and security (lower 4 levels ) concerns whose levels overlap. An 8-week Introduction to Informatics course was taught for 6 years to masters students of informatics and of public health, based on the Stack, with a Final Project continually filled in during the course, where students applied the Stack to existing reports describing health information systems and their deployments.

Results: Student feedback from 538 students working in 116 groups over 6 years shows near-universal appreciation that the Stack helped to organize their review of the report. Each student, from a wide variety of backgrounds, identified some level of the Stack as something they might have otherwise missed, and all levels were invoked by some student. Attributes identified by the students as missing from the Stack concerned the practicalities of system development.

Conclusion: The Stack is a broadly-encompassing heuristic whose application can be learned and applied by students from a wide variety of backgrounds in an 8-week course.

Keywords

Curriculum - professional education - informatics - information science - systems theory

1. Introduction: World and Role

In the world of informatics education, given that informatics is about structured knowledge, an informatics curriculum should exhibit informatics concepts in a structured form, and not just in content. For a competency-based program in an educational organization [[1]], an informatics training program should not only convey knowledge, but should impart practical skills. We developed the Informatics Stack to serve as the basis of a core skill that students in an introductory informatics course could apply already during the course. Because this skill is executed at the informatics bedside, rather than as a result of intense analysis, and because we do not claim it as a rigorous discipline of software engineering [[2]], we call it a “heuristic,” or rule of thumb [[3]]. The purpose of the Informatics Stack is to provide a heuristic tool to aids in the role of informatics practice that embodies a systems perspective at the health-IT “bedside” and that helps in the informatician’s task of translating between stakeholders and technical communities [[4]]. In the IMIA framework, we are targeting novice biomedical health informatics professionals in bachelor, master, or doctoral training [[5]].

As illustration of the tasks and workflow in using the Stack, we offer these example Use Cases. The first 2 are applicable prior to the start of a project, the third, in the course of a project and the fourth, after a project’s completion.

1. Readiness assessment. A domain stakeholder asks you to solve a need through health IT services. Is the request well formulated?

2. IT enthusiast. An IT enthusiast asks you to build or purchase new technology. Is there a need that matches that technology?

3. Formative evaluation. You are brought into an ongoing project and asked to comment on the effort. Is the project at risk?

4. Report review. You are reading a report about an information system and want to provide a critical review. Is the report complete? Do the elements of the project cohere?

In the current article, we lay out the contents and use of the Stack, illustrating specifics with details from an online course, Introduction to Biomedical and Public Health Informatics (ME 600.903) taught at Johns Hopkins for the past 6 years, which experience also provides a “verification” of the Stack (to use Friedman’s and Wyatt’s term for a subjectivist evaluation [[6]]).

2. Method: The Informatics Stack

The notion of a “Stack” is the metaphor of the OSI stack – itself, standard content in informatics education – where one level communicates to other levels only through intermediary levels [[7]]. The Informatics Stack is more like a Russian doll: higher levels encompass lower levels. So one reads ► [Figure 1] as follows: Technology supports the representation and use of Data, Information, and Knowledge (acted on by Algorithms); these activities are encapsulated in Modules which work together, in a system-of-systems as an Information System that supports Workflow, and with which human agents interact and who exhibit a range of Behaviors as a result of their level of technology Adoption. That workflow supports the Functions and Goals of a variety of participants playing their Roles, who participate in an Organization that functions in a particular World. The headings of the current article take their language from the Stack.

Figure 1 The Informatics Stack. The levels are defined in the text: lower levels support or are encompassed by higher levels. The bidirectional arrows indicate the specific focus of interchange or interoperability at each level. The domains of privacy, confidentiality, and security are indicated. Evaluation, Ethics, Law, Research have different identities at each level (not shown). The horizontal Line separates the what (Above) from the how (Below). Some Stack names are used as headers in the current article.

We here provide definitions. Example educational objectives, used in the Introductory course, are presented in ► Online Appendix 1.

World

The context (political, economic, policy) within which the organization operates.

Organization

The local context. May be a formal organization (e.g., health system) or informal (a family). For an organization that is part of another organization, the larger organization may be the “World” for the first organization.

Perspective/Role

A role may be filled by many kinds of people (e.g., physicians and nurse practitioners take on the role of care providers) while a single person may have multiple roles (e.g., a doctor is a care provider, a teacher, a learner, an administrator, and a researcher [[8]].

Goals/Functions

The goal is what needs to be accomplished (by role, to maintain an individual’s position within or simply to support the organization). They may decompose into objectives, as in the Logical Framework [[9]]. The functions are how the goal will be achieved. There may be many functions for a single goal.

Workflow/Behavior/Adoption

This level encompasses the work and behavior performed by a role (almost always in conjunction with other roles), and includes concerns like human-computer interaction, and adoption of technologies and new workflows.

Information System

The information ecology that supports the Workflow: how does the system appear to the user? May be explicitly a single system or may be, in reality, an interconnected system of systems. Electronic Health Record (EHR) Systems constitute the classic information system in clinical care, but so might a personal, wearable network, for consumer informatics, whose boundary may end at the smart phone or may extend into the patient portal, on the health system’s side.

Modules

Software (or other) entities that have a single information function. Modules may be information systems of their own. For instance, to the pediatrician, an Immunization Information System (IIS) is a module, supporting the doctor’s need for information about a child’s vaccine status; the IIS itself has multiple components. To the public health agency tasked with maximizing immunization rates, the IIS may be the Information System. Thus, an Information System (higher level) is and acts as a system of systems [[10]].

Data, Information, Knowledge, Wisdom (DIKW), Algorithms

Computer scientists separate data structures from algorithms, so we follow their lead here, and place DIKW on one side and Algorithms on the other, at this same level [[11], [12]]. The ways that data are organized, presented, or aggregated result in information. A representation of a patient’s data within the context of the patient’s presentation, e.g., as a JSON object (machine-oriented) or as a trend-line graph (user-oriented), is information. The rules by which that information is used for action constitutes knowledge. The understanding of when (and when not) to follow the machine’s rules is wisdom. The algorithms the machine uses to collect and store data, to create or display information from data, or to learn or apply knowledge to the information comprise the Algorithms.

Technology

Comprises the hardware, software, networking, and the like, through which the higher levels of the Stack are implemented. (E.g., the laptops, smartphones, used by students, and the servers used by the LMS.)

2.1 Above the Line and Below the Line

The Stack separates into upper and lower parts: Information System and up are the upper; Information System and down are the lower. We call the boundary between them (within the Information System level), the Line: Above the Line, the concerns are what will the information system accomplish; Below the Line, how it will accomplish them. The Line represents the task of translating for others, across the Line [[4]]. Classifying verbalized concerns by stakeholders as Above or Below the Line is an important competency in using the Stack in practice.

2.2 Use Cases, Revisited

Let us apply the Informatics Stack to our Use cases.

1. Readiness assessment. The informatician begins by understanding the context (World and Organization) of the stakeholder, then, ideally, works down the Stack to determine the Above the Line needs, in the language of the stakeholder; to determine the degree of health IT/informatics sophistication held by the stakeholder, to gauge the level of “translation” required below the Line; to monitor proposed designs for inconsistencies between the Below the Line IT service proposed and the Above the Line needs; and to figure out the real reason behind the request, in the case of inconsistencies.

2. IT enthusiast. The informatician’s goal is to figure out what is problem that the enthusiast thinks she is solving with the new technology, essentially working up the Stack.

3. Formative evaluation. The informatician is given both an Above-the-Line and a Below-the-Line reality, and the task is to assess their across-the-Line consistency, taking the stage of the project into account.

4. Report review. This use case in practice is similar to formative evaluation, except that there are no stakeholders to interview and functions more like a summative evaluation. This use case serves as the basis of the students’ Final Project. The Stack can also be used to structure a report comparing As–Is to To–Be solutions.

2.3 Vertical Issues of the Stack

There are “vertical” issues common across levels, although there may be specifics at any one level. Six such issues are standards, privacy/confidentiality/security, ethics, law, evaluation, and research. Current paradigms for informatics evaluation and research focus either on stage of development (e.g., conceptual, lab, field, practice [[13]]) or research method, divided into quantitative and qualitative methods [[6]]. The Table in ► Online Appendix 2 shows how those research and evaluation methods should be targeted to their position in the Stack. The table links scientific disciplines to each level; our list is informed by Shortliffe and Cimino [[14]] and by the IMIA educational framework [[5]] but we go the extra step of showing how and where the links are made.

2.4 Use in the Course

In our graduate course, ME 600.903 Introduction to Biomedical and Public Health Informatics, required of informatics masters students and elective for masters of public health, of information security, and of business administration, we structure the 8 weeks of topics according to the Stack, starting at the bottom of the Stack, after an Introduction. The summative exercise is a Final Project, where each group of 4–6 students reviews a system, generally a Davies award winner from the previous 5 years [[15]]. Davies awards are awarded by the Health Information Management Systems Society, based on extensive reports submitted by the candidate institutions, and so should reflect the best on those candidates. The students complete a structured review of that report in a group-shared wiki. The items for review comprise the Stack (see ► Online Appendix 2). Our review of their answers comprises qualitative evaluation of the Stack’s completeness and educational use.

The projects are scored (by the course director, HL) based on completeness, correctness, quality, and, most importantly, consistency across the levels. Finally, students are asked to reflect on what they saw in the report that they would not have seen without the Stack framework, and are asked what aspects they saw in the report that were not addressed by the Stack. Specifics of the Stack and the Project were changed over the years in response to these reflections. In particular, detailed instructions were provided, to overcome ambiguous earlier instructions and to reinforce that content of their report may change as they explore the target report in more detail; the assignment was pared down to 2 perspectives (preferably, one clinically related, the other, public health); an abstract was required, to help the students to summarize their analysis; clarifications were added on the differences between outputs and outcomes of an information system; the required method of describing workflow was limited to using swimlane diagrams; a section was added, highlighting information systems as systems of systems, with modules being systems in their own right; sections were added on testing and evaluation (and the differences between the two), interoperability, security, and ethics.

3. Results, Evaluation: Students’ Experience with the Stack

► [Table 1] shows the numbers of students and groups participating in the course. From 2013 and on, enrollment was a mix of graduate informatics, public health, clinical, computer science and business students.

Table 1

Enrollment and number of Final Project Group.

Year	N Students	N Groups
2010[*]	94	26
2011[*]	103	22
2012[*]	126	23
2013	59	11
2014	60	13
2015	62	13
2016	34	8
Total	538	116

* Training subsidized by workforce-development grant from the US Office of the National Coordinator for Health I T.

The reflections after 2013 were reviewed in detail, when the Stack details and the Final Project requirements were stable. (The Johns Hopkins Institutional Review Board judged this review as exempt from IRB review.) Of the surprises, every level of the Stack was identified by some student each year as something they might have missed. At times, two different students in the same group identified complementary levels (e.g., Below and Above the Line). Many described insight at how they previously conflated issues at different levels. (E.g., “I would mix up Goals/Functions with Workflow” [2014] or, “Without the frameworks I found it easy to be overwhelmed by the organizational ‘C-suite speak’.” [2014] Many students appreciated the systematic or structured examination. It “provided coherence as I went through the details of every layer of the stack.” [2014] Several reported difficulty in separating their analysis into the levels – and appreciated the benefit of having done so in the Project. Only one students reported, “I am not truly certain that the framework added to my comprehension.”

Of the issues not addressed, these fell into 3 components: First were issues that are not included in the Stack. Primarily, that the Stack does not account for temporal issues: the “nuts and bolts” of development or implementation (over time), “continuous process improvement” (and feedback), “sustainability,” “change management.” Other issues included “likelihood of success,” “dynamic interactions between the consumers of the system,” “community of users,” “scaling up,” and “cost.”

The second set of criticisms were issues the students said were not addressed, but actually were. Their raising these concerns pointed to possible weaknesses in the course (e.g., needing to stress certain points more), and not to the Stack itself. Thus, some students pointed to “adoption” and “human factors” as not addressed. Another, that interoperability was not addressed, despite an entire module devoted to standards; the layering of standards into levels shown in ► [Figure 1] was the response to that criticism. Yet another, that the “whole is often greater than the sum of its parts,” which is exactly the point of systems thinking embodied by the course.

Third were issues addressed in the Stack and in the course, but that the particular student missed or was missed in the particular report being critiqued. Fourth were observations, such as that, unlike the OSI stack, what happens at one levels of the Informatics Stack does affect choices made at all other levels. [2015] For instance, the nursing role affects which data are needed in a documentation module differently from a hospitalist–physician role; in the OSI stack, a packet of data at the datalink level is handled the same, whether the packet represents an email message or a Web page request.

Some students already applied the Stack outside the course: “I am using it to analyze another project, and it just threw light on me on what should be done right now and next.” [Computer science student]

Informally, alumni in both clinical and public health contexts have reported using the Stack in their work in exactly the “bedside” manner it was intended. Others have used it as a basis of consulting work, e.g., in creating a readiness-for-HIT checklist for a hospital.

4. Discussion

We submit the Stack as a heuristic for students to gain an initial purchase on how to tackle the complexity of an informatics environment in many informatics subdisciplines (e.g., clinical, public health). The Stack incorporates many concerns from other informatics curricula and informatics desiderata. We have demonstrated that an 8-week online course is able to transmit the core knowledge of the Stack and to have them demonstrate their ability to use it in the heuristic way it is intended.

We are not the first to suggest that informatics includes the levels described here. For instance [and we provide approximate Stack equivalents], the US Government’s Federal Enterprise Architecture includes the levels Performance Reference Model (PRM), Business Reference Model [Organization], Data Reference Model [DIKW], Application Reference Model [Modules], and Infrastructure Reference Model [Information Systems/Technology] [[2]]. The PRM addresses the temporal components not addressed by the Stack.

The more recent mHealth Assessment and Planning for Scale (MAPS) Toolkit [[16]] involves “Axes” of Groundwork [World], Partnerships [Organization], Financial Health [Organization], Technology & Architecture [Technology, DIKWA], Operations [Information System], and Monitoring & Evaluation [Evaluation]. While the Axes and the Stack levels appear comparable, the Toolkit is geared for the processes of deployment and system evolution.

The notion of separating what is to be accomplished from how it is to be accomplished is an essential component of project management [[17]], often itself a component of informatics curricula, and other areas, such as statistical analysis. But the distinction is not stressed in informatics education.

The Stack integrates information from many informatics textbooks currently available. Reference has already been made to Shortliffe and Cimino [[14]], with its set of broad foundational chapters. A core diagram shows the many sciences that contribute to informatics, but does not articulate how they do so, as in ► [Table 1]. The Hoyt and Yoshihari text is more practically oriented, with most chapters focused on a variety of clinical systems, resources, or functions (such as disease management), but without an organizing framework [[18]]. Enrico Coeria, who himself has written about complex adaptive systems and the “wicked problem” of healthcare [[19], [20]], published a textbook that is focused on the clinical perspective [[21]]. The recent Health Informatics: A Systems Perspective, means, by the term “system,” the healthcare organization, not the system of systems that is the focus of the Stack [[22]]. Another text takes the socio-technical perspective, one organizing principle of the Stack [[23]]. The most recent text on public health informatics includes notions of “context” and “science” of the field: government/legislation, information architecture, data sources, data tools, standards, privacy, EHRs, ethics, project management [[24]].

The article that defines the Core Contents for the US Clinical Informatics Sub-specialty includes the high-level outline, Fundamentals (which includes Ethics and Legal/regulatory issues), the Functions of Clinical Decision Making and Care Process Improvement, then Health Information Systems (which includes our Technology and DIKW levels), and Leading and Managing Change (the temporal components) [[25]]. The new competencies for Masters Degree CAHIIM accreditation in the United States employ the “foundations” of Health; Social and Behavioral Science; Information Science and Technology; and their intersections [[26]]. The 3 foundations correspond to the top, middle, and bottom levels of the Stack, respectively.

The limitations to the Stack were laid out by the students; most of them have to do with time and with the actual realities of building a system. We judged these concerns as having more to do with the dynamic process of software development, rather than the more static (single-time point) notion of global or formative evaluation or assessment intended here. Our curriculum addresses these concerns in two courses that focus on software development and deployment, ME 600.900 From Design to Deployment and ME 600.902 Leading Change Through Health IT. Further criticism is that it is broad with little depth. But such a trade off is appropriate for an introductory course. Finally, while the Stack is taught as focused on clinical and public health contexts, it applies to bioinformatics as well, if biologists (rather than subcellular components!) are the true focus. Similarly for “consumer informatics” or “precision medicine informatics,” and the patient.

5. Conclusion

The Informatics Stack is a compact method of conveying informatics principles and skills, and embodies informatics thinking by its very nature. It is a broadly-encom -passing heuristic whose application can be learned and applied by students from a wide variety of backgrounds in an 8-week course.

Acknowledgment

Thanks to Sara Hill for helping construct the first versions of this course, to Katie Washington for providing an early critical review, and to the students of ME 600.903, who took our content to heart and gave invaluable feedback for both the content and the process of the course.

^* Supplementary material published on our website https://doi.org/10.3414/ME16-01-0152

• References

• 1 Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med 2002; 77 (05) 361-367.
• 2 US Office of Management & Budget. Federal Enterprise Architecture Framework Version 2. 2013: 1-434. Available from: http://www.whitehouse.gov/omb/e-gov/fea/
• 3 Pearl J. Heuristics: Intelligent Search Strategies for Computer Problem Solving. New York: Addison-Wesley; 1984
• 4 Glaser JP. The role of the chief information officer in the health care organization in the 1990s. Top Health Inf Manage 1993; 13 (03) 62-68.
• 5 Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W, Hovenga E, Lun KC, Marin H, Martin-Sanchez F, Wright G. Recommendations of the international medical informatics association (IMIA) on education in biomedical and health informatics. Methods Inf Med 2010; 49 (02) 105-120.
• 6 Friedman CP, Wyatt JC. Evaluation methods in biomedical informatics. 2nd ed.. New York: Springer; 2006
• 7 ISO/IEC 7498 Information processing systems - Open Systems Interconnection - Basic Reference Model. Geneva: 1989
• 8 Medical School Objectives Project. Report II: Contemporary issues in medicine: Medical informatics and population health. Washington, DC: Association of American Medical Colleges; 1998
• 9 The Logical Framework. Practical Concepts Incorporated. 1979 Available from: http://pdf.usaid.gov/pdf_docs/PNABN963.pdf
• 10 Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, Gunter C, Musen M, Platt R, Stead W, Sullivan K, Van Houweling D. Toward a science of learning systems: a research agenda for the highfunctioning Learning Health System. J Am Med Inform Assoc 2015; 22 (01) 43-50.
• 11 Aho AV, Hopcroft JE, Ullman JD. Data Structures and Algorithms. Pearson; 1983
• 12 Goodrich MT. Roberto Tamassia. Data Structures and Algorithms in Java. Hoboken, NJ: Wiley; 2014
• 13 Stead WW, Sittig DF. Building a data foundation for tomorrow’s healthcare information management systems. Int J Biomed Comput 1995; 39: 127-131.
• 14 Shortliffe EH, Cimino JJ. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. New York: Springer; 2014
• 15 HIMSS Davies Awards. 2015 [cited 2015 Sep 27]. Available from: http://www.himss.org/library/davies-awards
• 16 The MAPS Toolkit: mHealth Assessment and Planning for Scale. Geneva: 2015
• 17 Wysocki R, Beck R, Crane D. Effective Project Management. New York: Wiley; 1995
• 18 Hoyt RE, Yoshihashi A. Health Informatics: Practical Guide for Healthcare and Information Technology Professionals. Pensacola, FL: Information Education; 2014
• 19 Westbrook JI, Braithwaite J, Georgiou A, Ampt A, Creswick N, Coiera E, Iedema R. Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory. J Am Med Inform Assoc 2007; 14 (06) 746-755.
• 20 Coiera E, Hovenga EJS. Building a sustainable health system. Yearb Med Inform. 2007: 11-18.
• 21 Coiera E. Guide to Health Informatics. 3rd ed. Boca Raton, FL: CRC; 2014
• 22 Brown GD, Patrick TB, Kalyan P. Health Informatics: A Systems Perspective. Arlington, VA: Health Administration Press; 2012
• 23 Whetton S. Health informatics: a socio-technical perspective. Oxford; New York: Oxford University Press; 2005
• 24 Magnuson JA, Fu PC. Public health informatics and information systems. 2nd ed. London: Springer; 2014
• 25 Gardner RM, Overhage JM, Steen EB, Munger BS, Holmes JH, Williamson JJ, Detmer DE. Core Content for the Subspecialty of Clinical Informatics. J Am Med Informatics Assoc 2009; 16 (02) 153-157.
• 26 AMIA Accreditation Committee. Foundational domains: Draft for public comment. 2016 [cited 2016 Dec 26]. Available from: https://www.cahiim.org/documentsSept%2015_Draft%20for%20Comment__AmiaAC%20Foundational%20Domains.pdf