Toward an Information Infrastructure for Global Health Improvement

Summary

Profound global challenges to individual and population health, alongside the opportunities to benefit from digital technology, have spawned the concept of the Learning Health System. Learning Health Systems (LHSs)--which can function at organizational, network, regional, and national levels of scale--have the capability of continuous data-driven self-study that promotes change and improvement. The LHS concept, which originated in the U.S. in 2007, is rapidly gaining attention around the world. LHSs require, but also transcend, the secondary use of health data. This paper describes the key features of LHSs, argues that effective and sustainable LHSs must be supported by infrastructures that allow them to function with economies of scale and scope, and describes the services that such infrastructures must provide. While it is relatively straightforward to describe LHSs, achieving them at the high level of capability necessary to promote significant health benefits will require advancements in science and engineering, engaging the field of informatics among a wider range of disciplines. It also follows from this vision that LHSs cannot be built from an imposed blueprint; LHSs will more likely evolve from efforts at smaller scales that compose into larger systems.

Keywords

Learning Health Systems - infrastructure - secondary use of health data - population health - acquisition, knowledge (computer) - data aggregation

Introduction

We live in an era of profound challenges and significant opportunities for global health. Among them, the rate of health data generation is increasing almost exponentially. For example, experts worldwide believe that, by 2020, people and health systems in the U.S. and around the world will generate data 50 times more rapidly than they did at the start of the decade [[1], [2], [3]]. Biomedical knowledge generation worldwide has been accelerating at a correspondingly profound rate [[4], [5]]. One estimate suggests that the doubling time of biomedical knowledge--which stood at 50 years in 1950 and 7 years in 1980--will decrease to an estimated 73 days by 2020 [[6]]. The year addressed by this IMIA Yearbook, 2017, coincidentally marks the 17^th anniversary of a study documenting the 17-year lag between the generation of new biomedical knowledge and its widespread application to health practice [[7]], and also the 17^th anniversary of a landmark study describing the prevalence and impact of patient safety issues that take a major toll in human lives [[8]]. These developments occur against a backdrop of declining individual and public health indicators coupled with pressures to decrease public and private health expenditures, but also a backdrop of potentially disruptive technological advancements [[9], [10], [11], [12], [13]].

In response to the challenges, as well as to seize the opportunities, the past decade has witnessed the introduction and evolution of the concept of Learning Health Systems (LHSs) [[9], [14], [15]], capable of continuous self-study and improvement. LHSs embrace but also transcend the secondary use of health data, the theme of the 2017 issue of the IMIA Yearbook, pointing to what may be possible through complete cyclical processes that mobilize health data, analyze it to create new knowledge, and apply that new knowledge to improve the health of individuals and populations. LHSs are supported by infrastructures that enable these processes to take place routinely and with efficiency of scale and scope. Augmenting their potential to improve health by analyzing data and acting on it, LHSs also hold the potential at a meta-level to improve their own capacity to learn.

In this essay, we describe our vision of LHSs, connect this vision to the concept of infrastructure, and argue for the development of LHSs that would occur primarily through coordinated composition of organizational, network, and regional initiatives as opposed to centralized, top-down, initiatives. While initiatives that are imposed top-down generate immediate attention and may lead to localized and short term benefits, we take the position that widespread enduring benefits to health will result from a more deliberate and evolutionary process of infrastructure co-production in which the full spectrum of stakeholders are directly engaged. Above all, this essay is an expression of hope and optimism for a future of better health for all people, suggesting a pathway--albeit a pathway fraught with challenges--to this future.

Learning Health Systems and Learning Cycles

The concept of LHS was first advanced by the U.S. Institute of Medicine (now the National Academy of Medicine) in 2007 [[16], [17]]. In the ensuing 10 years, the concept has gained increasing attention, initially in the U.S., but currently and progressively around the world. This is seen, in part, in a substantial literature with 1,940 Google Scholar citations retrieved from a search on the term “Learning Health System” conducted by the authors in January of 2017.

In the U.S., several recent developments point to increasing interest in and development of LHSs. A national LHS Summit, held in 2012, established consensus around 10 LHS Core Values that have been formally endorsed by 109 organizations [[18]] (see [Table 1]). In 2015, the U.S. Office of the National Coordinator for Health IT, citing these LHS Core Values, established a nationwide LHS as the pinnacle goal of its 10-year Interoperability Roadmap [[19]], and other federal agencies have expressed strong endorsement of the concept [[20], [21]]. The U.S. National Science Foundation (NSF) has supported workshops to identify the research challenges to achieving a high functioning LHS [[22], [23], [24], [25]]. More recently, the Computing Community Consortium (CCC) [[26]], which advises and works with national research policy makers to understand and shape the future of computing and computing research, has funded a series of three workshops on the broader topic of Cyber-Social Learning Systems (CSLSs), of which LHSs are a special case [[25]]. The Patient-Centered Outcomes Research Institute (PCORI) has developed a health data and analytics network at national scale [[27], [28]]. The literature reveals a panoply of reports of individual organizations seeking to achieve the capabilities associated with LHSs [[29], [30], [31], [32], [33], [34]].

Table 1

Learning Health System Consensus Core Values

1.)	Person-Focused	The LHS will protect and improve the health of individuals, families, groups, communities, and the general population by informing choices about health and care.
2.)	Privacy	The LHS will protect the privacy, confidentiality, and security of all data, as well as build trust among all stakeholders.
3.)	Inclusiveness	Every individual and organization committed to improving the health of individuals, communities, and diverse populations is invited and encouraged to participate.
4.)	Transparency	With a commitment to integrity, all aspects of LHS operations will be open and transparent to safeguard and deepen the trust of all stakeholders.
5.)	Accessibility	All should benefit from the public good derived from the LHS; therefore, the LHS should be available and should deliver value to all.
6.)	Adaptability	The LHS will be designed to enable iterative, rapid adaptation and incremental evolution to meet current and future needs of stakeholders.
7.)	Governance	The LHS will have that governance which is necessary to support its sustainable operation, to set required standards, and to build and maintain trust.
8.)	Cooperative and Participatory Leadership	The leadership of the LHS will be a multi-stakeholder collaboration across the public and private sectors.
9.)	Scientific Integrity	The LHS and its participants will share a commitment to the most rigorous application of science to ensure the validity and credibility of findings.
10.)	Value	The LHS will support learning activities that can serve to optimize both the quality and affordability of healthcare.

Interest in LHSs has spread across the globe [[35], [36], [37], [38]]. Specifically, in the European Community, the TRANSFoRm project has addressed some of the challenges of achieving a robust infrastructure for LHSs [[39], [40]]. The European Institute for Innovation through Health Data seeks “to tackle areas of challenge in the successful scaling up of innovations that critically rely on high-quality and interoperable health data” [[41]]. In the U.K., the LHS concept has become a beacon for health improvement [[36], [42], [43], [44]]. The Swiss government has recently announced a national LHS initiative [[45]]; and in Asia, collaborative efforts joining Japan to Taiwan have resulted in an incipient Consortium for Asia Pacific Learning Health Systems [[46], [47]].

We frequently refer to LHSs in the plural because there is no single accepted definition of the term, and it is almost certain that there will not, in the end, be a single monolithic LHS spanning the globe, but rather a global system of more and less tightly integrated LHSs. LHS proponents also differ in their interpretation of the “H” appearing in the abbreviation. Some interpret the “H” restrictively to stand for “health care”, focusing the LHS concept on improving the delivery of care to individuals and aligning it almost exclusively with health care quality, safety, and research. Others interpret the “H” more broadly to connote “health” in recognition that an LHS can promote public and population health.

Information is central to the LHS concept, and an LHS can improve health through successive iterations of “learning cycles”. The concept of iterative improvements achieved prominence through the efforts of W. Edwards Deming and has subsequently been elaborated in a robust literature [[48], [49], [50]]. At the highest level of abstraction, learning cycles convert data to knowledge (D2K), apply that knowledge to influence performance (K2P), and document changes in performance to generate new data that seeds the next iteration of the cycle (P2D) (see [Figure 1]). Early visions of Learning Healthcare Systems focused exclusively on the reuse of digital data generated by care delivery to drive the D2K component of the learning cycle [[16]]. The broader vision of Learning Health Systems recognizes that a much wider array of data sources--including purposefully collected data outside of care experience, such as geospatial data or data addressing the social determinants of health--can be important components of the learning process.

Fig. 1 The Learning Cycle

As an illustrative example, a learning cycle might be organized to reduce the incidence of falls by residents of long-term care facilities. Participating facilities--forming a learning community--could be those in a given region, those owned by a particular company, or a self-organized group of volunteers. The D2K segment of the cycle would engage participating facilities in describing what practices they currently employ to prevent falls. These data, aggregated across the community, would be analyzed in relation to each facility’s fall rate. The results of the analysis would likely identify some practices that could potentially reduce falls. The community would review these results and decide which have sufficient credibility to generate recommendations for change. This engages the K2P segment of the cycle, wherein each facility would receive recommendations, ideally customized to its own specific circumstances. Each facility would then respond to the recommendations, in whole or part, and perhaps make some additional changes stimulated by the recommendations. This engages the P2D segment of the cycle that would document the changes made at each facility, and resulting effects on fall rates, which initiates the next iteration of the cycle.

Characteristics of a Fully Functional Learning System

For the purposes of the discussion to follow, we will describe an LHS as a socio-technical system with the primary goal of significantly and safely improving health while reducing costs and other harms. An LHS is a cyber-social system composed of people and technology [[51]]. A fully functional LHS will exhibit all of the following five attributes [[52]]:

1. The health-related characteristics and experiences of very large numbers of persons, along with other relevant data types, are securely available as data to learn from. The D2K component of learning cycles requires data in sufficient quantity and of sufficiently high quality to generate credible findings. Efficient and scalable LHSs require all or a significant fraction of this data to be routinely collected and persisted in data marts that can be centralized or federated, institutionally- or patient-controlled. Data can describe individuals who play a variety of roles; it can also describe processes, organizations, and environments. Control over access to and use of personal data is governed in ways that protect the equities of all stakeholders including especially the individuals to whom the data pertain.

2. Best practice knowledge derived from these data is available to support health-related decisions and actions by individuals, care providers, as well as health service managers, planners, policy-makers, and payers. To enable the K2P function, the knowledge generated by LHSs must itself be represented and persisted in machine-executable forms. An LHS must know what it knows, and be able to compute from that knowledge advice to decision makers that is: specific to the decision setting, associated with a level of uncertainty, and, ideally, customized to the recipient of the advice [[53]]. The advice generated by the K2P function transcends the traditional clinical decision support paradigm focused on recommending care for a given patient. K2P can offer recommendations to all participants of the health ecosystem and address changes that may be implemented at the individual, group, organizational, or system levels.

3. Learning and health improvement are routine and continuous processes. Every health system is capable of self-study and improvement when faced with an externally imposed stimulus such as potential loss of accreditation, or an internal crisis. By contrast, in a learning health system, there is a shared recognition of the need for continuous improvement in the absence of crisis or external imposition, driven by a shared belief that the LHS approach is the best means to achieve health improvement at lower cost with greater safety. As such, an LHS will be characterized by multiple learning cycles ongoing simultaneously [[54], [55]].

4. Infrastructures enable the routine execution of multiple learning cycles. Even though each of the simultaneous learning cycles in an LHS addresses a unique health problem, all learning cycles depend on a common set of supportive services. The technologies, policies, and standards comprising these services constitute the infrastructure for the LHS. If these infrastructural services are shared across learning cycles, LHSs can function with economies of scale and scope, as discussed below.

5. Stakeholders within the system see value in the above activities and view them as part of their culture. Continuous learning is an intensely human process. Learning cycles will be undertaken by diverse groups of individuals, forming learning communities, who are driven by the motivation to solve problems that are barriers to improving health while reducing costs, harms, and other externalities. External motivations can incentivize participation in these learning communities, but the majority of the motivation must originate within the culture of the organization. If asked why he/she participates in the activities of an LHS, perhaps the best answer from a community member would be: “This is who we are and this is what we do.”

The above five characteristics can serve many purposes. They can serve as important guideposts to inform the strategy for realizing LHSs, for identifying key success factors, for quantifying and measuring progress and developing relevant success metrics, and for enabling LHSs to learn from one another. They can also be considered to comprise a set of continua suggestive of a capability-maturity model [[56]].

Infrastructures, Scale, and Informatics

The fundamental process of an LHS is the learning cycle as depicted in [Figure 1]. From an informatics perspective, a set of learning cycles, each addressing a specific health problem, can function as a learning system when they are supported by an infrastructure serving multiple simultaneous learning cycles, as depicted in [Figure 2]. Some learning cycles will progress more rapidly, others more slowly, as a function of the health problem being addressed. An infrastructure provides a set of integrated services, each service supporting a component of the learning cycle and all services together establishing a smoothly articulated workflow from each stage to the next in the learning process.

Fig. 2 Infrastructure Supporting Multiple Simultaneous Learning Cycles

As previously noted, infrastructure endows the LHS with economies of scale and scope. Without infrastructure, each learning cycle is figuratively a “tub on its own bottom” requiring its own concepts, methods, tools, and support systems to undertake the necessary activities of D2K, K2P, and P2D. Without infrastructure, implementing each new cycle requires people specific to that cycle and the deployment of cycle-specific methods, tools, and processes. Under that circumstance, implementation of the next cycle will cost almost as much as did its predecessor. Also, methods developed from an initial cycle may not transfer to successive ones because the methods of the initial cycle may be over-fitted to the health problem it addresses.

[Figure 3] illustrates the services required by an LHS infrastructure and provides a schematic representation of a complete infrastructure. The inner circumference of the figure displays the fundamental D2K, K2P, and P2D components of the learning cycle. The boxes on the outer circumference depict the infrastructure services required to support each stage of the complete cycle. The circle that connects the boxes illustrates the requirement that the services articulate into a smooth workflow.

Fig. 3 Prototypic LHS Infrastructure Services

The services depicted in [Figure 3] support and engage people, process, policy, and technology. In their totality, they emphasize that D2K infrastructure services are necessary to, but not sufficient for, supporting an LHS. Of special significance to a complete infrastructure are K2P services required to persist and manage knowledge so the system “knows what it knows”. K2P services enable the system to represent knowledge in machine-executable forms so it can be rapidly updated as the system learns, it rapidly generates advice to inform decisions, and it tailors the advice to the needs and characteristics of the recipients [[53]]. Methods drawn from the behavioral and implementation sciences to promote practice change complete the services necessary to support K2P [[57], [58]].

Learning cycles can occur at varying levels of scale. They can be undertaken by a single organization, by networks of otherwise independent organizations, by specialized disciplines that span organizational, legal, and geographic boundaries, and across geographical regions varying in size, from counties to states/provinces to entire nations. Because the actions necessary to execute learning cycles are, to a significant degree of approximation, invariant across these levels of scale, LHSs exhibit important fractal-like properties of self-similarity [[59], [60]]. The infrastructure supporting LHSs, displayed in [Figure 3], is capable of delivering the same services at any level of scale. This self-similarity property has strong implications for the evolution of LHSs at large scale, as will be discussed below.

Fully functional, integrated LHS infrastructures, as suggested by [Figure 3], do not yet exist, nor do architectures that would underpin such infrastructures. Nonetheless, pieces of LHS infrastructure that provide specific services shown in [Figure 3] are emerging. For example, PopMedNet and I2B2 are widely deployed D2K infrastructure components [[61], [62]]. Apervita [[63]], Semedy [[64]], and the Knowledge Grid [[53]], provide K2P services by representing and curating biomedical knowledge in machine executable forms and generating from that knowledge messages tailored to support decisions taken by care providers, patients, and/or managers. Systems such as Deliberative Dialogue can support the work of learning communities [[65]]. The European Community’s TRANSFoRm project [[39]] provides scalable infrastructure that supports both D2K and K2P. Further maturation of these existing services, development of infrastructure components supporting other needed services, and their integration into a coherent workflow are the fundamental challenges of an LHS, comprising some of the most interesting and important challenges faced by informatics in the coming decade. Integration of these services will require solutions to many additional problems in software and system design, including the formation of basic architectural design rules, the standardization of representation and especially the interpretation of data and knowledge that flow across the services, the distribution of local control and system-wide learning that both rely on and inform local control, the need to accommodate and account properly for data of varying quality, and the need to assure the trustworthiness of system operation in the face of daunting challenges, ranging from natural disasters to human error and attacks on data and system function coming from capable adversaries. Among other things, the LHS Core Value of Scientific Integrity (see [Table 1]) hinges on assaying the quality of data and the analytic methods used to learn from them; and societal trust in LHSs will in turn hinge on the presence of correct and compelling arguments that these systems have been constructed according to the highly demanding standards of quality, resilience, and integrity.

Toward a Health Improvement Ecosystem

We sit collectively at a propitious moment for the development of LHSs and the realization of their potential to improve individual and population health. As discussed previously, there is an intensifying collective interest in and enthusiasm for the LHS concept around the world. The vision of LHSs that are enabled by shared sets of socio-technical architectural and infrastructure elements points to how a global progression toward increasingly functional LHSs may occur.

To understand this, it is important to recognize that the LHS is much more than a “moonshot”, although it resembles the 1960’s moonshot in its ambitiousness and, perhaps also, the enthusiasm it is generating. Paraphrasing John F. Kennedy, the widespread achievement of LHSs will not be one organization “going to the moon”, it will be the entire world [[66]]. But unlike the original “moonshot”, LHS is not a direct build-out toward a single sharply defined end goal that we already largely know how to achieve modulo certain engineering details. LHSs are architectural, infrastructural, organizational, and ultimately societal means to achieve a wide range of ends that themselves comprise a moving and evolving set of targets. LHSs will never be complete and will always continue to evolve. LHSs represent a socio-technical, as opposed to almost purely technical, enterprise, far more complex than sending a human to the moon and safely back. There are deep open scientific, engineering, and design questions, invoking a broad range of sciences, that must be addressed to achieve highly functional LHSs [[22]]. By sharp contrast, when President Kennedy announced in May of 1961 the goal of putting a person on the moon by the end of the decade, he did so knowing that the Saturn booster rocket, representing the most fundamental engineering challenge, had been built and would be test-flown four months later [[67]].

This conception of LHSs as ultra-large scale, socio-technical systems is inconsistent with a belief that any one entity, any one approach, or any one design can be imposed upon the existing health ecosystem to create an LHS. We offer instead an evolutionary vision of the journey toward LHSs that will invoke different modes of thinking than have been customarily applied to health and informatics problems. These new modes of thinking will be multi-disciplinary [[68]] and will invoke important concepts such as co-production [[69], [70]]; “chaordic” organizations [[71]]; ultra-large-scale systems [[72], [73]]; co-opetition [[74], [75]]; evolution at whole-industry scale; social sensing [[76]]; learning over highly distributed, locally autonomous nodes [[77]]; and methods for developing ultra-high-assurance software-intensive systems, such as those employing emerging techniques of proof engineering [[78], [79], [80]].

Progress toward LHSs will be an incremental journey toward an ecosystem that exists only in fragmented elements at the present time. A vision for how this evolution may occur includes three channels of development: 1) continuing development of the existing elements, each of which, in its own way, comprises a component of LHS at scale, 2) integrating the elements into more complete and scalable working systems, and 3) developing an international-scale program of basic and applied multi-disciplinary, scientific, engineering, and design research to inform all of the above.

1. Continuing development of the existing elements. Existing LHS elements include organizations that have achieved LHS-like capabilities and deployed many of the LHS infrastructure services depicted in [Figure 3]. In the U.S., individual organizations such as Intermountain Healthcare [[81]] Johns Hopkins [[60]], and the Mayo Clinic [[82]], have developed programs following the LHS model; and many more, by virtue of endorsement [[18]] of the LHS Core Values, have signaled future steps in that direction. At a higher level of scale, organizational networks exhibiting limited LHS properties are rapidly appearing. These include many of the PCORI Clinical Data Research Networks [[83]], the NIH Collaboratory [[84]], disease-focused initiatives such as CancerLinQ [[30]], and the Collaborative Quality Initiatives (CQIs) in the State of Michigan [[85]]. European initiatives such as TRANSFoRm [[39]] and EHR4CR [[38]] complement this list.

2. Integration and scaling. A key necessary step toward scalable and high-functioning LHSs will require the adoption over time of common infrastructural elements and overarching cyber-social system architectures across levels of scale. As this process occurs, it will enable entities at one level of scale to function virtually at higher levels of scale, in much the same way that individual organizations participating in PCORI Clinical Data Research Networks, through their use of a common data model and deployment of the PopMedNet data aggregation software, and inherit the ability to conduct clinical research at national scale [[86]]. While local implementations will necessarily and desirably be unique to given institutions, they will increasingly reflect and benefit from shared architectural characteristics: support for shared APIs and processes, conformance to consensus design rules [[87]], standardized formats, and interpretations for data and procedures [[88]]. This in turn will require agreements on minimal constraints that all participants agree to be bound by so that the system can work as a whole even while substantial autonomy to act and evolve is left to the individual components. The processes of integration and scaling will require effective mechanisms of governance, which themselves will evolve with experience. In particular, the world will need to learn how LHS policy making and governance functions should be distributed across private sector and governmental entities. Experience with the U.S HITECH program [[89]], the European epSOS program [[90]], the U.K. National Information Governance Board [[91]], and other similarly-focused initiatives, may provide useful experience to guide the way to effective governance.

3. Expanding research. As noted previously, the challenge of achieving high-functioning LHSs raises a broad array of open research questions [[92], [93]]. Major advances in deep machine learning, artificial intelligence, parallel processing, connectivity, “big data”, and other areas of computing must be complemented by equally profound advances in behavioral, social, and organizational sciences embracing ethics, policy and governance, legal and regulatory mechanisms. Particularly in light of the Core Values of the LHS described earlier, ethical dimensions of LHSs are of particular importance and have been addressed in several recent publications [[94], [95]]. Knowledge derived from many disciplines, from computer science and engineering, to the social, behavioral, and economic sciences, to architecture and the humanities, will be necessary for LHSs to realize their positive disruptive potential. Research initiatives that join the public and private sectors, such as The Farr Institute of Health Informatics Research of the U.K. [[96]] may provide important models for stimulating and supporting the necessary level of interdisciplinary research.

In conclusion, the authors recognize that by conventional modes of thought, LHSs likely appear quixotic and the products of magical thinking. This essay has sought to bring LHSs into the realm of the achievable by describing mechanisms through which belief in the LHS concept, adoption of a multi-stakeholder evolutionary approach, careful attention to work already done, dedicated trial-and-error implementation, application of new ways of thinking, and rigorous research could combine to generate sustained progress toward increasingly functional LHSs in pursuit of improved global health. We indeed need to learn how to learn, and this will not in any way be easy, but momentum is increasing. To the extent that the international informatics community has been searching for an ultra-grand challenge to address--a challenge that invokes every aspect of the field, integrates with all other success-critical disciplines, and unites all members in a global enterprise--it has found one in the Learning Health System.

• References

• 1 Vesset D, Olofson CW, Nadkarni A, Zaidi A, Mc-Donough B, Schubmehl D. et al. IDC FutureScape: Worldwide Big Data and Analytics 2016 Predictions. International Data Corporation; 2015
• 2 Pandit K. Aureus Analytics. Big Data and Analytics Trends for 2015. 2015 Jan 12. http://blog.aureusanalytics.com/infographic-big-data-analytics-trends-2015/
• 3 Tractica LLC. Wearable Device Market Forecasts. 2016 https://www.tractica.com/research/wearable-device-market-forecasts/
• 4 Druss BG, Marcus SC. Growth and decentralization of the medical literature: implications for evidence-based medicine. JMLA 2005; Oct 1; 93 (04) 499 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1250328/
• 5 Alper BS, Hand JA, Elliott SG, Kinkade S. How much effort is needed to keep up with the literature relevant for primary care?. JMLA 2004; Oct 1; 92 (04) 429 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC521514/
• 6 Densen P. Challenges and opportunities facing medical education. Trans Am Clin Climatol Assoc 2011; 122: 48.
• 7 Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform 2000: Patient-centered systems. 2000 Jan 1.
• 8 Donaldson MS, Corrigan JM, Kohn LT. editors. To err is human: building a safer health system. Institute of Medicine (IOM). National Academies Press; 2000. Apr 1.
• 9 McGinnis JM, Stuckhardt L, Saunders R, Smith M. editors. Best care at lower cost: the path to continuously learning health care in America. 1st edn. Washington, DC: Institute of Medicine (IOM). National Academies Press; 2013. Jun 10. http://www.iom.edu/bestcare
• 10 European Commission. Europe 2020 – for a healthier EU. 2012 http://ec.europa.eu/health/europe_2020_en
• 11 European Commission. Innovative healthcare in the 21st century. 2015 Jun 19. https://ec.europa.eu/digital-single-market/en/innovative-healthcare-21st-century
• 12 Kruger K. Big data: Where can data and analytics take healthcare in Africa?. KPGM South Africa; 2016. May 11. https://www.sablog.kpmg.co.za/2016/05/big-data-can-data-analytics-take-healthcare-africa/
• 13 Jimenez-Marroquin MC, Deber R, Jadad AR. Information and communication technology (ICT) and eHealth policy in Latin America and the Caribbean: a review of national policies and assessment of socioeconomic context. Rev Panam Salud Publica 2014; Jun; 35 (5-6): 329-36 http://www.scielosp.org/scielo.php?pid=S1020-49892014000500003&script=sci_arttext
• 14 Institute of Medicine (IOM). Table of Characteristics of a Continuously Learning Health Care System. 2012 http://www.iom.edu/Reports/2012/Best-Care-at-Lower-Cost-The-Path-to-Continuously-Learning-Health-Care-in-America/Table.aspx
• 15 Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med 2010; Nov 10; 02 (57) 57cm29-
• 16 McGinnis JM, Aisner D, Olsen L. editors. The learning healthcare system: workshop summary (IOM roundtable on evidence-based medicine). National Academies Press; 2007. Jun 1. https://www.nap.edu/catalog/11903/the-learning-healthcare-system-workshop-summary-iom-roundtable-on-evidence
• 17 Institute of Medicine (IOM). National Academy of Medicine (NAM). Learning Health System Series. https://www.nationalacademies.org/hmd/∼/media/Files/Activity%20Files/Quality/VSRT/Core%20Documents/LearningHealthSystem.pdf
• 18 Learning Health Community. Endorsers of the Core Values Underlying a National-Scale Person-Centered Continuous Learning Health System. http://www.learninghealth.org/endorsers/
• 19 Connecting Health and Care for the Nation. A Shared Nationwide Interoperability Roadmap Final Version 1.0. 2014 https://www.health-it.gov/sites/default/files/hie-interoperability/nationwide-interoperability-roadmap-final-version-1.0.pdf
• 20 Califf RM. Evidence Generation: Moving into the Modern Era. U.S. Food and Drug Administration (FDA); http://nationalacademies.org/hmd/∼/media/Files/Activity%20Files/Research/DrugForum/2016-OCT-19/Califf_Presentation.pdf
• 21 Bindman A. Building a Research Strategy to Support Learning Health Systems. Agency for Healthcare Research and Quality (AHRQ). Presentation at the 10th Annual AMIA Health Policy Invitational Meeting; Washington, DC: 2016. Sep 22. https://www.amia.org/2016-health-policy-invitational-meeting
• 22 Friedman CP, Rubin JC, Brown JS, Buntin M, Corn M, Etheredge LM. et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc. 2014 Oct 23: amiajnl-2014. http://dx.doi.org/10.1136/amiajnl-2014-002977
• 23 National Science Foundation (NSF). Toward a Science of Learning Systems: The Research Challenges Underlying a National-Scale Learning Health System: Findings from a Multi-Disciplinary Workshop Supported by the National Science Foundation. Washington, DC: 2013. Nov11. http://lhs.medicine.umich.edu/sites/default/files/downloads/NSF%20Report%20Ver%2011.11.13.pdf
• 24 National Science Foundation (NSF). Learning Health System Symposium at the 48th Annual Hawaii International Conference on System Sciences (HICSS-48). Kauai, Hawaii: 2015. Jan5-6. http://lhs.medicine.umich.edu/lks/hicss48
• 25 Computing Community Consortium (CCC). Visioning Workshops: Cyber-Social Learning Systems (CSLS). 2016. Aug 29-30 Seattle, WA: 2016 Nov 2-3 (Ann Arbor, MI), and 2017 Jan 24-25 (Washington, DC) http://cra.org/ccc/visioning/visioning-activities/2016-activities/cyber-social-learning-systems/
• 26 Computing Research Association (CRA). About the Computing Community Consortium (CCC). http://cra.org/ccc/about/
• 27 Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc 2014; Jul 1; 21 (04) 578-82 http://dx.doi.org/10.1136/amiajnl-2014-002747
• 28 Collins FS, Hudson KL, Briggs JP, Lauer MS. PCORnet: turning a dream into reality. J Am Med INfrom Assoc 2014; 21 (04) 576-577 http://dx.doi.org/10.1136/amiajnl-2014-002864
• 29 Klein S, Hostetter M. In Focus: Learning Health Care Systems. http://www.commonwealthfund.org/publications/newsletters/quality-matters/2013/august-september/in-focus-learning-health-care-systems
• 30 CancerLinQ LLC, a subsidiary of the American Society of Clinical Oncology. http://www.cancerlinq.org
• 31 Michigan Surgical Quality Collaborative. http://www.msqc.org
• 32 Ovretveit J, Nelson E, James B. Building a learning health system using clinical registers: A non-technical introduction. J Health Organ Manag 2016; Oct 10; 30 (07) 1105-18.
• 33 Health Affairs. Using big data to transform care. 2014 Jul 1; Volume 33. Issue 7 http://content.healthaffairs.org/content/33/7.toc
• 34 Precision Medicine Initiative of the National Institutes of Health. https://www.nih.gov/precision-medicine-initiative-cohort-program
• 35 European Commission. EU and US strengthen their collaboration on eHealth IT. 2016 Jul 28. https://ec.europa.eu/digital-single-market/en/news/eu-and-us-strengthen-their-collaboration-ehealth-it
• 36 The Health Foundation UK. The Health Foundation’s Annual Conference 2017: Permanent beta -fast forward to a learning health care system. 2017 Jan. http://www.health.org.uk/events/permanent-beta-fast-forward-learning-health-care-system
• 37 Ainsworth J, Buchan I. Combining health data uses to ignite health system learning. Methods Inf Med 2015; Sep 17; 54 (06) 479-87.
• 38 Daniel C, Ouagne D, Sadou E, Paris N, Hussain S, Jaulent MC. et al. Cross border semantic interoperability for learning health systems: The EHR4CR semantic resources and services. Learning Health Systems. 2016 Oct 21. doi: 10.1002/lrh2.10014.
• 39 Delaney BC, Curcin V, Andreasson A, Arvanitis TN, Bastiaens H, Corrigan D. et al. Translational medicine and patient safety in Europe: TRANS-FoRm-architecture for the learning health system in Europe. Biomed Res Int 2015; 1 http://dx.doi.org/10.1155/2015/961526
• 40 Delaney BC. TRANSFoRm: translational medicine and patient safety in Europe. Digital Infrastructure for the Learning Health System 2011; Sep 21: 198.
• 41 Kalra D, Stroetmann V, Sundgren M, Dupont D, Schlünder I, Thienpont G. et al. The European Institute for Innovation through health data. Learning Health Systems. 2016 Jul 25. doi:10.1002/lrh2.10008.
• 42 ISQua. 34th International Conference. Learning at the System Level to Improve Healthcare Quality and Safety. London, UK: 2017. Oct 1-4. http://www.isqua.org/Events/london-2017
• 43 Smart W. DRAFT: Target Architecture: Outputs from the Interoperability and Population Health Summit. CIO for Health and Social Care in England; 2016. Nov.
• 44 Foley T. Background: Need for LHS. The Learning Healthcare Project UK. Newcastle University UK; 2015. http://www.learninghealthcareproject.org/section/background/need-for-lhs
• 45 University of Lucerne and collaborators. Projektgebundene Beiträge 2017-2020 nach HFKG. Projektantrag 4. 2015. Dec. (revision 27. 2016 Jan.) (einzureichen durch swiss universities bis Ende 2016 Feb). Projekttitel: Swiss Learning Health System (SLHS) – A national platform for health systems and services research, policy and practice;
• 46 Consortium for Asia Pacific Learning Health Systems (CAPLHS). Email correspondence dated 2016 Dec 1 from: 李 友 專 Yu-Chuan (Jack) Li, M.D., Ph.D., FACMI, Professor and Dean, College of Medical Science and Technology (CoMST), Taipei Medical University, Chair, Dermatology Dept, Wan Fang Hospital.
• 47 Tohuku Medical Megabank Organization. http://www.megabank.tohoku.ac.jp/english/time-line/20160527_01/
• 48 Deming WE, Edwards DW. Quality, productivity, and competitive position. Cambridge, MA: Massachusetts Institute of Technology (MIT), Center for advanced engineering study; 1982. Jun.
• 49 Senge PM. The fifth discipline: The art and practice of the learning organization. Broadway Business, Crown Pub; 2006
• 50 Buckler B. A learning process model to achieve continuous improvement and innovation. The Learning Organization 1996; Aug 1; 03 (03) 31-9 http://dx.doi.org/10.1108/09696479610119660
• 51 Sheth A, Anantharam P, Henson C. Physical-cyber-social computing: An early 21st century approach. IEEE Intell Syst 2013; Jan; 28 (01) 78-82.
• 52 Friedman CP, Allee NJ, Delaney BC, Flynn AJ, Silverstein JC, Sullivan KJ. et al. The science of Learning Health Systems: Foundations for a new journal. Learn Health Systems. 2016 Nov 29. doi: 10.1002/lrh2.10020.
• 53 Flynn AJ, Shi W, Fischer R, Friedman CP. Digital Knowledge Objects and Digital Knowledge Object Clusters: Unit Holdings in a Learning Health System Knowledge Repository. In: 2016 49th Hawaii International Conference on System Sciences (HICSS) Conference on. 2016. Jan 5, 3308-3317 IEEE;
• 54 James BC, Savitz LA. How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff 2011; Jun 1; 30 (06) 1185-91.
• 55 Milstein A. Code red and blue—safely limiting health care’s GDP footprint. New Engl J Med 2013; Jan 3; 368 (01) 1-3.
• 56 Paulk M. Capability maturity model for software. Encyclopedia of Software Engineering; 1993
• 57 Brehaut JC, Eva KW. Building theories of knowledge translation interventions: Use the entire menu of constructs. Implement Sci 2012; Nov 22; 07 (01) 114.
• 58 Landis-Lewis Z, Brehaut JC, Hochheiser H, Douglas GP, Jacobson RS. Computer-supported feedback message tailoring: theory-informed adaptation of clinical audit and feedback for learning and behavior change. Implement Sci 2015; Jan 21; 10 (01) 12.
• 59 Sturmberg JP, O’Halloran DM, Martin CM. Understanding health system reform–a complex adaptive systems perspective. J Eval Clin Pract 2012; Feb 1; 18 (01) 202-8 http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2753.2011.01792.x/full
• 60 Pronovost PJ, Mathews SC, Chute CG, Rosen A. Creating a purpose-driven learning and improving health system: The Johns Hopkins Medicine quality and safety experience. Learn Health Systems. 2016 Dec 15. doi: 10.1002/lrh2.10018.
• 61 Toh S, Platt R, Steiner JF, Brown JS. Comparative-Effectiveness Research in Distributed Health Data Networks. Clin Pharmacol Ther 2011; Dec 1; 90 (06) 883-7 doi:10.1038/clpt.2011.236.
• 62 Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, Kohane I. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc 2010; Mar 1; 17 (02) 124-30 http://dx.doi.org/10.1136/jamia.2009.000893
• 63 Apervita Platform. https://apervita.com/#apervita-platform
• 64 Semedy. Company: About Us. http://www.semedy.com/company/about-us
• 65 Boyko JA, Lavis JN, Abelson J, Dobbins M, Carter N. Deliberative dialogues as a mechanism for knowledge translation and exchange in health systems decision-making. Social Sci Med 2012; Dec 31; 75 (11) 1938-45 http://www.sciencedirect.com/science/article/pii/S0277953612005114
• 66 Space.com Staff. May 25, 1961 JFK’s Moon Shot Speech to Congress. Text of Speech by U.S. President John F. Kennedy, Delivered in Person Before a Joint Session of Congress - 1961 May 25. Space.com. New York: Posted 2011 May 25. http://www.space.com/11772-president-kennedy-historic-speech-moon-space.html
• 67 Parry D. Moonshot: The Inside Story of Mankind’s Greatest Adventure. Random House; 2009. Oct 6.
• 68 Kline SJ. Conceptual foundations for multidisciplinary thinking. Stanford University Press; 1995. http://www.sup.org/books/title/?id=2194
• 69 Berwick DM. Era 3 for Medicine and Health Care. JAMA 2016; 315 (13) 1329-30 doi:10.1001/jama.2016.1509.
• 70 Batalden M, Batalden P, Margolis P, Seid M, Armstrong G, Opipari-Arrigan L. et al. Coproduction of healthcare service. BMJ Qual Saf. 2015 Sep 16:bmjqs-2015. doi:10.1136/bmjqs-2015-004315.
• 71 Hock D. One from many: Visa and the rise of chaordic organization. Berrett-Koehler Publishers; 2009
• 72 Northrop L, Feiler P, Gabriel RP, Goodenough J, Linger R, Longstaff T. et al. Ultra-large-scale systems: The software challenge of the future. Carnegie-Mellon University. Pittsburgh, PA: Software Engineering Institute; 2006. Jun.
• 73 McGinnis JM, Powers B, Grossmann C. editors. Digital infrastructure for the learning health system: the foundation for continuous improvement in health and health care: workshop series summary. Institute of Medicine (IOM). National Academies Press; 2011. Sep 21.
• 74 Barretta A. The functioning of co-opetition in the health-care sector: An explorative analysis. Scandinavian Journal of Management 2008; Sep 30; 24 (03) 209-20.
• 75 Peng TJ, Bourne M. The coexistence of competition and cooperation between networks: implications from two Taiwanese healthcare networks. British Journal of Management 2009; Sep 1; 20 (03) 377-400.
• 76 Aggarwal CC, Abdelzaher T. Social sensing. In: Managing and mining sensor data. 2013: 237-297 Springer US; https://link.springer.com/chapter/10.1007/978-1-4614-6309-2_9#page-1
• 77 Gubbi J, Buyya R, Marusic S, Palaniswami M. Internet of Things (IoT): A vision, architectural elements, and future directions. Future Generation Computer Systems 2013; Sep 30; 29 (07) 1645-60 http://dx.doi.org/10.1016/j.future.2013.01.010
• 78 Leroy X. Formal verification of a realistic compiler. Communications of the ACM 2009; Jul 1; 52 (07) 107-15.
• 79 Fisher K. HACMS: high assurance cyber military systems. ACM SIGAda Ada Letters 2012; Dec 2; 32 (03) 51-2 http://dl.acm.org/citation.cfm?id=2402695
• 80 Malecha G, Morrisett G, Shinnar A, Wisnesky R. Toward a verified relational database management system. ACM Sigplan Notices 2010; Jan 17; 45 (01) 237-48 http://dl.acm.org/citation.cfm?id=1706329
• 81 James B. Video: We Count Our Success in Lives. 2014 May 19. https://www.youtube.com/watch?v=vHSjgslJVEk
• 82 Grohol J. Mayo Clinic leaders offer health reform vision, prescriptions: ‘Learning Organization’ is key to patient-centered, affordable health system. PsychCentral. 2016 Apr 30. http://psychcentral.com/news/archives/2006-04/mc-mcl033106.html
• 83 Clinical Data and Patient-Powered Research Networks – Awarded Projects. 2015 PCORI. http://www.pcori.org/research-results/pcornet-national-patient-centered-clinical-research-network/clinical-data-and-0
• 84 NIH Collaboratory. Health Care Systems Research Collaboratory. About Us. https://www.nihcollaboratory.org/about-us/Pages/default.aspx
• 85 Blue Cross Blue Shield Blue Care Network of Michigan. Collaborative Quality Initiatives (CQIs). http://www.bcbsm.com/providers/value-partnerships/collaborative-quality-initiatives.html
• 86 McGlynn EA, Lieu TA, Durham ML, Bauck A, Laws R, Go AS. et al. Developing a data infrastructure for a learning health system: the PORTAL network. J Am Med Inform Assoc 2014; Jul 1; 21 (04) 596-601 http://dx.doi.org/10.1136/amiajnl-2014-002746
• 87 Baldwin CY, Clark KB. Design Rules. Volume 1: The Power of Modularity, Volume 1. https://mitpress.mit.edu/books/design-rules
• 88 Sullivan KJ, Griswold WG, Cai Y, Hallen B. The structure and value of modularity in software design. ACM SIGSOFT Software Engineering Notes 2001; Sep 10; 26 (05) 99-108 https://dl.acm.org/citation.cfm?id=503224
• 89 Grossmann C, Powers B, McGinnis JM. Stewardship and Governance in the Learning Health System. 2011
• 90 Thorp J. Europe’s E-Health Initiatives: An Overview of European Interoperability Initiatives. J AHIMA 2010; Sep;81: 56-8 Available on epSOS EU website at: http://www.epsos.eu/fileadmin/content/images/press_section/Journal_of_AHI-MA_September_2010.pdf
• 91 Cayton H. Information Governance in the National Health Service (UK). Digital Infrastructure for the Learning Health System. 2011 Sep 21: 180.
• 92 Kharrazi H, Lasser EC, Yasnoff WA, Loonsk J, Advani A, Lehmann HP, Chin DC. et al. A proposed national research and development agenda for population health informatics: summary recommendations from a national expert workshop. J Am Med Inform Assoc 2017; Jan 1; 24 (01) 2-12 http://dx.doi.org/10.1093/jamia/ocv210
• 93 Etheredge LM. Rapid learning: a breakthrough agenda. Health Aff 2014; Jul 1; 33 (07) 1155-62.
• 94 Faden RR, Kass NE, Goodman SN, Pronovost PJ, Tunis S, Beauchamp TL. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Center Report 2013; Jan 1; 43 (s1): S16-27.
• 95 Sugarman J, Califf RM. Ethics and regulatory complexities for pragmatic clinical trials. JAMA 2014; Jun 18; 311 (23) 2381-2 doi:10.1001/jama.2014.4164.
• 96 The Farr Institute of Health Informatics Research (UK). About: Who We Are. http://www.farrinstitute.org/about/who-we-are