MIRACUM: Medical Informatics in Research and Care in University Medicine

Hans-Ulrich Prokosch; Till Acker; Johannes Bernarding; Harald Binder; Martin Boeker; Melanie Boerries; Philipp Daumke; Thomas Ganslandt; Jürgen Hesser; Gunther Höning; Michael Neumaier; Kurt Marquardt; Harald Renz; Hermann-Josef Rothkötter; Carmen Schade-Brittinger; Paul Schmücker; Jürgen Schüttler; Martin Sedlmayr; Hubert Serve; Keywan Sohrabi; Holger Storf

doi:10.3414/ME17-02-0025

Methods of Information in Medicine, Table of Contents

CC BY-NC-ND 4.0 · Methods Inf Med 2018; 57(S 01): e82-e91
DOI: 10.3414/ME17-02-0025

Focus Theme – Original Articles

Schattauer GmbH

MIRACUM: Medical Informatics in Research and Care in University Medicine

A Large Data Sharing Network to Enhance Translational Research and Medical Care

Hans-Ulrich Prokosch

¹Chair of Medical Informatics, Department of Medical Informatics, Biometrics and Epidemiology, Friedrich-Alexander-University Erlangen-Nürnberg, Erlangen, Germany

,

Till Acker

²Institute of Neuropathology, Justus-Liebig-University Giessen, Giessen, Germany

,

Johannes Bernarding

³Chair of Medical Informatics, Institute for Biometry and Medical Informatics, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany

,

Harald Binder

⁴Institute of Medical Biostatistics, Epidemiology and Informatics, University Medical Center of the Johannes Gutenberg University Mainz, Mainz, Germany

⁵Institute of Medical Biometry and Statistics, Medical Faculty and Medical Center – University of Freiburg, Freiburg, Germany

,

Martin Boeker

⁵Institute of Medical Biometry and Statistics, Medical Faculty and Medical Center – University of Freiburg, Freiburg, Germany

,

Melanie Boerries

⁶Institute of Molecular Medicine and Cell Research and Comprehensive Cancer Center Freiburg (CCCF), University Medical Center, Faculty of Medicine, University of Freiburg; German Cancer Research Center (DKFZ), Heidelberg and German Cancer Consortium (DKTK) partner site Freiburg, Freiburg, Germany

,

Philipp Daumke

⁷Averbis GmbH, Freiburg, Germany

,

Thomas Ganslandt

¹Chair of Medical Informatics, Department of Medical Informatics, Biometrics and Epidemiology, Friedrich-Alexander-University Erlangen-Nürnberg, Erlangen, Germany

⁸Department of Biomedical Informatics, University Medicine Mannheim, Ruprecht-Karls-University Heidelberg, Mannheim, Germany

,

Jürgen Hesser

⁹Experimental Radiation Oncology Department, University Medical Center Mannheim, Central Institute for Scientific Computing (IWR), Central Institute for Computer Engineering (ZITI), Heidelberg University, Mannheim, Germany

,

Gunther Höning

¹⁰Department of Information Technology, University Medical Center of the Johannes Gutenberg University Mainz, Mainz, Germany

,

Michael Neumaier

¹¹Chair for Clinical Chemistry, Medical Faculty Mannheim of Heidelberg University, Mannheim, Germany

,

Kurt Marquardt

¹²University Hospital of Giessen and Marburg, Giessen, Germany

,

Harald Renz

¹³Chair for Clinical Chemistry, Philipps University Marburg, Medical Director of the University Clinic Marburg, Marburg, Germany

,

Hermann-Josef Rothkötter

¹⁴Institute of Anatomy, Otto-von-Guericke-University Magdeburg, Dean of the Medical Faculty, Magdeburg, Germany

,

Carmen Schade-Brittinger

¹⁵Chair of the Coordinating Centre for Clinical Trials, Philipps University Marburg, Marburg, Germany

,

Paul Schmücker

¹⁶University of Applied Sciences Mannheim, Institute for Medical Informatics, Mannheim, Germany

,

Jürgen Schüttler

¹⁷Department of Anesthesiology, University of Erlangen-Nürnberg, Dean of the Medical Faculty, Erlangen, Germany

,

Martin Sedlmayr

¹Chair of Medical Informatics, Department of Medical Informatics, Biometrics and Epidemiology, Friedrich-Alexander-University Erlangen-Nürnberg, Erlangen, Germany

¹⁸Institute of Medical Informatics and Biometrics, Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany

,

Hubert Serve

¹⁹Department of Hematology and Oncology, University Hospital Frankfurt, Goethe University, Frankfurt am Main, Germany

,

Keywan Sohrabi

²⁰Faculty of Health Sciences, University of Applied Sciences – THM, Giessen, Germany

,

Holger Storf

²¹Medical Informatics Group, University Hospital Frankfurt, Goethe University, Frankfurt am Main, Germany

› Author Affiliations

Abstract

Summary

Introduction: This article is part of the Focus Theme of Methods of Information in Medicine on the German Medical Informatics Initiative. Similar to other large international data sharing networks (e.g. OHDSI, PCORnet, eMerge, RD-Connect) MIRACUM is a consortium of academic and hospital partners as well as one industrial partner in eight German cities which have joined forces to create interoperable data integration centres (DIC) and make data within those DIC available for innovative new IT solutions in patient care and medical research.

Objectives: Sharing data shall be supported by common interoperable tools and services, in order to leverage the power of such data for biomedical discovery and moving towards a learning health system. This paper aims at illustrating the major building blocks and concepts which MIRACUM will apply to achieve this goal.

Governance and Policies: Besides establishing an efficient governance structure within the MIRACUM consortium (based on the steering board, a central administrative office, the general MIRACUM assembly, six working groups and the international scientific advisory board), defining DIC governance rules and data sharing policies, as well as establishing (at each MIRACUM DIC site, but also for MIRACUM in total) use and access committees are major building blocks for the success of such an endeavor.

Architectural Framework and Methodology: The MIRACUM DIC architecture builds on a comprehensive ecosystem of reusable open source tools (MIRACOLIX), which are linkable and interoperable amongst each other, but also with the existing software environment of the MIRACUM hospitals. Efficient data protection measures, considering patient consent, data harmonization and a MIRACUM metadata repository as well as a common data model are major pillars of this framework. The methodological approach for shared data usage relies on a federated querying and analysis concept.

Use Cases: MIRACUM aims at proving the value of their DIC with three use cases: IT support for patient recruitment into clinical trials, the development and routine care implementation of a clinico-molecular predictive knowledge tool, and molecular-guided therapy recommendations in molecular tumor boards.

Results: Based on the MIRACUM DIC release in the nine months conceptual phase first large scale analysis for stroke and colorectal cancer cohorts have been pursued.

Discussion: Beyond all technological challenges successfully applying the MIRACUM tools for the enrichment of our knowledge about diagnostic and therapeutic concepts, thus supporting the concept of a Learning Health System will be crucial for the acceptance and sustainability in the medical community and the MIRACUM university hospitals.

Keywords

Data reuse - data sharing - Medical Informatics Initiative - MIRACUM - data integration centres

Full Text

References

References
1 Meystre SM, Lovis C, Bürkle T, Tognola G, Budrionis A, Lehmann CU. Clinical Data Reuse or Secondary Use: Current Status and Potential Future Progress. Yearb Med Inform 2017; 26 (01) 38-52.
2 Hripcsak G, Ryan PB, Duke JD, Shah NH, Park RW, Huser V, Suchard MA, Schuemie MJ, DeFalco FJ, Perotte A, Banda JM, Reich CG, Schilling LM, Matheny ME, Meeker D, Pratt N, Madigan D. Characterizing treatment pathways at scale using the OHDSI network. Proc Natl Acad Sci U S A 2016; 113 (27) 7329-7336.
3 Hripcsak G, Duke JD, Shah NH, Reich CG, Huser V, Schuemie MJ, Suchard MA, Park RW, Wong IC, Rijnbeek PR, van der Lei J, Pratt N, Norén GN, Li YC, Stang PE, Madigan D, Ryan PB. Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers. Stud Health Technol Inform 2015; 216: 574-578.
4 Collins FS, Hudson KL, Briggs JP, Lauer MS. PCORnet: turning a dream into reality. J Am Med Inform Assoc 2014; 21 (04) 576-577.
5 Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc 2014; 21 (04) 578-582.
6 Gottesman O, Kuivaniemi H, Tromp G, Faucett WA, Li R, Manolio TA, Sanderson SC, Kannry J, Zinberg R, Basford MA, Brilliant M, Carey DJ, Chisholm RL, Chute CG, Connolly JJ, Crosslin D, Denny JC, Gallego CJ, Haines JL, Hakonarson H, Harley J, Jarvik GP, Kohane I, Kullo IJ, Larson EB, McCarty C, Ritchie MD, Roden DM, Smith ME, Böttinger EP, Williams MS. beMERGE Network. The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future. Genet Med 2013; 15 (10) 761-771.
7 Gainotti S, Torreri P, Wang CM, Reihs R, Mueller H, Heslop E, Roos M, Badowska DM, de Paulis F, Kodra Y, Carta C, Martìn EL, Miller VR, Filocamo M, Mora M, Thompson M, Rubinstein Y, Posada de la Paz M, Monaco L, Lochmüller H, Taruscio D. The RD-Connect Registry & Biobank Finder: a tool for sharing aggregated data and metadata among rare disease researchers. Eur J Hum Genet. 2018 Forthcoming. doi:10.1038/s41431–017–0085-z.
8 Biobanking and Biomolecular Resources Infrastructure –European Research Infrastructure Consortium. [cited 2018 Mar 28]. Available from: http://www.bbmri-eric.eu/
9 ELIXIR: A distributed infrastructure for lifescience information. [cited 2018 Mar 28]. Available from: https://www.elixir-europe.org/
10 Eijssen L, Evelo C, Kok R, Mons B, Hooft R. other founding members of DTL Data (see Acknowledgements). The Dutch Techcentre for Life Sciences: Enabling data-intensive life science research in the Netherlands. Version 2. F1000Res 2015; 04: 33.
11 Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, Blomberg N, Boiten JW, da Silva Santos LB, Bourne PE, Bouwman J, Brookes AJ, Clark T, Crosas M, Dillo I, Dumon O, Edmunds S, Evelo CT, Finkers R, Gonzalez-Beltran A, Gray AJ, Groth P, Goble C, Grethe JS, Heringa J, tHoen PA, Hooft R, Kuhn T, Kok R, Kok J, Lusher SJ, Martone ME, Mons A, Packer AL, Persson B, Rocca-Serra P, Roos M, van Schaik R, Sansone SA, Schultes E, Sengstag T, Slater T, Strawn G, Swertz MA, Thompson M, van der Lei J, van Mulligen E, Velterop J, Waagmeester A, Wittenburg P, Wolstencroft K, Zhao J, Mons B. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 2016; 03: 160018.
12 Semler SC, Wissing F, Heyder R. German Medical Informatics Initiative: A national approach to integrating health data from patient care and medical research. Methods Inf Med 2018; 57 (Open): e50-e56.
13 MIRACUM (Medical Informatics for Research and Care in University Medicine). [cited 2018 Mar 28]. Available from: http://www.miracum.org/
14 Pommerening K, Drepper J, Helbing K, Ganslandt T. Leitfaden zum Datenschutz in medizinischen Forschungsprojekten. TMF Schriftenreihe. Berlin: Medizinische Wissenschaftliche Verlagsgesellschaft; 2014
15 Christoph J, Griebel L, Leb I, Engel I, Köpcke F, Toddenroth D, Prokosch HU, Laufer J, Marquardt K, Sedlmayr M. Secure Secondary Use of Clinical Data with Cloud-based NLP Services. Towards a Highly Scalable Research Infrastructure. Methods Inf Med 2015; 54 (03) 276-282.
16 Bialke M, Bahls T, Havemann C, Piegsa J, Weitmann K, Wegner T, Hoffmann W. MOSAIC – A Modular Approach to Data Management in Epidemiological Studies. Methods Inf Med 2015; 54 (04) 364-371.
17 Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, Kohane I. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc 2010; 17 (02) 124-130.
18 Kohane IS, Churchill SE, Murphy SN. A translational engine at the national scale: informatics for integrating biology and the bedside. J Am Med Inform Assoc 2012; 19 (02) 181-185.
19 Scheufele E, Aronzon D, Coopersmith R, McDuffie MT, Kapoor M, Uhrich CA, Avitabile JE, Liu J, Housman D, Palchuk MB. tranSMART: An Open Source Knowledge Management and High Content Data Analytics Platform. AMIA Jt Summits Transl Sci Proc 2014; 2014: 96-101.
20 Christoph J, Knell C, Naschberger E, Stürzl M, Maier C, Prokosch HU, Sedlmayr M. Two years of tranSMART in a university hospital for translational research and education. Stud Health Technol Inform 2017; 236: 70-79.
21 Christoph J, Knell C, Bosserhoff A, Naschberger E, Stürzl M, Rübner M, Seuss H, Ruh M, Prokosch HU, Sedlmayr B. Usability and Suitability of the Omics-Integrating Analysis Platform tranSMART for Translational Research and Education. ACI 2017; 08 (04) 1173-1183.
22 Marcus DS, Archie KA, Olsen TR, Ramaratnam M. The open-source neuroimaging research enterprise. J Digit Imaging 2007; 20 (Suppl. 01) 130-138.
23 Herrick R, Horton W, Olsen T, McKay M, Archie KA, Marcus DS. XNAT Central: Open sourcing imaging research data. NeuroImage 2016; 124 Part B: 1093-1096.
24 He S, Yong M, Matthews PM, Guo Y. tranSMARTXNAT Connector tranSMART-XNAT connector – image selection based on clinical phenotypes and genetic profiles. Bioinformatics. 2016: btw714.
25 Doiron D, Burton P, Marcon Y, Gaye A, Wolffenbuttel BH, Perola M, Stolk RP, Foco L, Minelli C, Waldenberger M, Holle R, Kvaløy K, Hillege HL, Tassé AM, Ferretti V, Fortier I. Data harmonization and federated analysis of population-based studies: the BioSHaRE project. Emerg Themes Epidemiol 2013; 10 (01) 12.
26 Fortier I, Burton PR, Robson PJ, Ferretti V, Little J, L’Heureux F, Deschenes M, Knoppers BM, Doiron D, Keers JC, Linksted P, Harris JR, Lachance G, Boileau C, Pedersen NL, Hamilton CM, Hveem K, Borugian MJ, Gallagher RP, McLaughlin J, Parker L, Potter JD, Gallacher J, Kaaks R, Liu B, Sprosen T, Vilain A, Atkinson SA, Rengifo A, Morton R, Metspalu A, Wichmann HE, Tremblay M, Chisholm RL, Garcia-Montero A, Hillege H, Litton JE, Palmer LJ, Perola M, Wolffenbuttel BH, Peltonen L, Hudson TJ. Quality, quantity and harmony: the DataSHaPER approach to integrating data across bioclinical studies. Int J Epidemiol 2010; 39 (05) 1383-1393.
27 Howe D, Costanzo M, Fey P, Gojobori T, Hannick L, Hide W, Hill DP, Kania R, Schaeffer M, St Pierre S. Big data: the future of biocuration. Nature 2008; 455 (7209): 47-50.
28 Spjuth O, Krestyaninova M, Hastings J, Shen HY, Heikkinen J, Waldenberger M, Langhammer A, Ladenvall C, Esko T, Persson MÅ, Heggland J, Dietrich J, Ose S, Gieger C, Ried JS, Peters A, Fortier I, de Geus EJ, Klovins J, Zaharenko L, Willemsen G, Hottenga JJ, Litton JE, Karvanen J, Boomsma DI, Groop L, Rung J, Palmgren J, Pedersen NL, McCarthy MI, van Duijn CM, Hveem K, Metspalu A, Ripatti S, Prokopenko I, Harris JR. Harmonising and linking biomedical and clinical data across disparate data archives to enable integrative cross-biobank research. Eur J Hum Genet 2016; 24 (04) 521-528.
29 Kadioglu D, Weingardt P, Ückert F, Wagner T. Samply.MDR – Ein Open-Source-Metadaten-Repository. HEC 2016: Health – Exploring Complexity. Joint Conference of GMDS, DGEpi, IEA-EEF, EFMI; München: 28.08.-02.09.2016. Düsseldorf: German Medical Science GMS Publishing House; 2016. DocAbstr. 425.
30 Lablans M, Kadioglu D, Muscholl M, Ückert F. Exploiting Distributed, Heterogeneous and Sensitive Data Stocks while Maintaining the Owner’s Data Sovereignty. Methods Inf Med 2015; 54 (04) 346-352.
31 Storf H, Schaaf J, Kadioglu D, Göbel J, Wagner TOF, Ückert F. [Registries for rare diseases: OSSE – An open-source framework for technical implementation]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2017 Mar 13. doi: 10.1007/s00103–017–2536–7.
32 Garza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform 2016; 64: 333-341.
33 MIRACUM Metadata Repository. M-MDR. [cited 2018 Mar 28]. Available from: http://mdr.miracum.de/
34 Mate S, Kadioglu D, Majeed RW, Stöhr MR, Folz M, Vormstein P, Storf H, Brucker DP, Keune D, Zerbe N, Hummel M, Senghas K, Prokosch HU, Lablans M. Proof-of-Concept Integration of Heterogeneous Biobank IT Infrastructures into a Hybrid Biobanking Network. Stud Health Technol Inform 2017; 243: 100-104.
35 Mate S, Vormstein P, Kadioglu D, Majeed RW, Lablans M, Prokosch HU, Storf H. On-The-Fly Query Translation Between i2b2 and Samply in the German Biobank Node (GBN) Prototypes. Stud Health Technol Inform 2017; 243: 42-46.
36 Gaye A, Marcon Y, Isaeva J, LaFlamme P, Turner A, Jones EM, Minion J, Boyd AW, Newby CJ, Nuotio ML, Wilson R, Butters O, Murtagh B, Demir I, Doiron D, Giepmans L, Wallace SE, Budin-Ljøsne I, Oliver Schmidt C, Boffetta P, Boniol M, Bota M, Carter KW, deKlerk N, Dibben C, Francis RW, Hiekkalinna T, Hveem K, Kvaløy K, Millar S, Perry IJ, Peters A, Phillips CM, Popham F, Raab G, Reischl E, Sheehan N, Waldenberger M, Perola M, van den Heuvel E, Macleod J, Knoppers BM, Stolk RP, Fortier I, Harris JR, Woffenbuttel BH, Murtagh MJ, Ferretti V, Burton PR. DataSHIELD: taking the analysis to the data, not the data to the analysis. Int J Epidemiol 2014; 43 (06) 1929-1944.
37 Cuggia M, Besana P, Glasspool D. Comparing semi-automatic systems for recruitment of patients to clinical trials. Int J Med Inform 2011; 80 (06) 371-388.
38 Dilts DM, Sandler AB. Invisible barriers to clinical trials: the impact of structural, infrastructural, and procedural barriers to opening oncology clinical trials. J Clin Oncol 2006; 24: 4545-4552.
39 Campbell MK, Snowdon C, Francis D. et al. Recruitment to randomised trials: strategies for trial enrollment and participation study. The STEPS study. Health Technol Assess 2007; 11 iii, ix-105.
40 Trinczek B, Köpcke F, Leusch T, Majeed RW, Schreiweis B, Wenk J, Bergh B, Ohmann C, Röhrig R, Prokosch HU, Dugas M. Design and multicentric implementation of a generic software architecture for patient recruitment systems re-using existing HIS tools and routine patient data. Appl Clin Inform 2014; 05 (01) 264-283.
41 Schreiweis B, Trinczek B, Köpcke F, Leusch T, Majeed RW, Wenk J, Bergh B, Ohmann C, Röhrig R, Dugas M, Prokosch HU. Comparison of electronic health record system functionalities to support the patient recruitment process in clinical trials. Int J Med Inform 2014; 83 (11) 860-868.
42 Köpcke F, Trinczek B, Majeed RW, Schreiweis B, Wenk J, Leusch T, Ganslandt T, Ohmann C, Bergh B, Röhrig R, Dugas M, Prokosch HU. Evaluation of data completeness in the electronic health record for the purpose of patient recruitment into clinical trials: a retrospective analysis of element presence. BMC Med Inform Decis Mak 2013; 13: 37.
43 De Moor G, Sundgren M, Kalra D, Schmidt A, Dugas M, Claerhout B, Karakoyun T, Ohmann C, Lastic PY, Ammour N, Kush R, Dupont D, Cuggia M, Daniel C, Thienpont G, Coorevits P. Using electronic health records for clinical research: the case of the EHR4CR project. J Biomed Inform 2015; 53: 162-173.
44 Doods J, Lafitte C, Ulliac-Sagnes N, Proeve J, Botteri F, Walls R, Sykes A, Dugas M, Fritz F. A European inventory of data elements for patient recruitment. Stud Health Technol Inform 2015; 210: 506-510.
45 Khalilia M, Choi M, Henderson A, Iyengar S, Braunstein M, Sun J. Clinical Predictive Modeling Development and Deployment through FHIR Web Services. AMIA Annu Symp Proc 2015; 2015: 717-726.
46 Hinderer M, Boeker M, Wagner SA, Binder H, Ückert F, Hülsemann JL, Neumaier M, Schade-Brittinger C, Acker T, Prokosch HU, Sedlmayr B. The experience of physicians in pharmacogenomic clinical decision support within eight German University Hospitals. Pharmacogenomics 2017; 18 (08) 773-785.
47 Hinderer M, Börries M, Haller F, Wagner S, Sollfrank S, Acker T, Prokosch HU, Christoph J. Supporting Molecular Tumor Boards in Molecularguided Decision-making – the Current Status of Five German University Hospitals. Stud Health Technol Inform 2017; 236: 48-54.
48 Hinderer M, Boeker M, Wagner SA, Lablans M, Newe S, Hülsemann JL, Neumaier M, Binder H, Renz H, Acker T, Prokosch HU, Sedlmayr M. Integrating clinical decision support systems for pharmacogenomic testing into clinical routine – a scoping review of designs of user-system interactions in recent system development. BMC Med Inform Decis Mak 2017; 17 (01) 81.
49 Klann JG, Abend A, Raghavan VA, Mandl KD, Murphy SN. Data interchange using i2b2. J Am Med Inform Assoc 2016; 23 (05) 909-915.
50 Haverkamp C, Gansland T, Horki P, Boeker M, Dörfler A, Schwab S, Berkefeld J, Pfeilschifter W, Niesen W-D, Egger K, Kaps M, Brockmann MA, Neumaier-Probst E, Szabo K, Skalej M, Bien S, Best C, Prokosch U, Urbach H. Regional differences in thrombectomy rates: secondary use of billing codes in the MIRACUM (Medical Informatics for Research and Care in University Medicine) Consortium. Clin Neuroradiol 2018; 28 (02) 225-234.
51 Maier C, Lang L, Storf H, Vormstein P, Bieber R, Bernarding J, Herrmann T, Haverkamp C, Horki P, Laufer J, Berger F, Höning G, Fritsch HW, Schüttler J, Ganslandt T, Prokosch HU, Sedlmayr M. Towards implementation of OMOP in a German university hospital consortium. Appl Clin Inform 2018; 09 (01) 54-61.
52 Available from: http://www.medizininformatikinitiative.de/de/konsortien/hd4cr-konzeptphase [cited 2018 Mar 22].
53 Prasser F, Kohlmayer F, Lautenschläger R, Kuhn KA. ARX – A Comprehensive Tool for Anonymizing Biomedical Data. AMIA Annu Symp Proc 2014; 2014: 984-993.
54 Prasser F, Kohlmayer F, Kuhn KA. Efficient and effective pruning strategies for health data de-identification. BMC Medical Informatics and Decision Making 2016; 16: 49.
55 Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, Gunter C, Musen M, Platt R, Stead W, Sullivan K, Van Houweling D. Toward a science of learning systems: a research agenda for the highfunctioning Learning Health System. J Am Med Inform Assoc 2015; 22 (01) 43-50.