Download PDF
Methods Inf Med 2021; 60(01/02): 032-048
DOI: 10.1055/s-0041-1731784
Original Article

Why Is the Electronic Health Record So Challenging for Research and Clinical Care?

John H. Holmes
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
James Beinlich
2   Information Technology Entity Services and Corporate Information Services, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Mary R. Boland
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Kathryn H. Bowles
3   Department of Biobehavioral Health Sciences, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, United States
,
Yong Chen
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Tessa S. Cook
4   Department of Radiology, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
George Demiris
3   Department of Biobehavioral Health Sciences, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, United States
,
Michael Draugelis
5   Department of Predictive Health Care, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Laura Fluharty
6   Clinical Research Operations, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Peter E. Gabriel
4   Department of Radiology, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Robert Grundmeier
7   Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
,
C. William Hanson
8   Department of Anesthesiology and Critical Care, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Daniel S. Herman
9   Department of Pathology and Laboratory Medicine, University of Pennsylvania Perelman School of Medicine Philadelphia, Pennsylvania, United States
,
Blanca E. Himes
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Rebecca A. Hubbard
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Charles E. Kahn Jr.
4   Department of Radiology, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Dokyoon Kim
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Ross Koppel
10   Department of Sociology, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Qi Long
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Nebojsa Mirkovic
11   Department of Research Analytics, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Jeffrey S. Morris
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Danielle L. Mowery
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Marylyn D. Ritchie
12   Department of Genetics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Ryan Urbanowicz
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Jason H. Moore
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
› Author Affiliations Funding Research reported in this publication was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under award number: UL1TR001878. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
› Further Information
    Permissions and Reprints

Abstract

Background The electronic health record (EHR) has become increasingly ubiquitous. At the same time, health professionals have been turning to this resource for access to data that is needed for the delivery of health care and for clinical research. There is little doubt that the EHR has made both of these functions easier than earlier days when we relied on paper-based clinical records. Coupled with modern database and data warehouse systems, high-speed networks, and the ability to share clinical data with others are large number of challenges that arguably limit the optimal use of the EHR

Objectives Our goal was to provide an exhaustive reference for those who use the EHR in clinical and research contexts, but also for health information systems professionals as they design, implement, and maintain EHR systems.

Methods This study includes a panel of 24 biomedical informatics researchers, information technology professionals, and clinicians, all of whom have extensive experience in design, implementation, and maintenance of EHR systems, or in using the EHR as clinicians or researchers. All members of the panel are affiliated with Penn Medicine at the University of Pennsylvania and have experience with a variety of different EHR platforms and systems and how they have evolved over time.

Results Each of the authors has shared their knowledge and experience in using the EHR in a suite of 20 short essays, each representing a specific challenge and classified according to a functional hierarchy of interlocking facets such as usability and usefulness, data quality, standards, governance, data integration, clinical care, and clinical research.

Conclusion We provide here a set of perspectives on the challenges posed by the EHR to clinical and research users.

Keywords

electronic health records - user-computer interface - standards - medical informatics - systems integration

Supplementary Material



Publication History

Received: 28 December 2020

Accepted: 22 May 2021

Article published online:
19 July 2021

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Office of the National Coordinator for Health Information Technology. Office-based physician electronic health record adoption. Accessed December 8, 2020 at: dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php
    Google Scholar
  • 2 Office of the National Coordinator for Health Information Technology. Percent of hospitals, by type, that possess certified health IT. Accessed December 8, 2020 at: dashboard.healthit.gov/quickstats/pages/certified-electronic-health-record-technology-in-hospitals.php
    Google Scholar
  • 3 Artis KA, Bordley J, Mohan V, Gold JA. Data omission by physician trainees on ICU rounds. Crit Care Med 2019; 47 (03) 403-409
    CrossrefPubMedGoogle Scholar
  • 4 Shenvi EC, Feupe SF, Yang H, El-Kareh R. “Closing the loop”: a mixed-methods study about resident learning from outcome feedback after patient handoffs. Diagnosis (Berl) 2018; 5 (04) 235-242
    CrossrefPubMedGoogle Scholar
  • 5 Khairat S, Coleman C, Newlin T. et al. A mixed-methods evaluation framework for electronic health records usability studies. J Biomed Inform 2019; 94: 103175
    CrossrefPubMedGoogle Scholar
  • 6 Cohen DJ, Dorr DA, Knierim K. et al. Primary care practices' abilities and challenges in using electronic health record data for quality improvement. Health Aff (Millwood) 2018; 37 (04) 635-643
    CrossrefPubMedGoogle Scholar
  • 7 Bristol AA, Nibbelink CW, Gephart SM, Carrington JM. Nurses' use of positive deviance when encountering electronic health records-related unintended consequences. Nurs Adm Q 2018; 42 (01) E1-E11
    CrossrefPubMedGoogle Scholar
  • 8 Gephart S, Carrington JM, Finley B. A systematic review of nurses' experiences with unintended consequences when using the electronic health record. Nurs Adm Q 2015; 39 (04) 345-356
    CrossrefPubMedGoogle Scholar
  • 9 Friedman A, Crosson JC, Howard J. et al. A typology of electronic health record workarounds in small-to-medium size primary care practices. J Am Med Inform Assoc 2014; 21 (e1): e78-e83
    CrossrefPubMedGoogle Scholar
  • 10 Schiff GD, Zucker L. Medical scribes: salvation for primary care or workaround for poor EMR usability?. J Gen Intern Med 2016; 31 (09) 979-981
    CrossrefPubMedGoogle Scholar
  • 11 Flanagan ME, Saleem JJ, Millitello LG, Russ AL, Doebbeling BN. Paper- and computer-based workarounds to electronic health record use at three benchmark institutions. J Am Med Inform Assoc 2013; 20 (e1): e59-e66
    CrossrefPubMedGoogle Scholar
  • 12 Hysong SJ, Sawhney MK, Wilson L. et al. Provider management strategies of abnormal test result alerts: a cognitive task analysis. J Am Med Inform Assoc 2010; 17 (01) 71-77
    CrossrefPubMedGoogle Scholar
  • 13 Menon S, Murphy DR, Singh H, Meyer AND, Sittig DF. Workarounds and test results follow-up in electronic health record-based primary care. Appl Clin Inform 2016; 7 (02) 543-559
    Article in Thieme ConnectPubMedGoogle Scholar
  • 14 Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57 (05) 805-834
    CrossrefPubMedGoogle Scholar
  • 15 Roman LC, Ancker JS, Johnson SB, Senathirajah Y. Navigation in the electronic health record: a review of the safety and usability literature. J Biomed Inform 2017; 67: 69-79
    CrossrefPubMedGoogle Scholar
  • 16 Reid PP, Compton WD, Grossman JH, Fanjiang Geds. National Academy of Engineering (US) and Institute of Medicine (US) Committee on Engineering and the Health Care System. Building a Better Delivery System: A New Engineering/Health Care Partnership. Washington, DC: National Academies Press (US); 2005
    Google Scholar
  • 17 Health information technology: standards, implementation specifications, and certification criteria for electronic health record technology. 2014 edition; revisions to the permanent certification program for health information technology. Accessed June 17, 2021 at: https://federalregister.gov/a/2012-20982
    Google Scholar
  • 18 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
    CrossrefPubMedGoogle Scholar
  • 19 Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med 2018; 178 (11) 1544-1547
    CrossrefPubMedGoogle Scholar
  • 20 Kearns M, Neel S, Roth A, Wu ZS. An Empirical Study of Rich Subgroup Fairness for Machine Learning. In: Proceedings of the Conference on Fairness, Accountability, and Transparency. FAT* '19. Association for Computing Machinery; 2019 100-109 DOI: 10.1145/3287560.3287592
    CrossrefGoogle Scholar
  • 21 Gijsberts CM, Groenewegen KA, Hoefer IE. et al. Race/ethnic differences in the associations of the framingham risk factors with carotid IMT and cardiovascular events. PLoS One 2015; 10 (07) e0132321
    CrossrefPubMedGoogle Scholar
  • 22 Eneanya ND, Yang W, Reese PP. Reconsidering the consequences of using race to estimate kidney function. JAMA 2019; 322 (02) 113-114
    CrossrefPubMedGoogle Scholar
  • 23 Rajkomar A, Hardt M, Howell MD, Corrado G, Chin MH. Ensuring fairness in machine learning to advance health equity. Ann Intern Med 2018; 169 (12) 866-872
    CrossrefPubMedGoogle Scholar
  • 24 Ehrmeyer SS, Laessig RH. Has compliance with CLIA requirements really improved quality in US clinical laboratories?. Clin Chim Acta 2004; 346 (01) 37-43
    CrossrefPubMedGoogle Scholar
  • 25 Greg Miller W, Myers GL, Lou Gantzer M. et al. Roadmap for harmonization of clinical laboratory measurement procedures. Clin Chem 2011; 57 (08) 1108-1117
    CrossrefPubMedGoogle Scholar
  • 26 Christenson RH, Jacobs E, Uettwiller-Geiger D. et al. Comparison of 13 commercially available cardiac troponin assays in a multicenter North American study. J Appl Lab Med 2017; 1 (05) 544-561
    CrossrefPubMedGoogle Scholar
  • 27 Huff SM, Rocha RA, McDonald CJ. et al. Development of the logical observation identifier names and codes (LOINC) vocabulary. J Am Med Inform Assoc 1998; 5 (03) 276-292
    CrossrefPubMedGoogle Scholar
  • 28 Cornet R, Chute CG. Health concept and knowledge management: twenty-five years of evolution. Yearb Med Inform 2016; 1 (9312666): (Suppl. 01) S32-S41
    PubMedGoogle Scholar
  • 29 Centers for Disease Control and Prevention. International Classification of Diseases, 10th Revision (ICD-10). Accessed June 17, 2021 at: https://www.cdc.gov/nchs/icd/icd10.htm
    Google Scholar
  • 30 Bodenreider O, Cornet R, Vreeman DJ. Recent Developments in Clinical Terminologies - SNOMED CT, LOINC, and RxNorm. Yearb Med Inform 2018; 27 (01) 129-139
    Article in Thieme ConnectPubMedGoogle Scholar
  • 31 Hripcsak G, Duke JD, Shah NH. et al. Observational health data sciences and informatics (OHDSI): opportunities for observational researchers. Stud Health Technol Inform 2015; 216: 574-578
    PubMedGoogle Scholar
  • 32 World Health Organization. International Statistical Classification of Diseases and Related Health Problems (ICD). ICD-11. Accessed June 17, 2021 at: https://www.who.int/classifications/icd/en/
    Google Scholar
  • 33 Smith SW, Koppel R. Healthcare information technology's relativity problems: a typology of how patients' physical reality, clinicians' mental models, and healthcare information technology differ. J Am Med Inform Assoc 2014; 21 (01) 117-131
    CrossrefPubMedGoogle Scholar
  • 34 Woodfield R, Grant I, Sudlow CL. UK Biobank Stroke Outcomes Group, UK Biobank Follow-Up and Outcomes Working Group. Accuracy of electronic health record data for identifying stroke cases in large-scale epidemiological studies: a systematic review from the UK Biobank Stroke Outcomes Group. PLoS One 2015; 10 (10) e0140533
    CrossrefPubMedGoogle Scholar
  • 35 Liao KP, Cai T, Savova GK. et al. Development of phenotype algorithms using electronic medical records and incorporating natural language processing. BMJ 2015; 350: h1885
    CrossrefPubMedGoogle Scholar
  • 36 Schulz S, Jansen L. Formal ontologies in biomedical knowledge representation. Yearb Med Inform 2013; 8 (9312666): 132-146
    PubMedGoogle Scholar
  • 37 Smith B, Ashburner M, Rosse C. et al; OBI Consortium. The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol 2007; 25 (11) 1251-1255
    CrossrefPubMedGoogle Scholar
  • 38 Zhang XA, Yates A, Vasilevsky N. et al. Semantic integration of clinical laboratory tests from electronic health records for deep phenotyping and biomarker discovery. NPJ Digit Med 2019; 2 (32) 32
    CrossrefPubMedGoogle Scholar
  • 39 Bona JP, Prior FW, Zozus MN, Brochhausen M. Enhancing clinical data and clinical research data with biomedical ontologies - insights from the knowledge representation perspective. Yearb Med Inform 2019; 28 (01) 140-151
    Article in Thieme ConnectPubMedGoogle Scholar
  • 40 Brochhausen M, Bona J, Blobel B. The role of axiomatically-rich ontologies in transforming medical data to knowledge. Stud Health Technol Inform 2018; 249: 38-49
    PubMedGoogle Scholar
  • 41 Bowles KH, Potashnik S, Ratcliffe SJ. et al. Conducting research using the electronic health record across multi-hospital systems: semantic harmonization implications for administrators. J Nurs Adm 2013; 43 (06) 355-360
    CrossrefPubMedGoogle Scholar
  • 42 Nordo AH, Levaux HP, Becnel LB. et al. Use of EHRs data for clinical research: Historical progress and current applications. Learn Health Syst 2019; 3 (01) e10076
    CrossrefPubMedGoogle Scholar
  • 43 Garza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform 2016; 64: 333-341
    CrossrefPubMedGoogle Scholar
  • 44 Collins FS, Hudson KL, Briggs JP, Lauer MS. PCORnet: turning a dream into reality. J Am Med Inform Assoc 2014; 21 (04) 576-577
    CrossrefPubMedGoogle Scholar
  • 45 Murphy SN, Weber G, Mendis M. et al. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc 2010; 17 (02) 124-130
    CrossrefPubMedGoogle Scholar
  • 46 Murtagh MJ, Blell MT, Butters OW. et al. Better governance, better access: practising responsible data sharing in the METADAC governance infrastructure. Hum Genomics 2018; 12 (01) 24
    CrossrefPubMedGoogle Scholar
  • 47 Meinert E, Alturkistani A, Brindley D, Knight P, Wells G, de Pennington N. Weighing benefits and risks in aspects of security, privacy and adoption of technology in a value-based healthcare system. BMC Med Inform Decis Mak 2018; 18 (01) 100
    CrossrefPubMedGoogle Scholar
  • 48 Dankar FK, Ptitsyn A, Dankar SK. The development of large-scale de-identified biomedical databases in the age of genomics-principles and challenges. Hum Genomics 2018; 12 (01) 19
    CrossrefPubMedGoogle Scholar
  • 49 Stahl BC, Rainey S, Harris E, Fothergill BT. The role of ethics in data governance of large neuro-ICT projects. J Am Med Inform Assoc 2018; 25 (08) 1099-1107
    CrossrefPubMedGoogle Scholar
  • 50 Powles J, Hodson H. Google deepmind and healthcare in an age of algorithms. Health Technol (Berl) 2017; 7 (04) 351-367
    CrossrefPubMedGoogle Scholar
  • 51 Roland M. Linking physicians' pay to the quality of care--a major experiment in the United Kingdom. N Engl J Med 2004; 351 (14) 1448-1454
    CrossrefPubMedGoogle Scholar
  • 52 Roland M, Guthrie B. Quality and outcomes framework: what have we learnt?. BMJ 2016; 354: i4060
    CrossrefPubMedGoogle Scholar
  • 53 United States District Court Northern District of Illinois Eastern Division. MATT DINERSTEIN, individually and on behalf of all others similarly situated, Plaintiff, v. Google, LLC, a Delaware limited liability company, THE UNIVERSITY OF CHICAGO MEDICAL CENTER, an Illinois not-for-profit corporation, and THE UNIVERSITY OF CHICAGO, an Illinois not-for-profit corporation. Accessed June 17, 2021 at: https://www.courtlistener.com/recap/gov.uscourts.ilnd.366172/gov.uscourts.ilnd.366172.85.0.pdf
    Google Scholar
  • 54 Ohno-Machado L. To share or not to share: that is not the question. Sci Transl Med 2012; 4 (165) 165cm15
    CrossrefPubMedGoogle Scholar
  • 55 Wu Y, Jiang X, Kim J, Ohno-Machado L. Grid Binary LOgistic REgression (GLORE): building shared models without sharing data. J Am Med Inform Assoc 2012; 19 (05) 758-764
    CrossrefPubMedGoogle Scholar
  • 56 Ohno-Machado L, Agha Z, Bell DS. et al; pSCANNER team. pSCANNER: patient-centered scalable national network for effectiveness research. J Am Med Inform Assoc 2014; 21 (04) 621-626
    CrossrefPubMedGoogle Scholar
  • 57 Duan R, Boland MR, Liu Z. et al. Learning from electronic health records across multiple sites: a communication-efficient and privacy-preserving distributed algorithm. J Am Med Inform Assoc 2020; 27 (03) 376-385
    CrossrefPubMedGoogle Scholar
  • 58 Duan R, Luo C, Schuemie MJ. et al. Learning from local to global: an efficient distributed algorithm for modeling time-to-event data. J Am Med Inform Assoc 2020; 27 (07) 1028-1036
    CrossrefPubMedGoogle Scholar
  • 59 Deng Y, Jiang X, Long Q. Privacy-preserving methods for vertically partitioned incomplete data. Annu Symp Am Med Inform Assoc 2021; 2020: 348-357
    PubMedGoogle Scholar
  • 60 Chang C, Deng Y, Jiang X, Long Q. Multiple imputation for analysis of incomplete data in distributed health data networks. Nat Commun 2020; 11 (01) 5467
    CrossrefPubMedGoogle Scholar
  • 61 Health Information Privacy. Guidance regarding methods for de-identification of protected health information in accordance with the Health Insurance Portability and Accountability Act (HIPAA) privacy rule. Accessed August 3, 2020. Accessed August 3, 2020 at: https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html
    Google Scholar
  • 62 Kushida CA, Nichols DA, Jadrnicek R, Miller R, Walsh JK, Griffin K. Strategies for de-identification and anonymization of electronic health record data for use in multicenter research studies. Med Care 2012; 50: S82-S101
    CrossrefPubMedGoogle Scholar
  • 63 Rocher L, Hendrickx JM, de Montjoye Y-A. Estimating the success of re-identifications in incomplete datasets using generative models. Nat Commun 2019; 10 (01) 3069
    CrossrefPubMedGoogle Scholar
  • 64 Meystre SM, Friedlin FJ, South BR, Shen S, Samore MH. Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC Med Res Methodol 2010; 10 (100968545): 70
    CrossrefPubMedGoogle Scholar
  • 65 Hayden EC. Privacy loophole found in genetic databases. Accessed June 17, 2021 at: https://www.nature.com/articles/nature.2013.12237
    Google Scholar
  • 66 Pecoraro F, Luzi D, Ricci FL. Designing ETL tools to feed a data warehouse based on electronic healthcare record infrastructure. Stud Health Technol Inform 2015; 210: 929-933
    PubMedGoogle Scholar
  • 67 Pecoraro F, Luzi D, Ricci FL. Secondary uses of EHR systems: A feasibility study. Published in: E-Health and Bioengineering Conference (EHB). 21–23 November. 2013; Iasi, Romania; 2013: 1-6
    Google Scholar
  • 68 Botsis T, Hartvigsen G, Chen F, Weng C. Secondary use of EHR: data quality issues and informatics opportunities. Summit On Translat Bioinforma 2010; 2010: 1-5
    PubMedGoogle Scholar
  • 69 Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013; 46 (05) 830-836
    CrossrefPubMedGoogle Scholar
  • 70 Kroth PJ, Morioka-Douglas N, Veres S. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2 (08) e199609-e199609
    Google Scholar
  • 71 National Academy of Medicine, Board on Population Health and Public Health Practice, Institute of Medicine. Capturing Social and Behavioral Domains and Measures in Electronic Health Records: Phase 2. Washington, DC: National Academies Press (US); 2015
    Google Scholar
  • 72 Bycroft C, Freeman C, Petkova D. et al. The UK Biobank resource with deep phenotyping and genomic data. Nature 2018; 562 (7726): 203-209
    CrossrefPubMedGoogle Scholar
  • 73 Hamilton CM, Strader LC, Pratt JG. et al. The PhenX Toolkit: get the most from your measures. Am J Epidemiol 2011; 174 (03) 253-260
    CrossrefPubMedGoogle Scholar
  • 74 McCarty CA, Berg R, Rottscheit CM. et al. Validation of PhenX measures in the personalized medicine research project for use in gene/environment studies. BMC Med Genomics 2014; 7: 3
    CrossrefPubMedGoogle Scholar
  • 75 Luzi D, Rocco I, Tamburis O, Corso B, Minicuci N, Pecoraro F. Variability in the assessment of children's primary healthcare in 30 European countries. Int J Qual Health Care 2021; 33 (01) mzab007
    CrossrefPubMedGoogle Scholar
  • 76 Deans KJ, Sabihi S, Forrest CB. Learning health systems. Semin Pediatr Surg 2018; 27 (06) 375-378
    CrossrefPubMedGoogle Scholar
  • 77 Sarafidis M, Tarousi M, Anastasiou A. et al. Data quality challenges in a learning health system. Stud Health Technol Inform 2020; 270: 143-147
    PubMedGoogle Scholar
  • 78 Chen JH, Alagappan M, Goldstein MK, Asch SM, Altman RB. Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets. Int J Med Inform 2017; 102: 71-79
    CrossrefPubMedGoogle Scholar
  • 79 Sculley D, Holt G, Golovin D. et al. Hidden technical debt in machine learning systems. Accessed June 17, 2021 at: https://papers.nips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
    Google Scholar
  • 80 Mathes T, Buehn S, Prengel P, Pieper D. Registry-based randomized controlled trials merged the strength of randomized controlled trails and observational studies and give rise to more pragmatic trials. J Clin Epidemiol 2018; 93: 120-127
    CrossrefPubMedGoogle Scholar
  • 81 Workman TA. Engaging Patients in Information Sharing and Data Collection: The Role of Patient-Powered Registries and Research Networks. Rockville, MD: Agency for Healthcare Research and Quality; 2013
    Google Scholar
  • 82 Wozniak L, Soprovich A, Rees S, Johnson ST, Majumdar SR, Johnson JA. Challenges in identifying patients with Type 2 Diabetes for quality-improvement interventions in primary care settings and the importance of valid disease registires. Can J Diabetes 2015; 39 (Suppl. 03) S77-S82
    CrossrefPubMedGoogle Scholar
  • 83 Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Inform Assoc 2013; 20 (01) 117-121
    CrossrefPubMedGoogle Scholar
  • 84 Xie S, Himes BE. Approaches to link geospatially varying social, economic, and environmental factors with electronic health record data to better understand asthma exacerbations. AMIA Annu Symp Proc 2018; 2018: 1561-1570
    PubMedGoogle Scholar
  • 85 Xie S, Greenblatt R, Levy MZ, Himes BE. Enhancing electronic health record data with geospatial information. AMIA Jt Summits Transl Sci Proc 2017; 2017: 123-132
    PubMedGoogle Scholar
  • 86 Goldstein BA, Bhavsar NA, Phelan M, Pencina MJ. Controlling for informed presence bias due to the number of health encounters in an electronic health record. Am J Epidemiol 2016; 184 (11) 847-855
    CrossrefPubMedGoogle Scholar
  • 87 Rusanov A, Weiskopf NG, Wang S, Weng C. Hidden in plain sight: bias towards sick patients when sampling patients with sufficient electronic health record data for research. BMC Med Inform Decis Mak 2014; 14 (01) 51
    CrossrefPubMedGoogle Scholar
  • 88 Wei W-Q, Teixeira PL, Mo H, Cronin RM, Warner JL, Denny JC. Combining billing codes, clinical notes, and medications from electronic health records provides superior phenotyping performance. J Am Med Inform Assoc 2016; 23 (e1): e20-e27
    CrossrefPubMedGoogle Scholar
  • 89 Hubbard RA, Tong J, Duan R, Chen Y. Reducing bias due to outcome misclassification for epidemiologic studies using EHR-derived probabilistic phenotypes. Epidemiology 2020; 31 (04) 542-550
    CrossrefPubMedGoogle Scholar
  • 90 Shortreed SM, Cook AJ, Coley RY, Bobb JF, Nelson JC. Challenges and opportunities for using big health care data to advance medical science and public health. Am J Epidemiol 2019; 188 (05) 851-861
    CrossrefPubMedGoogle Scholar
  • 91 Agniel D, Kohane IS, Weber GM. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ 2018; 361: k1479
    CrossrefPubMedGoogle Scholar
  • 92 Haneuse S, Daniels M. A general framework for considering selection bias in EHR-based studies: what data are observed and why?. EGEMS (Wash DC) 2016; 4 (01) 1203
    PubMedGoogle Scholar
  • 93 Wu J, Roy J, Stewart WF. Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning approaches. Med Care 2010; 48 (06) S106-S113
    CrossrefPubMedGoogle Scholar
  • 94 Pathak J, Kho AN, Denny JC. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J Am Med Inform Assoc 2013; 20 (e2): e206-e211
    CrossrefPubMedGoogle Scholar
  • 95 Robnik-Šikonja M, Kononenko I. Theoretical and empirical analysis of ReliefF and RReliefF. Mach Learn 2003; 53: 23-69
    CrossrefPubMedGoogle Scholar
  • 96 Madhavan S, Bastarache L, Brown JS. et al. Use of electronic health records to support a public health response to the COVID-19 pandemic in the United States: a perspective from 15 academic medical centers. J Am Med Inform Assoc 2021; 28 (02) 393-401
    CrossrefPubMedGoogle Scholar
  • 97 Haendel MA, Chute CG, Bennett TD. et al. The national COVID cohort collaborative (N3C): rationale, design, infrastructure, and deployment. J Am Med Inform Assoc 2021; 28 (03) 427-443
    CrossrefPubMedGoogle Scholar
  • 98 Brat GA, Weber GM, Gehlenborg N. et al. International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium. NPJ Digit Med 2020; 3: 109
    Google Scholar
  • 99 Burn E, You SC, Sena AG. et al. Deep phenotyping of 34,128 adult patients hospitalised with COVID-19 in an international network study. Nat Commun 2020; 11 (01) 5009
    CrossrefPubMedGoogle Scholar
  • 100 Friedman C, Rubin J, Brown J. et al. Toward a science of learning systems: a research agenda for the high-functioning learning health system. J Am Med Inform Assoc 2015; 22 (01) 43-50
    CrossrefPubMedGoogle Scholar
  • 101 Hutter F, Kotthoff L, Vanschoren J. Automated Machine Learning: Methods, Systems, Challenges. 1st ed.. Switzerland: Springer Nature; 2019
    CrossrefGoogle Scholar