Download PDF
Appl Clin Inform 2020; 11(03): 442-451
DOI: 10.1055/s-0040-1713133
Research Article
Georg Thieme Verlag KG Stuttgart · New York

Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs

Mark V. Mai
1   Department of Anesthesia and Critical Care Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
2   Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
,
Evan W. Orenstein
3   Department of Pediatrics, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
,
John D. Manning
4   Department of Emergency Medicine, Atrium Health's Carolinas Medical Center, Charlotte, North Carolina, United States
,
Anthony A. Luberti
2   Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
5   Department of Pediatrics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
,
Adam C. Dziorny
1   Department of Anesthesia and Critical Care Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
2   Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
› Author Affiliations Funding This work was supported by a Special Project Award from the Association of Pediatric Program Directors.
Further Information

Publication History

14 January 2020

05 May 2020

Publication Date:
24 June 2020 (online)

Also available at
   Permissions and Reprints

Abstract

Objective Patient attribution, or the process of attributing patient-level metrics to specific providers, attempts to capture real-life provider–patient interactions (PPI). Attribution holds wide-ranging importance, particularly for outcomes in graduate medical education, but remains a challenge. We developed and validated an algorithm using EHR data to identify pediatric resident PPIs (rPPIs).

Methods We prospectively surveyed residents in three care settings to collect self-reported rPPIs. Participants were surveyed at the end of primary care clinic, emergency department (ED), and inpatient shifts, shown a patient census list, asked to mark the patients with whom they interacted, and encouraged to provide a short rationale behind the marked interaction. We extracted routine EHR data elements, including audit logs, note contribution, order placement, care team assignment, and chart closure, and applied a logistic regression classifier to the data to predict rPPIs in each care setting. We also performed a comment analysis of the resident-reported rationales in the inpatient care setting to explore perceived patient interactions in a complicated workflow.

Results We surveyed 81 residents over 111 shifts and identified 579 patient interactions. Among EHR extracted data, time-in-chart was the best predictor in all three care settings (primary care clinic: odds ratio [OR] = 19.36, 95% confidence interval [CI]: 4.19–278.56; ED: OR = 19.06, 95% CI: 9.53–41.65' inpatient: OR = 2.95, 95% CI: 2.23–3.97). Primary care clinic and ED specific models had c-statistic values > 0.98, while the inpatient-specific model had greater variability (c-statistic = 0.89). Of 366 inpatient rPPIs, residents provided rationales for 90.1%, which were focused on direct involvement in a patient's admission or transfer, or care as the front-line ordering clinician (55.6%).

Conclusion Classification models based on routinely collected EHR data predict resident-defined rPPIs across care settings. While specific to pediatric residents in this study, the approach may be generalizable to other provider populations and scenarios in which accurate patient attribution is desirable.

Keywords

secondary use - patient–provider - data mining - workflow - medical education

Protection of Human and Animal Subjects

This study was reviewed and approved by the Children's Hospital of Philadelphia Institutional Review Board, Pennsylvania, United States.


 

  • References

  • 1 Herzke CA, Michtalik HJ, Durkin N. , et al. A method for attributing patient-level metrics to rotating providers in an inpatient setting. J Hosp Med 2018; 13 (07) 470-475
    CrossrefPubMedSearch in Google Scholar
  • 2 Schumacher DJ, Martini A, Holmboe E. , et al. Developing resident-sensitive quality measures: engaging stakeholders to inform next steps. Acad Pediatr 2019; 19 (02) 177-185
    CrossrefPubMedSearch in Google Scholar
  • 3 Mattana J, Kerpen H, Lee C. , et al. Quantifying internal medicine resident clinical experience using resident-selected primary diagnosis codes. J Hosp Med 2011; 6 (07) 395-400
    CrossrefPubMedSearch in Google Scholar
  • 4 Wanderer JP, Charnin J, Driscoll WD, Bailin MT, Baker K. Decision support using anesthesia information management system records and accreditation council for graduate medical education case logs for resident operating room assignments. Anesth Analg 2013; 117 (02) 494-499
    CrossrefPubMedSearch in Google Scholar
  • 5 Sebok-Syer SS, Chahine S, Watling CJ, Goldszmidt M, Cristancho S, Lingard L. Considering the interdependence of clinical performance: implications for assessment and entrustment. Med Educ 2018; 52: 970-980
    CrossrefPubMedSearch in Google Scholar
  • 6 Mehrotra A, Burstin H, Raphael C. Raising the bar in attribution. Ann Intern Med 2017; 167 (06) 434-435
    CrossrefPubMedSearch in Google Scholar
  • 7 McCoy RG, Bunkers KS, Ramar P. , et al. Patient attribution: why the method matters. Am J Manag Care 2018; 24 (12) 596-603
    PubMedSearch in Google Scholar
  • 8 Fishman E, Barron J, Liu Y. , et al. Using claims data to attribute patients with breast, lung, or colorectal cancer to prescribing oncologists. Pragmat Obs Res 2019; 10: 15-22
    PubMedSearch in Google Scholar
  • 9 Gupta K, Khanna R. . Getting the (right) doctor, right away. Available at: https://psnet.ahrq.gov/webmm/case/381/getting-the-right-doctor-right-away . Accessed August 29, 2018
    PubMed
  • 10 Vawdrey DK, Wilcox LG, Collins S. , et al. Awareness of the care team in electronic health records. Appl Clin Inform 2011; 2 (04) 395-405
    Thieme ConnectPubMedSearch in Google Scholar
  • 11 Dalal AK, Schnipper JL. Care team identification in the electronic health record: A critical first step for patient-centered communication. J Hosp Med 2016; 11 (05) 381-385
    CrossrefPubMedSearch in Google Scholar
  • 12 Arora VM. Harnessing the power of big data to improve graduate medical education: big idea or bust?. Acad Med 2018; 93 (06) 833-834
    CrossrefPubMedSearch in Google Scholar
  • 13 Weinstein DF. Optimizing GME by measuring its outcomes. N Engl J Med 2017; 377 (21) 2007-2009
    CrossrefPubMedSearch in Google Scholar
  • 14 Accreditation Council for Graduate Medical Organization. CLER Pathways to Excellence: Expectations for an Optimal Clinical Learning Environment to Achieve Safe and High-Quality Patient Care, Version 2.0. Available at: https://www.acgme.org/Portals/0/PDFs/CLER/1079ACGME-CLER2019PTE-BrochDigital.pdf . Accessed May 18, 2020
    PubMed
  • 15 Levin JC, Hron J. Automated reporting of trainee metrics using electronic clinical systems. J Grad Med Educ 2017; 9 (03) 361-365
    CrossrefPubMedSearch in Google Scholar
  • 16 Nagler J, Harper MB, Bachur RG. An automated electronic case log: using electronic information systems to assess training in emergency medicine. Acad Emerg Med 2006; 13 (07) 733-739
    CrossrefPubMedSearch in Google Scholar
  • 17 Simpao A, Heitz JW, McNulty SE, Chekemian B, Brenn BR, Epstein RH. The design and implementation of an automated system for logging clinical experiences using an anesthesia information management system. Anesth Analg 2011; 112 (02) 422-429
    CrossrefPubMedSearch in Google Scholar
  • 18 Smith TI, LoPresti CM. Clinical exposures during internal medicine acting internship: profiling student and team experiences. J Hosp Med 2014; 9 (07) 436-440
    CrossrefPubMedSearch in Google Scholar
  • 19 Sebok-Syer SS, Goldszmidt M, Watling CJ, Chahine S, Venance SL, Lingard L. Using electronic health record data to assess residents' clinical performance in the workplace: the good, the bad, and the unthinkable. Acad Med 2019; 94 (06) 853-860
    CrossrefPubMedSearch in Google Scholar
  • 20 Cebul RD. Using electronic medical records to measure and improve performance. Trans Am Clin Climatol Assoc 2008; 119: 65-75
    PubMedSearch in Google Scholar
  • 21 Hirsch AG, Jones JB, Lerch VR. , et al. The electronic health record audit file: the patient is waiting. J Am Med Inform Assoc 2017; 24 (e1): e28-e34
    CrossrefPubMedSearch in Google Scholar
  • 22 Hribar MR, Read-Brown S, Goldstein IH. , et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
    CrossrefPubMedSearch in Google Scholar
  • 23 Schumacher DJ, Wu DTY, Meganathan K. , et al. A feasibility study to attribute patients to primary interns on inpatient ward teams using electronic health record data. Acad Med 2019; 94 (09) 1376-1383
    CrossrefPubMedSearch in Google Scholar
  • 24 Dziorny AC, Orenstein EW, Lindell RB, Hames NA, Washington N, Desai B. Automatic detection of front-line clinician hospital shifts: a novel use of electronic health record timestamp data. Appl Clin Inform 2019; 10 (01) 28-37
    Thieme ConnectPubMedSearch in Google Scholar
  • 25 Hothorn T, Hornik K, Zeileis A. Unbiased recursive partitioning: a conditional inference framework. J Comput Graph Stat 2006; 15: 651-674
    CrossrefPubMedSearch in Google Scholar
  • 26 Hothorn T, Bühlmann P, Dudoit S, Molinaro A, van der Laan MJ. Survival ensembles. Biostatistics 2006; 7 (03) 355-373
    PubMedSearch in Google Scholar
  • 27 Strobl C, Boulesteix A-L, Zeileis A, Hothorn T. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics 2007; 8: 25
    CrossrefPubMedSearch in Google Scholar
  • 28 Strobl C, Boulesteix A-L, Kneib T, Augustin T, Zeileis A. Conditional variable importance for random forests. BMC Bioinformatics 2008; 9: 307
    CrossrefPubMedSearch in Google Scholar
  • 29 R Core Team. R: a language and environment for statistical computing. Available at: https://www.gbif.org/tool/81287/r-a-language-and-environment-for-statistical-computing . Accessed May 18, 2020
    PubMed
  • 30 Greiner M. Two-graph receiver operating characteristic (TG-ROC): update version supports optimisation of cut-off values that minimise overall misclassification costs. J Immunol Methods 1996; 191 (01) 93-94
    CrossrefPubMedSearch in Google Scholar
  • 31 Mai M, Orenstein EW. . Network analysis of EHR interactions to identify 360° evaluators. Available at: https://dblp.org/rec/conf/amia/MaiO18.html . Accessed May 18, 2020
    PubMed
  • 32 Tripp JS, Narus SP, Magill MK, Huff SM. Evaluating the accuracy of existing EMR data as predictors of follow-up providers. J Am Med Inform Assoc 2008; 15 (06) 787-790
    CrossrefPubMedSearch in Google Scholar
  • 33 Atlas SJ, Chang Y, Lasko TA, Chueh HC, Grant RW, Barry MJ. Is this “my” patient? Development and validation of a predictive model to link patients to primary care providers. J Gen Intern Med 2006; 21 (09) 973-978
    CrossrefPubMedSearch in Google Scholar
  • 34 Sequist TD, Singh S, Pereira AG, Rusinak D, Pearson SD. Use of an electronic medical record to profile the continuity clinic experiences of primary care residents. Acad Med 2005; 80 (04) 390-394
    CrossrefPubMedSearch in Google Scholar
  • 35 Wang JK, Hom J, Balasubramanian S. , et al. An evaluation of clinical order patterns machine-learned from clinician cohorts stratified by patient mortality outcomes. J Biomed Inform 2018; 86: 109-119
    CrossrefPubMedSearch in Google Scholar
  • 36 Singh A, Rhee KE, Brennan JJ, Kuelbs C, El-Kareh R, Fisher ES. Who's my doctor? Using an electronic tool to improve team member identification on an inpatient pediatrics team. Hosp Pediatr 2016; 6 (03) 157-165
    PubMedSearch in Google Scholar