Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs

Abstract

Objective Patient attribution, or the process of attributing patient-level metrics to specific providers, attempts to capture real-life provider–patient interactions (PPI). Attribution holds wide-ranging importance, particularly for outcomes in graduate medical education, but remains a challenge. We developed and validated an algorithm using EHR data to identify pediatric resident PPIs (rPPIs).

Methods We prospectively surveyed residents in three care settings to collect self-reported rPPIs. Participants were surveyed at the end of primary care clinic, emergency department (ED), and inpatient shifts, shown a patient census list, asked to mark the patients with whom they interacted, and encouraged to provide a short rationale behind the marked interaction. We extracted routine EHR data elements, including audit logs, note contribution, order placement, care team assignment, and chart closure, and applied a logistic regression classifier to the data to predict rPPIs in each care setting. We also performed a comment analysis of the resident-reported rationales in the inpatient care setting to explore perceived patient interactions in a complicated workflow.

Results We surveyed 81 residents over 111 shifts and identified 579 patient interactions. Among EHR extracted data, time-in-chart was the best predictor in all three care settings (primary care clinic: odds ratio [OR] = 19.36, 95% confidence interval [CI]: 4.19–278.56; ED: OR = 19.06, 95% CI: 9.53–41.65' inpatient: OR = 2.95, 95% CI: 2.23–3.97). Primary care clinic and ED specific models had c-statistic values > 0.98, while the inpatient-specific model had greater variability (c-statistic = 0.89). Of 366 inpatient rPPIs, residents provided rationales for 90.1%, which were focused on direct involvement in a patient's admission or transfer, or care as the front-line ordering clinician (55.6%).

Conclusion Classification models based on routinely collected EHR data predict resident-defined rPPIs across care settings. While specific to pediatric residents in this study, the approach may be generalizable to other provider populations and scenarios in which accurate patient attribution is desirable.

Keywords

secondary use - patient–provider - data mining - workflow - medical education

Protection of Human and Animal Subjects

This study was reviewed and approved by the Children's Hospital of Philadelphia Institutional Review Board, Pennsylvania, United States.

• References

• 1 Herzke CA, Michtalik HJ, Durkin N. , et al. A method for attributing patient-level metrics to rotating providers in an inpatient setting. J Hosp Med 2018; 13 (07) 470-475
• 2 Schumacher DJ, Martini A, Holmboe E. , et al. Developing resident-sensitive quality measures: engaging stakeholders to inform next steps. Acad Pediatr 2019; 19 (02) 177-185
• 3 Mattana J, Kerpen H, Lee C. , et al. Quantifying internal medicine resident clinical experience using resident-selected primary diagnosis codes. J Hosp Med 2011; 6 (07) 395-400
• 4 Wanderer JP, Charnin J, Driscoll WD, Bailin MT, Baker K. Decision support using anesthesia information management system records and accreditation council for graduate medical education case logs for resident operating room assignments. Anesth Analg 2013; 117 (02) 494-499
• 5 Sebok-Syer SS, Chahine S, Watling CJ, Goldszmidt M, Cristancho S, Lingard L. Considering the interdependence of clinical performance: implications for assessment and entrustment. Med Educ 2018; 52: 970-980
• 6 Mehrotra A, Burstin H, Raphael C. Raising the bar in attribution. Ann Intern Med 2017; 167 (06) 434-435
• 7 McCoy RG, Bunkers KS, Ramar P. , et al. Patient attribution: why the method matters. Am J Manag Care 2018; 24 (12) 596-603
• 8 Fishman E, Barron J, Liu Y. , et al. Using claims data to attribute patients with breast, lung, or colorectal cancer to prescribing oncologists. Pragmat Obs Res 2019; 10: 15-22
• 9 Gupta K, Khanna R. . Getting the (right) doctor, right away. Available at: https://psnet.ahrq.gov/webmm/case/381/getting-the-right-doctor-right-away . Accessed August 29, 2018
• 10 Vawdrey DK, Wilcox LG, Collins S. , et al. Awareness of the care team in electronic health records. Appl Clin Inform 2011; 2 (04) 395-405
• 11 Dalal AK, Schnipper JL. Care team identification in the electronic health record: A critical first step for patient-centered communication. J Hosp Med 2016; 11 (05) 381-385
• 12 Arora VM. Harnessing the power of big data to improve graduate medical education: big idea or bust?. Acad Med 2018; 93 (06) 833-834
• 13 Weinstein DF. Optimizing GME by measuring its outcomes. N Engl J Med 2017; 377 (21) 2007-2009
• 14 Accreditation Council for Graduate Medical Organization. CLER Pathways to Excellence: Expectations for an Optimal Clinical Learning Environment to Achieve Safe and High-Quality Patient Care, Version 2.0. Available at: https://www.acgme.org/Portals/0/PDFs/CLER/1079ACGME-CLER2019PTE-BrochDigital.pdf . Accessed May 18, 2020
• 15 Levin JC, Hron J. Automated reporting of trainee metrics using electronic clinical systems. J Grad Med Educ 2017; 9 (03) 361-365
• 16 Nagler J, Harper MB, Bachur RG. An automated electronic case log: using electronic information systems to assess training in emergency medicine. Acad Emerg Med 2006; 13 (07) 733-739
• 17 Simpao A, Heitz JW, McNulty SE, Chekemian B, Brenn BR, Epstein RH. The design and implementation of an automated system for logging clinical experiences using an anesthesia information management system. Anesth Analg 2011; 112 (02) 422-429
• 18 Smith TI, LoPresti CM. Clinical exposures during internal medicine acting internship: profiling student and team experiences. J Hosp Med 2014; 9 (07) 436-440
• 19 Sebok-Syer SS, Goldszmidt M, Watling CJ, Chahine S, Venance SL, Lingard L. Using electronic health record data to assess residents' clinical performance in the workplace: the good, the bad, and the unthinkable. Acad Med 2019; 94 (06) 853-860
• 20 Cebul RD. Using electronic medical records to measure and improve performance. Trans Am Clin Climatol Assoc 2008; 119: 65-75
• 21 Hirsch AG, Jones JB, Lerch VR. , et al. The electronic health record audit file: the patient is waiting. J Am Med Inform Assoc 2017; 24 (e1): e28-e34
• 22 Hribar MR, Read-Brown S, Goldstein IH. , et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
• 23 Schumacher DJ, Wu DTY, Meganathan K. , et al. A feasibility study to attribute patients to primary interns on inpatient ward teams using electronic health record data. Acad Med 2019; 94 (09) 1376-1383
• 24 Dziorny AC, Orenstein EW, Lindell RB, Hames NA, Washington N, Desai B. Automatic detection of front-line clinician hospital shifts: a novel use of electronic health record timestamp data. Appl Clin Inform 2019; 10 (01) 28-37
• 25 Hothorn T, Hornik K, Zeileis A. Unbiased recursive partitioning: a conditional inference framework. J Comput Graph Stat 2006; 15: 651-674
• 26 Hothorn T, Bühlmann P, Dudoit S, Molinaro A, van der Laan MJ. Survival ensembles. Biostatistics 2006; 7 (03) 355-373
• 27 Strobl C, Boulesteix A-L, Zeileis A, Hothorn T. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics 2007; 8: 25
• 28 Strobl C, Boulesteix A-L, Kneib T, Augustin T, Zeileis A. Conditional variable importance for random forests. BMC Bioinformatics 2008; 9: 307
• 29 R Core Team. R: a language and environment for statistical computing. Available at: https://www.gbif.org/tool/81287/r-a-language-and-environment-for-statistical-computing . Accessed May 18, 2020
• 30 Greiner M. Two-graph receiver operating characteristic (TG-ROC): update version supports optimisation of cut-off values that minimise overall misclassification costs. J Immunol Methods 1996; 191 (01) 93-94
• 31 Mai M, Orenstein EW. . Network analysis of EHR interactions to identify 360° evaluators. Available at: https://dblp.org/rec/conf/amia/MaiO18.html . Accessed May 18, 2020
• 32 Tripp JS, Narus SP, Magill MK, Huff SM. Evaluating the accuracy of existing EMR data as predictors of follow-up providers. J Am Med Inform Assoc 2008; 15 (06) 787-790
• 33 Atlas SJ, Chang Y, Lasko TA, Chueh HC, Grant RW, Barry MJ. Is this “my” patient? Development and validation of a predictive model to link patients to primary care providers. J Gen Intern Med 2006; 21 (09) 973-978
• 34 Sequist TD, Singh S, Pereira AG, Rusinak D, Pearson SD. Use of an electronic medical record to profile the continuity clinic experiences of primary care residents. Acad Med 2005; 80 (04) 390-394
• 35 Wang JK, Hom J, Balasubramanian S. , et al. An evaluation of clinical order patterns machine-learned from clinician cohorts stratified by patient mortality outcomes. J Biomed Inform 2018; 86: 109-119
• 36 Singh A, Rhee KE, Brennan JJ, Kuelbs C, El-Kareh R, Fisher ES. Who's my doctor? Using an electronic tool to improve team member identification on an inpatient pediatrics team. Hosp Pediatr 2016; 6 (03) 157-165