Refining Clinical Phenotypes to Improve Clinical Decision Support and Reduce Alert Fatigue: A Feasibility Study

Abstract

Background Chronic kidney disease (CKD) is common and associated with adverse clinical outcomes. Most care for early CKD is provided in primary care, including hypertension (HTN) management. Computerized clinical decision support (CDS) can improve the quality of care for CKD but can also cause alert fatigue for primary care physicians (PCPs). Computable phenotypes (CPs) are algorithms to identify disease populations using, for example, specific laboratory data criteria.

Objectives Our objective was to determine the feasibility of implementation of CDS alerts by developing CPs and estimating potential alert burden.

Methods We utilized clinical guidelines to develop a set of five CPs for patients with stage 3 to 4 CKD, uncontrolled HTN, and indications for initiation or titration of guideline-recommended antihypertensive agents. We then conducted an iterative data analytic process consisting of database queries, data validation, and subject matter expert discussion, to make iterative changes to the CPs. We estimated the potential alert burden to make final decisions about the scope of the CDS alerts. Specifically, the number of times that each alert could fire was limited to once per patient.

Results In our primary care network, there were 239,339 encounters for 105,992 primary care patients between April 1, 2018 and April 1, 2019. Of these patients, 9,081 (8.6%) had stage 3 and 4 CKD. Almost half of the CKD patients, 4,191 patients, also had uncontrolled HTN. The majority of CKD patients were female, elderly, white, and English-speaking. We estimated that 5,369 alerts would fire if alerts were triggered multiple times per patient, with a mean number of alerts shown to each PCP ranging from 0.07–to 0.17 alerts per week.

Conclusion Development of CPs and estimation of alert burden allows researchers to iteratively fine-tune CDS prior to implementation. This method of assessment can help organizations balance the tradeoff between standardization of care and alert fatigue.

Keywords

clinical decision support - chronic kidney disease - alert fatigue - electronic health record - data quality

Background and Significance

Electronic health records (EHRs), particularly those with computerized clinical decision support (CDS) that delivers evidence-based recommendations at the point of care, have been shown to improve the quality of care and clinical outcomes.[1] [2] [3] [4] [5] [6] [7] However, CDS can also lead to “alert fatigue,” also known as “pop-up fatigue,” limiting effectiveness.[8] [9] [10] [11]

In complex, data-intensive health care environments like primary care, alert fatigue can be particularly pronounced.[12] [13] [14] In addition, many believe EHRs are a major contributor to physician burnout.[13] [15] [16] [17] In addition to the impact on physician well-being, CDS can have unintended consequences on patient safety if alerts serve as a distraction from clinically important information.[18]

Multiple methods to reduce alert fatigue have been described in the Clinical Informatics literature. Healthcare systems have recognized the importance of a robust governance process to prioritize and limit CDS alerts.[19] Most commonly multidisciplinary committees consisting of physicians, pharmacists, and informaticians evaluate content through a consensus method.[20] Other methods include using clinician and end-user feedback to modify CDS, reporting, and visual dashboards.[21] [22] [23] [24] [25] [26] Many health care systems use real-world EHR data to determine which alerts fire most frequently.[27] [28] However, there are fewer methods to avoid alert fatigue before alerts are implemented. One health care system implemented drug-dose alerts in the silent mode prior to exposing clinicians to alerts and decreased drug-dose alerts from 12 to 3% of all medication orders.[29]

Though preventing alert burden prior to implementation is a major advance, preventing the development of alerts that could lead to alert burden would be more efficient. Alert burden can be estimated through the use of computable phenotypes (CPs), which are disease definitions or algorithms that allow the curation of disease populations using EHR data.[30] [31] [32] CPs are increasingly used as preliminary data to determine the feasibility of clinical trial enrollment.[33] [34] [35] In addition, health services researchers have used CPs to answer questions about the health care delivery system, such as, are there enough outpatient nephrologists to consult on patients with chronic kidney disease (CKD) in the primary care setting?[36] [37] In addition, best practices for designing and specifying CDS alerts include querying retrospective data to identify individual patients for whom a CDS alert would fire and then estimating the firing rates for the CDS alert prior to implementation.[38] These tasks constitute a feasibility study that can help to determine the technical and operational feasibility of CDS prior to the actual programming or build of the CDS. In addition, validation of data elements during a feasibility study can address potential errors in completeness, correctness, and timeliness (or “currency”) of the data driving the CDS.[38] [39]

We sought to assess the feasibility of deploying a set of CDS alerts for hypertension (HTN) management in CKD patients in the primary care setting prior to implementation for a planned clinical trial. We estimated the potential alert burden and used this information to define the scope of the CDS prior to implementation.

Methods

Clinical Domain

CKD is both prevalent and costly and may lead to end-stage renal disease and premature cardiovascular disease.[40] [41] [42] [43] [44] [45] [46] The majority of care for early CKD occurs in primary care settings. Although both CKD and uncontrolled blood pressure are not difficult to diagnose, both often go unrecognized and are suboptimally managed by primary care physicians (PCPs).[47] [48] [49] CDS could improve optimal management of CKD in primary care through tailored recommendations: blood pressure control is imperative in CKD; optimal management of elevated BP should include treatment with renin angiotensin-aldosterone system inhibitors, including angiotensin-converting enzyme inhibitor (ACEi) or angiotensin receptor blocker (ARB) agents.[50]

Patient Population

The population was limited to Brigham and Women's Primary Care Network of 15 primary care practices. The network of primary care practices includes practices within the academic medical center, community-based practices in the city and suburbs, and two urban community health centers. The population served is diverse in terms of socioeconomic status, education level, race, ethnicity, and languages spoken. We limited the population to include adult patients with a primary care encounter between April, 1, 2018, and April 1, 2019, with an attending physician, nurse practitioner, or physician assistant. The first primary care encounter was considered the index encounter. This study was reviewed and approved by the Mass General Brigham Institutional Review Board.

The definition of CKD was at least two estimated glomerular filtration rate (eGFR; calculated using CKD-EPI without race adjustment) less than 60 mL/min/1.73 m² or two urine-to-albumin creatinine ratio (UACR) greater than 30 mg/g at least 90 days apart in the 2 years preceding the index encounter.[51] The definition of uncontrolled HTN was at least two systolic blood pressure (SBP) values over 140 mm Hg measured in an ambulatory setting.

Development of Computable Phenotypes

We utilized clinical guidelines to map out the key decision points for clinicians and to define a set of potential actions.[48] This set of decision points and potential actions was reviewed and refined by a group of subject matter experts who also considered local practice patterns. For example, the National Quality Forum HTN measure uses 140 mm Hg as the threshold for controlled SBP. Even though some recent guidelines recommend a lower threshold, the group of subject matter experts decided to continue to use a threshold of 140 mm Hg with the plan of lowering the threshold in the future.

Pseudocode was developed ([Supplementary Tables S1]–[S5], available in the online version), and then translated to Structured Query Language.[52] The primary data source was the Mass General Brigham Enterprise Data Warehouse, which contains data extracted from Epic's Clarity Database.

The final set of decision points resulted in five CPs ([Fig. 1]). The first CP (1A) includes patients with CKD and uncontrolled SBP who do not have an ACEi on their medication list. The second CP (1B) includes patients with CKD and uncontrolled SBP who do not have an ARB on their medication list. The third CP (2A) includes patients who are currently on an ACEi but not at an optimal dose, while the fourth CP (2B) includes those who are on a suboptimal dose of an ARB. The fifth CP (3A) includes patients who are maximized on an ACEi or ARB but are not on a diuretic. We then conducted an iterative data analytic process consisting of database queries, data validation, and subject matter expert discussion, to make final decisions which led to a set of five CDS alerts ([Fig. 2]), along with alternate versions conditional on whether a patient is a female of childbearing age (teratogenicity warning) and whether a patient has a history of angioedema from ACEi (statement that ARB is not contraindicated).

Fig. 1 Flowchart to map out key decision points for clinicians.

Fig. 2 (A–E) Five CDS alerts resulting from five computable phenotypes and associated recommendations.

Iterative Refinement of Computable Phenotypes

For each CP, all required data elements to categorize patients were extracted, including encounter vital signs, laboratory values, allergies, and medications. To assess data quality and validate the data elements for the CDS, we randomly selected 10 patients at each decision point and performed a detailed chart review. The following data elements were reviewed: encounter type, encounter department, visit type, provider type, patient age, patient race, patient sex, blood pressure values, serum creatinine values, potassium values, medications, medication start and end dates, and allergies.

Chart review revealed data quality problems with completeness, correctness, and currency (or timeliness). We discovered problems with the completeness of the list of codes for encounter departments, which was supposed to include all primary care clinics. We were able to identify the issue by opening the clinic schedules of each network primary care clinic and reviewing the scheduled patients to determine whether each patient was captured in a data query designed to retrieve all encounters in primary care clinics. When we determined that there were entire clinics that were not captured, we discovered that two of the clinics were not included in the set of encounter departments named Brigham and Women's Hospital Primary Care and we were able to add the additional encounter departments for those clinics to our query. We also checked 10 charts from the list of encounters retrieved through this query to ensure that the encounter was a primary care visit. Another way that this process improved data quality was when we determined that the list of practicing PCPs obtained from the administrative office was not complete, correct, or current. To improve the data quality of the list of PCPs, we communicated directly with individual physicians who did not have encounters captured in the encounters data query to determine whether their clinic was missing from that query or whether they had left the practice. The process was particularly productive when we began to run queries related to laboratory results. For example, when identifying primary care patients fitting the CKD diagnosis criteria of two elevated UACR, chart review revealed laboratory results that were not retrieved by the data query for the laboratory test with the common name “MALB/CRE RATIO, RANDOM URINE.” By examining the individual laboratory results within patient charts we identified several other common names for the same test associated with thousands of individual laboratory results: “MICROALB/CREAT(480),” “MALB/CRE RATIO, URINE,” “MICROALBUMIN/CREATININE RATIO,” and “URINE MICROALBUMIN/CREAT RATIO EXTERNAL.” The data query was modified to include the additional laboratory tests and chart review was conducted. In effect, the process iteratively improved the likelihood that the CPs would accurately capture patients of interest. As previously described, we conducted a human-centered design process employing multiple methods for gathering user requirements and feedback on design and usability.[53] Factors such as informativeness, actionability, and information overload were also considered and discussed by subject matter experts. We discovered multiple situations where local practice norms, clinical workflow, or existing quality metrics were a consideration. We discussed clinical workflow issues such as inaccurate SBP readings at the beginning of the primary care visit. Many encounters included multiple SBP readings. In the usual visit workflow, the initial SBP is measured by a medical assistant and is often falsely elevated. If the index encounter included multiple SBP values, the SBP value with the latest timestamp was used. We also incorporated CDS to address uncommon situations: (1) females of childbearing age; (2) previous allergy reaction of angioedema to ACEi; and (3) abnormal potassium level. These considerations contributed qualitative data to the iterative refinement of CPs ([Supplementary Table S6], available in the online version).

The development of CPs 2A and 2B was more complicated than the development of 1A, 1B, or 3A. One consideration was whether these alerts should recommend titration of each medication to the maximum dose approved by the Food and Drug Administration. For example, the maximum dose of lisinopril is 80 mg once per day, but this dose carries a high risk of adverse events and is not commonly used in practice. We created a query to determine the most commonly prescribed dose ([Supplementary Table S7], available in the online version) and only recommended titration for doses below the most commonly prescribed dose. Another constraint was that we could not develop alerts with multiple conditional recommendations in Epic, meaning that we needed to create one alert for each dose of medication (i.e., one alert that could address each possible starting dose of lisinopril with an appropriate titration). Instead, each starting dose required a separate alert (e.g., one alert for a starting dose of 10 mg recommending titration to 20 mg and another alert for a starting dose of 20 mg recommending titration to 40 mg). We were able to determine that the majority of patients who were prescribed an ACEi were prescribed lisinopril and the majority of patients who were prescribed an ARB were on losartan. Based on local prescribing practices, alerts 2A and 2B recommended lisinopril for an ACE inhibitor and losartan for an ARB.

Baseline Metrics: Population Characteristics and Local Practice Patterns

We examined the baseline demographic and clinical characteristics of the patient population. We also examined baseline local practice patterns of HTN treatment. In particular, we were interested to know how aggressive PCPs were in HTN management at baseline. We examined the mean number of medications per patient, mean number of antihypertensive (anti-HTN) classes per patient, and proportion of patients prescribed 1, 2, 3, or >4 anti-HTN medications.

Estimation of Alert Burden

As the final, and most informative step in terms of determining feasibility, we estimated the alert burden that would result from CDS alerts targeting the five CPs. We estimated how many alerts would fire in a scenario where alerts were programmed to fire in each encounter where a patient met the criteria for one of the CPs. Then, we plotted the firing pattern of each alert to assess variability in firing rates week to week over the course of 1 year. We were able to identify which providers would see each alert and calculated the mean number of alerts per PCP per week.

Results

Population Characteristics and Local Practice Patterns

There were 105,992 primary care patients seen in 239,339 encounters with 281 PCPs between April 1, 2018, and April 1, 2019. The stage 3 and 4 CKD population consisted of 9,081 patients (8.6% of the primary care population). Encounters for patients with stage 3 to 4 CKD accounted for 28,242 (12%) of all primary care encounters. Among CKD patients, 4,191 had uncontrolled HTN. The majority of patients were female, elderly (mean age 77), about two-thirds were white and 83% were English speakers ([Table 1]).

Table 1 Demographic characteristics of patient population with stage 3 to 4 CKD and uncontrolled hypertension

	Patients (N = 4,191)
Gender male (%)	1,560 (37.2)
Age (mean [SD])	77.31 (11.79)
Race (%)
American Indian or Alaska Native	7 (0.2)
Asian	81 (1.9)
Black or African American	610 (14.6)
Hispanic or Latino	226 (5.4)
Other	379 (9.0)
Unknown or declined	195 (4.7)
White	2,693 (64.3)
Language (%)
English	3,477 (83.0)
Spanish	552 (13.2)
Other	162 (3.9)

Abbreviations: CKD, chronic kidney disease; SD, standard deviation.

Local practice patterns revealed that, on average, these patients were prescribed agents from two anti-HTN medication classes and 18% of these patients were prescribed agents from four or more anti-HTN medication classes ([Table 2]).

Table 2 Local practice patterns of HTN management for patient population with stage 3 and 4 CKD and uncontrolled HTN

	Patients
N	4,191
Medications (mean [SD])	10.06 (5.44)
Anti-HTN medication classes (N [SD])	2.3 (1.36)
Zero anti-HTN medications (N [%])	345 (8.23)
One anti-HTN medication (N [%])	879 (20.97)
Two anti-HTN medications (N [%])	1,215 (28.99)
Three anti-HTN medications (N (%))	1,007 (24.03)
Four or more anti-HTN medications (N [%])	745 (17.78)

Abbreviations: CKD, chronic kidney disease; HTN, hypertension.

Alert Burden: Volume of Alerts and Firing Rates

First, we estimated the volume of alerts if the alerts were programmed to fire at each encounter where a patient met the criteria for one of the five CPs. If alerts were to fire multiple times per patient over the course of 1 year, 5,369 alerts would fire overall. The mean number of alerts shown to each PCP would range from 0.07 to 0.17 alerts per week on average ([Table 3]). After reviewing these results, we decided to include a “lockout period” so that the alerts would only fire once per patient and would be suppressed for all future encounters. The logic of the BestPractice Advisory module allowed us to suppress the alert in subsequent encounters even if the patient continues to meet the criteria in subsequent encounters. If alerts were to fire once per patient over the course of 1 year, 2,524 alerts would fire overall ([Figs. 3] [4] [5]). Of note, the decision to limit the development of alerts 2A and 2B to just lisinopril and losartan, as opposed to all ACEi and ARB medications, resulted in the exclusion of 490 patients or 19% ([Fig. 4]).

Fig. 3 Estimated volume of alerts targeting patients who are not prescribed ACEi or ARB (CPs 1A and 1B).

Fig. 4 Estimated volume of alerts targeting patients prescribed a suboptimal dose of ACEi or ARB (CPs 2A and 2B).

Fig. 5 Estimated volume of alerts targeting CP 3A.

Table 3 Alert firing rate for each CP and mean alert rate per PCP per week if alerts were to be programmed to fire multiple times per patient

CP	Encounters triggering alert over 1 year	PCPs shown alerts	Mean alerts/PCP/week
1A	2,074	234	0.170
1B	587	155	0.073
2A	1,315	221	0.114
2B	801	189	0.082
3A	592	172	0.066

Abbreviations: CP, chronic kidney disease; PCP, primary care physicians.

In addition to the overall alert burden, we observed a large amount of week-to-week variability in alert firing rates, as much as fourfold for ACEi alerts (1A and 2A) and the hydrochlorothiazide alert (3A) but only a twofold variation for ARB alerts (1B and 2B; [Fig. 6]).

Fig. 6 Weekly alert firing rate by CP type.

Discussion

We conducted a study to determine the feasibility of a CDS intervention targeting early CKD populations in primary care and found that the potential alert burden was lower than expected. After estimating the alert burden, we decided to implement alerts targeting five CPs and only allowing the alerts to fire once per patient. We believed that at this level the alert burden would be acceptable.

Sophisticated CPs have been developed for CKD populations.[30] [32] To our knowledge, this is the first study to use the CPs for the design of CDS for CKD prior to implementation. One study used a similar method to refine drug–drug interaction alerts.[49] [54]

A secondary benefit of this feasibility study was that we achieved a nuanced understanding of local practice patterns and the complex population which the CDS would impact. Specifically, the baseline anti-HTN management of this elderly group with complex multimorbidity was more aggressive than expected, though still not in compliance with current guidelines.[55] One important aspect of anti-HTN management that we did not explore is patient tolerance of medication (adverse reactions that are not significant enough to be documented in the EHR) and patient preference. The high variability in firing rates from week to week was also unexpected. Increased frequency of alerts could contribute to alert fatigue and decrease compliance suggesting that interventions to decrease PCP alert burden should react to changes in the alert burden on a daily or continuous basis. The titration of alert burden should also take into account physician interaction with alerts, which can be assessed through audit logs.[56]

Limitations

This study has several limitations. First, this is a feasibility study which does not measure the alert burden in practice or the reception by PCPs. Second, the CDS targets SBP >140 mm Hg, which is not in accordance with current guidelines, but this decision was made in accordance with local expert opinion. It would be trivial to lower the SBP threshold in future implementations by changing just one rule within the alert logic. On the contrary, CDS governance policies often require in-depth clinical content review for similar changes, which would require significant time and effort. A similar limitation is that, due to limited resources for build effort, we limited CP 2A to lisinopril and 2B to losartan which excluded a group of patients. Another limitation is that current best practice favors chlorthalidone over hydrochlorothiazide as the preferred thiazide diuretic due to CKD-specific benefits and some guidelines recommend a calcium channel blocker.[57] [58] Secular trends in practice patterns or patient population could alter the rates of alert firing. Organizations should monitor alert firing over time. The sensitivity of the CPs was not measured as compared with gold standard. However, chart review of positive cases was used to iteratively improve the positive predictive value of the CPs by identifying incorrect data and missing data. The focus was on positive predictive value rather than sensitivity because our main concern was alert fatigue. Lastly, this study was limited to one content area within one hospital and the external validity of the CPs was not assessed.[59] Future studies utilizing these CPs should include an assessment of external validity at another institution.

Conclusion

Given the contribution of alert burden to physician burnout, there is a need for approaches such as the development of CPs and estimation of alert burden prior to implementation. These approaches will allow research investigators and vendors to iteratively fine-tune CDS during development. This method of assessment can help organizations to balance the tradeoff between standardization of care and alert fatigue.

Clinical Relevance Statement

This study addresses the overload of information that primary care physicians experience as a result of numerous electronic reminders to provide high-quality, evidence-based care. The study serves as a proof of concept that computable phenotypes and estimation of alert burden prior to implementation can reduce information overload.

Multiple-Choice Questions

• Q1. Computable phenotypes allow designers of clinical decision support to:

  • Identify patients meeting inclusion criteria for a chronic disease cohort.

  • Identify subgroups of chronic disease patients with a gap in evidence-based prescribing.

  • Prospectively determine the impact of an intervention to increase evidence-based prescribing. On clinical outcomes

  • Both a and b.

  Correct Answer: The correct answer is option d. Computable phenotypes rely on structured EHR data including laboratory results and vital signs to identify groups of patients with a given chronic disease diagnosis and can go further to identify patients within those cohorts who have not been prescribed evidence-based treatments. C is incorrect because computable phenotype studies are typically retrospective since characteristics of patients in a prospective clinical trial could change, excluding the patient from the computable phenotype (e.g., the patient's blood pressure improves).

• Q2. A clinical decision support designer could use a computable phenotype along with retrospective data about a healthcare organization's population to:

  • Estimate the total future alert burden for a clinical decision support alert per year for that entire healthcare organization.

  • Estimate the number of patients for whom the alert would fire over a period of 1 year.

  • Estimate the number of times that an individual physician would view the alert over a period of 1 year.

  • All of the above.

  Correct Answer: The correct answer is option d. By examining retrospective data, computable phenotypes can identify specific patients who would trigger a clinical decision support alert and these estimates can be aggregated to the physician or organization-level.

Conflict of Interest

None declared.

Acknowledgments

The authors would like to thank Ryan E. Jones, BA, and Emily J. Benjamin, BA, for assistance with chart review, Zoe Burns, MPH, for project management, Pamela Garabedian, MS, for expertise in user-centered design, Heather Baer, ScD, for expertise in epidemiology, and Sushrut S. Waikar, MD, MPH, for expertise in current nephrology practice.

Protection of Human and Animal Subjects

The Human Subjects Institutional Review Board at Brigham and Women's Hospital approved this study (IRB protocol no: 2018P000692).

Clinicaltrials.gov Trial Registration

This study is registered with Clinicaltrials.gov (identifier: NCT03679247).

• References

• 1 Bates DW, Teich JM, Lee J. et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6 (04) 313-321
• 2 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346
• 3 Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996; 3 (06) 399-409
• 4 Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999; 6 (02) 104-114
• 5 Murphy EV. Clinical decision support: effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J Biol Med 2014; 87 (02) 187-197
• 6 Beeler PE, Bates DW, Hug BL. Clinical decision support systems. Swiss Med Wkly 2014; 144: w14073
• 7 Samal L, Linder JA, Lipsitz SR, Hicks LS. Electronic health records, clinical decision support, and blood pressure control. Am J Manag Care 2011; 17 (09) 626-632
• 8 Embi PJ, Leonard AC. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Inform Assoc 2012; 19 (e1): e145-e148
• 9 Isaac T, Weissman JS, Davis RB. et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009; 169 (03) 305-311
• 10 Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce ‘alert fatigue’ while still minimizing the risk of litigation. Health Aff (Millwood) 2011; 30 (12) 2310-2317
• 11 Phansalkar S, van der Sijs H, Tucker AD. et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20 (03) 489-493
• 12 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 13 Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8 (03) 686-697
• 14 Perna G. Clinical alerts that cried wolf. As clinical alerts pose physician workflow problems, healthcare IT leaders look for answers. Healthc Inform 2012; 29 (04) 18-20 , 20
• 15 Kroth PJ, Morioka-Douglas N, Veres S. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2 (08) e199609
• 16 Linzer M, Smith CD, Hingle S. et al. Evaluation of work satisfaction, stress, and burnout among US internal medicine physicians and trainees. JAMA Netw Open 2020; 3 (10) e2018758
• 17 Melnick ER, Harry E, Sinsky CA. et al. Perceived electronic health record usability as a predictor of task load and burnout among US physicians: mediation analysis. J Med Internet Res 2020; 22 (12) e23382
• 18 Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record-based settings. JAMA Intern Med 2013; 173 (08) 702-704
• 19 McGreevey III JD, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing alert burden in electronic health records: state of the art recommendations from four health systems. Appl Clin Inform 2020; 11 (01) 1-12
• 20 Van Dort BA, Zheng WY, Sundar V, Baysari MT. Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals. J Am Med Inform Assoc 2021; 28 (01) 177-183
• 21 Bhakta SB, Colavecchia AC, Haines L, Varkey D, Garey KW. A systematic approach to optimize electronic health record medication alerts in a health system. Am J Health Syst Pharm 2019; 76 (08) 530-536
• 22 Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020; 11 (01) 46-58
• 23 Helmons PJ, Suijkerbuijk BO, Nannan Panday PV, Kosterink JG. Drug-drug interaction checking assisted by clinical decision support: a return on investment analysis. J Am Med Inform Assoc 2015; 22 (04) 764-772
• 24 Liberati EG, Ruggiero F, Galuppo L. et al. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci 2017; 12 (01) 113
• 25 Simpao AF, Ahumada LM, Desai BR. et al. Optimization of drug-drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard. J Am Med Inform Assoc 2015; 22 (02) 361-369
• 26 Zenziper Y, Kurnik D, Markovits N. et al. Implementation of a clinical decision support system for computerized drug prescription entries in a large tertiary care hospital. Isr Med Assoc J 2014; 16 (05) 289-294
• 27 Kawamanto K, Flynn MC, Kukhareva P. et al. A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: a case study. AMIA Annu Symp Proc 2018; 2018: 624-633
• 28 Shah T, Patel-Teague S, Kroupa L, Meyer AND, Singh H. Impact of a national QI programme on reducing electronic health record notifications to clinicians. BMJ Qual Saf 2019; 28 (01) 10-14
• 29 Saiyed SM, Greco PJ, Fernandes G, Kaelber DC. Optimizing drug-dose alerts using commercial software throughout an integrated health care system. J Am Med Inform Assoc 2017; 24 (06) 1149-1154
• 30 Frigaard M, Rubinsky A, Lowell L. et al. Validating laboratory defined chronic kidney disease in the electronic health record for patients in primary care. BMC Nephrol 2019; 20 (01) 3
• 31 Kennell Jr TI, Willig JH, Cimino JJ. Clinical informatics researcher's desiderata for the data content of the next generation electronic health record. Appl Clin Inform 2017; 8 (04) 1159-1172
• 32 Nadkarni GN, Gottesman O, Linneman JG. et al. Development and validation of an electronic phenotyping algorithm for chronic kidney disease. AMIA Annu Symp Proc 2014; 2014: 907-916
• 33 Ahmad FS, Ricket IM, Hammill BG. et al. Computable phenotype implementation for a national, multicenter pragmatic clinical trial: lessons learned from ADAPTABLE. Circ Cardiovasc Qual Outcomes 2020; 13 (06) e006292
• 34 Chapman M, Domínguez J, Fairweather E, Delaney BC, Curcin V. Using computable phenotypes in point-of-care clinical trial recruitment. Stud Health Technol Inform 2021; 281: 560-564
• 35 Kilgallon JL, Gannon M, Burns Z. et al. Multicomponent intervention to improve blood pressure management in chronic kidney disease: a protocol for a pragmatic clinical trial. BMJ Open 2021; 11 (12) e054065
• 36 Singh K, Waikar SS, Samal L. Evaluating the feasibility of the KDIGO CKD referral recommendations. BMC Nephrol 2017; 18 (01) 223
• 37 Duggal V, Montez-Rath ME, Thomas IC, Goldstein MK, Tamura MK. Nephrology referral based on laboratory values, kidney failure risk, or both: a study using veterans affairs health system data. Am J Kidney Dis 2021
• 38 Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc 2013; 20 (01) 144-151
• 39 Joseph A, Mullett C, Lilly C. et al. Coronary artery disease phenotype detection in an academic hospital system setting. Appl Clin Inform 2021; 12 (01) 10-16
• 40 Coresh J, Selvin E, Stevens LA. et al. Prevalence of chronic kidney disease in the United States. JAMA 2007; 298 (17) 2038-2047
• 41 Go AS, Chertow GM, Fan D, McCulloch CE, Hsu CY. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. N Engl J Med 2004; 351 (13) 1296-1305
• 42 Hirth RA. The organization and financing of kidney dialysis and transplant care in the United States of America. Int J Health Care Finance Econ 2007; 7 (04) 301-318
• 43 Hsu RK, Powe NR. Recent trends in the prevalence of chronic kidney disease: not the same old song. Curr Opin Nephrol Hypertens 2017; 26 (03) 187-196
• 44 Keith DS, Nichols GA, Gullion CM, Brown JB, Smith DH. Longitudinal follow-up and outcomes among a population with chronic kidney disease in a large managed care organization. Arch Intern Med 2004; 164 (06) 659-663
• 45 System USRD. USRDS 2009 Annual Data Report: Atlas of Chronic Kidney Disease And End-Stage Renal Disease in The United States. Bethesda, MD: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2009
• 46 Thompson S, James M, Wiebe N. et al; Alberta Kidney Disease Network. Cause of death in patients with reduced kidney function. J Am Soc Nephrol 2015; 26 (10) 2504-2511
• 47 Guessous I, McClellan W, Vupputuri S, Wasse H. Low documentation of chronic kidney disease among high-risk patients in a managed care population: a retrospective cohort study. BMC Nephrol 2009; 10 (01) 25
• 48 Kidney Disease: Improving Global Outcomes Blood Pressure Work G. KDIGO 2021 Clinical Practice Guideline for the Management of Blood Pressure in Chronic Kidney Disease. Kidney Int 2021; 99 (3S): S1-S87
• 49 Plantinga LC, Tuot DS, Powe NR. Awareness of chronic kidney disease among patients and providers. Adv Chronic Kidney Dis 2010; 17 (03) 225-236
• 50 Whelton PK, Carey RM, Aronow WS. et al. 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA Guideline for the prevention, detection, evaluation, and management of high blood pressure in adults: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Am Coll Cardiol 2018; 71 (19) e127-e248
• 51 Levey AS, Stevens LA. Estimating GFR using the CKD Epidemiology Collaboration (CKD-EPI) creatinine equation: more accurate GFR estimates, lower CKD prevalence estimates, and better risk predictions. Am J Kidney Dis 2010; 55 (04) 622-627
• 52 Boxwala AA, Rocha BH, Maviglia S. et al. A multi-layered framework for disseminating knowledge for computer-based decision support. J Am Med Inform Assoc 2011; 18 (suppl 1, suppl 1) i132-i139
• 53 Garabedian PM, Gannon MP, Aaron S, Wu E, Burns Z, Samal L. Human-centered design of clinical decision support for management of hypertension with chronic kidney disease. BMC Med Inform Decis Mak 2022; 22 (01) 217
• 54 Chou E, Boyce RD, Balkan B. et al. Designing and evaluating contextualized drug-drug interaction algorithms. JAMIA Open 2021; 4 (01) ooab023
• 55 Gannon MP, Wu E, McMahon GM. et al. Uncontrolled blood pressure and treatment of hypertension in older chronic kidney disease patients. J Am Geriatr Soc 2021; 69 (10) 2985-2987
• 56 Amroze A, Field TS, Fouayzi H. et al. Use of electronic health record access and audit logs to identify physician actions following noninterruptive alert opening: descriptive study. JMIR Med Inform 2019; 7 (01) e12650
• 57 Agarwal R, Sinha AD, Cramer AE. et al. Chlorthalidone for hypertension in advanced chronic kidney disease. N Engl J Med 2021; 385 (27) 2507-2519
• 58 Sinha AD, Agarwal R. Clinical pharmacology of antihypertensive therapy for the treatment of hypertension in CKD. Clin J Am Soc Nephrol 2019; 14 (05) 757-764
• 59 Lingren T, Thaker V, Brady C. et al. Developing an algorithm to detect early childhood obesity in two tertiary pediatric medical centers. Appl Clin Inform 2016; 7 (03) 693-706
• 60 Davis K, Gibson K, Bar-Shain D, Siff J, Gunzler D, Kaelber D. Using Clinical Decision Support to Decrease the Use of Teratogenic Antihypertensive Medications in Women of Childbearing Age. Paper presented at: AMIA 2019 Clinical Informatics Conference; 4/30/2019, 2019; Atlanta, GA

Publication History

Received: 12 July 2022

Accepted: 18 April 2023

Article published online:
12 July 2023