Partnering with Patients and Families to Improve Diagnostic Safety through the OurDX Tool: Effects of Race, Ethnicity, and Language Preference

Abstract

Background Patients and families at risk for health disparities may also be at higher risk for diagnostic errors but less likely to report them.

Objectives This study aimed to explore differences in race, ethnicity, and language preference associated with patient and family contributions and concerns using an electronic previsit tool designed to engage patients and families in the diagnostic process (DxP).

Methods Cross-sectional study of 5,731 patients and families presenting to three subspecialty clinics at an urban pediatric hospital May to December 2021 who completed a previsit tool, codeveloped and tested with patients and families. Prior to each visit, patients/families were invited to share visit priorities, recent histories, and potential diagnostic concerns. We used logistic regression to determine factors associated with patient-reported diagnostic concerns. We conducted chart review on a random subset of visits to review concerns and determine whether patient/family contributions were included in the visit note.

Results Participants provided a similar mean number of contributions regardless of patient race, ethnicity, or language preference. Compared with patients self-identifying as White, those self-identifying as Black (odds ratio [OR]: 1.70; 95% confidence interval [CI]: [1.18, 2.43]) or “other” race (OR: 1.48; 95% CI: [1.08, 2.03]) were more likely to report a diagnostic concern. Participants who preferred a language other than English were more likely to report a diagnostic concern than English-preferring patients (OR: 2.53; 95% CI: [1.78, 3.59]. There were no significant differences in physician-verified diagnostic concerns or in integration of patient contributions into the note based on race, ethnicity, or language preference.

Conclusion Participants self-identifying as Black or “other” race, or those who prefer a language other than English were 1.5 to 2.5 times more likely than their counterparts to report potential diagnostic concerns when proactively asked to provide this information prior to a visit. Actively engaging patients and families in the DxP may uncover opportunities to reduce the risk of diagnostic errors and potential safety disparities.

Keywords

error management and prevention - vulnerable populations - patient safety - patient–provider communication

Background and Significance

Diagnostic errors/delays and health care disparities are “twin challenges” in health care, each worsened by the coronavirus disease 2019 pandemic.[1] [2] [3] [4] [5] [6] Individuals at risk of health care disparities may also be at increased risk of safety events including diagnostic error due to language barrier, reduced self-efficacy, or biased care.[7] [8] [9] [10] [11] [12] Diagnostic errors are generally underreported—a particular hazard for patients identifying as racial and/or ethnic minorities, those with limited English proficiency or health literacy, and those who are un- or underinsured.[10] [13]

Understanding and learning from the experiences of patients/families with diagnostic error is critical to improve patient safety.[14] Patients and families identify components of diagnostic error that may not otherwise be recognized by clinicians or health care systems.[14] [15] [16] Individuals with limited English proficiency further report unique contributing factors to diagnostic error.[7] However, feedback about care concerns from patients/families at risk of disparities is limited.[17] [18] [19] [20] Underreporting of diagnostic errors and lack of patient/family perspectives mean missed safety events and compromised organizational learning.[10] [15] These findings increase the urgency to establish systematic ways to broadly engage patients and families in the diagnostic process (DxP).[13]

Advances in health care technology and policy, such as federally mandated access to electronic health information in the 2021 Cures Act Final Rule, raise new opportunities to partner with patients/families in diagnosis. Studies demonstrate that sharing visit notes can help engage patients/families as diagnostic partners.[21] Patients/families who read notes report better remembering and completing diagnostic tests and referrals,[22] improved relationships with their providers,[22] [23] and identification of breakdowns related to the DxP.[15] [24] [25] Access to notes also enables patients/families to learn more about their health conditions, formulate questions without the time pressure or stressors of the clinical environment, and participate as informed members of the health care team.[26]

Shared electronic health information may also help clinicians understand patients and improve shared decision-making in diagnosis.[27] Recent studies on patient contributions prior to the visit through the patient portal or a waiting room iPad show improved communication and visit efficiency.[28] [29] [30] In some instances, such contributions were used by clinicians to cogenerate visit notes.[29] [30] Incorporating the patient story into the clinic note in this way may be empowering, especially for patients whose voices are not otherwise adequately heard or for those who may experience “systemic oppression or disenfranchisement.”[31]

However, access to the patient portal is itself subject to disparities, and interventions to engage patients/families online may have unintended consequences due to selective participation.[32] [33] [34] [35] One study comparing the use of a portal-based online previsit survey between patients in safety net clinics versus patients in nonsafety net clinics demonstrated lower participation in the former group.[29] Innovations to engage patients in diagnosis therefore require a focus on promoting equity and principled data collection to examine disparities in use.[16] [36] To date little is known about how to engage patients and families of diverse races, ethnicities, or language preferences in the DxP.

We developed an online tool called Our Diagnosis (OurDX) to engage patients and families in the ambulatory DxP. OurDX was designed with patients and families, using evidence-based items and common patient-reported diagnostic concerns previously described in the literature.[7] [15] [29] [37]

Objectives

The objective of the study was to examine differences in sociodemographic factors (self-reported race, ethnicity, and language preference) on patient and family contributions to an electronic previsit tool (OurDX) designed to engage patients and families in the DxP. We hypothesized that OurDX use would vary by race, ethnicity, and language preference including: (1) identification of patient/family diagnostic concerns (primary outcome); (2) the number of patient/family contributions; (3) clinician verification of patient/family diagnostic concerns, and (4) clinician integration of patient/family contributions into the visit note.

Methods

Study Population

Patients and families with scheduled visits in three participating medical and surgical subspecialty clinics in a northeast U.S. academic pediatric hospital during May 1, 2021 to December 31, 2021 were invited to complete OurDX as part of a previsit survey via email. Patient demographics, including legal sex, race, ethnicity, age, preferred language, and interpreter services preference were extracted from the electronic health record (EHR). If language preference was missing in the designated administrative data field, we conducted chart review to identify preferred language.

OurDX Tool

OurDX was codeveloped by patients, families, clinicians, researchers, and experts in user-centered-design and diagnostic safety. Through OurDX, patients and families were invited to contribute visit priorities, recent medical histories, and potential diagnostic concerns (such as feeling their main concern was not heard or problems or delays with tests or referrals). The intervention did not rely on patients and families registering for a patient portal, since OurDX was accessed through email.[38] [39] Eligible participants were emailed a link to the OurDX survey via a third-party vendor survey platform (Tonic Health, Murray, Utah, United States) 5 days prior to their appointment with reminders 3 and 1 day prior to the visit. Completed surveys were automatically imported into the EHR and available for clinician review within the EHR's usual visit workflow dashboard. Further details of OurDX development and implementation are available elsewhere.[37] [40]

Patient/Family Diagnostic Concerns

We defined patient/family diagnostic concerns as: “A problem or delay reported by patients/families that could map to any part of the DxP, as outlined in the National Academy of Medicine conceptual model.”[15] These included problems or concerns related to access to care, inaccurate medical history, delayed tests or referrals, communication breakdowns, and problems or delays with diagnosis or next steps. We calculated the frequency of patient/family diagnostic concerns (primary outcome) from the entire study population.

Chart Review

We conducted chart review, randomly selecting a subset of visits from each participating clinic. We used chart review to further characterize patients and to determine whether patient/family OurDX contributions were incorporated into the visit note, as in previously published methodology.[31] Chart review was completed by a research assistant with support from a pediatrician, using REDcap. We used the chart review sample for qualitative analyses including the secondary outcomes (number of patient/family contribution categories, clinician verification of patient/family concerns, and integration of patient/family contributions into the visit note).

Coding Process

Two physician–researchers reviewed all patient/family reports in the chart review sample. We started with a deductive approach, using the Framework for Patient-Reported DxP-related Breakdowns to code all patient content in OurDX reports, including information provided in the visit priorities, recent medical history, and diagnostic concerns.[15] We then used an inductive approach to describe and label any new categories emerging from the data, beyond the framework categories. Using constant comparison and in-depth discussions to reach consensus, we identified and defined three additional categories, testing once again for saturation of codes in the data. When no new categories emerged, we finalized 10 coding categories: access to care, medical history/symptoms, information on medications related to main concern, recent visits for the same problem, multidisciplinary information, tests/referrals, explanation (diagnosis) or next steps, care coordination, communication concern, or other, each supported as important diagnostic information in the literature.[15] [31] [41] [42] We defined a patient/family contribution as any content that was coded in these 10 categories and provided by the patient/family across all fields in the OurDX tool.

The two physician–researchers also reviewed OurDX reports in detail in the chart review sample to verify patient/family concerns. We defined a patient/family diagnostic concern as verified if physician–researcher review of the patient/family information in the OurDX report and the accompanying chart review and visit note confirmed a probable diagnostic safety opportunity for clinicians. In other words, the patient/family provided information that clinicians could respond to at the point of care to improve DxP safety. Examples include the opportunity to provide specific test results the patient/family had not received, assist with delayed referrals, or help ensure that patient/family main concerns were correctly heard and understood, to help prevent potential downstream diagnostic errors or delays. Throughout the coding process the physicians were blinded to the patient's sociodemographic information.

To test intercoder reliability, we used 20% of the chart review sample and calculated the AC1 and kappa statistics. We used AC1 because some categories were used more frequently than others. However, we also calculated the kappa statistic because it is a more conservative measure and more commonly used. We considered agreement coefficients 0.61 to 0.8 as good agreement and 0.81 to 1.00 as excellent agreement. Intercoder reliability testing demonstrated good to excellent agreement: AC1 0.94 (0.89, 0.98) and kappa 0.84 (0.74, 0.94) for patient contributions and AC1 0.83 (0.76, 0.89); kappa 0.79 (0.72, 0.87) for patient diagnostic concern verification. Based on this agreement, one physician coded the remainder of reports in the sample.

Analysis

We used descriptive statistics to compare patient sociodemographics between respondents and nonrespondents in the participating clinics during the study period. We used chi-squared analysis to compare the mean number of contributions and clinician verification of patient/family diagnostic concerns, by race, ethnicity, and language preference. We used logistic regression to examine potential sociodemographic patient factors associated with patient/family identification of potential diagnostic concerns and integration of patient priorities into the clinician's note. For patients with >1 visit during the study period (<20%), we randomly selected one visit to include in our analyses using established methodology.[43]

Results

Study Population

Among 18,129 visits during the study period, 7,075 (39.0%) OurDX reports were submitted by 5,731 patients or parents/proxies (“family”), approximating the response rate of clinical previsit surveys across all ambulatory clinics in our organization (35%). Patient characteristics are shown in [Table 1]. Compared with nonparticipating patients, participating patients were more likely to self-identify as White and English-preferring ([Supplementary Appendixes A] and [B], available in the online version), consistent with the overall sociodemographics of previsit survey users in our organization. We conducted a total of 324 chart reviews. Patient characteristics in the chart review sample were similar overall to the whole patient population and organizational previsit survey respondents.

Table 1 Patient characteristics

Patient characteristics	All participants (N = 5,731)	Chart review participants (N = 320)
Age, y (mean, SD)	7.14 (7.56)	7.96 (8.34)
Gender
Male	3,234 (56.43%)	184 (57.50%)
Female	2,497 (43.57%)	136 (42.50%)
Race
White	3,806 (66.41%)	202 (63.13%)
Black	262 (4.57%)	17 (5.31%)
Asian	223 (3.89%)	11 (3.44%)
Other	496 (8.65%)	33 (10.31%)
Unknown	944 (16.47%)	57 (17.81%)
Ethnicity
Hispanic	349 (6.09%)	25 (7.81%)
Non-Hispanic	4,168 (72.73%)	225 (70.31%)
Unknown	1,214 (21.18%)	70 (21.88%)
Preferred language
English	5,518 (96.28%)	304 (95.00%)
Other language	213 (3.72%)	16 (5.00%)
Total number of OurDX reports
1	4,634 (80.86%)	316 (98.75%)
2	907 (15.83%)	4 (1.25%)
≥3	190 (3.32%)	0
Total number of chronic conditions (mean, SD)	N/A	1.77 (1.17)
Total number of medications (mean, SD)	N/A	0.91 (1.30)

Abbreviations: N/A, not applicable; SD, standard deviation.

Note: number of chronic conditions and medications were extracted on chart review and therefore were not available (n/a) for the entire patient population.

Patient/Family Contributions

Participants made multiple contributions to the DxP, reflected in the visit priorities, recent medical history, and potential diagnostic concerns, across all 10 categories including: access problems, medical history, information on medications, interdisciplinary information, recent visits at other health care centers, problems/delays with tests/referrals, communication issues, care coordination, explanation/next steps, or other. A comparison of the mean number of patient/family contribution categories is shown in [Table 2]. Overall, patients and families contributed information in a mean of 2 to 3 categories, with a range from 1 to 8. We did not observe statistically significant differences in the mean number of contribution categories by race, ethnicity, or preferred language.

Table 2 Patient contributions in OurDX by patient sociodemographic factors

Patient characteristics	N = 314	Mean number of contributions	Standard deviation	p-Value
Race				0.079
White	198	3.06	1.50
Black/African American	17	2.41	0.87
Other	32	2.66	1.45
Asian	11	2.09	0.83
Unknown	56	3.00	1.62
Ethnicity				0.584
Non-Hispanic	220	3.00	1.49
Hispanic	25	2.76	1.61
Unknown	69	2.83	1.41
Language preference				0.385
English	298	2.96	1.46
Other than English	16	2.63	1.86

Notes: Contributions were coded from patient reports with actionable information. Of the 320 participants in the chart review sample, 6 (1.9%) did not have reports with actionable information, resulting in n = 314.

Patient/Family Diagnostic Concerns

Overall, 10.6% of unique participants identified at least one potential diagnostic concern, with a total of 609 participants reporting 735 potential diagnostic concerns during the study. The most common patient diagnostic concerns included problems/delays with tests or referrals (379/735; 51.6%), problems/delays related to diagnosis or next steps (257/735; 35.0%), and patients/families feeling their main concern was not heard (232/735; 31.6%). Factors associated with reporting a potential diagnostic concern are shown in [Table 3]. Compared with 9.2% of White respondents, 14.0 to 15.3% of respondents self-identifying as Black, Asian, or “other” race reported a diagnostic concern. Patients self-identifying as Black or “other” race were significantly more likely to report a potential diagnostic concern than those self-identifying as white (odds ratio: [OR]: 1.70; 95% confidence interval [CI]: [1.18, 2.43] and OR: 1.48; 95% CI: [1.08, 2.03], respectively). Similarly, compared with 10.1% of English-preferring participants, 25.1% of individuals who preferred a language other than English reported a potential diagnostic concern; (OR: 2.53; 95% CI: [1.78, 3.59]. Notably, reports from participants who preferred a language other than English were five times as likely to indicate that the main concern had not been heard, as opposed to reports from English-preferring participants (3.4% English-preferring vs. 16.9% with another language preference, [Supplementary Appendix C], available in the online version). We did not observe any significant differences by patient gender or ethnicity.

Table 3 Multiple logistic regression of sociodemographic factors associated with identification of patient diagnostic concerns (N = 5731)

Variable	% patients with diagnostic concerns	OR	95% CI		p-Value
Age, y (mean, SD)	7.39 (8.30)	1.006	0.995	1.017	0.253
Gender					0.208
Female	10.00%	0.895	0.753	1.064
Male (ref)	11.11%
Race					0.006
White (ref)	9.18%
Asian	14.03%	1.411	0.943	2.112
Black	14.94%	1.692	1.181	2.426
Other	15.32%	1.478	1.078	2.028
Unknown	11.90%	1.258	0.920	1.719
Ethnicity					0.935
Non-Hispanic or Latino (ref)	10.14%
Unknown	11.29%	1.035	0.776	1.381
Hispanic	14.08%	0.954	0.656	1.386
Language preference					<0.0001
English (ref)	10.07%
Other than English	25.12%	2.528	1.783	3.586

Abbreviations: CI, confidence interval; OR, odds ratio; ref, reference.

Note: Mean and standard deviation of age with at least one patient diagnostic concern were reported.

Clinician Verification of Patient Diagnostic Concerns

Across patient population groups, the majority (61.5–84.6%) had diagnostic concerns that were confirmed on physician–researcher review ([Table 4]). We did not observe any significant differences in the proportion of confirmed patient diagnostic concerns when compared by patient sociodemographic characteristics, although the total number of patient-reported diagnostic concerns were very small in some subgroups.

Table 4 Clinician verification of patient diagnostic concerns in chart review sample

Patient characteristic	N = 213	Clinician-verified patient diagnostic concern, n (%)	p-Value
Race			0.9057
White	129	91 (70.54%)
Black/African American	13	8 (61.54%)
Other	24	16 (66.67%)
Asian	8	6 (75.00%)
Unknown	39	29 (74.36%)
Ethnicity			0.96
Non-Hispanic	146	102 (69.86%)
Hispanic	17	12 (70.59%)
Unknown	50	36 (72.00%)
Language preference			0.2472
English	200	139 (69.50%)
Another language	13	11 (84.62%)

Integration of Patient/Family Priorities into the Visit Note

In total, 294 (90.7%) OurDX reports in the 324 chart reviews provided visit priorities. Among these, 191(65.0%) of clinician notes included all patient/family documented priorities. In addition, 97 (33.0%) included some patient/family priorities. We did not observe any differences in the likelihood of clinicians including all patient/family priorities in the note by patient race, ethnicity, or preferred language ([Table 5]).

Table 5 Multiple logistic regression of patient/family priorities included in clinician note by sociodemographic characteristics (n = 320)

Variable	% of patients with all priorities included in note	OR	95% CI		p-Value
Age, y	7.64 (8.58)	0.987	0.959	1.016	0.365
Gender					0.109
Female	58.87%	0.663	0.400	1.097
Male (ref)	70.06%
Race					0.585
White (ref)	64.74%
Black/African American	83.33%	2.483	0.52	11.852
Other	71.43%	1.021	0.385	2.075
Asian	50.00%	0.598	0.143	2.498
Unknown	62.26%	0.645	0.250	1.661
Ethnicity					0.219
Hispanic	84.21%	2.876	0.697	11.871
Unknown	67.79%	1.643	0.671	4.027
Non-Hispanic (ref)	62.80%
Language preference					0.753
Other than English	77.78%	1.329	0.225	7.835
English (ref)	64.89%

Abbreviations: CI, confidence interval; OR, odds ratio; ref, reference.

Discussion

Our study of over 5,700 patients and families attending 7,075 ambulatory visits with medical and surgical subspecialists demonstrates that OurDX can serve as a structured tool to invite contributions to the DxP from patients/families of varying backgrounds. Although responses were more common among individuals identifying as White or English-preferring, when patients/families at greater risk of health care disparities did participate, they provided important contributions and were more likely to report potential diagnostic concerns than their counterparts. Although clinicians may be skeptical about the clinical relevance or interpretability of diagnostic concerns reported by some patients at risk of health disparities—such as those with limited English proficiency—the majority were confirmed on clinician review and we observed no differences in the likelihood of clinician verification by patient sociodemographics, although the small numbers in some subgroups require further study.

Our findings underscore that the ability of patients/families to identify DxPs at risk may depend on how the question is asked. Despite known underreporting of error among patient populations potentially at risk of health disparities,[10] [20] eliciting process measures like whether patients felt heard or experienced specific problems or delays related to the DxP actually yielded higher reporting of diagnostic concerns from patients and families self-identifying as Black or other race and among patients and families preferring a language other than English compared with their counterparts, respectively. This may be explained at least in part by patients who may not understand the term “error”; may not be sure about whether their experience constitutes an error; or may harbor greater concerns for reporting an error, due to fear of retaliation or other ill-effects on the patient–clinician relationship.[7] [15] [17] [44] [45] A tool that empowers patients/families by routinely eliciting their DxP concerns in basic terms (i.e., “did you feel heard?”) and in the comfort and safety of their own home may help overcome some of these barriers, although additional support to participate is needed.

Because patient/family reported diagnostic concerns in OurDX reports were available to the clinician at the time of the visit, these reports present a unique opportunity to act upon potential diagnostic safety opportunities at the point of care, thus helping to prevent downstream diagnostic errors. Recognizing and addressing patient concerns such as not feeling heard and problems or delays in tests, referrals, diagnosis, or next steps are critical to improve diagnostic safety[7] [14] [15] and may be particularly vital for patients and families from historically marginalized communities, who may be at greater risk for diagnostic error.[11] For example, our findings indicate that reports from patients/families who prefer a language other than English were more than five times as likely to indicate that the patient/family's main concern was not heard. Identification of such patient/family diagnostic concerns may prompt clinicians to listen more intently, check for understanding, ensure the use of interpreters when needed, or use “teach back” principles to ensure greater alignment between patients/families and clinicians,[46] [47] [48] [49] behaviors that were not measured in this study but that may particularly benefit individuals at risk of diagnostic error. Eliciting information before the visit might also help ameliorate potential implicit bias on the part of the provider, although further research is needed.[50] Finally, systematically asking patient/family priorities and concerns through OurDX before the visit may help tackle disparities by better aligning patient–clinician agendas and shared understanding. Taken together, these factors suggest that OurDX may help clinicians identify and address at least some equity gaps in diagnostic safety.

We did not find significant differences in clinician integration of patient/family priorities into the visit note by patient sociodemographic characteristics, although larger studies are needed. This is an important issue because cultural differences, language barrier, implicit bias, or miscomprehension may otherwise result in misalignment between patients/families and clinicians regarding the significance of patient symptoms or concerns.[36] Such misalignments between patients and clinicians have been associated with diagnostic delay and diagnostic blindspots.[16] [51] Documenting priorities and histories in patient or family member's own words may also help to improve the accuracy of notes.[52] This may be of particular benefit to patients at risk of healthcare disparities since inaccurate records were an important contributing factor cited by patients with limited English language health literacy or disadvantaged socioeconomic position who reported a diagnostic error in a U.S. population-based survey.[7]

Finally, recent data demonstrate that negative descriptors are more commonly found in the EHRs of patients self-identifying as a racial minority and may exacerbate health care disparities.[53] [54] [55] Stigmatizing language can be transmitted in the EHR, affecting the attitudes and practices of other clinicians.[56] [57] Sharing and cogenerating notes with patients and families may help raise awareness about more neutral and respectful EHR language. Further research is needed to test whether incorporating patient and family priorities and histories in notes may help mitigate this disparity.

Strengths and Limitations

Although the study included over 7,000 patient/family reports, the overall sample size for marginalized populations was small. In addition, the response rate in our study was limited, although it exceeded the response rate typical of online surveys.[58] [59] Like many studies focusing on health disparities related to health information technology use, it was inherently limited by nonresponse bias, although patients/families self-identifying a race other than White or a language preference other than English showed a >25% response rate, exceeding many email surveys.[60] Similar to prior studies using digital surveys, overall response rates to OurDX were the highest among patients and families who self-identified as White or English-preferring, highlighting ongoing challenges in addressing the barriers and inequities in accessing digital tools and digital health literacy, and missed opportunities to learn from patients, especially those who use interpreters.[61] [62] [63] [64] [65] [66] While we tested the intervention at three different medical and surgical subspecialty clinics, the study involved one site, limiting generalizability.

To mitigate known challenges in patient portal registration among patients and families facing health disparities, we sent a survey link directly via email, bypassing the need for a patient portal account and alleviating a potential barrier to survey access.[39] [67] The survey was written in English, and response rates could be improved with translation to other languages, and EHR tools that better support patients whose primary language is not English.[68] Additional factors driving participation that were raised during our study and others include broader support for speaking up—especially among patients vulnerable to health disparities, provider encouragement to participate, and reassurance that providers read patient/family contributions.[17] [20] [45] [69] Patients who prefer a language other than English may already be at a disadvantage in reviewing notes and may not be able to thus identify and speak up about note inaccuracies. Our study did not examine socioeconomic factors or other social determinants of health that may also affect participation. Far more sweeping changes in structural racism, social justice, health literacy, and information technology are needed to achieve “Techquity,” “the strategic development and deployment of technology in health care and health to achieve health equity.”[62] Interventions like OurDX are a humble step and must be further developed in concert with these broader policy efforts and community participants. Larger studies with greater diversity are needed to build upon these exploratory findings as well as qualitative studies that may provide additional rich context to the interpretation of these findings and future tool refinement.

Conclusion

Emerging research suggests underreporting of medical errors among patient populations at risk of healthcare disparities. In this exploratory study, use of OurDX—an online tool to engage patients and families in the DxP—resulted in significantly greater identification of patient-reported DxP concerns among patients and families from racial minorities or those who preferred a language other than English, compared with their respective counterparts. For example OurDX reports among participants preferring a language other than English were more than 5 times as likely to indicate that the patient/family's main concern was not heard. Because patient contributions through OurDX are available at the time of the visit, clinicians may have a greater opportunity to identify and act on patient/family concerns at the point of care before they may lead to diagnostic errors. We did not observe differences in the number of DxP contributions, the proportion of patient/family-reported diagnostic concerns verified on clinician review, or the likelihood of integrating patient/family contributions into clinician visit note by clinicians by race, ethnicity, or language preference among those who used the tool. Greater solicitation and integration of priorities, perspectives, and concerns of patients at risk of healthcare disparities into the medical record may help engage more diverse patients in the DxP and ultimately improve diagnostic safety, but further research with broader patient populations and more in-depth qualitative studies are needed to address disparity gaps.

Clinical Relevance Statement

Patients and families historically at higher risk of healthcare disparities were more likely to report diagnostic concerns through an online diagnostic safety tool as compared with their counterparts, providing an opportunity to engage broader patient populations in safety, improve under-reporting of concerns, and potentially prevent diagnostic errors and safety disparities at the point of care, if implemented alongside broader organizational equity efforts.

Multiple-Choice Questions

1. ALL of the following patient-reported diagnostic concerns were most commonly reported by patients and families, EXCEPT:

   • Problems or delays with tests or referrals

   • Problems or delays related to diagnosis or next steps

   • Not feeling their main concern was heard by clinicians

   • Delay in medication refill requests

   Correct Answer: The correct answer is option d. The most commonly reported patient diagnostic concerns included problems/delays with tests or referrals, problems/delays related to diagnosis or next steps, and patients feeling their main concern was not heard.

2. As compared with English-preferring participants, how much more likely were participants preferring a language other than English to report not feeling heard?

   • 1.5 times more likely

   • 5 times more likely

   • 2.5 times more likely

   • 6 times more likely

   Correct Answer: The correct answer is option b. 16.9% of individuals who preferred a language other than English reported not feeling heard as compared with 3.4% of English-preferring participants.

Conflict of Interest

None declared.

Acknowledgments

The authors thank the patients and families who participated in the study for their insights and contributions. They also thank Kendall Harcourt for her assistance with the manuscript and project management.

Protection of Human and Animal Subjects

The study was approved by Boston Children's Hospital Institutional Review Board.

• References

• 1 Ibrahim SA, Pronovost PJ. Diagnostic errors, health disparities, and artificial intelligence: a combination for health or harm?. JAMA Health Forum 2021; 2 (09) e212430-e212430
• 2 Chin MH. Advancing health equity in patient safety: a reckoning, challenge and opportunity. BMJ Qual Saf 2020;
• 3 Woolf SH, Masters RK, Aron LY. Effect of the COVID-19 pandemic in 2020 on life expectancy across populations in the USA and other high income countries: simulations of provisional mortality data. BMJ 2021; 373 (1343) n1343
• 4 Rosen A, Carter D, Applebaum JR. et al. Critical care clinicians' experiences of patient safety during the COVID-19 pandemic. J Patient Saf 2022; 18 (08) e1219-e1225
• 5 Gandhi TK, Singh H. Reducing the risk of diagnostic error in the COVID-19 era. J Hosp Med 2020; 15 (06) 363-366
• 6 Joseph JW, Landry AM, Kennedy M. et al. Association of race and ethnicity with triage emergency severity index scores and total visit work relative value units for emergency department patients. JAMA Netw Open 2022; 5 (09) e2231769-e2231769
• 7 Bell SK, Dong J, Ngo L, McGaffigan P, Thomas EJ, Bourgeois F. Diagnostic error experiences of patients and families with limited English-language health literacy or disadvantaged socioeconomic position in a cross-sectional US population-based survey. BMJ Qual Saf 2022;
• 8 Wasserman M, Renfrew MR, Green AR. et al. Identifying and preventing medical errors in patients with limited English proficiency: key findings and tools for the field. J Healthc Qual 2014; 36 (03) 5-16
• 9 Divi C, Koss RG, Schmaltz SP, Loeb JM. Language proficiency and adverse events in US hospitals: a pilot study. Int J Qual Health Care 2007; 19 (02) 60-67
• 10 Schulson LB, Novack V, Folcarelli PH, Stevens JP, Landon BE. Inpatient patient safety events in vulnerable populations: a retrospective cohort study. BMJ Qual Saf 2020; 1-8
• 11 Michelson KA, McGarghan FLE, Patterson EE, Samuels-Kalow ME, Waltzman ML, Greco KF. Delayed diagnosis of serious paediatric conditions in 13 regional emergency departments. BMJ Qual Saf 2022; x: x
• 12 Khan A, Yin HS, Brach C. et al; Patient and Family Centered I-PASS Health Literacy Subcommittee. Association between parent comfort with English and adverse events among hospitalized children. JAMA Pediatr 2020; 174 (12) e203215-e203215
• 13 Balogh EP, Miller BT, Ball JR. et al. Improving Diagnosis in Health Care. National Academies Press (US); 2015.
• 14 Giardina TD, Haskell H, Menon S. et al. Learning from patients' experiences related to diagnostic errors is essential for progress in patient safety. Health Aff (Millwood) 2018; 37 (11) 1821-1827
• 15 Bell SK, Bourgeois F, DesRoches CM. et al. Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care. BMJ Qual Saf 2022; 31 (07) 526-540
• 16 Bell SK, Bourgeois F, Dong J. et al. Patient identification of diagnostic safety blindspots and participation in “good catches” through shared visit notes. Milbank Q 2022; 100 (04) 1121-1165
• 17 Lam BD, Bourgeois F, Dong ZJ, Bell SK. Speaking up about patient-perceived serious visit note errors: patient and family experiences and recommendations. J Am Med Inform Assoc 2021; 28 (04) 685-694
• 18 Martino SC, Elliott MN, Mallett JS, Orr N, Gaillot S, Haviland AM. Speaking up and walking out: are vulnerable patients less likely to disagree with or change doctors?. Med Care 2018; 56 (09) 749-754
• 19 Schlesinger M, Mitchell S, Elbel B. Voices unheard: barriers to expressing dissatisfaction to health plans. Milbank Q 2002; 80 (04) 709-755 , iv–v
• 20 Khan A, Parente V, Baird JD. et al; Patient and Family Centered I-PASS SCORE Scientific Oversight Committee. Association of patient and family reports of hospital safety climate with language proficiency in the US. JAMA Pediatr 2022; 176 (08) 776-786
• 21 Blease CR, Bell SK. Patients as diagnostic collaborators: sharing visit notes to promote accuracy and safety. Diagnosis (Berl) 2019; 6 (03) 213-221
• 22 Bell SK, Folcarelli P, Fossa A. et al. Tackling ambulatory safety risks through patient engagement: what 10,000 patients and families say about safety-related knowledge, behaviors, and attitudes after reading visit notes. J Patient Saf 2021; 17 (08) e791-e799
• 23 Bell SK, Gerard M, Fossa A. et al. A patient feedback reporting tool for OpenNotes: implications for patient-clinician safety and quality partnerships. BMJ Qual Saf 2017; 26 (04) 312-322
• 24 Bell SK, Delbanco T, Elmore JG. et al. Frequency and types of patient-reported errors in electronic health record ambulatory care notes. JAMA Netw Open 2020; 3 (06) e205867-e205867
• 25 Giardina TD, Choi DT, Upadhyay DK. et al. Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes. J Am Med Inform Assoc 2022; 29 (06) 1091-1100
• 26 Fossa AJ, Bell SK, DesRoches C. OpenNotes and shared decision making: a growing practice in clinical transparency and how it can support patient-centered care. J Am Med Inform Assoc 2018; 25 (09) 1153-1159
• 27 Berger ZD, Brito JP, Ospina NS. et al. Patient centred diagnosis: sharing diagnostic decisions with patients in clinical practice. BMJ 2017; 359: j4218
• 28 Anderson MO, Jackson SL, Oster NV. et al. Patients typing their own visit agendas into an electronic medical record: pilot in a safety-net clinic. Ann Fam Med 2017; 15 (02) 158-161
• 29 Walker J, Leveille S, Kriegel G. et al. Patients contributing to visit notes: mixed methods evaluation of OurNotes. J Med Internet Res 2021; 23 (11) e29951
• 30 Kumah-Crystal YA, Stein PM, Chen Q. et al. Before-visit questionnaire: a tool to augment communication and decrease provider documentation burden in pediatric diabetes. Appl Clin Inform 2021; 12 (05) 969-978
• 31 Shucard H, Muller E, Johnson J. et al. Clinical use of an electronic pre-visit questionnaire soliciting patient visit goals and interim history: a retrospective comparison between safety-net and non-safety-net clinics. Health Serv Res Manag Epidemiol 2022 ;9:23333928221080336
• 32 Gordon NP, Hornbrook MC. Differences in access to and preferences for using patient portals and other eHealth technologies based on race, ethnicity, and age: a database and survey study of seniors in a large health plan. J Med Internet Res 2016; 18 (03) e50
• 33 Perzynski AT, Roach MJ, Shick S. et al. Patient portals and broadband internet inequality. J Am Med Inform Assoc 2017; 24 (05) 927-932
• 34 Veinot TC, Mitchell H, Ancker JS. Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc 2018; 25 (08) 1080-1088
• 35 Lyles CR, Sarkar U. Health literacy, vulnerable patients, and health information technology use: where do we go from here?. J Gen Intern Med 2015; 30 (03) 271-272
• 36 Sabin JA. Tackling implicit bias in health care. N Engl J Med 2022; 387 (02) 105-107
• 37 Bell SK, Bourgeois FC, Liu SK, Thomas E, Lowe B, Salmi L. Co-development of OurDX—an online tool to facilitate patient and family engagement in the diagnostic process. BMJ Opinion 2021 . Accessed March 3, 2023 at: https://blogs.bmj.com/bmj/2021/10/14/co-development-of-ourdx-an-online-tool-to-facilitate-patient-and-family-engagement-in-the-diagnostic-process/
• 38 Marshall A, Mondowney G, Sartini P. et al; Centers for Medicare & Medicaid Services (CMS), HHS. Medicare program; revisions to payment policies under the physician fee schedule and other revisions to part B for CY 2018; Medicare shared savings program requirements; and Medicare diabetes prevention program. Final rule. Fed Regist 2017; 82 (219) 52976-53371
• 39 LeLaurin JH, Nguyen OT, Thompson LA. et al. Disparities in pediatric patient portal activation and feature use. JAMIA Open 2021; 4 (03) ooab086
• 40 Bell SK, Dong ZJ, Desroches CM. et al. Partnering with patients and families living with chronic conditions to coproduce diagnostic safety through OurDX: a previsit online engagement tool. J Am Med Inform Assoc 2023; 30 (04) 692-702
• 41 El-Kareh R, Hasan O, Schiff GD. Use of health information technology to reduce diagnostic errors. BMJ Qual Saf 2013; 22 (Suppl. 02) ii40-ii51
• 42 Murphy DR, Meyer AND, Sittig DF, Meeks DW, Thomas EJ, Singh H. Application of electronic trigger tools to identify targets for improving diagnostic safety. BMJ Qual Saf 2019; 28 (02) 151-159
• 43 Bettenhausen JL, Richardson T, Herzig SJ, Hall M. Methodologic progress note: a clinician's guide to logistic regression. J Hosp Med 2021; 16 (11) 672-674
• 44 Delbanco T, Bell SK. Guilty, afraid, and alone–struggling with medical error. N Engl J Med 2007; 357 (17) 1682-1683
• 45 Mazor KM, Smith KM, Fisher KA, Gallagher TH. Speak up! Addressing the paradox plaguing patient-centered care. Ann Intern Med 2016; 164 (09) 618-619
• 46 Agency for Healthcare Research and Quality. Teach-back: intervention. U.S. Department of Health and Human Services; 2021. . Accessed September 10, 2022 at: https://www.ahrq.gov/patient-safety/reports/engage/interventions/teachback.html
• 47 Boissy A. How—and—why to listen until someone feels heard. 2018
• 48 Maletsky KD, Worsley D, Tran Lopez K. et al. Communication experiences of caregivers using a language other than English on inpatient services. Hosp Pediatr 2023; 13 (06) 471-479
• 49 Rojas CR, Chamberlain JM, Cohen JS. et al. Undertriage for children with caregivers preferring languages other than English. Pediatrics 2023; 151 (06) e2022059386
• 50 Maina IW, Belton TD, Ginzberg S, Singh A, Johnson TJ. A decade of studying implicit racial/ethnic bias in healthcare providers using the implicit association test. Soc Sci Med 2018; 199: 219-229
• 51 Amelung D, Whitaker KL, Lennard D. et al. Influence of doctor-patient conversations on behaviours of patients presenting to primary care with new or persistent symptoms: a video observation study. BMJ Qual Saf 2020; 29 (03) 198-208
• 52 Sokol-Hessner L, Kane GJ, Annas CL. et al. Development of a framework to describe patient and family harm from disrespect and promote improvements in quality and safety: a scoping review. Int J Qual Health Care 2019; 31 (09) 657-668
• 53 Sun M, Oliwa T, Peek ME, Tung EL. Negative patient descriptors: documenting racial bias in the electronic health record. Health Aff (Millwood) 2022; 41 (02) 203-211
• 54 Himmelstein G, Bates D, Zhou L. Examination of stigmatizing language in the electronic health record. JAMA Netw Open 2022; 5 (01) e2144967-e2144967
• 55 Beach MC, Saha S, Park J. et al. Testimonial injustice: linguistic bias in the medical records of Black patients and women. J Gen Intern Med 2021; 36 (06) 1708-1714
• 56 P Goddu A, O'Conor KJ, Lanzkron S. et al. Do words matter? Stigmatizing language and the transmission of bias in the medical record. J Gen Intern Med 2018; 33 (05) 685-691
• 57 Kelly JF, Westerhoff CM. Does it matter how we refer to individuals with substance-related conditions? A randomized study of two commonly used terms. Int J Drug Policy 2010; 21 (03) 202-207
• 58 Centers for Medicare and Medicaid Services. HCAHPS: patients' perspectives of care survey. CMS.gov. 2013 . Accessed May 29, 2020 at: http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/HospitalHCAHPS.html
• 59 Forcino RC, Barr PJ, O'Malley AJ. et al. Using CollaboRATE, a brief patient-reported measure of shared decision making: results from three clinical settings in the United States. Health Expect 2018; 21 (01) 82-89
• 60 Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 2008; 11 (02) 322-333
• 61 Clark CR, Akdas Y, Wilkins CH. et al. TechQuity is an imperative for health and technology business: let's work together to achieve it. J Am Med Inform Assoc 2021; 28 (09) 2013-2016
• 62 Rhee K, Dankwa-Mullan I, Brennan V, Clark C. What is techquity?. J Health Care Poor Underserved 2021; 32 (02) xiii-xviii
• 63 Pew Research Center. Internet/broadband fact sheet. 2021 Accessed April 20, 2023 at: https://www.pewresearch.org/internet/fact-sheet/internet-broadband/
• 64 van Kessel R, Wong BLH, Clemens T, Brand H. Digital health literacy as a super determinant of health: More than simply the sum of its parts. Internet Interv 2022; 27: 100500
• 65 Rodriguez JA, Clark CR, Bates DW. Digital health equity as a necessity in the 21st century cures act era. JAMA 2020; 323 (23) 2381-2382
• 66 Johnson C, Richwine C, Patel V. Individuals' access and use of patient portals and smartphone health apps, 2020; 2021. Accessed May 16, 2023 at: https://www.healthit.gov/data/data-briefs/individuals-access-and-use-patient-portals-and-smartphone-health-apps-2020
• 67 Bush RA, Vemulakonda VM, Richardson AC, Deakyne Davies SJ, Chiang GJ. Providing access: differences in pediatric portal activation begin at patient check-in. Appl Clin Inform 2019; 10 (04) 670-678
• 68 Payne TH, Lehmann CU, Zatzick AK. The voice of the patient and the electronic health record. Appl Clin Inform 2023; 14 (02) 254-257
• 69 Irizarry T, DeVito Dabbs A, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015; 17 (06) e148

Publication History

Received: 02 March 2023

Accepted: 24 July 2023

Article published online:
15 November 2023