Creation and Evaluation of a Clinical Informatics Match: Initial Findings

Abstract

Background Clinical Informatics (CI) fellowship programs utilize the Electronic Residency Application Service (ERAS) to gather applications but until recently used an American Medical Informatics Association (AMIA) member-developed, simultaneous offer–acceptance process to match fellowship applicants to programs. In 2021, program directors collaborated with the AMIA to develop a new match to improve the process.

Objective Describe the results of the first 2 years of the match and address opportunities for improvement.

Methods We obtained applicant data for fellowship applicants in 2021 and 2022 from the ERAS and match data for the same years from the AMIA. We analyzed our data using descriptive statistics.

Results There were 159 unique applicants over the 2-year period. Applicants submitted 2,178 applications with a median of 10 per applicant (interquartile range [IQR] 3–20). One hundred and four applicants (65.4%) participated in the match and ranked a median of seven programs (2–12). Forty-two programs in 2021 and 47 programs in 2022 offered a combined total 153 positions in the match. Participating programs ranked a median of eight applicants per year (IQR 5–11). Of participating applicants, 95 (91.3%) successfully matched and of those 66 (69.5%) received their top choice. Thirty-two programs (76.2%) matched at least one candidate in 2021 and 33 programs (70.2%) matched at least one candidate in 2022. In both years, 24 programs filled all available slots (57.1% in 2021 and 51.1% in 2022).

Conclusion Applicants were extremely successful in the new match, which successfully addressed most of the challenges of the simultaneous offer–acceptance process identified by program directors. However, applicant attrition resulted in a quarter of programs going unmatched. Although many programs still filled slots outside the match, fellowship slots may remain unfilled while the CI practice pathway remains open.

Keywords

accreditation and licensure - clinical informatics - education - physician - workforce

Background and Significance

In 2005, the American Medical Informatics Association (AMIA) sponsored the creation of a formal certification program for physicians in Clinical Informatics (CI) through the American Board of Preventive Medicine (ABPM) with cosponsorship from the American Board of Pathology.[1] In 2009, AMIA published a white paper proposing the requirements for CI fellowship training,[2] which were approved by the Accreditation Council for Graduate Medical Education (ACGME). The American Board of Medical Specialties approved CI as a subspecialty in 2011[3] and in 2014, the ACGME approved the first four CI fellowship programs.[4] According to ACGME rules, the primary specialty departments (home department) that may host a CI fellowship program include Anesthesiology, Diagnostic Radiology, Emergency Medicine, Family Medicine, Internal Medicine, Medical Genetics and Genomics, Pathology, Pediatrics, or Preventive Medicine.[5]

AMIA established a formal community of Clinical Informatics Program Directors (CIPD) to support communication and resource sharing among training programs. This group recognized that an uncontrolled competitive process to recruit fellows was not desirable for applicants or programs, and agreed on the need for a fair process to recruit fellows. Aiming to ultimately join an existing match program, like the National Resident Matching Program (NRMP), the CIPD initially developed and used a simultaneous offer–acceptance process with the first instance occurring in December 2015 for fellows starting July 2016.[6] Despite multiple efforts to collaborate with the NRMP, the San Francisco Match, and others, no organization was willing or able to support the unique needs of the CI subspecialty and their home departments[6] resulting in the simultaneous offer–acceptance process serving the needs of fellowship applicants and programs for 6 years.

While the simultaneous offer–acceptance process helped to establish a collaborative process for fellow recruitment, it was an interim step and not the desired end solution. Based on feedback from applicants and programs on the initial process, and in an attempt to address the limitations of established home departments of CI programs,[6] the CIPD developed its own match algorithm based on the Gale–Shapely algorithm used by the NRMP.[7] [8] In 2021, AMIA hosted the first official CI Match using the new algorithm for the class of fellows starting July 2022.

The ABPM recently extended the practice pathway for board certification by 3 years reportedly to allow those derailed by the coronavirus disease pandemic to become certified and to allow time for continued growth of ACGME-accredited CI fellowships.[9] [10] If CI fellowship training is to be the sole path toward physician certification, significant efforts are needed to support programs in marketing, recruitment, and funding. To maximize these efforts, program directors must be able to quantify the pipeline of candidates and understand what drives students and residents toward subspecialization in CI.

While the Electronic Residency Application Service (ERAS) publishes data on the number of applicants annually, because these data are aggregated based on the home department for the fellowship, since, it is difficult to ascertain the total number of applicants across all programs.[11] Prior studies have made efforts to evaluate the pool of CI applicants,[12] [13] however these data are incomplete as they fail to quantify the number of applicants that chose to participate in the simultaneous offer–acceptance process.

Objectives

Our objectives were to describe the results of the AMIA match in its first 2 years, to assess how well the AMIA match addressed previously identified challenges, and to identify ongoing opportunities for improvement.

Methods

Data Sources

We obtained data on applicants to CI fellowships in 2021 and 2022 from ERAS. The dataset included applicant name, ERAS ID, program name, program location, and ACGME ID. We obtained match data for the same years from AMIA, including applicant name, ERAS ID, program name, and ACGME ID. We de-identified the applicant name and ERAS ID using a random number generator prior to analysis.

Analysis

We analyzed the data using descriptive statistics. Due to non-normal distribution, we used the Wilcoxon signed-rank test to compare medians with p < 0.05 considered significant. We used the Fisher's exact test to compare outcomes for programs with constraints to those without constraints. We used Microsoft Excel 2016 and RStudio 2022.12.0 + 353 to conduct the data analysis. To understand the geographic distribution of CI applicants, programs, and fellows without divulging program-specific data, we grouped results by United States Census region (Midwest, Northeast, South, and West).

Ethics

The Boston Children's Hospital Institutional Review Board deemed our study exempt from human subject review.

Results

Applications

We identified 159 unique applicants over the 2-year period (2021: n = 76; 2022: n = 83) who submitted 2,178 applications (2021: n = 900; 2022: n = 1,278). [Fig. 1] shows the distribution of applicants, programs, and matches by region. The median number of applications per applicant was 10, with an interquartile range (IQR) of 3 to 20 (2021: median = 6, IQR 2–14; 2022: median = 11, IQR 5–24.5). There were 44 applicants who applied to three or fewer programs (2021: n = 28; 2022: n = 16) and 14 who applied to over 40 programs (2021: n = 5; 2022: n = 9). In general, applicants who ultimately matched successfully applied to more programs than those who did not ([Fig. 2A], matched median = 14 programs, IQR 8–31.75 vs. unmatched median = 4 programs, IQR 1–9; p < 0.001). However, even applicants who applied to three or fewer programs matched 27% of the time (2021: n = 7, 25%; 2022: n = 5, 31%).

Fig. 1 Distribution of Clinical Informatics (A) programs, (B) applicants, and (C) matches by United States Census region.

Fig. 2 (A) Box plots showing the number of applications submitted by applicants who ultimately matched compared with those who went unmatched. (B) Box plots showing the number of applications received by programs that ultimately matched at least one candidate versus programs that went unmatched.

Forty-four programs received applications through ERAS in 2021 and 49 in 2022. Programs received a median of 23 applications per year with an IQR 16 to 30 (2021: median = 20, IQR 14.75–25; 2022: median = 26, IQR 17–35). Programs that matched at least one applicant received more applications than those that went unmatched ([Fig. 2B], matched median = 26 applications, IQR 18–35 vs. unmatched median = 18 applications, IQR 14–23.75; p < 0.001). Of the 39 programs that participated in both years, all but four saw an increase in the number of applications.

Match Participation and Ranking

Applicants and programs participated in the match through a secure website hosted by AMIA. In both years combined, 104 applicants (65.4%) participated in the AMIA match (2021: n = 48, 63.2%; 2022: n = 56, 67.5%). There were 42 participating programs in 2021 offering a total of 69 positions (mean 1.6 positions per program) and 47 programs in 2022 offering 84 positions (mean 1.8 positions per program). Applicants ranked a median of seven programs IQR 2 to 12 (2021: median = 5, IQR = 2–12; 2022: median = 7, IQR = 3–12.25). There were 18 (17.3%) applicants who ranked only 1 program (2021: n = 10, 20.8%; 2022: n = 8, 14.3%) and 29 (27.9%) who ranked over 10 (2021: n = 13, 27.1% ; 2022: n = 16, 28.6%). Of applicants who ranked three or fewer programs, 78.4% matched, compared with 98.5% of applicants who ranked more than three programs (2021: <3 = 78.9% vs. >3 = 96.6%; 2022: ≤3 = 77.8% vs. >3 = 100%).

Programs ranked a median of eight applicants per year IQR 5 to 11 (2021: median = 7, IQR 5–10; 2022: median = 9, IQR 5.5–12). Programs that matched at least one applicant ranked more applicants compared with those that went unmatched (matched median = 9 applicants, IQR 6–12 vs. unmatched median = 5, IQR 2.5–8; p < 0.01). In 2021, there were seven programs (16%) that entered constraints (e.g., primary specialty of the applicant must be Pediatrics) into the match system, and in 2022 there were nine (18%). All constraints were handled correctly and honored by the match algorithm. Programs with constraints were as likely to match at least one applicant compared with those without any constraints (Fisher's exact test, p = 0.2445).

Match Outcomes

Once all applicants and programs submitted their rank order lists, the match algorithm was run and validated manually by an independent source. There were no deadlocks or delays, and match results were made available to applicants and programs via the secure website on the agreed upon date and time. Of applicants who participated in the match, 95 (91.3%) were successfully paired with a program (2021: n = 43, 89.6%; 2022: n = 52, 92.9%). Of applicants who matched, 66 (69.5%) received their top choice (2021: n = 29, 67.4%; 2022: n = 37, 71.2%). Of the participating programs, 65 (73%) matched at least one candidate (2021: n = 32, 76.2%; 2022: n = 33, 70.2%) and 51 (57.3%) filled all available slots (2021: n = 24, 57.1%; 2022: n = 24, 51.1%). [Fig. 3] shows the frequency of a given ranked choice match for applicants ([Fig. 3A]) and programs ([Fig. 3B]). There were 58 positions (38%) that remained unfilled (2021: n = 26, 38%; 2022: n = 32, 38%) at the conclusion of the match.

Fig. 3 (A) Histogram showing the frequency of applicants matching with their first, second, third, etc. choice. (B) Histogram showing the frequency of programs matching with their first, second, third, etc. choice. Programs filling more than one slot by definition had to select at least one candidate beyond the first choice (n slots = nth choice).

Training Distribution

[Fig. 1] shows the distribution of CI programs, ERAS applications, and AMIA matches by geographical region. The total number of programs, applications, and matches increased modestly from 2021 to 2022, with all United States census regions seeing an increase in applications. Most CI programs reside in the Midwest and Northeast, however, the West received the most applications and matched the most fellows. California leads all other states in number of programs, applications, and successful matches. The South has the fewest programs, applications, and matches. Notably, more than half of states do not have a CI program participating in ERAS or the AMIA match.

Discussion

This is the first manuscript to describe the design and examine the results of the AMIA match program for CI fellowships. Our research adds to existing literature on CI fellowship applicants and programs.

American Medical Informatics Association Match Results

The AMIA match algorithm was adapted from the Gale–Shapely algorithm and designed to be applicant-favoring. Our data demonstrated that the match was extremely successful for applicants, who chose to participate, with nearly all candidates finding a match and with more than two-thirds of matched applicants matching with their top choice. Therefore, we conclude that the match algorithm met its intended purpose. Applicants, who applied to and ranked more programs, were more likely to match. However, even applicants who applied to three or fewer programs still matched successfully about a quarter of the time.

The number of applications submitted per applicant varied widely as did the number received by each program. However, given that there were fewer match participants than available positions, a significant number of programs did not fill all available positions—particularly in 2022 when just over half of programs filled all their available slots. Programs that received more applications and ranked more applicants were more likely to match at least one applicant, although even several programs that received <10 applications found a match.

We found a good distribution of programs across U.S. Census regions, however more than half of states do not have a CI training program. Reassuringly, the number of applications grew across every census region from 2021 to 2022, and the number of matches either remained the same or increased.

Did the American Medical Informatics Association Match Address Known Challenges?

Compared with the simultaneous offer–acceptance process, the AMIA match addressed most of the key challenges identified by Pageler et al.[6] The old process could lead to candidates not matching with a preferred program, which was corrected with most fellows matching with their program of choice. Likewise, the process eliminated the problem with multiple programs being deadlocked on the top applicants. In addition, the AMIA match mitigated risks for applicants and programs alike. Previously, applicants may have been faced with the decision to accept an offer from one of their lesser ranked programs or to decline it in the hopes of another offer. Similarly, programs needed to consider the risk of their offer being declined given the risk of failing to match other applicants while awaiting a response. The AMIA Match appears to have also met the needs of programs regarding home department constraints as programs with constraints were no less likely to match than those without constraints. Unique challenges remain for pathology resident applicants, who typically apply for fellowship 2 years prior to their start date and often combine 1 year of CI with 1 year of a second fellowship with a CI emphasis, thus far the flexibility of the AMIA match has allowed CI fellowships with a preference for pathologists to participate as needed. In terms of preventing match manipulation, AMIA added a disclaimer for the 2022 match to remind applicants and programs that they were required to abide by the results of the match. However, program directors are aware of several instances of participants not abiding by the match results or of applicants withdrawing from the match but then matriculating with a CI fellowship.

Opportunities for Improvement

Based on prior publications, discussing the CI applicant pool[12] [13] and data from ERAS and AMIA, we observed trends in the CI fellowship application life cycle over time ([Fig. 4]). Because the simultaneous offer–acceptance process from 2016 to 2021 had no registration process, the number of participants is unknown. In addition, the authors did not have access to ERAS data on the total number of applicants from 2021.

Fig. 4 Trends in Clinical Informatics fellowships, positions, applications, and matches over time. *Participants not required to register; no data available. ^¥ERAS applicant data unavailable for 2021. ERAS, Electronic Residency Application Service.

While the number of CI programs and positions continued to rise, the number of applicants plateaued in 2022. The CI community must continue to market the CI training programs to medical students, residents, and the general public to increase the pool of available applicants and train enough fellows to support the growing needs of the health care community. Based on regional data, programs could consider targeting medical schools and residencies in states where no CI fellowship program currently exists.

Only 65% of ERAS CI fellowship applicants participated in the AMIA match. One reason for this attrition is applicants accepting offers to CI fellowship outside of the match, after initially applying through ERAS. Based on publicly available data of currently employed CI fellows, we estimate that approximately 15 applicants over the past 2 years found positions outside of the match.

While almost all programs that accepted applications through ERAS participated in the match, they represent only approximately 80 to 85% of ACGME-accredited programs. There are good reasons that a program might accept applications through ERAS but not participate in the match, such as unexpected loss of funding. Other special circumstances include targeted recruitment of an internal applicant, who is not applying in any other program or recruiting an applicant pursuing combined training experiences in conjunction with another department. While it may seem reasonable to offer a position to an internal candidate outside of the match, it could put undue pressure on the trainee to accept that position as opposed to exploring other options. Similarly, applicants interested in dual training experiences in CI and another subspecialty face the challenge of having to decide which programs to apply to and whether to participate in the AMIA CI match, the clinical program's match (if there is one), both, and neither. For the AMIA Match to continue to meet the needs of applicants and programs it is critical to identify these special cases and create specific guidelines for applicants and programs to follow, which accommodate these needs. This may also include the need to delineate sanctions for those who deviate from the guidelines.

With respect to applicant attrition, there are other potential explanations as well. While applicants applying to many CI programs around the country are clearly committed to a career in informatics, those applying to only a few programs may be considering multiple career paths or have personal constraints. Some applicants may wish to explore CI through the application process but ultimately decide to pursue clinical training in another specialty or accept a job offer in lieu of additional training.

The extension of the practice pathway also undoubtedly contributed to applicant decisions not to participate in the match. As a debate at the 2022 AMIA Annual Symposium concluded, the majority of AMIA members believe the CI fellowship training program to be the preferred path toward certification and superior to the practice path.[14] Indeed, Turer et al[10] argue that the extension of the practice pathway allows candidates to opt for less rigorous training, diverting potential candidates away from CI fellowship training. This further underscores the need for the CI community to market the benefits of a CI fellowship training to applicants and the public at large and the need to end the practice pathway.

Limitations

Our study has several limitations, most notably the lack of data on the interview process, including how many interview offers each applicant received and how many they chose to attend. This likely is a critical factor in the decision of applicants and programs to rank one another in the match. In addition, we lack data on why applicants choose specific programs to apply to in the first place and what factors might influence that decision. Although we were able to estimate the number of applicants matriculating to CI programs outside of the match based on publicly available data, we do not have complete data on what happened to all CI applicants who did not participate. Access to this information could help further identify marketing opportunities. Lastly, because we focused on ACGME-accredited training programs, we may be missing applicants who applied to nonaccredited programs as well as those who applied to accredited programs but ultimately matriculated to a nonaccredited program. Though likely a small number of individuals, this limitation could be addressed in the future by identifying newly board-certified CI physicians and comparing with the list of graduates from accredited fellowships, especially once the practice pathway has closed.

Conclusion

CI applicants had great success in the AMIA Match. The combination of a plateauing number of CI applicants, applicant attrition from the match and prioritization of applicant ranking over program ranking resulted in many programs not matching a fellow or going unfilled. The CI community should support efforts to close the practice pathway and must continue work to market training programs to potential applicants and to develop guidelines to ensure a fair and transparent matching process.

Clinical Relevance Statement

Our findings demonstrate the AMIA match program functions as intended and add to the existing literature on CI fellowship programs. Trends in program growth, applications and matches over time suggest the CIPD community should support efforts to close the practice pathway and market CI training to potential applicants to increase the number of match participants and fill available training slots.

Multiple Choice Questions

1. Which of these statements is correct regarding CI fellowship regional training distribution?

   • a) The West has the most CI fellowship programs applicants and matches.

   • b) All U.S. states have at least one training program.

   • c) The South has the fewest programs, applicants, and matches.

   • d) The number of applications to Midwestern programs dropped from 2021 to 2022.

   Correct Answer: The correct answer is option c. CI fellowships are spread across all U.S. Census regions; however, the South has the fewest number of programs, applicants, and matches. Although California has the most programs, applications, and matches, the West overall has fewer programs than the Midwest and Northeast. The number of applications grew from 2021 to 2022 in each census region but more than half of states have no CI fellowship program at all.

2. What is the reason for unmatched CI fellowship programs in the match?

   • a) Not enough applicants

   • b) Not enough match participants

   • c) Applicants ranking a small number of high-quality programs

   • d) Programs entering too many constraints

   Correct Answer: The correct answer is option b. Initially, there were more applicants than available fellowship slots. Over time the number of available CI fellowship slots has caught up to the number of fellowship applicants, however, not all the applicants participate in the match. As a result, there is a surplus of available slots compared with match participants. If applicants were ranking a small number of high-quality programs you would expect to see more unmatched applicants, which was not the case. Regardless of the number of constraints, if there are not enough match participants there will be unfilled programs.

Conflict of Interest

All authors are current or past members of the Clinical Informatics Program Directors Match Subcommittee.

J.D.H. chairs the Match Subcommittee.

M.L. is the past chair of the Match Subcommittee and helped develop the AMIA match algorithm.

Acknowledgments

Many thanks to RaeLynn Gochnauer, Michelle Martin, James Keagy, and Samuel Yang for their contributions to developing and maintaining the AMIA Match process.

Human Subject Protections

The Boston Children's Hospital Institutional Review Boards deemed our study exempt from human subject review.

• References

• 1 Detmer DE, Munger BS, Lehmann CU. Clinical informatics board certification: history, current status, and predicted impact on the clinical informatics workforce. Appl Clin Inform 2010; 1 (01) 11-18
• 2 Safran C, Shabot MM, Munger BS. et al; AMIA Board of Directors. Program requirements for fellowship education in the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; 16 (02) 158-166
• 3 American Medical Informatics Association. Clinical Informatics Becomes a Board-certified Medical Subspecialty Following ABMS Vote. 2011. Accessed October 9, 2023 at: https://amia.org/news-publications/clinical-informatics-becomes-board-certified-medical-subspecialty-following-abms
• 4 Longhurst CA, Pageler NM, Palma JP. et al. Early experiences of accredited clinical informatics fellowships. J Am Med Inform Assoc 2016; 23 (04) 829-834
• 5 Lehmann CU, Gundlapalli AV, Williamson JJ. et al. Five years of clinical informatics board certification for physicians in the United States of America. Yearb Med Inform 2018; 27 (01) 237-242
• 6 Pageler NM, Elkin PL, Kannry J, Leu MG, Levy B, Lehmann CU. a clinical informatics program directors' proposal to the American Board of Preventive Medicine. Appl Clin Inform 2020; 11 (03) 483-486
• 7 Gale D, Shapley LS. College admissions and the stability of marriage. Am Math Mon 1962; 69 (01) 9
• 8 Wu Y, Taravati P, Yanagihara RT. et al. Inefficiencies in residency matching associated with Gale-Shapley algorithms. J Acad Ophthalmol 2021; 13 (02) e175-e182
• 9 American Board of Preventive Medicine. The American Board of Preventive Medicine Announces a Three-Year Extension of the Practice Pathway for Clinical Informatics 2022. Accessed March 9, 2023 at: https://www.theabpm.org/2022/03/02/the-american-board-of-preventive-medicine-announces-a-three-year-extension-of-the-practice-pathway-for-clinical-informatics/
• 10 Turer RW, Levy BP, Hron JD. et al. An open letter arguing for closure of the practice pathway for clinical informatics medical subspecialty certification. Appl Clin Inform 2022; 13 (01) 301-303
• 11 AAMC. ERAS Statistics. Accessed March 9, 2023 at: https://www.aamc.org/data-reports/interactive-data/eras-statistics-data
• 12 Bell DS, Baldwin K, Bell EJ. et al. Characteristics of the National Applicant Pool for Clinical Informatics Fellowships (2016–2017). AMIA Annu Symp Proc 2018; 2018: 225-231 Accessed August 15, 2019 at: http://www.ncbi.nlm.nih.gov/pubmed/30815060
• 13 Bell DS, Baldwin K, Bell III EJ. Characteristics of the National Applicant Pool for Clinical Informatics Fellowships (2018–2020). AMIA Annu Symp Proc 2023; 2022: 241-248
• 14 Kim E, Lehmann C, Hersh W, Fuller C, Levy B. A debate on the extension of the Practice Pathway for ABMS clinical informatics board certification for physicians in the United States. Paper presented at: American Medical Informatics Association Annual Symposium; November 8, 2022; Washington DC.

Publication History

Received: 14 August 2023

Accepted: 12 October 2023

Article published online:
13 December 2023