On July 1, 2023, Family Medicine residencies entered the new world of competency-based residency education. The new residency standards1 greatly reduce the number of month or hour based requirements and, instead, expect residencies to assess competence explicitly across a wide variety of clinical areas. In parallel, the ABFM announced its requirements for competency based board eligibility in June.2 Starting in June 2024, program directors, supported by their clinical competency committees (CCCs), will need to attest that residents are competent in specific core outcomes; this requirement will be phased in over 3 years to include all core outcomes.3
This is a big lift for residencies, program directors, faculty, and residents—and it will require new thinking about assessment, faculty development and a growth mindset among both residents and faculty. The Society of Teachers of Family Medicine (STFM), Association of Family Medicine Residency Directors (AFMRD), and many other organizations are developing and rolling out a comprehensive strategy to support residency redesign and the shift to competency-based residency education. An important first question, however, is what assessments are family medicine residencies currently using? Going into the biggest changes in residency education since our founding, what is our baseline?
To answer this question, ABFM used a confidential short survey of program directors as they participated in the attestation process in the summer of 2023. Traditionally, family medicine residency directors have used an online process to attest that each individual resident has completed residency and is ready for autonomous practice. This is a key component of ABFM Board eligibility. This summer, we added a short survey in which we asked what kinds of assessments were being used currently in a “checkbox” list. We also asked program directors how many assessments each of the graduates in 2023 had received over the course of their residencies. Residency Directors were told that individual residency data were confidential and would not be used for certification of individuals nor shared with the Review Committee.
We obtained responses from 652 (92%) of residencies in our files. The remaining 8% of residencies are new residencies without any graduates and or perhaps a few whom had no residents seeking ABFM board certification. Figure 1 depicts the types of assessments used by more than half of family medicine residencies. Not unexpectedly, the in-training examination, rotation evaluations, procedure assessments and resident self-assessments are used in most residencies. Interestingly, 446 of residencies are also using individual patient satisfaction reports and peer assessments.
Types of assessments used by more than half (>50%) of family medicine residencies.
Going forward, we believe that all residencies should consider using the assessments listed in Figure 1. In addition, optimizing rotation evaluations by anchoring them in specific observable clinical behaviors and emphasize entrustment for independent clinical practice could improve resident learning. This emphasis on specificity of evaluations to practice competencies would require residency faculty to decide what the core competencies or core outcomes are for specific rotations. For example, an inpatient medical rotation evaluation might assess entrustability for key components of inpatient care such as initial assessment of acutely ill patients, management of multiple clinical problems simultaneously over a hospitalization, effective transition of care, communications with patients, nurses, team members and consultants and professionalism and trustworthiness. A well-designed rotation evaluation can summarize many discrete direct observations and/or multi-source feedback from many team members, and a global judgment of trustworthiness over a sustained and intense period of observation of 2 to 4 weeks.
Figure 2 depicts assessments that are used in 25 to 50% of family medicine residencies—that is, many but not most residencies. These include individual quality and safety reports and care continuity reports. Of note, many residencies are incorporating OSCEs, AV reviews, and simulations. Surprisingly, at a time in which many residencies are under fiscal pressure and for which a major goal of the new standards is the “practice as the curriculum”, feedback on billing and coding is done in only 48% of residencies.
Types of assessments that are used in 25 to 50% of family medicine residencies. Abbreviation: OSCE, objective structured clinical examination.
More broadly, these data suggest that many residencies have learned how to incorporate these additional assessments into their ongoing work. This is an opportunity for dialog and evaluation by our specialty—what assessments are valuable and most time effective? In this context, the commitment of AFMRD and STFM to feature Competency Based Medical Education (CBME) in their spring meetings will be a key support for faculty development in our specialty. In addition, emerging residency learning networks1,4 could provide an ideal context for residencies to learn about competency assessment together over time.
Figure 3 depicts the assessments that <25% of residencies use—and may represent experimental assessments by innovators in the specialty. We need to learn from these innovators! Many of the assessments are focused on defining and assessing important and higher standards of practice for family physicians—for example, effective use of medications, efficiency in patient care, access to care, and appropriateness of referrals. These residencies are trying to capture and measure what the ACGME and ABFM have defined as core outcomes of excellence as a personal physician,3 a core outcome for which ABFM has phased in requirements over all 3 years. Although less than 25% of residencies report using these assessments now, there are more than 50 examples for each—enough for our community to learn from!
Types of assessments that are <25% of residencies use.
How many written assessments should a resident receive over their residency? If there is 1 per rotation month, the low end would be 36 for a 3-year residency, assuming all them are completed. At the other end, experience from Canada suggests that many more written assessments are necessary, perhaps 1 per week or even 1 per day. Figure 4 suggests that the typical US resident graduating in 2023 had received between 50 and 75 by the end of residency. Interestingly, there are a few residencies in the US that routinely give many more assessments than the median. What can we learn from them? What the right number is remains unclear, but more assessments have the potential to drive more learning and may add validity to the overall summative judgment of trustability. Of course, much feedback given to residents is oral, but the act of writing and the habit of specificity requires intentionality and supports long-term reflection and learning. The burden of documentation is also an important barrier. An option to decrease the burden of higher number of evaluations is a digital app with a user-friendly interface. For example, the M3 app5,6 in Family Medicine and the SIMPL app7 used in Surgery and soon in Pediatrics, need to be available. Ideally these assessments would be contained within a Learning Management System that would allow effective summarization of trends and prevent the extra work of double entry. The system would also need to provide analytics that allow effective summarization of trends. The STFM Task Force on Competency Based Education is leading the charge toward implementation of similar applications.
Number of assessments at 2023 graduation.
Of course, assessments are only part of the overall picture of tracking the development of residents. Faculty must have a shared mental model8 about what specific components of the core outcomes are most important, as well as the faculty development to deliver assessments effectively and equitably. The Clinical Competency Committee role9 is also critical, not only in assessment of milestones and developing a summative assessment of competence in the core outcomes, but also in reviewing the sampling of assessments and assuring that assessments capture a broad scope of practice and across the continuum of care. This function will be critical to the validity of the CCC’s assessment of competency over all outcomes at the end of residency.
More broadly, faculty roles are evolving. When faculty precept in clinic or attend on the hospital service, a core component of what they do should be to produce assessments. This role is in contrast to coaching, in which a faculty member reviews assessments with the resident regularly (eg, quarterly) and identifies gaps and needed clinical experiences, while also mentoring around career development. Residency faculty time for education is another potential barrier, and it is critical that residencies measure faculty time carefully and take advantage of ACGME’s restoration of pre-2019 standards for time required for education for core faculty.
Taken together, these data represent an almost complete census of assessment practices in Family Medicine residencies. Of note, although there is no independent assessment of how well or how consistently the assessments are being done, and although residency directors may be biased in what they have reported, this is likely to represent the best available baseline data for our specialty. We are hopeful that the rigor and frequency of assessments will increase as our faculty leaders adapt to the new world of competency assessment, as our community learns what is most effective, and as we work together to reduce the burden of documentation and review of competence.
ABFM thanks you all for your effort and creativity and welcomes the dialog in the future.
Notes
Conflict of interest: The authors are employees of the ABFM.
To see this article online, please go to: http://jabfm.org/content/37/1/155.full.
