Family Medicine Certification Longitudinal Assessment Becomes Permanent ======================================================================= * Warren P. Newton * Elizabeth Baxley * Thomas O'Neill * Kevin Rode * Roger Fain * Keith Stelter On April 29, 2021, the American Board of Medical Specialties (ABMS) Committee on Continuing Certification approved the American Board of Family Medicine's (ABFM's) proposal to make the Family Medicine Certification Longitudinal Assessment (FMCLA) a permanent alternative to the 1-day examination for summative assessment of cognitive expertise in family medicine. We anticipate that the ABMS Board of Directors will formally approve this in June. This editorial lays out what we have learned about FMCLA and what our next steps will be. ## FMCLA—a Description FMCLA was conceived as an alternative for ABFM Diplomates to meet their cognitive expertise examination requirement. While serving a primarily summative role in assessing a Diplomate's cognitive expertise, the design was purposefully selected because it is also formative, consistent with what we know about best practices in adult learning and retention. ABFM launched a 2-year pilot on January 1, 2019, patterned after the earlier work of the American Board of Anesthesiology and American Board of Pediatrics. FMCLA provides participants 25 examination questions each quarter, which can be completed at any time, in any location, with 5 minutes per question and access to references. Three hundred completed questions are required for a pass/fail decision; up to 4 years is allowed, which allows for participants to delay 1 to 4 quarters or to answer fewer than 25 questions in a given quarter. From the beginning, FMCLA has been very popular with Diplomates. More than 73% of eligible Diplomates selected FMCLA in 2019, 2020, and 2021, with nearly 100% retention rate (98% and 99.3% for the first 2 cohorts). Family physicians participating in the pilot are very similar to those taking the traditional 1-day examination, in demographic terms of gender, degree, scope, and type of practices; however, they tend to be slightly younger and have slightly lower test scores on their most recent examination. With more than 11,000 continuously participating, evaluations of the information technology platform, ease of use, and tracking progress are all more than 95% positive, and 96% reported less anxiety and a preference for FMCLA over the 1-day examination. The average time to complete each question has been 2 minutes, 21 seconds. ## Summative Assessment As we have described in prior editorials,1⇓–3 the most important function of FMCLA is to assure the public that ABFM Diplomates have the cognitive expertise necessary to be board certified—a summative function. Given the strong evidence that physicians are ineffective at assessing their own knowledge, independent assessment is necessary.4,5 Our traditional 1-day examination meets rigorous psychometric standards and includes ongoing assessment for bias with respect to race and ethnicity. Thus, the most important question for ABFM has been whether the new format—longitudinal, with access to references—is similarly rigorous. The examination blueprint and the questions themselves are similar to the 1-day examination, and we follow the same procedures to pretest questions, score the exams, and assess for bias. What is different is the more flexible delivery, more time per question, and access to references, leading to 1 of the most important questions to be answered by the pilot: are the 2 processes comparable psychometrically in assessing the cognitive expertise of family physicians? To meet this standard, item hierarchy—that is, the relative difficulty of individual questions compared with other questions, must be maintained. We know now that FMCLA generally preserves item hierarchy. With references and more time, some items get easier, and a few get harder, but our routine quality control procedures are sufficient to identify “drifting” items and adjust their difficulty to bring them back in line. We have learned that the framing of the question makes some questions easier in the longitudinal format. If the clinical stem gives the diagnosis, or names a specific clinical guideline, the question is easier to look up in real time. Adjustment of item writing will be necessary going forward. A second test of summative validity is whether FMCLA performs similarly in assessing physicians. We compared estimates from the approximately 9000 Diplomates of the first 2 FMCLA cohorts and estimated how their passing rate would compare with the 1-day examination. The resulting estimates—90.5% and 90.9%—are well within the historic range of performance of the 1-day examination. These estimates will become more precise as more questions are answered. Importantly, for the cohort with 2 years of experience, the passing rate increased somewhat to 93% as participants responded to the ongoing feedback. This is still in the historic range of the ABFM examination, and we consider this improvement over time to be one of the benefits of FMCLA. ## Formative Assessment FMCLA also supports Diplomate learning. Over the last generation, there has been increasing understanding that assessment itself is a powerful driver of learning,6 and one of our hopes for FMCLA is that the immediate feedback, critiques, and references provided after each question would more effectively contribute to ongoing learning and retention than the 1-day examination. The pilot provides ample evidence of this effect. Nearly two thirds of the participants in the first cohort reported using references when answering 11 or more items, with only 5% not looking at any reference. Importantly, 84% sought information after the test, and 82% indicated that they had changed their practice as the result of participating in FMCLA. Some 89% reported incorporating FMCLA into their approach to keeping up with medical knowledge, complementing formal continuing medical education (CME) and other types of learning. Our next area of development is to increase the effectiveness of learning through FMCLA. Traditionally, with our 1-day examination, we have given general feedback that comes with the final score report and is organized around an organ system-based blueprint. There is evidence that the process of preparing for the 1-day examination, including CME courses, self-study, and the feedback provided, does support learning.7 Yet, feedback on gaps in knowledge in a specific organ system is imprecise statistically, because the test is powered to have maximum precision about the pass/fail judgment. To address this problem, we have begun to identify testing points for each question and provided participants with this information. We have also begun to give candidates an indication of how they are doing compared with the cut score for passing the examination at the end of the first 100 items. Over the longer term, we have started to develop a new blueprint that will be organized around Diplomates' clinical activities, which will allow more effective feedback and a better handoff to CME activities developed by the American Academy of Family Physicians and other partners. We believe it is crucially important for physicians to learn more self-awareness of their confidence in responding to a particular clinical scenario and use that feedback to learn more about what they believe they know but in fact do not know. FMCLA participants indicate whether the subject of each question is relevant to their practice and whether they feel confident in the answer. We are piloting providing individualized feedback for participants, in which they learn the testing points of the questions they reported as relevant to their practice or areas that they are confident in—but answered incorrectly. Early feedback on these new evaluation forms has been extremely positive, though uptake to date has been low. In summary, our pilot with more than 11,000 Diplomates has demonstrated strong support of self-reported learning. Support of learning will become a key part of ABFM's strategy for the future, along with our current revisions of our Knowledge Self-Assessment activities, the new Journal Club activity, and enhanced linkages to formal CME activities. ## Next Steps ABFM will now incorporate FMCLA into our permanent certification portfolio. The mechanics of the assessment will be relatively unchanged from the pilot—25 questions a quarter, 5 minutes per question, use of references, and 300 items needed to score. We are currently considering options to expand eligibility for using FMCLA and to create greater flexibility within the first year of participation. More fundamental will be changes in our item-writing process. Currently our multiple-choice questions are developed by family physicians and edited by our medical editors, are reviewed by 2 committees of practicing family physicians, and then tested extensively in other testing environments. We have added a rereview by 2 family physicians before final deployment and asked item writers to reduce the number of questions that are easy lookups. The new draft ABMS Standards for Continuing Certification8 call for knowledge assessments that provide both formative and summative functions, cycles of assessment no less frequent than 5 years apart, and an opportunity for remediation before a consequential decision is made. FMCLA was designed with those requirements in mind and will play a major role in meeting a number of the new ABMS Standards. Importantly, we do still plan to continue to offer the 1-day examination. Despite the enthusiastic response to FMCLA, up to 24% of Diplomates have chosen the 1-day examination; if the rest of our Diplomates make similar choices, this represents about 24,000 Diplomates! There are opportunities to more fully integrate different aspects of certification—also a goal of the new ABMS Standards. We will be considering how to allow requirements for 2 or more requirements to be met for 2 or more activities in 1 setting. For now, we commit to doing everything we can to assure relevance, value, and lower burden for all ABFM activities and work to support Diplomates and eligible family physicians on their continuous certification journey. We look forward to your continued feedback as we design the future. ## Notes * *Conflict of interest:* The authors are employees of the American Board of Family Medicine. * To see this article online, please go to: [http://jabfm.org/content/34/4/879.full](http://jabfm.org/content/34/4/879.full). ## References 1. 1.O'Neill C, Newton W, Brady J, Spogen D. Using the Family Medicine Certification Longitudinal Assessment to make summative decisions. J Am Board Fam Med 2019;32:951–3. [FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMzIvNi85NTEiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzM0LzQvODc5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 2. 2.Newton W, Rode K, O'Neill C, Fain R, Baxley EG. Longitudinal assessment: where we are and why it is important. J Am Board Fam Med 2019;32:448–50. [FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMzIvMy80NDgiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzM0LzQvODc5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 3. 3.Newton WP, Rode K, O'Neill T, Fain R, Baxley E, Peterson L. Family Medicine Certification Longitudinal Assessment after one year. J Am Board Fam Med 2020;33:344–6. [FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMzMvMi8zNDQiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzM0LzQvODc5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 4. 4.Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094–102. [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1001/jama.296.9.1094&link_type=DOI) [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=16954489&link_type=MED&atom=%2Fjabfp%2F34%2F4%2F879.atom) [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000240298400011&link_type=ISI) 5. 5.Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med 1991;66:762–9. [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1097/00001888-199112000-00012&link_type=DOI) [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=1750956&link_type=MED&atom=%2Fjabfp%2F34%2F4%2F879.atom) [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=A1991GW93000013&link_type=ISI) 6. 6.Zakrajsek T, Newton W. Promoting active learning in residency didactic sessions. Fam Med. In press. 7. 7.Chesluk B, Gray B, Eden A, Hansen E, Lynn L, Peterson L. "That was pretty powerful": a qualitative study of what physicians learn when preparing for their maintenance-of-certification exams. J Gen Intern Med 2019;34:1790–6. 8. 8.Draft Standards for Continuing Certification —Call for Comments. 2021 [cited 2021, June 9]. Available from: [https://www.abms.org/news-events/draft-standards-for-continuing-certification-call-for-comments/](https://www.abms.org/news-events/draft-standards-for-continuing-certification-call-for-comments/).