Добавил:
kiopkiopkiop18@yandex.ru t.me/Prokururor I Вовсе не секретарь, но почту проверяю Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз: Предмет: Файл:
Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_2725_Библиотеки_им_академика_М_И_Перельмана.pdf
Скачиваний:
0
Добавлен:
31.08.2026
Размер:
31 Мб
Скачать
88
D. Keister and V. Brohm
that are characteristic of CBME.These include focusing on outcomes, emphasizing abilities, de-emphasizing time-based training, and promoting greater learner-centeredness (Table11.2). Education that follows these principles ensures that every graduate is prepared for practice.
Expanding on these key concepts, Van Melle convened a group of experts using a Delphi method to create a CBME Core Components Framework to describe the ideal implemen­tation for CBME. In traditional medical education, the pro­gram structure is the primary focus with the learning, assessment, and outcomes seen as necessary products of an ideal structure. The Van Melle framework centers clearly artic­ulated outcome competencies that are described as a sequenced progression in a triangle that both create and are inuenced by
Table 11.2 Key principles that are characteristic of CBME. (From Frank etal. [26])
Main principle Elaboration
Focusing on outcomes
In an era of greater public accountability, medical curricula must ensure that all graduates are competent in all essential domains
Emphasizing abilities
Medical curricula must emphasize the abilities to be acquired There is too much emphasis on knowledge and not enough on skills,
De-emphasizing time-based training
Medical education can shift from a focus on the time a learner spends on an educational unit to a focus on the learning actually attained
Promoting greater learner-centeredness
Medical education can promote greater learner engagement in training A curriculum of competencies provides clear goals for learners
three other factors: (1) Programmatic assessment that “sup- port and document the developmental acquisition of compe­tencies,” (2) Tailored learning experiences that “facilitate the developmental acquisition of competencies,” and (3) Competency-focused instruction that “promote(s) the develop­mental acquisition of skills.” Van Melle describes how each component might appear in practice, the principle behind the components, and the conceptual frameworks from educational theory that support them in an excellent resource for readers who care to investigate this topic in more detail (2019).
The ultimate goal of CBME is to assure the development of competent physicians who are master adaptive learners [16] who will maintain their expertise throughout their careers.
Not all current curricula explicitly dene desired outcomes Not all current curricula address all of the desired outcomes Not all current curricula assess or ensure that graduates have acquired all of the necessary abilities In the health professions, assessment scores should not be compensatory from one domain to another (i.e., excellent knowledge does not compensate for poor communication skills) Medical education needs to be transparent for learners, teachers, and the public with respect to its goals and effectiveness Standards must be criterion-oriented Medical education tends to emphasize process issues (e.g., instructional methods) over outcomes (e.g., graduate performance and satisfaction) Medical education must prepare trainees for practice Content that does not contribute to preparation for practice should be dropped
attitudes, and their synthesis into observable competencies An emphasis on the abilities of learners should be derived from the needs of those served by graduates (i.e., societal needs) Educational objectives as an organizing framework should be replaced with a hierarchy of competencies
Time is a resource to be tailored to the needs of teachers and learners Current curricula and credentialing tend to emphasize xed times spent in training Learners may progress at different rates and may achieve threshold competencies faster or slower than the average peer Greater emphasis should be placed on the developmental progression of abilities and on measures of performance Greater exibility may make some curricula more efcient and engaging
A roadmap of milestones provides a transparent path to achieve the competencies An individual learner can adjust their own learning using the milestones
11 Designing Assessment toMeet theChallenge ofCompetency-Based Medical Education
Laying theFoundation forCBME
The International Competency-Based Medical Education (ICBME) Collaborators created the fundamental principles of CBME at their 2013 summit. These three principles include: education must be based on the health needs of the populations served; the primary focus of education and training should be the desired outcomes for learners rather than the structure and process of the educational system; the formation of a physician should be seamless across the continuum of education, training, and practice [13]. The shift to CBME utilizing these principles will ensure learn­ers are assessed on their application of medical knowledge in alignment with their local population and system needs, which vary widely within countries and across the globe [13]. For there to be seamless transitions in the advance­ment of learners, all entities including UGME, GME, and board certifying bodies need to share a common image of a suitable physician, starting with the “end in mind” [51, 59]. Carraccio et al. describe the commitments required for medical educators to successfully implement CBME, including the commitments to: Teaching, assessing, and
role modeling the broad range of identied competencies; Supervision that balances patient safety with the profes­sional development of learners; Transparency with all stakeholders; Commitment to the empowerment of learn­ers; The effectiveness and efciency of assessment strate­gies and tools; Basing transition decisions on competence rather than time; Advancing CBME through workplace assessment, program evaluation, and research; Faculty development; and Collaboration [13].
Utilizing CBME requires medical educators to empower learners to play an active role in their education. These commitments correlate with key concepts from the Master Adaptive Learning (MAL) model, which promotes the resi­dents’ ability to recognize and address knowledge gaps through transparent and specic feedback from all stake­holders in resident education, including self-evaluation [15,
16]. The MAL model (Fig.11.1) contains four gears which
are essential activities for the learner: planning phase, learn­ing phase, assessing phase, and adjusting phase. During the planning phase, learners identify gaps in their knowledge/ skills/ attitudes, prioritize opportunities for learning, create a goal, and search for resources. Throughout the learning phase, residents employ active learning strategies and criti­cally appraise resources. In the assessment phase, the resi­dents implement what was learned or “try it out.” Next, they compare their unguided self-assessment with an informed self-assessment utilizing external feedback such as feedback from faculty, patients, staff, colleagues, and patient outcomes [16]. Lastly, in the adjustment phase, the learner incorporates what was learned into their practice.
Fig. 11.1 The master adaptive learner model. (From Cutrer etal. [15])
The gears on this model are powered by the coaching of medical educators (depicted as a rheostat) combined with the internal learner characteristic “batteries” of curiosity, moti­vation, mindset, and resilience [16]. These batteries depict a learner’s growth mindset through their desire for knowledge and willingness to engage despite any setbacks or obstacles that may occur. Although these characteristics are internal, they can be augmented or suppressed by any number of fac­tors in the learner’s life or in the overarching learning envi­ronment. One challenge for FM residents is that their learning environment is constantly changing (e.g., outpatient clinic, hospital rotations, off-service rotations, lecture hall, virtual space, etc.). A key component in MAL is for the learning environment to be safe. The ACGME Clinical Learning Environment Review (CLER) Program ensures appropriate environments by providing sponsoring institutions feedback on patient safety, health care quality, teaming, supervision, well-being, and professionalism.
The active participation in learning described in the MAL model is critical to CBME [30]. The classic model of medi-
89
90
D. Keister and V. Brohm
cal education curriculum design and assessment is a hierar­chical creation of curriculum implemented upon learners with minimal variation to address individual learner needs. In contrast, CBME requires the co-production of the curricu­lum and assessment between learners and instructors. The informed self-assessment required for learners to identify and plan their learning “requires that learners proactively seek out assessment from faculty and members of the health­care team, perform aspects of their own assessment such as clinical performance reviews or evidence-based practice and actively engage their assessment data for professional devel­opment as part of their own accountability.” Clinicians with this type of engagement in continual learning are essential to the improvement in healthcare quality that CBME promises.
Recommended steps in the planning of the CBME cur­riculum will be discussed in the remainder of this chapter. They include: (1) Identifying the abilities needed from grad-
uates; (2) Explicitly dening the required competency and their components; (3) Dening milestones for the competen­cies along a developmental path; (4) Selecting educational activities, experiences, and instructional methods; (5) Selecting assessment tools to measure progress along the milestones; and (6) Designing an outcomes evaluation of the program [26].
Clarifying Competencies, Milestones, EPAs, andCore Outcomes
Given the new terminology necessary for the implementa­tion of CBME, educators can get confused about the differ­ence between competencies, sub-competencies, milestones, EPAs and core outcomes. The ACGME Core Competencies, patient care; medical knowledge; practice-based learning and improvement; professionalism; interpersonal skills and communication; and systems-based practice are general characteristics of a learner that imply knowledge, skills and attitudes that a learner in any medical specialty must have to be successful in practice [10, 70]. Each discipline has parsed the core competencies into sub-competencies that describe the critical components of practice for that discipline (Family med milestones). For example, within the patient care com­petency in family medicine, the rst sub-competency is “Patient care 1: Care of the acutely ill patient” (Fig.11.2). Other patient care sub-competencies in family medicine include care for patients with chronic illnesses, health pro­motion, care of patients with undifferentiated signs and sys­tems, and procedural care. Within each sub-competency, each discipline has dened a series of milestones, which are observable descriptions of the knowledge, skills, attitudes, and other attributes of elements of a sub-competency orga-
Fig. 11.2 ACGME milestone sub-competency of patient care #1. (From Family Medicine Milestones Workgroup [20])
11 Designing Assessment toMeet theChallenge ofCompetency-Based Medical Education
91
nized in a developmental framework (Family med mile­stones). The sequenced progression of sub-competency acquisition within the milestones describes a common path to growth in learners of every discipline. Indeed, most chil­dren do crawl before they walk. However, outliers exist in normal development. Therefore, learners might acquire more advanced skills prior to demonstrating basic skills. Similarly, a normal learning curve might include a delay in a given sub­competency. The milestones detail a common path to skill acquisition. They are not a perfect description of the learning journey for every student.
In contrast, Entrustable Professional Activities (EPAs) are descriptions of discrete tasks or responsibilities that faculty empower a trainee to perform without supervision when they have demonstrated the necessary skill [23]. Achieving a level of ability to be trusted with an EPAs generally requires the demonstration of multiple milestones within several sub­competencies [71]. For example, one EPA described for fam­ily medicine is “Manage inpatient care, discharge planning, and transitions of care.” To be entrusted with this activity, a resident must demonstrate skill within the care of acutely ill patients, a sub-competency of patient care, as well as sub­competencies within medical knowledge, systems-based practice, professionalism, and interpersonal communication skills. Thus, as faculty members determine that they trust residents to perform a given EPA, achievement of milestones within the family medicine sub-competency grids can rea­sonably be documented.
Core outcomes can be understood in the context of CBME as being equivalent to EPAs. The ABFM uses “the term ‘core outcomes’ because ABFM research (in 2022) showed that only approximately 40% of family medicine program direc­tors reported they are using the term Entrustable Professional Activities (EPAs), and what they mean by EPAs varies greatly [51].” The term core outcome was felt to be more intuitively clear, and the ABFM hoped that the term would avoid confusion due to the multiple denitions of EPA already in use by program directors. However, the ABFM description of their core outcomes mirrors the denition of EPAs described within this chapter [51, 52].
Assessment: TheKey toCBME
Ultimately, the success of CBME lies in high-quality assess­ment. Accurate and timely feedback allows learners the for­mative information that encourages them to grow and gives the leaders the educational program the information they need to provide the learner with the appropriate resources to advance their training. Although most GME programs have begun to implement such assessments, there will need to be
substantial growth in the quality of assessments to deliver on the potential of CBME.
The theoretical model used most often in considering skill acquisition in medical learners was developed by Hubert and Stuart Dreyfus [10]. The model details ve stages of development: novice, advanced beginner, compe­tent, procient, and expert. The stages were originally used to consider the entire journey of medical education with a novice being considered a starting medical student and an expert being a physician in practice. More recently, the ACGME has found benets in applying the model to the stages of resident growth with a novice equating to mile­stone level 1 and an expert to milestone level 5 [18]. Regardless of the scope of the journey included within the stages, the Dreyfus model allows educators a framework to describe the development of skills with Novice learners thinking analytically with little skill in prioritizing infor­mation; Advanced Beginners having the ability to apply analytic and nonanalytic thinking for common problems; Competent learners being able to see the big picture but falling back to analytic thinking for complex problems; Procient learners developing the ability to extrapolate knowledge to novel situations and live with ambiguity, and Experts having skill with subtle variations even in complex problems [18].
Because the ACGME milestones are intended to offer a formative assessment of resident growth along the Dreyfus developmental trajectory, it is hoped that they might allow educators to identify a resident who is behind their peers. Holmboe etal. [33] investigated the use of the milestones to facilitate this process in residents from family medicine, internal medicine, and emergency medicine. Assuming the goal that graduating residents should achieve milestone level 4 (correlating to a Dreyfus level of procient) in all sub-com­petencies, Holmboe looked to nd a predictive level to sug­gest that a resident was unlikely to reach that goal. In their analysis, they found that residents who had not reached level 3 by the end of PGY2 were less likely to reach the goal of milestone level 4. However, the predictive probabilities across the 24 FM sub-competencies ranged from 32% to 67%. Similar ranges of predictive probability existed for the other specialties. Although the milestones have promise for this level of predictability, we clearly need to rene this pro­cess, either through further revision of the milestones or through improvement into the assessments that contribute to milestones assignment.
The ACGME Assessment Guidebook provides an excel­lent resource for competency-based assessment [32]. The Guidebook includes a review of commonly used assessment techniques, describing the pros and cons of each. It also includes suggestions for the best assessment methods to use
92
D. Keister and V. Brohm
for each ACGME competency. It is unclear how familiar FM residency program leaders are with the Guidebook or the degree to which it is used in developing assessment systems in FM residencies.
Building aSystem ofAssessment
A robust system of assessments across multiple settings is imperative for sufcient data and feedback in CBME.It is important to make mindful, intentional choices around the
Fig. 11.3 States of assessment. (From Norcini etal. [53])
combination of methods utilized for assessment [76]. Traditionally “too many assessment tools presumably designed for a competency-based approach are overly reduc­tionist (e.g., a tick box exercise) and there has been a general lack of a programmatic approach to assessment” [30]. Figure11.3 shows the various states of assessment, revealing the best coverage of content evaluation in a system of assessment.
The ACGME assessment guidebook has recommenda­tions for the necessary systems of assessment noted in Fig.11.4.
Fig. 11.4 ACGME recommendations for systems of assessment. (From Holmboe and Iobst [32])
11 Designing Assessment toMeet theChallenge ofCompetency-Based Medical Education
93
Educators must commit to a routine review of assessment strategies and faculty development of resident assessment to collect the most valid competency data. The goal is for all graduating family medicine residents to reach a level 4 on all milestones as this correlates with prociency in the Dreyfus model and readiness for unsupervised practice [14]. A vari­ety of assessment tools are discussed in detail in the ACGME Assessment Guidebook and descriptions of each tool, includ­ing the competencies assessed, validity, feasibility, accept­ability, and catalytic effect can be reviewed. Examples of assessments for each core competency are listed in Table11.3.
The creation of perfect assessments to evaluate every­thing a resident should know/demonstrate by the time of graduation is unrealistic [34]. In a 2023 Webinar discussion of the implementation of competency-based ABFM board eligibility Warren Newton stated, “Remember that the per­fect is the enemy of the good. Simple, practical, scalable are priorities” [52].
It is important to recognize the following assessment pearls for CBME: All assessments are samples; The higher
the stakes, the more samples are needed; Assess what is important, not just what is easy; All assessment involves judgment; Quantitative and qualitative methods complement one another; No single assessment tool can capture all aspects of clinical competence; Feedback is an essential ele-
ment of assessment; Assessment drives learning; Validity is the most important characteristic of assessment data; and Perfect assessment is an illusion [34].
Low-stake assessments have limited consequences to the learner regarding promotion, whereas high-stake assess­ments have signicant impact on advancement (Van Der Vleuten [77] t for purpose). While low-stake assessments require a lean number of samples, a high-stake assessment should have multiple assessments from diverse samples. Quantitative assessments can be considered straight forward and easily standardized but should be used in a complimen­tary way to qualitative methods. All assessment involves judgment, which is why it is imperative for regular faculty development. ACGME Core Competencies such as Professionalism or Interpersonal and Communication skills may feel difcult to assess, but necessary. When learners rec­ognize that all core competencies are equally important aspects of becoming a physician through assessment, it will drive their learning to become procient [34, 77].
The Role oftheClinical Competency Committee
The ACGME requires that all training programs have a Clinical Competency Committee (CCC) with the purpose
Table 11.3 Assessments for each ACGME core competency. (From Holmboe and Iobst [32])
Competency Competency-based assessment options Medical knowledge In-training exam
Faculty work-based assessments Chart stimulated recall, assessment of reasoning tool, and others
Patient care and procedural skills Work-based clinical assessment through direct observation of the
individual during care delivery Faculty and peer assessment Standardized assessments Simulation
Professionalism Informed self-assessment
Multisource feedback, such as a 360-degree evaluation Patient experience surveys
Interpersonal and communication skills Patient-reported feedback and experience surveys
Multisource feedback, such as a 360-degree evaluation, especially regarding interprofessional care
Practice-based learning and improvement Evaluation of knowledge, skills, and attitudes from participation in
systematic efforts to improve the quality, safety, or value of health care services Audit and feedback of the medical record Review of medical errors and patient safety events Evidence-based practice logs
Systems-based practice Feedback from multiple faculty evaluations regarding ability to
practice in a complex health care system Multisource feedback, such as a 360-degree evaluation, especially regarding interprofessional care Assessment of cost-conscious care
94
Fig. 11.5 The assessment system. (From Andolsek etal. [5])
D. Keister and V. Brohm
of creating a shared understanding of resident performance expectations and assessment [2]. A CCC must be appointed by the program director and, at a minimum, include three members of the program faculty, at least one of whom is a core faculty member (ACGME). Additional members must be faculty or other health professionals who have extensive contact and experience with the program’s residents (ACGME). According to ACGME, the literature suggests that a group of ve to seven diverse individuals is ideal; therefore, larger programs may need multiple CCCs.Depicted in Fig.11.5 is The Assessment System in which the CCC is depicted as the Data Synthesis Committee. The CCC must review all resident qualitative and quantitative assessments at least semi-annually to determine milestone progress and convey individualized educational recommendations to the program director and residents. In doing so, the CCC plays an integral role in CBME by ensuring that the appropriate combination of evaluation tools is utilized in competency determinations, while also identifying gaps in the curriculum and/or fac­ulty development regarding feedback. All twice- yearly residency milestones are submitted to the ACGME
ReviewCommittee for review on the individual and pro­gram levels.
Implementation ofCompetency-Based Assessment
Currently, medical education defaults to a block-based model where progression is dependent on completion. Implementing CBME will require a more exible approach to reect the individualized learning curves of residents. Additionally, mov­ing to a longitudinal curriculum/rotation schedule will allow teaching continuity between trainees and faculty, enhancing coaching relationships and allowing for more robust feedback on progression [56]. Instead of competence being a static view based on time, Frank etal. [26] propose modiers that specify domains of ability, context, and stage of practice (Fig.11.6).
Timing of assessments should reect learning trajectories as knowledge, skills, and behaviors evolve over time [53]. Utilizing a system such as EPAs follows the development of competence while allowing learners to move at their own pace [53].
11 Designing Assessment toMeet theChallenge ofCompetency-Based Medical Education
Fig. 11.6 Proposed denitions of CBME and related terms by the ICBME collaborators. (From Frank etal. [26])
95
Informed Self-Assessment
Epstein et al. [19] acknowledge that the “The power of self- assessment lies in two major domains—the integra­tion of high-quality external and internal data to assess current performance and promote future learning, and the capacity for ongoing self-monitoring during everyday clinical practice.” It is imperative that learner’s informed self-assessment is fed by clear, timely, specic, construc­tive feedback, preferably offered by trusted, credible supervisors in a safe environment [63]. Additionally, a time for critical and deliberate reection on these assess­ments is a key activity [63], as external feedback that is inconsistent with self-assessment may be discounted. This is because one’s performance is integrally linked to one’s sense of self [63]. Guidance and facilitation will enhance this process of reection and the R2C2 Facilitated Feedback Model can be used to do so through building rapport and relationship, exploring reactions to perfor­mance data, exploring understanding of the content, and coaching for performance change. The ARCH model can also be used to provide effective feedback to learners: Allow/Ask for self- assessment, Reinforce what is being done well (Knowledge, Skills, and Attitude), Conrm what needs correction or improvement, and Help the learner with an action plant for improvement and coach as needed [9].
Faculty Development
The success of CBME requires that learners receive accurate feedback. Therefore, it is important that faculty are trained properly and supported to deliver high-quality formative and summative assessments (Nousiainen). Validity is the most important characteristic of assessment data [34] and characterizes the extent to which a measurement is sound/ correct. The validity of non-standardized assessment resides in the users rather than the instrument [77]. Because the use of non-standardized assessment is needed to evaluate Dreyfus prociency and expertise levels, it is imperative that investment in faculty education occurs. This is particularly important for community faculty members who are used in many FM residencies, but who have minimal formal training in teaching or assessment skills [64].
The Critical Question: What Happens When aLearner Does Not Meet Core Outcomes?
The most vexing question for residency programs is what to do with a learner who fails to meet the competency-based requirements. The answer is obvious: if a residency program director (PD) cannot attest that a resident meets the ABFM core outcomes, the learner should not graduate from FM residency [51]. However, the ramications of this decision
96
D. Keister and V. Brohm
are massive for both the learner and the program. Despite the difculties, PDs must have the courage to uphold the social contract that is inherent in their position as the nal arbiters of which physicians are eligible for board certication [59]. Fortunately, the development of robust systems of assess­ment will make this task easier. Certainly, it would be unjust for a PD to choose not to graduate a resident without any prior discussion of shortfalls in a core outcome. Thus, the system of assessment with clear developmental markers is critical. The goal of CBME is that any learner who is falling short of a core outcome would be identied early and assisted in creating a program of study to help them to succeed. This remediation may require additional time in residency or an alteration from the residency’s usual curriculum, but most learners will succeed in meeting core outcomes with this support [8, 28].
Evaluating Systems ofAssessment
In order to meet the promise of CBME, GME programs must include a process to evaluate their systems of assessment with a goal of continuous quality improvement [36, 51]. It is logical to begin with simple, practical solutions in the short term, and it will be critical to rene such measures as new evidence emerges, both internal within and external from training programs. There are several critical factors that need to be considered when evaluating systems of assessment.
Most importantly, we must recall that residency programs are complex systems with many interdependent parts [10,
32]. An evaluation of any component of a GME program
must consider this fact. Bowe and Armstrong [11] offer a holistic framework for such evaluation, which is simple and elegant.
At level 1, individual components of the assessment sys­tem are evaluated formatively on a frequent basis. For exam­ple, an assessment form for an inpatient family medicine rotation is reviewed by learners, faculty advisors, and the residency director as it is completed and shared. Residency management software makes such reviews simple, automati­cally delivering completed forms to designated individuals. If a residency director nds that the form offers minimal dif­ferentiation between learners, and that few narrative com­ments are included, they speak informally to the chief residents and the head of the FM inpatient service about the issue. The form is adapted based on feedback from these key stakeholders and all stakeholders monitor future forms to assess the impact of the change. This example highlights sev­eral key components of level 1 evaluation. In this example, the director identies a problem based on their observation of their regular workow, though the problem could easily have been reported at any time by the chief residents, FM
service leader, and inpatient faculty. In all these scenarios, systems that are already in place lead to the identication of a problem. The problem is addressed by key stakeholders in partnership with program leaders. A change is made based on feedback from stakeholders, and the change is monitored for effectiveness. This process of continuous quality improve­ment is familiar to most physicians from the use of the plan­do- study-act (PDSA) method for clinical improvement [60].
At level 2, the system of assessment itself is monitored for problems. Through formal and informal processes, educa­tional leaders evaluate the system as a whole. Bowe states, “comprehensive analyses on combinations of assessment data are planned and interpreted to identify patterns of com­ponent interrelationships, trends in learning trajectories, root causes of unexpected outcomes, and the predictive value of various assessment approaches and criteria” [11]. While this process sounds formal and complex, the reality is it can be quite simple. Residencies must seek feedback from residents and faculty about the quality of core learning experiences and from residents on faculty members’ performance. Such assessments should be reviewed by the residency’s leader­ship team to analyze trends. In addition, the review of this evaluation data is a core function of the Program Evaluation Committee (PEC). For example, a residency program direc­tor (PD) may receive feedback about a PG2 cardiology expe­rience that had previously been well received. The rst sign might be poor assessment of resident performance, and this might be followed by resident feedback that the learning experience is overly challenging. Following these trends, the PD would bring the concerns to the PEC, and residents might report that a core cardiology faculty member who had previ­ously taken responsibility for didactics on the rotation has been out on an unexpected leave of absence. Another member of the PEC might realize that a recent change to the residen­cy’s formal teaching schedule moved an ECG-reading lec­ture previously given in the PG1 year to PG2. While the specic mechanisms that programs use to detect this type of systemic problem that relates to multiple parts of the resi­dency may be different, the idea that a group is reviewing the trends and looking into concerning ndings must be a part of every program. It is important to note the value of both quali­tative and quantitative data in this process. Because the PEC’s work is in practice improvement, the data analysis need not be at the level of publishable research. Rather, level 2 evaluation needs to allow exibility, rapid detection, and the ability to respond quickly to problems, as would be typi­cal during PDSA cycles in practice-based quality improvement.
Finally, level 3 “evaluations occur at periodic intervals to comprehensively review longitudinal system performance and determine system readiness to prepare learners for evolv­ing changes in the medical education training continuum and
11 Designing Assessment toMeet theChallenge ofCompetency-Based Medical Education
97
health care system.” The ACGME’s NAS included a require­ment for a 10-yr Self-Study that was an excellent example of level 3 analysis [47, 49, 58, 62]. However, the ACGME logistically was unable to continue site visits associated with these Self-Studies [1]. The future of ACGME-mandated Self-Studies remains unclear. Regardless, level 3 processes remain an important step for the success of CBME.Therefore, the steps of the ACGME Self-Study will serve as an example of a level 3 process. Philibert and Lieh-Lai [58] recommend that an evaluation group, which could be the program’s PEC or another subset of stakeholders within the residency, assembles and aggregates data from Annual Program Evaluations (APEs). The APE analysis can simply be a review of the documents themselves with an eye toward trends that inform a longitudinal assessment of the program’s improvement efforts. Using the initial analysis as back­ground, the evaluation group meets with stakeholders within and external to the residency to consider the program’s aims. Once the aims are reviewed and updated (as necessary), the evaluation team performs a SWOT analysis, identifying strengths, weaknesses, opportunities, and threats for the pro­gram [22]. The evaluation team then presents the ndings of the Self-Study with the program leadership and constituents to come to consensus on improvement priorities for the next several years. The process is completed by documenting the ndings and recommendations of the study. As an example, the PEC for a residency program could schedule an extra meeting to review the APEs from the last several years. This could be followed by a resident-faculty meeting to review the identied trends and discuss the aims of the program. A SWOT analysis could be conducted by the residents and fac­ulty at that meeting. External stakeholders such as the DIO, leaders of key off-service rotations, and operational leaders at the FMP could review the SWOT analysis asynchronously and offer additional suggestions. The PEC chair could create a document listing all the ndings to date and share it with the program leadership, working together to draft a list of key improvement goals for the next several years. These goals could be shared with the PEC at its next meeting. Following a discussion, the APE for the next year could include an action plan reecting those goals. This process could occur over a year and would only require a few hours of time from the PEC, the PD, and a resident-faculty meet­ing. However, the power of such a process to anticipate future changes and prepare the program to seize opportuni­ties and respond to threats allows programs to excel, even amidst uncertain times.
One additional factor that must be mentioned when dis­cussing the evaluation and evolution of GME program assessment is the difculty of implementation of novel ideas
within complex systems. Given the focus of the ACGME and ABFM on the implementation of CBME, we can expect the emergence of evidence about successful assessments pro­cesses and systems over the coming years. It is tempting for a residency leader to think, “I wish they would just tell me what assessment program to implement and how,” and equally enticing to imagine that others will do the work of nding the ideal system that all programs can implement. However, even if such an ideal system existed, the imple­mentation and optimization of the assessment system within an individual program would require an understanding of these principles and a process for evaluation and continuous improvement. This is true because complex adaptive systems can create drastically different outcomes even when the same process is implemented in systems that appear similar [32]. Therefore, the implementation of CBME will require all GME programs to implement systems of assessment that are unique to their environment. This caution is not to say that GME leaders should not look externally for ideas for improvement. Rather, we must expunge the fallacy that a process that has been successfully implemented elsewhere can be implemented locally without change or monitoring.
CBME andTime Variable GME
Traditional medical education is centered around time spent in a variety of clinical environments [26], usually in block­based rotations [55]. However, learners come to training with varying levels of competency and acquire skills at vari­able rates [26]. One appeal of CBME is an invitation to con­sider the possibility that learners could complete GME programs in differing amounts of time [55]. All experienced GME faculty members have experienced residents who required more than the required duration of training to meet the program’s requirements. This may occur for a variety of reasons, and most residents who require additional time are able to successfully complete training [8, 28]. CBME offers promise to detect resident dyscompetency or incompetency earlier and support remediation by giving specic targets for improvement [55]. In addition, several pilots demonstrate the possibility that CBME might allow learners to shorten their time in training with robust curricula and assessment systems [48, 56]. Family medicine has begun the journey into time variable GME in partnership with the ACGME in the family medicine Advancing Innovation in Residency Education (AIRE) program that allows longer training and facilitates innovation in residency curricula [50]. This pilot began in 2023 and will continue for an uncertain duration as it assesses possibilities for the future of our discipline.