Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_112_библиотеки_им_акад_М_И_Перельмана
.pdf
early lessons from three specialties. Acad Med. 2020
Jan;95(1):97-103. doi:10.1097/ACM.0000000000002899.
73. Han M, Hamstra SJ, Hogan SO, et al. Trainee
physician milestone ratings and patient complaints in
early post-training practice. JAMA Netw Open. 2023
Apr 3;6(4):e237588.
doi:10.1001/jamanetworkopen.2023.7588.
74. Kim JG, Rodriguez HP, Holmboe ES, et al. The
reliability of graduate medical education quality of
care clinical performance measures. J Grad Med Educ.
2022 Jun;14(3):281-288. doi:10.4300/JGME-D-21-
00706.1.
75. US Department of Education, Office of Educational
Technology Education. Enhancing teaching and
learning through educational data mining and
learning analytics: an issue brief. October 2012.
Accessed September 16, 2022. https://tech.ed.gov/wp-
content/uploads/2014/03/edm-la-brief.pdf.
76. Ellaway RH, Pusic MV, Galbraith RM, Cameron T.
Developing the role of big data and analytics in health
professional education. Med Teach. 2014;36(3):216-
222. doi:10.3109/0142159X.2014.874553.
77. Kogan JR, Hess BJ, Conforti LN, Holmboe ES. What
drives faculty ratings of residents’ clinical skills? The
impact of faculty’s own clinical skills. Acad Med.
2010;85(10 suppl):S25-S28.
doi:10.1097/ACM.0b013e3181ed1aa3.
78. Kogan JR, Conforti LN, Iobst WF, Holmboe ES.
Reconceptualizing variable rater assessments as both
an educational and clinical care problem. Acad Med.
2014;89(5):721-727.
https://t.me/med1917
doi:10.1097/ACM.0000000000000221.
79. Lucey CR, Hauer KE, Boatright D, Fernandez A.
Medical education’s wicked problem: achieving
equity in assessment for medical learners. Acad Med.
2020;95(12 suppl):S98-S108.
doi:10.1097/ACM.0000000000003717.
80. Alexis DA, Kearney MD, Williams JC, Xu C,
Higginbotham EJ, Aysola J. Assessment of
perceptions of professionalism among faculty,
trainees, staff, and students in a large universitybased health system. JAMA Netw Open.
2020;3(11):e2021452.
doi:10.1001/jamanetworkopen.2020.21452.
81. Forscher PS, Lai CK, Axt JR, et al. A meta-analysis of
procedures to change implicit measures. J Pers Soc
Psychol. 2019;117(3):522-559.
doi:10.1037/pspa0000160.
82. Morsy L. Carnegie and Rockefeller’s philanthropic
legacy: exclusion of African Americans from
medicine. Acad Med. 2023 Mar 1;98(3):313-316.
doi:10.1097/ACM.0000000000005092.
83. Forscher PS, Mitamura C, Dix EL, Cox WTL, Devine
PG. Breaking the prejudice habit: Mechanisms, time
course, and longevity. J Exp Soc Psychol. 2017;72:133-
146. doi:10.1016/j.jesp.2017.04.009.
84. Dickey CC, Thomas C, Feroze U, Nakshabandi F,
Cannon B. Cognitive demands and bias: challenges
facing clinical competency committees. J Grad Med
Educ. 2017 Apr;9(2):162-164. doi:10.4300/JGME-D-16-
00411.1.
85. Englander R, Holmboe E, Batalden P, et al.
https://t.me/med1917
Coproducing health professions education: a
prerequisite to coproducing health care services?
Acad Med. 2020;95(7):1006-1013.
doi:10.1097/ACM.0000000000003137.
86. Damschroder LJ, Aron DC, Keith RE, Kirsh SR,
Alexander JA, Lowery JC. Fostering implementation
of health services research findings into practice: a
consolidated framework for advancing
implementation science. Implement Sci. 2009;4:50.
doi:10.1186/1748-5908-4-50.
87. Hall AK, Rich J, Dagnone JD, et al. It’s a marathon,
not a sprint: rapid evaluation of competency-based
medical education program implementation. Acad
Med. 2020;95(5):786-793.
doi:10.1097/ACM.0000000000003040.
88. Rich JV, Luhanga U, Fostaty Young S, et al.
Operationalizing programmatic assessment: The
CBME Programmatic Assessment Practice Guidelines.
Acad Med. 2022 May 1;97(5):674-678.
doi:10.1097/ACM.0000000000004574.
89. Van Der Vleuten CPM, Schuwirth LWT, Driessen
EW, Govaerts MJB, Heeneman S. Twelve tips for
programmatic assessment. Med Teach. 2015
Jul;37(7):641-646. doi:10.3109/0142159X.2014.973388.
90. Schuwirth L, van der Vleuten C, Durning SJ. What
programmatic assessment in medical education can
learn from healthcare. Perspect Med Educ. 2017
Aug;6(4):211-215. doi:10.1007/s40037-017-0345-1.
91. Misra S, Iobst WF, Hauer KE, Holmboe ES. The
importance of competency-based programmatic
assessment in graduate medical education. J Grad
https://t.me/med1917
Med Educ. 2021 Apr;13(2 Suppl):113-119.
doi:10.4300/JGME-D-20-00856.1.
92. Iobst WF, Holmboe ES. Programmatic assessment:
the secret sauce of effective CBME implementation. J
Grad Med Educ. 2020 Aug;12(4):518-521.
doi:10.4300/JGME-D-20-00702.1.
93. Holmboe ES, Osman NY, Murphy CM, Kogan JR.
The urgency of now: rethinking and improving
assessment practices in medical education programs.
Acad Med. 2023 Apr 18.
doi:10.1097/ACM.0000000000005251.
https://t.me/med1917
4
Evaluation Frameworks,
Assessment Forms, and Rating
Scales
Louis N. Pangaro, MD, Steven J. Durning, MD, PhD, Eric S.
Holmboe, MD
https://t.me/med1917
Chapter Outline
Introduction
Evaluation Forms and Frameworks
Analytic Frameworks
Developmental Frameworks
Synthetic Model
Achieving Construct Alignment Through Simplicity
Descriptive Terminology for Narratives in Evaluation
Complementary Frameworks: ACGME General
Competencies and RIME
Frameworks: Concluding Thoughts
Rating Scales
Rating Scales: Basic Design
Purposes and Advantages of Evaluation Forms
Narrative-Based Assessment
Evaluation Sessions
Psychometric Issues
Reliability
Validity
Rating Errors
Rater Accuracy
Faculty Development and Evaluation Forms
Performance Dimension Training and RIME
Conclusions
Annotated Bibliography
References
Introduction
https://t.me/med1917
As noted in earlier chapters, assessing trainee performance in
the care of “real” patients in actual practice settings is critical
to effective evaluation.* Central to this process is the faculty
member who is the observer of the clinical care and of the
specific interactions of the learner with a patient, the
patient’s family, and other members of the healthcare team.
Central to the concept of “competency-based” education is
that observations and the judgments made about a learner’s
progress and readiness for advancement are explicit, public,
and demonstrable; further that they are applied with
consistency and with sufficient sampling to ensure readiness
for the next stage of professional development. This chapter
concerns itself with how concepts of competence, rating
scales, and other evaluation forms are intended to guide the
observer in what to look for and how to interpret what is
seen. It is now understood that the faculty observer is not a
dispassionate servomechanism using the evaluation
(assessment) rubric provided, but is engaged in a social
judgment involving complex interactions1 and emotions that
can also be affected by various biases. In other words, the
problem of assessing learners in the clinical setting to achieve
accuracy and equity2 is not simply cognitive (a choice of the
rating rubric) but equally a social and emotional process for
the teacher, as well as logistic in the time the faculty member
has to observe the learner while at the same time ensuring
the patient’s well-being. Central to this chapter is the
understanding that the program or clerkship director cannot
simply provide the teacher-rater with an evaluation form or
rating scale and expect that summative judgments (e.g.,
assessment of learning) with good validity evidence can be
made. We hope that reviewing the premises and uses of
https://t.me/med1917
frameworks and forms will give program directors
confidence in their process of providing feedback and in
making advancement decisions.
To achieve a summative judgment of readiness for
advancement, these in vivo observations are combined with
standardized and typically quantified in vitro assessments of
knowledge and procedural and/or interpersonal skills.
Educators may label this activity as in-training assessment
3
or, more recently, workplace-based assessment (WBA).
Effective assessment of overall clinical performance requires
a multidimensional approach using both descriptive and
quantified tools to describe the ability to provide patient
care, and these assessments may include input from both
faculty and nonphysician observers.4 Such evaluation or
assessment involves judgment, often using a rating scale and
narrative, about a learner’s performance. This chapter will
discuss the frameworks underlying evaluation, systems of
descriptive assessment yielding narrative evaluation, rating
systems yielding a quantified evaluation, and the use and
pitfalls of rating scales. Evaluation forms and rating scales
may initially be applied at the level of the interaction
between the learner and a single patient, and the form
provides the framework that allows teachers to convert their
observations into information that can be used for feedback
to the learner and to the program as part of its ongoing
quality improvement efforts. To achieve an eventual decision
about advancement of the learner to the next level of
responsibility, there must be numerous observations in many
contexts (i.e., clinical situations) by many observers that
ultimately allow the process to achieve a level of confidence
https://t.me/med1917
sufficient for a high-stakes (“summative”) decision. To do
this, the evaluation framework, the assessment form, and the
rating scale must be robust; that is, have strong reliability
and validity evidence in each use (see Chapter 2).
The most common method used by faculty for evaluating
overall performance has been the global rating scale included
as part of an evaluation form, in which the term global
implies an all-inclusive synthesis of performance dimensions
over some period of time; the term scale refers to a linear and
ordinal analog (often a line with numbers) for distinguishing
levels or steps of performance; and the term rating refers to
the act of locating a person’s performance on the continuum
or at a specific level.
5,6
The rating scale is one component of
an evaluation form, which typically also expects descriptive,
written comments.7 The intention is that the final evaluation
documented on the form should be the aggregate of the scale
rating(s) on specific competencies plus descriptive, written
comments. The expectation is that the ratings on individual
scales and the overall evaluation of the learner will reflect the
institution’s framework of assessment, and for graduate
medical education (GME) will in turn reflect the expectations
of the specialty’s national organization, such as the
Accreditation Council for Graduate Medical
Education/American Board of Medical Specialties
(ACGME/ABMS) Milestones in the United States. In other
words, the premises about what successful performance
looks like—the “construct” underlying the assessment—
should be shared on the evaluation form in both the rating
scale and the rater’s observations; these are all aligned with a
concept, construct, or mental model that is shared by the
https://t.me/med1917
program, learners, and teachers.8 As we will emphasize in
this chapter, the more a rater accepts the premises of the
rating scale—a process of construct alignment—the more
successful the rating process will be.
9
Where do evaluation forms fit into a medical education
evaluation system? Comprehensive evaluation of a trainee is
a multidimensional composite that is authored by the
director of the academic program (“academic director”; i.e.,
fellowship director, program director, or clerkship director)
and often includes both summary evaluations of learners by
one or more teachers and a series of quantified
measurements. Evaluations by individual teachers in turn
may be their own syntheses of multiple observations over
days or weeks, with or without direct observations of
competence at individual tasks.
It is the role of academic managers10 such as fellowship or
residency program, clerkship, or course directors to achieve
consistently credible evaluations of trainees, for the sake of
both society and future patients (summative evaluation) and
to enhance the improvement of trainees through feedback
(formative evaluation) (see Chapter 1). Formative evaluation,
“assessment for learning,” is most important for professional
development, feedback, and coaching. The design and
systematic use of evaluation forms, with or without rating
scales, is one strategy academic managers can use to foster
credible evaluations that are valid and not arbitrary (i.e., the
evaluations are based on societal, professional, and
institutional goals and not on goals that may be idiosyncratic
https://t.me/med1917
Соседние файлы в папке Библиотека им академика М.И. Перельмана