Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_112_библиотеки_им_акад_М_И_Перельмана
.pdf
FIG. 1.6 Coproduction to support learning trajectories through
effective programmatic assessment.
Pusic and colleagues have noted that learning curves, or
trajectories, are in fact sigmoidal, and that learners during
UME and GME training will spend almost the entirety of
their time on the steep part of the curve.98 Ultimately, workbased assessment will need to be the predominate
component in a program of assessment, especially for
postgraduate programs and ongoing professional
development in practice. Educators need to recognize this
developmental sequence when designing an assessment
system and it will be critical to ensure that the chosen method
is suitable to the task. Finally, programs and educational
leaders should adopt a coproduction mindset that is aligned
with the core components framework described earlier.
Coproduction sees learner as partners in the educational
program and sees assessment as something faculty do with
learners instead of to them. This involves a series of
coproduced learning cycles to support learners’ professional
development.
https://t.me/med1917
Criteria for Choosing an
Assessment Method
Decisions about which method of assessment to use in a
particular circumstance have traditionally rested on validity
and reliability. Validity is the degree to which the inferences
based on the results of an assessment are correct. Valid
inferences regarding a particular test score or assessment
result depend upon the reliability of these outcomes, and
reliability is a component in more “modern” concepts of
validity such as those by Kane and Messick, discussed in
Chapter 2.
For purposes of assessment in medical education, Van Der
Vleuten added educational impact, cost-effectiveness, and
acceptability as other key factors in their utility index to
guide the choice or revision of assessments. The utility index
is represented by the equation Validity × Reliability ×
Educational Impact × Cost-Effectiveness × Acceptability =
Utility.99 Utility is a useful concept as programs choose and
implement assessment methods. It is also important to note
that Utility is a multiplicative construct; if any of the terms, or
variables, are zero, utility is, by definition, zero.
In terms of educational impact, Van Der Vleuten argues that
trainees will work hard in preparation for an assessment.
90
Consequently, the method should direct them to study in the
most relevant way. For example, if an educational objective is
for trainees to know the differential diagnoses for a particular
chief complaint, then assessment using extended matching
questions will likely induce better learning than assessment
based on standardized patients.
Cost-effectiveness is the extent to which an assessment
https://t.me/med1917
method is affordable and efficient. Although high-fidelity
simulations might be a good way to assess procedural
competence, the use of a method such as direct observation
of procedural skills (DOPS), which is based on faculty
observation, is likely to be more cost-effective in most
graduate training settings.
100
Acceptability is the degree to
which the trainees and faculty believe that the method
produces valid results. This factor will influence motivation
of faculty to use the method and enhance trainees’ distrust of
the results. It is important that educational leaders not
underestimate trainees’ knowledge and understanding of
assessment and their ability to participate in decisions
regarding assessment practices.
An international group of assessment experts updated the
utility index and created a list of criteria for good
assessment.
101
Validity and acceptability were retained as
separate categories, and for validity the importance of
coherence (a body of evidence that hangs together to support
the results for a specific purpose) was highlighted. Reliability
was essentially split into two new categories: (1)
reproducibility (repeatability) and consistency (all items or
components of the test intercorrelate in the same direction),
and (2) equivalence (all students are treated equally).
Educational impact was split into educational effect (the
assessment motivates learners to prepare for it in a fashion
that has educational benefit) and catalytic effect (the
assessment provides results and feedback in a fashion that
creates, enhances, and supports education; it drives future
learning forward). Finally, the last new category was
feasibility, namely that the assessment should be practical,
realistic, and sensible.
101
https://t.me/med1917
Educational Effect>
“The assessment
motivates those who
take it to prepare in a
fashion that has
educational benefit.”
Catalytic Effect>
“The assessment provides results
and feedback in a fashion that
creates, enhances, and supports
education; it drives future
learning forward.”
In addition to the factors highlighted in the two versions of
criteria for good assessment, it is important to consider how a
particular method fits into the overall system for assessment.
The same method can (and arguably should) be used to
assess more than one competency. For example, peer
assessment can provide a measure of both professionalism
and interpersonal skills. Likewise, two different methods can
be used to capture information on the same competency,
thereby increasing confidence in the results. For example,
patient care can be assessed using both the single-encounter
mini-CEX (clinical evaluation exercise) and summary ratings
by attending physicians derived from a more longitudinal
experience (e.g., clinical rotation).
Educational effect, catalytic effect, feasibility, and
acceptability are not easily quantifiable, nor is the
relationship among methods of assessment in a system.
However, these factors plus reliability and validity should be
weighed interactively when considering selection of a
particular method.
Elements of Effective Faculty
Development
Faculty members play a particularly critical role in
assessment in the clinical setting because such assessment is
https://t.me/med1917
often based on observation. And by faculty we mean any
health professional, at a minimum, who participates in an
assessment system. Recall that Miller placed “does,” meaning
the care of actual patients, at the top of the pyramid. Envision
the pyramid as a spear and at the tip of that spear are
patients. Using this metaphor helps faculty appreciate the
central role of observation in both assuring trainee
competence (at a minimum) and guaranteeing that patients
receive high-quality, safe care in the context of training (see
Chapter 5
). Most important is the fact that the actual
measurement instrument is the faculty, not the assessment
tool. We cannot emphasize enough throughout this book that
assessment in the workplace is essential and relies on
informed, expert judgment (see Chapters 4, 5, and 9).
Assessment methods and tools are only as good as the
individuals using them. Although there has been substantial
progress in creating many new methods and tools,
significantly less attention has been paid to the development
of approaches to training faculty in how to use them most
effectively. This omission continues to occur despite repeated
studies over time demonstrating significant problems with
the quality of faculty assessments
102–104
(see Chapter 5).
There are three significant reasons why faculty training is
urgently needed. First, to perform quality assessment, faculty
members must possess sufficient knowledge, skill, and
attitudes in the competency targeted by the assessment. For
example, a decline of clinical skills teaching in the workplace
was noted by George Engel in 1976
105
and has resulted in
many of today’s educators failing to acquire a high level of
clinical skills needed for effective care and teaching. This
likely limits the degree to which they can validly assess
https://t.me/med1917
clinical performance, and recent research adds evidence to
the importance of the faculty’s own underlying clinical
skills.49 Faculty development and subsequent teaching and
assessment can restore this gap.
Second, competencies that are considered essential for
every physician have evolved and will continue to evolve
and change over time. Witness the birth of the competency
domains of Practice-based Learning and Improvement and
Systems-based Practice in the ACGME/ABMS competency
framework, and the change of the role of manager in
CanMEDS to that of leader in 2015.
106
The majority of faculty
today never received during their training any formal
instruction in many of the competencies and
subcompetencies now needed for modern practice. Many
faculty actually acquire new knowledge and skills alongside
their trainees (i.e., through colearning).
Finally, assessment is a core tenet of professionalism for
medical educators. Too often, faculty members view
assessment as someone else’s job, especially when a negative
performance appraisal is involved (see Chapter 16). Faculty
development reinforces the importance of assessment and
provides medical educators the opportunity to develop
common standards for performance. The medical profession
has the obligation to uphold and protect its own standards,
implying a commitment for physicians to evaluate
themselves and their colleagues.
107
To make effective use of the methods of assessment,
educational institutions must commit the necessary resources
for faculty development. However, too often faculty
development translates into a project or a brief workshop. If
faculty development is to be truly successful, medical
https://t.me/med1917
educators need to embrace new strategies that embed faculty
development in real-time teaching and clinical activities. For
example, Hemmer and colleagues embed faculty frame-ofreference training into formal evaluation sessions for
students.
108
Faculty development, like quality improvement
and maintenance of competence, must become a continuous
process and appropriately rewarded. As noted earlier, the
quality and safety of patient care depends on it (see Chapter
5).
Medical educators must also end their quest for the perfect
assessment tool with the perfect rating form imbued with
special powers to solve all measurement needs. Assessment
is a sophisticated skill that requires hard work and a
multifaceted approach. Lindy and Farr, in a landmark article
in the performance appraisal field over 40 years ago, pleaded
with researchers to redirect development efforts from a
search for the perfect rating form to training the assessors.
109
Researchers in this field subsequently developed several rater
training approaches that can lead to better assessments.
Chapter 5 provides guidance on several practical faculty
training methods.
Milestones and EPAs, described later in the chapter, require
special consideration. Using Milestones and EPAs for
curriculum development and assessment requires a shift in
thinking among faculty and an infrastructure to support new
assessment practices. Both individual faculty and committees
must get acquainted and experienced with entrustment
decision-making for EPAs and their conditions.
110
Training in
the dimensions to be used in assessment and in the criteria
for decisions is needed, and specific tools related to EPAbased assessment continue to be developed and
https://t.me/med1917
implemented.
111,112
If anything, sufficient and adequate
supervision and feedback is key to entrustment decisions,
which requires longitudinal relationships with program
leaders, advisors, and coaches.
113,114
This does not necessarily
mean huge investments in time for advising, but an efficient
use of any encounter that advisors and advisees have, for the
benefit of learning. Group process will also likely enhance the
effectiveness of Milestones and EPAs as part of an
assessment system, and faculty will need training in effective
group process (see Chapter 16).
Overview of Assessment Methods
Traditional Measures
(See Chapters 2 and 7.)
Traditional measures will continue to play an important
role in the assessment of clinical proficiency. Specifically,
written methods such as MCQs and standardized patients
will be foundational components of assessment programs for
the near future, especially in undergraduate medical
education. All of these methods can be improved and work
on each must continue.
Methods Based on Observation
(See Chapters 4–6, 8, and 9.)
Assessment methods based on the observation of routine
encounters in the clinical setting offer a rich and feasible
target for assessment. Continued refinement of the methods
themselves is needed, as is faculty development, which is a
key to their successful use. Furthermore, the opportunity for
educational feedback and coaching as part of these methods
https://t.me/med1917
is probably as important as their assessment potential.
Simulation
(See Chapters 6
, 7, and 13.)
Improvements in technology have spurred the
development of a series of simulators that re-create reality
with high fidelity, but even lower-tech simulations can be
very valuable. The use of simulation in assessment is
growing, but much of the technology remains expensive and
several developments are needed before their widespread
adoption and use. Researchers will need to continue to focus
on identifying appropriate scoring methods, optimizing the
generalizability of scores, and ensuring their relevance to
performance in practice.
115
Recent research of mastery-based
simulation in specific skills such as central venous lines and
other bedside procedures has clearly shown translation of
skills acquired at a mastery-defined level in a simulation to
the patient bedside. These methods offer the ability to test
under a variety of conditions without concern for harm to
patients and they improve care at the bedside. Evidence is
growing that mastery-based approaches combined with
simulation-based deliberate practice can translate into
improve patient care and outcomes.
116–118
Educators will
confront difficult decisions requiring them to balance the
cost, variable fidelity of individual simulation methods, and
potential risks to patients (and trainees) in making decisions
regarding how best to assess procedural skills.
119
Workplace-Based Assessment
(See Chapters 3–5, 8–12, 16, and 17.)
The assessment of physicians’ performance at work (mostly
https://t.me/med1917
the “does” level of Miller’s pyramid) is the area of assessment
undergoing the most change and development. While
learners may try to “perform” when under direct observation
(“show how”), most learners acclimate quickly, and even if
what the faculty observe is “best behavior” there is still much
utility in the assessment and ensuring that the patient
receives safe, effective, patient-centered care. The day-to-day
performance of physicians is being used increasingly in the
settings of continuous quality improvement and physician
accountability. Assessment in this context is a matter of
identifying the basis for the judgments (e.g., outcomes,
process of care), deciding how the data will be gathered, and
avoiding threats to validity and reliability (e.g., patient mix,
patient complexity, attribution, and numbers of patients).
120
The patient is also playing a much greater role in workplacebased assessment, predominantly through patient experience
surveys.
121
In addition, patient-reported outcome measures
(PROMs) are being increasingly used by health systems to
judge functional outcomes for patients (see Chapter 11).
While substantial research continues around quality and
safety measures, patient experience surveys, and PROMs,
much work remains to be done as noted earlier in this
chapter. However, given this is ultimately what patients and
the public care most about, educational programs need to
embrace workplace-based assessments as part of an overall
assessment program.
Emerging Directions in
Assessment
Implementation of competency-based medical education
https://t.me/med1917
Соседние файлы в папке Библиотека им академика М.И. Перельмана