Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_2823_Библиотеки_им_академика_М_И_Перельмана
.pdf
information about the learner. While multiple perspectives
can be valuable, we believe that acceptable medical care is
bounded, and as such, not all assessments are of equal
quality. In other words, there are important limits for what is
acceptable patient care based on the patient’s specific context,
needs and desires and not that of the learner.
Some variation in assessments is beneficial, such as when
an assessor leverages one of their areas of expertise
associated with effective practice (i.e., warranted variation).
In contrast, assessments driven by idiosyncrasies or bias that
is not evidence based represent unwarranted variation.
Unwarranted variation is an underappreciated problem in
assessment.
144
Unwarranted variation can be harmful and
can contribute to suboptimal and variable educational
outcomes. Unwarranted variation at the program and
institutional levels also can affect educational outcomes and,
ultimately, the quality of care graduates deliver.
145
The impact of unwarranted variation in assessment is
depicted in Fig. 5.8.
144
Care delivered to patients can either be
acceptable (represented in turquoise), questionable (light
blue), or unacceptable (white). A much smaller amount of
acceptable care is supported by best evidence (purple).
Imagine there are two faculty (Faculty 1 and Faculty 2) who
observe and assess the same learner with a patient. The care
the learner provides to the patient is shown in yellow. In this
example, the learner provided some best-evidence care, some
acceptable care, and some unacceptable care. Faculty 1
observed some, but not all, of the learner’s best-evidence care
and some, but not all, of the acceptable care. Faculty 1 missed
the learner’s unacceptable care. In contrast, Faculty 2
identified some of the learner’s best-evidence care but also
https://t.me/med1917
believe they observed best-evidence care and acceptable care
that was not actually done by the learner. However, Faculty 2
did pick up on the learner’s unacceptable care. This example
shows how faculty can make errors of omission and
commission. This diagram also underscores that poor
accuracy in assessment is not just an educational issue but a
patient quality of care issue as well, as the patient potentially
receives unacceptable care that is not identified.
FIG. 5.8 Warranted and unwarranted variation in clinical skills
assessment (see text for details).
Overview of Faculty Development
Approaches to Improve Assessment
Quality
Being a good clinician and teacher does not necessarily
https://t.me/med1917
equate to skill completing WBA. The quality of WBA is
primarily dependent on faculty’s ability to make and
document high-quality observations of the learner and
provide the learner with feedback. As mentioned previously,
faculty are the assessment instrument in WBA. Therefore
time developing faculty is paramount to ensure effective
observation and assessment of clinical skills.
71,146–148
Said
another way, improving WBA quality requires faculty
development.
147
There are only a few studies that have explored the
effectiveness of rater training to improve WBA.
99,149–152
Therefore there is not an extensive body of “evidence” to
guide approaches to rater training faculty development.
Additionally, rater training studies have showed mixed
effects. For example, a randomized controlled trial (RCT) of
an 8-hour faculty development intervention that included
live practice with standardized residents and patients led to
meaningful changes in faculty ratings.
149
Another RCT of two
3-hour workshops followed by three asynchronous online
spaced learning modules improved rating accuracy and the
accuracy and quality of narrative assessment.99 However, a
study with a briefer 2-hour training intervention and no
hands-on practice failed to show meaningful change.
150
One-time faculty development interventions are less robust
than longitudinal interventions.
153
Too often, faculty
development occurs as a one-time event where participants
are taught information. However, without reinforcement,
acquired skills diminish with time. For rater training faculty
development to be effective over time, assessors need to be
“recalibrated.”
154
Longitudinal faculty development also
enables participants to share successes, discuss challenges,
https://t.me/med1917
learn new skills, and refresh previously learned skills.
Another benefit of longitudinal faculty development is that it
can also create learning communities or “communities of
practice.”
155
The goal of rater training is to improve the quality of WBA.
The rater training approaches described in this chapter are
informed by our qualitative and quantitative research studies
and our experience running faculty development
programs.
99,111,116,151
In the next part of this chapter, we will
describe eight parts of rater training: (1) motivating faculty to
participate in rater training, (2) performance dimension
training, (3) frame of reference training, (4) recognizing
inference, (5) prioritizing observations and creating a
summary statement, (6) mitigating bias, (7) creating
opportunities for additional practice, and (8) practicing
feedback after direct observation. Appendix 5.1 has examples
of 1.5-hour, 3-hour, and full-day rater training workshops,
and examples of longitudinal training.
Motivating Faculty to Participate in Rater
Training
It is sometimes necessary to motivate faculty to participate in
rater training. It can be helpful to review the importance of
assessing clinical skills before starting rater training. While
discussing the importance of assessing clinical skills may
encourage direct observation, it is unlikely to motivate
faculty to improve the quality of their assessments. Many
faculty erroneously believe they effectively and accurately
assess learners and, as a result, may not be motivated to
participate in rater training. To motivate faculty to participate
in rater training, we describe the problems with WBA
https://t.me/med1917
accuracy, reliability, and validity. However, “showing” the
need to improve assessment is more effective than “telling.”
Therefore we show faculty a video of a standardized resident
performing a history or counseling a standardized patient. In
the video the resident demonstrates
“intermediate/satisfactory” clinical skills. (See Appendix 5.2
on the Expert Consult website; video scenarios for medical
interviewing [Videos 5.1
–5.3]; physical examination [Videos
5.4–5.6], and counseling [Videos 5.7–5.9]). Participants are not
told how the resident is scripted. Participants watch the
video, identify the resident’s strengths and weaknesses, circle
a mini-CEX rating, and indicate how they arrived at their
overall rating (normative standard, self as standard, gestalt,
etc.). Next, we poll faculty on the rating they selected.
Inevitably, there is a range of ratings (for example, rating
scores ranging from 2–8), which demonstrates low interrater
reliability. Participants are usually surprised by the degree of
rating variability (particularly when participants are more
experienced educators) and are interested in hearing how
other faculty selected their ratings. This nicely sets the stage
for performance dimension training.
Performance Dimension Training
Performance dimension training (PDT) is a critical element
for all rater training programs.
156–158
The overarching goal of
PDT is to ensure that faculty understand the definitions and
criteria for the competency of interest and build shared
mental models of competence. The steps of PDT are
summarized in Appendix 5.3. PDT starts with the facilitator
asking participants to define, for a specific skill (e.g., starting
a new medication), all the criteria and learner behaviors that
https://t.me/med1917
constitute superior skills (i.e., expertise and mastery) from
the perspective of patient outcomes. Faculty can use the
video they just watched to help them with PDT. For example,
faculty list the verbal and nonverbal behaviors necessary for
effective counseling. All items should be observable. That is,
if empathy is important, faculty should identify the behaviors
that demonstrate empathy. PDT exercises should be
performed in small groups of five to eight individuals. Small
groups can share their results with the larger group. Group
differences lead to productive discussions about what are the
criteria for competence in counseling. After groups generate
their list of behaviors, we distribute evidence-based
standards of the skill (frameworks) (Appendix 5.3). Examples
include the SEGUE model for medical interviewing,
Braddock’s elements or the Agency for Healthcare Research
and Quality’s (AHRQ) SHARE framework for shared
decision-making, and the SPIKES model for breaking bad
news.
32
,78,159,160
Faculty compare their list of behaviors to
those listed on the framework. This step further standardizes
and calibrates faculty on the criteria associated with that skill.
Faculty feel validated when they identify the same skills that
have been published.
151
When faculty identify a list of
behaviors before seeing the framework, they will consider
skills on the distributed frameworks they had not identified
on their own.
151
Faculty then rewatch the video encounter
using the new criteria (the behavioral framework) to guide
their observations. The PDT exercise takes approximately 30
minutes for a single clinical skill. Interactive group
conservations during PDT help create a “shared mental
model” for assessment that faculty buy into.
151
Although the
group process takes time, faculty prefer creating their list of
https://t.me/med1917
skills before being handed a list of behaviors constituting the
skill.
151
Faculty are skeptical of assessment frameworks they
have not developed.
151
The video scenarios (Appendix 5.2,
Videos 5.1
–5.9) on the Expert Consult website can be used for
PDT for history taking, physical exam, and counseling.
There are several benefits of PDT
151
(Box 5.2). When faculty
agree on assessment criteria using a group process, it
validates the skills they believe are important. Sometimes
faculty question whether they are the only one who believes
a particular skill is essential for competence. Group process
during PDT often reveals that their peers also believe the skill
is essential. Additionally, faculty find it helpful to create and
use the list of behaviors as a guideline to inform their holistic
judgment and guide feedback content. Faculty describe how
PDT helps them approach direct observation in a more
standardized, systematic way, paying attention to a broader
array of skills, including interpersonal and communication
skills.
151
PDT also helps faculty assess skills underpinning
high-quality patient care that they might not have focused on
prior to training.
151
Faculty believe PDT helps them give
more effective feedback after direct observation by increasing
the breadth of skills they can discuss. PDT can also provide
them more granular vocabulary which helps them give more
specific, constructive feedback.
151
PDT can help faculty
deconstruct their holistic assessments to provide more
specific feedback.
151
The goal of PDT is not to create a
checklist that is used as a reductionistic, mindless tool.
Rather, the value lies in the process of creating the
framework to understand the components of a skill and
develop a shared mental model around a patient-focused
entrustment outcome.
https://t.me/med1917
Box 5.2
Benefits of Performance Dimension
Training and Frame of Reference Training
•. Creates a shared mental model for important
assessment criteria
•. Validates skills that faculty believe are required for
competence
•. Creates buy-in for skills originally not considered to
be important
•. Enables more standardized, systematic approach to
direct observation
•. Encourages attention to broader array of skills during
direct observation
•. Facilitates assessment of skills underpinning highquality patient care
•. Improves accuracy of entrustment supervision rating
•. Improves accuracy and specificity of narrative
comments
•. Increases breadth of skills discussed during feedback
•. Provides more granular vocabulary for feedback
•. Increases self-efficacy giving specific, constructive
feedback
•. Helps deconstruct holistic assessments for feedback
•. Helps faculty learn new skills or refresh previously
learned skills
•. Increases faculty mindfulness during patient care
•. Helps faculty identify gaps in the care they provide
to patients
Earlier in this chapter we described how direct observation
quality may be limited by deficiencies in faculty’s clinical
skills. PDT can improve assessment while also allowing
https://t.me/med1917
faculty to reflect on their own clinical skills. PDT can help
faculty improve their clinical skills, either through learning
new skills, identifying gaps in their own skills, reviewing
previously learned skills, or simply becoming more reflective
and mindful in their clinical care.
151
For example, faculty may
learn important skills about motivational interviewing that
they had not been taught prior to PDT. PDT creates potential
economies of scale by simultaneously developing faculty’s
own skills as clinicians and as assessors. This is particularly
appealing given the pressures to justify the cost of medical
education.
161–163
There is a profound lack of this type of
professional development for faculty.
Frame of Reference Training
Frame of reference training (FoRT) targets rating
accuracy.
156
,157
FoRT is an extension of PDT. Using the results
of PDT, the main goal of FoRT’ is to have faculty consistently
apply the performance criteria to distinguish between skill
levels. FoRT supports the developmental model of
competency-based medical education by helping faculty
distinguish between skill levels and identify where a learner
is on the “developmental spectrum.” Appendix 5.4 provides
an overview of the FoRT process.
Through FoRT, faculty define the minimal criteria for
satisfactory skills. We recommend faculty define satisfactory
as what is needed for safe, effective, patient-centered care
unsupervised.72 This definition of satisfactory is consistent
with the Institute of Medicine’s (now NAM) definition of
high-quality care and practices associated with improved
patient outcomes.
5,50–52
Alternatively, satisfactory can be
defined as the learner being able to competently perform the
https://t.me/med1917
skill unsupervised.
164
These criteria for competence then
anchor marginal and unsatisfactory performance. Once the
group defines marginal criteria, by default any lesser
performance is ineffective, not patient centered, and/or
unsafe (i.e., “unsatisfactory”). The next steps in FoRT are to
show additional videos of learners demonstrating different
levels of skill. Participants are asked to write their
observations (narrative assessment) and select an
entrustment rating. Participants share their observation and
ratings, and the facilitator provides the group with feedback
on the expert rating (and rationale) and narrative assessment.
Participants discuss differences between their observations,
narrative assessments, and ratings with those of the
facilitator or expert. This exercise helps faculty differentiate
learners with different skill levels.
There are multiple benefits of FoRT (Box 5.2
). FoRT
promotes deliberate practice of assessment skills.
165
Watching a clinical encounter performed at three learner skill
levels helps faculty discriminate between performance levels.
This is beneficial because assessors have difficulty
distinguishing between unsatisfactory and satisfactory
performance (calling for direct and indirect supervision,
respectively) or between satisfactory and aspirational
performance (no supervision needed).
165
Comparing and
contrasting learners across a video series helps faculty better
understand the range of behaviors for or variable execution
of a given skill.
165
For example, across the video series,
participants can see a range of how much and how well a
resident explores the physical, psychological, and emotional
impact of a symptom on a patient. Watching a three-video
series can also help faculty gain clarity about the importance
https://t.me/med1917
Соседние файлы в папке Библиотека им академика М.И. Перельмана