Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_112_библиотеки_им_акад_М_И_Перельмана
.pdf
Use Learning Analytics and Big Data
Learning analytics using both local and national data (e.g., “big
data”) must be embedded in programs of assessment.46 Learning
analytics refers to “the interpretation of a wide range of data
produced by and gathered on behalf of students in order to assess
academic progress, predict future performance, and spot potential
issues.”75 Programs must become more facile using all their available
quantitative and qualitative assessment data to create a meaningful
dashboard of professional development for program leaders and
learners. Most learning management systems can create these
dashboards, which provide both criterion and normative-referenced
snapshots of learner development.76 However, learning analytics and
“big data” must account for issues of bias that can be unwittingly
incorporated into assessment analytics.
75,76
This will be especially
important in using artificial intelligence (AI), machine learning (ML),
and large language machine learning (LLML) techniques.
AI/ML/LLML holds tremendous promise if used wisely and
thoughtfully within and across programs and will be the next
frontier in assessment to identify patterns of learners’ developmental
trajectories more accurately. For example, locally collected GME
assessment data can now be compared to national GME “big data.”
Residency programs can use Milestones’ nationally based predictive
probability values (PPVs) to identify specific subcompetencies where
individual learners may be struggling.72 However, the critical point
is that the quality of the assessment is defined by the interpretation of
the data. A PPV, for example, does not itself tell you what is actually
happening with the learner. It is simply a quantitative “signal” that
something concerning may be present. Proper interpretation of this
“signal” requires further investigation and conversation with the
learner and the educational team, using mostly qualitative (narrative)
data. Chapter 16 provides an example of what a longitudinal
https://t.me/med1917
dashboard can look like to guide programs; such a dashboard should
contain both quantitative and qualitative data elements.
Explicitly Define Assessors’ Roles and
Responsibilities in the Assessment System
Thus far we have focused on the purpose and role of assessment at
the program level. However, the majority of WBAs are performed by
individual faculty focused on individual learners who are fed into
programs of assessment. And here we define “faculty” more broadly
to include any healthcare professional involved in the training of
another healthcare professional. It is therefore imperative that
everyone involved in the assessment program clearly understands
their roles and responsibilities.
All frontline health professionals who contribute to learners’
professional development are the backbone of any program of
assessment. While physician faculty will likely perform most of the
assessments for physician training, for example, training programs
must also look to learners’ interprofessional faculty colleagues as a
rich source of education and assessment. The core assessment
responsibilities for faculty are (1) perform direct observation of
clinical skills; (2) provide rich narrative descriptions of performance
and accurate ratings; (3) provide ongoing feedback and coaching; (4)
provide robust assessment information to program leadership and
the CCC to support the learner’s professional development; and (5)
ensure that all patients receive safe, effective, equitable, patientcentered care through appropriate learner supervision.
There are multiple issues in faculty assessment that must be
considered. In addition to those already highlighted, other issues
include poor reliability and accuracy, raters’ own clinical abilities,
and inaccurate use of inference
77
(see Chapters 1, 2, and 5). As a
result, learners must, often independently, make sense of disparate
https://t.me/med1917
assessments and feedback from faculty. What feedback should
learners trust and prioritize? Is feedback grounded in EBP and based
on sufficient interaction time to capture the learner’s abilities? Are
assessments affected by bias? The most frequently proposed solution
to poor reliability and accuracy is ensuring that enough assessments
are performed by multiple raters completing multiple assessments
longitudinally. Obtaining multiple assessments leverages
psychometric science and, from a programmatic perspective,
supports better summative entrustment judgments of learners if
multiple assessments are collected longitudinally. However,
supervision decisions based on inaccurate and unreliable
assessments by individual faculty may lead to poor patient care
when faculty entrust a learner to function with less oversight when,
in reality, more is needed. To ensure the probability that high-quality
care is delivered as frequently as possible, faculty must make valid
and accurate assessments. How can the current situation be
improved?
Assessors, particularly faculty, need ongoing training in
assessment. Unfortunately, a certain level of nihilism surrounds
faculty development in assessment. Time, financial costs, and
perceived ineffectiveness are often cited as the major barriers to
assessment training. However, if assessors are not trained, what are
the costs to patients and learners? Patients, followed next in line by
learners, are the stakeholders most affected by poor assessment
practices and unwarranted variation in assessments. Most clinical
supervisors have to make judgments and provide feedback even if
they don’t like it. An investment in faculty training is an investment
in efficiency and effectiveness in their educational role while
concomitantly having to manage and provide patient care in busy
clinical settings. One factor that hampers faculty development
process even further is the experience faculty had with education
when they were the learners themselves. This sometimes leads them
https://t.me/med1917
to believe they understand education because they have experienced
it and with the mindset, “I turned out OK, right?”
For example, a faculty member with years of experience reading
the literature doesn’t alone make them a clinician specialist,
especially if that experience dates from decades ago and therefore
runs the risk of being outdated. After all, we don’t practice medicine
like we did 25 years ago, so why would we think we should practice
education like we did 25 years ago? The failure of faculty to
recognize and acknowledge that things have changed, and will
continue to change, requires an investment in faculty development to
improve efficiency and effectiveness in both educational and clinical
practice.
Faculty development in assessment is hard and requires sustained,
ongoing effort. Much like the challenges of implicit bias, where
patterns of behavior become subconscious, ingrained habits, faculty
members’ assessment habits can follow a similar pattern. Faculty
very often use themselves as their primary frame of reference, or
standard, when assessing learners’ clinical skills77 (see Chapter 5).
However, faculty’s abilities in the very clinical skills they assess are
variable. Going forward, the primary frame of reference, or minimal
standard, for WBAs should be whether the patient received safe,
effective, equitable, and patient-centered care.
78
Faculty development programs should also teach faculty how to
account for contextual factors that affect a learner’s ability to provide
equitable, high-quality, safe care (Fig. 3.5). This does not mean there
should always be an adjustment of an individual learner’s
assessment, as there will be times when the learner must adapt to the
circumstances to provide effective, safe, equitable, and patientcentered care. For example, residents will need to adapt their
counseling for patients with lower health literacy to meet those
patients’ needs, even if the healthcare setting lacks helpful resources.
https://t.me/med1917
Assessors and programs should assess learner adaptability as a core
competency. Chapter 5 provides a wealth of suggested faculty
development approaches to improve assessment by faculty.
FIG. 3.5 Assessment: complex and situated in context.
Similar to addressing bias in assessment, improving faculty
development in assessment must also be treated as a translational
activity. McGaghie adapted the clinical translation framework for
medical education, and all training programs should keep the
translational steps and goals in mind as it creates, implements, and
continuously refines its program of assessment.29 It is well past time
to make a more sustained effort in faculty development, grounded in
evidence, to concomitantly improve assessment for the purpose of
improving professional development and clinical care.
Address Bias in Assessment
While I have previously described some specific issues regarding
bias, I want to emphasize the urgent need to address the many facets
https://t.me/med1917
of bias in medical education that threaten learners’ professional
development: bias and inequity in the learning environment, bias
inherent in the assessment instruments, bias on the part of individual
evaluators who assess learners, and bias on the part of CCCs or other
groups that review assessments and make high-stakes decisions.
79
Going forward, as described earlier, assessment tools should include
additional domains essential to the physician role, such as advocacy
for at-risk populations. Furthermore, it will be important to ensure
that the definition of competence is not biased. For example, there is
increasing concern that the assessment of professionalism is
predicated on gender- and race-normative constructs and
definitions.
80
Despite the current lack of robust evidence supporting
interventions to reduce bias and prejudice, faculty development
remains essential to address the multiple types and foci of bias.
81
Programs can attend to the individual or interpersonal effects of bias
by offering faculty development about the history of bias and racism
in medicine, writing bias-free narratives, and engaging in individual
mindfulness practice to mitigate the effects of assessors’ own bias.
The scope and focus of bias (i.e., which URiM groups are most
impacted by the bias) will vary by country.
Medical education should leverage existing literature around biasreducing interventions, despite current limitations in the evidence
within medical education, to build on and study the effectiveness of
specific techniques from bias reduction research in patient care.
Intervention procedures that induce threat or emotions in
participants are minimally effective. A recent metaanalysis found
that one-time, limited-focus interventions had little impact on
reducing bias or prejudice.55 This should not be surprising as implicit
biases and prejudices often become personal habits that require
repeated attention and practice to change.82 Multifaceted and
https://t.me/med1917
longitudinal interventions show more promise and should be
studied in medical education.83 Table 3.4 lists five promising
strategies along with potential examples in medical education.
83
Table 3.4
Five Possible Strategies Programs and Faculty Can Try to Reduce Bias
in Assessment
Strategy Description Assessment Example
Stereotype
replacement
Recognizing
when a
stereotype has
been activated,
thinking about
why, and then
actively
substituting
nonstereotypical
thoughts
When completing a narrative
assessment of a female learner,
the assessor stops to consider if
they may be using gender-laden
language or uses an online tool to
assess for gender bias. If bias is
found, the assessor substitutes
evidence-based behavioral skills
that are more neutral.
Perspective
taking
Considering
what it would
be like to be a
member of the
minoritized
group
During rounds, faculty witness a
difficult interaction between a
learner from a URiM group with
a discriminatory patient. Faculty
should ask themselves: What
must that be like for the learner?
How will I intervene in this
situation?
Individuation Recognizing
when you have
stereotyped
someone
according to
their group
affiliation and
instead
thinking about
what makes
A faculty member watches a
learner from another country
struggle to interview a patient
with a possible sexually
transmitted disease and initially
stereotypes the learner as from a
group “uncomfortable talking
about sex.” Instead, the faculty
sees an individual learner
struggling and seeks to
https://t.me/med1917
them an
individual
understand why they are
struggling as an individual.
Counterstereotypic
imaging
Imagining an
individual or
situation that
counteracts a
stereotypical
reaction in
detail
A faculty member starts with an
assumption that women are not
strong enough to perform
orthopedic procedures and then
instead thinks about successful
women who are orthopedic
surgeons
Increased
opportunities
for contact
Increasing
opportunities
for contact
with members
of a
stereotyped
group
Programs and faculty can spend
meaningful time with URiM
trainees to listen and learn more
about their lived experiences and
their path to the current training
program.
Adapted from Holmboe ES, Osman NY, Murphy CM, Kogan JR. The urgency of now:
rethinking and improving assessment practices in medical education programs. Acad Med.
2023 Apr 18. doi:10.1097/ACM.0000000000005251
.
URiM, Underrepresented in medicine.
Educators should also be aware of other cognitive biases that affect
assessment. These include the well-known correlational and
distributive type rating errors, cognitive load, and many more.
84
Correlational errors refer to the phenomenon where performance on
one competency domain significantly influences the rating in other
competency domains. An example of such an error is the halo effect,
where everything is rated high based on perceived strong
performance in one domain such as medical knowledge.
Distributional errors involve raters using limited ranges of a scale
such as leniency error (everything rated higher) or stringency error
(everything rated lower). Cognitive load entails asking the assessor
to judge too many items in a short period of time. Dickey and
colleagues provide a nice overview of other cognitive biases, and
https://t.me/med1917
Chapters 4 and 5 cover other cognitive biases in more detail.
84
Research in reducing bias through faculty development and
program evaluation (see Chapter 16) should use the translational
science approach described earlier (i.e., bench to bedside). The first
step is to identify whether there is bias in the assessment program.
Programs can examine their assessment data (ratings and bias-laden
narrative comments) looking for differences in learners by identity
subgroups compared to the majority group. CCCs can ask
nonparticipant observers to watch and listen for bias-laden terms,
descriptions, or decisions. Once a needs assessment has occurred, T1
studies can be done to identify theory-supported strategies (Fig. 3.6)
to reduce implicit bias among educators. Future efforts might
specifically teach learning strategies to reduce implicit bias in direct
observation assessments. Implicit prejudice toward patients, for
example, manifests itself in verbal and nonverbal communication
behaviors. The same may hold true between learners and faculty and
is an opportunity to target communication-based interventions with
faculty to reduce implicit bias. T2 studies target effectiveness among
a larger group of educators and other assessments. Finally, T3
studies target wider implementation as the new assessment practice
designed to reduce implicit bias is embedded across the assessment
program. We have much to learn about how to address bias most
effectively in assessment, but early research in clinical practice
provides a foundation to design faculty development around more
effective communication strategies in assessment.
https://t.me/med1917
FIG. 3.6 Health professions education as translational science. (From
McGaghie WC, Barsuk JH, Wayne DB, eds. Comprehensive Healthcare
Simulation: Mastery Learning in Health Professions Education. Springer;
2020.)
Recognize the Importance of Coproduction
With Learners
For programmatic assessment to be fully effective, learners must be
active participants with individual agency. Medical students,
residents, fellows, and other health professionals learn in, and sit at,
the center of bidirectional and mutually beneficial relationships with
the leaders and faculty of their training programs. These
relationships require a coproduction mindset. In a coproduction
mindset, assessments take place in a psychologically safe
environment in which assessments are predominantly done with
learners instead of to them.
36
Coproduction has gained a strong foothold in clinical care.
Coproduction of healthcare is defined as “the interdependent work of
users and professionals to design, create, develop, deliver, assess,
and improve the relationships and actions that contribute to the
health of individuals and populations.”39 Recently, Englander and
https://t.me/med1917
Соседние файлы в папке Библиотека им академика М.И. Перельмана