Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_2823_Библиотеки_им_академика_М_И_Перельмана
.pdf
Assigning Responsibility for Direct
Observation
Early in training, direct observation should be used to
identify learners who have outlier performance. Program
directors and faculty should “frontload” observation as part
of learners’ baseline needs assessment. This is especially
important when learners transition to new roles and
responsibilities (i.e., July in the United States and Canada,
which is the start of the academic year). Early observation
can identify learners who need additional support, coaching,
and remediation. Just four observations can detect outlier
performance.77 Over time, the focus of direct observation can
shift to assessment and feedback for ongoing skill
development. Most importantly, learners should be
evaluated multiple times, in multiple contexts, across
multiple evaluators, and longitudinally over time. Although
rater training can improve the quality of assessments, the
best validity evidence occurs when there is broad sampling of
skills.
At the programmatic level, it is helpful to identify how the
responsibility for direct observation snapshots can be shared.
Consider adopting a “divide and conquer” approach. Parse
out what skills will be observed and assessed on what
rotations. For example, faculty attending on the geriatric
service might be responsible for observing the geriatric
functional assessment. Faculty in the intensive care unit
might be responsible for assessing breaking bad news and
goals of care discussions. Outpatient preceptors could be
responsible for assessing agenda setting and the
musculoskeletal exam. Consider asking core faculty to
identify the skills that should be prioritized for direct
https://t.me/med1917
observation in their specialty. Including core faculty in this
way promotes buy-in to WBA because faculty then observe
skills that they believe are important. As previously
discussed, observations should be aligned with or inform
milestones or EPAs.
Program directors should decide who is responsible for
initiating observation snapshots: learners or faculty. There
are pros and cons to each approach. Placing the responsibility
for direct observation on faculty emphasizes that direct
observation is valued by the program. However, learners
may then feel that direct observation is happening “to them,”
not “for them” or “with them.” Placing the responsibility on
learners can give learners control and ownership of their skill
development. However, learners may become frustrated
when faculty do not observe them when requested. Ideally
faculty and learners should take mutual responsibility and
ownership of the process. That is, faculty initiate direct
observation and learners also ask to be observed when they
need feedback to further develop a clinical skill.
181
Tracking Observations
Programs should create a plan to monitor and track whether
observation is happening. Online evaluation systems
facilitate tracking and allow data to be aggregated by learner
and evaluator. If observations are still recorded on paper, a
process to tally observations is needed. A simple strategy is
posting a piece of paper in the rounding or precepting room
that lists all learners, the minimum number of snapshots
required for each learner, and a place for faculty to initial and
date each time they have done an observation. Smartphone
apps or forms linked to QR codes can also track observations.
https://t.me/med1917
One example is the System for Improving and Measuring
Procedural Learning (SIMPL) app that is designed for
observation of surgical procedures using an entrustment type
scale.
182
,183
This tool has a significant amount of research
behind it and is currently being widely piloted in the United
States. The ACGME also hosts the Direct Observation of
Clinical Care (DOCC) app.
184
Key Messages About Faculty
Development and Implementation
Direct observation/WBA, which has always been important
in medical education, is a key assessment approach in
competency-based education. Direct observation of clinical
skills followed by feedback is necessary for deliberate
practice and for high-quality supervision. Because there are
multiple threats to frequent, high-quality, valid assessments,
programs need to invest in faculty development to maximize
the effectiveness of direct observation. Faculty development
should utilize rater training techniques that help faculty
develop a shared mental model of clinical skills that is
aligned with high-quality patient-centered care. To improve
direct observation frequency, faculty development should
help faculty identify observation snapshots that are
meaningful for the learner and could improve patient care
quality. Programs need to discuss the importance of direct
observation and feedback with learners. Finally, faculty
development programs should include longitudinal training
and practice since direct observation, assessment, and
feedback are complex skills that require ongoing practice.
https://t.me/med1917
Annotated Bibliography
1. Anderson HL, Kurtz J, Kirkpatrick E.
Implementation and use of workplace-based
assessment in clinical learning environments: a
scoping review. Acad Med. 2021 Nov
1;96(11S):S164-S174.
doi:10.1097/ACM.0000000000004366. This is a
scoping review of the barriers and enablers of
WBA. Identified themes include lack of trainee
and assessor engagement in design, time
constraints of the clinical environment, and
distilling the complex language of competencybased assessments into terms and parameters
assessors and trainees can easily use. This
article discusses the importance of technology
solutions and areas for future research and
innovation.
2. Gingerich A, Kogan J, Yeates P, Govaerts M,
Holmboe E. Seeing the ‘black box’ differently:
assessor cognition from three research
perspectives. Med Educ. 2014 Nov;48(11):1055-
1068. doi:10.1111/medu.12546. This perspective
piece from an international group of
researchers presents key findings in assessor
cognition research focused on WBA. The piece
explores different perspectives for variability in
assessment judgments. Three prevailing
approaches to assessor cognition research are
reviewed: assessor as trainable, assessor as
fallible, and assessor as meaningfully
idiosyncratic. The implications for assessor
https://t.me/med1917
training/faculty development are addressed.
3. Kogan JR, Conforti L, Bernabeo E, Iobst W,
Holmboe E. Opening the black box of clinical
skills assessment via observation: a conceptual
model. Med Educ. 2011 Oct;45(10):1048–1060.
doi:10.1111/j.1365-2923.2011.04025.x. This is a
qualitative study using a grounded theory
approach that was designed to create a
conceptual framework identifying the factors
impacting faculty’s judgments and ratings of
residents after direct observation with patients.
Participants were 44 internal medicine faculty
outpatient preceptors from 16 internal
medicine residency programs in the United
States. Four factors were identified that
explained variability of faculty’s ratings of
residents: variable frames of reference, high
levels of inference, variable approaches to
translating observations to numerical ratings,
and institutional/cultural factors. This article
summarizes the findings in a conceptual model
that describes factors influencing the variability
of observations and judgment during
workplace-based assessment.
4. Kogan JR, Dine CJ, Conforti LN, Holmboe ES.
Can rater training improve the quality and
accuracy of workplace-based assessment
narrative comments and entrustment ratings?
A randomized controlled trial. Acad Med. 2023
Feb 1;98(2):237-247.
doi:10.1097/ACM.0000000000004819. This is a
https://t.me/med1917
multiinstitution, single blind RCT of a rater
training intervention using the techniques of
performance dimension training, frame of
reference training, and spaced learning. The
purpose of this study was to determine
whether rater training could improve WBA
narrative comment quality and accuracy. A
secondary aim was to assess impact on
entrustment rating accuracy. The quality and
specificity of narrative comments improved
with rater training; the effect was mitigated by
inappropriate stringency. Training improved
accuracy of prospective entrustmentsupervision ratings but the effect was more
limited.
5. Kogan JR, Hatala R, Hauer KE, Holmboe E.
Guidelines: the do’s, don’ts and don’t knows of
direct observation of clinical skills in medical
education. Perspect Med Educ. 2017
Oct;6(5):286-305. doi:10.1007/s40037-017-0376-7.
This evidence-informed guideline outlines
approaches faculty, programs, and learners can
take to improve the frequency and quality of
direct observation. The article includes factors
that undermine WBA and areas for future
research.
6. Pelgrim EA, Kramer AW, Mokkink HG, van
den Elsen L, Grol RPTM, van der Vleuten
CPM. In-training assessment using direct
observation of single-patient encounters: a
literature review. Adv Health Sci Educ Theory
https://t.me/med1917
Pract. 2011 Mar;16(1):131-142.
doi:10.1007/s10459-010-9235-6. This is a
systematic review that describes the feasibility,
reliability, validity, and educational effect of
workplace-based assessment instruments and
describes validity evidence of the mini-CEX.
The review also highlights the lack of research
on educational effects of direct observation
tools.
7. Ten Cate O, Carraccio C, Damodaran A, et al.
Entrustment decision making: extending
Miller’s pyramid. Acad Med. 2021 Feb
1;96(2):199-204.
doi:10.1097/ACM.0000000000003800. In this
perspective the authors discuss how assessing
is important in competency-based medical
education and discuss adding “trusted” to the
apex of Miller’s pyramid.
8. Ten Cate O, Schwartz A, Chen HC. Assessing
trainees and making entrustment decisions: on
the nature and use of entrustment-supervision
scales. Acad Med. 2020 Nov;95(11):1662-1669.
doi:10.1097/ACM.0000000000003427. In this
perspective the authors describe entrustment
scales and use of entrustment as a component
of WBA. The authors describe the distinction
between ad hoc entrustment decisions and
summative entrustment decisions. The authors
discuss prospective and retrospective
entrustment-supervision scales.
9. Young JQ, Sugarman R, Schwartz J, O’Sullivan
https://t.me/med1917
PS. Faculty and resident engagement with
workplace-based assessment tool: use of
implementation science to explore enablers and
barriers. Acad Med. 2020 Dec;95(12):1937-1944.
doi:10.1097/ACM.0000000000003543. This
qualitative study explores faculty and resident
enablers and barriers to the use of a direct
observation tool. Enabling factors include the
need for ongoing training, design features of
the assessment tool, predisposing beliefs, and
dedicated faculty time. Barriers include length
of the assessment form, discomfort with
feedback, and variability in the quality of
delivered feedback.
References
1. American Association of Medical Colleges.
Core Entrustable Professional Activities for
Entering Residency. Accessed January 16, 2023.
https://store.aamc.org/downloadable/download/sample/sample_id/63
2. Liaison Committee of Medical Education.
Functions and Structure of a Medical School.
Accessed January 16, 2023. http://lcme.org.
3. Accreditation Council for Graduate Medical
Education. Common Program Requirements.
Accessed January 16, 2023.
http://www.acgme.org.
4. American Board of Medical Specialties.
Accessed January 16, 2023.
http://www.abms.org.
https://t.me/med1917
5. Institute of Medicine. Crossing the Quality
Chasm: A New Health System for the 21st
Century. National Academy Press; 1999.
6. Carraccio C, Wolfsthal SD, Englander R,
Ferentz K, Martin C. Shifting paradigms: from
Flexner to competencies. Acad Med. 2002
May;77(5):361-367. doi:10.1097/00001888-
200205000-00003.
7. Govaerts MJB, van der Vleuten CPM,
Schuwirth LWT, Muijtjens AMM. Broadening
perspectives on clinical performance
assessment: rethinking the nature of in-training
assessment. Adv Health Sci Educ Theory Pract.
2007 May;12(2):239-260. doi:10.1007/s10459-
006-9043-1.
8. Swanwick T, Chana N. Workplace-based
assessment. Br J Hosp Med. 2009
May;70(5):290-293.
doi:10.12968/hmed.2009.70.5.42235.
9. Miller GE. The assessment of clinical
skills/competence/performance. Acad Med.
1990 Sep;65(9 Suppl):S63-S67.
doi:10.1097/00001888-199009000-00045.
10. Ten Cate O, Carraccio C, Damodaran A, et al.
Entrustment decision making: extending
Miller’s pyramid. Acad Med. 2021 Feb
1;96(2):199-204.
doi:10.1097/ACM.0000000000003800.
11. Ram P, van der Vleuten C, Rethans JJ, Grol R,
Aretz K. Assessment of practicing family
physicians: comparison of observation in a
https://t.me/med1917
multiple-station examination using
standardized patients with observation of
consultations in daily practice. Acad Med. 1999
Jan;74(1):62-69. doi:10.1097/00001888-
199901000-00020.
12. Kopelow ML, Schnabl GK, Hassard TH, et al.
Assessing practicing physicians in two settings
using standardized patients. Acad Med. 1992
Oct;67(10 Suppl):S19-S21.
doi:10.1097/00001888-199210000-00026.
13. Rethans JJ, Sturmans F, Drop R, van der
Vleuten C, Hobus P. Does competence of
general practitioners predict their
performance? Comparison between
examination setting and actual practice. BMJ.
1991 Nov 30;303(6814):1377-1380.
doi:10.1136/bmj.303.6814.1377.
14. Hodges B, Regehr G, McNaughton N,
Tiberius R, Hanson M. OSCE checklists do not
capture increasing levels of expertise. Acad
Med. 1999 Oct;74(10):1129-1134.
doi:10.1097/00001888-199910000-00017.
15. Regehr G, MacRae H, Reznick RK, Szalay D.
Comparing the psychometric properties of
checklists and global rating scales for assessing
performance on an OSCE-format examination.
Acad Med. 1998 Sep;73(9):993-997.
doi:10.1097/00001888-199809000-00020.
16. Hawkins R, MacKrell Gaglione M, LaDuca T,
et al. Assessment of patient management skills
and clinical skills of practising doctors using
https://t.me/med1917
Соседние файлы в папке Библиотека им академика М.И. Перельмана