116 S.B. Goldin
Designing and Implementing High Quality
Surgical Education Research
Developing Good Research Questions
Empirical research has been characterized by several attributes, with an important one being parsimony. Parsimony can be defined as the simplest application of a theory, or in the interpretation of data, in accordance with the rule of Ockham’s (also spelled Occam’s) razor. Ockham’s razor states that the explanation of any phenomenon should make as few assumptions as possible and eliminate those that make no difference to the hypothesis. Thus, when two or more hypotheses are equal in most respects, selecting the hypothesis to test that introduces the fewest assumptions and postulates the fewest entities while providing actionable results is preferred.
Good research questions have several attributes. They should seek new knowledge, be based upon the scientific method, build upon existing scientific knowledge, and pick up where existing theories and data end. Further, they should meet the criterion of “scientific equipoise,” meaning that among several alternatives (such as alternative curricula), the most preferable is not known, and thus research is required to identify the best curriculum. Good questions represent a specific, testable hypothesis that can be answered on the basis of collectable and analyzable data. These answers should provide information that is “generalizable,” meaning that it is applicable to a definable population of individuals, and not just the participants enrolled in the study. Consultation with experts trained in study design will help prevent common errors that include developing hypotheses that do not reflect scientific equipoise, answering questions about a specific group of individuals that are not generalizable to a larger population, and developing questions that cannot be answered because appropriate data cannot be collected.
Chapter 8. Surgical Educational Research: Getting Started |
117 |
Beginning the Study Design Process
Educational research must be designed, and this design process, as well as the implementation process, must be well documented. This process is the same for all research whether studying biological mechanisms in benchtop research, studying teaching methods, or studying participant attitudes. All research involves asking a question and obtaining an answer in a systematic and thoughtful way. Educational research focuses on describing and documenting the transmission of knowledge and competency, as well as understanding which teaching methods are best and can lead to better practice.
A summary of the steps involved in designing a research project is listed in Table 8.1. A summary of the features of rigorous studies is listed in Table 8.2. There are several aspects of these tables that deserve further comment.
Although the steps for designing and implementing a research project (Table 8.1) appear straightforward, their implementation can be fraught with error. Clinicians are usually in a good position to identify areas that need further study to clarify unknown concerns or advance the current understanding of a topic. Group meetings and conference attendance to discuss the literature about these concerns are often helpful. However, turning these good ideas into research generally requires interdisciplinary research teams.
Developing a Research Team
Once an area of study and a few candidate research questions are developed,a research team should be assembled.Research teams should contain individuals with various skill sets that can improve the overall project starting from step 1. The types of individuals invited to serve on the research team should bring expertise needed to develop and implement a rigorous study to answer the research question. As an example, suppose an academic surgeon wanted to study the quality
118 |
S.B. Goldin |
|
TABLE 8.1 Research steps |
||
|
|
|
Research steps in the |
|
|
scientific method |
Activities in surgical education research |
|
|
|
|
Define the question |
The question (or questions) should |
|
|
|
state clear goals |
|
|
The question (or questions) should offer a |
|
|
null and alternate hypothesis |
Gather information |
Review the literature |
|
and resources |
Have discussions with colleagues |
|
(observe) |
Network and have discussions at |
|
|
|
conferences |
Form hypothesis
Perform experiment and collect data
Analyze data
Interpret data and draw conclusions that serve as a starting point for a new hypothesis
Publish results
Carefully word all research questions or hypotheses in such a way that data may be collected and analyzed to provide answers/address hypotheses
Develop a research protocol that specifies all techniques or interventions
Obtain regulatory approval and funding for the research protocol
Pilot test data collection to refine the process From pilot test results, select the best data collection methods to provide appropriate data for analysis
Record data electronically and continue to steward data throughout the study
Once data collection is complete, perform statistical analysis on the data collected as described in the research protocol
Referring to the hypothesis (or hypotheses) set forth in the research protocol, conclude whether the evidence gathered supports or does not support each hypothesis
When early findings are available, present at peer-reviewed conferences and obtain feedback from colleagues early in interpretation
After returning from the conference, referring to the research protocol, immediately develop the full manuscript from the abstract presented at the conference. Submit for publication
Chapter 8. Surgical Educational Research: Getting Started |
119 |
||
TABLE 8.1 (continued) |
|
|
|
|
|
|
|
Research steps in the |
Activities in surgical education research |
|
|
scientific method |
|
|
|
|
|
|
|
|
|
Continue to publish results as set forth in |
|
|
|
the original research protocol |
|
|
|
If post-hoc questions come up that can be |
|
|
|
answered with the study dataset, obtain |
|
|
|
regulatory approval for continued post- |
|
|
|
hoc analysis, and develop a manuscript for |
|
|
|
publication |
|
Retest (frequently |
Offer to collaborate with other |
|
|
done by other |
scientists and educators, both internally |
|
|
scientists) |
and externally, to further develop |
|
|
|
|
measurement instruments, study designs, |
|
|
|
and statistical approaches, as well as |
|
|
|
further the common line of research |
|
TABLE 8.2 Features of rigorous studies |
|
||
|
|
|
|
1. |
Large numbers of participants |
|
|
2. |
Multi-institutional enrollment of participants |
|
|
3. |
A control or appropriate comparison group |
|
|
4. |
Measurement of an objective outcome other than self- |
|
|
|
assessments by participants. This may include: |
|
|
(a)Tests of knowledge
(b)Course grades
(c)Objective-structured clinical examinations, observations made by standardized patient interaction assessments of performance, or with real patients such as graded observation of patient interactions or patient feedback
(d)Objective clinical outcomes such as blood glucose control
(e)Performance with mannequins, computer simulations, or laboratory animals
(f)Psychological inventories
5. |
Measurement of one or more outcomes at least 1 month |
|
after the intervention |
6. |
Performance of the intervention more than once |
7. |
Estimate of statistical power |