Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_6027_Библиотеки_им_академика_М_И_Перельмана.pdf
116 S. Viswanathan
Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33, 587–606.
Gigerenzer, G. (2018). Statistical rituals: The replication delusion and how we got there. Advances
in
Methods and Practices in Psychological Science, 1, 198–218.
Groppe, D. M., Urbach, T. P., & Kutas, M. (2011). Mass univariate analysis of event-related brain
potentials/
Hommelsen, M., Viswanathan, S., & Daun, S. (2022). Robustness of individualized inferences
from longitudinal resting state EEG dynamics. European Journal of Neuroscience, 56, 1– 32.
Kanwisher, N. (2010). Functional specificity in the human brain: A window into the functional
architecture
of America, 107, 11163–11170.
Keil, A., Debener, S., Gratton, G., Junghofer, M., Kappenman, E. S., Luck, S. J., Luu, P., Miller,
G.
studies using electroencephalography and magnetoencephalography. Psychophysiology, 51,
1–21.
Keil, A., Bernat, E. M., Cohen, M. X., Ding, M., Fabiani, M., Gratton, G., Kappenman, E. S., Maris,
E.,
guidelines for studies using frequency domain and time-frequency domain analyses of neural
time series. Psychophysiology, 59, e14052.
King, J. R., & Dehaene, S. (2014). Characterizing the dynamics of mental representations: The
temporal
Lachaux, J.-P., Rodriguez, E., Martinerie, J., & Varela, F. J. (1999). Measuring phase synchrony in
brain
Lang, T. A., & Altman, D. G. (2015). Basic statistical reporting for articles published in biomedical
journals:
guidelines. International Journal of Nursing Studies, 52, 5– 9.
Luck, S. J., & Gaspelin, N. (2017). How to get statistically significant effects in any ERP
experiment
Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of EEG- and MEG-data.
Journal
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E. J. (2011). Erroneous analyses of interac-
tions in neuroscience: A problem of significance. Nature Neuroscience, 14, 1105–1107.
Ostasiewicz, W. (2014). The emergence of statistical science. Śląski Przegląd Statystyczny, 12,
76
Pernet, C. R., Chauveau, N., Gaspar, C., & Rousselet, G. A. (2011). LIMO EEG: A toolbox for
hierarchical
and Neuroscience, 2011, 831409.
Pernet, C. R., Latinus, M., Nichols, T. E., & Rousselet, G. A. (2015). Cluster-based computational
methods
study. Journal of Neuroscience Methods, 250, 85– 93.
Picton, T. W., Bentin, S., Berg, P., Donchin, E., Hillyard, S. A., Johnson, R., Miller, G. A., Ritter,
W.,
related potentials to study cognition: Recording standards and publication criteria. Psychophys-
iology, 37, 127– 152.
Stevens, S. S. (1968). Measurement, statistics and the schemapiric view. Science, 161, 849–856.
Van de Schoot, R., Depaoli, S., King, R., Kramer, B., Märtens, K., Tadesse, M. G., Vannucci, M.,
Gelman,
Nature Reviews Methods Primers, 1, 1 .
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and
purpose.
Wasserstein, R. L., Schirm,
American Statistician, 73, 1– 19.
Wickham, H.
fields I: A critical tutorial review. Psychophysiology, 48, 1711–1725.
of the mind. Proceedings of the National Academy of Sciences of the United States
A., & Yee, C. M. (2014). Committee report: Publication guidelines and recommendations for
Mathewson, K. E., Ward, R. T., & Weisz, N. (2022). Recommendations and publication
generalization method. Trends in Cognitive Sciences, 18, 203–210.
signals. Human Brain Mapping, 208, 194–208.
The “statistical analyses and methods in the published literature” or the SAMPL
(and why you shouldn’t). Psychophysiology, 54, 146–157.
of Neuroscience Methods, 164, 177–190.
–77.
LInear MOdeling of ElectroEncephaloGraphic data. Computational Intelligence
for mass univariate analyses of event-related brain potentials/fields: A simulation
Ruchkin, D. S., Rugg, M. D., & Taylor, M. J. (2000). Guidelines for using human event-
A., Veen, D., Willemsen, J., & Yau, C. (2021). Bayesian statistics and modelling.
The American Statistician, 70, 129–133.
A. L., & Lazar, N. A. (2019). Moving to a world beyond “p < 0.05”.
(2014). Tidy data. Journal of Statistical Software, 59, 1–23.
Chapter 10
Pilot Testing
Paulo Rodrigo Bazán
Abstract Pilot testing is a crucial preliminary phase in electroencephalography
(EEG)
analysis pipelines before full-scale data collection. This chapter discusses the significance of pilot testing in ensuring high-quality EEG data. First, we define what
pilot testing is and clarify what it is not. We then explore the various study
characteristics that can be assessed during the pilot phase, such as task parameters,
equipment setup, signal quality, and experimental procedures. The chapter further
examines the preparation required for the pilot phase, emphasizing the definition of
criteria to assess the experiment. Finally, we discuss the iterative nature of pilot
testing and its integration into the study preparation proces s.
Keywords Pilot testing · EEG studies · Experimental design · EEG data
acquis
studies, aimed at optimizing experimental design, data acquisition, and
ition · Equipment setup · Experimental procedures
10.1 What Pilot Testing Is
Pilot testing refers to the preliminary phase of a study; its purpose is to evaluate
several aspects of the study before fully investing in and proceeding with data
acquisition. It involves the internal checks and fine-tuning of the experiment,
which should be part of every research study (Barbosa et al., 2022; Boudewyn
et
al., 2023). In a broader sense, it evaluates the feasibility of the proposed study and
provi
des helpful information for future studies with similar tasks or techniques.
Therefore, it can be relevant as a study on its own, and there are even journals
dedicated to pilot studies (Lancaster, 2015).
Although it is common to find published articles with “pilot” or “feasibility” in
titles, the exact definition of these terms is not universally agreed. Some authors
their
use the terms as synonyms (Ruel et al., 2016), and others suggest that one is a subset
P. R. Bazán (*)
Brain Products GmbH, Gilching, Germany
e-mail:
paulo-rodrigo.bazan@brainproducts.com
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026
T.
Warbrick (ed.), The EEG Handbook,
https://doi.org/10.1007/978-3-032-20450-9_10
117
118 P. R. Bazán
of the other (Eldridge et al., 2016), and yet others suggest specific and mutually
exclusive definitions (Thabane et al.,
pilot
testing can be seen as a pretesting of a specific task, questionnaire, or data
acquisition instrument (van Teijlingen & Hundley, 2002). This pretesting focuses on
validation of such an instrument (Ruel et al., 2016).
the
In electroencephalography (EEG) studies, the pilot phase is used to optimize the
experiment, from the design and EEG data acquisition procedures to the analysis
pipeline. The aim is to ensure high-q uality EEG data and adequate control of the
experimental environment in a safe and efficient way (as covered in Chap. 16:
Practical Aspects of EEG Data Acquisition”). Here, piloting can be very helpful
“
to check if all the needed triggers are working, and if the needed synchrony between
equipment is being achieved (Boudewyn et al.,
improvem
initial check to confirm whether it is possible to study the research question with the
planned analysis method and experimental design. It is also possible to use pilot
testing to validate new EEG equipment or technologies, but this is beyond the scope
of this chapter. We will focus on pilot testing as part of every EEG study , as the
initial check and optimization of the planned experimental protocol.
ent of the analysis methods and codes. Furthermore, it also allows an
2010; O’Cathain et al., 2015). Additionally,
2023). It also allows testing and
10.1.1 Why Pilot Testing Matters
Imagine doing hundreds of data acquisition sessions over a few years, only to realize
that the EEG signal was too noisy. Or that the data cannot be analyzed because there
are significant jitters in the markers, improper task instructions, or incorrect parameters. This would be a disappointing outcome and a waste of valuable time and
resources. Although this is an extreme scenario, it illustrates how important pilot
testing can be, as it could easily detect and eventually solve such issues.
The pilot test is a tool to improve the quality of the research project: different
eristics can be assessed, and the best parameters can be selected. Addition-
charact
ally, pilot testing can provide a better estimate of logistical requirements and of the
resources needed, which are helpful for funding requests (van Teijlingen & Hundley,
2002). It can also evaluate other aspects, such as the recruitment steps, consent and
reten
tion rates, and randomization procedures when preparing clinical trials (Leon
et al.,
2011; Lancaster, 2015).
All this
can confirm whether the EEG feature of interest is elicited by the experiment design
and detected by the analysis pipeline. This increases the chance of being able to fully
execute the study as prereg istered (Paul et al.,
information is very relevant for preregistering your study. The pilot test
2021).
10 Pilot Testing 119
10.1.2 What Pilot Testing Is Not, and Important
Considerations
Pilot tests are not shortcuts, and they do not replace the necessary steps in preparing a
research study, such as reviewing the literature, defining a research question, designing the experiment, and planning the analysis to answer the question (Barbosa et al.,
2022). To properly plan and perform the pilot phase, an initial proposition of the
with a clear hypothesis is necessary. Piloting will help to re fine the experiment
study
design and the analysis. It can also help to determine whether the research question
can be assessed with the proposed methodology.
Pilot testing is not just a study with a small sample size. Its goals are different
from
those of the main study. For example, pilots should not be used for inferences
related to the main hypothesis to be tested in the main study (Leon et al.,
Lanca
ster, 2015). Variable estimates taken from pilot data will not have the required
sion due to the small sample. For example, it is generally no longer
preci
recommended to use pilot data to define the sample size for the study. As this relies
on estimating the effect size of the main outcome, this can yield an inappropriate
sample size calculation (Leon et al., 2011; In, 2017). Estimation of other parameters,
such
as the standard deviation of the variable of interest, will also be suboptimal
(Sim,
2019). However, there is still some debate as to whether some outcomes could
be considered to estimate sample size (Lewis et al., 2021).
Is it appropriate to include the pilot data in the final study analysis? There are two
to be considered here. First, it is likely that the pilotin g phase will result in
issues
adjustments to the experimental procedure. Therefore, data collected in the pilot tests
will be different from the data from the main study. This variation can have an
impact on the outcomes (Leon et al.,
pilot sample into the final study, the preliminary analysis must be unrelated
the
(orthogonal) to the study hypotheses (Barbosa et al.,
statistical
preliminary analysis of the pilot data involves checking the presence of the expected
EEG signature in the data and testing the desired analysis pipeline. Therefore, a
dedicated sample is needed for the pilot phase in most cases.
testing of the hypothesis with the pilot data are not recommended, the
2011; In, 2017). Second, to be able to include
2022). Although inferences and
2011;
10.2 How to Prepare and Run the Pilot Testing
Now that we know what pilot testing can and cannot do, we can discuss practical tips
for properly planning and executing the pilot phase. First, we need to define the main
goals of the pilot phase. We must consider whether the plan is to publish it or to use it
to define the parameters to be reported in the preregistration of the main study.
Alternatively, it could be executed only as an internal step. In any case, we must
define the characteristics to be checked and optimized for the experiment to work.
Once these are defined, we can choose measurable outcomes to assess them. This
120 P. R. Bazán
Preparing
Pilot Testing
Define the goals of pilot testing
(characteristics to be checked)
Main experiment characteristics
Signal Quality
Task Parameters
Instructions
Participant
Experience
Equipment Setup
Procedures
Questionnaires
Define measurable outcomes
(evaluation criteria)
Visual Inspection
Signal-to-Noise ratio
EEG analysis pipeline
results as expected
Number/Percentage
of excluded trials
Task analysis pipeline
as expected
Feedback from
participants
Marker Count
Marker delays and
jitter
Feedback from data
acquisition team
Time to setup the
experiment
Accidents/incidents
Total session time
Consistency and
reliability Metrics
Questionnaire analysis
pipeline as expected
Define the Pilot Testing Steps
Main stepsMain criteria
Testing characteristics
individually
Additional intermidiate steps
(gradually increment the pilot test)
Simulating a data
acquisition session
Define Pilot Sample(s)
(characteristics and size)
Main types of samples
You rs el f
Lab Members
Control Population
Experimental
Population(s)
Fig. 10.1 Preparing pilot testing flowchart. The flowchart illustrates the process of preparing pilot
testing, including defining the study’s goals, measurable outcomes, testing steps, and pilot sample
characteristics. The key experiment characteristics can be optimized (from signal quality to questionnaires) by assessing the suggested criteria (arrow connections). Almost all these goals can have
an impact on signal quality (except for the questionnaires). The goals and outcomes help define the
steps and the samples used in pilot testing. Different phases are associated with different sample
groups, including lab members, control populations, and experimental participants. These phases go
from testing the characteristics individually to simulating a data acquisition session for thorough
preparation
also requires clear evaluation criteria to confirm we can continue with our experiment. All these points will help design the pilot testing steps and the required pilot
samples. An overview of the piloting phase preparation is given in Fig.
steps
are further discussed below.
10.1. Its main
10 Pilot Testing 121
10.2.1 Define Characteristics and Evaluation Criteria
The pilot phase can involve variations of the experimental task or of the equipment
setup, which can be compared to decide which is more suitable. For this, we need to
know which characteristics we want to test. There is some room for exploration as
well, as one goal of the pilot tests is to evaluate factors that were not accounted for
during the design of the experiment. However, having a pilot phase without clear
outcomes will likely result in difficulties in interpreting the pilot data. It is important
to define the criteria that will be used to conclude the pilot and proceed with the next
research phases. These criteria will prevent both proceeding without a proper
experimental protocol and being trapped in an optimization loop (there will always
be points to be improved in scientific progress). Here we highlight some of the main
characteristics to be assessed in the pilot phase of EEG studies (see Fig.
t criteria to assess them.
sugges
10.2.1.1 Signal Quality
This is a key point for all EEG studies, as the signal quality can be affected by many
facto
rs. We want to ensure a low noise level in the data acquisition environment,
which can be evaluated first with a visual inspection of the signal. It is helpful to
monitor the signal during the pilot test to identify the source of the noise. Monitoring
the filtered signal can provide an initial idea of whether the frequency filters can
handle the noise. Using Fast-Fourier Transform will also be helpful to identify the
source, as the main noise frequencies will be highlighted. The signal quality evaluation should also check whether the expected EEG feature of interest is observed.
For example, when doing an oddball task, a P300 evoked potential should be
detected; when using visual stimuli, visually evoked potentials should be present,
such as P100.
An objective way to assess EEG signal quality is the signal-to-noise ratio. The
old for acceptable signal quality depends on the signal of interest and on the
thresh
planned analysis. For example, if a study is focused on the alpha and beta frequency
range and there is some 50 Hz line noise, it may not impact the analysis. Therefore,
running the desired analysis pipeline is important as it will show if it can properly
handle the observed noise. Further, by running the pipeline, we evaluate how many
trials were excluded in preprocessing. Both the number and the percentage are
relevant metrics to assess signal quality. We want to make sure that a minimum
number of trials is available for analysis, but we also want to make the experiment
efficient. This is useful when evaluating the feasibility of data acquisition in a noisy
environment, or during a task that is expected to elicit more noise due to movement.
Most aspects
the signal quality must also be assessed during the optimization of such characteristics, as discussed below.
of the experiment can have an impact on the EEG signal. Therefore,
10.1) and
122 P. R. Bazán
10.2.1.2 Task Parameters
Many characteristics of the stimuli and task parameters can be adjusted during pilot
testing.
For example, task duration, number of trials, stimulus size or volume,
interval between trials, difficulty, the sequence of the stimuli, and the interval
between stimuli and between trials (not just the length but the distribution of the
interval). We must define what will be carefully tuned. As an initial step, it is
important to confirm that the participants can understand and perform the task.
Specific experimental tasks can be programmed and used in the pilot phase to
optimize these parameters. For example, the size of a cue stimulus can be varied,
and the performance associated with each size can be evaluated. This will determine
the optimal size to use in the main study.
One way to evaluate the task parameters is by assessing behavioral performance.
number and the percentage of excluded trials are again relevant. Task results are
The
also important, such as reaction times and the number and percentage of correct
responses. These metrics help confirm that the outcomes of the task are consistently
and properly recorded. For example, a big variation over time or between experimental blocks may be related to fatigue or learning effects, or it may indicate
improper coding of the task responses (Barbos a et al., 2022).
One helpful tool to optimize the task, as well as other parts of the experiment, is a
structured questionnaire for the pilot phase. The questionnaire should include open
questions to allow participants to express their overall perception and experience of
the study (Barbosa et al., 2022). Open questions can assess the strategy used by the
participa
nts, as sometimes they can find an alternative and easier way to do the task,
such as blurring their vision during a Stroop test. The questionnaire should also
contain objective and scale rating questions to quantify and compare the tested
experimental variations.
The EEG signal quality should also be evaluated during the adjustment of task
parameters. For example, fatigue, learning, and the strategy used by the participant
can also affect the event-related potentials and the EEG results overall. Further, the
EEG signal-to-noise ratio and the analysis pipeline provide important information
for defining the number of trials.
10.2.1.3 Instructions
How well
participants understand the task will have an impact on their performance.
Therefore, how the task is explained to them is crucial and must be optimized.
Failures in task comprehension or adopting strategies other than the one intended for
the task can usually be detected in the task results, which will be outside of the
expected range. Dedicated questionnaires are also effective for detecting communication issues. For example, we can ask the participant if the instructions were clear
and to rate the clarity of the instructions. It is important to check the EEG signal and
expected results, as miscommunication can lead to different cognitive components
being involved in the task than originally planned in the study design (Amaro &
Barker,
2006).
10 Pilot Testing 123
10.2.1.4 Participant Experience
It is important to make the experiment as pleasant as possible and always provide a
kind
and respectful environment (Barbosa et al., 2022). This can be measured via a
dedicated questionnaire, asking the participants about their comfort during or their
satisfaction with the experiment. The EEG data can be inspected for indicators of
emotional state or alertness, for example.
10.2.1.5 Equipment Setup
Several aspects of equipment setup play a role in a successful experiment, from the
position
of the equipment to the recording software parameters. These should be
considered during pilot testing.
In general, it is important to ensure correct and precise triggers (as discussed in
Chap.
14: “Triggers”). For this, the total marker count and the jitter in the markers
are
effective metrics. The count must match exactly the intended number (e.g.,
number of trials). The maximum acceptable jitter depends on the EEG analysis
pipeline, although it should be very small overall. If these metrics are outside of the
expected levels, the trigger connections should be checked, as well as the task
programming.
The setup should also be practical and efficient. We can evaluate it by measuring
time it takes to set up the experiment, as well as the total session time. Once
the
optimized, these will be informative for scheduling the data acquisition of the study.
The input from the research group can help identify what needs improvement. The
research group can use a questionnaire (e.g., grade the equipment setup) or a
checklist of characteristics to discuss together if adjustments are needed.
The equipment positioning and the data acquisition setup can have an impact on
the
data quality, and therefore on the study outcomes. Visual inspection of the data
quality and signal-to-noise ratio can be used as outcomes. If there is significant noise
in the data, try to adjust and identify the source, either by checking nearby electrical
equipment or confirming if the noise is specific to the room. Also check if the data
acquisition notebook is connected to the power supply (this can, in some cases,
generate noise). Further tips to optimize equipment positioning are presented in
Chap.
16: “Practical Aspects of EEG Data Acquisition.”
Further, it is important to consider the safety of the setup. It is expected that no
accide
nt or incident will happen, but we must control and register them properly. In
case something happens, having detailed information will allow more effective
adjustments to the setup.
10.2.1.6 Procedures
The overall protocol or study work flow can be optimized during pilot testing. A
ul tool is feedback from the participants about their experience during the
helpf
experiment. The goal is to evaluate the full procedure, including the instructions
124 P. R. Bazán
given to the participant before the day of the experiment, the routine on the data
acquisition day, and the entire experience of the participant during the experiment.
Similarly, the feedback from the research staff during the tests is also very
relevant. Confirm that the role of each person during the data acquisition is clear,
and that each lab member is comfortable with their role—all the people involved in
the experiment should feel respected, not only the participant. Aside from adjustments to the protocol, consider increasing the personnel involved in data acquisition
if necessary. Additional training can be provided to lab members if needed. The
initial pilot tests, possibly with lab members as participants, can be a good training
opportunity. However, it is important to simulate the real data acquisition and avoid
different procedures due to knowing the participant/colleague.
As additional outcomes, you can assess:
. Data loss during the pilot testing
. Quality of the acquired data
. Consistency of experimental procedures between different pilot test data
acquis
itions
. The duration of the experimental session and of each step separately.
10.2.1.7 Questionnaires
Although questionnaires do not directly influence the EEG signal quality, they can
ement the neurophysiological measures and the task performance. For exam-
compl
ple, it can help during participant screening to control variables that can influence the
study outcomes. A dedicated short, structured interview or questionnaire can be used
to confirm that the questions are clear and properly understood by the participants
(van Teijlingen & Hundley, 2002). If necessary, the study questionnaire can be
adjus
ted and the testing reiterated. Pilot studies can also be used to assess the
reliability of the questionnaire as part of the validation process (Bujang et al.,
2022). It is also recommended to have different questions to evaluate the same
re. This will allow consistency and reliability of the questionnaire to be assessed
featu
to see if the desired aspect is really being measured by the questionnaire.
10.2.2 Define the Different Steps of Your Pilot Tests
The pilot tests are best optimized when organized in several steps (Fig. 10.2). For
example, an initial step would be to check the main characteristics of the experiment
separately, such as the individual instruments used in the experiment, the equipment
setup, and the tasks and stimuli being used. We can check the noise level in the EEG
recorded in the experiment room. We can, in parallel, check the task parameters
considering only the behavior. The preliminary testing steps are especially valuable
when either the task or the instruments used are not validated within the study
10 Pilot Testing 125
Preparing
Pilot Testing
Define the goals of pilot testing
(characteristics to be checked)
Define measurable outcomes
(evaluation criteria)
Define the Pilot Testing Steps
Define Pilot Sample(s)
(characteristics and size)
Doing
Pilot Testing
Testing characteristics/
criteria individually
Testing characteristics in
partial setups
Test together using the draft
version of the experiment
procedure
Simulating a data acquisition
session
After
Pilot Testing
Do study
Preregistration
Conduct the Study
Publish the Pilot
Testing Results
Fig. 10.2 Flowchart of the main phases in pilot testing. Three phases of pilot testing are outlined:
preparation, execution, and post-testing steps. The diagram presents the steps within each phase and
their order. The preparation phase (preparing) includes defining study goals, evaluation criteria,
steps of the testing, and samples used. The execution (doing) requires testing individual characteristics, partial setups, testing the experiment procedure, and then simulating data acquisition session.
It can require a return to previous steps or even to the previous phase to prepare new pilot tests. The
steps after pilot testing involve publishing the results (optional), preregistering the study, and then
conducting the full study. Optional steps and returns to previous steps are presented in dashed lines
population (Nebe et al., 2023 ). Once each main characteristic is tested, combined
partial setups can be made. For example, testing the task with the EEG to check the
triggers and the task parameters based on the EEG signal. Then the steps progress by
incorporating more parts of the experiment and testing the initial versions of the
protocol. The last step is simulating a complete data acquisition session, to confirm
the experiment is ready.
Pilot testing steps are executed iteratively until the desired optimization is
achieved (Barbosa et al., 2022). The results from one step can lead to a return to
previ
ous steps. For example, after a partial setup assessing the EEG signal during the
task, we may need to go back to adjusting the task by itself. The pilot results can even
suggest a return to prior study phases, such as refining the research question, if the
pilot testing results suggest it cannot be properly assessed.
Please keep
in mind that the steps can be adjusted to each type of study. For
example, when doing EEG-fMRI, it is recommended to divide the signal quality
check into three steps: First, check the signal outside the scanner room, to test that
the triggers and clock synchrony are correct; then test inside the scanner room
without an fMRI sequence, to evalua te the artifacts generated by the MR; and then
test the signal with a sequence (Störmer et al.,
2016).