List of Figures
1.1The (general) human-machine closed loop model. The model’s signal processing + pattern recognition component, denoted in gray, is the component on which this monograph will focus (for more detail, see Figure 1.2). Within the scope of this monograph, the model’s domain of application is affective
computing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2 The signal processing + pattern recognition pipeline. . . . . . . . . . . . . . . . 16
2.1 Recordings of Heart Rate (HR), ElectroDermal Activity (EDA), and a person’s activity for a period of 30 minutes, in a real world setting. . . . . . . . . . . . . 31
3.1Left: The points indicate the electrodes that were placed on the face of the participants to determine the EMG signals. The EMG signals of the frontalis, corrugator supercilii, and zygomaticus major were respectively measured
through electrodes 1 − 2, 3 − 4, and 5 − 6. Right: The points at which the |
|
electrodes were placed on the hands of the participants to determine the EDA |
|
signal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
48 |
4.1 The skewness measure of the galvanic skin response / ElectroDermal Activity |
|
EDA) for each of the eight film clips. . . . . . . . . . . . . . . . . . . . . . . . . |
61 |
4.2The behavior of the mean EDA signal over time, for each of the eight film fragments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.3The behavior of the mean electromyography (EMG) signal of the frontalis
over time, for each of the eight film fragments. . . . . . . . . . . . . . . . . . . 64
4.4The behavior of the mean electromyography (EMG) signal of the corrugator supercilii over time, for each of the eight film fragments. . . . . . . . . . . . . 65
4.5The behavior of the mean electromyography (EMG) signal of the zygomaticus major over time, for each of the eight film fragments. . . . . . . . . . . . . . . 66
5.1A screendump of the Digital Rating System (DRS) used in this research; see Section 5.4. An IAPS picture (category: relaxed) is shown [374]. Below the 11 point (0-10) Likert scale with radio buttons is shown augmented with three
Self-Assessment Mannequin (SAM) images. With these images the experienced arousal was assessed as indicated by both the SAM images and the text “Calm vs. Excited scale”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
xi
Contents
5.2The processing scheme of Unobtrusive Sensing of Emotions (USE). It shows how the physiological signals (i.e., speech and the ECG), the emotions as denoted by people, personality traits, people’s gender, and the environment are all combined in one ANOVA. Age was determined but not processed. Note that the ANOVA can also be replaced by a classifier or an agent, as a module of an AmI [694].
Explanation of the abbreviations: ECG: electrocardiogram; HR: heart rate; F0: fundamental frequency of pitch; SD: standard deviation; MAD: mean abso-
lute deviation; and ANOVA: ANalysis Of VAriance. . . . . . . . . . . . . . . . 83
5.3Two samples of speech signals from the same person (an adult man) and their accompanying extracted fundamental frequencies of pitch (F0) (Hz), energy of speech (Pa), and intensity of air pressure (dB). In both cases, energy and intensity of speech show a similar behavior. The difference in variability of F0
between (a) and (b) indicates the difference in experienced emotions. . . . . . 85
5.4A schematic representation of an electrocardiogram (ECG) denoting four R- waves, from which three R-R intervals can be determined. Subsequently, the heart rate and its variance (denoted as standard deviation (SD), variability, or mean absolute deviation (MAD)) can be determined. . . . . . . . . . . . . . . 86
7.1The complete processing scheme, as applied in the current research. Legenda: EMG: electromyography EDA: electrodermal activity; ANOVA of
variance; LOOCV: leave-one-out cross validation . . . . . . . . . . . . . . . . . 119
7.2Samples of the electromyography (EMG) in µV of the frontalis, the corrugator supercilii, and the zygomaticus major as well as of the electrodermal activity (EDA) in µV , denoted by the skin conductance level (SCL). All these signals were recorded in parallel, with the same person. . . . . . . . . . . . . . . . . . 120
8.1Overview of both the design of the research and the relations (dotted lines) investigated. The two studies, SPS and RL, are indicated, each consisting of a happy and a stress/anxiety-inducing session. In addition, baseline measurements were done, before and after the two studies. . . . . . . . . . . . . . . . . 135
8.2Speech signal processing scheme, as applied in this research.
Abbreviations: F0: fundamental frequency, HF: high frequency. . . . . . . . . . 137
8.3A sample of the speech signal features of a Post-Traumatic Stress Disorder (PTSD) patient from the re-living (RL) study. The dotted lines denote the mean and +/- 1 standard deviation. The patient’s SUD scores for this sam- ple were: 9 (left) and 5 (right). Power (dB) (top) denotes the power and the
High Frequency (HF) power (dB) (bottom). . . . . . . . . . . . . . . . . . . . . 138
8.4Reported stress over time per session (i.e., anxiety triggering and happy) for
the Stress-Provoking Stories (SPS) study. . . . . . . . . . . . . . . . . . . . . . . 142
8.5Reported stress over time per session (i.e., anxiety triggering and happy) for
the Re-Living (RL) study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
9.1The overall relation between the reported Subjective Unit of Distress (SUD) and the relative correct classification using 11 principal components based on
28 parameters of speech features. . . . . . . . . . . . . . . . . . . . . . . . . . . 159
xii
Contents
10.1 Four hours ambulatory EDA recordings, with its minimum and mean baseline. 173 10.2 A 30 minute time window of an EDA signal, which is a part near the end of
the signal presented in Figure 10.1. Three close-ups around the event near 3.3 hours are presented in Figure 10.3. . . . . . . . . . . . . . . . . . . . . . . . . . 175
10.3Three close-ups around the event presented in Figure 10.2. The statistics accompanying the three close-ups can be found in Table 10.5. . . . . . . . . . . . 176
10.4A typical sample of lost data with an EDA signal, as frequently occurs in real-
world recordings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 A.1 Visualization of the first three principal components of all six possible com-
binations of two emotion classes. The emotion classes are plotted per two to facilitate the visual inspection. The plots illustrate how difficult it is to sep- arate even two emotion classes, where separating four emotion classes is the aim. However, note that the emotion category neutral can be best separated from the other three categories: mixed, negative, and positive emotions, as is
illustrated in b), c), and d). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
xiii
Contents
xiv