- •Stellingen
- •Propositions
- •List of Figures
- •List of Tables
- •1 Introduction
- •Introduction
- •Affect, emotion, and related constructs
- •Affective Computing: A concise overview
- •The closed loop model
- •Three disciplines
- •Human-Computer Interaction (HCI)
- •Health Informatics
- •Three disciplines, one family
- •Outline
- •2 A review of Affective Computing
- •Introduction
- •Vision
- •Speech
- •Biosignals
- •A review
- •Time for a change
- •3 Statistical moments as signal features
- •Introduction
- •Emotion
- •Measures of affect
- •Affective wearables
- •Experiment
- •Participants
- •Equipment and materials
- •Procedure
- •Data reduction
- •Results
- •Discussion
- •Comparison with the literature
- •Use in products
- •4 Time windows and event-related responses
- •Introduction
- •Data reduction
- •Results
- •Mapping events on signals
- •Discussion and conclusion
- •Interpreting the signals measured
- •Looking back and forth
- •5 Emotion models, environment, personality, and demographics
- •Introduction
- •Emotions
- •Modeling emotion
- •Ubiquitous signals of emotion
- •Method
- •Participants
- •International Affective Picture System (IAPS)
- •Digital Rating System (DRS)
- •Signal processing
- •Signal selection
- •Speech signal
- •Heart rate variability (HRV) extraction
- •Normalization
- •Results
- •Considerations with the analysis
- •The (dimensional) valence-arousal (VA) model
- •The six basic emotions
- •The valence-arousal (VA) model versus basic emotions
- •Discussion
- •Conclusion
- •6 Static versus dynamic stimuli
- •Introduction
- •Emotion
- •Method
- •Preparation for analysis
- •Results
- •Considerations with the analysis
- •The (dimensional) valence-arousal (VA) model
- •The six basic emotions
- •The valence-arousal (VA) model versus basic emotions
- •Static versus dynamic stimuli
- •Conclusion
- •IV. Towards affective computing
- •Introduction
- •Data set
- •Procedure
- •Preprocessing
- •Normalization
- •Baseline matrix
- •Feature selection
- •k-Nearest Neighbors (k-NN)
- •Support vector machines (SVM)
- •Multi-Layer Perceptron (MLP) neural network
- •Discussion
- •Conclusions
- •8 Two clinical case studies on bimodal health-related stress assessment
- •Introduction
- •Post-Traumatic Stress Disorder (PTSD)
- •Storytelling and reliving the past
- •Emotion detection by means of speech signal analysis
- •The Subjective Unit of Distress (SUD)
- •Design and procedure
- •Features extracted from the speech signal
- •Results
- •Results of the Stress-Provoking Story (SPS) sessions
- •Results of the Re-Living (RL) sessions
- •Overview of the features
- •Discussion
- •Stress-Provoking Stories (SPS) study
- •Re-Living (RL) study
- •Stress-Provoking Stories (SPS) versus Re-Living (RL)
- •Conclusions
- •9 Cross-validation of bimodal health-related stress assessment
- •Introduction
- •Speech signal processing
- •Outlier removal
- •Parameter selection
- •Dimensionality Reduction
- •k-Nearest Neighbors (k-NN)
- •Support vector machines (SVM)
- •Multi-Layer Perceptron (MLP) neural network
- •Results
- •Cross-validation
- •Assessment of the experimental design
- •Discussion
- •Conclusion
- •10 Guidelines for ASP
- •Introduction
- •Signal processing guidelines
- •Physical sensing characteristics
- •Temporal construction
- •Normalization
- •Context
- •Pattern recognition guidelines
- •Validation
- •Triangulation
- •Conclusion
- •11 Discussion
- •Introduction
- •Hot topics: On the value of this monograph
- •Applications: Here and now!
- •TV experience
- •Knowledge representations
- •Computer-Aided Diagnosis (CAD)
- •Visions of the future
- •Robot nannies
- •Digital Human Model
- •Conclusion
- •Bibliography
- •Summary
- •Samenvatting
- •Dankwoord
- •Curriculum Vitae
- •Publications and Patents: A selection
- •Publications
- •Patents
- •SIKS Dissertation Series
1
Introduction
Abstract
The quest towards an in-depth understanding of affective computing begins here. This is needed as advances in computing and electrical engineering seem to show that the unthinkable (e.g., huggable computers) is possible (in time). I will start with a brief general introduction in Section 1.1. Subsequently, Sections 1.2–1.4 will introduce three core elements of this monograph: I) Affect, emotion, and related constructs, II) affective computing, and III) Affective Signal Processing (ASP). Next, in Section 1.5, the working model used in this monograph will be presented: a closed loop model. The model’s signal processing and pattern recognition pipeline will be discussed, as this forms the (technical) foundation of this monograph. Section 1.6 will denote the relevance of ASP for computer science, as will be illustrated through three of its disciplines: human-computer interaction, artificial intelligence, and health informatics. This provides us with the ingredients for the quest for guidelines for ASP as described in this monograph. As such, I hope that this monograph will become a springboard for research on and applications of affective computing. I will end with an outline of this monograph.
Parts of this chapter are taken from:
Broek, E.L. van den, Nijholt, A., & Westerink, J.H.D.M. (2010). Unveiling Affective Signals. In E. Barakova, B. de Ruyter, and A.J. Spink (Eds.), ACM Proceedings of Measuring Behavior 2010: Selected papers from the 7th international conference on methods and techniques in behavioral research, Article No. a6. August 24–27, Eindhoven – The Netherlands.
and on the first three sections of:
Broek, E. L. van den, Janssen, J.H., Zwaag, M.D. van der, Westerink, J.H.D.M., & Healey, J.A. Affective Signal Processing (ASP): A user manual. [in preparation]
which already appeared partially as:
Broek, E.L. van den et al. (2009/2010/2011). Prerequisites for Affective Signal Processing (ASP) - Parts I-V. In A. Fred, J. Filipe, and H. Gamboa, Proceedings of BioSTEC 2009/2010/2011: Proceedings of the International Joint Conference on Biomedical Engineering Systems and Technologies. January, Porto, Portugal / Valencia, Spain / Rome, Italy.
1.1 Introduction
1.1 Introduction
Originally, computers were invented for highly trained operators, to help them do massive numbers of calculations [149, 610]. However, this origin dates from the first half of the previous century, and much has changed since. nowadays, everybody uses them in one of their many guises. Whereas previously computers were stationary entities the size of a room, today we are in touch with various types of computers throughout our normal daily lives, including our smart phones [3, 122, 135, 381, 460, 610, 713]. Computation is on track to become even smaller and more pervasive. For example, microrobots can already flow through your blood vessels and identify and treat physical damage [2, 165, 214, 479]. Moreover, from dedicated specialized machines, computers have become our window to both the world and our social life [145, 472, 532].
Computers are slowly becoming dressed, huggable, and tangible and our reality will become augmented by virtual realities [50, 594]. Artificial entities are becoming personalized and are expected to understand more and more of their users’ feelings, emotions, and moods [286, 594, 671]. Consequently, concepts such as emotions, that were originally the playing field of philosophers, sociologists, and psychologists [302] have become entangled in computer science as well [210]. This topic was baptized affective computing by Rosalind W. Picard [520, 521]. Picard identified biosignals as an important covert channel to capture human emotions, in addition to channels such as speech and computer vision.
Biosignals (or physiological signals) can be defined as (bio)electrical signals recorded on the body surface, although both non-electrical biosignals and invasive recording techniques exist as well. These bio(electrical) signals are related to ionic processes that arise as a result of electrochemical activity of cells of specialized tissue (e.g., the nervous system). This results in (changes in) electric currents produced by the sum of electrical potential differences across the tissue. This property is similar regardless of the part of the body the cells are located (e.g., the heart, muscles, or the brain) [245, 620]. For an overview of biosignals used for affective computing, I refer to Table 1.1.
There have been many studies that have investigated the use of biosignals for affective computing in the last decade. In Section 1.3 an overview of relevant handbooks will be provided and in Chapter 2 an exhaustive review of research articles will be provided. The handbooks and articles have in common that they illustrate, as I will also show later on (i.e., Chapter 2), that the results on affective computing have been slightly disappointing at best. Hence, I believe a careful examination of the current state-of-the-art can help to provide new insights for future progress. In sum, the goal of this monograph is to i) review the progress made on the processing of biosignals related to emotions (i.e., Affective Signal Processing (ASP) ), ii) conduct necessary additional research, and iii) provide guidelines on issues that need to be tackled in order to improve ASP’s performance.
5
1 Introduction
Table 1.1: An overview of common physiological signals and features used in ASP. The reported response times are the absolute minimum; in practice longer time windows are applied to increase the recording’s reliability.
Physiological response |
Features |
Unit |
Response time |
|
|
|
|
Cardiovascular activity |
Heart rate (HR) |
beats / min |
0.67-1.5 sec |
through ElectroCardioGram (ECG) |
SD IBIs, RMSSD IBIs |
s |
0.67-1.5 sec |
or Blood Volume Pulse (BVP) |
Low Frequency (LF) power (0.05Hz - 0.15Hz) |
ms2 |
0.67-1.5 sec |
(per beat) [43, 44, 349] |
High Frequency (HF) power (0.15HZ - 0.40Hz), RSA |
ms2 |
0.67-1.5 sec |
|
Very Low Frequency (VLF) power ( < 0.05Hz) |
ms2 |
0.67-1.5 sec |
|
LF/HF |
ms2 |
0.67-1.5 sec |
|
Pulse Transit Time (PTT) |
ms |
0.67-1.5 sec |
Electrodermal Activity (EDA) |
Mean, SD SCL |
µS |
after 2-10 sec |
[62] |
Nr of SCRs |
nr / min |
after 2-10 sec |
|
SCR amplitude |
µS |
after 2-10 sec |
|
SCR 1/2 recovery time, SCR rise time |
s |
after 2-10 sec |
Skin temperature (ST) |
Mean |
oC |
after 15-20 sec |
Respiration (per breath) |
rate |
nr / min |
4-15 sec |
[55, 238] |
amplitude |
a.u. |
4-15 sec |
|
ins, exh |
sec |
4-15 sec |
|
total duty cycle |
ins / cycle sec |
4-15 sec |
|
ins exh |
ins / exh sec |
4-15 sec |
Muscle activity [548] |
Mean, SD EMG* |
µV |
< 1 sec |
through ElectroMyoGram (EMG) |
Mean, SD inter-blink interval |
ms |
< 1 sec |
Movements / Posture [201, 403] |
Alternating Current component (motion) |
Hz |
< 1 sec |
through Accelerometer [124, 190] |
Direct Current component (posture) |
Hz |
< 1 sec |
Impedance Cardiography |
Left-ventricular ejection time (LVET) |
sec |
per beat |
[606, 623] |
Prepreejection period (PEP) |
sec |
per beat |
|
Stroke Volume (SV) |
ml |
per beat |
|
Cardiac Output (CO) |
liters/min |
1 minute |
|
Total peripheral resistance (TPR) |
MAP*80/CO |
per beat |
Blood Pressure (BP) |
both systolic and diastolic |
mmHg |
per beat |
Legend: SD: Standard deviation; RMSSD: Root Mean Sum of Square Differences; IBI: Inter-beat interval; ins: inspiration; exh: exhalation; RSA: Respiratory Sinus Arrhythmia; SCL: Skin Conductance Level; SCR: Skin Conductance Response. * Most often the EMG of the corrugator supercilii, zygomaticus major, frontalis, and upper trapezius are used for ASP.
6
