- •Stellingen
- •Propositions
- •List of Figures
- •List of Tables
- •1 Introduction
- •Introduction
- •Affect, emotion, and related constructs
- •Affective Computing: A concise overview
- •The closed loop model
- •Three disciplines
- •Human-Computer Interaction (HCI)
- •Health Informatics
- •Three disciplines, one family
- •Outline
- •2 A review of Affective Computing
- •Introduction
- •Vision
- •Speech
- •Biosignals
- •A review
- •Time for a change
- •3 Statistical moments as signal features
- •Introduction
- •Emotion
- •Measures of affect
- •Affective wearables
- •Experiment
- •Participants
- •Equipment and materials
- •Procedure
- •Data reduction
- •Results
- •Discussion
- •Comparison with the literature
- •Use in products
- •4 Time windows and event-related responses
- •Introduction
- •Data reduction
- •Results
- •Mapping events on signals
- •Discussion and conclusion
- •Interpreting the signals measured
- •Looking back and forth
- •5 Emotion models, environment, personality, and demographics
- •Introduction
- •Emotions
- •Modeling emotion
- •Ubiquitous signals of emotion
- •Method
- •Participants
- •International Affective Picture System (IAPS)
- •Digital Rating System (DRS)
- •Signal processing
- •Signal selection
- •Speech signal
- •Heart rate variability (HRV) extraction
- •Normalization
- •Results
- •Considerations with the analysis
- •The (dimensional) valence-arousal (VA) model
- •The six basic emotions
- •The valence-arousal (VA) model versus basic emotions
- •Discussion
- •Conclusion
- •6 Static versus dynamic stimuli
- •Introduction
- •Emotion
- •Method
- •Preparation for analysis
- •Results
- •Considerations with the analysis
- •The (dimensional) valence-arousal (VA) model
- •The six basic emotions
- •The valence-arousal (VA) model versus basic emotions
- •Static versus dynamic stimuli
- •Conclusion
- •IV. Towards affective computing
- •Introduction
- •Data set
- •Procedure
- •Preprocessing
- •Normalization
- •Baseline matrix
- •Feature selection
- •k-Nearest Neighbors (k-NN)
- •Support vector machines (SVM)
- •Multi-Layer Perceptron (MLP) neural network
- •Discussion
- •Conclusions
- •8 Two clinical case studies on bimodal health-related stress assessment
- •Introduction
- •Post-Traumatic Stress Disorder (PTSD)
- •Storytelling and reliving the past
- •Emotion detection by means of speech signal analysis
- •The Subjective Unit of Distress (SUD)
- •Design and procedure
- •Features extracted from the speech signal
- •Results
- •Results of the Stress-Provoking Story (SPS) sessions
- •Results of the Re-Living (RL) sessions
- •Overview of the features
- •Discussion
- •Stress-Provoking Stories (SPS) study
- •Re-Living (RL) study
- •Stress-Provoking Stories (SPS) versus Re-Living (RL)
- •Conclusions
- •9 Cross-validation of bimodal health-related stress assessment
- •Introduction
- •Speech signal processing
- •Outlier removal
- •Parameter selection
- •Dimensionality Reduction
- •k-Nearest Neighbors (k-NN)
- •Support vector machines (SVM)
- •Multi-Layer Perceptron (MLP) neural network
- •Results
- •Cross-validation
- •Assessment of the experimental design
- •Discussion
- •Conclusion
- •10 Guidelines for ASP
- •Introduction
- •Signal processing guidelines
- •Physical sensing characteristics
- •Temporal construction
- •Normalization
- •Context
- •Pattern recognition guidelines
- •Validation
- •Triangulation
- •Conclusion
- •11 Discussion
- •Introduction
- •Hot topics: On the value of this monograph
- •Applications: Here and now!
- •TV experience
- •Knowledge representations
- •Computer-Aided Diagnosis (CAD)
- •Visions of the future
- •Robot nannies
- •Digital Human Model
- •Conclusion
- •Bibliography
- •Summary
- •Samenvatting
- •Dankwoord
- •Curriculum Vitae
- •Publications and Patents: A selection
- •Publications
- •Patents
- •SIKS Dissertation Series
1 Introduction
1.6 Three disciplines
The relation between emotions and computer science lays hold of various branches of computer science. The computer science disciplines that most noteworthy lay hold on emotions are:
1.HCI and related disciplines (e.g., user experience (UX) and interaction design, and cognitive ergonomics) [24, 264, 315, 545, 687, 735];
2.AI [292, 568], including agents and avatars [35, 38, 71, 217, 242, 297, 444, 686], robotics [69, 70, 672], and cognitive science and neuroscience [15, 147, 148, 162, 184, 431, 505]; and
3.Health Informatics, including e-health, and, more in general, health technology (including mobile technology) [18, 173, 296, 297, 321, 326, 702].
In the next three subsections, I will explain the relation of ASP with each of these three branches of computer science.
1.6.1 Human-Computer Interaction (HCI)
In the 90s of the previous century, Nass and colleagues [475, 549] touched upon a new level of HCI: a personal, intimate, and emotional level. Together with the work of Picard [520, 521] their work positioned affective processes firmly as an essential ingredient of HCI.
The importance of affect for HCI can be well explained by denoting its effects on three cognitive processes, which are important in HCI context:
1.Attention: Affective processes take hold on several aspects of our cognitive processing [118] and, hence, HCI [695]. One of the most prominent effects of affect lies in its ability to capture attention. Affective processes have a way of being completely absorbing. Functionally, they direct and focus our attention on those objects and situations that have been appraised as important to our needs and goals [695]. This attention-getting function can be used advantageously in HCI context [594, Chapter 4].
2.Memory: However, it should be noted that such effects also has implications for learning and memory [49, 64]. Events with an affective load are generally remembered better than events without such a load, with negative events being dominant over positive events [549]. Further, affect improves memory for core information, while undermining memory for background information [396, Chapter 37].
3.Decision making: Affective processes also have their influence on our flexibility and efficiency of thinking and problem solving [396, Chapter 34]. It has also been shown
18
1.6 Three disciplines
that affect can (heavily) influence judgment and decision making [39, 593]. Affective processes tend to bias thoughts by way of an affect-filter. Thoughts are directed to a affect-consistent position, which can also increase the risk of distractions.
This triplet of cognitive processes illustrates that a careful consideration of affect in HCI can be instrumental in creating interfaces that are both efficient and effective as well as enjoyable and satisfying [594, Chapter 4].
1.6.2 Artificial Intelligence (AI)
Almost half a century ago, the American psychologist Ulric Neisser [478] stated that “Human thinking begins in an intimate association with emotions and feelings which is never entirely lost”. Nobel prize winner and recipient of the ACM’s Turing Award, Herbert A. Simon had similar ideas on this topic: “. . . how motivational and emotional controls over cognition can be incorporated into an information-processing system . . . ’” [611, p. 29].
Nonetheless, in the decades that followed AI aimed at understanding human cognition without taking emotion into account [568]. Although emotions were sometimes denoted as important (e.g., [454, 455]), it took until the publication of Picard’s book Affective computing [521] before they received center stage attention. Even though AI has made it possible that a computer can beat the world’s best chess players [89] and can win quizzes such as Jeopardy! [195], the general opinion is that AI has failed [426] (cf. [111, 332]). This is likely to be (partly) because of a lack of focus on emotions. So, almost 50 years after Ulric Neisser’s words, with the user more demanding than ever, perhaps now is the time to bring emotions to the front line of AI research and practice.
1.6.3 Health Informatics
In 1935, Flanders Dunbar noted that the “Scientific study of emotion and of the bodily changes that accompany diverse emotional experience marks a new era in medicine”. We now know that many physiological processes that are of profound significance for health can be influenced by way of emotions (e.g., [183, 305, 326]). For example, it has been shown that emotions influence our cardiovascular system [276, 400, 493, 495, 496, 589, 659] and, consequently, can shorten or prolong life [204, 205, 298, 299, 423, 598]. Moreover, emotions also play an important role with chronic diseases [42, 46], cancer (e.g., coping strategies) [343, 645], and rehabilitation [489], to mention three. [174, p. vii]. Nevertheless, emotions remained rather spiritual and human’s health has usually been explained in physical (e.g., injuries) and physiological terms (e.g., bacteria and viruses). It is only since the last decades that it is generally acknowledged that emotions have their impact on health and illness [326, 493, 692].
Now they have been acknowledged by traditional medicine, emotions are now being
19
