1 Introduction
given a position in health informatics. This shift was accelerated with the general increase in the need for health informatics that has emerged due to the massive growth of the market for new systems that improve productivity, cut costs, and support the transition of health care from hospital to the home [137, 173, 290, 296, 321, 393, 476, 605, 638, 702]. This transition relies heavily on (ethical) issues such as trust, persuasion, and communication [112] that have emotion as common denominator. Health informatics is already or will soon be applied for the support/assistance of independent living, chronic disease management, facilitation of social support, and to bring the doctor’s office to people’s homes. Par excellence, this is where health informatics and affective computing blend together.
1.6.4 Three disciplines, one family
The three disciplines described above are not mutually independent. For one thing, health informatics regularly applies AI techniques [568]; for example, the expert systems Eliza [711] and MYCIN [608] and their successors for various (sub)domains in medicine (e.g., [173, 677]). Also beyond medicine, expert systems have shown their added value, which is best illustrated by the fact that the user’s role shifted from controller to supervisor [385]; for example, recently in high-end automobiles (e.g., Audi, BMW, and Mercedes-Benz). In all these cases, the systems interact with their users; hence, HCI takes a prominent place. This all stresses the relations between the three branches of computer science.
Besides the triplet discussed in this section, many other disciplines within computer science should take emotions into account as well; for example, virtual reality (VR) [86, 474, 488, 616], color processing [539], ambient intelligence (AmI) and ubiquitous computing (UbiComp) [1, 207, 471, 540, 668, 669, 676], multimedia [21, 595, 741], the World Wide Web (WWW) [199], and information retrieval (IR) [282, 283, 485]. To conclude, I hope that I have shown the substantial impact emotions have on many of the disciplines within computer science.
1.7 Outline
This monograph will be divided into five parts:
I A prologue,
II Basic research on baseline-free ASP that uses statistical moments,
III Basic research on bi-modal ASP that explores various aspects,
IV Three studies affective computing, and
V An epilogue.
20
1.7 Outline
A wide range of statistical techniques will be employed in the various chapters throughout this monograph. Appendix A will present these techniques in their simplest forms, will denote their characteristics, and will identify the relations between them.
I. Prologue: This part has started with the current chapter and will continue with: Chapter 2, the second and last chapter of the prologue. In this chapter I will introduce affective computing and, more in particular, ASP. The three dominant modalities in this field (i.e., vision, speech, and biosignals) will be introduced. Next, I will provide the first exhaustive review on biosignal-based affective computing. Its advantages and disadvantages will be denoted as well as the reasons for the rapidly increasing interest in this modality.
II. Baseline-free ASP : This part will include two chapters in which I shall explore the feasibility of baseline-free ASP (i.e., without normalization in any phase of processing) using statistical moments:
Chapter 3. This chapter will cover research for which I used dynamic real world stimuli (i.e., movie scenes) to induce emotions. The EMG of three facial muscles was recorded, which is often done to establish a ground truth measurement. In addition, the participants’ EDA was recorded. EDA is a robust and well documented biosignal that reveals the level of experienced arousal [62, 163].
Chapter 4. The research reported here consisted of analyses on the same data set as Chapter 3. The studies differed in the choice of time windows, which enabled research towards the impact and usage of this parameter for ASP. Where the analyses in Chapter 3 were executed on the complete signals accompanying the movie scenes, in this study 10 sec. time windows were used. Moreover, events in the movie scenes were annotated and the participants’ affective responses that accompanied them were recorded.
III. Bi-modal ASP : Two studies will be presented that employed bi-modal ASP and deviate only with respect to the stimuli that were used for emotion elicitation. The research in these two chapters also assessed the influence of emotion representations by analyzing the obtained data using both the dimensional valence-arousal model [105, 176, 202, 452, 567, 647] and the six basic emotions [116, 181, 391]. Moreover, the impact of the environment (or context), the personality traits neuroticism and extroversion, and demographics on ASP was explored.
Chapter 5 will report research that employed a (or perhaps even the) reference set for affective computing: the International Affective Picture System (IAPS). The bi-modal ASP approach utilized the rare combination of ECG and speech. To the author’s knowledge, this combination has further only been explored by Kim and colleagues [336, 337, 339, 340].
Chapter 6. In this chapter, I will present a study that is identical to the one in Chapter 5 except for the stimuli that have been applied to induce emotions in the participants. The type and selection of stimuli has recently (again) been shown to be a factor of importance [8]. In this study the same set of movie fragments was used as in Chapters 3 and 4. This enabled a
21
1 Introduction
comparison of static versus dynamic stimuli and, as such, assessed their validity.
IV. Towards affective computing: In these three chapters I will explore the feasibility of affective computing using ASP.
Chapter 7. In this chapter, I will go through the complete signal processing + pattern recognition pipeline (see Figure 1.2), using the data that is also presented in Chapters 3 and 4. In the quest for an optimal processing pipeline, several signal processing aspects and classification methods (see also Appendix A) will be explored. As such, the feasibility of emotionaware systems will be assessed.
Chapter 8. In this chapter two studies will be presented that bring us from lab research to clinical practice. For these studies, I employed only the speech signal since direct biosignals were considered to be too obtrusive. The studies’ aim was to lay a foundation for the development of a Computer-Aided Diagnosis (CAD) of patients with a Post-Traumatic Stress Disorder (PTSD).
Chapter 9. In this chapter the data from the two studies presented in Chapter 8 will be fed to the same complete signal processing + pattern recognition pipeline as was already employed in Chapter 7. This explores the true feasibility of the envisioned emotion-aware systems, in this case: ASP-based Computer-Aided Diagnosis (CAD) for mental health care.
V. Epilogue: This part consists of a set of guidelines for ASP and a general discussion. Chapter 10. In this chapter I will present the lessons learned while working on the research presented in this monograph. Considerations and guidelines for processing affective signals and classifying the features derived from these signals in terms of emotions will be introduced. These guidelines will indicate possible problems, will provide solutions for them, and will provide research directives for affective computing. As such, this will perhaps be the most important chapter of this monograph.
Chapter 11 will consist of seven sections. First, I will look back on the work conducted and draw some brief conclusions from this. Second, I will place the work presented in this monograph in a historical perspective. Third, I will weight this monograph’s contribution to emotion science’s 10 hot topics as has been recently identified [236]. Fourth, I will introduce affective computing’s I/O. Fifth, I will describe three consumer applications that can be developed here and now! Sixth, I will stretch the horizon and describe two visions of the future: robot nannies and digital human models. Seventh and last, I will draw some final conclusions and close the monograph.
22