Static versus dynamic stimuli

Abstract

This chapter presents a study that replicated the study presented in the previous chapter, with as its only difference the type of stimuli used to elicit the participants’ emotions. So, this study also explores the rare combination of speech, electrocardiogram, and a revised SelfAssessment Mannequin (SAM) to assess people’s emotions. Forty people watched movie scenes as were introduced in Chapters 3 and 4 in either an office or a living room environment. Additionally, their scores for the personality traits neuroticism and extroversion and demographic information (i.e., gender, nationality, and level of education) was included in the analysis. The resulting data was analyzed using both basic emotion categories and the valence-arousal model, which enabled a comparison between both representations. It was shown that, when people’s gender is taken into account, both heart rate variability (HRV, derived from the ECG) and the standard deviation of the fundamental frequency of speech indicate people’s experienced valence and arousal, in parallel. As such, both measures seem to validate each other. However, the explained variance is much lower than on the data of the previous chapter. For the valence-arousal model, the explained variance was reasonable: 43%. In contrast, for the basic emotions, the explained variance was low: 21%. So, in line with the previous chapter, this study also supports in favor of the valence-arousal model and questions the six basic emotions. Further comparison of both studies confirmed that, independent of emotion representation, the bimodal ASP approach taken is robust in penetrating emotions. In particular, this is confirmed by the combination of features from the two modalities. The distinct features from speech and HRV reveal a more subtle picture in which the several factors do appear to play their role. An exception is the personality trait extroversion, which seems to be of hardly any influence at all.

This chapter is a thoroughly revised version of:

Broek, E.L. van den, Schut, M.H., Westerink, J.H.D.M., & Tuinenbreijer, K. (2009). Unobtrusive Sensing of Emotions (USE). Journal of Ambient Intelligence and Smart Environments, 1(3), 287– 299. [Thematic Issue: Contribution of Artificial Intelligence to AmI]

6.1 Introduction

6.1 Introduction

A decade ago, Ducatel, Bogdanowicz, Scapolo, Leijten, and Burgelman [169] expressed a similar concern in “Scenarios for Ambient Intelligence in 2010” on behalf of the EU’s IST Advisory Group. Two of their key notions were already assessed in the previous chapter and will be further explored in the current chapter: emotion and unobtrusive measurements.The lessons learned in Artificial Intelligence (AI), Cybernetics, psychophysiology, and other disciplines will be taken into account, which will also make it a truly interdisciplinary research.

Before continuing this research, let me take a step back ... in time. Let me cherish remarks made on machine intelligence, either long or not so lang ago. AI pioneer Herbert A. Simon [611] was the first to denote the importance of emotion for AI. Minsky [454, Chapter 16, p. 163] confirmed this by stating: The question is not whether intelligent machines can have emotions, but whether machines can be intelligent without emotions. Nevertheless, in practice emotions were mostly ignored in the quest for intelligent machines until Picard [521] introduced the field affective computing. Since then, the importance of emotion for AI was slowly acknowledged [455]. However, it needs to be stressed that emotions are not only of crucial importance for true AI but are at least as important for Ambient Intelligence (AmI). This has already been acknowledged by Emile Aarts [1, p. 14]: Ubiquitous-computing environments should exhibit some form of emotion to make them truly intelligent. To this end, the system’s self-adaptive capabilities should detect user moods and react accordingly.

This chapter continues the quest for ubiquitous affective computing. In line with the previous chapter, this study also respects the complexity of emotions as well as the current limitations of unobtrusive physiological measurement. The study reported in this chapter is a replication of the study reported in the previous chapter, except for the stimuli used to elicit the emotions from the participants. To refrain from major redundancies in this monograph, this chapter is a compressed version of the article it originates from.

First, I will introduce the construct emotion (Section 6.2) by taking a step back and briefly discussing the work that served as the foundation of the definition of emotion used in the previous chapter. Next, in Section 6.3, I will denote the aspects on which the study reported in this section deviates from that reported in the previous chapter. Sections 6.4 and 6.5 will describe respectively the preparation of the analysis and its results. Subsequently, in Section 6.6 the current study will be compared with the one presented in the previous chapter. Last, in Section 6.7, I will close this chapter with some final words on the work presented here.

101

6 Static versus dynamic stimuli

6.2 Emotion

A lengthy debate on the topic of emotion would be justified; however, this falls way beyond the scope of the current chapter, it even falls beyond the scope of this monograph. Hence, in this chapter there will be no overview of the various emotion theories and the levels on which emotions can be described either. Instead, I will take the elaboration on the definition of emotion as provided in the previous chapter as starting point. The definition presented in Chapter 5 was based on recent work by Izard [302] and the comments that followed this work [302]. Carroll E. Izard [302] took the work Kleinginna and Kleinginna [350] as starting point. For a thoroughly composed definition this will be used as a starting point. In line with Izard [302], I will also go back to this work.

Kleinginna and Kleinginna [350] compiled a list of 92 definitions and 9 skeptical statements about emotion. Regrettably, they had to conclude that psychologists cannot agree on many distinguishing characteristics of emotions. Therefore, they proposed a working definition: Emotion is a complex set of interactions among subjective and objective factors, mediated by neural/hormonal systems, which can (a) give rise to affective experiences such as feelings of arousal and pleasure / displeasure; (b) generate cognitive processes such as emotionally relevant perceptual effects, appraisals, labeling processes; (c) activate widespread physiological adjustments to the arousing conditions; and (d) lead to behavior that is often, but not always, expressive, goal directed, and adaptive. I will now adopt this definition as working definition, instead of that of Izard [302], as presented in Chapter 5.

Kleinginna and Kleinginna [350] also addressed the influence of emotions on people’s cognitive processes: issues (b) and (d). Hence, emotions by themselves should be taken into account; but, so should their effect on cognitive processes (e.g., attention, visual perception, and memory) and, thereby, our functioning. This emphasizes the importance of taking emotions into account in AmI. Moreover, Kleinginna and Kleinginna [350] addressed the influence of emotions on our physiology. This is nicely in line with the main objective of this monograph: Affective Signal Processing (ASP): Unraveling the mystery of emotions.

In line with the frequently adopted circumplex or valence-arousal (VA) model of emotions [372, 443, 535, 647], the definition of Kleinginna and Kleinginna [350] distinguishes arousal and valence (i.e., pleasure / displeasure). The valence-arousal model denotes valence and arousal as two independent bipolar factors that describe emotions. Although the VA model is successful, it has two severe limitations. First, no emotions are identified with high scores, either positive or negative, on either the valence or the arousal axis [372]. Second, the model cannot handle mixed emotions; that is, parallel experience of both positive and negative valence [79] (see also Chapters 3 and 4).

To enable the identification of mixed emotions and provide a suitable processing scheme, the valence-arousal model is sometimes extended [79, 357] (cf. Chapters 3-5). Such

102