6 Static versus dynamic stimuli
6.6 Static versus dynamic stimuli
More than anything else, the current study and its replicate presented in Chapter 5, provide the opportunity to determine the possible influence of the type of stimuli. The current study used dynamic stimuli (i.e., movie scenes) [570], as were already used in the first studies reported in this monograph. In contrast, the study presented in the previous chapter used static IAPS pictures [374, 452]. On the one hand, the movie scenes are claimed to be more ecologically valid [235, 237]. The IAPS images can be questioned with respect to their ecological validity as they present, for example, exaggerated images that are also possibly culturally biased. On the other hand, the IAPS pictures have been validated and have been used frequently and, consequently, have become a sort of standard for emotion elicitation. This having been said, the question remains what (type of) stimuli to use.
When the results of both studies are compared, the first thing that one notices is the difference in the number of effects found. For the VA model and the basic emotions, the results of the study presented in Chapter 5 reported respectively > 2× and 1.5× the number of results as were reported in the current chapter. For the univariate analysis, a similar trend was shown for the VA model. In contrast, no significant difference in the number of effects was found for the six basic emotions. More interesting than the number of effects is the amount of variance the effects explained. For the VA model, the difference in explained variance between both types of stimuli is enormous: 90% (IAPS pictures) versus 43% (movie scenes). In contrast, for the basic emotions, the difference in explained variance between both types of stimuli was marginal: 18% (IAPS pictures) versus versus 21% (movie scenes). It is noteworthy that these differences are opposite.
The univariate ANOVAs of both studies show a similar trend over both emotion representations. With the VA model many more results were found than with the six emotion categories. This effect seems to have been rather independent of the type of stimuli used. However, more important, the univariate analyses of both studies showed remarkable differences. With the IAPS pictures used as emotion elicitation (see Chapter 5), the speech feature intensity (I) has shown to have a remarkably high discriminative power for the VA model. This result was not confirmed at all in the current study, which employed the movie scenes. Given the fact that everything except the stimuli has been controlled over both studies, this is an astonishing effect. Although it should be noted that neither of the studies revealed any effect of intensity (I) of speech on the six basic emotions.
The study presented here and the one presented in the previous chapter explored more than biomodal emotion elicitation using two distinct types of stimuli. The studies also included various other factors of which it has been posed that they are of influence on people’s emotion experience: environment (or context), the personality traits neuroticism and extroversion, and demographic information, most noteworthy gender. In line with what would
110
6.7 Conclusion
be expected, these factors were shown to interact significantly. This stresses the need for a holistic approach towards ASP, towards a digital human profile, which will be denoted more extensively in the next part of this monograph.
Both studies confirmed that, independent of emotion representation, the bimodal ASP approach taken is robust in penetrating emotions. In particular, this is confirmed by the MANOVAs of both studies, see Tables 5.3 and 5.5 as well as Tables 6.3 and 6.5. The ANOVAs reveal a more subtle picture in which the several factors do appear to play their role. An exception is the personality trait extroversion, which seems to be of hardly any influence. Independent of the emotion representation, the personality trait neuroticism had a significant influence on the emotions experienced by the participants when viewing IAPS pictures. Surprisingly, such an effect was not found with emotion elicitation using movie scenes. So, this suggests that the personality trait neuroticism is (also) dependent on the stimuli type and not or not only on the emotions meant to be elicited. Demographic information was shown to be of little value when conducting ASP, except for the gender of the participants. Over both studies and both emotion representations, gender was frequently shown to be of influence.
Perhaps the most important conclusion of this one-on-one comparison of the two studies is that, independent of the emotion representation and the type of stimuli used, the speech feature SD F0 and the ECG feature HRV have shown a significant power in penetrating the participants’ emotions. Follow-up research should be conducted to see whether or not this effect can be generalized to other types of stimuli and even to other modalities. For the time being, however, the inclusion of both SD F0 and HRV is advised for research and applications that envision emotion-awareness. Additional research should be conducted on the true value of the speech feature intensity (I) and its relation to both emotion representations as used in the two studies discussed here.
6.7 Conclusion
Both the F0 of speech and the HRV can be considered as physiological parameters that can be determined indirectly or at least unobtrusively. This makes them par excellence suitable for AmI purposes. This study and the study reported in the previous chapter were two of the first studies that reported the use of both signals simultaneously to unravel the user’s emotional state. To my knowledge, Kim and colleagues [336, 337, 339, 340] are the only ones who have reported on this combination before. The results of this study show that the combination of these measures provides a reliable, robust, and unobtrusive method to penetrate the user’s emotional state. Moreover, the signals validate each other. Both HRV and SD F0 seem to indicate influences of experienced valence and arousal in parallel. This also confirmed the findings reported in the previous chapter.
111
6 Static versus dynamic stimuli
How emotion should be described and modeled remains a topic of debate, despite the work presented in the current and previous chapters. In this chapter, we have adopted the definition of Kleinginna and Kleinginna [350]. However, even in the same decade, various seminal works on emotion were published; for example, Frijda (1986) [208] and Orotony, Clore, and Collins (1988) [502]. Both of these works included their own definition of emotion; for example, Orotony, Clore, and Collins [502, Chapter 1, p. 13 and Chapter 5, p.
191] defined emotions as: valenced reactions to events, agents, or objects, with their particular nature being determined by the way in which the eliciting situation is construed. Since the 80s of the previous century, a vast number of books, opinions, and research papers have been published, illustrating the lack of a generally accepted, multidisciplinary theory on emotions. For a concise, more recent overview of the various theories on emotions, we refer to [144, 396, 535, 582].
This chapter closes Part III of this monograph, in which I explored methods and techniques as well as several additional factors to unravel their influence on unveiling emotions. In the next part of the monograph, Part IV, I will present three chapters that explore the feasibility of affective computing. In the next chapter, Chapter 7, I will go through the complete signal processing + pattern recognition pipeline, using the data that was also presented in Chapters 3 and 4 and, as such, address the feasibility of emotion-aware systems in a completely different way and will reveal many of its future challenges. Lab research is brought to clinical practice in the two chapters that follow Chapter 7. In Chapter 8 two studies will be presented that explore the feasibility of Computer-Aided Diagnosis (CAD) for mental health care. In these studies, I will employ only the speech signal since direct biosignals were considered to be too obtrusive for the application at hand. After that, in Chapter 9, the complete signal processing + pattern recognition pipeline will be applied on the data derived from the studies discussed in Chapter 8. The resulting analyses can serve as the ASP foundation for Computer-Aided Diagnosis (CAD) in mental health care settings.
112