1.2 Affect, emotion, and related constructs

In the next section, I will provide a concise introduction on this monograph’s core constructs affect, emotion, and related constructs. Subsequently, in Section 1.3, I will provide a concise overview of affective computing by providing both a definition of the field and a list of its representative handbooks. Section 1.4 will provide a definition of Affective Signal Processing (ASP) and will introduce its research rationale. Next, in Section 1.5, my working model for automatizing the recognition of emotion from biosignals will be introduced: a closed loop model for affective computing. One component of the model receives our main attention: the signal processing + pattern recognition processing pipeline. In Section 1.6, I will describe how the work presented in this monograph is embedded in computer science. Last, in Section 1.7, I will provide an outline of this monograph.

1.2 Affect, emotion, and related constructs

In 1993, Robert C. Solomon noted in the Handbook of Emotions [396, Chapter 1, p. 3, 1st ed.] that “ “What is an emotion?” is the question that “was asked in precisely that form by William James, as the title of an essay he wrote for Mind well over 100 years ago (James, 1884). . . . But the question “What is an emotion?” has proved to be as difficult to resolve as the emotions have been to master. Just when it seems that an adequate definition is in place, some new theory rears its unwelcome head and challenges our understanding.” Regrettably, there is no reason to assume that this could not be the case for the concise theoretical framework that will be presented here (cf. [302]). Nevertheless, we need such a framework to bring emotion theory to affective computing practice.

In 2003, 10 years after Solomon’s notion, in the journal Psychological Review, James A. Russell characterized the state-of-the-art of emotion (related) research as follows: “Most major topics in psychology and every major problem faced by humanity involve emotion. Perhaps the same could be said of cognition. Yet, in the psychology of human beings, with passions as well as reasons, with feelings as well as thoughts, it is the emotional side that remains the more mysterious. Psychology and humanity can progress without considering emotion – about as fast as someone running on one leg.” [567, p. 145]. Where Solomon [396, Chapter 1, p. 3, 1st ed.] stressed the complexity of affect and emotions, Russell [567, p. 145] stressed the importance to take them into account. Indeed, affect and emotions are of importance psychology and humanity but also for (some branches of) science and engineering, as we will argue in this monograph.

Solomon’s [396, Chapter 1, p. 3, 1st ed.] and Russell’s [567, p. 145] quotes perfectly points towards the complexity of the constructs at hand (i.e., affect and emotion, amongst other things). It is well beyond the scope of this monograph to provide an exhaustive overview of theory on affect, emotion, and related constructs. However, a basic understanding and stipulative definitions are needed, as they are the target state affective computing and ASP are aiming at. This section will provide the required definitions. Since this mono-

7

1 Introduction

graph aims at affective computing and ASP, I will focus on affect as the key construct, which is, from a taxonomic perspective, a convenient choice as well. Affect is an umbrella construct that, instead of emotions, incorporates all processes I am interested in, as we will see in the remaining section.

Core affect is a neurophysiological state that is consciously accessible as a primitive, universal, simple (i.e., irreducible on the mental plane), nonreflective feeling evident in moods and emotions [531, 567]. It can exist with or without being labeled, interpreted, or attributed to any cause [567]. People are always and continuously in a state of core affect, although it is suggested that it disappears altogether from consciousness when it is neutral and stable [567]. Affect influences our attitudes, emotions, and moods and as such our feelings, cognitive functioning, behavior, and physiology [236, 567]; see also Table 1.2. As such, affect is an umbrella construct, a superordinate category [236].

Affect is similar to Thayer’s activation [647], Watson and Tellegen’s affect [707], and Morris’ mood [462] as well as what is often denoted as a feeling [567]. As such, core affect is an integral blend of hedonic (pleasure-displeasure) and arousal (sleepy-activated) values; in other words, it can be conveniently mapped onto the valence-arousal model [372, 566, 567, 647]. However, note that the term “affect” is used throughout the literature in many different ways [531]. Often it is either ill defined or not defined at all. However, affect has also been positioned on another level than that just sketched; for example, as referring to behavioral aspects of emotion [236].

With affect being defined, we are left with a variety of related constructs. To achieve a concise but proper introduction to these constructs, we adopt Scherer’s table of psychological constructs related to affective phenomena [58, Chapter 6]; see Table 1.2. It provides concise definitions, examples, and seven dimensions on which the constructs can be characterized. Together this provides more than rules of thumb, it demarcates the constructs up to a reasonable and workable level. Suitable usage of Table 1.2 and the theoretical frameworks it relies on opens affect’s black box and makes it a gray box [323, 517], which should be conceived as a huge progress. The relations affective processes have with cognitive processes are also interested in this perspective. These will be discussed in Section 1.6.

1.3 Affective Computing: A concise overview

Affect and its related constructs (see Section 1.2) have already been a topic of research for centuries. In contrast, computers were developed only a few decades ago [149, 610]. At a first glance, these two topics seem to be worlds apart; however, as denoted in Section 1.1, emotions and computers have become entangled and, in time, will inevitably embrace each other. Their relation, however, is fresh and still needs to mature.

8

1 Introduction

In 1995, Rosalind W. Picard wrote a technical report [520], which was a thought-paper that presented her initial thinking on affective computing. In a nutshell, this report identifies a number of crucial notions which are still relevant. Moreover, Picard provided an initial definition of affective computing: “. . . a set of ideas on what I call “affective computing,” computing that relates to, arises from, or influences emotions.” [520, p. 1]

In 2005, the first International Conference on Affective Computing and Intelligent Interaction (ACII) was organized. Two of the conference chairs, Tao and Tan, wrote a review on affective computing in which they defined it as: “Affective computing is trying to assign computers the human-like capabilities of observation, interpretation and generation of affect features.”

(cf. [639]). As such, they assured a one-on-one mapping of affect onto the traditional computer science / Human-Computer Interaction (HCI) triplet input (i.e., observation), processing (i.e., interpretation), and output (i.e., generation).

In 2010, the IEEE Transactions on Affective Computing were launched. Its inaugural issue contained a review by Rafael A. Calvo and Sidney D’Mello [87] who characterized the rationale of affective computing with: “automatically recognizing and responding to a user’s affective states during interactions with a computer can enhance the quality of the interaction, thereby making a computer interface more usable, enjoyable, and effective.”

For this monograph, however, we will define affective computing as: “the scientific understanding and computation of the mechanisms underlying affect and their embodiment in machines”. This definition is inspired by the short definition of Artificial Intelligence (AI) provided by the Association for the Advancement of Artificial Intelligence (AAAI) . Drawing upon this definition, I have compiled an overview of books (see Table 1.3) that can be considered as handbooks on or related to affective computing. As such, Table 1.3 provides a representative overview of the work conducted in this field.

I have chosen to exclude M.Sc./Ph.D.-theses from Table 1.3. However, three Ph.D.- theses from the early years of affective computing should be mentioned: Jennifer A. Healey’s (2000) “Wearable and automotive systems for affect recognition from physiology” [269], Maja Pantic’s (2001) “Facial expression analysis by computational intelligence techniques” [509], and Marc Schröder’s (2004) “Speech and emotion research: An overview of research frameworks and a dimensional approach to emotional speech synthesis” [588], which are complementary with respect to the signals used. Healey [269], Pantic [509], and Schröder [588] utilized respectively biosignals, computer vision technology, and the speech signal. In the next chapter, I will discuss this triplet in more depth. Additionally, the numerous (edited) volumes of Klaus R. Scherer and colleagues, starting with [581] and [583] up to the more recent [582] and [584], should be acknowledged. His work is of tremendous importance for affective computing; however, only a minority of his work includes a computing component [578].

Association for the Advancement of Artificial Intelligence (AAAI)’s URL: http://www.aaai.org/

10

1.3 Affective Computing: A concise overview

Table 1.3: An overview of 24 handbooks on affective computing. Selection criteria: i) on emotion and/or affect, ii) either a significant computing or engineering element or an application-oriented approach, and iii) proceedings, M.Sc.-theses, Ph.D.-theses, books on text-analyses, and books on solely theoretical logic-based approaches were excluded.

11