- •Stellingen
- •Propositions
- •List of Figures
- •List of Tables
- •1 Introduction
- •Introduction
- •Affect, emotion, and related constructs
- •Affective Computing: A concise overview
- •The closed loop model
- •Three disciplines
- •Human-Computer Interaction (HCI)
- •Health Informatics
- •Three disciplines, one family
- •Outline
- •2 A review of Affective Computing
- •Introduction
- •Vision
- •Speech
- •Biosignals
- •A review
- •Time for a change
- •3 Statistical moments as signal features
- •Introduction
- •Emotion
- •Measures of affect
- •Affective wearables
- •Experiment
- •Participants
- •Equipment and materials
- •Procedure
- •Data reduction
- •Results
- •Discussion
- •Comparison with the literature
- •Use in products
- •4 Time windows and event-related responses
- •Introduction
- •Data reduction
- •Results
- •Mapping events on signals
- •Discussion and conclusion
- •Interpreting the signals measured
- •Looking back and forth
- •5 Emotion models, environment, personality, and demographics
- •Introduction
- •Emotions
- •Modeling emotion
- •Ubiquitous signals of emotion
- •Method
- •Participants
- •International Affective Picture System (IAPS)
- •Digital Rating System (DRS)
- •Signal processing
- •Signal selection
- •Speech signal
- •Heart rate variability (HRV) extraction
- •Normalization
- •Results
- •Considerations with the analysis
- •The (dimensional) valence-arousal (VA) model
- •The six basic emotions
- •The valence-arousal (VA) model versus basic emotions
- •Discussion
- •Conclusion
- •6 Static versus dynamic stimuli
- •Introduction
- •Emotion
- •Method
- •Preparation for analysis
- •Results
- •Considerations with the analysis
- •The (dimensional) valence-arousal (VA) model
- •The six basic emotions
- •The valence-arousal (VA) model versus basic emotions
- •Static versus dynamic stimuli
- •Conclusion
- •IV. Towards affective computing
- •Introduction
- •Data set
- •Procedure
- •Preprocessing
- •Normalization
- •Baseline matrix
- •Feature selection
- •k-Nearest Neighbors (k-NN)
- •Support vector machines (SVM)
- •Multi-Layer Perceptron (MLP) neural network
- •Discussion
- •Conclusions
- •8 Two clinical case studies on bimodal health-related stress assessment
- •Introduction
- •Post-Traumatic Stress Disorder (PTSD)
- •Storytelling and reliving the past
- •Emotion detection by means of speech signal analysis
- •The Subjective Unit of Distress (SUD)
- •Design and procedure
- •Features extracted from the speech signal
- •Results
- •Results of the Stress-Provoking Story (SPS) sessions
- •Results of the Re-Living (RL) sessions
- •Overview of the features
- •Discussion
- •Stress-Provoking Stories (SPS) study
- •Re-Living (RL) study
- •Stress-Provoking Stories (SPS) versus Re-Living (RL)
- •Conclusions
- •9 Cross-validation of bimodal health-related stress assessment
- •Introduction
- •Speech signal processing
- •Outlier removal
- •Parameter selection
- •Dimensionality Reduction
- •k-Nearest Neighbors (k-NN)
- •Support vector machines (SVM)
- •Multi-Layer Perceptron (MLP) neural network
- •Results
- •Cross-validation
- •Assessment of the experimental design
- •Discussion
- •Conclusion
- •10 Guidelines for ASP
- •Introduction
- •Signal processing guidelines
- •Physical sensing characteristics
- •Temporal construction
- •Normalization
- •Context
- •Pattern recognition guidelines
- •Validation
- •Triangulation
- •Conclusion
- •11 Discussion
- •Introduction
- •Hot topics: On the value of this monograph
- •Applications: Here and now!
- •TV experience
- •Knowledge representations
- •Computer-Aided Diagnosis (CAD)
- •Visions of the future
- •Robot nannies
- •Digital Human Model
- •Conclusion
- •Bibliography
- •Summary
- •Samenvatting
- •Dankwoord
- •Curriculum Vitae
- •Publications and Patents: A selection
- •Publications
- •Patents
- •SIKS Dissertation Series
Publications and Patents: A selection
F Publications and Patents: A selection
292
F Publications and Patents: A selection
A selection of publications and patent applications directly related to the work presented in this thesis. A complete list of publications is available upon request.
Publications
Journal & Magazine papers
50.Broek, E.L. van den, Janssen, J.H., Zwaag, M.D. van der, Healey, J.A., & Westerink, J.H.D.M. Affective Signal Processing (ASP): A user manual. ACM Computing Surveys [in preparation]
49.Broek, E.L. van den, Sluis, F. van der, & Dijkstra, T. (2011/2). Modeling an acoustic profile for ubiquitous stress assessment. Personal and Ubiquitous Computing, 15/6.
[Special issue: Mental health and the impact of ubiquitous technologies; in press]
48.Janssen, J.H., Broek, E.L. van den, & Westerink, J.D.H.M. (2011/2). Tune in to your emotions: A robust personalized affective music player. User Modeling and UserAdapted Interaction, 21/2. [in press]
47.Broek, E.L. van den (2011). Ubiquitous emotion-aware computing. Personal and Ubiquitous Computing, 15. [in press]
46.Zwaag, M.D. van der, Westerink, J.D.H.M., & Broek, E.L. van den (2011). Emotional and psychophysiological responses to tempo, mode, and percussiveness. Musicae Scientiae, 15(2), 250–269.
45.Broek, E.L. van den (2011). Safeguards in a world of ambient intelligence: A social, economic, legal, and ethical perspective. Journal of Ambient Intelligence and Smart Environments, 3(2), 179–180.
44.Broek, E.L. van den (2011). Multimedia techniques for device and ambient intelligence: A continuing endeavor. Journal of Ambient Intelligence and Smart Environments, 3(1), 67–68.
43.Bialoskorski, L.S.S., Westerink, J.H.D.M., & Broek, E.L. van den (2010). Experiencing affective interactive art. International Journal of Arts and Technology, 3(4), 341–356. [invited]
42.Broek, E.L. van den (2010). Beyond Biometrics. Procedia Computer Science, 1(1), 2505– 2513. [invited]
41.Broek, E.L. van den (2010). Robot nannies: Future or fiction? Interaction Studies, 11(2), 274–282.
40. Broek, E.L. van den, Sluis, F. van der, & Schouten, Th.E. (2010). User-centered digital
293
F Publications and Patents: A selection
preservation of multimedia. European Research Consortium for Informatics and Mathematics (ERCIM) News, No. 80 (January), 45-47. [Special issue: Digital Preservation]
39.Broek, E.L. van den & Westerink, J.H.D.M. (2009). Considerations for emotion-aware consumer products. Applied Ergonomics, 40(6), 1055–1064. [Special issue: Psychophysiology in Ergonomics]
38.Broek, E.L. van den, Schut, M.H., Westerink, J.H.D.M., & Tuinenbreijer, K. (2009). Unobtrusive Sensing of Emotions (USE). Journal of Ambient Intelligence and Smart Environments, 1(3), 287–299. [Thematic Issue: Contribution of Artificial Intelligence to AmI]
37.Drunen, A. van, Broek, E.L. van den, Spink, A.J., & Heffelaar, T. (2009). Exploring workload and attention measurements with uLog mouse data. Behavior Research Methods, 41(3), 868–875. [invited]
36.Broek, E.L. van den, Pieters, J., Bruikman, H., Sluis, F. van der, Drunen, A. van, & Luursema, J.-M. (2009). Cognitieve Ergonomie. Perspectief, 4(3), 16–19. [invited]
35.Bialoskorski, L.S.S., Westerink, J.H.D.M., and Broek, E.L. van den (2009). Mood Swings: Design and Evaluation of Affective Interactive Art. New Review of Hypermedia and Multimedia, 15(2), 173–191.
34.Westerink, J., Broek, E.L. van den, Herk, J. van, Tuinenbreijer, K., and Schut, M. (2006). Statistical processing of facial electromyography (EMG) signals in emotional film scenes. Psychophysiology, 43 (S1), S106.
33.Broek, E.L. van den (2004). Emotional Prosody Measurement (EPM): A voice-based evaluation method for psychological therapy effectiveness. Studies in Health Technology and Informatics (Medical and Care Compunetics), 103, 118–125.
32.Broek, E.L. van den (2003). A stress marker in speech. Toegepaste Taalwetenschap in Artikelen, 69(1), 143–153 & 172.
Books & Book chapters
31.Broek, E.L. van den (2012). Handbook of Affective Computing. Series: Pan Stanford Series in Artificial Intelligence. Pan Stanford Publishing. [in preparation]
30.Broek, E.L. van den (2011). Commentary on Affective Computing by Kristina Höök. In M. Soegaard (Ed.), interaction-design.org Encyclopedia. [in preparation]
29.Broek, E.L. van den, Sluis, F. van der, and Dijkstra, T. (2011). Telling the story and re-living the past: How speech analysis can reveal emotions in post-traumatic stress disorder (PTSD) patients. In J.H.D.M. Westerink, M. Krans, and M. Ouwerkerk (Eds.),
Sensing Emotions: The impact of context on experience measurements (Chapter 10), p. 153– 180. Series: Philips Research Book Series, Vol. 12. Dordrecht, The Netherlands:
294
F Publications and Patents: A selection
Springer Science+Business Media B.V. [invited]
28.Broek, E.L. van den, Lisý, V., Janssen, J.H., Westerink, J.D.H.M., Schut, M.H., and Tuinenbreijer, K. (2010). Affective Man-Machine Interface: Unveiling human emotions through biosignals. In A. Fred, J. Filipe & H. Gamboa (Eds.), BioMedical Engineering Systems and Technologies (series: Communications in Computer and Information Science, Vol. 52), p. 21–47. Berlin/Heidelberg, Germany: Springer-Verlag. [invited]
27.Westerink, J.H.D.M., Broek, E.L. van den, Schut, M.H., Herk, J. van, and Tuinenbreijer, K. (2008). Computing emotion awareness through galvanic skin response and facial electromyography. In J.H.D.M. Westerink, M. Ouwerkerk, T. Overbeek, W.F. Pasveer, and B. de Ruyter (Eds.), Probing Experience: From Academic Research to Commercial Propositions (Part II: Probing in order to feed back), Chapter 14, p. 137–150. Series: Philips Research Book Series , Vol. 8. Dordrecht, The Netherlands: Springer Science + Business Media B.V. [invited]
Full papers in proceedings (peer reviewed)
26.Sluis, F. van der, Broek, E.L. van den, & Dijkstra, T. (2011). Towards an artificial therapy assistant: Measuring Excessive Stress from Speech. In V. Traver, A. Fred, J. Filipe, and H. Gamboa (Eds.), Health Informatics 2011: Proceedings of the International Conference on Health Informatics, p. 357–363. January 26-29, Rome – Italy.
25.Broek, E.L. van den, Janssen, J.H., Zwaag, M.D. van der, Westerink, J.H.D.M., & Healey, J.A. (2011). Prerequisites for Affective Signal Processing (ASP) – Part V: A response to comments and suggestions. In F. Babiloni, A. Fred, J. Filipe, and H. Gamboa (Eds.), Biosignals 2011: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, p. 301–306. January 26-29, Rome – Italy.
24.Broek, E.L. van den, Nijholt, A., & Westerink, J.H.D.M. (2010). Unveiling Affective Signals. In E. Barakova, B. de Ruyter, and A.J. Spink (Eds.), ACM Proceedings of Measuring Behavior 2010: Selected papers from the 7th international conference on methods and techniques in behavioral research, Article No. a6. August 24–27, Eindhoven – The Netherlands.
23.Broek, E.L. van den, Nijholt, A., & Westerink, J.H.D.M. (2010). Unveiling Affective Signals. In A.J. Spink, F. Grieco, O.E. Krips, L.W.S. Loijens, L.P.J.J. Noldus, & P.H. Zimmerman (Eds.), Proceedings of Measuring Behavior 2010: The 7th International Conference on Methods and Techniques in Behavioral Research, p. 95–98. August 24–27, Eindhoven – The Netherlands.
22.Broek, E.L. van den, Nijholt, A., & Westerink, J.H.D.M. (2010). Symposium: Unveiling Affective Signals. In A.J. Spink, F. Grieco, O.E. Krips, L.W.S. Loijens, L.P.J.J. Noldus, &
295
F Publications and Patents: A selection
P.H. Zimmerman (Eds.), ACM Proceedings of Measuring Behavior 2010: The 7th International Conference on Methods and Techniques in Behavioral Research, p. 94. August 24–27, Eindhoven – The Netherlands.
21.Zwaag, M.D. van der, Broek, E.L. van den, & Janssen, J.H. (2010). Guidelines for biosignal-driven HCI. In A. Girouard, R. Mandryk, L. Nacke, E.T. Solovey, D. Tan, & R.J.K. Jacob (Eds.), Proceedings of ACM CHI2010 workshop - Brain, Body, and Bytes: Physiological user interaction, p. 77–80. April 11, Atlanta, GA, USA.
20.Broek, E.L. van den, Zwaag, M.D. van der, Healey, J.A., Janssen, J.H., & Westerink, J.H.D.M. (2010). Prerequisites for Affective Signal Processing (ASP) - Part IV. In J. Kim & P. Karjalainen (Eds.), Proceedings of the 1st International Workshop on Bio-inspired Human-Machine Interfaces and Healthcare Applications – B-Interface 2010, p. 59–66. January 21, Valencia - Spain.
19.Schut, M.H., Tuinenbreijer, K., Broek, E.L. van den, Westerink, J.H.D.M. (2010). Biometrics for Emotion Detection (BED): Exploring the combination of Speech and ECG. In J. Kim & P. Karjalainen (Eds.), Proceedings of the 1st International Workshop on Bioinspired Human-Machine Interfaces and Healthcare Applications – B-Interface 2010, p. 51–58. January 21, Valencia - Spain.
18.Broek, E.L. van den, Janssen, J.H., Healey, J.A., & Zwaag, M.D. van der. (2010). Prerequisites for Affective Signal Processing (ASP) - Part II. In A. Fred, J. Filipe & H. Gamboa (Eds.), Biosignals 2010: Proceedings of the Third International Conference on Bio-Inspired Systems and Signal Processing, p. 188–193. January 20-23, Valencia - Spain.
17.Sluis, F. van der, Broek, E.L. van den, & Dijkstra, T. (2010). Towards semi-automated assistance for the treatment of stress disorders. In A. Fred, J. Filipe & H. Gamboa (Eds.),
HealthInf 2010: Proceedings of the Third International Conference on Health Informatics, p. 446–449. January 20-23, Valencia - Spain.
16.Broek, E.L. van den, Janssen, J.H., & Westerink, J.D.H.M. (2009). Guidelines for Affective Signal Processing (ASP): From lab to life. In J. Cohn, A. Nijholt, & M. Pantic (Eds.), Proceedings of the IEEE International Conference on Affective Computing & Intelligent Interaction, Vol. I, p. 704–709. September 10-12, Amsterdam - The Netherlands.
[special session introduction]
15.Janssen, J.H., Broek, E.L. van den, & Westerink, J.D.H.M. (2009). Personalized affective music player. In J. Cohn, A. Nijholt, & M. Pantic (Eds.), Proceedings of the IEEE International Conference on Affective Computing & Intelligent Interaction, Vol. I, p. 472–477. September 10-12, Amsterdam - The Netherlands.
14.Zwaag, M.D. van der, Westerink, J.D.H.M., & Broek, E.L. van den (2009). Deploying music characteristics for an affective music player. In J. Cohn, A. Nijholt, & M. Pantic (Eds.), Proceedings of the IEEE International Conference on Affective Computing & Intelligent
296
F Publications and Patents: A selection
Interaction, Vol. I, p. 459–465. September 10-12, Amsterdam - The Netherlands.
13.Broek, E.L. van den, Sluis, F. van der, & Dijkstra, T. (2009). Therapy Progress Indicator: Combining speech parameters and the subjective unit of distress. In J. Cohn, A. Nijholt, & M. Pantic (Eds.), Proceedings of the IEEE International Conference on Affective Computing & Intelligent Interaction, Vol. I, 381-386. September 10-12, Amsterdam - The Netherlands.
12.Janssen, J.H. & Broek, E.L. van den (2009). Guidelines for Mobile Emotion Measurement. In A. Geven, M. Tscheligi & L. Noldus (Eds.), Proceedings of the ACM MobileHCI workshop – Measuring Mobile Emotions: Measuring the Impossible?, p. 17–20. September 15, Bonn - Germany.
11.Bialoskorski, L.S.S., Westerink, J.H.D.M., & Broek, E.L. van den (2009). Mood Swings: An Affective Interactive Art System. Lecture Notes of ICST, 9, 181–186.
10.Bialoskorski, L.S.S., Westerink, J.H.D.M., & Broek, E.L. van den (2009). Come, see and experience affective interactive art. In A. Nijholt, D. Reidsma, and H. Hondorp (Eds.), Program and demonstration papers of the Third International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN), 1–3. June 22-24, Amsterdam - The Netherlands.
9.Broek, E.L. van den, Janssen, J.H., Westerink, J.H.D.M., & Healey, J.A. (2009). Prerequisites for Affective Signal Processing (ASP). In P. Encarnação and A. Veloso (Eds.), Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, p. 426–433. January 14-17, Porto - Portugal.
8.Broek, E.L. van den, Lisý, V., Westerink, J.D.H.M., Schut, M.H., & Tuinenbreijer, K. (2009). Biosignals as an Advanced Man-Machine Interface. In P. Encarnação and A. Veloso (Eds.), Biosignals 2009: Proceedings of the International Conference on BioInspired Systems and Signal Processing, p. IS15–IS24. January 14-17, Porto - Portugal.
[invited]
7.Broek, E.L. van den, Lisý, V., Westerink, J.D.H.M., Schut, M.H., & Tuinenbreijer, K. (2009). Biosignals as an Advanced Man-Machine Interface. In T.F.B. Filho and H. Gamboa (Eds.), Biodevices 2009: Proceedings of the International Conference on Biomedical Electronics and Devices, p. IS15–IS24. January 14-17, Porto - Portugal. [invited]
6.Broek, E.L. van den, Lisý, V., Westerink, J.D.H.M., Schut, M.H., & Tuinenbreijer, K. (2009). Biosignals as an Advanced Man-Machine Interface. In L. Azevedo and A.R. Londral (Eds.), Healthinf 2009: Proceedings of the International Conference on Health Informatics, p. IS15–IS24. January 14-17, Porto - Portugal. [invited]
5.Bialoskorski, L.S.S., Westerink, J.H.D.M., and Broek, E.L. van den (2008). Introducing Mood Swings. In P. Markopoulos, J. Hoonhout, I. Soute, and J. Read (Eds), Interna-
297
