This document describes an emotion-based interactive installation that uses an Emotion-Driven Music Engine (EDME) to perceive emotions from participants and express emotions through music and images. The installation uses a camera to recognize the number of participants and their movement, which determines the emotional state of the system. EDME then generates appropriate music and selects relevant images based on the system's emotional state. Experiments found that participants successfully experienced and influenced the system's emotions through their actions.

1. Emotion-Based Interactive Installation
FranciscoVentura,António Pedro Oliveira and Amílcar Cardoso
University of Coimbra, Portugal

2. 2
Outline
 Introduction
 EDME
 Interactive Installation
 Experiments
 Demonstration
 Conclusion

3. 3
Outline
 Introduction
 EDME
 Interactive Installation
 Experiments
 Demonstration
 Conclusion

4. Introduction
4
 We prepared an installation to explore the interactive
capabilities of an Emotion-Driven Music Engine (EDME)
 This installation allows participants to experience and
influence the emotional state of a computational system
 The system perceives emotions from the environment by
using a camera and expresses emotions to the environment
with music and images by using, respectively, loudspeakers
and a projector

5. 5
Outline
 Introduction
 EDME
 Interactive Installation
 Experiments
 Demonstration
 Conclusion

6. EDME
6
 EDME is a computer program that intends to produce
music expressing a desired emotion
 It is composed by an offline stage and a real-time stage
 At the offline stage, pre-composed music is segmented,
emotionally classified and stored in a music base
 At the real-time stage, EDME selects from a music base the
segments that are closer to a desired emotion

7. EDME
7
 Transformation and sequencing algorithms are applied on
selected segments to form song-like structures
 These song-like structures are then synthesized and given
to the listener
Real-Time
Stage
Offline Stage

8. 8
Outline
 Introduction
 EDME
 Interactive Installation
 Experiments
 Demonstration
 Conclusion

9. Interactive Installation
 The installation is composed by a computer system that
uses a camera to take pictures of the environment

10. Interactive Installation
 The installation is composed by a computer system that
uses a camera to take pictures of the environment
 The system analyses these pictures to recognize the
number of participants and quantity of movement
System recognizes:
• Number of Participants
• Quantity of Movement
System recognizes:
• Number of Participants
• Quantity of Movement

11. Interactive Installation
 The installation is composed by a computer system that
uses a camera to take pictures of the environment
 The system analyses these pictures to recognize the
number of participants and quantity of movement
 These two variables are used to define, respectively, the
values of valence and arousal of the emotional state of the
system
System recognizes:
• Number of Participants
• Quantity of Movement
System recognizes:
• Number of Participants
• Quantity of Movement
Emotional State
of the System
Emotional State
of the System

12. Interactive Installation
12
 The installation is composed by a computer system that
uses a camera to take pictures of the environment
 The system analyses these pictures to recognize the
number of participants and quantity of movement
 These two variables are used to define, respectively, the
values of valence and arousal of the emotional state of the
system
 The emotional state influences the expression of music and
images to the environment
System recognizes:
• Number of Participants
• Quantity of Movement
System recognizes:
• Number of Participants
• Quantity of Movement
Emotional State
of the System
Emotional State
of the System
Emotional
expression
(Music & Images)
Emotional
expression
(Music & Images)

13. Interactive Installation
13
 The number of participants is determined by the
percentage of change between scenario (image taken when
the system starts) and current image
 Quantity of movement is determined by the change
between current and last image

14. Interactive Installation
14
 EDME generates music according to the values of valence
and arousal of the system
 As images are emotionally classified with valence, the
selection of images is only based on this emotional
dimension

15. Interactive Installation
15
 The emotional behaviour of the system is controlled by a
multi-agent system

16. 16
Outline
 Introduction
 EDME
 Interactive Installation
 Experiments
 Demonstration
 Conclusion

17. Experiments
17
 We made a preliminary experiment with students with the
objective of observing the interaction between the system
and participants

18. Experiments
18
 The second experiment was focused in obtaining
participants feedback via a questionnaire of 7 questions
 Example of questions:
 Music expressed expected emotion?
 Importance of images in the emotional expression?
 System expressed expected emotion?

19. Experiments
19
 Answers from the questionnaire show us that:
 the system correctly related arousal with the amount of movement
and valence with the number of participants
 the system was successful in the expression of emotions with both
music and images

20. 20
Outline
 Introduction
 EDME
 Interactive Installation
 Experiments
 Demonstration
 Conclusion

21. 21
Outline
 Introduction
 EDME
 Interactive Installation
 Experiments
 Demonstration
 Conclusion

22. Conclusion
22
 EDME is prepared to be used in interactive contexts
 People experienced and influenced the emotional state of
the system
 Both music and images are relevant in the emotional
expression of the system
 Installation assumes no variation of environmental
conditions; however, it can be adapted to deal with this
situation by changing the scenario