1. Interactive Socio-Mobile Systems for Active
Experience of Audiovisual Content
Antonio Camurri and Alberto Massari
{toni, alby}@infomus.org
Casa Paganini – InfoMus Research Centre
University of Genoa, Italy
www.infomus.org
www.siempre.infomus.org

2. Summary
• The Casa Paganini – InfoMus Research
Centre of University of Genoa
• How S&T research may benefit from the arts
• The European Project SIEMPRE
• The EyesWeb software platform
• Research perspectives

3. Real-time measure of expressive gesture and non
verbal social signals (entrainment, leadership).
The Virtual Binocular to explore AV content
(Socio-mobile) active music listening
…Research

4. Examples of scientific projects at Casa Paganini - InfoMus
based on cross-fertilization of ICT and art
From artistic project… …to S&T research
Music Theatre Opera “Outis”
Luciano Berio
Teatro Alla Scala di Milano (1996)
Invisible interfaces for on-stage interaction
and synchronization with audio
Music Theatre Opera “Cronaca del Luogo”,
Luciano Berio, opening
Salzburg Festival (July 1999)
Real-time analysis of full-body movement,
non-verbal expressive behaviour qualities.
EyesWeb software platform.
Music Theatre Opera “Un Avatar del
Diavolo”, Roberto Doati,
La Biennale Venezia (2005)
Tangible acoustic interfaces: give the
sense of touch to everyday objects
Museum “Enrico Caruso”,
permanent interactive installation “Sala
della Musica”, Firenze (2011-)
Non-verbal behaviour analysis for
individual and group interaction with
cultural heritage content
EU FET11 Closing Performance: TanGO
Touching Music” (6 May 2011)
Performance built upon scientific results of
the European ICT FET SIEMPRE Project.
Study of music joint performance: string
quartets, orchestra sections with
conductor, audience response
S&T research in EU ICT FET SIEMPRE
Project

5. Casa Paganini – InfoMus Research Centre
The monumental building of Santa Maria delle Grazie La Nuova

6. The European Project SIEMPRE
Social Interaction and Entrainment
using Music PeRformance Experimentation
May 2010 – June 2013
Consortium:
Casa Paganini - InfoMus, Univ of Genoa (Coordinator)
Queens University of Belfast
Swiss Center for Affective Sciences, University of Geneve
Pompeu Fabra University, Barcelona
Italian Institute of Technology, Genoa
INCO (non EU) Partners:
Stanford University, USA
Waseda University, Tokyo, Japan
Virginia Tech, USA

7. SIEMPRE: experimentation centered
on music
• Music as a window into complex integrative brain
processes: exploited to study brain mechanisms
• String quartet representative of the Self-Managed
Team (studies in economy and management) in an
ecologically valid framework: e.g., as a proxy for a
group of employees having a total responsibility for a
defined project.
• Orchestra: a scenario characterized by social sub-
groups (e.g., first and second violins sections) with a
leader (the conductor). An orchestra is an example
of inter-team cooperation inside an organization.
• Audience exposed to artistic stimuli.

8. SIEMPRE research challenges
• What factors drive interpersonal synchronisation? How do they
vary according to physical, emotional and social contexts?
• Can specific roles (e.g., leadership) be identified inside a
group? Can general principles be discerned concerning the
influence of individuals over others?
• What are the factors that determine feelings of group cohesion
or a sense of shared meaning? Can the validity of such reports
be supported or confirmed by expressive Movement Audio and
Physiological (eMAP) measures?
• How does social context affect individual intrapersonal
synchronisation of eMAP and vice-versa? How does the
emotion of an individual affect a collaborative creative product?
• Which are the neurophysiological foundations of creative group
communication?
www.siempre.infomus.org

9. SIEMPRE: collaborations with
world-level artists
• Conductors: Riccardo Muti and his
«Youth Orchestra Luigi Cherubini»,
Pietro Borgonovo, Sera Tokay
• Renaud Capuçon, violinist
• Quartetto di Cremona, string quartet
Experiments also with music students
from Music Conservatories

10. String Quartet experiments
“Self-Managed Team”: all musicians contribute equally to
the task [Gilboa & Tal-Shmotkin 2010]
Picture: Quartetto di Cremona during an experiment.

11. SIEMPRE string quartet scenario: research questions
• Which multimodal cues explain the difference between
playing alone or in ensemble ?

12. SIEMPRE string quartet scenario: research questions
• Which multimodal cues explain who is the leader ?
• Which multimodal cues explain the difference between
a performance engaging an audience and a simply
correct (but «cold») performance ?

13. SIEMPRE scenarios: orchestra with conductor

14. Orchestra scenario: a study
• Studying the effectiveness of an established
leader through a group measure of entrainment
• Defining a metric for quantifying such
effectiveness
• Validating the metric with raters (experts)
Boerner et al. (2004) model of leadership in orchestra:
• conductor’s leadership style is not participative or delegative, but directive
• Goal-directed leadership: goal is artistic quality

15. Soft entrainment
• beauty in music lies largely in artistic deviation
from the exact or rigid (Seashore)
• Expressivity is characterized by such deviations
• In joint music performance, players do not reach
and maintain the maximum level of entrainment
• Soft-entrainment: alternation of high and low
entrainment (Yoshida):

16. Orchestra: Experiment
• Recordings of Motion Capture and audio on two
string sections and conductor
• Focus on the z-component of bow trajectory of
string sections (z-direction assumed as the
predominant direction of bow movement)
• Segmentation of music phrases

17. The obtained measures
-> Synchronization Index
-> Effectiveness of Leadership, in terms of a
weighted combination of entrainment and
soft-entrainment

18. The obtained measures
Synchronization Index and Effectiveness of
Leadership seem to explain the rating of
expert listeners, asked to judge the quality
of performance of the same music
fragments played by the same orchestra
but with different conductors

19. Future research

20. The SIEMPRE EyesWeb Platform
Supports the synchronous recording of multimodal channels

22. Recent publications
• www.infomus.org www.eyesweb.org
• G.Varni, G.Volpe, A.Camurri (2010) A System for Real-Time Multimodal
Analysis of Nonverbal Affective Social Interaction in User-Centric Media.
IEEE Transactions on Multimedia, Vol.12, No.6, pp.576-590.
• G.Varni, M.Mancini, G.Volpe, A.Camurri (2010) A system for mobile active
music listening based on social interaction and embodiment, ACM Mobile
Networks and Applications Intl Journal (ACM MONET), 16(3).
• D.Glowinski, N.Dael, A.Camurri, G.Volpe, M.Mortillaro, K.Scherer (2011)
Towards a Minimal Representation of Affective Gestures. IEEE
Transactions on Affective Computing, March 2011, 2(2).
• G.Volpe, A.Camurri (2011) A system for embodied social active listening
to sound and music content. ACM Journal on Computing and Cultural
Heritage, 4(1).
• Research team: Antonio Camurri, Gualtiero Volpe, Corrado
Canepa, Paolo Coletta, Nicola Ferrari, Simone Ghisio, Giorgio Gnecco,
Donald Glowinski, Maurizio Mancini, Alberto Massari, Francesca Odone,
Stefano Piana, Marcello Sanguineti, Alessandra Staglianò, Giovanna Varni

23. 4th EyesWeb Week
• A Tutorial and Workshop on EyesWeb: how
to design interactive experiments? How to
master the EyesWeb platform?
• Casa Paganini, spring 2014
• See www.infomus.org for updates
• www.youtube.com/InfoMusLab
• www.siempre.infomus.org

24. future research directions ?
• aesthetic appreciation is one of the most
intangible aspects of higher cognition
• music as a window into complex integrative
brain processes: arts exploited to study brain
mechanisms (Zeki & Lamb)
• exploring the rules governing aesthetic
experience has great relevance for
neuroscience and future ICT
• social aesthetics, quality of experience,
the perceived quality of music performance