Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Interactive Socio-Mobile Systems for ActiveExperience of Audiovisual ContentAntonio Camurri and Alberto Massari{toni, alby...
Summary• The Casa Paganini – InfoMus ResearchCentre of University of Genoa• How S&T research may benefit from the arts• Th...
Real-time measure of expressive gesture and nonverbal social signals (entrainment, leadership).The Virtual Binocular to ex...
Examples of scientific projects at Casa Paganini - InfoMusbased on cross-fertilization of ICT and artFrom artistic project...
Casa Paganini – InfoMus Research CentreThe monumental building of Santa Maria delle Grazie La Nuova
The European Project SIEMPRESocial Interaction and Entrainmentusing Music PeRformance ExperimentationMay 2010 – June 2013C...
SIEMPRE: experimentation centeredon music• Music as a window into complex integrative brainprocesses: exploited to study b...
SIEMPRE research challenges• What factors drive interpersonal synchronisation? How do theyvary according to physical, emot...
SIEMPRE: collaborations withworld-level artists• Conductors: Riccardo Muti and his«Youth Orchestra Luigi Cherubini»,Pietro...
String Quartet experiments“Self-Managed Team”: all musicians contribute equally tothe task [Gilboa & Tal-Shmotkin 2010]Pic...
SIEMPRE string quartet scenario: research questions• Which multimodal cues explain the difference betweenplaying alone or ...
SIEMPRE string quartet scenario: research questions• Which multimodal cues explain who is the leader ?• Which multimodal c...
SIEMPRE scenarios: orchestra with conductor
Orchestra scenario: a study• Studying the effectiveness of an establishedleader through a group measure of entrainment• De...
Soft entrainment• beauty in music lies largely in artistic deviationfrom the exact or rigid (Seashore)• Expressivity is ch...
Orchestra: Experiment• Recordings of Motion Capture and audio on twostring sections and conductor• Focus on the z-componen...
The obtained measures-> Synchronization Index-> Effectiveness of Leadership, in terms of aweighted combination of entrainm...
The obtained measuresSynchronization Index and Effectiveness ofLeadership seem to explain the rating ofexpert listeners, a...
Future research
The SIEMPRE EyesWeb PlatformSupports the synchronous recording of multimodal channels
Recent publications• www.infomus.org www.eyesweb.org• G.Varni, G.Volpe, A.Camurri (2010) A System for Real-Time Multimodal...
4th EyesWeb Week• A Tutorial and Workshop on EyesWeb: howto design interactive experiments? How tomaster the EyesWeb platf...
future research directions ?• aesthetic appreciation is one of the mostintangible aspects of higher cognition• music as a ...
Interactive Socio-Mobile Systems for Active Experience of Audiovisual Content
Upcoming SlideShare
Loading in …5
×

Interactive Socio-Mobile Systems for Active Experience of Audiovisual Content

549 views

Published on

Published in: Technology, Business
  • Be the first to comment

Interactive Socio-Mobile Systems for Active Experience of Audiovisual Content

  1. 1. Interactive Socio-Mobile Systems for ActiveExperience of Audiovisual ContentAntonio Camurri and Alberto Massari{toni, alby}@infomus.orgCasa Paganini – InfoMus Research CentreUniversity of Genoa, Italywww.infomus.orgwww.siempre.infomus.org
  2. 2. Summary• The Casa Paganini – InfoMus ResearchCentre of University of Genoa• How S&T research may benefit from the arts• The European Project SIEMPRE• The EyesWeb software platform• Research perspectives
  3. 3. Real-time measure of expressive gesture and nonverbal social signals (entrainment, leadership).The Virtual Binocular to explore AV content(Socio-mobile) active music listening…Research
  4. 4. Examples of scientific projects at Casa Paganini - InfoMusbased on cross-fertilization of ICT and artFrom artistic project… …to S&T researchMusic Theatre Opera “Outis”Luciano BerioTeatro Alla Scala di Milano (1996)Invisible interfaces for on-stage interactionand synchronization with audioMusic Theatre Opera “Cronaca del Luogo”,Luciano Berio, openingSalzburg Festival (July 1999)Real-time analysis of full-body movement,non-verbal expressive behaviour qualities.EyesWeb software platform.Music Theatre Opera “Un Avatar delDiavolo”, Roberto Doati,La Biennale Venezia (2005)Tangible acoustic interfaces: give thesense of touch to everyday objectsMuseum “Enrico Caruso”,permanent interactive installation “Saladella Musica”, Firenze (2011-)Non-verbal behaviour analysis forindividual and group interaction withcultural heritage contentEU FET11 Closing Performance: TanGOTouching Music” (6 May 2011)Performance built upon scientific results ofthe European ICT FET SIEMPRE Project.Study of music joint performance: stringquartets, orchestra sections withconductor, audience responseS&T research in EU ICT FET SIEMPREProject
  5. 5. Casa Paganini – InfoMus Research CentreThe monumental building of Santa Maria delle Grazie La Nuova
  6. 6. The European Project SIEMPRESocial Interaction and Entrainmentusing Music PeRformance ExperimentationMay 2010 – June 2013Consortium:Casa Paganini - InfoMus, Univ of Genoa (Coordinator)Queens University of BelfastSwiss Center for Affective Sciences, University of GenevePompeu Fabra University, BarcelonaItalian Institute of Technology, GenoaINCO (non EU) Partners:Stanford University, USAWaseda University, Tokyo, JapanVirginia Tech, USA
  7. 7. SIEMPRE: experimentation centeredon music• Music as a window into complex integrative brainprocesses: exploited to study brain mechanisms• String quartet representative of the Self-ManagedTeam (studies in economy and management) in anecologically valid framework: e.g., as a proxy for agroup of employees having a total responsibility for adefined project.• Orchestra: a scenario characterized by social sub-groups (e.g., first and second violins sections) with aleader (the conductor). An orchestra is an exampleof inter-team cooperation inside an organization.• Audience exposed to artistic stimuli.
  8. 8. SIEMPRE research challenges• What factors drive interpersonal synchronisation? How do theyvary according to physical, emotional and social contexts?• Can specific roles (e.g., leadership) be identified inside agroup? Can general principles be discerned concerning theinfluence of individuals over others?• What are the factors that determine feelings of group cohesionor a sense of shared meaning? Can the validity of such reportsbe supported or confirmed by expressive Movement Audio andPhysiological (eMAP) measures?• How does social context affect individual intrapersonalsynchronisation of eMAP and vice-versa? How does theemotion of an individual affect a collaborative creative product?• Which are the neurophysiological foundations of creative groupcommunication?www.siempre.infomus.org
  9. 9. SIEMPRE: collaborations withworld-level artists• Conductors: Riccardo Muti and his«Youth Orchestra Luigi Cherubini»,Pietro Borgonovo, Sera Tokay• Renaud Capuçon, violinist• Quartetto di Cremona, string quartetExperiments also with music studentsfrom Music Conservatories
  10. 10. String Quartet experiments“Self-Managed Team”: all musicians contribute equally tothe task [Gilboa & Tal-Shmotkin 2010]Picture: Quartetto di Cremona during an experiment.
  11. 11. SIEMPRE string quartet scenario: research questions• Which multimodal cues explain the difference betweenplaying alone or in ensemble ?
  12. 12. SIEMPRE string quartet scenario: research questions• Which multimodal cues explain who is the leader ?• Which multimodal cues explain the difference betweena performance engaging an audience and a simplycorrect (but «cold») performance ?
  13. 13. SIEMPRE scenarios: orchestra with conductor
  14. 14. Orchestra scenario: a study• Studying the effectiveness of an establishedleader through a group measure of entrainment• Defining a metric for quantifying sucheffectiveness• Validating the metric with raters (experts)Boerner et al. (2004) model of leadership in orchestra:• conductor’s leadership style is not participative or delegative, but directive• Goal-directed leadership: goal is artistic quality
  15. 15. Soft entrainment• beauty in music lies largely in artistic deviationfrom the exact or rigid (Seashore)• Expressivity is characterized by such deviations• In joint music performance, players do not reachand maintain the maximum level of entrainment• Soft-entrainment: alternation of high and lowentrainment (Yoshida):
  16. 16. Orchestra: Experiment• Recordings of Motion Capture and audio on twostring sections and conductor• Focus on the z-component of bow trajectory ofstring sections (z-direction assumed as thepredominant direction of bow movement)• Segmentation of music phrases
  17. 17. The obtained measures-> Synchronization Index-> Effectiveness of Leadership, in terms of aweighted combination of entrainment andsoft-entrainment
  18. 18. The obtained measuresSynchronization Index and Effectiveness ofLeadership seem to explain the rating ofexpert listeners, asked to judge the qualityof performance of the same musicfragments played by the same orchestrabut with different conductors
  19. 19. Future research
  20. 20. The SIEMPRE EyesWeb PlatformSupports the synchronous recording of multimodal channels
  21. 21. Recent publications• www.infomus.org www.eyesweb.org• G.Varni, G.Volpe, A.Camurri (2010) A System for Real-Time MultimodalAnalysis of Nonverbal Affective Social Interaction in User-Centric Media.IEEE Transactions on Multimedia, Vol.12, No.6, pp.576-590.• G.Varni, M.Mancini, G.Volpe, A.Camurri (2010) A system for mobile activemusic listening based on social interaction and embodiment, ACM MobileNetworks and Applications Intl Journal (ACM MONET), 16(3).• D.Glowinski, N.Dael, A.Camurri, G.Volpe, M.Mortillaro, K.Scherer (2011)Towards a Minimal Representation of Affective Gestures. IEEETransactions on Affective Computing, March 2011, 2(2).• G.Volpe, A.Camurri (2011) A system for embodied social active listeningto sound and music content. ACM Journal on Computing and CulturalHeritage, 4(1).• Research team: Antonio Camurri, Gualtiero Volpe, CorradoCanepa, Paolo Coletta, Nicola Ferrari, Simone Ghisio, Giorgio Gnecco,Donald Glowinski, Maurizio Mancini, Alberto Massari, Francesca Odone,Stefano Piana, Marcello Sanguineti, Alessandra Staglianò, Giovanna Varni
  22. 22. 4th EyesWeb Week• A Tutorial and Workshop on EyesWeb: howto design interactive experiments? How tomaster the EyesWeb platform?• Casa Paganini, spring 2014• See www.infomus.org for updates• www.youtube.com/InfoMusLab• www.siempre.infomus.org
  23. 23. future research directions ?• aesthetic appreciation is one of the mostintangible aspects of higher cognition• music as a window into complex integrativebrain processes: arts exploited to study brainmechanisms (Zeki & Lamb)• exploring the rules governing aestheticexperience has great relevance forneuroscience and future ICT• social aesthetics, quality of experience,the perceived quality of music performance

×