Interactive Socio-Mobile Systems for Active Experience of Audiovisual Content

386 views
334 views

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
386
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates joint action. Here we compare the expressive movement of violinists when playing solo Vs. in the string quartet music ensemble. promising context for investigating expressive and adaptive interactions [6], [9] as all four musicians contribute equally to the performance and no explicit asymmetry as it can be observed in the orchestra case computing a set of non-verbal expressive cues to characterize individual performance characteristics (e.g., entropy) and hence giving an insight on why some musician succeed in causing other’s behaviour
  • SIEMPRE feasibility study (May 2009): the preparation of the preliminary experimental setup of the “string quartet scenario”, in one of the ecological settings (Casa Pa- ganini at UNIGE) available during the whole duration of the project for experiments. Part- ners involved here were UNIGE and QUB. The red circles show two high-speed and high- resolution video-cameras used for movement analysis. Audio recording: two environmental microphones, and one contact microphone on each instrument for separate musical instru- ments recordings (for audio analysis).
  • SIEMPRE feasibility study (May 2009): the preparation of the preliminary experimental setup of the “string quartet scenario”, in one of the ecological settings (Casa Pa- ganini at UNIGE) available during the whole duration of the project for experiments. Part- ners involved here were UNIGE and QUB. The red circles show two high-speed and high- resolution video-cameras used for movement analysis. Audio recording: two environmental microphones, and one contact microphone on each instrument for separate musical instru- ments recordings (for audio analysis).
  • Support for fine-grain synchronized recording and analysis of different multimodal chanelsTime stampQualysis used for MoCap data and analog sensors (EMG)EW XMI used for synchronizationRepoVizz: web app and repository to browse and reuse the SIEMPRE archive of multimodal dataIllustration of the synchronized audio-video material used for the perceptual experiment. On left side, a view of the Multimodal Setup for the recording of the Quartetto di Cremona at Casa Paganini (University of Genoa). The selected material consisted in (i) audio from piezo-electric microphone attached to the first violinist instrument (ii) High-quality video centered on the first violinist (iii) smtpe time code managed through the EyesWeb software platform to synchronize the audio-video streams.
  • Interactive Socio-Mobile Systems for Active Experience of Audiovisual Content

    1. 1. Interactive Socio-Mobile Systems for ActiveExperience of Audiovisual ContentAntonio Camurri and Alberto Massari{toni, alby}@infomus.orgCasa Paganini – InfoMus Research CentreUniversity of Genoa, Italywww.infomus.orgwww.siempre.infomus.org
    2. 2. Summary• The Casa Paganini – InfoMus ResearchCentre of University of Genoa• How S&T research may benefit from the arts• The European Project SIEMPRE• The EyesWeb software platform• Research perspectives
    3. 3. Real-time measure of expressive gesture and nonverbal social signals (entrainment, leadership).The Virtual Binocular to explore AV content(Socio-mobile) active music listening…Research
    4. 4. Examples of scientific projects at Casa Paganini - InfoMusbased on cross-fertilization of ICT and artFrom artistic project… …to S&T researchMusic Theatre Opera “Outis”Luciano BerioTeatro Alla Scala di Milano (1996)Invisible interfaces for on-stage interactionand synchronization with audioMusic Theatre Opera “Cronaca del Luogo”,Luciano Berio, openingSalzburg Festival (July 1999)Real-time analysis of full-body movement,non-verbal expressive behaviour qualities.EyesWeb software platform.Music Theatre Opera “Un Avatar delDiavolo”, Roberto Doati,La Biennale Venezia (2005)Tangible acoustic interfaces: give thesense of touch to everyday objectsMuseum “Enrico Caruso”,permanent interactive installation “Saladella Musica”, Firenze (2011-)Non-verbal behaviour analysis forindividual and group interaction withcultural heritage contentEU FET11 Closing Performance: TanGOTouching Music” (6 May 2011)Performance built upon scientific results ofthe European ICT FET SIEMPRE Project.Study of music joint performance: stringquartets, orchestra sections withconductor, audience responseS&T research in EU ICT FET SIEMPREProject
    5. 5. Casa Paganini – InfoMus Research CentreThe monumental building of Santa Maria delle Grazie La Nuova
    6. 6. The European Project SIEMPRESocial Interaction and Entrainmentusing Music PeRformance ExperimentationMay 2010 – June 2013Consortium:Casa Paganini - InfoMus, Univ of Genoa (Coordinator)Queens University of BelfastSwiss Center for Affective Sciences, University of GenevePompeu Fabra University, BarcelonaItalian Institute of Technology, GenoaINCO (non EU) Partners:Stanford University, USAWaseda University, Tokyo, JapanVirginia Tech, USA
    7. 7. SIEMPRE: experimentation centeredon music• Music as a window into complex integrative brainprocesses: exploited to study brain mechanisms• String quartet representative of the Self-ManagedTeam (studies in economy and management) in anecologically valid framework: e.g., as a proxy for agroup of employees having a total responsibility for adefined project.• Orchestra: a scenario characterized by social sub-groups (e.g., first and second violins sections) with aleader (the conductor). An orchestra is an exampleof inter-team cooperation inside an organization.• Audience exposed to artistic stimuli.
    8. 8. SIEMPRE research challenges• What factors drive interpersonal synchronisation? How do theyvary according to physical, emotional and social contexts?• Can specific roles (e.g., leadership) be identified inside agroup? Can general principles be discerned concerning theinfluence of individuals over others?• What are the factors that determine feelings of group cohesionor a sense of shared meaning? Can the validity of such reportsbe supported or confirmed by expressive Movement Audio andPhysiological (eMAP) measures?• How does social context affect individual intrapersonalsynchronisation of eMAP and vice-versa? How does theemotion of an individual affect a collaborative creative product?• Which are the neurophysiological foundations of creative groupcommunication?www.siempre.infomus.org
    9. 9. SIEMPRE: collaborations withworld-level artists• Conductors: Riccardo Muti and his«Youth Orchestra Luigi Cherubini»,Pietro Borgonovo, Sera Tokay• Renaud Capuçon, violinist• Quartetto di Cremona, string quartetExperiments also with music studentsfrom Music Conservatories
    10. 10. String Quartet experiments“Self-Managed Team”: all musicians contribute equally tothe task [Gilboa & Tal-Shmotkin 2010]Picture: Quartetto di Cremona during an experiment.
    11. 11. SIEMPRE string quartet scenario: research questions• Which multimodal cues explain the difference betweenplaying alone or in ensemble ?
    12. 12. SIEMPRE string quartet scenario: research questions• Which multimodal cues explain who is the leader ?• Which multimodal cues explain the difference betweena performance engaging an audience and a simplycorrect (but «cold») performance ?
    13. 13. SIEMPRE scenarios: orchestra with conductor
    14. 14. Orchestra scenario: a study• Studying the effectiveness of an establishedleader through a group measure of entrainment• Defining a metric for quantifying sucheffectiveness• Validating the metric with raters (experts)Boerner et al. (2004) model of leadership in orchestra:• conductor’s leadership style is not participative or delegative, but directive• Goal-directed leadership: goal is artistic quality
    15. 15. Soft entrainment• beauty in music lies largely in artistic deviationfrom the exact or rigid (Seashore)• Expressivity is characterized by such deviations• In joint music performance, players do not reachand maintain the maximum level of entrainment• Soft-entrainment: alternation of high and lowentrainment (Yoshida):
    16. 16. Orchestra: Experiment• Recordings of Motion Capture and audio on twostring sections and conductor• Focus on the z-component of bow trajectory ofstring sections (z-direction assumed as thepredominant direction of bow movement)• Segmentation of music phrases
    17. 17. The obtained measures-> Synchronization Index-> Effectiveness of Leadership, in terms of aweighted combination of entrainment andsoft-entrainment
    18. 18. The obtained measuresSynchronization Index and Effectiveness ofLeadership seem to explain the rating ofexpert listeners, asked to judge the qualityof performance of the same musicfragments played by the same orchestrabut with different conductors
    19. 19. Future research
    20. 20. The SIEMPRE EyesWeb PlatformSupports the synchronous recording of multimodal channels
    21. 21. Recent publications• www.infomus.org www.eyesweb.org• G.Varni, G.Volpe, A.Camurri (2010) A System for Real-Time MultimodalAnalysis of Nonverbal Affective Social Interaction in User-Centric Media.IEEE Transactions on Multimedia, Vol.12, No.6, pp.576-590.• G.Varni, M.Mancini, G.Volpe, A.Camurri (2010) A system for mobile activemusic listening based on social interaction and embodiment, ACM MobileNetworks and Applications Intl Journal (ACM MONET), 16(3).• D.Glowinski, N.Dael, A.Camurri, G.Volpe, M.Mortillaro, K.Scherer (2011)Towards a Minimal Representation of Affective Gestures. IEEETransactions on Affective Computing, March 2011, 2(2).• G.Volpe, A.Camurri (2011) A system for embodied social active listeningto sound and music content. ACM Journal on Computing and CulturalHeritage, 4(1).• Research team: Antonio Camurri, Gualtiero Volpe, CorradoCanepa, Paolo Coletta, Nicola Ferrari, Simone Ghisio, Giorgio Gnecco,Donald Glowinski, Maurizio Mancini, Alberto Massari, Francesca Odone,Stefano Piana, Marcello Sanguineti, Alessandra Staglianò, Giovanna Varni
    22. 22. 4th EyesWeb Week• A Tutorial and Workshop on EyesWeb: howto design interactive experiments? How tomaster the EyesWeb platform?• Casa Paganini, spring 2014• See www.infomus.org for updates• www.youtube.com/InfoMusLab• www.siempre.infomus.org
    23. 23. future research directions ?• aesthetic appreciation is one of the mostintangible aspects of higher cognition• music as a window into complex integrativebrain processes: arts exploited to study brainmechanisms (Zeki & Lamb)• exploring the rules governing aestheticexperience has great relevance forneuroscience and future ICT• social aesthetics, quality of experience,the perceived quality of music performance

    ×