Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction
Upcoming SlideShare
Loading in...5

Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction



Despite its central role in interactive media, guidelines for designing and evaluating rhythmic interactions are hard to find. The primary objective of our tutorial is that the participants will learn ...

Despite its central role in interactive media, guidelines for designing and evaluating rhythmic interactions are hard to find. The primary objective of our tutorial is that the participants will learn the fundamentals of the design and evaluation of rhythmicity in ambient and ubiquitous multimodal mobile interaction.

The tutorial is organized by Cumhur Erkut and Antti Jylhä from the Aalto University. The target audience is the interaction specialists and software developers, who want to enhance their interactive applications with rhythmicity. A particularly suitable group is the mobile application developers. Students and general audience with background in HCI, interaction design, or audio signal processing were also welcome to attend. About 10 HCI, game, and interaction design researchers have followed the tutorial and created a lively discussion.



Total Views
Slideshare-icon Views on SlideShare
Embed Views



2 Embeds 6

http://paper.li 5
http://www.docshut.com 1


Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction Document Transcript

    • 29/09/2011  Rhythmic BlueprintsDesign and Evaluation of Rhythmic InteractionCumhur Erkut and Antti JylhäAalto University, School of Electrical EngineeringDepartment of Signal Processing and AcousticsMindTrek 2011 29/09/2011Tampere, FinlandOverview•  Getting know each other•  Sensitizing: interactive rhythms in various scales –  Musical rhythms –  Social rhythms•  Elements of rhythm –  Pulse, beat, meter, and tempo –  Resonance, synchronization, entrainment•  Design and evaluation models, and how we use them•  Conclusions: Directions and Guidelines•  Remember: http://blogs.aalto.fi/rhythmicity/ 29/09/2011 2 1  
    • 29/09/2011  Michael H. Thaut,in Rhythm, Music, and the Brain, pp. 16-17 If we return briefly to the importance of temporal regulation for all our higher cognitive and motor functions, we may have very good reason to believe that rhythm in music, the element of temporal order, has a unique and profound influence on our perceptual processes related to cognition, affect, and motor function. Rhythm may enhance our brain operations through providing structure and anticipation in time. Rhythm may be one of the central processors to optimize our gestalt formation in the basic process of learning and perception. 29/09/2011 3Rhythmic interaction•  Time is of the essence•  A natural human capability –  Walking, hammering, talking… –  Anticipation, mutual coordination•  Examples –  Personal Orchestra {Borchers:2004hx} –  Virtual interactive humanoids {Nijholt:2008ty} –  Percussion robot Haile {Weinberg:2006wl} –  B-keeper {Robertson:2007tf} –  Hand clap interface {Jylha:2009gq} 29/09/2011 4 2  
    • 29/09/2011  Demo: Sonically Augmented Table andRhythmic Interaction {Pesonen:2010ur} 29/09/2011 5Elements of Musical Rhythms•  Pulses: events in a “pulse train” with regular temporal spacing –  The inter-pulse interval always the same –  The basis of rhythm perception {Thaut:2005te}.•  Beats: audible pulse markings –  Sequenced events –  May deviate from exact pulse timings in slight shifts 29/09/2011 6 3  
    • 29/09/2011  Elements of Musical Rhythms•  Tempo –  Metric of (musical) rhythmic “speed” (rate of pulses/beats) –  Inversely proportional to inter-pulse interval –  Often measured as beats per minute (BPM) –  In music, never completely stable•  Meter –  Defines the rhythmic structure in music (and vice versa) –  Often expressed as beats per measure •  E.g., 3/4, 4/4 –  Relates to accentuation 29/09/2011 7Related concepts•  Vibration: Mechanical response of a body to an external stimuli•  Frequency: Vibrations per second•  Resonance: Tendency to vibrate at a certain frequency•  Entrainment: A process between multiple bodies to align their rhythmic resonances•  Synchronization: The quasi-stable state of entrainment 29/09/2011 9 4  
    • 29/09/2011  Time-sensitive crowded scenes in movies 29/09/2011 10Laws are simple: use in media? 29/09/2011 11 5  
    • 29/09/2011  Sound synthesis, control, and hierarchicalevents 29/09/2011 12 ClaPD 29/09/2011 13 6  
    • 29/09/2011   29/09/2011 14Tutorial in a Nutshell … 29/09/2011 15 7  
    • 29/09/2011   Hand clapping interface for sonic interactions (AM ’08 / CHI ’09) 29/09/2011 16Applications•  Potential in –  Games and entertainment –  Sound design –  New HCI schemes•  Three example cases 1.  Hand-clap driven sampler 2.  Controlling music tempo 3.  Synchronizing a virtual audience 29/09/2011 17 8  
    • 29/09/2011  System dataflow Tempo estimation Sound Hand clap Application User input detection Clap type identification Sonic feedback 29/09/2011 18Music tempo control•  The user claps to control the tempo of music –  BPM of the user s clapping is mapped to that of the music29/09/2011 19 9  
    • 29/09/2011  Virtual audience•  The user claps to synchronize a virtual audience with her clapping•  The user can sync the audience with or without reference music•  Interaction is immediate –  The user is part of the clapping crowd29/09/2011 20Demo video•  http://www.youtube.com/watch?v=7HLYGkayAGA 29/09/2011 21 10  
    • 29/09/2011  Interface implementation with PD•  Clap detection: [bonk~] (Puckette98)!•  Tempo estimation: [rhythm_estimator] (Seppänen01)!•  Combination: [clap_tracker]!29/09/2011 22Evaluation•  Informal evaluation –  2 subjects tested the example applications –  Interface was found easy to use –  Both subjects found that the tempo of the virtual audience or the music drove their clapping•  Some latency appears in the system –  Mostly from buffering and computations –  Not reported as disturbing by the subjects 29/09/2011 23 11  
    • 29/09/2011  Negotiation•  Perceived tempo affected subjects clapping –  The user negotiates with the computer to set the tempo •  Mutual coordination, ”dual-drive” –  Process analogue: musical ensemble 29/09/2011 24Explorations of rhythmic interaction withdancers {Erkut:2009wu} 29/09/2011 25 12  
    • 29/09/2011  Design Blueprints: our version Based on Z. Obrenovic, J. Abascal, and D. Starcevic, “Universal cite{Erkut:2011ta} accessibility as a multimodal design issue,” Communications of the ACM, 29/09/2011 vol. 50, no. 5, pp. 83–88, 2007. 28Rhythmic musical interface 29/09/2011 29 13  
    • 29/09/2011   29/09/2011 30iPalmas (AM ‘09)•  An interactive Flamenco rhythm tutor for hand clapping cite{Jylha:2009uc} 29/09/2011 31 14  
    • 29/09/2011  Flamenco 29/09/2011 32Rhythm in Flamenco•  Compas = meter in Flamenco•  Usually 12-beat cycles –  Accentuation on certain beats depending on style, e.g. bulerias: X - - X--X-X-X-•  Lots of percussion –  Footwork, hand clapping, instruments…•  Focal performer leads –  Improvisation requires communication 29/09/2011 33 15  
    • 29/09/2011  Palmas•  Supporting the compas•  Palmero/-a•  Two types –  Hard and soft•  Follow the focal performer http://www.youtube.com/watch?v=Yo38h7Wdc88 29/09/2011 34Palmas skill requirements1.  Sordas and fuertes2.  Steady and accurate basic accompaniment3.  Decorative clapping4.  Starting to clap after silence5.  Reacting to tempo changes6.  Reacting to rhythmical cues 29/09/2011 35 16  
    • 29/09/2011  System (1)•  Flamenco palmas synthesis and tutor –  Teaches the skills –  Provides synthetic examples and accompaniment•  User’s clapping as input•  Auditory and visual feedback 29/09/2011 36System (2) Clap User analysis Virtual GUI tutor Clap synthesis Visual feedback 29/09/2011 37 17  
    • 29/09/2011  System (3) 29/09/2011 38Synthetic palmas•  ClaPD hand clap synthesis engine (Peltola et al. 2007) –  Enveloped noise burst –  Band-pass filter defines clap type –  Coupled oscillator model for rhythmic interaction•  Pre-defined palmas patterns set synthesis parameters•  User and/or tutor set the tempo 29/09/2011 39 18  
    • 29/09/2011  Analysis of clapping•  Clap type –  Classification by template matching ( [bonk~] )•  Accent detection –  Loudness-based•  Tempo –  Probabilistic estimate based on inter-onset intervals•  Tempo steadiness –  Temporal variance•  Correct accentuation –  Comparison of user’s and tutor’s accent patterns 29/09/2011 40Feedback (1)•  Clapping sounds from the system –  Synchrony, accents, tempo changes•  Numeric feedback and sliders –  Performance metrics•  Visualization –  Dancing circles –  Transcription of the pattern 29/09/2011 41 19  
    • 29/09/2011  Feedback (2) 29/09/2011 42Prototype•  Built on Pure Data (Pd)•  Synthesis of virtual palmeros•  Clap type training•  Basic compas training –  Tutor speeds up when the user gets better -> Tempo changes trained as well 29/09/2011 43 20  
    • 29/09/2011  Interlude: evaluation Human System 29/09/2011 44Interlude: evaluation Human Interaction System 29/09/2011 45 21  
    • 29/09/2011  What can/should be evaluated•  Human factors –  Rhythmic capabilities •  Perception •  Production•  System factors –  I/O latency –  Computational complexity•  Interaction –  Fluency, naturalness –  Etc.•  The whole triptych 29/09/2011 46Methods of evaluation•  Qualitative –  Characterization of the capabilities/properties/phenomena •  Understanding –  Interviews, observations, etc.•  Quantitative –  Measurable quantities •  For quantifying the qualitative attributes •  Also can be used to derive qualitative results 29/09/2011 47 22  
    • 29/09/2011  Human factors•  Sensorimotor synchronization (SMS {Repp:2005tb}) –  Synchronization between sensory stimulus and motor response –  Often metronome-based evaluation or simple rhythmic tasks •  “tapping to the beat” –  Also, movement-to-music evaluation •  How movement trajectories correlate with the rhythmic elements•  Rhythmic capabilities –  Production: e.g., rhythmic stability, accentuation –  Perception: modal acuity –  Some people can be actually “rhythm-deaf” (Phillips-Silver et al. 2011) 29/09/2011 48System factors•  I/O latency –  Plays a role in rhythmic applications •  In principle a constraint, but can be overcome in cyclic interactions•  I/O modalities –  Auditory, visual, haptic –  Streaming (continous) vs. event-based (discrete)•  Computational properties –  Complexity, required processing power, memory, … 29/09/2011 49 23  
    • 29/09/2011  Evaluating interaction•  Qualitative studies•  User tests –  Often based on simple tasks•  In-performance evaluation•  Hybrid methods 29/09/2011 50However…•  … evaluation of individual components is valuable, but does not necessarily explain everything  Wholesome evaluation needed 29/09/2011 51 24  
    • 29/09/2011  Case example: iPalmas evaluation (CMJ ’11)•  Subjective experiment AND objective description –  Both qualitative and quantitative measures cite{Jylha2011:CMJ} 29/09/2011 52iPalmas revisited Audio recording Metrics: - Tempo Detection and - Accent Text file log Hand claps analysis - Deviation User - Time stamp Audio Synthetic feedback Tempo Post-analysis hand claps, Visual Transcription, feedback circles, Tutor numeric feedback Audio Mostly 2011 Jylhä & Erkut 29/09/2011 53 25  
    • 29/09/2011  Subjective experiment•  To evaluate 1.  Human factors 2.  System 3.  Interaction•  Rhythmic tasks with realistic rhythmic patterns –  Training, testing•  Logging of measurable quantities•  Observations throughout the experiment•  Interviews, verbal comments•  Post-experiment questionnaire 29/09/2011 54Evaluation of the first iPalmas system•  Performed to evaluate rhythmic interaction and the system (Jylhä et al. 2011)•  16 subjects•  4 patterns•  4 tutors (audio and audiovisual, adaptive and fixed tempo)•  Training phase and test phase, recall phase Audio Mostly 2011 Jylhä & Erkut 29/09/2011 55 26  
    • 29/09/2011  Evaluation of the first iPalmas system Audio Mostly 2011 Jylhä & Erkut 29/09/2011 56Evaluation of the first iPalmas system•  Auditory information (clapping) the key to learning and performing –  The reverb disturbing to some•  Transcription helpful•  Circles pretty but not very helpful•  Numeric FB useful only to some –  Helped in “tuning in” to clapping the accents•  Temporal variation in clapping –  Tutor stops  subject speeds up•  Adaptive mode helps performance Audio Mostly 2011 Jylhä & Erkut 29/09/2011 57 27  
    • 29/09/2011  Tempo speed-up and fluctuation Audio only tutor (steady) Audio only tutor (adaptive) 200 200 BPM BPM 180 180 Subject11 160 Subject1 160 0 20 40 60 0 20 40 60 Time (sec) Time (sec) Audiovisual tutor (steady) Audiovisual tutor (adaptive) 200 200 BPM BPM 180 180 160 160 0 20 40 60 0 20 40 60 Time (sec) Time (sec) cite{Jylha2011:CMJ} 29/09/2011 58Quantitative findings•  Average time to start clapping: 10 cycles (40 s)•  Accentuation correctness –  Audio-only: 68.1 % –  Audiovisual: 73.7 % –  Fixed tempo: 67.6 % –  Adaptive tempo: 74.2 %•  Indication: for accentuated beats, IOI slightly longer (344.7 ms) than for non-accentuated beats (343 ms) –  More prominent with the adaptive tutor cite{Jylha2011:CMJ} 29/09/2011 59 28  
    • 29/09/2011  Evaluation Blueprints: SummarizeQualitative, quantitative, and metrics cite{Erkut:2011ta} 29/09/2011 60Direction1: SID http://en.wikipedia.org/wiki/Sonic_interaction_design 29/09/2011 61 29  
    • 29/09/2011  Direction 2: Interaction Gestalts andAttributes User Interactive experience artifact •  User experience •  Artifact qualities properties Interaction gestalt {Lim:2009tw} 29/09/2011 62Directions 1+2 Combined: New basic sonicinteraction design http://www.room50.org/stefanodellemonache See, cite{Rocchesso:2009wi}, 29/09/2011 and cite{Franinovic:2009wl} 63 30  
    • 29/09/2011  Other directions•  Social rhythms: H. Lefebvre, “Rhythmanalysis: Space, Time and Everyday Life,” Book, pp. 1–129, 2004 => CSCW => Social games•  Rhythms and emotions: Thaut’05, but also recent CHI papers, e.g. cite{Epp:2011hz}•  Gamification!•  What else? 29/09/2011 64References•  C. Epp, M. Lippold, and R. L. Mandryk, “Identifying emotional states using keystroke dynamics,” in CHI’11, Vancouver, BC, Canada, 2011, pp. 715–724.•  C. Erkut, A. Jylhä, and R. Discioglu, “A structured design and evaluation model with application to rhythmic interaction displays,” in Proc. New Interfaces for Musical Expression (NIME), Oslo, Norway, 2011, pp. 477–480.•  C. Erkut, A. Jylhä, and I. Ekman, “Recent advances in exploring self-induced sonic interactions in the context of performing arts,” in Intl. Workshop on Haptic and Audio Interaction Design, 2009, pp. 1–2.•  K. Franinovic, “Toward Basic Interaction Design,” available online at http://tdd.elisava.net/coleccion/25/franinovic-en•  A. Jylhä and C. Erkut, “A hand clap interface for sonic interaction with the computer,” CHI-EA, Apr. 2009.•  A. Jylhä, C. Erkut, I. Ekman, and K. Tahiroglu, “iPalmas - An interactive flamenco rhythm machine,” Proc. Audio Mostly, pp. 1–2, May. 2009.•  A. Jylhä, I. Ekman, C. Erkut, and K. Tahiroglu, “Design and Evaluation of Rhythmic Interaction with an Interactive Tutoring System,” Computer Music Journal, vol. 35, no. 2, pp. 36–48. 2011. 29/09/2011 68 31  
    • 29/09/2011  References•  Y.-K. Lim, E. Stolterman, H. Jung, and J. Donaldson, “Interaction gestalt and the design of aesthetic interactions,” Proc.. Conf. Designing Pleasurable Products and Interfaces, pp. 239–254, 2007.•  M. Pesonen, Sonically Augmented Table and Rhythmic Interaction, Master’s thesis, Aalto Univerity, School of Electrical Engineering, 2010.•  B. Repp, “Sensorimotor synchronization: A review of the tapping literature,” Psychonomic Bulletin and Review, vol. 12, no. 6, pp. 969–992, 2005.•  A. Robertson and M. Plumbley, “B-Keeper: A Beat-Tracker for Live Performance,” Proc. NIME, pp. 234–237, 2007.•  D. Rocchesso, P. Polotti, and S. D. Delle Monache, “Designing Continuous Sonic Interaction,” Intl. J. Design, vol. 3, no. 3, pp. 13–25, May. 2009.•  M. Thaut, Rhythm, Music, and the Brain. New York, NY, USA: Routledge, 2005.•  G. Weinberg and S. Driscoll, “Robot-human interaction with an anthropomorphic percussionist,” CHI 06, Apr. 2006. 29/09/2011 69 Discussion •  (Edward T) Halls studies (on cultural rhythms) and (Saul) Greenbergs work on social rhythms (awareness) •  Should the tutorial be called "interaction with rhythms" instead of rhythmic interaction? •  Synesthesia as a resource? •  Behavioral change: negotiation already used in game design (transfer of adaptation) •  Adaptive coaching •  Language learning and prononciation 29/09/2011 70 32