Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions

Mixing Interaction, Soniﬁcation, Rendering and Design - The art of creating sonic interactions Eoin Brazil EG_IT 22 & 23 October, Verona. Italy

Soniﬁcation

Using Sound to Effectively Convey Information Ben Cohen, co- founder of Ben & Jerry’s Ice-cream Company Talking against the size and cost of the nuclear stockpile in the United States.

203 Cartooniﬁcation evel models: bouncing, breaking, rolling, crumpling, pouring Maisy drinking with a straw (From “Happy Birthday, Maisy” by Lucy Cousins, Candlewick Press, 1998.)

Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8 2.5 0.333 1 0.333 0.692 0.176 0.138 0.354 0.058 0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 2.1 1.028 1 1.028 0.576 0.070 0.077 2.401 0.162 1.7 0.217 1 0.217 0.885 0.259 1.087 0.739 1.005 1.3 0.507 1 0.507 0.862 0.198 0.309 0.578 1.107 0.570 1 0.570 0.866 0.301 0.434 0.526 0.674 0.8 0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 0.4 0.530 1 0.530 0.042 0.852 0.250 0.053 0.185 0.497 1 0.497 1.590 1.365 1.175 1.278 0.101 0 Area-1 Area-2 Area-3 Area-4 0.167 1 0.167 1.261 0.031 0.658 0.306 1.289 Area-5 Area-6 Area-7 Area-8 An Example of Complex datasets - Adding sound can elaborate streams Auditory Graphing and temporality of the data

Sonic Interaction Design • Sound and interaction are both process oriented • Interactive systems are complex - sound is able to convey complex information very quickly • Interactive systems are “disappearing”, sound can tell us about their presence and their status • We have learned to understand complex soundscapes -> ubiquitous, visualisation, and many areas of computing may proﬁt from the use of sound

SHAPE Exhibition Interactive Sound Object Models for Exploration & Education Re-Tracing the Past

Solids Liquids Gases Fracture Impact Friction Fluid ﬂowing Bubble Turbulence Explosion Low Level Model Model Model Model Model Model Model Models Fracturing Crumpling Hitting Rolling Rubbing Braking Flowing Burbing Dripping Whoosing Burning Exploding Popping Basic Events and Textures Pouring Filling Splashing Sucking Derived Walking Crushing Bouncing Dropping Breaking Sliding Squeaking Processes Rolling / Squeaking Rubbed Vacuum Footsteps Can Falling on gravel crushing coin breaking wheel door glass cleaner Simulation Paper Struck Glass Bowed Examples Struck bar crumpling string cleaning string SOb-ifying Impact Models for Hitting, Rolling, or Crumbling Textures / the Exhibition Basic Events

Virtual Touch Machine Interacting with one of the four `mysterious’ objects Using a multimodal (Audio- Haptic-Visual) interface to explore its materiality and size Real-Time Interaction

SOb-ifying the Exhibition

SOb - Sounding Interactive Sound Object Models for Exploration & Education Object Project

Museum Installations - Lessons & Considerations • Enhance • Design activities / interpretation consultations ✦ observational • Fun yet interactive, studies spark ✦ consult docents / • Active not passive experts environment / ✦ understand interfaces physical/spatial constraints

Soundscapes & J. Tardieu, P. Susini, F. Poisson, P. Lazareff, and S. McAdams Train Stations France, 2008

Connecting L. Pellarin, N. Bottcher, J. M. Olsen, O. Gregersen, S. Seraﬁn, Strangers at a and M. Guglielmi Denmark, 2005 Train Station Bouncing Frequency / Airplane Mirror Balls Amplitude Synth Triggers

Ping Photon Wind Shepard Drive Tones Shannon Portal - Interactive Sounds for Auditory Display for Communication & Entertainment Public Spaces

Analysis of the Sound & Space common fate’ and ‘good continuation’ [11].

casual users where auditory display was a significant element to facilitate user interaction. We used an iterative design process, Brainstorms, Concept from simple onomatopoeicPublic Spacescomplex Auditory representations, to Auditory starting sound object models in Pure Data. The system was evaluated at Generation each stage, from the lab to the final public setting. The problems Linking ubiquitious auditory addressed covered the representations of left-right, up-down, displays design to evaluation in Display Design and the amount of movement by the user or groups of users. In public environments addition to this, it was important that the auditory display would attract attention when users were within control range of the system, i.e. an affordance that invited and allowed users to Process for Sonic Sketching sound using discover functionality. Mood clips (mood boards or Boards onomatopoeia) 1. INTRODUCTION In our Shared Worlds project we explored designing ubiquitous computing for public environments. Part of this Public Spaces Figure 1: Shannon Portal installation exploration was to design, build and evaluate installations in Video Bodystorming / public environments. The work reported in this paper is about the Shannon Portal, a complex Prototyping interactive installation in Roleplay the Shannon Airport in the West of Ireland. The overall design is Inteaction described in [1], while in this paper we focus on the design of the auditory display element of the installation. One of the Evaluate issues that we wanted to explore in the project was the use of Evaluate large ambient displays and public interaction with such displays. As the installation at Shannon required an interactive image gallery to be displayed, we designed a purpose-built back-projection system. An overhead camera Evaluate tracked users’ Prototype using PD / movements in front of the display, allowing the users to move a virtual Max/MSP or similar magnifying glass across the image gallery on screen, by moving their body in front of the display. See Figure 1 and 2. Over several iterations we developed hardware and software for the display and explored different mappings of users’ h = Pitc actions to movement of the virtual magnifying glass on the ack tion projection screen. For reasons outside the scope of this paper, it t-B c F ron Dire was decided that it was not the users’ location that would Evaluate Left-Right Panning = Figure 2: Back-projection, loudspeakers 2 video and Code interaction and control the movement, but the users’ movement. The rationale 1 Amplitude Panning camera for this sound as there was only one virtual magnifying glass was that object model but there could be multiple simultaneous users, this mapping 3 Movement within space = Loudness would encourage collaboration and engagement. 1.1. The challenge of used for fourth prototype public Mappings auditory display in While testing and evaluating the system in the lobby outside environments our lab in the Interaction Design Centre at the University of (Shepard Tones) Limerick, we noted that users who occasionally passed by the Is has been noted in numerous papers (e.g. [2-4]) that

Public Spaces - Lessons & Considerations • Engaging (virtuality) • Design activities / consultations • Visitors / Working within the space ✦ observational studies • Privacy / Intrusion ✦ interviews • Masking and ✦ understanding the soundscape issues physical /cultural / (peak vs non-peak spatial constraints times)

Hanging Gardens - Auditory Installation Collaboration with Jurgen Simpson Linked two gallery spaces in Limerick and Belfast Explore circadian rhythms of the spaces Motion sensors and time of day, site-speciﬁc

Artistic Installations - Lessons & Considerations • Interpretation vs • Design activities / artistic consultations • Engaging ✦ aesthetic issues ✦ distance / • Group / Individual repairability ✦ understand • Consider temporal physical/spatial constraints constraints

Action Sound Function No touch N/A Touch area outside button N/A HCI Design and Enter button area Tick Interactive Move finger on button Friction sound Sonification for Fingers and Ears Exit button area Tack Lift finger off button Tock Select / Activate function

Where and How to start designing with Sound • How to select and classify suitable sounds for a particular interaction design ? • TaDa - select sounds to communicate the dimensions and directions with relation to their actions and to that of the system e.g. ears-lead-eyes design pattern • Sounds linked to physical objects as dynamic entities with manipulation / interaction • Multidimensional information carrier - size, shape, material, distance, speed, and emotional expression

Selection of Approaches Depending on Question • Evaluating Perceptual Qualities of Single Auditory Stimuli • Identiﬁcation Tasks, Context-Based Ratings, Attribute Ratings • Evaluating Relations Among Auditory Stimuli • Discrimination Trials, Similarity Ratings, Sorting Tasks, MDS, CA, PCA, Pathﬁnder Analysis • Evaluation of an application or an interface • Surveys, Verbal Protocols, Heuristics, Task lists, etc.

Sound Creation 1 Context and Auditory Display Definition Review of existing Everyday Musical Sounds Sounds 2 2 Considerations Considerations Activity, Sound Like, Activity, Mapping, Vocalisations / Onomatopoeia Instruments Find Closest Match in F/X Library / Use Foley 2 Explore Existing Earcon Hierarchies / Create New Hierarchy 2 approaches, methods, and 2 3&4 Analyse Sound and Create Earcon Family Consider Action/s and and Test - Does it Objects making it sound right? Inconsistent Attributes or Mappings techniques. 3&4 Confused Sounds Create the Sound and Test - Does it sound right? 5 6 Evaluate Scalings / Auditory Characterisation Mappings of Story / Scene / Account Inconsistent Attributes 7 8 9 8 Elicit Descriptors & TaDa Analysis Narrative Sound Sonic Map Creation for or Mappings Constructs Artefact Creation Account / Scene / Story Confused Sounds Sound Creation 10 Rating of Constructs & 11 Descriptor Categorisation Structuring Of Concepts Hearsay Analysis Personal Constructs Groups of Concepts Descriptor Categories Associated Ratings 17 Evaluation Inconsistent Attributes 12 13 or Mappings Causal Uncertainty Measures Structuring Of Constructs Results Confused Sounds Confused Sounds Sound Analysis Groups of Constructs 14 Definition Of Attributes Construction Of Scales Attribute Scales 15 Validation Of Scales Preliminary Results Mismatched Sounds 16 Category Refinement 17 Evaluation Results Sound Analysis

Stage 1 - Develop consensus Select Listening Panel, Sounds Select Listeners, Sounds & Group or Stage 1 - Develop individual & Attributes to explore Attributes to explore Individual descriptive language descriptive language based Play audio stimuli (~100-200) Play audio stimuli (~100-200) approaches Determine attribute scales, Determine attribute scales, descriptors, key audio groups descriptors, key audio groups ranking / rating of data Stage 2 - Participant Stage 2 - Participant performance data Gather global Gather end Participant ranks, rates, or scores from user groups data naive preference participants data to preferences Stage 3 - Mapping Stage 3 - Mapping of data Map preference / performance Mapping of data using data using statistical methods statistical methods

CLOSED project, 2008, Functional Artefacts Ircam, Verona, Berlin Sonic Interaction Functional Artefacts Design Process for Linking design to evaluation Functional Artefacts including basic design methods Analyse, Shape, Describe Interaction Materials Combine Experience Interaction Interaction in Context Gestalt Artefact Evaluate Rolling Clapping 1 Examples: Clapping 2

CLOSED project, 2008, Functional Artefacts Ircam, Verona, Berlin Sonic Interaction Functional Artefacts Design Process for Linking design to evaluation Functional Artefacts including basic design methods Analyse, Shape, Describe Interaction Materials Combine Experience Interaction Interaction in Context Gestalt Artefact Rolling Evaluate Clapping 1 Examples: Clapping 2

Explorations and Methods • Listening test • Causal approaches uncertainty • Textual • 2-D perceptual descriptor scaling analysis • Repertory grid • Sonic Mapping technique

Graham Coleman, Sonic Mapping PhD Thesis Scotland, 2008 Listening Classifying Background Sounds Foreground Sounds * Ac als als Ac tio gn gn tio + ns ? Si Si ns 0 0 * Emotions Contextual Emotions Sounds Written descriptions of Sorting Visible Hidden sounds and their type Ac als ti gn on Si * Everyday s + Speech 0 Music Emotions ? Abstract / Unknown

Graham Coleman, Sonic Mapping PhD Thesis Scotland, 2008 Background Sounds Foreground Sounds Listening Classifying * Ac als als Ac tio gn gn tio + ns ? Si Si ns 0 0 * Emotions Contextual Emotions Sounds Written descriptions of Visible Hidden sounds and their type Ac als tio gn ns * Everyday Si + Speech Sorting 0 Music Emotions ? Abstract / Unknown

High Arousal 1 2 3 4 5 6 7 8 9 9 9 Zizi - The alarmed excited astonished 8 afraid close/sit delighted 8 7 angry stroke happy 7 Affectionate Couch Displeasure 6 annoyed pleased 6 Pleasure petting 5 frustrated content 5 4 miserable none serene 4 Stephen Barrass, Linda Davy 3 depressed calm 3 & Kerry Richens 2 bored relaxed 2 Australia, 2003 1 tired sleepy 1 1 2 3 4 5 6 7 8 9 Low Arousal

High Arousal 1 2 3 4 5 6 7 8 9 9 alarmed excitedZizi - The 9 astonished 8 afraid Affectionate8 delighted close/sit 7 angry stroke Couch 7 happy Displeasure 6 annoyed 6 pleased Pleasure Stephen Barrass, Linda Davy & Kerrypetting Richens 5 frustrated content Australia, 2003 5 4 miserable none serene 4 3 depressed calm 3 2 bored relaxed 2 1 tired sleepy 1 1 2 3 4 5 6 7 8 9 Low Arousal

George Kelly, Repertory Grid USA, 1955 Listening Classifying Sorting

Dendogram from George Kelly, USA, 1955 Repertory Grid Source / Object / Type of Interaction Clusters of `related’ or `similar’ sounds

Advice and tips for achieving better sound designs Selecting and classify suitable sounds Verify your mappings Rapid prototyping with tools such as PD, SuperCollider, Chuck, etc. Contexts / Users / Environment

Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions

0 comments

Notes on slide 1

Favorites, Groups & Events