Mixing Interaction, Sonification, Rendering and
Design - The art of creating sonic interactions

                 Eoin Braz...
Sonification
Using Sound to Effectively
   Convey Information


                       Ben Cohen, co-
                       founder of...
203
                          Cartoonification
evel models: bouncing, breaking, rolling, crumpling, pouring




           ...
Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8
                                                          2.5
 0.3...
Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8
                                                          2.5
 0.3...
Sonic Interaction Design

• Sound and interaction are both process oriented

• Interactive systems are complex - sound is ...
SHAPE Exhibition    Interactive Sound Object Models
                      for Exploration & Education
Re-Tracing the Past
Solids                                                                                                  Liquids           ...
Virtual
     Touch
    Machine
Interacting with one of the four
`mysterious’ objects

Using a multimodal (Audio-
Haptic-Vi...
Virtual
     Touch
    Machine
Interacting with one of the four
`mysterious’ objects

Using a multimodal (Audio-
Haptic-Vi...
SOb-ifying the Exhibition
SOb - Sounding    Interactive Sound Object Models
                  for Exploration & Education
 Object Project
Museum Installations - Lessons & Considerations

• Enhance                • Design activities /
  interpretation          ...
Soundscapes &     J. Tardieu, P. Susini, F. Poisson,
                  P. Lazareff, and S. McAdams
 Train Stations   Franc...
Soundscapes &     J. Tardieu, P. Susini, F. Poisson,
                  P. Lazareff, and S. McAdams
 Train Stations   Franc...
Connecting        L. Pellarin, N. Bottcher, J. M.
                         Olsen, O. Gregersen, S. Serafin,
    Strangers a...
Ping    Photon    Wind          Shepard
        Drive                   Tones

   Shannon Portal -
                       ...
Ping    Photon    Wind          Shepard
        Drive                   Tones

   Shannon Portal -
                       ...
Ping    Photon    Wind          Shepard
        Drive                   Tones

   Shannon Portal -
                       ...
Ping    Photon    Wind          Shepard
        Drive                   Tones

   Shannon Portal -
                       ...
Ping    Photon    Wind          Shepard
        Drive                   Tones

   Shannon Portal -
                       ...
Analysis of the Sound & Space
common fate’ and ‘good continuation’ [11].
casual users where auditory display was a significant element to
                 facilitate user interaction. We used an ...
Public Spaces - Lessons & Considerations

• Engaging (virtuality) • Design activities /
                          consulta...
Hanging Gardens -
  Auditory Installation
Collaboration with Jurgen Simpson

Linked two gallery spaces in Limerick and Bel...
Artistic Installations - Lessons & Considerations

• Interpretation vs        • Design activities /
  artistic            ...
Action                      Sound            Function
                   No touch                    N/A
                 ...
Where and How to start designing with Sound

• How to select and classify suitable sounds for a particular
  interaction d...
Selection of Approaches Depending on Question

• Evaluating Perceptual Qualities of Single Auditory Stimuli
  • Identificat...
Sound Creation
                                                                           1
                              ...
Stage 1 - Develop consensus
                                                     Select Listening Panel, Sounds    Select ...
CLOSED project, 2008,
    Functional Artefacts                                             Ircam, Verona, Berlin

 Sonic I...
CLOSED project, 2008,
  Functional Artefacts                       Ircam, Verona, Berlin

 Sonic Interaction              ...
CLOSED project, 2008,
  Functional Artefacts                       Ircam, Verona, Berlin

 Sonic Interaction              ...
CLOSED project, 2008,
  Functional Artefacts                       Ircam, Verona, Berlin

 Sonic Interaction              ...
Explorations and Methods

• Listening test           • Causal
  approaches                 uncertainty

• Textual         ...
Graham Coleman,
   Sonic Mapping                 PhD Thesis
                                 Scotland, 2008

Listening    ...
Graham Coleman,
           Sonic Mapping                        PhD Thesis
                                               ...
High Arousal

                    1           2            3         4       5        6         7            8        9
  ...
High Arousal

                    1           2            3         4       5        6          7          8         9
  ...
George Kelly,
Repertory Grid     USA, 1955



       Listening




                           Classifying




 Sorting
Dendogram from     George Kelly,
                   USA, 1955
  Repertory Grid

                                    Source...
Advice and tips for achieving better sound designs

 Selecting and classify suitable sounds

 Verify your mappings

 Rapid...
Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions
Upcoming SlideShare
Loading in …5
×

Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions

1,673 views

Published on

This talk will focus on acoustic rendering, the rendering of scenarios and contexts from the viewpoint of sonic interaction design. Sonic interaction design sits at the crossroads between interaction design, auditory display, acoustics, interactive arts, and ubiquitous computing. The area focuses on the exploration and exploitation of sound to communicate meaning, information, and aesthetical / emotional qualities in interactive contexts. The practise and study of sonic rendering within the interaction loop between users and artefacts or environments for both functional and artistic applications will be discussed. Examples of sonic interventions and of sonic interactive installations are discussed to illustrate how human centric and experiential aspects within the interaction loop. This will show how interaction design techniques and acoustic production are used to create engaging sonic experiences. Case studies covered will include work in museum installations, auditory displays designed for public spaces, and artistic interventions. The talk will show how sonification and sonic interaction design supplements the related areas of sound production and of gestural interfaces. An overview of the techniques and methods will show how this occurs in practise.

What is Sonic Interaction Design ?
Examples include aesthetic and sonic quality assessment in product and interaction design: workshops and interaction with projects (e.g., SOb, CLOSED, COST SID).
Creative exploration of context and opportunities for interactive sound artefacts and for acoustic interventions.

Where does Sonic Interaction Design inform Acoustic Rendering ?
Room acoustics and Site sound - sound within a space or interface.
Context and Scenario, how these inform the rendering (how it is acoustically rendered, what is rendered, how will the rendering be interpreted)

Examples includes installations and interventions from various locations including various museums, airports, train stations, and galleries. Related work in products sound design will also be covered.

Published in: Education, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,673
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
40
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions

  1. 1. Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions Eoin Brazil EG_IT 22 & 23 October, Verona. Italy
  2. 2. Sonification
  3. 3. Using Sound to Effectively Convey Information Ben Cohen, co- founder of Ben & Jerry’s Ice-cream Company Talking against the size and cost of the nuclear stockpile in the United States.
  4. 4. 203 Cartoonification evel models: bouncing, breaking, rolling, crumpling, pouring Maisy drinking with a straw (From “Happy Birthday, Maisy” by Lucy Cousins, Candlewick Press, 1998.)
  5. 5. Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8 2.5 0.333 1 0.333 0.692 0.176 0.138 0.354 0.058 0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 2.1 1.028 1 1.028 0.576 0.070 0.077 2.401 0.162 1.7 0.217 1 0.217 0.885 0.259 1.087 0.739 1.005 1.3 0.507 1 0.507 0.862 0.198 0.309 0.578 1.107 0.570 1 0.570 0.866 0.301 0.434 0.526 0.674 0.8 0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 0.4 0.530 1 0.530 0.042 0.852 0.250 0.053 0.185 0.497 1 0.497 1.590 1.365 1.175 1.278 0.101 0 Area-1 Area-2 Area-3 Area-4 0.167 1 0.167 1.261 0.031 0.658 0.306 1.289 Area-5 Area-6 Area-7 Area-8 An Example of Complex datasets - Adding sound can elaborate streams Auditory Graphing and temporality of the data
  6. 6. Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8 2.5 0.333 1 0.333 0.692 0.176 0.138 0.354 0.058 0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 2.1 1.028 1 1.028 0.576 0.070 0.077 2.401 0.162 1.7 0.217 1 0.217 0.885 0.259 1.087 0.739 1.005 1.3 0.507 1 0.507 0.862 0.198 0.309 0.578 1.107 0.570 1 0.570 0.866 0.301 0.434 0.526 0.674 0.8 0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 0.4 0.530 1 0.530 0.042 0.852 0.250 0.053 0.185 0.497 1 0.497 1.590 1.365 1.175 1.278 0.101 0 Area-1 Area-2 Area-3 Area-4 0.167 1 0.167 1.261 0.031 0.658 0.306 1.289 Area-5 Area-6 Area-7 Area-8 An Example of Complex datasets - Adding sound can elaborate streams Auditory Graphing and temporality of the data
  7. 7. Sonic Interaction Design • Sound and interaction are both process oriented • Interactive systems are complex - sound is able to convey complex information very quickly • Interactive systems are “disappearing”, sound can tell us about their presence and their status • We have learned to understand complex soundscapes -> ubiquitous, visualisation, and many areas of computing may profit from the use of sound
  8. 8. SHAPE Exhibition Interactive Sound Object Models for Exploration & Education Re-Tracing the Past
  9. 9. Solids Liquids Gases Fracture Impact Friction Fluid flowing Bubble Turbulence Explosion Low Level Model Model Model Model Model Model Model Models Fracturing Crumpling Hitting Rolling Rubbing Braking Flowing Burbing Dripping Whoosing Burning Exploding Popping Basic Events and Textures Pouring Filling Splashing Sucking Derived Walking Crushing Bouncing Dropping Breaking Sliding Squeaking Processes Rolling / Squeaking Rubbed Vacuum Footsteps Can Falling on gravel crushing coin breaking wheel door glass cleaner Simulation Paper Struck Glass Bowed Examples Struck bar crumpling string cleaning string SOb-ifying Impact Models for Hitting, Rolling, or Crumbling Textures / the Exhibition Basic Events
  10. 10. Virtual Touch Machine Interacting with one of the four `mysterious’ objects Using a multimodal (Audio- Haptic-Visual) interface to explore its materiality and size Real-Time Interaction
  11. 11. Virtual Touch Machine Interacting with one of the four `mysterious’ objects Using a multimodal (Audio- Haptic-Visual) interface to explore its materiality and size Real-Time Interaction
  12. 12. SOb-ifying the Exhibition
  13. 13. SOb - Sounding Interactive Sound Object Models for Exploration & Education Object Project
  14. 14. Museum Installations - Lessons & Considerations • Enhance • Design activities / interpretation consultations ✦ observational • Fun yet interactive, studies spark ✦ consult docents / • Active not passive experts environment / ✦ understand interfaces physical/spatial constraints
  15. 15. Soundscapes & J. Tardieu, P. Susini, F. Poisson, P. Lazareff, and S. McAdams Train Stations France, 2008
  16. 16. Soundscapes & J. Tardieu, P. Susini, F. Poisson, P. Lazareff, and S. McAdams Train Stations France, 2008
  17. 17. Connecting L. Pellarin, N. Bottcher, J. M. Olsen, O. Gregersen, S. Serafin, Strangers at a and M. Guglielmi Denmark, 2005 Train Station Bouncing Frequency / Airplane Mirror Balls Amplitude Synth Triggers
  18. 18. Ping Photon Wind Shepard Drive Tones Shannon Portal - Interactive Sounds for Auditory Display for Communication & Entertainment Public Spaces
  19. 19. Ping Photon Wind Shepard Drive Tones Shannon Portal - Interactive Sounds for Auditory Display for Communication & Entertainment Public Spaces
  20. 20. Ping Photon Wind Shepard Drive Tones Shannon Portal - Interactive Sounds for Auditory Display for Communication & Entertainment Public Spaces
  21. 21. Ping Photon Wind Shepard Drive Tones Shannon Portal - Interactive Sounds for Auditory Display for Communication & Entertainment Public Spaces
  22. 22. Ping Photon Wind Shepard Drive Tones Shannon Portal - Interactive Sounds for Auditory Display for Communication & Entertainment Public Spaces
  23. 23. Analysis of the Sound & Space common fate’ and ‘good continuation’ [11].
  24. 24. casual users where auditory display was a significant element to facilitate user interaction. We used an iterative design process, Brainstorms, Concept from simple onomatopoeicPublic Spacescomplex Auditory representations, to Auditory starting sound object models in Pure Data. The system was evaluated at Generation each stage, from the lab to the final public setting. The problems Linking ubiquitious auditory addressed covered the representations of left-right, up-down, displays design to evaluation in Display Design and the amount of movement by the user or groups of users. In public environments addition to this, it was important that the auditory display would attract attention when users were within control range of the system, i.e. an affordance that invited and allowed users to Process for Sonic Sketching sound using discover functionality. Mood clips (mood boards or Boards onomatopoeia) 1. INTRODUCTION In our Shared Worlds project we explored designing ubiquitous computing for public environments. Part of this Public Spaces Figure 1: Shannon Portal installation exploration was to design, build and evaluate installations in Video Bodystorming / public environments. The work reported in this paper is about the Shannon Portal, a complex Prototyping interactive installation in Roleplay the Shannon Airport in the West of Ireland. The overall design is Inteaction described in [1], while in this paper we focus on the design of the auditory display element of the installation. One of the Evaluate issues that we wanted to explore in the project was the use of Evaluate large ambient displays and public interaction with such displays. As the installation at Shannon required an interactive image gallery to be displayed, we designed a purpose-built back-projection system. An overhead camera Evaluate tracked users’ Prototype using PD / movements in front of the display, allowing the users to move a virtual Max/MSP or similar magnifying glass across the image gallery on screen, by moving their body in front of the display. See Figure 1 and 2. Over several iterations we developed hardware and software for the display and explored different mappings of users’ h = Pitc actions to movement of the virtual magnifying glass on the ack tion projection screen. For reasons outside the scope of this paper, it t-B c F ron Dire was decided that it was not the users’ location that would Evaluate Left-Right Panning = Figure 2: Back-projection, loudspeakers 2 video and Code interaction and control the movement, but the users’ movement. The rationale 1 Amplitude Panning camera for this sound as there was only one virtual magnifying glass was that object model but there could be multiple simultaneous users, this mapping 3 Movement within space = Loudness would encourage collaboration and engagement. 1.1. The challenge of used for fourth prototype public Mappings auditory display in While testing and evaluating the system in the lobby outside environments our lab in the Interaction Design Centre at the University of (Shepard Tones) Limerick, we noted that users who occasionally passed by the Is has been noted in numerous papers (e.g. [2-4]) that
  25. 25. Public Spaces - Lessons & Considerations • Engaging (virtuality) • Design activities / consultations • Visitors / Working within the space ✦ observational studies • Privacy / Intrusion ✦ interviews • Masking and ✦ understanding the soundscape issues physical /cultural / (peak vs non-peak spatial constraints times)
  26. 26. Hanging Gardens - Auditory Installation Collaboration with Jurgen Simpson Linked two gallery spaces in Limerick and Belfast Explore circadian rhythms of the spaces Motion sensors and time of day, site-specific
  27. 27. Artistic Installations - Lessons & Considerations • Interpretation vs • Design activities / artistic consultations • Engaging ✦ aesthetic issues ✦ distance / • Group / Individual repairability ✦ understand • Consider temporal physical/spatial constraints constraints
  28. 28. Action Sound Function No touch N/A Touch area outside button N/A HCI Design and Enter button area Tick Interactive Move finger on button Friction sound Sonification for Fingers and Ears Exit button area Tack Lift finger off button Tock Select / Activate function
  29. 29. Where and How to start designing with Sound • How to select and classify suitable sounds for a particular interaction design ? • TaDa - select sounds to communicate the dimensions and directions with relation to their actions and to that of the system e.g. ears-lead-eyes design pattern • Sounds linked to physical objects as dynamic entities with manipulation / interaction • Multidimensional information carrier - size, shape, material, distance, speed, and emotional expression
  30. 30. Selection of Approaches Depending on Question • Evaluating Perceptual Qualities of Single Auditory Stimuli • Identification Tasks, Context-Based Ratings, Attribute Ratings • Evaluating Relations Among Auditory Stimuli • Discrimination Trials, Similarity Ratings, Sorting Tasks, MDS, CA, PCA, Pathfinder Analysis • Evaluation of an application or an interface • Surveys, Verbal Protocols, Heuristics, Task lists, etc.
  31. 31. Sound Creation 1 Context and Auditory Display Definition Review of existing Everyday Musical Sounds Sounds 2 2 Considerations Considerations Activity, Sound Like, Activity, Mapping, Vocalisations / Onomatopoeia Instruments Find Closest Match in F/X Library / Use Foley 2 Explore Existing Earcon Hierarchies / Create New Hierarchy 2 approaches, methods, and 2 3&4 Analyse Sound and Create Earcon Family Consider Action/s and and Test - Does it Objects making it sound right? Inconsistent Attributes or Mappings techniques. 3&4 Confused Sounds Create the Sound and Test - Does it sound right? 5 6 Evaluate Scalings / Auditory Characterisation Mappings of Story / Scene / Account Inconsistent Attributes 7 8 9 8 Elicit Descriptors & TaDa Analysis Narrative Sound Sonic Map Creation for or Mappings Constructs Artefact Creation Account / Scene / Story Confused Sounds Sound Creation 10 Rating of Constructs & 11 Descriptor Categorisation Structuring Of Concepts Hearsay Analysis Personal Constructs Groups of Concepts Descriptor Categories Associated Ratings 17 Evaluation Inconsistent Attributes 12 13 or Mappings Causal Uncertainty Measures Structuring Of Constructs Results Confused Sounds Confused Sounds Sound Analysis Groups of Constructs 14 Definition Of Attributes Construction Of Scales Attribute Scales 15 Validation Of Scales Preliminary Results Mismatched Sounds 16 Category Refinement 17 Evaluation Results Sound Analysis
  32. 32. Stage 1 - Develop consensus Select Listening Panel, Sounds Select Listeners, Sounds & Group or Stage 1 - Develop individual & Attributes to explore Attributes to explore Individual descriptive language descriptive language based Play audio stimuli (~100-200) Play audio stimuli (~100-200) approaches Determine attribute scales, Determine attribute scales, descriptors, key audio groups descriptors, key audio groups ranking / rating of data Stage 2 - Participant Stage 2 - Participant performance data Gather global Gather end Participant ranks, rates, or scores from user groups data naive preference participants data to preferences Stage 3 - Mapping Stage 3 - Mapping of data Map preference / performance Mapping of data using data using statistical methods statistical methods
  33. 33. CLOSED project, 2008, Functional Artefacts Ircam, Verona, Berlin Sonic Interaction Functional Artefacts Design Process for Linking design to evaluation Functional Artefacts including basic design methods Analyse, Shape, Describe Interaction Materials Combine Experience Interaction Interaction in Context Gestalt Artefact Evaluate Rolling Clapping 1 Examples: Clapping 2
  34. 34. CLOSED project, 2008, Functional Artefacts Ircam, Verona, Berlin Sonic Interaction Functional Artefacts Design Process for Linking design to evaluation Functional Artefacts including basic design methods Analyse, Shape, Describe Interaction Materials Combine Experience Interaction Interaction in Context Gestalt Artefact Rolling Evaluate Clapping 1 Examples: Clapping 2
  35. 35. CLOSED project, 2008, Functional Artefacts Ircam, Verona, Berlin Sonic Interaction Functional Artefacts Design Process for Linking design to evaluation Functional Artefacts including basic design methods Analyse, Shape, Describe Interaction Materials Combine Experience Interaction Interaction in Context Gestalt Artefact Rolling Evaluate Clapping 1 Examples: Clapping 2
  36. 36. CLOSED project, 2008, Functional Artefacts Ircam, Verona, Berlin Sonic Interaction Functional Artefacts Design Process for Linking design to evaluation Functional Artefacts including basic design methods Analyse, Shape, Describe Interaction Materials Combine Experience Interaction Interaction in Context Gestalt Artefact Rolling Evaluate Clapping 1 Examples: Clapping 2
  37. 37. Explorations and Methods • Listening test • Causal approaches uncertainty • Textual • 2-D perceptual descriptor scaling analysis • Repertory grid • Sonic Mapping technique
  38. 38. Graham Coleman, Sonic Mapping PhD Thesis Scotland, 2008 Listening Classifying Background Sounds Foreground Sounds * Ac als als Ac tio gn gn tio + ns ? Si Si ns 0 0 * Emotions Contextual Emotions Sounds Written descriptions of Sorting Visible Hidden sounds and their type Ac als ti gn on Si * Everyday s + Speech 0 Music Emotions ? Abstract / Unknown
  39. 39. Graham Coleman, Sonic Mapping PhD Thesis Scotland, 2008 Background Sounds Foreground Sounds Listening Classifying * Ac als als Ac tio gn gn tio + ns ? Si Si ns 0 0 * Emotions Contextual Emotions Sounds Written descriptions of Visible Hidden sounds and their type Ac als tio gn ns * Everyday Si + Speech Sorting 0 Music Emotions ? Abstract / Unknown
  40. 40. High Arousal 1 2 3 4 5 6 7 8 9 9 9 Zizi - The alarmed excited astonished 8 afraid close/sit delighted 8 7 angry stroke happy 7 Affectionate Couch Displeasure 6 annoyed pleased 6 Pleasure petting 5 frustrated content 5 4 miserable none serene 4 Stephen Barrass, Linda Davy 3 depressed calm 3 & Kerry Richens 2 bored relaxed 2 Australia, 2003 1 tired sleepy 1 1 2 3 4 5 6 7 8 9 Low Arousal
  41. 41. High Arousal 1 2 3 4 5 6 7 8 9 9 alarmed excitedZizi - The 9 astonished 8 afraid Affectionate8 delighted close/sit 7 angry stroke Couch 7 happy Displeasure 6 annoyed 6 pleased Pleasure Stephen Barrass, Linda Davy & Kerrypetting Richens 5 frustrated content Australia, 2003 5 4 miserable none serene 4 3 depressed calm 3 2 bored relaxed 2 1 tired sleepy 1 1 2 3 4 5 6 7 8 9 Low Arousal
  42. 42. George Kelly, Repertory Grid USA, 1955 Listening Classifying Sorting
  43. 43. Dendogram from George Kelly, USA, 1955 Repertory Grid Source / Object / Type of Interaction Clusters of `related’ or `similar’ sounds
  44. 44. Advice and tips for achieving better sound designs Selecting and classify suitable sounds Verify your mappings Rapid prototyping with tools such as PD, SuperCollider, Chuck, etc. Contexts / Users / Environment

×