Successfully reported this slideshow.
Engaging Spect-actorswith Multimodal Digital Puppetry                       Céline Coutrix (presenter), Giulio Jacucci,   ...
Introduction•  (Digital) Puppetry                         1
Introduction•  Field study  -    Describe an example of how a computer       can be used as a medium for engagement  -    ...
Outline1. The system studied2. Data collection and analysis3. Findings  a.  Multimodality  b.  Performative structures  c....
Outline1. The system studied2. Data collection and analysis3. Findings  a.  Multimodality  b.  Performative structures  c....
The system: Euclideaudience, in front of puppet	   pupeteer	                                                5
The system: Euclide       Euclide	       system	                      6
The system: Euclide•  Has been successfully used for engagement   in the Science museum in Naples, Italy•  Why? How could ...
The system: Euclide•  Puppeteer•  Audience                          8
The system: Puppeteer     puppet      on distant screen	                                        9         pupeteer
The system: Puppeteer•  Puppeteer  -     Voice  -     Face  -     Body       ·    Global movement       ·    Local movemen...
The system: Puppeteer•  Voice                          11
The system: Puppeteer•  Face                          12
The system: Puppeteer•  Face                          13
The system: Puppeteer•  Face                          14
The system: Puppeteer•  Face                          15
The system: Puppeteer•  Face                          16
The system: Puppeteer•  Face                          17
The system: Puppeteer•  Face                          18
The system: Puppeteer•  Face                          19
The system: Puppeteer•  Body: Global movement                           20
The system: Puppeteer•  Body: Global movement                           21
The system: Puppeteer•  Body: Global movement                           22
The system: Puppeteer•  Body: Local movement                          23
The system: Puppeteer•  Body: Local movement                          24
The system: Puppeteer•  Body: Local movement                          25
The system: Puppeteer•  Body                          26
The system: Puppeteer•  Scenery             transparency	                               27
The system: Puppeteer•  Scenery orientation	                          28
The system: Puppeteer•  Scenery                 Zoom+Position	                                    29
The system: Euclide•  Puppeteer•  Audience                          30
The system: Euclide             distant	            pupeteer	                                    31  audience, in front of...
The system: Euclide•  Compromise between  -     Museum’s requirements for cheap, robust,        generic and flexible hardw...
Outline1. The system studied2. Data collection and analysis3. Findings  a.  Multimodality  b.  Performative structures  c....
Data Collection•  Video+audio recordings  -    Over several months  -    Four viewpoints  -    Three different pupeteers• ...
Data Analysis•  Constant comparison analysis•  Top-down and Bottom-up approach  -    Findings originates from both the dat...
Outline1. The system studied2. Data collection and analysis3. Findings  a.  Multimodality  b.  Performative structures  c....
Findings: Multimodal Puppetry •  Most used features are controlled    through more accessible devices, due    to design   ...
Findings: Multimodal Puppetry           face	      body	      scenery	    •    Mostly animation of face,    •    second co...
Findings: Multimodal Puppetry  Simultaneous	use of modalities	                       Single modality	   •  Important multi...
Findings: Multimodal Puppetry 1. Speech        synchronization’s problem 2. Speech+lips 3. Lips                           ...
Findings: Multimodal Puppetry Speech durations	       Silence durations	 • Lot of very short speeches and silences   in or...
Outline1. The system studied2. Data collection and analysis3. Findings  a.  Multimodality  b.  Performative structures  c....
Findings: Emergence of  Performative Structure•  Engaging the Audience Throughout the   Sessions: Evidences  -     People ...
Findings: Emergence of       Performative Structure                 Number and length of sessions (mins)10 8 6 4 2 0      ...
Findings: Emergence of  Performative Structure•  Structure of sessions  -    Approach: enter the interaction area;        ...
Findings: Emergence of  Performative Structure•  Structure of sessions  -    Approach, Testing, Playing, Ending  -    Long...
Findings: Emergence ofPerformative Structure                         47
Findings: Emergence of      Performative Structure•    the audience starts interacting happily but calm,•    then gets exc...
Outline1. The system studied2. Data collection and analysis3. Findings  a.  Multimodality  b.  Performative structures  c....
Findings: Construction of       Engagement•  The Puppet’s Multimodal Resources:   Effects (face, body and scenery   animat...
Findings: Construction of       Engagement•  Pure special effects  -    Non verbal ressources are predominant       modali...
Findings: Construction of       Engagement•  Non-verbal turns: Contribution in an   interaction that includes speech, e.g....
Findings: Construction of       Engagement•  Verbal accompaniements: Non-verbal   actions are accompagnied by speech, e.g....
Findings: Construction of       Engagement•  Verbal accompaniements: ExamplePuppet:     Ho una molla             I’ve got ...
Findings: Construction of        Engagement•  Virtual gesturing: priority is given to   speech; visual features are used i...
Findings: Construction of       Engagement•  Virtual gesturing: Example  Visitor:     Dove sei?             Where are you?...
Findings: Construction of       Engagement•  Multimodal interaction for puppeteer   is essential  -    Helps avoiding the ...
Findings: Construction of           Engagement   •  Response to verbal abuse: ExampleChild:  (Scemo)         DumbPuppet: V...
Findings: Construction of       Engagement•  Talk = Main ressource,   Multimodal animation = instrument•  Combination attr...
Outline1. The system studied2. Data collection and analysis3. Findings  a.  Multimodality  b.  Performative structures  c....
Conclusion•  Why is digital puppetry a promising application   field for HCI?  -     On the contrary to a fully computer-d...
Conclusion•    Why is digital puppetry a promising application field     for HCI?     -     Performative Structures for Br...
Engaging Spect-actorswith Multimodal Digital Puppetry
Upcoming SlideShare
Loading in …5
×

Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]

650 views

Published on

  • Be the first to comment

  • Be the first to like this

Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]

  1. 1. Engaging Spect-actorswith Multimodal Digital Puppetry Céline Coutrix (presenter), Giulio Jacucci, Anna Spagnolli, Lingyi Ma, Matti Helin, Gabriela Richard, Lorenza Parisi, Stefano Roveda, Prayag Narula
  2. 2. Introduction•  (Digital) Puppetry 1
  3. 3. Introduction•  Field study -  Describe an example of how a computer can be used as a medium for engagement -  Understand why would this be a promising field for HCI applications 2
  4. 4. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 3
  5. 5. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 4
  6. 6. The system: Euclideaudience, in front of puppet pupeteer 5
  7. 7. The system: Euclide Euclide system 6
  8. 8. The system: Euclide•  Has been successfully used for engagement in the Science museum in Naples, Italy•  Why? How could it be improved? What can we learn for future puppetry systems? 7
  9. 9. The system: Euclide•  Puppeteer•  Audience 8
  10. 10. The system: Puppeteer puppet on distant screen 9 pupeteer
  11. 11. The system: Puppeteer•  Puppeteer -  Voice -  Face -  Body ·  Global movement ·  Local movement -  Scenery 10
  12. 12. The system: Puppeteer•  Voice 11
  13. 13. The system: Puppeteer•  Face 12
  14. 14. The system: Puppeteer•  Face 13
  15. 15. The system: Puppeteer•  Face 14
  16. 16. The system: Puppeteer•  Face 15
  17. 17. The system: Puppeteer•  Face 16
  18. 18. The system: Puppeteer•  Face 17
  19. 19. The system: Puppeteer•  Face 18
  20. 20. The system: Puppeteer•  Face 19
  21. 21. The system: Puppeteer•  Body: Global movement 20
  22. 22. The system: Puppeteer•  Body: Global movement 21
  23. 23. The system: Puppeteer•  Body: Global movement 22
  24. 24. The system: Puppeteer•  Body: Local movement 23
  25. 25. The system: Puppeteer•  Body: Local movement 24
  26. 26. The system: Puppeteer•  Body: Local movement 25
  27. 27. The system: Puppeteer•  Body 26
  28. 28. The system: Puppeteer•  Scenery transparency 27
  29. 29. The system: Puppeteer•  Scenery orientation 28
  30. 30. The system: Puppeteer•  Scenery Zoom+Position 29
  31. 31. The system: Euclide•  Puppeteer•  Audience 30
  32. 32. The system: Euclide distant pupeteer 31 audience, in front of puppet
  33. 33. The system: Euclide•  Compromise between -  Museum’s requirements for cheap, robust, generic and flexible hardware -  Puppeteers’ requirements for large number of functionalities -  Usability ·  e.g. mouth control moved from index to thumb to avoid tiredness 32
  34. 34. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 33
  35. 35. Data Collection•  Video+audio recordings -  Over several months -  Four viewpoints -  Three different pupeteers•  Interview of head pupeteer 34
  36. 36. Data Analysis•  Constant comparison analysis•  Top-down and Bottom-up approach -  Findings originates from both the data and our research questions•  Coding scheme built through three steps: open, axial, selective coding•  Clips selected together, then divided between coders•  Discussion and revision of coding until agreements > 0.6 35
  37. 37. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 36
  38. 38. Findings: Multimodal Puppetry •  Most used features are controlled through more accessible devices, due to design 37
  39. 39. Findings: Multimodal Puppetry face body scenery •  Mostly animation of face, •  second comes body, •  last comes scenery 38
  40. 40. Findings: Multimodal Puppetry Simultaneous use of modalities Single modality •  Important multimodal use 39
  41. 41. Findings: Multimodal Puppetry 1. Speech synchronization’s problem 2. Speech+lips 3. Lips 40
  42. 42. Findings: Multimodal Puppetry Speech durations Silence durations • Lot of very short speeches and silences in order to maintain engagement •  Puppet active even when silent 41
  43. 43. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 42
  44. 44. Findings: Emergence of Performative Structure•  Engaging the Audience Throughout the Sessions: Evidences -  People in the audience: ·  Don’t talk to each other 99% of the time ·  Don’t pay attention to the outside 98% of the time ·  Talk to the puppet 65% of the time ·  Pleasure and Arousal annotated as positive all the time ·  Number increase, then decrease 43
  45. 45. Findings: Emergence of Performative Structure Number and length of sessions (mins)10 8 6 4 2 0 < 1 min 1 min < < 2 min 2 min < < 9 min > 9 min •  Rather short, but: -  Sessions interrupted by teachers -  Group of children continue interacting 44 from the next puppet station
  46. 46. Findings: Emergence of Performative Structure•  Structure of sessions -  Approach: enter the interaction area; observe -  Testing: Try to interact to find out which actions have an effect -  Playing: Interact in an aware, active, involved way; climax -  Ending: attention diverted; leave 45
  47. 47. Findings: Emergence of Performative Structure•  Structure of sessions -  Approach, Testing, Playing, Ending -  Long playing phases, with scenery animations 46
  48. 48. Findings: Emergence ofPerformative Structure 47
  49. 49. Findings: Emergence of Performative Structure•  the audience starts interacting happily but calm,•  then gets excited,•  finishes the interaction happily and calm again 48
  50. 50. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 49
  51. 51. Findings: Construction of Engagement•  The Puppet’s Multimodal Resources: Effects (face, body and scenery animations), like natural gestures, in relation to speech -  Pure special effects -  Non-verbal turns -  Verbal accompaniements -  Virtual gesturing 50
  52. 52. Findings: Construction of Engagement•  Pure special effects -  Non verbal ressources are predominant modality for interacting with the audience -  Used to attract visitors that cannot be reached by speech (babies, far away visitors) 51
  53. 53. Findings: Construction of Engagement•  Non-verbal turns: Contribution in an interaction that includes speech, e.g. -  wearing accessory like glasses on special request from the audience and subsequently commenting on it -  Explaining a fact by displaying a picture of it 52
  54. 54. Findings: Construction of Engagement•  Verbal accompaniements: Non-verbal actions are accompagnied by speech, e.g. -  Acrobatic movement with onomatopeic sound → Communication relies on the nonverbal resource, which dictates its structure and meaning 53
  55. 55. Findings: Construction of Engagement•  Verbal accompaniements: ExamplePuppet: Ho una molla I’ve got a spring Mi serve per saltare I need it to jump ⎡ Dong:: dong:: ⎤ Dong dong ⎣((Starts jumping))⎦ 54
  56. 56. Findings: Construction of Engagement•  Virtual gesturing: priority is given to speech; visual features are used in synch with speech, following its structure and duration, e.g. -  Head movement, like pointing gesture 55
  57. 57. Findings: Construction of Engagement•  Virtual gesturing: Example Visitor: Dove sei? Where are you? Puppet: Sono⎡::((pause))⎤ I’m ⎣((Looking down))⎦ ⎡qui dentro,⎤ inside here ⎣((Looking around))⎦ non mi vedi? can’t you see me 56
  58. 58. Findings: Construction of Engagement•  Multimodal interaction for puppeteer is essential -  Helps avoiding the low status of a conversational agent, e.g. verbal abuse 57
  59. 59. Findings: Construction of Engagement •  Response to verbal abuse: ExampleChild: (Scemo) DumbPuppet: Vabbeh adesso basta. All right; let’s stop thisChild: ((turns back, surprised))Puppet: Ogni volta che scemo, ogni volta che scemo. All the times dumb, all the times dumb. E tu invece come sei? And what about you instead, what are you?Child: ((goes away)) 58
  60. 60. Findings: Construction of Engagement•  Talk = Main ressource, Multimodal animation = instrument•  Combination attracts visitors effectively 59
  61. 61. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 60
  62. 62. Conclusion•  Why is digital puppetry a promising application field for HCI? -  On the contrary to a fully computer-driven system ·  Engagement stems from combination of multimodal ressources by puppeteer ·  Users from the audience inspire the narrative by the puppeteer and feel like they have an active role 61
  63. 63. Conclusion•  Why is digital puppetry a promising application field for HCI? -  Performative Structures for Brief Interactions repetitive element of an entertainment interface could be further taken into account -  Multimodal Use of the System by the Puppeteer possibility to introduce advanced interface techniques ·  Easing lip/speech sync, ·  Mapping expressive hand gestures to puppet expressions, ·  Etc. 62
  64. 64. Engaging Spect-actorswith Multimodal Digital Puppetry

×