Your SlideShare is downloading. ×
Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]

340
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
340
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Engaging Spect-actorswith Multimodal Digital Puppetry Céline Coutrix (presenter), Giulio Jacucci, Anna Spagnolli, Lingyi Ma, Matti Helin, Gabriela Richard, Lorenza Parisi, Stefano Roveda, Prayag Narula
  • 2. Introduction•  (Digital) Puppetry 1
  • 3. Introduction•  Field study -  Describe an example of how a computer can be used as a medium for engagement -  Understand why would this be a promising field for HCI applications 2
  • 4. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 3
  • 5. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 4
  • 6. The system: Euclideaudience, in front of puppet pupeteer 5
  • 7. The system: Euclide Euclide system 6
  • 8. The system: Euclide•  Has been successfully used for engagement in the Science museum in Naples, Italy•  Why? How could it be improved? What can we learn for future puppetry systems? 7
  • 9. The system: Euclide•  Puppeteer•  Audience 8
  • 10. The system: Puppeteer puppet on distant screen 9 pupeteer
  • 11. The system: Puppeteer•  Puppeteer -  Voice -  Face -  Body ·  Global movement ·  Local movement -  Scenery 10
  • 12. The system: Puppeteer•  Voice 11
  • 13. The system: Puppeteer•  Face 12
  • 14. The system: Puppeteer•  Face 13
  • 15. The system: Puppeteer•  Face 14
  • 16. The system: Puppeteer•  Face 15
  • 17. The system: Puppeteer•  Face 16
  • 18. The system: Puppeteer•  Face 17
  • 19. The system: Puppeteer•  Face 18
  • 20. The system: Puppeteer•  Face 19
  • 21. The system: Puppeteer•  Body: Global movement 20
  • 22. The system: Puppeteer•  Body: Global movement 21
  • 23. The system: Puppeteer•  Body: Global movement 22
  • 24. The system: Puppeteer•  Body: Local movement 23
  • 25. The system: Puppeteer•  Body: Local movement 24
  • 26. The system: Puppeteer•  Body: Local movement 25
  • 27. The system: Puppeteer•  Body 26
  • 28. The system: Puppeteer•  Scenery transparency 27
  • 29. The system: Puppeteer•  Scenery orientation 28
  • 30. The system: Puppeteer•  Scenery Zoom+Position 29
  • 31. The system: Euclide•  Puppeteer•  Audience 30
  • 32. The system: Euclide distant pupeteer 31 audience, in front of puppet
  • 33. The system: Euclide•  Compromise between -  Museum’s requirements for cheap, robust, generic and flexible hardware -  Puppeteers’ requirements for large number of functionalities -  Usability ·  e.g. mouth control moved from index to thumb to avoid tiredness 32
  • 34. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 33
  • 35. Data Collection•  Video+audio recordings -  Over several months -  Four viewpoints -  Three different pupeteers•  Interview of head pupeteer 34
  • 36. Data Analysis•  Constant comparison analysis•  Top-down and Bottom-up approach -  Findings originates from both the data and our research questions•  Coding scheme built through three steps: open, axial, selective coding•  Clips selected together, then divided between coders•  Discussion and revision of coding until agreements > 0.6 35
  • 37. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 36
  • 38. Findings: Multimodal Puppetry •  Most used features are controlled through more accessible devices, due to design 37
  • 39. Findings: Multimodal Puppetry face body scenery •  Mostly animation of face, •  second comes body, •  last comes scenery 38
  • 40. Findings: Multimodal Puppetry Simultaneous use of modalities Single modality •  Important multimodal use 39
  • 41. Findings: Multimodal Puppetry 1. Speech synchronization’s problem 2. Speech+lips 3. Lips 40
  • 42. Findings: Multimodal Puppetry Speech durations Silence durations • Lot of very short speeches and silences in order to maintain engagement •  Puppet active even when silent 41
  • 43. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 42
  • 44. Findings: Emergence of Performative Structure•  Engaging the Audience Throughout the Sessions: Evidences -  People in the audience: ·  Don’t talk to each other 99% of the time ·  Don’t pay attention to the outside 98% of the time ·  Talk to the puppet 65% of the time ·  Pleasure and Arousal annotated as positive all the time ·  Number increase, then decrease 43
  • 45. Findings: Emergence of Performative Structure Number and length of sessions (mins)10 8 6 4 2 0 < 1 min 1 min < < 2 min 2 min < < 9 min > 9 min •  Rather short, but: -  Sessions interrupted by teachers -  Group of children continue interacting 44 from the next puppet station
  • 46. Findings: Emergence of Performative Structure•  Structure of sessions -  Approach: enter the interaction area; observe -  Testing: Try to interact to find out which actions have an effect -  Playing: Interact in an aware, active, involved way; climax -  Ending: attention diverted; leave 45
  • 47. Findings: Emergence of Performative Structure•  Structure of sessions -  Approach, Testing, Playing, Ending -  Long playing phases, with scenery animations 46
  • 48. Findings: Emergence ofPerformative Structure 47
  • 49. Findings: Emergence of Performative Structure•  the audience starts interacting happily but calm,•  then gets excited,•  finishes the interaction happily and calm again 48
  • 50. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 49
  • 51. Findings: Construction of Engagement•  The Puppet’s Multimodal Resources: Effects (face, body and scenery animations), like natural gestures, in relation to speech -  Pure special effects -  Non-verbal turns -  Verbal accompaniements -  Virtual gesturing 50
  • 52. Findings: Construction of Engagement•  Pure special effects -  Non verbal ressources are predominant modality for interacting with the audience -  Used to attract visitors that cannot be reached by speech (babies, far away visitors) 51
  • 53. Findings: Construction of Engagement•  Non-verbal turns: Contribution in an interaction that includes speech, e.g. -  wearing accessory like glasses on special request from the audience and subsequently commenting on it -  Explaining a fact by displaying a picture of it 52
  • 54. Findings: Construction of Engagement•  Verbal accompaniements: Non-verbal actions are accompagnied by speech, e.g. -  Acrobatic movement with onomatopeic sound → Communication relies on the nonverbal resource, which dictates its structure and meaning 53
  • 55. Findings: Construction of Engagement•  Verbal accompaniements: ExamplePuppet: Ho una molla I’ve got a spring Mi serve per saltare I need it to jump ⎡ Dong:: dong:: ⎤ Dong dong ⎣((Starts jumping))⎦ 54
  • 56. Findings: Construction of Engagement•  Virtual gesturing: priority is given to speech; visual features are used in synch with speech, following its structure and duration, e.g. -  Head movement, like pointing gesture 55
  • 57. Findings: Construction of Engagement•  Virtual gesturing: Example Visitor: Dove sei? Where are you? Puppet: Sono⎡::((pause))⎤ I’m ⎣((Looking down))⎦ ⎡qui dentro,⎤ inside here ⎣((Looking around))⎦ non mi vedi? can’t you see me 56
  • 58. Findings: Construction of Engagement•  Multimodal interaction for puppeteer is essential -  Helps avoiding the low status of a conversational agent, e.g. verbal abuse 57
  • 59. Findings: Construction of Engagement •  Response to verbal abuse: ExampleChild: (Scemo) DumbPuppet: Vabbeh adesso basta. All right; let’s stop thisChild: ((turns back, surprised))Puppet: Ogni volta che scemo, ogni volta che scemo. All the times dumb, all the times dumb. E tu invece come sei? And what about you instead, what are you?Child: ((goes away)) 58
  • 60. Findings: Construction of Engagement•  Talk = Main ressource, Multimodal animation = instrument•  Combination attracts visitors effectively 59
  • 61. Outline1. The system studied2. Data collection and analysis3. Findings a.  Multimodality b.  Performative structures c.  Construction of engagement4. Conclusion 60
  • 62. Conclusion•  Why is digital puppetry a promising application field for HCI? -  On the contrary to a fully computer-driven system ·  Engagement stems from combination of multimodal ressources by puppeteer ·  Users from the audience inspire the narrative by the puppeteer and feel like they have an active role 61
  • 63. Conclusion•  Why is digital puppetry a promising application field for HCI? -  Performative Structures for Brief Interactions repetitive element of an entertainment interface could be further taken into account -  Multimodal Use of the System by the Puppeteer possibility to introduce advanced interface techniques ·  Easing lip/speech sync, ·  Mapping expressive hand gestures to puppet expressions, ·  Etc. 62
  • 64. Engaging Spect-actorswith Multimodal Digital Puppetry

×