1.
1
å
You as a Puppet
Evaluation of Telepresence User Interface for Puppetry
Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai

2.
Introduction
TV-shows Attractions
3
“Jim Henson's Legacy: A "Rainbow Connection" with UMD”
Merrill College of Journalism Press Releases. (CC BY-NC 2.0)
Communication Education
Library teaches children the fun of reading | Stripes Guam
http://guam.stripes.com/base-info/library-teaches-children-fun-reading

3.
Introduction — Previous Puppetry Ways
4
Glove-puppet
“German children performing a tale with selfmade puppets” Jürgen Heimann.
(CC BY 3.0)
“Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006.
(CC BY 2.0)
String-puppet

4.
5
Under the Desk…
¸¸¸¸¸¸¸¸¸¸¸

5.
6
Image by www.traveljunction.com
Character Costume

6.
7
http://i843.photobucket.com/albums/zz352/loaloauk/Disney%20Planets/Mickey1.jpg
Character Costume

7.
8
Performer’s location
Physical limitations…
Puppet’s size and shape
“German children performing a tale with selfmade puppets” Jürgen Heimann.
(CC BY 3.0)
“Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006.
(CC BY 2.0)
Image by www.traveljunction.com

8.
Related Work — Input System for Puppet
9
[Slyper et al. 2015] [Oshita et al. 2013][Slyper et al. 2011]
Webcam Apple iSight
Nicolas Esposito from France (CC BY 2.0)
Web Camera Tongue Joystick Hand Gesture

9.
Related Work — Computational Puppet
10
[Sekiguchi et al. 2001] [Bonanni et al. 2005] [Lee et al. 2008]
PlayPals HuggableRobotPHONE

10.
Introduction — Concept of Our System
11
Remote Place
Objective Place
Feedback Transmission

11.
Implementation
13
Audiovisual Feedback
Sensing & Transmission
Performer to Puppet
Puppet to Performer

12.
Implementation — Sensing & Transmission
14
Head Pose

13.
Implementation — Sensing & Transmission
15
Arms’ Movements

14.
Implementation — Sensing & Transmission
16
Photoreflectors

15.
Implementation — Sensing & Transmission
17
Mouth State

16.
Implementation — Audiovisual Feedback
18
“Bones” of the Puppet
Camera
Microphone

17.
Implementation — “Bones” of the Puppet
19

18.
20
How does
the telepresence system
affect the performer and
the audience?

19.
User Study
21
Performer Side Audience Side
Enjoyable?
Cute?
Lifelike?
Enjoyable?
Fatigue?
Useful?

20.
User Study — Performer Side
22
Purpose
To investigate how the participants feel when
using the telepresence system and compare
the results between the experts and beginners.
Participants
N = 14 (age 18-24)
5 professional puppeteers
9 beginners

21.
User Study — Performer Side
23
1. We prepare a simple stage with two puppets for puppetry.
2. Each participant manipulates a puppet freely for
approximately five minutes using our system, and only
beginners take time to manipulate a puppet by hand.
3. We ask them to rate each question on a seven-point scale,
and write free descriptions.
Procedure

22.
User Study — Puppet’s POV / Performer Side
24
Q. How useful was it to operate the puppet
from the puppet’s point of view?
Professional
Beginner
M SD
5.60
5.44
0.80
0.68
Not at all Very useful

23.
User Study — Puppet’s POV / Performer Side
25
“Usually (by hand), I manipulate a puppet by imagining
the puppet’s point of view, but it’s difficult to do that, and
the puppet’s point of view sometimes goes in a different
direction, contrary to my intention. So I feel it was useful
that I could see things from the puppet’s point of view.”
“I felt like I could became a character. It was similar to
the feeling when I wear a costume.”
— Professional
— Beginner

24.
Q. How useful was it to
operate by moving your own body?
Professional
Beginner
M SD
4.60
6.44
1.20
0.68
User Study — Moving body / Performer Side
26
Significant difference
Not at all Very useful

25.
User Study — Moving body / Performer Side
27
“I could experience it as if I had become the puppet
itself… I think it has more potential if the performers
practice, since it is easier to use than by hand.”
“When I was asked to manipulate a puppet in a
conventional manner, I attempted to move its hands
using my thumb and little finger. However, I did not have
a technique for moving its hands up and down.”
— Professional
— Beginner

26.
Q. Overall, how useful was this system?
Professional
Beginner
M SD
4.80
6.33
0.98
0.47
Significant difference
User Study — Overall Usefullness/Performer Side
28
Not at all Very useful

27.
“With the system, I could express my feelings in the
puppet. When I manipulate a puppet directly by hand, it
feels like I ‘manipulate’ a puppet, but with the system,
I felt like I became a puppet.”
— Beginner
User Study — Overall Usefullness/Performer Side
29

28.
Q. Overall, how enjoyable was it to operate
a puppet with this system?
Professional
Beginner
M SD
6.40
6.44
0.49
0.50
User Study — Overall Enjoyment / Performer Side
30
Not at all Very enjoyable

29.
“I enjoyed that the mouth of the puppet moved in
accordance with my voice.”
“I enjoyed seeing the audience, because people come
closer to the robot puppet, and they don’t normally do
that. This made me laugh when our puppets
communicated with each other.”
— Professional
— Beginner
User Study — Overall Enjoyment / Performer Side
31

30.
User Study — Audience Side
32
Purpose
To investigate how the participants feel about
the puppetry played by the telepresence
system and compare the results between the
hand-based and telepresence-based puppetry.
Participants
N = 22 (age 18-24)

31.
1. We record videos of the hand-based
and telepresence-based puppetry
acted by professional puppeteers.
2. Participants watch the two videos.
3. We ask them to rate each question
about these puppet shows on a
seven-point scale, and write free
descriptions.
Procedure
User Study — Audience Side
33

32.
34
User Study — Audience Side
Q. How enjoyable
did you feel the puppetry was?
Q. How cute did you feel
the manipulated puppets were?
Q. How lifelike did you feel
the manipulated puppets were?
Didn’t feel at all Felt very much
Ours
Hand
Ours
Hand
Ours
Hand
4.27
4.55
4.59
5.32
4.00
4.77
1.62
1.67
1.70
1.39
1.60
1.51
M SD

33.
Limitations
35

34.
Limitations — PositionofaPuppetonStageCannotbeChanged
36
“This robot cannot move its position, because it
doesn’t have legs. I will get troubled when I use
some props for detailed performance.”
— Professional
Possible Solution
Gravity detection system like a NintendoWii Fit
Other orientations detected by Kinect

35.
Limitations — PerformerCannotConfirmHowaPuppetMoves
37
“…I couldn’t understand how much I could
move a puppet when I moved my body. …”
— Professional
Possible Solution
Display of the puppet’s state in the HMD
Display of images from a fixed-point camera

36.
Limitations — MovementsLookSomewhatMechanicalorArtificial
38
“I felt that the conventional puppets were
more lifelike than the system-controlled ones…”
“The hand-manipulated puppets’ movements
were subtle and seemed more lifelike.”
Possible Solution
Extra servo motors / Background musics

37.
Limitations — Stress & Fatigue on a Performer
39
“When I moved my body, the video that I saw
was shaken a lot and I felt sick.”
— Beginner
Possible Solution
Omnidirectional cameras

38.
Limitations — Cost & Space
40
Possible Solution
Pose estimation from RGB images / Google Cardboard

39.
Future Work
41
Interactive Attractions Telecommunication Computational Design

40.
Introduction — Summary of Contribution
42
1. We implemented an immersive telepresence
system for puppetry.
2. We reported the effectiveness of the system
for both the performer and audience side.
3. We discussed the limitations of the system and
future work beyond the puppetry.

41.
Research Group
43
Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai
Supported by

42.
44
å
You as a Puppet
Evaluation of Telepresence User Interface for Puppetry
Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai
Demo Reception
Please visit our booth in
Today, 6pm - 9pm
@Villeray, De Tourny