SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 30 day free trial to unlock unlimited reading.
The possibility of personality extraction using skeletal information in hip-hop dance by human or machine
INTERACT2019で発表した際のプレゼン用スライドです。
This slide is for the presentation when presented at INTERACT2019.
「The possibility of personality extraction using skeletal information in hip-hop dance by human or machine」
INTERACT2019で発表した際のプレゼン用スライドです。
This slide is for the presentation when presented at INTERACT2019.
「The possibility of personality extraction using skeletal information in hip-hop dance by human or machine」
The possibility of personality extraction using skeletal information in hip-hop dance by human or machine
1.
The Possibility of
Personality Extraction
Using Skeletal Information
in Hip-Hop Dance
by Human or Machine
Saeka Furuichi (Meiji University)
Kazuki Abe, Satoshi Nakamura (Meiji University)
2.
Dance is very popular all over the world
Dance
There has also been a huge demand
for learning how to dance as well
Breakdance
Added as a new competition
at 2018 Youth Olympics
3.
How to improve dancing
n To learn from skilled instructors
Learners can practice choreographies
that matches their level
n To learn from dance videos on the Web
…Popular with beginners and students
because it can be easily challenged
• Learner need to find appropriate
choreographies themselves
• Difficult to objectively identify
whether the choreography
suits them or not
4.
System flow
Target dancer who learning from dance videos
Develop a system that can search for
dance videos match their own personality
User
System
3. Analyzes personality
1. Dance in front of the camera
2. Upload dance videos to
the system
4. System recommends videos
with similar personalities
For you!
5.
Personality
One of the key elements of dance
Clear expression
becomes
more attractive
Unclear expression
tends to be
superficial and boring
In the previous breakdance example,
personality is included as one of the judging criteria
(Trivium Value System)
It is important for dancers
to develop their own personality
6.
Related work
Many dance studies using body motion data
• Developed a method to make dance motions natural
on VR by HMMs. [Mousas ‘18]
• Classified dance motions related to emotions using LMA.
[Aristidou ‘15]
• Proposed a system for emotional behavior recognition.
[Senecal ‘16]
Few studies focused on dance personality
The specific elements when the personality
appears are not clear
7.
Purpose
The purpose of research
To search for characteristic which can be
personality from skeletal information
in hip-hop dance
1. Can human distinguish personality?
2. Can machine distinguish personality?
Easy to obtain dance skeletons by
using Kinect, OpenPose and so on
8.
Dataset Construction
Skeletal information
• Extracted from dance movement by Kinect
• Composed of 15-points at 3D coordinates
Participants
• 22 university students (7 males, 15 females)
• Let them dance the particular
choreography five times
9.
Dataset Construction
Participants
• Dance experience
…Ranging from five months to six years (avg. 2.4 years)
• Practice
…Had 1-hour practice sessions twice a week for three weeks
Particular choreography
…Including many movements using the whole body
such as raising the legs, squatting, turning, and hitting the chest
Dataset: 22 participants × 5 times = 110 items
(15 sec. × 110 items = 1650 sec.)
10.
Dataset Construction
Choreography used
• Used from actually videos on the Web
• Participants practiced in the club for school
festival's performance
• Music : Traila$ong’s “Gravity”
Instructions
• Asked participants to reproduce the choreography
of the dance video
• Asked participants to practice everyone together to
make the same choreography
11.
Let participants to select the dance that they
thought their own one, and rank the skeletons
from first to third place
→ To clarify whether it is possible to distinguish
one’s own dance from video of only skeleton
1. Can human distinguish personality?
select the dance 5 times
12.
1. Can human distinguish personality?
• The selection of dance × 5 times
• To rank the skeletons from first to third place
• 1st=5 point, 2nd=3 point, 3rd=1 point
The average point of each skeleton = “Score”
𝑆𝑐𝑜𝑟𝑒 =
𝑃𝑜𝑖𝑛𝑡 1 + 𝑃𝑜𝑖𝑛𝑡 2 + ⋯ + 𝑃𝑜𝑖𝑛𝑡 4 + 𝑃𝑜𝑖𝑛𝑡(5)
5
13.
Let participants to select the dance that they
thought their own one, and rank the skeletons
from first to third place
→ To clarify whether it is possible to distinguish
one’s own dance from video of only skeleton
Interviewed for element
to determine the distinction
→ To clarify where the personality
appears in the skeleton
1. Can human distinguish personality?
14.
1. Can human distinguish personality?
Divided participants into two groups
the manner of expressing personality differs
depending on the level of experience
Little experience Rich experience
11 participants 11 participants
Dance
experience
How much they can express the personality
depends on the level of their dance experience
15.
1. Can human distinguish personality?
Divided participants into two groups
Little experience
・Never learned from
a skilled teacher
・Delightful dance as
a hobby
Rich experience
・Learning from
skilled instructors
・A lot of practice
11 participants 11 participants
Dance
experience
16.
A's B's C's D's E's F's G's H's I's J's K's L's
A 2.8 1.2 0.0 0.6 0.0 1.2 0.6 1.0 0.2 0.0 0.8 0.6
B 0.0 3.6 0.2 0.2 1.8 0.2 2.0 0.8 0.0 0.0 0.0 0.2
C 0.0 0.6 0.8 0.0 0.0 0.4 2.0 1.8 1.0 1.0 0.6 0.8
D 1.2 1.0 0.0 0.4 0.2 1.6 0.0 3.2 0.0 0.2 0.0 1.2
E 0.2 0.8 2.2 0.0 2.6 1.6 0.0 1.2 0.0 0.4 0.0 0.0
F 0.0 0.6 0.0 2.2 1.2 3.2 0.0 1.2 0.4 0.2 0.0 0.0
G 2.0 0.0 0.2 0.6 1.0 0.0 2.4 0.4 0.6 0.0 1.8 0.0
… … … … … … … … … … … … …
K 0.0 0.2 0.0 0.0 0.6 2.8 0.0 3.4 0.0 1.0 0.0 1.0
1. ResultsEach participant
The skeletal information that each participant
thinks the answer
• Participant A chose the
answers 5 times
• Add the score(average point)
to each skeletal information
17.
A's B's C's D's E's F's G's H's I's J's K's L's
A 2.8 1.2 0.0 0.6 0.0 1.2 0.6 1.0 0.2 0.0 0.8 0.6
B 0.0 3.6 0.2 0.2 1.8 0.2 2.0 0.8 0.0 0.0 0.0 0.2
C 0.0 0.6 0.8 0.0 0.0 0.4 2.0 1.8 1.0 1.0 0.6 0.8
D 1.2 1.0 0.0 0.4 0.2 1.6 0.0 3.2 0.0 0.2 0.0 1.2
E 0.2 0.8 2.2 0.0 2.6 1.6 0.0 1.2 0.0 0.4 0.0 0.0
F 0.0 0.6 0.0 2.2 1.2 3.2 0.0 1.2 0.4 0.2 0.0 0.0
G 2.0 0.0 0.2 0.6 1.0 0.0 2.4 0.4 0.6 0.0 1.8 0.0
… … … … … … … … … … … … …
K 0.0 0.2 0.0 0.0 0.6 2.8 0.0 3.4 0.0 1.0 0.0 1.0
1. ResultsEach participant
The skeletal information that each participant
thinks the answer
• Highlight the highest score
• Participant A chose her own
skeleton as the final answer
18.
A's B's C's D's E's F's G's H's I's J's K's L's
A 2.8 1.2 0.0 0.6 0.0 1.2 0.6 1.0 0.2 0.0 0.8 0.6
B 0.0 3.6 0.2 0.2 1.8 0.2 2.0 0.8 0.0 0.0 0.0 0.2
C 0.0 0.6 0.8 0.0 0.0 0.4 2.0 1.8 1.0 1.0 0.6 0.8
D 1.2 1.0 0.0 0.4 0.2 1.6 0.0 3.2 0.0 0.2 0.0 1.2
E 0.2 0.8 2.2 0.0 2.6 1.6 0.0 1.2 0.0 0.4 0.0 0.0
F 0.0 0.6 0.0 2.2 1.2 3.2 0.0 1.2 0.4 0.2 0.0 0.0
G 2.0 0.0 0.2 0.6 1.0 0.0 2.4 0.4 0.6 0.0 1.8 0.0
H 0.0 0.2 0.0 0.6 0.4 0.2 0.0 2.6 0.0 0.0 4.2 0.8
I 0.2 1.2 0.8 0.6 0.0 0.6 1.0 2.6 0.0 1.6 0.2 0.2
J 0.0 0.6 0.2 0.0 2.0 1.0 1.0 0.8 0.6 0.6 2.2 0.0
K 0.0 0.2 0.0 0.0 0.6 2.8 0.0 3.4 0.0 1.0 0.0 1.0
a's b's c's d's e's f's g's h's i's j's k's l's
a 0.4 3.0 0.6 0.0 0.0 0.0 1.2 1.8 0.2 0.0 1.2 0.6
b 0.0 0.6 0.0 0.0 1.6 0.6 0.4 0.0 0.8 0.0 4.2 0.8
c 0.0 0.6 2.4 0.6 1.0 0.0 2.0 1.0 0.4 0.2 0.8 0.0
d 0.2 0.6 1.2 0.6 1.0 0.2 0.0 1.0 1.2 1.2 1.0 0.8
e 0.0 0.0 0.2 0.0 1.4 0.2 0.0 3.2 0.6 0.4 2.0 1.0
f 0.0 0.0 0.0 0.6 0.0 0.8 1.6 0.4 0.6 0.2 2.8 2.0
g 0.2 0.0 0.0 0.0 0.0 3.2 0.6 0.0 0.2 0.2 2.4 2.2
h 0.0 0.2 0.6 0.0 0.0 0.0 0.2 2.6 0.0 0.2 1.2 4.0
i 0.0 0.0 0.0 0.2 0.0 0.0 1.6 0.0 4.6 0.2 2.4 0.0
j 0.2 0.6 0.0 1.2 0.8 0.0 0.0 2.2 0.0 0.0 2.2 1.8
k 0.0 1.0 0.6 0.0 0.0 2.2 2.2 0.2 2.0 0.0 0.8 0.0
1. Results
Little experience
2 out of 11 participants gave
the highest score
Rich experience
5 out of 11 participants gave
the highest score
In both groups, the participants who did not choose their
own dance tended to choose the specific one
19.
nBody part used for discrimination
nElement used for discrimination
Hands Feet Chest Whole body Other
Little 10 2 1 10 3
Rich 17(11) 4 2 8 3
Total 27 6 3 18 6
1. Results(interview)
Shape Movement Speed Timing Habit Direction
Center of
gravity
Little 8(6) 7 3 2 0 2 3
Rich 20(10) 1 5 3 4 2 1
Total 28 8 8 5 4 4 4
Little –avg. 2.3 cases ・Rich –avg. 3.2 cases
20.
Little Rich
Percentage of participants
who chose their own dance
18%
(2/11)
45%
(5/11)
Noticed body part Hand, Whole body
Noticed elements Movement Shape
1. Results
21.
Little Rich
Percentage of participants
who chose their own dance
18%
(2/11)
45%
(5/11)
Noticed body part Hand, Whole body
Noticed elements Movement Shape
1. Results
Rich experience group were able to identify
their own dance more accurately
22.
Little Rich
Percentage of participants
who chose their own dance
18%
(2/11)
45%
(5/11)
Noticed body part Hand, Whole body
Noticed elements Movement Shape
1. Results
Rich experience group were able to identify
their own dance more accurately
Generate machine learning's features using
these two elements of concentrated answers
e.g., movement
of arms, knees e.g., angle of hands
23.
2. Can machine distinguish personality?
Classify the dance by generating feature quantities
based on discrimination factors‘
Used the 3D skeletal information obtained by Kinect
nMovement -> Movement amount features
Amount of spatial movement
of each skeletal point every second
nShape -> Joint angle features
Average of joint angles
every second (30 frames)
24.
2. Can machine distinguish personality?
• Divided participants into the same two groups
as the subjective evaluation experiment
• Used Random Forest as the classification algorithm
• Performed 12-value classification learning
• To determine how well dancers can be identified
from skeletal information
Little experience Rich experience
25.
2. Generating feature quantities
Movement features
…Generated 13D vectors by calculating the amount
of spatial movement
182 𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑠 𝑜𝑓 14 𝑠𝑒𝑐𝑜𝑛𝑑𝑠 × 13 𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑠
shoulderelbow
knee
hand
hip
foot
torso
26.
2. Generating feature quantities
Angle features
…Generated 6D vectors by calculating the joint angles
shoulder
elbow
knee
28.
2. Generating feature quantities
As two countermeasures for noise
• Linear interpolation from the frame before and after
the defect frame
• Smoothing by the method of exponential moving
average
Before After
2D
29.
2. Can machine distinguish personality?
10 (= 𝐶@ A) combinations of training and test data
can be made from data per person
• Learned 10 patterns each time
• Calculated classification probability and
average accuracy rate from the results
Dance
1
Dance
2
Dance
3
Dance
4
Dance
5
Training data Test data
37.
2. Results
The average classification accuracy
Machine learning can distinguish individuals
Little Rich
Movement feature 95.4% 89.5%
Angle feature 99.1% 92.0%
38.
2. The reason rich experience
group was worse
• Compared to the little experience group,
the rich experience group included more data with
missing values
• The dance that was wrongly estimated was
concentrated on the data had missing values
39.
2. Correctly estimated number
In the rich group
Except for BCEF where all dances were correct for both features
Movement features Angle features
1 2 3 4 5
A 1 4 4 4 4
D 4 4 1 4 4
G 4 4 4 4 3
H 4 4 4 4 4
I 4 0 4 1 4
J 3 4 4 4 4
K 4 4 3 4 4
L 3 4 4 2 4
1 2 3 4 5
A 4 4 3 4 2
D 4 3 3 4 2
G 4 3 2 4 0
H 4 4 4 2 4
I 4 0 4 4 4
J 4 4 0 4 3
K 4 4 4 4 4
L 4 4 4 4 4
40.
When separated by a dance count instead of every second
with features delimited every 4 counts (64 frames)
with features delimited every 2 counts (32 frames)
Little Rich
Movement 87% 72%
Angle 94% 78%
Little Rich
Movement 94% 87%
Angle 99% 85%
2. Results
41.
2. Results(effective features)
The random forest is ensemble learning that learns
by a set of decision trees
The importance of each feature vector can be evaluated
by comparing each decision tree
Training
Data
Result 1
Result 2
Result 3
Final
Result
42.
2. Results(effective features)
n Movement features
The upper body such as the chest and right shoulder etc.
n Angle features
The left and right knees
43.
Discuss the personality
nCan human distinguish personality?
Human can find features subjectively from dances
with only skeletal information
→Many did not correctly identify their own dance
nCan machine distinguish personality?
Machine can discriminate dances with high
accuracy regardless of the features used
It should become easier to find the personality
of one’s own dance by using these features
44.
Where is the personality?
Consider the position and elements of the body
where personality tends to appear
nCan human distinguish personality?
nCan machine distinguish personality?
Movement
Angle
Hands Whole body Shape Movement
Little 10 10 8 7
Rich 17 8 20 1
Considered to correspond to the whole body
emphasized in the subjective evaluation,
because the ratio occupied in the skeleton
used for the subjective evaluation is large
45.
To make this study
Several challenges
• Participants danced specific choreography
→ Necessary to conduct experiments with variety
of choreography, and to clarify whether it is
possible to identify the dancer
46.
To make this study
Several challenges
• Participants danced specific choreography
→ Necessary to conduct experiments with variety
of choreography, and to clarify whether it is
possible to identify the dancer
Future work
• Clarify what constitutes personality in dance
• Examine the application method of the extracted
personality
• Consider the method to search for dance videos
that match the personality of the user
47.
Conclusion
Examined hip-hop dance to clarify the possibility to
extract personality
nCan human distinguish personality?
• Human can distinguish the specific dance in the
only skeletal information
• The little group focused the movement
• The rich group focused the shape
nCan machine distinguish personality?
• Machine can discriminate individuals with high
accuracy using either of two features,
movement amount and joint angle