1.
Automatic System for Editing Dance Videos
Recorded Using Multiple Cameras
Shuhei Tsuchida, Satoru Fukayama, Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST)
ACE 2017

2.
Do you think this video is attractive?

3.
Multi-view
Single-view

4.
Camera A
Camera B
Camera C

5.
Time
Camera A
Camera B
Camera C

6.
Camera A
Camera B
? ? ?
?
Time
Camera C

7.
Purpose
Editing dance videos automatically more attractive and sophisticated

8.
Proposed system

9.
System Flow

10.
System Flow
System

11.
System Flow
1. Preprocessing
2. Camera shot
selection

12.
System Flow
1. Preprocessing
2. Camera shot
selection

13.
Preprocessing: Align time axis

14.
Preprocessing: Align time axis

15.
Preprocessing: Beat tracking

16.
Preprocessing: Beat tracking
Beat postion
Beat postion

17.
Preprocessing: Extract dance part
Dance part

18.
Preprocessing: Crop

19.
Preprocessing: Crop

20.
Preprocessing: Crop

21.
Preprocessing: Crop

22.
Preprocessing: Crop

23.
Preprocessing: Crop

24.
System Flow
1. Preprocessing
2. Camera shot
selection

25.
System Flow
1. Preprocessing
2. Camera shot
selection

26.
Camera A
Camera B
Camera C
1beat 1beat 1beat 1beat 1beat 1beat
Formulation as an optimization problem to select a path
to choose an effective camera transition.
Camera shot selection
Path

27.
Camera shot selection
Compute probability

28.
Camera shot selection
Compute probability

29.
Camera shot selection
Find optimal path

30.
Camera shot selection
Find optimal path

31.
Camera shot selection
Find optimal path

32.
Camera shot selection
Having the higher probability

33.
How to compute probability

34.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

35.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

36.
Principle 1: Switch camera according to beat.
Principles of Video Editing: Principle 1

37.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

38.
Principle 2: The maximum length and minimum length of a cut are
changed according to user preferences ．
Principles of Video Editing: Principle 2
Short Cut Long Cut

39.
Camera A
Min Max
1 2 3 4 5 6
Principle 2: The maximum length and minimum length of a cut are
changed according to user preferences ．
Camera B
Camera C
Principles of Video Editing: Principle 2

40.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

41.
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
．
Similar composition Different composition
Principles of Video Editing: Principle 3
BAD GOOD

42.
C
A
B
Wc
WA
WB
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
Principles of Video Editing: Principle 3
Averaged Optical
flow

43.
Switch
C
A
B
Wc
WA
WB
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
Principles of Video Editing: Principle 3
Averaged Optical
flow
Large
Small
Switch the camera from larger optical flow to smaller optical flow.
Switch the camera from smaller optical flow to larger optical flow.

44.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

45.
Principles of Video Editing: Principle 4
Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Shot : Close
Cut length: Short
Shot : Distant
Cut length: Long

46.
Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Principles of Video Editing: Principle 4

47.
Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Large
Small
Optical flow is larger, the shorter cut length has a higher probability.
Optical flow is smaller, the longer cut length has a higher probability.
Short
cut
length
Long
cut
length
Principles of Video Editing: Principle 4

48.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

49.
Main camera
Principles of Video Editing: Principle 5
Principle 5: Make the main camera recognizable.

50.
Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5

51.
Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5

52.
Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5
Main camera

53.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

54.
Not intense movement part Intense movement part
Principle 6: Decide what to emphasize and when to vary the pace.
Principles of Video Editing: Principle 6
Prefer more distant shot Prefer closer shot

55.
Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow

56.
Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Threshold
Averaged optical flow per beat
Averaged
optical
flow
Beat

57.
Threshold
Averaged optical flow per beat
Beat
Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Averaged
optical
flow
Closer shot
More distant shot

58.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

59.
Principle 2-6 probability distribution: 𝑃!, 𝑃", 𝑃#, 𝑃$, 𝑃%
Camera combination: 𝑥&:( = 𝑥&, 𝑥!, … , 𝑥( (𝑇 = 𝑡)*+)
Evaluation Function
𝑂(𝑥&:() = +
,-&
(
𝑙𝑜𝑔𝑃(𝑥,)
𝑃 𝑥! = 𝑎"𝑃" 𝑥! + 𝑎#𝑃# 𝑥! + 𝑎$𝑃$ 𝑥! + 𝑎%𝑃% 𝑥! + 𝑎&𝑃& 𝑥!
𝑎" + 𝑎# + 𝑎$ + 𝑎% + 𝑎& = 1
Probabilistic model

60.
Probabilistic model

61.
Evaluation Experiment
Purpose
To investigate whether it is capable of generating
more attractive and sophisticated video.
Evaluation object
Automatic edited four-genre dance videos (10 cameras)
・Proposed System : 6
・Baseline : 1
Total: 7 conditions x 4 genres = 28 videos
10 experts(up to 20 year experience)
7 Likert scale
Evaluation method

62.
Evaluation Experiment
Q: This editing is attractive.
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

63.
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

64.
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

65.
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

66.
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

67.
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

68.
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Parameter setting is important for this automatic system.

69.
Conclusion
• We proposed a system that automatically generates
more attractive and sophisticated dance videos.
• The system is based on probabilistic framework that
uses six principles obtained from professional dance
editing expert.
Future Work
Live-performance streaming.

70.
Skeleton
information
x
y
(0, 0)
Screen
Example of acquisition of
skeletal information
The amount of skeleton information
not obtained in a frame : bf
The amount of skeleton information available
: B
The total number of frames : F
Variance value of x : V(xall)
Variance value of y : V(yall)
𝛾 =
1
𝐹
)
'()
*
𝑏'
𝐵
The rate of undetected
skeleton information :
𝑚 =
𝑉 𝑥*.. + 𝑉(𝑦*..)
𝛾
Principles of Video Editing ‒ Probabilistic model of Principle 5
Principle 5: Make the main camera recognizable.