As social media has matured, uploading video content has increased. Multiple videos of physical performances, such as dance, are difficult to integrate into high-quality videos without knowledge of video-editing principles. In this study, we present a system that automatically edits dance-performance videos taken from multiple viewpoints into a more attractive and sophisticated dance video. Our system can crop the frame of each camera appropriately by using the performer’s behavior and skeleton information. The system determines the camera switches and cut lengths following a probabilistic model of general cinematography guidelines and of knowledge extracted from expert experience. In this study, our system automatically edited a dance video of four performers taken from multiple viewpoints, and ten video-production experts evaluated the generated video. As a result of a comparison of another automatic editing system, our system tended to be performed better.

1. Automatic System for Editing Dance Videos
Recorded Using Multiple Cameras
Shuhei Tsuchida, Satoru Fukayama, Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST)
ACE 2017

2. Do you think this video is attractive?

3. Multi-view
Single-view

4. Camera A
Camera B
Camera C

5. Time
Camera A
Camera B
Camera C

6. Camera A
Camera B
? ? ?
?
Time
Camera C

7. Purpose
Editing dance videos automatically more attractive and sophisticated

8. Proposed system

10. System Flow

11. System Flow
System

12. System Flow
1. Preprocessing
2. Camera shot
selection

13. System Flow
1. Preprocessing
2. Camera shot
selection

14. Preprocessing: Align time axis

15. Preprocessing: Align time axis

16. Preprocessing: Beat tracking

17. Preprocessing: Beat tracking
Beat postion
Beat postion

18. Preprocessing: Extract dance part
Dance part

19. Preprocessing: Crop

20. Preprocessing: Crop

21. Preprocessing: Crop

22. Preprocessing: Crop

23. Preprocessing: Crop

24. Preprocessing: Crop

25. System Flow
1. Preprocessing
2. Camera shot
selection

26. System Flow
1. Preprocessing
2. Camera shot
selection

27. Camera A
Camera B
Camera C
1beat 1beat 1beat 1beat 1beat 1beat
Formulation as an optimization problem to select a path
to choose an effective camera transition.
Camera shot selection
Path

28. Camera shot selection
Compute probability

29. Camera shot selection
Compute probability

30. Camera shot selection
Find optimal path

31. Camera shot selection
Find optimal path

32. Camera shot selection
Find optimal path

33. Camera shot selection
Having the higher probability

34. How to compute probability

35. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

36. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

37. Principle 1: Switch camera according to beat.
Principles of Video Editing: Principle 1

38. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

39. Principle 2: The maximum length and minimum length of a cut are
changed according to user preferences ．
Principles of Video Editing: Principle 2
Short Cut Long Cut

40. Camera A
Min Max
1 2 3 4 5 6
Principle 2: The maximum length and minimum length of a cut are
changed according to user preferences ．
Camera B
Camera C
Principles of Video Editing: Principle 2

41. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

42. Principle 3: Switch to a camera with different composition to
prevent Jump cut .
．
Similar composition Different composition
Principles of Video Editing: Principle 3
BAD GOOD

43. C
A
B
Wc
WA
WB
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
Principles of Video Editing: Principle 3
Averaged Optical
flow

44. Switch
C
A
B
Wc
WA
WB
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
Principles of Video Editing: Principle 3
Averaged Optical
flow
Large
Small
Switch the camera from larger optical flow to smaller optical flow.
Switch the camera from smaller optical flow to larger optical flow.

45. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

46. Principles of Video Editing: Principle 4
Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Shot : Close
Cut length: Short
Shot : Distant
Cut length: Long

47. Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Principles of Video Editing: Principle 4

48. Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Large
Small
Optical flow is larger, the shorter cut length has a higher probability.
Optical flow is smaller, the longer cut length has a higher probability.
Short
cut
length
Long
cut
length
Principles of Video Editing: Principle 4

49. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

50. Main camera
Principles of Video Editing: Principle 5
Principle 5: Make the main camera recognizable.

51. Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5

52. Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5

53. Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5
Main camera

54. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

55. Not intense movement part Intense movement part
Principle 6: Decide what to emphasize and when to vary the pace.
Principles of Video Editing: Principle 6
Prefer more distant shot Prefer closer shot

56. Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow

57. Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Threshold
Averaged optical flow per beat
Averaged
optical
flow
Beat

58. Threshold
Averaged optical flow per beat
Beat
Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Averaged
optical
flow
Closer shot
More distant shot

59. Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.

60. Principle 2-6 probability distribution: 𝑃!, 𝑃", 𝑃#, 𝑃$, 𝑃%
Camera combination: 𝑥&:( = 𝑥&, 𝑥!, … , 𝑥( (𝑇 = 𝑡)*+)
Evaluation Function
𝑂(𝑥&:() = +
,-&
(
𝑙𝑜𝑔𝑃(𝑥,)
𝑃 𝑥! = 𝑎"𝑃" 𝑥! + 𝑎#𝑃# 𝑥! + 𝑎$𝑃$ 𝑥! + 𝑎%𝑃% 𝑥! + 𝑎&𝑃& 𝑥!
𝑎" + 𝑎# + 𝑎$ + 𝑎% + 𝑎& = 1
Probabilistic model

61. Probabilistic model

62. Evaluation Experiment
Purpose
To investigate whether it is capable of generating
more attractive and sophisticated video.
Evaluation object
Automatic edited four-genre dance videos (10 cameras)
・Proposed System : 6
・Baseline : 1
Total: 7 conditions x 4 genres = 28 videos
10 experts(up to 20 year experience)
7 Likert scale
Evaluation method

63. Evaluation Experiment
Q: This editing is attractive.
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

64. Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

65. Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

66. Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

67. Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

68. Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set

69. Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Parameter setting is important for this automatic system.

70. Conclusion
• We proposed a system that automatically generates
more attractive and sophisticated dance videos.
• The system is based on probabilistic framework that
uses six principles obtained from professional dance
editing expert.
Future Work
Live-performance streaming.

72. Skeleton
information
x
y
(0, 0)
Screen
Example of acquisition of
skeletal information
The amount of skeleton information
not obtained in a frame : bf
The amount of skeleton information available
: B
The total number of frames : F
Variance value of x : V(xall)
Variance value of y : V(yall)
𝛾 =
1
𝐹
)
'()
*
𝑏'
𝐵
The rate of undetected
skeleton information :
𝑚 =
𝑉 𝑥*.. + 𝑉(𝑦*..)
𝛾
Principles of Video Editing ‒ Probabilistic model of Principle 5
Principle 5: Make the main camera recognizable.