Successfully reported this slideshow.

Automatic System for Editing Dance Videos Recorded Using Multiple Cameras

0

Share

Loading in …3
×
1 of 72
1 of 72

Automatic System for Editing Dance Videos Recorded Using Multiple Cameras

0

Share

Download to read offline

As social media has matured, uploading video content has increased. Multiple videos of physical performances, such as dance, are difficult to integrate into high-quality videos without knowledge of video-editing principles. In this study, we present a system that automatically edits dance-performance videos taken from multiple viewpoints into a more attractive and sophisticated dance video. Our system can crop the frame of each camera appropriately by using the performer’s behavior and skeleton information. The system determines the camera switches and cut lengths following a probabilistic model of general cinematography guidelines and of knowledge extracted from expert experience. In this study, our system automatically edited a dance video of four performers taken from multiple viewpoints, and ten video-production experts evaluated the generated video. As a result of a comparison of another automatic editing system, our system tended to be performed better.

As social media has matured, uploading video content has increased. Multiple videos of physical performances, such as dance, are difficult to integrate into high-quality videos without knowledge of video-editing principles. In this study, we present a system that automatically edits dance-performance videos taken from multiple viewpoints into a more attractive and sophisticated dance video. Our system can crop the frame of each camera appropriately by using the performer’s behavior and skeleton information. The system determines the camera switches and cut lengths following a probabilistic model of general cinematography guidelines and of knowledge extracted from expert experience. In this study, our system automatically edited a dance video of four performers taken from multiple viewpoints, and ten video-production experts evaluated the generated video. As a result of a comparison of another automatic editing system, our system tended to be performed better.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Automatic System for Editing Dance Videos Recorded Using Multiple Cameras

  1. 1. Automatic System for Editing Dance Videos Recorded Using Multiple Cameras Shuhei Tsuchida, Satoru Fukayama, Masataka Goto National Institute of Advanced Industrial Science and Technology (AIST) ACE 2017
  2. 2. Do you think this video is attractive?
  3. 3. Multi-view Single-view
  4. 4. Camera A Camera B Camera C
  5. 5. Time Camera A Camera B Camera C
  6. 6. Camera A Camera B ? ? ? ? Time Camera C
  7. 7. Purpose Editing dance videos automatically more attractive and sophisticated
  8. 8. Proposed system
  9. 9. System Flow
  10. 10. System Flow System
  11. 11. System Flow 1. Preprocessing 2. Camera shot selection
  12. 12. System Flow 1. Preprocessing 2. Camera shot selection
  13. 13. Preprocessing: Align time axis
  14. 14. Preprocessing: Align time axis
  15. 15. Preprocessing: Beat tracking
  16. 16. Preprocessing: Beat tracking Beat postion Beat postion
  17. 17. Preprocessing: Extract dance part Dance part
  18. 18. Preprocessing: Crop
  19. 19. Preprocessing: Crop
  20. 20. Preprocessing: Crop
  21. 21. Preprocessing: Crop
  22. 22. Preprocessing: Crop
  23. 23. Preprocessing: Crop
  24. 24. System Flow 1. Preprocessing 2. Camera shot selection
  25. 25. System Flow 1. Preprocessing 2. Camera shot selection
  26. 26. Camera A Camera B Camera C 1beat 1beat 1beat 1beat 1beat 1beat Formulation as an optimization problem to select a path to choose an effective camera transition. Camera shot selection Path
  27. 27. Camera shot selection Compute probability
  28. 28. Camera shot selection Compute probability
  29. 29. Camera shot selection Find optimal path
  30. 30. Camera shot selection Find optimal path
  31. 31. Camera shot selection Find optimal path
  32. 32. Camera shot selection Having the higher probability
  33. 33. How to compute probability
  34. 34. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  35. 35. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  36. 36. Principle 1: Switch camera according to beat. Principles of Video Editing: Principle 1
  37. 37. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  38. 38. Principle 2: The maximum length and minimum length of a cut are changed according to user preferences . Principles of Video Editing: Principle 2 Short Cut Long Cut
  39. 39. Camera A Min Max 1 2 3 4 5 6 Principle 2: The maximum length and minimum length of a cut are changed according to user preferences . Camera B Camera C Principles of Video Editing: Principle 2
  40. 40. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  41. 41. Principle 3: Switch to a camera with different composition to prevent Jump cut . . Similar composition Different composition Principles of Video Editing: Principle 3 BAD GOOD
  42. 42. C A B Wc WA WB Principle 3: Switch to a camera with different composition to prevent Jump cut . Principles of Video Editing: Principle 3 Averaged Optical flow
  43. 43. Switch C A B Wc WA WB Principle 3: Switch to a camera with different composition to prevent Jump cut . Principles of Video Editing: Principle 3 Averaged Optical flow Large Small Switch the camera from larger optical flow to smaller optical flow. Switch the camera from smaller optical flow to larger optical flow.
  44. 44. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  45. 45. Principles of Video Editing: Principle 4 Principle 4: The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Shot : Close Cut length: Short Shot : Distant Cut length: Long
  46. 46. Principle 4: The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. C A B Wc WA WB Averaged Optical flow Principles of Video Editing: Principle 4
  47. 47. Principle 4: The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. C A B Wc WA WB Averaged Optical flow Large Small Optical flow is larger, the shorter cut length has a higher probability. Optical flow is smaller, the longer cut length has a higher probability. Short cut length Long cut length Principles of Video Editing: Principle 4
  48. 48. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  49. 49. Main camera Principles of Video Editing: Principle 5 Principle 5: Make the main camera recognizable.
  50. 50. Principle 5: Make the main camera recognizable. Principles of Video Editing: Principle 5
  51. 51. Principle 5: Make the main camera recognizable. Principles of Video Editing: Principle 5
  52. 52. Principle 5: Make the main camera recognizable. Principles of Video Editing: Principle 5 Main camera
  53. 53. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  54. 54. Not intense movement part Intense movement part Principle 6: Decide what to emphasize and when to vary the pace. Principles of Video Editing: Principle 6 Prefer more distant shot Prefer closer shot
  55. 55. Principles of Video Editing: Principle 6 Principle 6: Decide what to emphasize and when to vary the pace. C A B Wc WA WB Averaged Optical flow
  56. 56. Principles of Video Editing: Principle 6 Principle 6: Decide what to emphasize and when to vary the pace. C A B Wc WA WB Averaged Optical flow Threshold Averaged optical flow per beat Averaged optical flow Beat
  57. 57. Threshold Averaged optical flow per beat Beat Principles of Video Editing: Principle 6 Principle 6: Decide what to emphasize and when to vary the pace. C A B Wc WA WB Averaged Optical flow Averaged optical flow Closer shot More distant shot
  58. 58. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  59. 59. Principle 2-6 probability distribution: 𝑃!, 𝑃", 𝑃#, 𝑃$, 𝑃% Camera combination: 𝑥&:( = 𝑥&, 𝑥!, … , 𝑥( (𝑇 = 𝑡)*+) Evaluation Function 𝑂(𝑥&:() = + ,-& ( 𝑙𝑜𝑔𝑃(𝑥,) 𝑃 𝑥! = 𝑎"𝑃" 𝑥! + 𝑎#𝑃# 𝑥! + 𝑎$𝑃$ 𝑥! + 𝑎%𝑃% 𝑥! + 𝑎&𝑃& 𝑥! 𝑎" + 𝑎# + 𝑎$ + 𝑎% + 𝑎& = 1 Probabilistic model
  60. 60. Probabilistic model
  61. 61. Evaluation Experiment Purpose To investigate whether it is capable of generating more attractive and sophisticated video. Evaluation object Automatic edited four-genre dance videos (10 cameras) ・Proposed System : 6 ・Baseline : 1 Total: 7 conditions x 4 genres = 28 videos 10 experts(up to 20 year experience) 7 Likert scale Evaluation method
  62. 62. Evaluation Experiment Q: This editing is attractive. 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  63. 63. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  64. 64. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  65. 65. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  66. 66. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  67. 67. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  68. 68. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set Parameter setting is important for this automatic system.
  69. 69. Conclusion • We proposed a system that automatically generates more attractive and sophisticated dance videos. • The system is based on probabilistic framework that uses six principles obtained from professional dance editing expert. Future Work Live-performance streaming.
  70. 70. Skeleton information x y (0, 0) Screen Example of acquisition of skeletal information The amount of skeleton information not obtained in a frame : bf The amount of skeleton information available : B The total number of frames : F Variance value of x : V(xall) Variance value of y : V(yall) 𝛾 = 1 𝐹 ) '() * 𝑏' 𝐵 The rate of undetected skeleton information : 𝑚 = 𝑉 𝑥*.. + 𝑉(𝑦*..) 𝛾 Principles of Video Editing ‒ Probabilistic model of Principle 5 Principle 5: Make the main camera recognizable.

×