SlideShare a Scribd company logo
1 of 72
Download to read offline
Automatic System for Editing Dance Videos
Recorded Using Multiple Cameras
Shuhei Tsuchida, Satoru Fukayama, Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST)
ACE 2017
Do you think this video is attractive?
Multi-view
Single-view
Camera A
Camera B
Camera C
Time
Camera A
Camera B
Camera C
Camera A
Camera B
? ? ?
?
Time
Camera C
Purpose
Editing dance videos automatically more attractive and sophisticated
Proposed system
System Flow
System Flow
System
System Flow
1. Preprocessing
2. Camera shot
selection
System Flow
1. Preprocessing
2. Camera shot
selection
Preprocessing: Align time axis
Preprocessing: Align time axis
Preprocessing: Beat tracking
Preprocessing: Beat tracking
Beat postion
Beat postion
Preprocessing: Extract dance part
Dance part
Preprocessing: Crop
Preprocessing: Crop
Preprocessing: Crop
Preprocessing: Crop
Preprocessing: Crop
Preprocessing: Crop
System Flow
1. Preprocessing
2. Camera shot
selection
System Flow
1. Preprocessing
2. Camera shot
selection
Camera A
Camera B
Camera C
1beat 1beat 1beat 1beat 1beat 1beat
Formulation as an optimization problem to select a path
to choose an effective camera transition.
Camera shot selection
Path
Camera shot selection
Compute probability
Camera shot selection
Compute probability
Camera shot selection
Find optimal path
Camera shot selection
Find optimal path
Camera shot selection
Find optimal path
Camera shot selection
Having the higher probability
How to compute probability
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Principle 1: Switch camera according to beat.
Principles of Video Editing: Principle 1
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Principle 2: The maximum length and minimum length of a cut are
changed according to user preferences .
Principles of Video Editing: Principle 2
Short Cut Long Cut
Camera A
Min Max
1 2 3 4 5 6
Principle 2: The maximum length and minimum length of a cut are
changed according to user preferences .
Camera B
Camera C
Principles of Video Editing: Principle 2
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
.
Similar composition Different composition
Principles of Video Editing: Principle 3
BAD GOOD
C
A
B
Wc
WA
WB
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
Principles of Video Editing: Principle 3
Averaged Optical
flow
Switch
C
A
B
Wc
WA
WB
Principle 3: Switch to a camera with different composition to
prevent Jump cut .
Principles of Video Editing: Principle 3
Averaged Optical
flow
Large
Small
Switch the camera from larger optical flow to smaller optical flow.
Switch the camera from smaller optical flow to larger optical flow.
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Principles of Video Editing: Principle 4
Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Shot : Close
Cut length: Short
Shot : Distant
Cut length: Long
Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Principles of Video Editing: Principle 4
Principle 4: The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Large
Small
Optical flow is larger, the shorter cut length has a higher probability.
Optical flow is smaller, the longer cut length has a higher probability.
Short
cut
length
Long
cut
length
Principles of Video Editing: Principle 4
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Main camera
Principles of Video Editing: Principle 5
Principle 5: Make the main camera recognizable.
Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5
Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5
Principle 5: Make the main camera recognizable.
Principles of Video Editing: Principle 5
Main camera
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Not intense movement part Intense movement part
Principle 6: Decide what to emphasize and when to vary the pace.
Principles of Video Editing: Principle 6
Prefer more distant shot Prefer closer shot
Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Threshold
Averaged optical flow per beat
Averaged
optical
flow
Beat
Threshold
Averaged optical flow per beat
Beat
Principles of Video Editing: Principle 6
Principle 6: Decide what to emphasize and when to vary the pace.
C
A
B
Wc
WA
WB
Averaged Optical
flow
Averaged
optical
flow
Closer shot
More distant shot
Switch camera according to beat.
Principles of Video Editing
The maximum length and minimum length of a cut are changed
according to user preferences
Switch to a camera with different composition to prevent Jump cut .
The closer the shot, the shorter the cut length.
The more distant the shot, the longer the cut length.
Make the main camera recognizable.
Decide what to emphasize and when to vary the pace.
Principle 2-6 probability distribution: 𝑃!, 𝑃", 𝑃#, 𝑃$, 𝑃%
Camera combination: 𝑥&:( = 𝑥&, 𝑥!, … , 𝑥( (𝑇 = 𝑡)*+)
Evaluation Function
𝑂(𝑥&:() = +
,-&
(
𝑙𝑜𝑔𝑃(𝑥,)
𝑃 𝑥! = 𝑎"𝑃" 𝑥! + 𝑎#𝑃# 𝑥! + 𝑎$𝑃$ 𝑥! + 𝑎%𝑃% 𝑥! + 𝑎&𝑃& 𝑥!
𝑎" + 𝑎# + 𝑎$ + 𝑎% + 𝑎& = 1
Probabilistic model
Probabilistic model
Evaluation Experiment
Purpose
To investigate whether it is capable of generating
more attractive and sophisticated video.
Evaluation object
Automatic edited four-genre dance videos (10 cameras)
・Proposed System : 6
・Baseline : 1
Total: 7 conditions x 4 genres = 28 videos
10 experts(up to 20 year experience)
7 Likert scale
Evaluation method
Evaluation Experiment
Q: This editing is attractive.
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Q: This editing is attractive.
Evaluation Experiment
1 2 3 4 5 6 B 1 2 3 4 5 6 B
1 2 3 4 5 6 B 1 2 3 4 5 6 B
Parameter set
Parameter setting is important for this automatic system.
Conclusion
• We proposed a system that automatically generates
more attractive and sophisticated dance videos.
• The system is based on probabilistic framework that
uses six principles obtained from professional dance
editing expert.
Future Work
Live-performance streaming.
Skeleton
information
x
y
(0, 0)
Screen
Example of acquisition of
skeletal information
The amount of skeleton information
not obtained in a frame : bf
The amount of skeleton information available
: B
The total number of frames : F
Variance value of x : V(xall)
Variance value of y : V(yall)
𝛾 =
1
𝐹
)
'()
*
𝑏'
𝐵
The rate of undetected
skeleton information :
𝑚 =
𝑉 𝑥*.. + 𝑉(𝑦*..)
𝛾
Principles of Video Editing ‒ Probabilistic model of Principle 5
Principle 5: Make the main camera recognizable.

More Related Content

Similar to Automatic System for Editing Dance Videos Recorded Using Multiple Cameras

Dslr workshop camera audit form blank 2019 (1)
Dslr workshop camera audit form blank 2019 (1)Dslr workshop camera audit form blank 2019 (1)
Dslr workshop camera audit form blank 2019 (1)
HarryAllinson2
 

Similar to Automatic System for Editing Dance Videos Recorded Using Multiple Cameras (20)

Parts of a Digital Cinema Camera
Parts of a Digital Cinema CameraParts of a Digital Cinema Camera
Parts of a Digital Cinema Camera
 
DSLR video
DSLR videoDSLR video
DSLR video
 
Camera Audit Form Blank.docx
Camera Audit Form Blank.docxCamera Audit Form Blank.docx
Camera Audit Form Blank.docx
 
How to get the perfect camera exposure with PTZOptics
How to get the perfect camera exposure with PTZOpticsHow to get the perfect camera exposure with PTZOptics
How to get the perfect camera exposure with PTZOptics
 
Video camera analysis (1)
Video camera analysis (1)Video camera analysis (1)
Video camera analysis (1)
 
Training Videovigilancia IP: What, Why, When and How
Training Videovigilancia IP: What, Why, When and HowTraining Videovigilancia IP: What, Why, When and How
Training Videovigilancia IP: What, Why, When and How
 
Unit 2 Lesson 08
Unit 2 Lesson 08Unit 2 Lesson 08
Unit 2 Lesson 08
 
Szdalos
SzdalosSzdalos
Szdalos
 
How to select the best industrial camera
How to select the best industrial cameraHow to select the best industrial camera
How to select the best industrial camera
 
Tenvis IPROBOT 3 English User Manual
Tenvis IPROBOT 3 English User ManualTenvis IPROBOT 3 English User Manual
Tenvis IPROBOT 3 English User Manual
 
Digital Photo Taking
Digital  Photo  TakingDigital  Photo  Taking
Digital Photo Taking
 
Dslr workshop camera audit form blank 2019 (1)
Dslr workshop camera audit form blank 2019 (1)Dslr workshop camera audit form blank 2019 (1)
Dslr workshop camera audit form blank 2019 (1)
 
Brian Elliott's "Camera Basics" Lecture
Brian Elliott's "Camera Basics" LectureBrian Elliott's "Camera Basics" Lecture
Brian Elliott's "Camera Basics" Lecture
 
Jarrod Crawford's "Basic Editing Concepts" Lecture.
Jarrod Crawford's "Basic Editing Concepts" Lecture.Jarrod Crawford's "Basic Editing Concepts" Lecture.
Jarrod Crawford's "Basic Editing Concepts" Lecture.
 
Digital Photography Introduction
Digital Photography IntroductionDigital Photography Introduction
Digital Photography Introduction
 
Live Streaming Checklist
Live Streaming ChecklistLive Streaming Checklist
Live Streaming Checklist
 
Video Editing and Encoding
Video Editing and EncodingVideo Editing and Encoding
Video Editing and Encoding
 
10 tips for live streaming an event on location
10 tips for live streaming an event on location10 tips for live streaming an event on location
10 tips for live streaming an event on location
 
Camera Basics
Camera BasicsCamera Basics
Camera Basics
 
Video camera analysis
Video camera analysis Video camera analysis
Video camera analysis
 

More from Shuhei Tsuchida

ダンスをマスターした自身の映像を先に見ることによるダンス学習支援
ダンスをマスターした自身の映像を先に見ることによるダンス学習支援ダンスをマスターした自身の映像を先に見ることによるダンス学習支援
ダンスをマスターした自身の映像を先に見ることによるダンス学習支援
Shuhei Tsuchida
 
ダンスパフォーマンスにおける自走型スクリーンを用いた 隊形練習支援 (WISS 2012 tsuchida)
ダンスパフォーマンスにおける自走型スクリーンを用いた隊形練習支援 (WISS 2012 tsuchida)ダンスパフォーマンスにおける自走型スクリーンを用いた隊形練習支援 (WISS 2012 tsuchida)
ダンスパフォーマンスにおける自走型スクリーンを用いた 隊形練習支援 (WISS 2012 tsuchida)
Shuhei Tsuchida
 
移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)
移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)
移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)
Shuhei Tsuchida
 

More from Shuhei Tsuchida (20)

パフォーマンスを拡張する研究
パフォーマンスを拡張する研究パフォーマンスを拡張する研究
パフォーマンスを拡張する研究
 
ダンスステップ学習における分離学習の適用
ダンスステップ学習における分離学習の適用 ダンスステップ学習における分離学習の適用
ダンスステップ学習における分離学習の適用
 
神戸大学男女共同参画推進室主催「2021年度神戸大学ダイバーシティフォーラム ~男性の育児休業から考えるワーク・ライフ・バランス~」
神戸大学男女共同参画推進室主催「2021年度神戸大学ダイバーシティフォーラム ~男性の育児休業から考えるワーク・ライフ・バランス~」神戸大学男女共同参画推進室主催「2021年度神戸大学ダイバーシティフォーラム ~男性の育児休業から考えるワーク・ライフ・バランス~」
神戸大学男女共同参画推進室主催「2021年度神戸大学ダイバーシティフォーラム ~男性の育児休業から考えるワーク・ライフ・バランス~」
 
ダンスをマスターした自身の映像を先に見ることによるダンス学習支援
ダンスをマスターした自身の映像を先に見ることによるダンス学習支援ダンスをマスターした自身の映像を先に見ることによるダンス学習支援
ダンスをマスターした自身の映像を先に見ることによるダンス学習支援
 
ダンス情報処理
ダンス情報処理ダンス情報処理
ダンス情報処理
 
Exit20151212
Exit20151212Exit20151212
Exit20151212
 
複数の移動ロボットを用いた身体パフォーマンスの印象評価(EC2016)
複数の移動ロボットを用いた身体パフォーマンスの印象評価(EC2016)複数の移動ロボットを用いた身体パフォーマンスの印象評価(EC2016)
複数の移動ロボットを用いた身体パフォーマンスの印象評価(EC2016)
 
球体型自走ロボットを用いた ダンスパフォーマンス環境の構築(WISS2015 presen)
球体型自走ロボットを用いた ダンスパフォーマンス環境の構築(WISS2015 presen)球体型自走ロボットを用いた ダンスパフォーマンス環境の構築(WISS2015 presen)
球体型自走ロボットを用いた ダンスパフォーマンス環境の構築(WISS2015 presen)
 
ダンスパフォーマンスにおける自走型スクリーンを用いた 隊形練習支援 (WISS 2012 tsuchida)
ダンスパフォーマンスにおける自走型スクリーンを用いた隊形練習支援 (WISS 2012 tsuchida)ダンスパフォーマンスにおける自走型スクリーンを用いた隊形練習支援 (WISS 2012 tsuchida)
ダンスパフォーマンスにおける自走型スクリーンを用いた 隊形練習支援 (WISS 2012 tsuchida)
 
A System for Practicing Formations in Dance Performance Supported by Self-Pro...
A System for Practicing Formations in Dance Performance Supported by Self-Pro...A System for Practicing Formations in Dance Performance Supported by Self-Pro...
A System for Practicing Formations in Dance Performance Supported by Self-Pro...
 
A Dance Performance Environment in which Performers Dance with Multiple Robot...
A Dance Performance Environment in which Performers Dance with Multiple Robot...A Dance Performance Environment in which Performers Dance with Multiple Robot...
A Dance Performance Environment in which Performers Dance with Multiple Robot...
 
Mimebot: Sphere-shaped Mobile Robot Imitating Rotational Movement (MoMM2016 p...
Mimebot: Sphere-shaped Mobile Robot Imitating Rotational Movement (MoMM2016 p...Mimebot: Sphere-shaped Mobile Robot Imitating Rotational Movement (MoMM2016 p...
Mimebot: Sphere-shaped Mobile Robot Imitating Rotational Movement (MoMM2016 p...
 
移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)
移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)
移動ロボットを用いたダンスパフォーマンス環境の構築 (20161006 CEATEC 未踏ブース ストリートプレゼン 土田)
 
人と球体自律走行ロボット群の協調パフォーマンスシステムの開発 (20160706 jst start presen tsuchida)
人と球体自律走行ロボット群の協調パフォーマンスシステムの開発 (20160706 jst start presen tsuchida)人と球体自律走行ロボット群の協調パフォーマンスシステムの開発 (20160706 jst start presen tsuchida)
人と球体自律走行ロボット群の協調パフォーマンスシステムの開発 (20160706 jst start presen tsuchida)
 
回転移動を模したテクスチャ表示機能をもつ球体型移動ロボット (インタラクション 2016)
回転移動を模したテクスチャ表示機能をもつ球体型移動ロボット (インタラクション 2016)回転移動を模したテクスチャ表示機能をもつ球体型移動ロボット (インタラクション 2016)
回転移動を模したテクスチャ表示機能をもつ球体型移動ロボット (インタラクション 2016)
 
多視点ダンス映像のインタラクティブ編集システム
多視点ダンス映像のインタラクティブ編集システム多視点ダンス映像のインタラクティブ編集システム
多視点ダンス映像のインタラクティブ編集システム
 
Query-by-Dancing: 身体動作の類似性に基づくダンス楽曲検索システム
Query-by-Dancing: 身体動作の類似性に基づくダンス楽曲検索システムQuery-by-Dancing: 身体動作の類似性に基づくダンス楽曲検索システム
Query-by-Dancing: 身体動作の類似性に基づくダンス楽曲検索システム
 
AIST Dance Video Database: Multi-Genre, Multi-Dancer, and Multi-Camera Databa...
AIST Dance Video Database: Multi-Genre, Multi-Dancer, and Multi-Camera Databa...AIST Dance Video Database: Multi-Genre, Multi-Dancer, and Multi-Camera Databa...
AIST Dance Video Database: Multi-Genre, Multi-Dancer, and Multi-Camera Databa...
 
AIST Dance Video Database:  ダンス情報処理研究のためのストリートダンス動画データベース
AIST Dance Video Database:  ダンス情報処理研究のためのストリートダンス動画データベースAIST Dance Video Database:  ダンス情報処理研究のためのストリートダンス動画データベース
AIST Dance Video Database:  ダンス情報処理研究のためのストリートダンス動画データベース
 
DanceUnisoner: グループダンスパフォーマンス動画作成インタフェース
DanceUnisoner: グループダンスパフォーマンス動画作成インタフェースDanceUnisoner: グループダンスパフォーマンス動画作成インタフェース
DanceUnisoner: グループダンスパフォーマンス動画作成インタフェース
 

Recently uploaded

Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Precisely
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 

Recently uploaded (20)

ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 

Automatic System for Editing Dance Videos Recorded Using Multiple Cameras

  • 1. Automatic System for Editing Dance Videos Recorded Using Multiple Cameras Shuhei Tsuchida, Satoru Fukayama, Masataka Goto National Institute of Advanced Industrial Science and Technology (AIST) ACE 2017
  • 2. Do you think this video is attractive?
  • 6. Camera A Camera B ? ? ? ? Time Camera C
  • 7. Purpose Editing dance videos automatically more attractive and sophisticated
  • 9.
  • 12. System Flow 1. Preprocessing 2. Camera shot selection
  • 13. System Flow 1. Preprocessing 2. Camera shot selection
  • 17. Preprocessing: Beat tracking Beat postion Beat postion
  • 18. Preprocessing: Extract dance part Dance part
  • 25. System Flow 1. Preprocessing 2. Camera shot selection
  • 26. System Flow 1. Preprocessing 2. Camera shot selection
  • 27. Camera A Camera B Camera C 1beat 1beat 1beat 1beat 1beat 1beat Formulation as an optimization problem to select a path to choose an effective camera transition. Camera shot selection Path
  • 33. Camera shot selection Having the higher probability
  • 34. How to compute probability
  • 35. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 36. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 37. Principle 1: Switch camera according to beat. Principles of Video Editing: Principle 1
  • 38. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 39. Principle 2: The maximum length and minimum length of a cut are changed according to user preferences . Principles of Video Editing: Principle 2 Short Cut Long Cut
  • 40. Camera A Min Max 1 2 3 4 5 6 Principle 2: The maximum length and minimum length of a cut are changed according to user preferences . Camera B Camera C Principles of Video Editing: Principle 2
  • 41. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 42. Principle 3: Switch to a camera with different composition to prevent Jump cut . . Similar composition Different composition Principles of Video Editing: Principle 3 BAD GOOD
  • 43. C A B Wc WA WB Principle 3: Switch to a camera with different composition to prevent Jump cut . Principles of Video Editing: Principle 3 Averaged Optical flow
  • 44. Switch C A B Wc WA WB Principle 3: Switch to a camera with different composition to prevent Jump cut . Principles of Video Editing: Principle 3 Averaged Optical flow Large Small Switch the camera from larger optical flow to smaller optical flow. Switch the camera from smaller optical flow to larger optical flow.
  • 45. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 46. Principles of Video Editing: Principle 4 Principle 4: The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Shot : Close Cut length: Short Shot : Distant Cut length: Long
  • 47. Principle 4: The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. C A B Wc WA WB Averaged Optical flow Principles of Video Editing: Principle 4
  • 48. Principle 4: The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. C A B Wc WA WB Averaged Optical flow Large Small Optical flow is larger, the shorter cut length has a higher probability. Optical flow is smaller, the longer cut length has a higher probability. Short cut length Long cut length Principles of Video Editing: Principle 4
  • 49. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 50. Main camera Principles of Video Editing: Principle 5 Principle 5: Make the main camera recognizable.
  • 51. Principle 5: Make the main camera recognizable. Principles of Video Editing: Principle 5
  • 52. Principle 5: Make the main camera recognizable. Principles of Video Editing: Principle 5
  • 53. Principle 5: Make the main camera recognizable. Principles of Video Editing: Principle 5 Main camera
  • 54. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 55. Not intense movement part Intense movement part Principle 6: Decide what to emphasize and when to vary the pace. Principles of Video Editing: Principle 6 Prefer more distant shot Prefer closer shot
  • 56. Principles of Video Editing: Principle 6 Principle 6: Decide what to emphasize and when to vary the pace. C A B Wc WA WB Averaged Optical flow
  • 57. Principles of Video Editing: Principle 6 Principle 6: Decide what to emphasize and when to vary the pace. C A B Wc WA WB Averaged Optical flow Threshold Averaged optical flow per beat Averaged optical flow Beat
  • 58. Threshold Averaged optical flow per beat Beat Principles of Video Editing: Principle 6 Principle 6: Decide what to emphasize and when to vary the pace. C A B Wc WA WB Averaged Optical flow Averaged optical flow Closer shot More distant shot
  • 59. Switch camera according to beat. Principles of Video Editing The maximum length and minimum length of a cut are changed according to user preferences Switch to a camera with different composition to prevent Jump cut . The closer the shot, the shorter the cut length. The more distant the shot, the longer the cut length. Make the main camera recognizable. Decide what to emphasize and when to vary the pace.
  • 60. Principle 2-6 probability distribution: 𝑃!, 𝑃", 𝑃#, 𝑃$, 𝑃% Camera combination: 𝑥&:( = 𝑥&, 𝑥!, … , 𝑥( (𝑇 = 𝑡)*+) Evaluation Function 𝑂(𝑥&:() = + ,-& ( 𝑙𝑜𝑔𝑃(𝑥,) 𝑃 𝑥! = 𝑎"𝑃" 𝑥! + 𝑎#𝑃# 𝑥! + 𝑎$𝑃$ 𝑥! + 𝑎%𝑃% 𝑥! + 𝑎&𝑃& 𝑥! 𝑎" + 𝑎# + 𝑎$ + 𝑎% + 𝑎& = 1 Probabilistic model
  • 62. Evaluation Experiment Purpose To investigate whether it is capable of generating more attractive and sophisticated video. Evaluation object Automatic edited four-genre dance videos (10 cameras) ・Proposed System : 6 ・Baseline : 1 Total: 7 conditions x 4 genres = 28 videos 10 experts(up to 20 year experience) 7 Likert scale Evaluation method
  • 63. Evaluation Experiment Q: This editing is attractive. 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  • 64. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  • 65. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  • 66. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  • 67. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  • 68. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set
  • 69. Q: This editing is attractive. Evaluation Experiment 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B 1 2 3 4 5 6 B Parameter set Parameter setting is important for this automatic system.
  • 70. Conclusion • We proposed a system that automatically generates more attractive and sophisticated dance videos. • The system is based on probabilistic framework that uses six principles obtained from professional dance editing expert. Future Work Live-performance streaming.
  • 71.
  • 72. Skeleton information x y (0, 0) Screen Example of acquisition of skeletal information The amount of skeleton information not obtained in a frame : bf The amount of skeleton information available : B The total number of frames : F Variance value of x : V(xall) Variance value of y : V(yall) 𝛾 = 1 𝐹 ) '() * 𝑏' 𝐵 The rate of undetected skeleton information : 𝑚 = 𝑉 𝑥*.. + 𝑉(𝑦*..) 𝛾 Principles of Video Editing ‒ Probabilistic model of Principle 5 Principle 5: Make the main camera recognizable.