SlideShare a Scribd company logo
1 of 30
Download to read offline
Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs
.
YALMS: Review
.
YALMS....
..
....stands for Yet Another Lecture Management System
• basically, a class robot
◦ robot: set of automation scripts and frameworks
• first presented in ET研2013.07 04
• a long-term project with 4-5 main components -- see the roadmap at the end
04 M.Zhanikeev+1 "YALMS: Yet Another Lecture Management System..." IEICE ET研 (2013.09)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 2/28
...
2/28
.
YALMS: Overall Design
Step 1: Record and Stream
V
A
S
Lecturer
Viewers
Content
Server
HTML5
HTML5
Muxer
RecordScreen
Streaming
Step 2: Annotate
Lecturer
S
Auto-
annotate
C
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 3/28
...
3/28
.
YALMS: Annotation
VIDEO
VAS
Annotate
Pack/Encode
Screen
VIDEO
VAS
VA
Realtime Screen
VIDEO
S
Mux
A V
MISC. C
C
PNG
Replay Screen
VIDEO
A V
MISC. C
Annotate
S
S
S
The focus of
this study
YALMS
Robot
S
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 4/28
...
4/28
.
Fuzzy Environments
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 5/28
...
5/28
.
Sources of Fuzziness
1. unusual screen sizes in presentation mode
◦ this PDF was created in 16:9mode to fit my notebook perfectly, will NOT fit
others
2. unpredictable or unavoidable margins when saving your slides as PDF
◦ PPTX! no way to avoid white margins in print
◦ tex2pdf creates perfect PDFs! --- look the same in PDF and in presentation mode
3. slide animation
◦ PPTX animation is really BAD! (confusing, difficult to print, ...)
◦ latex/beamer creates sequences of pages for print -- same page number, many pages
◦ the point is ... animation should look good both in print and on screen
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 6/28
...
6/28
.
Objectives and Assumptions
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 7/28
...
7/28
.
Objectives and Assumptions
.
YALMS should be able...
..
.... to retain automation capability in fuzzy environments
• ASSUMPTION 1: no rules for slide PDFs -- only the PDF format
itself
• ASSUMPTION 2: no rules for screen video -- black margins, wierd
sizes -- all OK
• ASSUMPTION 3: no additional information is requested,
annotation should happen with only (1) and (2)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 8/28
...
8/28
.
Tools and Actual Software
1. pdftk : a nice library for handing PDFs 07
2. ImageMagic : the best graphics library (basically free Photoshop) 08
3. VLC for screen capture, ffmpeg will also work
4. ffmpeg for video editing 06
5. all scripting is done in php -- suitable for both CLI and web server
07 "pdftk: The PDF toolkit" http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit (2013)
08 "ImageMagic Homepage" http://www.imagemagick.org/script/index.php (2013)
06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 9/28
...
9/28
.
Solutions
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 10/28
...
10/28
.
Solution 1: Simple Scene Detection
• the simplest way to detect scene detection is to extract all I-Frames 06
• creates more images than actual scene changes, but redundancy is not a
problem
$ ffmpeg -i screen.mp4 -vf select="eq(pict_type¥,I)"
-vsync 0 -an frames.%03d.png
$ stands for command prompt, all commands are single-line
06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 11/28
...
11/28
.
Solution 2: Trim the Black Margins
• deal with fuzzy margins and sizes using ImageMagic's
convert
• OK to add a small border because trim does not cut everything anyway
• force frame resize
$ convert -colorspace gray -bordercolor black
-border 2x2 -trim +repage -resize 1280x720!
frames.004.png frames.004b.png
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 12/28
...
12/28
.
Solution 2: Trim Performance
Nothing to
trim
Trim
success!
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 13/28
...
13/28
.
Solution 3: Bring to Common Size
• cannot compare without same size
$ convert -colorspace gray -bordercolor black
-border 8x5 -resize 1280x720! -density 300x300
-quality 100 pages.002.pdf pages.002.png
$ stands for command prompt, all commands are single-line
$ convert -colorspace gray -bordercolor black
-border 2x2 -trim +repage -resize 1280x720!
frames.004.png frames.004b.png
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 14/28
...
14/28
.
Solution 3: Size Automation
• easy to find page size in command line, so, automation OK
platypus@platypus-PC /local/yalms.content
$ identify -format "%w %h" frames.004.png
1280 720
platypus@platypus-PC /local/yalms.content
$ identify -format "%w %h" frames.004c.png
1156 720
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 15/28
...
15/28
.
Solution 4: Compare using the AE Metric
• AE metric: simply the absolute pixel diff between two images
• alternatives: PNSR, MLSE ...
$ compare -colorspace gray -metric AE -fuzz 50%
pages.002.png frames.004b.png diff.png
7021
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 16/28
...
16/28
.
Solutions: Example
Video frame
Video frame
(trim, resize)
PDF page
(border, resize)
Diff image
AE: 7019
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 17/28
...
17/28
.
Parameters and Performance
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 18/28
...
18/28
.
Performance: Experimental Setup
• a 1.5h class
• screen capture and slide PDF
• fuzzy: 16:10 screen size (black margin)
• fuzzy: many ALT-TAB switches between applications
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 19/28
...
19/28
.
Performance: Parameters and Metrics
• fuss: used by ffmpeg as comparison threshold
• pace : minimum frequency of page changes -- possible to remove jitter
• performance: annotation success
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 20/28
...
20/28
.
Performance: Frame Diffs
• generally, good
trend, because
only one frame
matches well
• not ideal performance
0 20 40 60 80 100
Ordered list of frame diffs
1.2
2.4
3.6
4.8
6
log(framediff)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 21/28
...
21/28
.
Performance: Results
• thickness:
fuss value
(between 10% and 90%)
• changing fuss can help
sometimes
• some cases are
persistent
failures
0 2 4 6 8 10 12 14 16 18
Page sequence
0
0.5
1
1.5
2
2.5
3
3.5
log(timediff)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 22/28
...
22/28
.
Performance: Room for Improvement?
• multi-fuss comparison, pick the best
• when failing consistently, just set marker between two neighboring
successes
◦ mark it as failed and ask for human
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 23/28
...
23/28
.
YALMS.annotation: The Robot
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 24/28
...
24/28
.
YALMS.annotation : The Robot
screen
video
Slides
PDF
LaTeX/Beamer
Keynote
PowerPoint
…
VLC
ffmpeg
I-Frames I-Frames
pages PNGs
ANNOTATION
slices
PNGs
timestamps
End product
scenes cleanup
split
pdf2png
03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 25/28
...
25/28
.
YALMS.annotation : The Script
$ php robot.php
PURPOSE: to process two files: SLIDES PDF and
SCREEN CAPTURE MOVIE
OUTCOME: ANNOTATIONS for the movie -- times for each
PDF page in the movie
[pace] minimum time gap between page changes
-- will ignore fickers below that threshold
[wdir] full path to working directory
[screen] filename of the screen capture movie
-- should be in $wdir
[slides] filename of the slides PDF file -- should be in $wdir
$
$ stands for command prompt, all commands are single-line
03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 26/28
...
26/28
.
YALMS: The Roadmap
YALMS: Intro
YALMS.annotation
2013.07・ET研
2013.09・ET研
YALMS.streaming
2014.01・ET研
YALMS.hiperf
2013.10・ITS-MMS研
ET研
その他
YALMS.cloud
2013.11・IN研
YALMS.format
YALMS.done!
DASH Adaptive HTML5
PDF slices
VM sync groupdrive
SSD
multicore
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 27/28
...
27/28
.
That’s all, thank you ...
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28
...
28/28
.
[01] (2013)
The enPiT Project
http://www.enpit.jp
[02] (2013)
YALMS Project Home
https://github.com/maratishe/yalms
[03] (2013)
YALMS Annotation Robot (software behind this paper)
https://github.com/maratishe/yalms.annotation
[04] M.Zhanikeev+1 (2013.09)
YALMS: Yet Another Lecture Management System...
IEICE ET研
[05] (2013)
Homepage of edubase Stream at NII
http://stream.edubase.jp
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28
...
28/28
.
[06] (2013)
Homepage of ffmpeg and ffserver tools
http://www.ffmpeg.org
[07] (2013)
pdftk: The PDF toolkit
http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit
[08] (2013)
ImageMagic Homepage
http://www.imagemagick.org/script/index.php
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28
...
28/28

More Related Content

Similar to Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs

Practical Groovy Domain-Specific Languages
Practical Groovy Domain-Specific LanguagesPractical Groovy Domain-Specific Languages
Practical Groovy Domain-Specific LanguagesGuillaume Laforge
 
4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...
4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...
4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...Youness Lahdili
 
Linux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADTLinux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADTSchlomo Schapiro
 
Level Up With Parallelism in FME Server
Level Up With Parallelism in FME ServerLevel Up With Parallelism in FME Server
Level Up With Parallelism in FME ServerSafe Software
 
Gatling - JUGL, 2012-09-13
Gatling  - JUGL, 2012-09-13Gatling  - JUGL, 2012-09-13
Gatling - JUGL, 2012-09-13Nicolas Rémond
 
Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012slandelle
 
JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...
JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...
JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...Andrey Sadovykh
 
Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...
Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...
Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...AugmentedWorldExpo
 
Gui Input Tools for Math [UKMC09]
Gui Input Tools for Math [UKMC09]Gui Input Tools for Math [UKMC09]
Gui Input Tools for Math [UKMC09]Greg TAPPERO
 
Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...
Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...
Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...Amazon Web Services
 
Test Driven Development: Part 2
Test Driven Development: Part 2Test Driven Development: Part 2
Test Driven Development: Part 2CodeAndroid
 
Clientside/Offline (onefile) Lecture Player in a Web Browser
Clientside/Offline (onefile) Lecture Player in a Web BrowserClientside/Offline (onefile) Lecture Player in a Web Browser
Clientside/Offline (onefile) Lecture Player in a Web BrowserTokyo University of Science
 
Video Processing Thesis MATLAB Projects
Video Processing Thesis MATLAB ProjectsVideo Processing Thesis MATLAB Projects
Video Processing Thesis MATLAB ProjectsPhdtopiccom
 
Solid Edge Simulation From Plm World 2009
Solid Edge Simulation From Plm World 2009Solid Edge Simulation From Plm World 2009
Solid Edge Simulation From Plm World 2009Mark Burhop
 
UcamX-Integr8tor YELO - English
UcamX-Integr8tor YELO - EnglishUcamX-Integr8tor YELO - English
UcamX-Integr8tor YELO - EnglishWerner Biliet
 
Resume-Rohit_Vijay_Bapat_December_2016
Resume-Rohit_Vijay_Bapat_December_2016Resume-Rohit_Vijay_Bapat_December_2016
Resume-Rohit_Vijay_Bapat_December_2016Rohit Bapat
 
Presentation1.2.pptx
Presentation1.2.pptxPresentation1.2.pptx
Presentation1.2.pptxpranaykusuma
 

Similar to Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs (20)

Practical Groovy Domain-Specific Languages
Practical Groovy Domain-Specific LanguagesPractical Groovy Domain-Specific Languages
Practical Groovy Domain-Specific Languages
 
4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...
4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...
4 - Simulation and analysis of different DCT techniques on MATLAB (presented ...
 
Linux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADTLinux tag 2013 Data Center Automation with YADT
Linux tag 2013 Data Center Automation with YADT
 
azw
azwazw
azw
 
AJ-ch01.intro Zeid.ppt
AJ-ch01.intro Zeid.pptAJ-ch01.intro Zeid.ppt
AJ-ch01.intro Zeid.ppt
 
Level Up With Parallelism in FME Server
Level Up With Parallelism in FME ServerLevel Up With Parallelism in FME Server
Level Up With Parallelism in FME Server
 
Gatling - JUGL, 2012-09-13
Gatling  - JUGL, 2012-09-13Gatling  - JUGL, 2012-09-13
Gatling - JUGL, 2012-09-13
 
Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012
 
JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...
JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...
JUNIPER: Towards Modeling Approach Enabling Efficient Platform for Heterogene...
 
Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...
Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...
Andreas Zeitler (Vuframe): Virtual & Augmented Business: How to Discover and ...
 
Gui Input Tools for Math [UKMC09]
Gui Input Tools for Math [UKMC09]Gui Input Tools for Math [UKMC09]
Gui Input Tools for Math [UKMC09]
 
Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...
Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...
Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimizati...
 
Test Driven Development: Part 2
Test Driven Development: Part 2Test Driven Development: Part 2
Test Driven Development: Part 2
 
Clientside/Offline (onefile) Lecture Player in a Web Browser
Clientside/Offline (onefile) Lecture Player in a Web BrowserClientside/Offline (onefile) Lecture Player in a Web Browser
Clientside/Offline (onefile) Lecture Player in a Web Browser
 
Video Processing Thesis MATLAB Projects
Video Processing Thesis MATLAB ProjectsVideo Processing Thesis MATLAB Projects
Video Processing Thesis MATLAB Projects
 
Solid Edge Simulation From Plm World 2009
Solid Edge Simulation From Plm World 2009Solid Edge Simulation From Plm World 2009
Solid Edge Simulation From Plm World 2009
 
UcamX-Integr8tor YELO - English
UcamX-Integr8tor YELO - EnglishUcamX-Integr8tor YELO - English
UcamX-Integr8tor YELO - English
 
Resume-Rohit_Vijay_Bapat_December_2016
Resume-Rohit_Vijay_Bapat_December_2016Resume-Rohit_Vijay_Bapat_December_2016
Resume-Rohit_Vijay_Bapat_December_2016
 
TS-1679 Excerpt
TS-1679 ExcerptTS-1679 Excerpt
TS-1679 Excerpt
 
Presentation1.2.pptx
Presentation1.2.pptxPresentation1.2.pptx
Presentation1.2.pptx
 

More from Tokyo University of Science

A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...
A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...
A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...Tokyo University of Science
 
Ultrasound Relative Positioning for IoT Devices in Dense Wireless Spaces
Ultrasound Relative Positioning for IoT Devices in Dense Wireless SpacesUltrasound Relative Positioning for IoT Devices in Dense Wireless Spaces
Ultrasound Relative Positioning for IoT Devices in Dense Wireless SpacesTokyo University of Science
 
Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...
Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...
Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...Tokyo University of Science
 
What if We Atomize Student Data and Apps and Put Them on Docker Containers?
What if We Atomize Student Data and Apps and Put Them on Docker Containers?What if We Atomize Student Data and Apps and Put Them on Docker Containers?
What if We Atomize Student Data and Apps and Put Them on Docker Containers?Tokyo University of Science
 
Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...
Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...
Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...Tokyo University of Science
 
On Performance Under Hotspots in Hadoop versus Bigdata Replay Platforms
On Performance Under Hotspots in Hadoop versus Bigdata Replay PlatformsOn Performance Under Hotspots in Hadoop versus Bigdata Replay Platforms
On Performance Under Hotspots in Hadoop versus Bigdata Replay PlatformsTokyo University of Science
 
Taking the Step from Software to Product Development \\ when teaching PBL at ...
Taking the Step from Software to Product Development \\ when teaching PBL at ...Taking the Step from Software to Product Development \\ when teaching PBL at ...
Taking the Step from Software to Product Development \\ when teaching PBL at ...Tokyo University of Science
 
Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...
Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...
Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...Tokyo University of Science
 
The Switchboard Optimization Problem and Heuristics for Cut-Through Networking
The Switchboard Optimization Problem and Heuristics for Cut-Through NetworkingThe Switchboard Optimization Problem and Heuristics for Cut-Through Networking
The Switchboard Optimization Problem and Heuristics for Cut-Through NetworkingTokyo University of Science
 
The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...
The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...
The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...Tokyo University of Science
 
Bulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless Spaces
Bulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless SpacesBulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless Spaces
Bulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless SpacesTokyo University of Science
 
Fog Cloud Caching at Network Edge via Local Hardware Awareness Spaces
Fog Cloud Caching at Network Edge via Local Hardware Awareness SpacesFog Cloud Caching at Network Edge via Local Hardware Awareness Spaces
Fog Cloud Caching at Network Edge via Local Hardware Awareness SpacesTokyo University of Science
 
On a Hybrid Packets-and-Circuits Switching Logic
On a Hybrid Packets-and-Circuits Switching LogicOn a Hybrid Packets-and-Circuits Switching Logic
On a Hybrid Packets-and-Circuits Switching LogicTokyo University of Science
 
Image-Related Uses for Roadside Infrastructure \\ based on Wireless Beacons
Image-Related Uses for Roadside Infrastructure \\ based on Wireless BeaconsImage-Related Uses for Roadside Infrastructure \\ based on Wireless Beacons
Image-Related Uses for Roadside Infrastructure \\ based on Wireless BeaconsTokyo University of Science
 
Complexity Resolution Control for Context Based on Metromaps
Complexity Resolution Control for Context Based on MetromapsComplexity Resolution Control for Context Based on Metromaps
Complexity Resolution Control for Context Based on MetromapsTokyo University of Science
 
The Declarative-Coordinated Model for Self-Optimization of Service Networks
The Declarative-Coordinated Model for Self-Optimization of Service NetworksThe Declarative-Coordinated Model for Self-Optimization of Service Networks
The Declarative-Coordinated Model for Self-Optimization of Service NetworksTokyo University of Science
 
3-Way Scripts as a Practical Platform for Secure Distributed Code in Clouds
3-Way Scripts as a Practical Platform for Secure Distributed Code in Clouds3-Way Scripts as a Practical Platform for Secure Distributed Code in Clouds
3-Way Scripts as a Practical Platform for Secure Distributed Code in CloudsTokyo University of Science
 
3-Way Scripts as a Base Unit for Flexible Scale-Out Code
3-Way Scripts as a Base Unit for Flexible Scale-Out Code3-Way Scripts as a Base Unit for Flexible Scale-Out Code
3-Way Scripts as a Base Unit for Flexible Scale-Out CodeTokyo University of Science
 
Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback
Towards Social Robotics on Smartphones with Simple XYZV Sensor FeedbackTowards Social Robotics on Smartphones with Simple XYZV Sensor Feedback
Towards Social Robotics on Smartphones with Simple XYZV Sensor FeedbackTokyo University of Science
 
Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...
Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...
Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...Tokyo University of Science
 

More from Tokyo University of Science (20)

A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...
A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...
A Method for Cloud-Assisted Secure Wireless Grouping of Client Devices at Net...
 
Ultrasound Relative Positioning for IoT Devices in Dense Wireless Spaces
Ultrasound Relative Positioning for IoT Devices in Dense Wireless SpacesUltrasound Relative Positioning for IoT Devices in Dense Wireless Spaces
Ultrasound Relative Positioning for IoT Devices in Dense Wireless Spaces
 
Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...
Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...
Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Tra...
 
What if We Atomize Student Data and Apps and Put Them on Docker Containers?
What if We Atomize Student Data and Apps and Put Them on Docker Containers?What if We Atomize Student Data and Apps and Put Them on Docker Containers?
What if We Atomize Student Data and Apps and Put Them on Docker Containers?
 
Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...
Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...
Large-Scale Crowdsourcing by Vehicular Data Packets in a Sparse Roadside Infr...
 
On Performance Under Hotspots in Hadoop versus Bigdata Replay Platforms
On Performance Under Hotspots in Hadoop versus Bigdata Replay PlatformsOn Performance Under Hotspots in Hadoop versus Bigdata Replay Platforms
On Performance Under Hotspots in Hadoop versus Bigdata Replay Platforms
 
Taking the Step from Software to Product Development \\ when teaching PBL at ...
Taking the Step from Software to Product Development \\ when teaching PBL at ...Taking the Step from Software to Product Development \\ when teaching PBL at ...
Taking the Step from Software to Product Development \\ when teaching PBL at ...
 
Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...
Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...
Design and Implementation of a 3-Party Cloud-Backed Handshake for Secure Grou...
 
The Switchboard Optimization Problem and Heuristics for Cut-Through Networking
The Switchboard Optimization Problem and Heuristics for Cut-Through NetworkingThe Switchboard Optimization Problem and Heuristics for Cut-Through Networking
The Switchboard Optimization Problem and Heuristics for Cut-Through Networking
 
The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...
The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...
The Switchboard Traffic Engineering Problem for Mixed Contention/Cut-Through ...
 
Bulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless Spaces
Bulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless SpacesBulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless Spaces
Bulk-n-Pick Method for One-to-Many Data Transfer in Dense Wireless Spaces
 
Fog Cloud Caching at Network Edge via Local Hardware Awareness Spaces
Fog Cloud Caching at Network Edge via Local Hardware Awareness SpacesFog Cloud Caching at Network Edge via Local Hardware Awareness Spaces
Fog Cloud Caching at Network Edge via Local Hardware Awareness Spaces
 
On a Hybrid Packets-and-Circuits Switching Logic
On a Hybrid Packets-and-Circuits Switching LogicOn a Hybrid Packets-and-Circuits Switching Logic
On a Hybrid Packets-and-Circuits Switching Logic
 
Image-Related Uses for Roadside Infrastructure \\ based on Wireless Beacons
Image-Related Uses for Roadside Infrastructure \\ based on Wireless BeaconsImage-Related Uses for Roadside Infrastructure \\ based on Wireless Beacons
Image-Related Uses for Roadside Infrastructure \\ based on Wireless Beacons
 
Complexity Resolution Control for Context Based on Metromaps
Complexity Resolution Control for Context Based on MetromapsComplexity Resolution Control for Context Based on Metromaps
Complexity Resolution Control for Context Based on Metromaps
 
The Declarative-Coordinated Model for Self-Optimization of Service Networks
The Declarative-Coordinated Model for Self-Optimization of Service NetworksThe Declarative-Coordinated Model for Self-Optimization of Service Networks
The Declarative-Coordinated Model for Self-Optimization of Service Networks
 
3-Way Scripts as a Practical Platform for Secure Distributed Code in Clouds
3-Way Scripts as a Practical Platform for Secure Distributed Code in Clouds3-Way Scripts as a Practical Platform for Secure Distributed Code in Clouds
3-Way Scripts as a Practical Platform for Secure Distributed Code in Clouds
 
3-Way Scripts as a Base Unit for Flexible Scale-Out Code
3-Way Scripts as a Base Unit for Flexible Scale-Out Code3-Way Scripts as a Base Unit for Flexible Scale-Out Code
3-Way Scripts as a Base Unit for Flexible Scale-Out Code
 
Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback
Towards Social Robotics on Smartphones with Simple XYZV Sensor FeedbackTowards Social Robotics on Smartphones with Simple XYZV Sensor Feedback
Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback
 
Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...
Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...
Back to Rings but not Tokens: Physical and Logical Designs for Distributed Fi...
 

Recently uploaded

RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 

Recently uploaded (20)

RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 

Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs

  • 2. . YALMS: Review . YALMS.... .. ....stands for Yet Another Lecture Management System • basically, a class robot ◦ robot: set of automation scripts and frameworks • first presented in ET研2013.07 04 • a long-term project with 4-5 main components -- see the roadmap at the end 04 M.Zhanikeev+1 "YALMS: Yet Another Lecture Management System..." IEICE ET研 (2013.09) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 2/28 ... 2/28
  • 3. . YALMS: Overall Design Step 1: Record and Stream V A S Lecturer Viewers Content Server HTML5 HTML5 Muxer RecordScreen Streaming Step 2: Annotate Lecturer S Auto- annotate C M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 3/28 ... 3/28
  • 4. . YALMS: Annotation VIDEO VAS Annotate Pack/Encode Screen VIDEO VAS VA Realtime Screen VIDEO S Mux A V MISC. C C PNG Replay Screen VIDEO A V MISC. C Annotate S S S The focus of this study YALMS Robot S M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 4/28 ... 4/28
  • 5. . Fuzzy Environments M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 5/28 ... 5/28
  • 6. . Sources of Fuzziness 1. unusual screen sizes in presentation mode ◦ this PDF was created in 16:9mode to fit my notebook perfectly, will NOT fit others 2. unpredictable or unavoidable margins when saving your slides as PDF ◦ PPTX! no way to avoid white margins in print ◦ tex2pdf creates perfect PDFs! --- look the same in PDF and in presentation mode 3. slide animation ◦ PPTX animation is really BAD! (confusing, difficult to print, ...) ◦ latex/beamer creates sequences of pages for print -- same page number, many pages ◦ the point is ... animation should look good both in print and on screen M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 6/28 ... 6/28
  • 7. . Objectives and Assumptions M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 7/28 ... 7/28
  • 8. . Objectives and Assumptions . YALMS should be able... .. .... to retain automation capability in fuzzy environments • ASSUMPTION 1: no rules for slide PDFs -- only the PDF format itself • ASSUMPTION 2: no rules for screen video -- black margins, wierd sizes -- all OK • ASSUMPTION 3: no additional information is requested, annotation should happen with only (1) and (2) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 8/28 ... 8/28
  • 9. . Tools and Actual Software 1. pdftk : a nice library for handing PDFs 07 2. ImageMagic : the best graphics library (basically free Photoshop) 08 3. VLC for screen capture, ffmpeg will also work 4. ffmpeg for video editing 06 5. all scripting is done in php -- suitable for both CLI and web server 07 "pdftk: The PDF toolkit" http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit (2013) 08 "ImageMagic Homepage" http://www.imagemagick.org/script/index.php (2013) 06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 9/28 ... 9/28
  • 10. . Solutions M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 10/28 ... 10/28
  • 11. . Solution 1: Simple Scene Detection • the simplest way to detect scene detection is to extract all I-Frames 06 • creates more images than actual scene changes, but redundancy is not a problem $ ffmpeg -i screen.mp4 -vf select="eq(pict_type¥,I)" -vsync 0 -an frames.%03d.png $ stands for command prompt, all commands are single-line 06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 11/28 ... 11/28
  • 12. . Solution 2: Trim the Black Margins • deal with fuzzy margins and sizes using ImageMagic's convert • OK to add a small border because trim does not cut everything anyway • force frame resize $ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 12/28 ... 12/28
  • 13. . Solution 2: Trim Performance Nothing to trim Trim success! M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 13/28 ... 13/28
  • 14. . Solution 3: Bring to Common Size • cannot compare without same size $ convert -colorspace gray -bordercolor black -border 8x5 -resize 1280x720! -density 300x300 -quality 100 pages.002.pdf pages.002.png $ stands for command prompt, all commands are single-line $ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 14/28 ... 14/28
  • 15. . Solution 3: Size Automation • easy to find page size in command line, so, automation OK platypus@platypus-PC /local/yalms.content $ identify -format "%w %h" frames.004.png 1280 720 platypus@platypus-PC /local/yalms.content $ identify -format "%w %h" frames.004c.png 1156 720 $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 15/28 ... 15/28
  • 16. . Solution 4: Compare using the AE Metric • AE metric: simply the absolute pixel diff between two images • alternatives: PNSR, MLSE ... $ compare -colorspace gray -metric AE -fuzz 50% pages.002.png frames.004b.png diff.png 7021 $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 16/28 ... 16/28
  • 17. . Solutions: Example Video frame Video frame (trim, resize) PDF page (border, resize) Diff image AE: 7019 M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 17/28 ... 17/28
  • 18. . Parameters and Performance M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 18/28 ... 18/28
  • 19. . Performance: Experimental Setup • a 1.5h class • screen capture and slide PDF • fuzzy: 16:10 screen size (black margin) • fuzzy: many ALT-TAB switches between applications M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 19/28 ... 19/28
  • 20. . Performance: Parameters and Metrics • fuss: used by ffmpeg as comparison threshold • pace : minimum frequency of page changes -- possible to remove jitter • performance: annotation success M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 20/28 ... 20/28
  • 21. . Performance: Frame Diffs • generally, good trend, because only one frame matches well • not ideal performance 0 20 40 60 80 100 Ordered list of frame diffs 1.2 2.4 3.6 4.8 6 log(framediff) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 21/28 ... 21/28
  • 22. . Performance: Results • thickness: fuss value (between 10% and 90%) • changing fuss can help sometimes • some cases are persistent failures 0 2 4 6 8 10 12 14 16 18 Page sequence 0 0.5 1 1.5 2 2.5 3 3.5 log(timediff) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 22/28 ... 22/28
  • 23. . Performance: Room for Improvement? • multi-fuss comparison, pick the best • when failing consistently, just set marker between two neighboring successes ◦ mark it as failed and ask for human M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 23/28 ... 23/28
  • 24. . YALMS.annotation: The Robot M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 24/28 ... 24/28
  • 25. . YALMS.annotation : The Robot screen video Slides PDF LaTeX/Beamer Keynote PowerPoint … VLC ffmpeg I-Frames I-Frames pages PNGs ANNOTATION slices PNGs timestamps End product scenes cleanup split pdf2png 03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 25/28 ... 25/28
  • 26. . YALMS.annotation : The Script $ php robot.php PURPOSE: to process two files: SLIDES PDF and SCREEN CAPTURE MOVIE OUTCOME: ANNOTATIONS for the movie -- times for each PDF page in the movie [pace] minimum time gap between page changes -- will ignore fickers below that threshold [wdir] full path to working directory [screen] filename of the screen capture movie -- should be in $wdir [slides] filename of the slides PDF file -- should be in $wdir $ $ stands for command prompt, all commands are single-line 03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 26/28 ... 26/28
  • 27. . YALMS: The Roadmap YALMS: Intro YALMS.annotation 2013.07・ET研 2013.09・ET研 YALMS.streaming 2014.01・ET研 YALMS.hiperf 2013.10・ITS-MMS研 ET研 その他 YALMS.cloud 2013.11・IN研 YALMS.format YALMS.done! DASH Adaptive HTML5 PDF slices VM sync groupdrive SSD multicore M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 27/28 ... 27/28
  • 28. . That’s all, thank you ... M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28
  • 29. . [01] (2013) The enPiT Project http://www.enpit.jp [02] (2013) YALMS Project Home https://github.com/maratishe/yalms [03] (2013) YALMS Annotation Robot (software behind this paper) https://github.com/maratishe/yalms.annotation [04] M.Zhanikeev+1 (2013.09) YALMS: Yet Another Lecture Management System... IEICE ET研 [05] (2013) Homepage of edubase Stream at NII http://stream.edubase.jp M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28
  • 30. . [06] (2013) Homepage of ffmpeg and ffserver tools http://www.ffmpeg.org [07] (2013) pdftk: The PDF toolkit http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit [08] (2013) ImageMagic Homepage http://www.imagemagick.org/script/index.php M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28