.
YALMS: Review
.
YALMS....
..
....stands for Yet Another Lecture Management System
• basically, a class robot
◦ robot: se...
.
YALMS: Overall Design
Step 1: Record and Stream
V
A
S
Lecturer
Viewers
Content
Server
HTML5
HTML5
Muxer
RecordScreen
Str...
.
YALMS: Annotation
VIDEO
VAS
Annotate
Pack/Encode
Screen
VIDEO
VAS
VA
Realtime Screen
VIDEO
S
Mux
A V
MISC. C
C
PNG
Repla...
.
Fuzzy Environments
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot...
.
Sources of Fuzziness
1. unusual screen sizes in presentation mode
◦ this PDF was created in 16:9mode to fit my notebook ...
.
Objectives and Assumptions
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotati...
.
Objectives and Assumptions
.
YALMS should be able...
..
.... to retain automation capability in fuzzy environments
• ASS...
.
Tools and Actual Software
1. pdftk : a nice library for handing PDFs 07
2. ImageMagic : the best graphics library (basic...
.
Solutions
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 10/28
.....
.
Solution 1: Simple Scene Detection
• the simplest way to detect scene detection is to extract all I-Frames 06
• creates ...
.
Solution 2: Trim the Black Margins
• deal with fuzzy margins and sizes using ImageMagic's
convert
• OK to add a small bo...
.
Solution 2: Trim Performance
Nothing to
trim
Trim
success!
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Alg...
.
Solution 3: Bring to Common Size
• cannot compare without same size
$ convert -colorspace gray -bordercolor black
-borde...
.
Solution 3: Size Automation
• easy to find page size in command line, so, automation OK
platypus@platypus-PC /local/yalm...
.
Solution 4: Compare using the AE Metric
• AE metric: simply the absolute pixel diff between two images
• alternatives: P...
.
Solutions: Example
Video frame
Video frame
(trim, resize)
PDF page
(border, resize)
Diff image
AE: 7019
M.Zhanikeev/H.Koi...
.
Parameters and Performance
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotati...
.
Performance: Experimental Setup
• a 1.5h class
• screen capture and slide PDF
• fuzzy: 16:10 screen size (black margin)
...
.
Performance: Parameters and Metrics
• fuss: used by ffmpeg as comparison threshold
• pace : minimum frequency of page ch...
.
Performance: Frame Diffs
• generally, good
trend, because
only one frame
matches well
• not ideal performance
0 20 40 60...
.
Performance: Results
• thickness:
fuss value
(between 10% and 90%)
• changing fuss can help
sometimes
• some cases are
p...
.
Performance: Room for Improvement?
• multi-fuss comparison, pick the best
• when failing consistently, just set marker b...
.
YALMS.annotation: The Robot
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotat...
.
YALMS.annotation : The Robot
screen
video
Slides
PDF
LaTeX/Beamer
Keynote
PowerPoint
…
VLC
ffmpeg
I-Frames I-Frames
page...
.
YALMS.annotation : The Script
$ php robot.php
PURPOSE: to process two files: SLIDES PDF and
SCREEN CAPTURE MOVIE
OUTCOME...
.
YALMS: The Roadmap
YALMS: Intro
YALMS.annotation
2013.07・ET研
2013.09・ET研
YALMS.streaming
2014.01・ET研
YALMS.hiperf
2013.1...
.
That’s all, thank you ...
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotatio...
.
[01] (2013)
The enPiT Project
http://www.enpit.jp
[02] (2013)
YALMS Project Home
https://github.com/maratishe/yalms
[03]...
.
[06] (2013)
Homepage of ffmpeg and ffserver tools
http://www.ffmpeg.org
[07] (2013)
pdftk: The PDF toolkit
http://www.pd...
Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs
Upcoming SlideShare
Loading in …5
×

Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs

740 views

Published on

We start with a PDF file with presentation slides. We capture the video of the screen (screen streaming) while making an arbitrarily long presentation and store it in a file. We also keep the original PDF file. Then, the problem this paper attempts to solve is how to find frames in the video which correspond to transitions between pages in the slides during presentation. The method needs to be automatic and robust to complex transitions like going backwards, switching between applications, etc. This process is referred to as video slicing or video annotation and has many practical uses. The main intended uses are (1) automatic video slicing and (2) creating short sketches of longer videos.

Published in: Technology, Art & Photos
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
740
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs

  1. 1. . YALMS: Review . YALMS.... .. ....stands for Yet Another Lecture Management System • basically, a class robot ◦ robot: set of automation scripts and frameworks • first presented in ET研2013.07 04 • a long-term project with 4-5 main components -- see the roadmap at the end 04 M.Zhanikeev+1 "YALMS: Yet Another Lecture Management System..." IEICE ET研 (2013.09) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 2/28 ... 2/28
  2. 2. . YALMS: Overall Design Step 1: Record and Stream V A S Lecturer Viewers Content Server HTML5 HTML5 Muxer RecordScreen Streaming Step 2: Annotate Lecturer S Auto- annotate C M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 3/28 ... 3/28
  3. 3. . YALMS: Annotation VIDEO VAS Annotate Pack/Encode Screen VIDEO VAS VA Realtime Screen VIDEO S Mux A V MISC. C C PNG Replay Screen VIDEO A V MISC. C Annotate S S S The focus of this study YALMS Robot S M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 4/28 ... 4/28
  4. 4. . Fuzzy Environments M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 5/28 ... 5/28
  5. 5. . Sources of Fuzziness 1. unusual screen sizes in presentation mode ◦ this PDF was created in 16:9mode to fit my notebook perfectly, will NOT fit others 2. unpredictable or unavoidable margins when saving your slides as PDF ◦ PPTX! no way to avoid white margins in print ◦ tex2pdf creates perfect PDFs! --- look the same in PDF and in presentation mode 3. slide animation ◦ PPTX animation is really BAD! (confusing, difficult to print, ...) ◦ latex/beamer creates sequences of pages for print -- same page number, many pages ◦ the point is ... animation should look good both in print and on screen M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 6/28 ... 6/28
  6. 6. . Objectives and Assumptions M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 7/28 ... 7/28
  7. 7. . Objectives and Assumptions . YALMS should be able... .. .... to retain automation capability in fuzzy environments • ASSUMPTION 1: no rules for slide PDFs -- only the PDF format itself • ASSUMPTION 2: no rules for screen video -- black margins, wierd sizes -- all OK • ASSUMPTION 3: no additional information is requested, annotation should happen with only (1) and (2) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 8/28 ... 8/28
  8. 8. . Tools and Actual Software 1. pdftk : a nice library for handing PDFs 07 2. ImageMagic : the best graphics library (basically free Photoshop) 08 3. VLC for screen capture, ffmpeg will also work 4. ffmpeg for video editing 06 5. all scripting is done in php -- suitable for both CLI and web server 07 "pdftk: The PDF toolkit" http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit (2013) 08 "ImageMagic Homepage" http://www.imagemagick.org/script/index.php (2013) 06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 9/28 ... 9/28
  9. 9. . Solutions M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 10/28 ... 10/28
  10. 10. . Solution 1: Simple Scene Detection • the simplest way to detect scene detection is to extract all I-Frames 06 • creates more images than actual scene changes, but redundancy is not a problem $ ffmpeg -i screen.mp4 -vf select="eq(pict_type¥,I)" -vsync 0 -an frames.%03d.png $ stands for command prompt, all commands are single-line 06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 11/28 ... 11/28
  11. 11. . Solution 2: Trim the Black Margins • deal with fuzzy margins and sizes using ImageMagic's convert • OK to add a small border because trim does not cut everything anyway • force frame resize $ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 12/28 ... 12/28
  12. 12. . Solution 2: Trim Performance Nothing to trim Trim success! M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 13/28 ... 13/28
  13. 13. . Solution 3: Bring to Common Size • cannot compare without same size $ convert -colorspace gray -bordercolor black -border 8x5 -resize 1280x720! -density 300x300 -quality 100 pages.002.pdf pages.002.png $ stands for command prompt, all commands are single-line $ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 14/28 ... 14/28
  14. 14. . Solution 3: Size Automation • easy to find page size in command line, so, automation OK platypus@platypus-PC /local/yalms.content $ identify -format "%w %h" frames.004.png 1280 720 platypus@platypus-PC /local/yalms.content $ identify -format "%w %h" frames.004c.png 1156 720 $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 15/28 ... 15/28
  15. 15. . Solution 4: Compare using the AE Metric • AE metric: simply the absolute pixel diff between two images • alternatives: PNSR, MLSE ... $ compare -colorspace gray -metric AE -fuzz 50% pages.002.png frames.004b.png diff.png 7021 $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 16/28 ... 16/28
  16. 16. . Solutions: Example Video frame Video frame (trim, resize) PDF page (border, resize) Diff image AE: 7019 M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 17/28 ... 17/28
  17. 17. . Parameters and Performance M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 18/28 ... 18/28
  18. 18. . Performance: Experimental Setup • a 1.5h class • screen capture and slide PDF • fuzzy: 16:10 screen size (black margin) • fuzzy: many ALT-TAB switches between applications M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 19/28 ... 19/28
  19. 19. . Performance: Parameters and Metrics • fuss: used by ffmpeg as comparison threshold • pace : minimum frequency of page changes -- possible to remove jitter • performance: annotation success M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 20/28 ... 20/28
  20. 20. . Performance: Frame Diffs • generally, good trend, because only one frame matches well • not ideal performance 0 20 40 60 80 100 Ordered list of frame diffs 1.2 2.4 3.6 4.8 6 log(framediff) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 21/28 ... 21/28
  21. 21. . Performance: Results • thickness: fuss value (between 10% and 90%) • changing fuss can help sometimes • some cases are persistent failures 0 2 4 6 8 10 12 14 16 18 Page sequence 0 0.5 1 1.5 2 2.5 3 3.5 log(timediff) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 22/28 ... 22/28
  22. 22. . Performance: Room for Improvement? • multi-fuss comparison, pick the best • when failing consistently, just set marker between two neighboring successes ◦ mark it as failed and ask for human M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 23/28 ... 23/28
  23. 23. . YALMS.annotation: The Robot M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 24/28 ... 24/28
  24. 24. . YALMS.annotation : The Robot screen video Slides PDF LaTeX/Beamer Keynote PowerPoint … VLC ffmpeg I-Frames I-Frames pages PNGs ANNOTATION slices PNGs timestamps End product scenes cleanup split pdf2png 03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 25/28 ... 25/28
  25. 25. . YALMS.annotation : The Script $ php robot.php PURPOSE: to process two files: SLIDES PDF and SCREEN CAPTURE MOVIE OUTCOME: ANNOTATIONS for the movie -- times for each PDF page in the movie [pace] minimum time gap between page changes -- will ignore fickers below that threshold [wdir] full path to working directory [screen] filename of the screen capture movie -- should be in $wdir [slides] filename of the slides PDF file -- should be in $wdir $ $ stands for command prompt, all commands are single-line 03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 26/28 ... 26/28
  26. 26. . YALMS: The Roadmap YALMS: Intro YALMS.annotation 2013.07・ET研 2013.09・ET研 YALMS.streaming 2014.01・ET研 YALMS.hiperf 2013.10・ITS-MMS研 ET研 その他 YALMS.cloud 2013.11・IN研 YALMS.format YALMS.done! DASH Adaptive HTML5 PDF slices VM sync groupdrive SSD multicore M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 27/28 ... 27/28
  27. 27. . That’s all, thank you ... M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28
  28. 28. . [01] (2013) The enPiT Project http://www.enpit.jp [02] (2013) YALMS Project Home https://github.com/maratishe/yalms [03] (2013) YALMS Annotation Robot (software behind this paper) https://github.com/maratishe/yalms.annotation [04] M.Zhanikeev+1 (2013.09) YALMS: Yet Another Lecture Management System... IEICE ET研 [05] (2013) Homepage of edubase Stream at NII http://stream.edubase.jp M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28
  29. 29. . [06] (2013) Homepage of ffmpeg and ffserver tools http://www.ffmpeg.org [07] (2013) pdftk: The PDF toolkit http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit [08] (2013) ImageMagic Homepage http://www.imagemagick.org/script/index.php M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28

×