Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs

  • 522 views
Uploaded on

We start with a PDF file with presentation slides. We capture the video of the screen (screen streaming) while making an arbitrarily long presentation and store it in a file. We also keep the......

We start with a PDF file with presentation slides. We capture the video of the screen (screen streaming) while making an arbitrarily long presentation and store it in a file. We also keep the original PDF file. Then, the problem this paper attempts to solve is how to find frames in the video which correspond to transitions between pages in the slides during presentation. The method needs to be automatic and robust to complex transitions like going backwards, switching between applications, etc. This process is referred to as video slicing or video annotation and has many practical uses. The main intended uses are (1) automatic video slicing and (2) creating short sketches of longer videos.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
522
On Slideshare
522
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. . YALMS: Review . YALMS.... .. ....stands for Yet Another Lecture Management System • basically, a class robot ◦ robot: set of automation scripts and frameworks • first presented in ET研2013.07 04 • a long-term project with 4-5 main components -- see the roadmap at the end 04 M.Zhanikeev+1 "YALMS: Yet Another Lecture Management System..." IEICE ET研 (2013.09) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 2/28 ... 2/28
  • 2. . YALMS: Overall Design Step 1: Record and Stream V A S Lecturer Viewers Content Server HTML5 HTML5 Muxer RecordScreen Streaming Step 2: Annotate Lecturer S Auto- annotate C M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 3/28 ... 3/28
  • 3. . YALMS: Annotation VIDEO VAS Annotate Pack/Encode Screen VIDEO VAS VA Realtime Screen VIDEO S Mux A V MISC. C C PNG Replay Screen VIDEO A V MISC. C Annotate S S S The focus of this study YALMS Robot S M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 4/28 ... 4/28
  • 4. . Fuzzy Environments M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 5/28 ... 5/28
  • 5. . Sources of Fuzziness 1. unusual screen sizes in presentation mode ◦ this PDF was created in 16:9mode to fit my notebook perfectly, will NOT fit others 2. unpredictable or unavoidable margins when saving your slides as PDF ◦ PPTX! no way to avoid white margins in print ◦ tex2pdf creates perfect PDFs! --- look the same in PDF and in presentation mode 3. slide animation ◦ PPTX animation is really BAD! (confusing, difficult to print, ...) ◦ latex/beamer creates sequences of pages for print -- same page number, many pages ◦ the point is ... animation should look good both in print and on screen M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 6/28 ... 6/28
  • 6. . Objectives and Assumptions M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 7/28 ... 7/28
  • 7. . Objectives and Assumptions . YALMS should be able... .. .... to retain automation capability in fuzzy environments • ASSUMPTION 1: no rules for slide PDFs -- only the PDF format itself • ASSUMPTION 2: no rules for screen video -- black margins, wierd sizes -- all OK • ASSUMPTION 3: no additional information is requested, annotation should happen with only (1) and (2) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 8/28 ... 8/28
  • 8. . Tools and Actual Software 1. pdftk : a nice library for handing PDFs 07 2. ImageMagic : the best graphics library (basically free Photoshop) 08 3. VLC for screen capture, ffmpeg will also work 4. ffmpeg for video editing 06 5. all scripting is done in php -- suitable for both CLI and web server 07 "pdftk: The PDF toolkit" http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit (2013) 08 "ImageMagic Homepage" http://www.imagemagick.org/script/index.php (2013) 06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 9/28 ... 9/28
  • 9. . Solutions M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 10/28 ... 10/28
  • 10. . Solution 1: Simple Scene Detection • the simplest way to detect scene detection is to extract all I-Frames 06 • creates more images than actual scene changes, but redundancy is not a problem $ ffmpeg -i screen.mp4 -vf select="eq(pict_type¥,I)" -vsync 0 -an frames.%03d.png $ stands for command prompt, all commands are single-line 06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 11/28 ... 11/28
  • 11. . Solution 2: Trim the Black Margins • deal with fuzzy margins and sizes using ImageMagic's convert • OK to add a small border because trim does not cut everything anyway • force frame resize $ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 12/28 ... 12/28
  • 12. . Solution 2: Trim Performance Nothing to trim Trim success! M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 13/28 ... 13/28
  • 13. . Solution 3: Bring to Common Size • cannot compare without same size $ convert -colorspace gray -bordercolor black -border 8x5 -resize 1280x720! -density 300x300 -quality 100 pages.002.pdf pages.002.png $ stands for command prompt, all commands are single-line $ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 14/28 ... 14/28
  • 14. . Solution 3: Size Automation • easy to find page size in command line, so, automation OK platypus@platypus-PC /local/yalms.content $ identify -format "%w %h" frames.004.png 1280 720 platypus@platypus-PC /local/yalms.content $ identify -format "%w %h" frames.004c.png 1156 720 $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 15/28 ... 15/28
  • 15. . Solution 4: Compare using the AE Metric • AE metric: simply the absolute pixel diff between two images • alternatives: PNSR, MLSE ... $ compare -colorspace gray -metric AE -fuzz 50% pages.002.png frames.004b.png diff.png 7021 $ stands for command prompt, all commands are single-line M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 16/28 ... 16/28
  • 16. . Solutions: Example Video frame Video frame (trim, resize) PDF page (border, resize) Diff image AE: 7019 M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 17/28 ... 17/28
  • 17. . Parameters and Performance M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 18/28 ... 18/28
  • 18. . Performance: Experimental Setup • a 1.5h class • screen capture and slide PDF • fuzzy: 16:10 screen size (black margin) • fuzzy: many ALT-TAB switches between applications M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 19/28 ... 19/28
  • 19. . Performance: Parameters and Metrics • fuss: used by ffmpeg as comparison threshold • pace : minimum frequency of page changes -- possible to remove jitter • performance: annotation success M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 20/28 ... 20/28
  • 20. . Performance: Frame Diffs • generally, good trend, because only one frame matches well • not ideal performance 0 20 40 60 80 100 Ordered list of frame diffs 1.2 2.4 3.6 4.8 6 log(framediff) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 21/28 ... 21/28
  • 21. . Performance: Results • thickness: fuss value (between 10% and 90%) • changing fuss can help sometimes • some cases are persistent failures 0 2 4 6 8 10 12 14 16 18 Page sequence 0 0.5 1 1.5 2 2.5 3 3.5 log(timediff) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 22/28 ... 22/28
  • 22. . Performance: Room for Improvement? • multi-fuss comparison, pick the best • when failing consistently, just set marker between two neighboring successes ◦ mark it as failed and ask for human M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 23/28 ... 23/28
  • 23. . YALMS.annotation: The Robot M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 24/28 ... 24/28
  • 24. . YALMS.annotation : The Robot screen video Slides PDF LaTeX/Beamer Keynote PowerPoint … VLC ffmpeg I-Frames I-Frames pages PNGs ANNOTATION slices PNGs timestamps End product scenes cleanup split pdf2png 03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 25/28 ... 25/28
  • 25. . YALMS.annotation : The Script $ php robot.php PURPOSE: to process two files: SLIDES PDF and SCREEN CAPTURE MOVIE OUTCOME: ANNOTATIONS for the movie -- times for each PDF page in the movie [pace] minimum time gap between page changes -- will ignore fickers below that threshold [wdir] full path to working directory [screen] filename of the screen capture movie -- should be in $wdir [slides] filename of the slides PDF file -- should be in $wdir $ $ stands for command prompt, all commands are single-line 03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013) M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 26/28 ... 26/28
  • 26. . YALMS: The Roadmap YALMS: Intro YALMS.annotation 2013.07・ET研 2013.09・ET研 YALMS.streaming 2014.01・ET研 YALMS.hiperf 2013.10・ITS-MMS研 ET研 その他 YALMS.cloud 2013.11・IN研 YALMS.format YALMS.done! DASH Adaptive HTML5 PDF slices VM sync groupdrive SSD multicore M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 27/28 ... 27/28
  • 27. . That’s all, thank you ... M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28
  • 28. . [01] (2013) The enPiT Project http://www.enpit.jp [02] (2013) YALMS Project Home https://github.com/maratishe/yalms [03] (2013) YALMS Annotation Robot (software behind this paper) https://github.com/maratishe/yalms.annotation [04] M.Zhanikeev+1 (2013.09) YALMS: Yet Another Lecture Management System... IEICE ET研 [05] (2013) Homepage of edubase Stream at NII http://stream.edubase.jp M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28
  • 29. . [06] (2013) Homepage of ffmpeg and ffserver tools http://www.ffmpeg.org [07] (2013) pdftk: The PDF toolkit http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit [08] (2013) ImageMagic Homepage http://www.imagemagick.org/script/index.php M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28 ... 28/28