YALMS (Yet Another Lecture Management System) is a lecture automation and annotation system. It uses various tools like ffmpeg and ImageMagick to extract frames from lecture videos, trim black margins, resize pages to a common size, and compare frames to PDF slides to automatically generate annotations of which slide corresponds to each video frame. The paper describes the challenges of annotating "fuzzy" videos and slides and presents solutions to handle differences in sizes and formats to accurately align video lectures with presentation slides.
Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs
2. .
YALMS: Review
.
YALMS....
..
....stands for Yet Another Lecture Management System
• basically, a class robot
◦ robot: set of automation scripts and frameworks
• first presented in ET研2013.07 04
• a long-term project with 4-5 main components -- see the roadmap at the end
04 M.Zhanikeev+1 "YALMS: Yet Another Lecture Management System..." IEICE ET研 (2013.09)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 2/28
...
2/28
3. .
YALMS: Overall Design
Step 1: Record and Stream
V
A
S
Lecturer
Viewers
Content
Server
HTML5
HTML5
Muxer
RecordScreen
Streaming
Step 2: Annotate
Lecturer
S
Auto-
annotate
C
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 3/28
...
3/28
6. .
Sources of Fuzziness
1. unusual screen sizes in presentation mode
◦ this PDF was created in 16:9mode to fit my notebook perfectly, will NOT fit
others
2. unpredictable or unavoidable margins when saving your slides as PDF
◦ PPTX! no way to avoid white margins in print
◦ tex2pdf creates perfect PDFs! --- look the same in PDF and in presentation mode
3. slide animation
◦ PPTX animation is really BAD! (confusing, difficult to print, ...)
◦ latex/beamer creates sequences of pages for print -- same page number, many pages
◦ the point is ... animation should look good both in print and on screen
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 6/28
...
6/28
8. .
Objectives and Assumptions
.
YALMS should be able...
..
.... to retain automation capability in fuzzy environments
• ASSUMPTION 1: no rules for slide PDFs -- only the PDF format
itself
• ASSUMPTION 2: no rules for screen video -- black margins, wierd
sizes -- all OK
• ASSUMPTION 3: no additional information is requested,
annotation should happen with only (1) and (2)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 8/28
...
8/28
9. .
Tools and Actual Software
1. pdftk : a nice library for handing PDFs 07
2. ImageMagic : the best graphics library (basically free Photoshop) 08
3. VLC for screen capture, ffmpeg will also work
4. ffmpeg for video editing 06
5. all scripting is done in php -- suitable for both CLI and web server
07 "pdftk: The PDF toolkit" http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit (2013)
08 "ImageMagic Homepage" http://www.imagemagick.org/script/index.php (2013)
06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 9/28
...
9/28
11. .
Solution 1: Simple Scene Detection
• the simplest way to detect scene detection is to extract all I-Frames 06
• creates more images than actual scene changes, but redundancy is not a
problem
$ ffmpeg -i screen.mp4 -vf select="eq(pict_type¥,I)"
-vsync 0 -an frames.%03d.png
$ stands for command prompt, all commands are single-line
06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 11/28
...
11/28
12. .
Solution 2: Trim the Black Margins
• deal with fuzzy margins and sizes using ImageMagic's
convert
• OK to add a small border because trim does not cut everything anyway
• force frame resize
$ convert -colorspace gray -bordercolor black
-border 2x2 -trim +repage -resize 1280x720!
frames.004.png frames.004b.png
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 12/28
...
12/28
13. .
Solution 2: Trim Performance
Nothing to
trim
Trim
success!
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 13/28
...
13/28
14. .
Solution 3: Bring to Common Size
• cannot compare without same size
$ convert -colorspace gray -bordercolor black
-border 8x5 -resize 1280x720! -density 300x300
-quality 100 pages.002.pdf pages.002.png
$ stands for command prompt, all commands are single-line
$ convert -colorspace gray -bordercolor black
-border 2x2 -trim +repage -resize 1280x720!
frames.004.png frames.004b.png
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 14/28
...
14/28
15. .
Solution 3: Size Automation
• easy to find page size in command line, so, automation OK
platypus@platypus-PC /local/yalms.content
$ identify -format "%w %h" frames.004.png
1280 720
platypus@platypus-PC /local/yalms.content
$ identify -format "%w %h" frames.004c.png
1156 720
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 15/28
...
15/28
16. .
Solution 4: Compare using the AE Metric
• AE metric: simply the absolute pixel diff between two images
• alternatives: PNSR, MLSE ...
$ compare -colorspace gray -metric AE -fuzz 50%
pages.002.png frames.004b.png diff.png
7021
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 16/28
...
16/28
17. .
Solutions: Example
Video frame
Video frame
(trim, resize)
PDF page
(border, resize)
Diff image
AE: 7019
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 17/28
...
17/28
19. .
Performance: Experimental Setup
• a 1.5h class
• screen capture and slide PDF
• fuzzy: 16:10 screen size (black margin)
• fuzzy: many ALT-TAB switches between applications
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 19/28
...
19/28
20. .
Performance: Parameters and Metrics
• fuss: used by ffmpeg as comparison threshold
• pace : minimum frequency of page changes -- possible to remove jitter
• performance: annotation success
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 20/28
...
20/28
21. .
Performance: Frame Diffs
• generally, good
trend, because
only one frame
matches well
• not ideal performance
0 20 40 60 80 100
Ordered list of frame diffs
1.2
2.4
3.6
4.8
6
log(framediff)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 21/28
...
21/28
22. .
Performance: Results
• thickness:
fuss value
(between 10% and 90%)
• changing fuss can help
sometimes
• some cases are
persistent
failures
0 2 4 6 8 10 12 14 16 18
Page sequence
0
0.5
1
1.5
2
2.5
3
3.5
log(timediff)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 22/28
...
22/28
23. .
Performance: Room for Improvement?
• multi-fuss comparison, pick the best
• when failing consistently, just set marker between two neighboring
successes
◦ mark it as failed and ask for human
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 23/28
...
23/28
25. .
YALMS.annotation : The Robot
screen
video
Slides
PDF
LaTeX/Beamer
Keynote
PowerPoint
…
VLC
ffmpeg
I-Frames I-Frames
pages PNGs
ANNOTATION
slices
PNGs
timestamps
End product
scenes cleanup
split
pdf2png
03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 25/28
...
25/28
26. .
YALMS.annotation : The Script
$ php robot.php
PURPOSE: to process two files: SLIDES PDF and
SCREEN CAPTURE MOVIE
OUTCOME: ANNOTATIONS for the movie -- times for each
PDF page in the movie
[pace] minimum time gap between page changes
-- will ignore fickers below that threshold
[wdir] full path to working directory
[screen] filename of the screen capture movie
-- should be in $wdir
[slides] filename of the slides PDF file -- should be in $wdir
$
$ stands for command prompt, all commands are single-line
03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 26/28
...
26/28
27. .
YALMS: The Roadmap
YALMS: Intro
YALMS.annotation
2013.07・ET研
2013.09・ET研
YALMS.streaming
2014.01・ET研
YALMS.hiperf
2013.10・ITS-MMS研
ET研
その他
YALMS.cloud
2013.11・IN研
YALMS.format
YALMS.done!
DASH Adaptive HTML5
PDF slices
VM sync groupdrive
SSD
multicore
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 27/28
...
27/28
28. .
That’s all, thank you ...
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28
...
28/28
29. .
[01] (2013)
The enPiT Project
http://www.enpit.jp
[02] (2013)
YALMS Project Home
https://github.com/maratishe/yalms
[03] (2013)
YALMS Annotation Robot (software behind this paper)
https://github.com/maratishe/yalms.annotation
[04] M.Zhanikeev+1 (2013.09)
YALMS: Yet Another Lecture Management System...
IEICE ET研
[05] (2013)
Homepage of edubase Stream at NII
http://stream.edubase.jp
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28
...
28/28
30. .
[06] (2013)
Homepage of ffmpeg and ffserver tools
http://www.ffmpeg.org
[07] (2013)
pdftk: The PDF toolkit
http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit
[08] (2013)
ImageMagic Homepage
http://www.imagemagick.org/script/index.php
M.Zhanikeev/H.Koide -- maratishe@gmail.com -- Methods and Algorithms for YALMS Video Annotation Robot 28/28
...
28/28