Track4K is an open-source video analysis and tracking system that post-processes a 4K video to track the presenter and generate a cropped version of the video using smooth panning movements.

1. ||
Stephen Marquard, University of Cape Town
1
Track4K in production

2. ||
Track4K is an open-source video analysis and tracking system that post-
processes a 4K video to track the presenter and generate a cropped version of
the video using smooth panning movements.
Track4K was developed as a 4th-year Computer Science student project at the
University of Cape Town in 2016, released as open source in Jan 2017 and
deployed as part of UCT's Opencast system in September 2017.
Created by Charles Fitzhenry, Maximillian Hahn and Tanweer Khatieb
Supervisors: Patrick Marais, Stephen Marquard
14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 2
Track4K Overview

3. || 14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 3
Track4K works best in medium-sized venues

4. ||
1. Record in 4K, process to 1080p or 720p, publish smaller sizes
(UCT production workflow)
2. Record in 4K, create tracking data only, publish in 4K with
player auto-tracking (paella prototype)
3. Record in 4K, process to 1080p or 720p, publish 4K original (untracked)
and smaller sizes (tracked)
14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 4
Three ways to use Track4K

5. ||
 Record the lecture with a 4K camera
 track4k analyzes the input video and produces a tracking file, with frame-by-frame positioning information
 cropvid reads the tracking file and input video and produces an output video
 Because the tracking is not real-time, the field of view changes smoothly and only when necessary
14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 5
How it works

6. ||
 Originally, track4k processed everything itself, from input video to output video
 OpenCV only handles video files, and will not preserve an audio track.
 That’s still fine if the 4K video is recorded using a fixed frame rate:
you can mux the resulting video with the input video audio track.
 But if you have a video recorded with a variable frame rate (typical for H264 streams from IP
cameras), this creates audio synchronization issues, because OpenCV does not provide access
to per-frame timing information (no PTS value is available for each frame), so the output video is
written as if it had a fixed frame rate.
 To resolve this:
 track4k reads the video and writes a tracking file (frame-by-frame).
 cropvid reads the input video and writes the tracked output video, preserving frame timing information
 The tracking file can also be used stand-alone for other applications (player-side tracking)
14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 6
Why does Track4K come in two parts?

7. ||
[track1]
name = audio
device = pulse
flavor = presenter
location = …
file = audio.flac
player = True
audioencoder = flacenc
active = True
delay = 0.2
14/02/2018 7
Track4K at UCT: Galicaster configuration (Axis 1428)
[track2]
name = presenter
device = rtp
flavor = presenter4k
location = rtspt://m209-cam01.uct.ac.za/axis-media/media.amp
file = presenter.mkv
cameratype = h264
audio = False
muxer = matroskamux
caps-preview = video/x-raw,framerate=1/1
active = True
[basic]
custom_flavors=presenter4k presentation2

8. ||
Uses the Opencast Execute-
Many workflow operation
handler.
After ingest, if there is a
presenter4k flavor in the
mediapackage, then run track4k
and create a presenter/source
tracked video.
We do not archive the 4K
version, so only one presenter
video is retained in the
mediapackage.
14/02/2018 8
Track4K at UCT: workflow configuration

9. || 14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 9

10. ||
The wrapper script
uses ${event_location}
to look up the venue in
a config file, and
optionally set the target
resolution and vertical
crop position.
14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 10
Track4K at UCT: wrapper script

11. ||
Limitations
 Only does panning movements
 Tracking can sometimes be distracted by audience members
(workaround: apply top and bottom image masks on the camera)
 Run-time is approximately 3 x real-time (track4k and cropvid)
Future work
More sophisticated tracking and virtual cinematography is the subject of Master’s
theses by Charles Fitzhenry and Tanweer Khatieb (completion during 2018).
14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 11

12. ||
Source:
https://github.com/LectureTracking/trackhd
Wrapper script:
https://github.com/cilt-uct/trackhd/tree/master/examples/opencast
Track4K Student Project:
http://track4k.co.za
Technical paper: A Virtual Cinematographer for Presenter Tracking in 4K Lecture Videos
http://pubs.cs.uct.ac.za/archive/00001178
News article:
http://www.cilt.uct.ac.za/cilt/news/Computer-Science-students-improve-UCT-lecture-recording
14/02/2018First name Surname (edit via “Insert” > “Header & Footer”) 12
Get Track4K