1. MiPSO: Multi-Period Per-Scene Optimization For
HTTP Adaptive Streaming
Phani Malladi†, Christian Timmerer†‡ and Hermann Hellwagner†
†Alpen-Adria-Universitat Klagenfurt and †‡Bitmovin Inc.
International Conference on Multimedia and Expo 2020
1 / 25

2. Outline
1 Introduction
2 Background and Related Work
3 Multi-Period Per-Scene Optimization Framework
4 Experimental Evaluation
5 Conclusion
6 Acknowledgement
2 / 25

3. Introduction
Over-the-top (OTT) video services such as Netflix, YouTube, Disney+
and many others continue to grow in popularity.
According to the Cisco Visual Networking Index forecast, global Internet
video traffic is expected to exceed 82% in 2022.
Dynamic Adaptive Streaming over HTTP (DASH) has become de facto
standard in recent years for delivering videos over the Internet.
The current DASH specification defines a hierarchical data model for
Media Presentation Descriptions (MPDs).
Periods
Adaptation Sets
Representations
Segments
3 / 25

4. Introduction
In the days of “one-size-fits-all” video delivery, all service providers used
a single set of representations for distributing all their content.
The traditional approach (a.k.a fixed bitrate ladder) employs a fixed
bitrate for a given spatial resolution.
The fixed bit rate ladder completely ignores video-specific characteris-
tics and applies the same encoding settings to all video content.
The sub-optimality entailed by traditional approach can be significantly
improved by customizing the resolution-bitrate pairs of each video.
The service providers have slowly started tailoring the streams in an
automated and scalable way for each piece of video content.
4 / 25

5. Introduction
Table 1: Example of a Conventional Encoding Ladder1
.
Profile Resolution
Bitrate
(kbps)
1 320x240 235
2 384x288 375
3 512x384 560
4 512x384 750
5 640x480 1050
6 720x480 1750
7 1280x720 2350
8 1280x720 3000
9 1920x1080 4300
10 1920x1080 5800
1
Netflix. https://medium.com/netflix-techblog/per-title-encode-
optimization-7e99442b62a2.
5 / 25

6. Introduction
A Multi-Period Per-Scene Optimization (MiPSO) framework is pro-
posed to examine multiple periods in on-demand scenarios of DASH.
The proposed framework utilizes multiple periods in MPD to provide
different encoded representations of a video at
Maximum possible quality
Minimum possibe bitrate
In each period, MiPSO adjusts the video representations by taking into
account the complexities of the video content.
The experimental evaluation with a test video dataset demonstrates
that MiPSO can
reduce average bitrate of streams by approximately 10% at the same
visual quality
increase visual quality of streams by at least 1 dB in terms of PSNR at
the same bitrate
6 / 25

7. Background and Related Work
A number of proprietary solutions are proposed in the recent literature
for “encoding ladder” (encoding set) customization.
Different names — per-title, content-aware, context-aware, content
adaptive, encoding optimization, ...
For determining an optimal encoding set, all of these solutions perform
multiple test encodings of the video content at different resolutions
with various quantization parameter combinations.
From the rate-quality curves, a specific bitrate-resolution pair is derived
for the video such that the encoded representation aims to maximize
the video quality or minimize the bitrate.
7 / 25

8. Background and Related Work
Each solution has its customization in
adjusting the total number of representations
adjusting the bitrate of each representation
adjusting the minimum and maximum bitrates
adjusting output specific resolutions
adjusting number of passes
All of the solutions proposed in the literature are proprietary and hence,
it is not possible to make direct comparisons across these solutions.
Lastly, all of the existing solutions apply their “encoding ladder” over
the whole video content for optimizing either bitrate or quality.
Since characteristics of each scene within a video varies which results
in significant number of quality fluctuations across the scenes.
8 / 25

9. Background and Related Work
The current DASH specification accepts for multiple periods within a
MPD and, thus, allows to remove, add, or modify certain representa-
tions within each period.
Although the applicability of multi-period MPDs is demonstrated in
live streaming scenarios, it is not fully exploited in VoD scenarios.
Complementary to the existing state-of-the art approaches, in this pa-
per, a Multi-Period Per-Scene Optimization (MiPSO) framework is pro-
posed.
The proposed framework adjusts video representations in each period
and there by utilizing the current specification of DASH to its full
potential for VoD services.
9 / 25

10. Multi-Period Per-Scene Optimization Framework
Figure 1: Flowchart
10 / 25

11. Multi-Period Per-Scene Optimization Framework
MiPSO adopts a sequence of segment encodings to estimate the com-
plexity of the video at different bitrate-resolution combinations.
The segments of a video are obtained by sampling the video at regular
intervals such that they are spread throughout the whole video.
MiPSO subsequently constructs a bitrate-quality model of the video
in terms of the averaged values of the segment bitrates and qualities
which serve as input for the encodings of each scene.
The accuracy of the bitrate-quality model will be improved with the
increase in either duration of segments or sampling frequency.
11 / 25

12. Multi-Period Per-Scene Optimization Framework
MiPSO subsequently divides the whole video into a number of scenes.
Our current implementation for scene detection utilizes PySceneDe-
tect2 as an algorithm operated in threshold-based mode.
For estimating the scene complexity, MiPSO later adopts a sequence
of segments encodings from every individual scene in the video.
Thereafter, the bitrate-quality curves of each scene are constructed
from the encodings of the representative segments.
A set of bitrate-resolution pairs is further derived for every scene in the
video by selecting bitrate-quality closest to the convex hull.
2
PySceneDetect. https://pyscenedetect.readthedocs.io/en/latest/.
12 / 25

13. Multi-Period Per-Scene Optimization Framework
Figure 2: Convex Hull Construction
13 / 25

14. Multi-Period Per-Scene Optimization Framework
For quality optimization the upper envelope of convex hull, while for
bitrate optimization the lower envelope of convex hull are considered.
Finally, a multi-period MPD is obtained after packaging each scene as
a period such that each period will have adaptive encoding ladder.
MiPSO adjusts the resolution of every scene in the video under the
constraints of either bitrate or quality.
MiPSO always leads to either higher perceptual quality at the same
bitrate or to lowest bitrate at the same perceptual quality streams.
14 / 25

15. Experimental Evaluation
Efficacy of MiPSO is evaluated, along with fixed encoding ladder
average bitrate and total storage space required for a given target quality
average quality delivered for a given target bitrate
The proposed framework is evaluated experimentally using H.264/AVC,
in particular, encodings are performed using FFmpeg3 and libx2644.
For the fixed ladder, each video is encoded at different bitrate combi-
nations as in Table 1.
For MiPSO each scene is encoded at the respective optimal resolution-
bitrate ladder as obtained by quality and bitrate optimizations.
For the fixed ladder the encoded resolution does not vary throughout
the representation, whereas with MiPSO the encoded resolution varies.
3
FFmpeg. https://www.ffmpeg.org/.
4
x264. http://www.videolan.org/developers/x264.html.
15 / 25

16. Experimental Evaluation
Table 2: Description of Test Video Set.
Video Title Resolution
Duration Frame Rate
M
(sec) (fps)
Big Buck Bunny 1920x1080 596 25 10
Tears of Steel 1920x1080 653 25 13
Sintel 1920x1080 852 25 13
Table 3: Settings for the Video Segment Encodes
Codec Resolutions
Bitrates
(kbps)
H.264
320x240, 352x288, 512x384, 235, 375, 560, 750,
640x480, 720x480 1050, 1750, 2350,
1280x720, 1920x1080 3000, 4300, 5800
16 / 25

17. Experimental Evaluation
Table 4: Fixed vs. MiPSO Ladder (Quality Optimization)
Bitrate Test Content
Fixed MiPSO
VMAF
Y-PSNR
VMAF
Y-PSNR
(kbps) (dB) (dB)
1943.93 Big Buck Bunny 74.19 37.94 81.88 39.69
1958.28 Tears of Steel 72.20 36.16 78.41 37.23
1934.71 Sintel 73.95 38.42 81.24 40.28
Table 5: Fixed vs. MiPSO Ladder (Bitrate Optimization)
Y-PSNR Test Content
Fixed MiPSO
Bitrate Storage Bitrate Storage
(dB) (kbps) (MB) (kbps) (MB)
37.94 Big Buck Bunny 1943.93 1415.37 1697.73 880.33
36.16 Tears of Steel 1958.28 1684.81 1799.88 1288.50
38.42 Sintel 1934.71 2013.31 1751.16 1532.03
17 / 25

18. Experimental Evaluation
(a) Quality Optimization
Target Bitrate = 1750 kbps
(b) Bitrate Optimization
Target PSNR = 37.5 dB
Figure 3: Frame Level Y-PSNR values
18 / 25

19. Conclusion
MiPSO adjusts the bitrate-resolution ladder according to the scene
complexity, thereby enabling either highest quality or lowest bitrate
representations for every scene in the video.
Since MiPSO is codec agnostic, it can be applied to any other codecs
for further encoding optimization.
In future work, we plan to avoid the encoding of representative seg-
ments while preserving the benefits of the proposed framework by using
machine learning techniques.
Our dataset is available online at http://ftp.itec.aau.at/datasets/
icme20/
19 / 25

20. Conclusion
Figure 4: Multi-Period MPD
20 / 25

21. Acknowledgement
This research has been supported in part by the Christian Doppler
Laboratory ATHENA https://athena.itec.aau.at/
21 / 25

22. References I
Cisco Visual Networking Index, “Global Mobile Data Traffic Forecast
Update, 2017–2022 White Paper,” Cisco: San Jose, CA, USA, 2018.
Thomas Stockhammer, “Dynamic Adaptive Streaming over HTTP –
Standards and Design Principles,” in ACM Conference on Multimedia
Systems, 2011, pp. 133–144.
Roger Pantos and William May, “HTTP Live Streaming, RFC 8216,”
https://tools.ietf.org/html/rfc8216, 2017.
“Information Technology–Dynamic Adaptive Streaming Over HTTP
(DASH)–Part 1: Media Presentation Description and Segment For-
mats, ISO/IEC 23009-1:2019,” https://standards.iso.org/ittf/
PubliclyAvailableStandards/c075485_ISO_IEC_23009-1_2019.
zip, 2019.
22 / 25

23. References II
Chao Chen, Yao-Chung Lin, Steve Benting, and Anil Kokaram, “Opti-
mized Transcoding for Large Scale Adaptive Streaming using Playback
Statistics,” in IEEE International Conference on Image Processing,
2018, pp. 3269–3273.
Yuriy A Reznik, Karl Olav Lillevold, Abhijith Jagannath, Justin Greer,
and Jon Corley, “Optimal Design of Encoding Profiles for ABR Stream-
ing,” in ACM Packet Video Workshop, 2018, pp. 43–47.
Angeliki V Katsenou, Joel Sole, and David R Bull, “Content-gnostic
Bitrate Ladder Prediction for Adaptive Video Streaming,” in Picture
Coding Symposium, 2019.
Jan De Cock, Zhi Li, Megha Manohara, and Anne Aaron, “Complexity-
Based Consistent-Quality Encoding in the Cloud,” in IEEE International
Conference on Image Processing, 2016, pp. 1484–1488.
23 / 25

24. References III
Masaru Takeuchi, Shintaro Saika, Yusuke Sakamoto, Tatsuya Na-
gashima, Zhengxue Cheng, Kenji Kanai, Jiro Katto, Kaijin Wei,
Ju Zengwei, and Xu Wei, “Perceptual Quality Driven Adaptive Video
Coding Using JND Estimation,” in Picture Coding Symposium, 2018,
pp. 179–183.
Ioannis Katsavounidis, “Dynamic Optimizer–A Perceptual Video En-
coding Optimization Framework,” The Netflix Tech Blog, 2018.
Namgi Kim and Byoung-Dai Lee, “Analysis and Improvement of MPEG-
DASH-based Internet Live Broadcasting Services in Real-world Environ-
ments,” KSII Transactions on Internet & Information Systems, vol. 13,
no. 5, 2019.
24 / 25

25. Thanks!
Questions ?
25 / 25