Best Practices of building a highly scalable, available, resilient live streaming architecture

1. 1©2019
Confidential
Best Practices for Live
Streaming
Tokyo Video Tech Meetup
April 22nd, 2019

2. 2©2019
Confidential
Ashok Lalwani
Head of Solutions & Partnerships,
Video Streaming
ashok@fastly.com

3. 3©2019
Confidential
Introduction

4. 7©2019
Confidential
Fastly’s Edge Cloud network Network Footprint *
● 60 POPs on 6 continents
● 68 Internet Exchange
Points
● 45 Tbps connected capacity
Platform Highlights
● Single network/platform
● Write your own config
● Instant global config
changes
● 150 ms mean global purge
time
● Stream logs to 1 or many
endpoints simultaneously
● TLS everywhere
* Network stats as of 3/31/2019

5. 8©2019
Confidential
Take control of the edge
Execute more logic at the edge, closer to your viewers
Fastly Edge Cloud
Client Devices
& Apps
Datacenter /
Cloud
Fastly PlatformFastly VCL
Monitor or write scripts to
take actionable insights
100% of logs are
streamed in real time
Control most
aspects of an
HTTP request
or response
Integrate into your
existing CI/CD
workflows
Protocol
Terminations
(HTTP/2, TLS, HTTP/3
QUIC soon)
Advanced Network
Optimizations
(origin PNI, BBR, N/W
Route resiliency)
Edge
Authentication
(JWT, time-based,
OAuth2)
Content
Targeting
(Geolocation,
Device)
Load
Balancing
(L7 GSLB & LSLB,
Multi-Origin Request
Iteration)
Edge
Logic
(Dictionaries, ACLs,
ESI, A/B testing, Cache
Key customization)

6. 9©2019
Confidential
Agenda
1. Making your Live
Streaming
architecture resilient
2. Invest in end to end
monitoring
3. Best practices to
improve user
experience

7. 10©2019
Confidential
Making your
architecture resilient

8. 11©2019
Confidential
Typical Live
Stream Workflow
Encode/
Trancode
Origin Live Feed
Video Playback QoS

9. 12©2019
Confidential
12
Subscriber growth happens with:
● Marketing ¥¥¥ spend on
subscriber acquisition
● Investment in exclusive content
(Licensed or Originals)
● Access to OTT service on more
devices/platforms
● Cord-cutters / nevers / shavers
Growth Phase
Requires continued focus in building
highly scalable and resilient
architecture

10. 13©2019
Confidential
Add more CDNs
Typical First Step
Origin Live
Feed
Encode/
Trancode
CDN
Selector
Video
Playback
QoS

11. 14©2019
Confidential
14
● Regional availability &
capacity
● Cost management
● QoS metrics
● Failover
● All of the Above
When to use
multiple CDNs
Tradeoff:
Cost / Complexity vs. Quality of
Experience (QoE)

12. 15©2019
Confidential
Wait… Did we solve the problem?
May be for scaling the streaming service to growing number of users but
what about overall resiliency/availability?
What about them?
Origin Live
Feed
Encode/
Trancode
CDN
Selector
Video
Playback
QoS
What about them?

13. 16©2019
Confidential
Here’s one reference architecture

14. 17©2019
Confidential
Here’s another reference architecture (CBS Sports)

15. 18©2019
Confidential
But all that comes at a cost…
...maintaining high availability can be very expensive but is
necessary for mission critical services
Tradeoff:
Availability / Resiliency vs. Total Cost of Ownership

16. 19©2019
Confidential
Unlike CAP
theorem, this one
is not a hard
problem
Live Linear or 24/7 channels vs.
High Profile Live Streaming Events
Total Cost of
Ownership
Availability /
Resiliency
QoE Metrics

17. 20©2019
Confidential
What your peers are doing
today:
• Maintain different architecture for Live linear/24x7
channels and Live events
• Manage their own live streaming architecture
thereby allowing them to experiment and iterate

18. 21©2019
Confidential
Invest in end to end
monitoring

19. 22©2019
Confidential
The internet is terrible
Too many intermediaries
Datacentre
Cloud 1
Cloud 2
IXP
ISP
Transit
Provider
IXP
Transit
Provider
Transit
Provider
Transit
Provider
Mobile

20. 23©2019
Confidential
The internet is terrible wonderful
Too many intermediaries also makes internet resilient
Datacentre
Cloud 1
Cloud 2
IXP
ISP
Transit
Provider
IXP
Transit
Provider
Transit
Provider
Transit
Provider
Mobile

21. 24©2019
Confidential
Typical QoS metrics monitored
Are these metrics sufficient to find issues during a
live event?
• Rebuffer Ratio (most common monitored)
• Startup Time (aka Time To First Frame)
• Avg. Throughput
• Avg. Up/Down Switch
• Video Start Failures (aka VSF) or Exits Before Video Starts (aka EBVS)

22. 25©2019
Confidential
The internet is terrible wonderful terrible
Both of them would show up as rebuffering
Datacentre
Cloud 1
Cloud 2
IXP
ISP
Transit
Provider
IXP
Transit
Provider
Transit
Provider
Transit
Provider
Mobile

23. 26©2019
Confidential
Build your end to end data pipeline
Origin Live FeedEncode/
Trancode
CDN
Selector
Data Pipeline

24. 27©2019
Confidential
Example (CBSi)

25. 28©2019
Confidential
What we need is
• Well defined of metrics across all
components not just on client side
• Standardized way to exchange those metrics
with various systems
• Proactive & Reactive tools/templates to
identify, troubleshoot and mitigate issues

26. 29©2019
Confidential
Best practices to
improve your end user
experience

27. 30©2019
Confidential
We will cover some commonly
used techniques
• Bitrate Ladder
• Video manifest variants for devices
• HLS or DASH or both???

28. 31©2019
Confidential
HLS Manifest example
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:BANDWIDTH=1425745,RESOLUTION=768x432,FR
AME-RATE=29.970,CODECS="avc1.4D401F,mp4a.40.5"
desktop_1.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=345977,RESOLUTION=400x224,FRA
ME-RATE=29.970,CODECS="avc1.42C01E,mp4a.40.5"
desktop_2.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=620677,RESOLUTION=512x288,FRA
ME-RATE=29.970,CODECS="avc1.42C01E,mp4a.40.5"
desktop_3.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=950712,RESOLUTION=640x360,FRA
ME-RATE=29.970,CODECS="avc1.4D401F,mp4a.40.5"
desktop_4.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2085745,RESOLUTION=960x540,FR
AME-RATE=29.970,CODECS="avc1.4D401F,mp4a.40.5"
desktop_5.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3955779,RESOLUTION=1280x720,F
RAME-RATE=29.970,CODECS="avc1.4D401F,mp4a.40.2"
desktop_6.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5978451,RESOLUTION=1920x1080,
FRAME-RATE=29.970,CODECS="avc1.4D4028,mp4a.40.2"
desktop_7.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:7
#EXT-X-MEDIA-SEQUENCE:4300078
#EXTINF:6.006,
desktop_1_4300078.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300079.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300080.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300081.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300082.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300083.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300084.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300085.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300086.ts?m=1549344461
#EXTINF:6.006,
desktop_1_4300087.ts?m=1549344461

29. 32©2019
Confidential
Device-aware manifests
#EXTM3U
#EXT-X-STREAM-INF: BANDWIDTH=1425745,RESOLU
TION=768x432,FRAME-RATE=29.970,CODECS="av
c1.4D401F,mp4a.40.5"
desktop_1.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=345977,RESOLUT
ION=400x224,FRAME-RATE=29.970,CODECS="avc
1.42C01E,mp4a.40.5"
desktop_2.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=620677,RESOLUT
ION=512x288,FRAME-RATE=29.970,CODECS="avc
1.42C01E,mp4a.40.5"
desktop_3.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=950712,RESOLUT
ION=640x360,FRAME-RATE=29.970,CODECS="avc
1.4D401F,mp4a.40.5"
desktop_4.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=2085745,RESOLU
TION=960x540,FRAME-RATE=29.970,CODECS="av
c1.4D401F,mp4a.40.5"
desktop_5.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=3955779,RESOLU
TION=1280x720,FRAME-RATE=29.970,CODECS="a
vc1.4D401F,mp4a.40.2"
desktop_6.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=5978451,RESOLU
TION=1920x1080,FRAME-RATE=29.970,CODECS="
avc1.4D4028,mp4a.40.2"
desktop_7.m3u8
#EXTM3U
#EXT-X-STREAM-INF: BANDWIDTH=1425745,RESOLU
TION=768x432,FRAME-RATE=29.970,CODECS="av
c1.4D401F,mp4a.40.5"
tablet_1.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=345977,RESOLUT
ION=400x224,FRAME-RATE=29.970,CODECS="avc
1.42C01E,mp4a.40.5"
tablet_2.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=620677,RESOLUT
ION=512x288,FRAME-RATE=29.970,CODECS="avc
1.42C01E,mp4a.40.5"
tablet_3.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=950712,RESOLUT
ION=640x360,FRAME-RATE=29.970,CODECS="avc
1.4D401F,mp4a.40.5"
tablet_4.m3u8
#EXT-X-STREAM-INF:B ANDWIDTH=2085745,RESOLU
TION=960x540,FRAME-RATE=29.970,CODECS="av
c1.4D401F,mp4a.40.5"
table_5.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=3955779,RESOLU
TION=1280x720,FRAME-RATE=29.970,CODECS="a
vc1.4D401F,mp4a.40.2"
tablet_6.m3u8
#EXTM3U
#EXT-X-STREAM-INF: BANDWIDTH=950712,RESOLUT
ION=640x360,FRAME-RATE=29.970,CODECS="avc
1.4D401F,mp4a.40.5"
iphone_1.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=345977,RESOLUT
ION=400x224,FRAME-RATE=29.970,CODECS="avc
1.42C01E,mp4a.40.5"
iphone_2.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=620677,RESOLUT
ION=512x288,FRAME-RATE=29.970,CODECS="avc
1.42C01E,mp4a.40.5"
iphone_3.m3u8
#EXT-X-STREAM-INF: BANDWIDTH=1425745,RESOLU
TION=768x432,FRAME-RATE=29.970,CODECS="av
c1.4D401F,mp4a.40.5"
iphone_4.m3u8
#EXT-X-STREAM-INF:B ANDWIDTH=2085745,RESOLU
TION=960x540,FRAME-RATE=29.970,CODECS="av
c1.4D401F,mp4a.40.5"
iphone_5.m3u8
Browser Tablet iPhone

30. 33©2019
Confidential
HLS or DASH or both???
• HLS predominantly used in Live Streaming. Support DASH only if:
– DRM is a requirement
– Device support
– SSAI support for DASH is still not mature yet
• Typically 4s or 6s segment duration used
• Low latency implementation standardization for HLS first demo’ed during NAB
2019
– No native iOS support for Low Latency yet

31. 34©2019
Confidential
Thank you!