This document discusses building a content delivery network (CDN) to efficiently stream live video to millions of viewers. It begins by outlining the types of live streams served, such as sporting events and citizen journalism. It then describes how live streaming works and the challenges of delivering continuous, high-bandwidth streams globally. The document evaluates using multiple commercial CDNs but finds them too costly. It proposes building out the company's own network capacity instead, using peering, transit lines and monitoring to efficiently route viewers. Metrics from millions of viewer reports are analyzed to predict quality of service and optimize routing decisions in real-time. The current network configuration and future plans to expand dynamically are outlined.

1. Building our own CDN
How to serve millions of viewers
efficiently
Zoltán Németh

2. at a
glance
Live streaming
Free broadcastingSoftware as a Service
Big events
• Sony Playstation
• Nintendo
• Lady Gaga
• Festivals
Citizen journalism
• Rescue of Chilean miners
• Earthquake in Japan
• Revolutions in Egypt, Syria
• Protests in Ukraine
• Obama campaign
• Climate Reality
• Football
• Justin Bieber
Concurrent viewership peaks at almost 2 million

3. Streaming  Definitions
 Broadcaster, viewer
 Media
 Buffering
 History
 FMS, RTMP
 Edge
 UHS, HTTP

4. How we use the
network
 Live streaming properties
 Continuous connection vs chunk
download
 Big bandwidth
 Stability critical
 Large audience
 Global delivery
 Providers performance
 Mobile networks

5. First approach:
CDN
 Pros
 Scaling, flexibility
 Bigger capacity
 Cons
 Costs
 Origin requests
 Limitations
 Regional and provider differences
 Capacity race conditions

6. Multiple CDNs  Global coverage solved
 Costs high
 Unified configuration problem
 How to select which one to use

7. Own capacity  Why
 Cost-effectiveness
 Keep CDNs for peaks
 Transit lines
 Peering
 Billing model
 95% / Flat rate

8. How to utilize own
network
 Basic decision logic
 Peering first
 Transit second
 CDNs last
 Cache layer
 Varnish
 Monitoring
 POPs, Edge clusters

9. Viewer side
switching - QoS
 List all providers
 In case of problem – switch
 Challenges:
 How to detect a problem
 Sync
 Broadcaster side problems

10. Server side
prediction
 Initial guess for QoS
 Minimize useless switches (UX)
 Cost optimization
 How?
 Viewers report metrics
 Server side processing and
aggregation

11. UMS  Ustream Media Server
 Continuous connection to all clients
 Real-time push-based updates
 First use: viewer number display
 Channel status poll
 Viewer authentication and
authorization
 Stream information
 Java, kernel tweaks

12. UMS  Frontend layer
 Up to 150.000 clients / machine
 3 IP addresses per machine
 1.5 billion outgoing messages per
machine per day (10 billion for the
live cluster)
 Aggregation / logic layer
 Current: 400.000 clients per
media limit
 Future: distributed

13. The data source:
Streamstat
 Viewer reports
 UMS process
 Log
 Historic data processing
 From log
 Hadoop + Tableau
 Realtime processing in UMS
 Chandra
{"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business
Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPD
ATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtoc
ol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamPr
otocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc-
uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static-
cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN
13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat
Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_U
PDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","netwo
rkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241,
"url":"http://uhs-akamai.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117
,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs-
akamai.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214
,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs-
akamai.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575,"
streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs-
level3.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static-
cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN
11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN
N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UP
DATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProto
col":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamPr
otocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams-
uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static-
cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}

14. Streamstat reports  Types of reports
 Playing
 Buffering
 Bandwidth
 Errors, QoS events
 Verticals
 Geo
 AS number
 Media

15. Chandra  Real-time data aggregation layer
 Replaceable data store
 Redis
 Simple logic: based on increments
 Easily scalable

16. Chandra demo

17. Decisions  Ways of routing viewers
 Initial setup
 Updates
 Stream priority
 Stream module
 Ways of decision
 Rule-based
 Dynamic

18. Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists

19. Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists

20. Stream module
generation flow
Rule blocking UCDN

21. Routing on own
network
 Provider resolution service
 Reject to CDN if full
 Capacity monitoring
 Lines
 Clusters

22. What we have
now
 3 big CDN providers
 ~5 smaller providers
 UMS clusters
 Core in SJC
 POPs: NRT, AMS, BUD
 Exchanges
 QoS and static rules

24. Future plans  Growth 
 Dynamic rules
 QoS and MBR
 UMS Elastic Logic
 Open sourcing
 Framework
 Chandra

25. Questions
syntaxerror@ustream.tv