1. Building our own CDN
How to serve millions of viewers
efficiently
Zoltán Németh

2. at a
glance
Live streaming
Free broadcastingSoftware as a Service
Big events
• Sony Playstation
• Nintendo
• Lady Gaga
• Festivals
Citizen journalism
• Rescue of Chilean miners
• Earthquake in Japan
• Revolutions in Egypt, Syria
• Protests in Ukraine
• Obama campaign
• Climate Reality
• Football
• Justin Bieber
Concurrent viewership peaks at almost 2 million

3. Streaming  Definitions
 Broadcaster, viewer
 Media
 Buffering
 History
 FMS, RTMP
 Edge
 UHS, HTTP

4. How we use the
network
 Live streaming properties
 Continuous connection vs chunk
download
 Big bandwidth
 Stability critical
 Large audience
 Global delivery
 Providers performance
 Mobile networks

5. First approach:
CDN
 Pros
 Scaling, flexibility
 Bigger capacity
 Cons
 Costs
 Origin requests
 Limitations
 Regional and provider differences
 Capacity race conditions

6. Multiple CDNs  Global coverage solved
 Costs high
 Unified configuration problem
 How to select which one to use

7. Own capacity  Why
 Cost-effectiveness
 Keep CDNs for peaks
 Transit lines
 Peering
 Billing model
 95% / Flat rate

8. How to utilize own
network
 Basic decision logic
 Peering first
 Transit second
 CDNs last
 Cache layer
 Varnish
 Monitoring
 POPs, Edge clusters

9. Viewer side
switching - QoS
 List all providers
 In case of problem – switch
 Challenges:
 How to detect a problem
 Sync
 Broadcaster side problems

10. Server side
prediction
 Initial guess for QoS
 Minimize useless switches (UX)
 Cost optimization
 How?
 Viewers report metrics
 Server side processing and
aggregation

11. UMS  Ustream Media Server
 Continuous connection to all clients
 Real-time push-based updates
 First use: viewer number display
 Channel status poll
 Viewer authentication and
authorization
 Stream information
 Java, kernel tweaks

12. UMS  Frontend layer
 Up to 150.000 clients / machine
 3 IP addresses per machine
 1.5 billion outgoing messages per
machine per day (10 billion for the
live cluster)
 Aggregation / logic layer
 Current: 400.000 clients per
media limit
 Future: distributed

13. The data source:
Streamstat
 Viewer reports
 UMS process
 Log
 Historic data processing
 From log
 Hadoop + Tableau
 Realtime processing in UMS
 Chandra
{"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business
Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPD
ATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtoc
ol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamPr
otocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc-
uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static-
cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN
13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat
Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_U
PDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","netwo
rkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241,
"url":"http://uhs-akamai.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117
,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs-
akamai.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214
,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs-
akamai.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575,"
streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs-
level3.ustream.tv/ams/ams-
uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static-
cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN
11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN
N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UP
DATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProto
col":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamPr
otocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams-
uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static-
cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}

14. Streamstat reports  Types of reports
 Playing
 Buffering
 Bandwidth
 Errors, QoS events
 Verticals
 Geo
 AS number
 Media

15. Chandra  Real-time data aggregation layer
 Replaceable data store
 Redis
 Simple logic: based on increments
 Easily scalable

16. Chandra demo

17. Decisions  Ways of routing viewers
 Initial setup
 Updates
 Stream priority
 Stream module
 Ways of decision
 Rule-based
 Dynamic

18. Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists

19. Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists

20. Stream module
generation flow
Rule blocking UCDN

21. Routing on own
network
 Provider resolution service
 Reject to CDN if full
 Capacity monitoring
 Lines
 Clusters

22. What we have
now
 3 big CDN providers
 ~5 smaller providers
 UMS clusters
 Core in SJC
 POPs: NRT, AMS, BUD
 Exchanges
 QoS and static rules

24. Future plans  Growth 
 Dynamic rules
 QoS and MBR
 UMS Elastic Logic
 Open sourcing
 Framework
 Chandra

25. Questions
syntaxerror@ustream.tv