Building our own CDN
How to serve millions of viewers
efficiently
Zoltán Németh
at a
glance
Live streaming
Free broadcastingSoftware as a Service
Big events
• Sony Playstation
• Nintendo
• Lady Gaga
• F...
Streaming  Definitions
 Broadcaster, viewer
 Media
 Buffering
 History
 FMS, RTMP
 Edge
 UHS, HTTP
How we use the
network
 Live streaming properties
 Continuous connection vs chunk
download
 Big bandwidth
 Stability c...
First approach:
CDN
 Pros
 Scaling, flexibility
 Bigger capacity
 Cons
 Costs
 Origin requests
 Limitations
 Regio...
Multiple CDNs  Global coverage solved
 Costs high
 Unified configuration problem
 How to select which one to use
Own capacity  Why
 Cost-effectiveness
 Keep CDNs for peaks
 Transit lines
 Peering
 Billing model
 95% / Flat rate
How to utilize own
network
 Basic decision logic
 Peering first
 Transit second
 CDNs last
 Cache layer
 Varnish
 M...
Viewer side
switching - QoS
 List all providers
 In case of problem – switch
 Challenges:
 How to detect a problem
 S...
Server side
prediction
 Initial guess for QoS
 Minimize useless switches (UX)
 Cost optimization
 How?
 Viewers repor...
UMS  Ustream Media Server
 Continuous connection to all clients
 Real-time push-based updates
 First use: viewer numbe...
UMS  Frontend layer
 Up to 150.000 clients / machine
 3 IP addresses per machine
 1.5 billion outgoing messages per
ma...
The data source:
Streamstat
 Viewer reports
 UMS process
 Log
 Historic data processing
 From log
 Hadoop + Tableau
...
Streamstat reports  Types of reports
 Playing
 Buffering
 Bandwidth
 Errors, QoS events
 Verticals
 Geo
 AS number...
Chandra  Real-time data aggregation layer
 Replaceable data store
 Redis
 Simple logic: based on increments
 Easily s...
Chandra demo
Decisions  Ways of routing viewers
 Initial setup
 Updates
 Stream priority
 Stream module
 Ways of decision
 Rule-...
Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists
Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists
Stream module
generation flow
Rule blocking UCDN
Routing on own
network
 Provider resolution service
 Reject to CDN if full
 Capacity monitoring
 Lines
 Clusters
What we have
now
 3 big CDN providers
 ~5 smaller providers
 UMS clusters
 Core in SJC
 POPs: NRT, AMS, BUD
 Exchang...
Future plans  Growth 
 Dynamic rules
 QoS and MBR
 UMS Elastic Logic
 Open sourcing
 Framework
 Chandra
Questions
syntaxerror@ustream.tv
Building our own CDN
Upcoming SlideShare
Loading in...5
×

Building our own CDN

1,127

Published on

Published in: Internet, Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,127
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
7
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Building our own CDN

  1. 1. Building our own CDN How to serve millions of viewers efficiently Zoltán Németh
  2. 2. at a glance Live streaming Free broadcastingSoftware as a Service Big events • Sony Playstation • Nintendo • Lady Gaga • Festivals Citizen journalism • Rescue of Chilean miners • Earthquake in Japan • Revolutions in Egypt, Syria • Protests in Ukraine • Obama campaign • Climate Reality • Football • Justin Bieber Concurrent viewership peaks at almost 2 million
  3. 3. Streaming  Definitions  Broadcaster, viewer  Media  Buffering  History  FMS, RTMP  Edge  UHS, HTTP
  4. 4. How we use the network  Live streaming properties  Continuous connection vs chunk download  Big bandwidth  Stability critical  Large audience  Global delivery  Providers performance  Mobile networks
  5. 5. First approach: CDN  Pros  Scaling, flexibility  Bigger capacity  Cons  Costs  Origin requests  Limitations  Regional and provider differences  Capacity race conditions
  6. 6. Multiple CDNs  Global coverage solved  Costs high  Unified configuration problem  How to select which one to use
  7. 7. Own capacity  Why  Cost-effectiveness  Keep CDNs for peaks  Transit lines  Peering  Billing model  95% / Flat rate
  8. 8. How to utilize own network  Basic decision logic  Peering first  Transit second  CDNs last  Cache layer  Varnish  Monitoring  POPs, Edge clusters
  9. 9. Viewer side switching - QoS  List all providers  In case of problem – switch  Challenges:  How to detect a problem  Sync  Broadcaster side problems
  10. 10. Server side prediction  Initial guess for QoS  Minimize useless switches (UX)  Cost optimization  How?  Viewers report metrics  Server side processing and aggregation
  11. 11. UMS  Ustream Media Server  Continuous connection to all clients  Real-time push-based updates  First use: viewer number display  Channel status poll  Viewer authentication and authorization  Stream information  Java, kernel tweaks
  12. 12. UMS  Frontend layer  Up to 150.000 clients / machine  3 IP addresses per machine  1.5 billion outgoing messages per machine per day (10 billion for the live cluster)  Aggregation / logic layer  Current: 400.000 clients per media limit  Future: distributed
  13. 13. The data source: Streamstat  Viewer reports  UMS process  Log  Historic data processing  From log  Hadoop + Tableau  Realtime processing in UMS  Chandra {"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPD ATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtoc ol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamPr otocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc- uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"} {"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_U PDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","netwo rkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241, "url":"http://uhs-akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117 ,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs- akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214 ,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs- akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575," streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs- level3.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"} {"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UP DATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProto col":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamPr otocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams- uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}
  14. 14. Streamstat reports  Types of reports  Playing  Buffering  Bandwidth  Errors, QoS events  Verticals  Geo  AS number  Media
  15. 15. Chandra  Real-time data aggregation layer  Replaceable data store  Redis  Simple logic: based on increments  Easily scalable
  16. 16. Chandra demo
  17. 17. Decisions  Ways of routing viewers  Initial setup  Updates  Stream priority  Stream module  Ways of decision  Rule-based  Dynamic
  18. 18. Stream module generation flow  Filter the Stream Priority list  Dedications / whitelists
  19. 19. Stream module generation flow  Filter the Stream Priority list  Dedications / whitelists
  20. 20. Stream module generation flow Rule blocking UCDN
  21. 21. Routing on own network  Provider resolution service  Reject to CDN if full  Capacity monitoring  Lines  Clusters
  22. 22. What we have now  3 big CDN providers  ~5 smaller providers  UMS clusters  Core in SJC  POPs: NRT, AMS, BUD  Exchanges  QoS and static rules
  23. 23. Future plans  Growth   Dynamic rules  QoS and MBR  UMS Elastic Logic  Open sourcing  Framework  Chandra
  24. 24. Questions syntaxerror@ustream.tv
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×