Successfully reported this slideshow.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Building our own CDN

  1. 1. Building our own CDN How to serve millions of viewers efficiently Zoltán Németh
  2. 2. at a glance Live streaming Free broadcastingSoftware as a Service Big events • Sony Playstation • Nintendo • Lady Gaga • Festivals Citizen journalism • Rescue of Chilean miners • Earthquake in Japan • Revolutions in Egypt, Syria • Protests in Ukraine • Obama campaign • Climate Reality • Football • Justin Bieber Concurrent viewership peaks at almost 2 million
  3. 3. Streaming  Definitions  Broadcaster, viewer  Media  Buffering  History  FMS, RTMP  Edge  UHS, HTTP
  4. 4. How we use the network  Live streaming properties  Continuous connection vs chunk download  Big bandwidth  Stability critical  Large audience  Global delivery  Providers performance  Mobile networks
  5. 5. First approach: CDN  Pros  Scaling, flexibility  Bigger capacity  Cons  Costs  Origin requests  Limitations  Regional and provider differences  Capacity race conditions
  6. 6. Multiple CDNs  Global coverage solved  Costs high  Unified configuration problem  How to select which one to use
  7. 7. Own capacity  Why  Cost-effectiveness  Keep CDNs for peaks  Transit lines  Peering  Billing model  95% / Flat rate
  8. 8. How to utilize own network  Basic decision logic  Peering first  Transit second  CDNs last  Cache layer  Varnish  Monitoring  POPs, Edge clusters
  9. 9. Viewer side switching - QoS  List all providers  In case of problem – switch  Challenges:  How to detect a problem  Sync  Broadcaster side problems
  10. 10. Server side prediction  Initial guess for QoS  Minimize useless switches (UX)  Cost optimization  How?  Viewers report metrics  Server side processing and aggregation
  11. 11. UMS  Ustream Media Server  Continuous connection to all clients  Real-time push-based updates  First use: viewer number display  Channel status poll  Viewer authentication and authorization  Stream information  Java, kernel tweaks
  12. 12. UMS  Frontend layer  Up to 150.000 clients / machine  3 IP addresses per machine  1.5 billion outgoing messages per machine per day (10 billion for the live cluster)  Aggregation / logic layer  Current: 400.000 clients per media limit  Future: distributed
  13. 13. The data source: Streamstat  Viewer reports  UMS process  Log  Historic data processing  From log  Hadoop + Tableau  Realtime processing in UMS  Chandra {"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPD ATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtoc ol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamPr otocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc- uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"} {"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_U PDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","netwo rkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241, "url":"http://uhs-akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117 ,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs- akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214 ,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs- akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575," streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs- level3.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"} {"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UP DATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProto col":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamPr otocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams- uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}
  14. 14. Streamstat reports  Types of reports  Playing  Buffering  Bandwidth  Errors, QoS events  Verticals  Geo  AS number  Media
  15. 15. Chandra  Real-time data aggregation layer  Replaceable data store  Redis  Simple logic: based on increments  Easily scalable
  16. 16. Chandra demo
  17. 17. Decisions  Ways of routing viewers  Initial setup  Updates  Stream priority  Stream module  Ways of decision  Rule-based  Dynamic
  18. 18. Stream module generation flow  Filter the Stream Priority list  Dedications / whitelists
  19. 19. Stream module generation flow  Filter the Stream Priority list  Dedications / whitelists
  20. 20. Stream module generation flow Rule blocking UCDN
  21. 21. Routing on own network  Provider resolution service  Reject to CDN if full  Capacity monitoring  Lines  Clusters
  22. 22. What we have now  3 big CDN providers  ~5 smaller providers  UMS clusters  Core in SJC  POPs: NRT, AMS, BUD  Exchanges  QoS and static rules
  23. 23. Future plans  Growth   Dynamic rules  QoS and MBR  UMS Elastic Logic  Open sourcing  Framework  Chandra
  24. 24. Questions syntaxerror@ustream.tv

×