Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Building our own CDN
How to serve millions of viewers
efficiently
Zoltán Németh
at a
glance
Live streaming
Free broadcastingSoftware as a Service
Big events
• Sony Playstation
• Nintendo
• Lady Gaga
• F...
Streaming  Definitions
 Broadcaster, viewer
 Media
 Buffering
 History
 FMS, RTMP
 Edge
 UHS, HTTP
How we use the
network
 Live streaming properties
 Continuous connection vs chunk
download
 Big bandwidth
 Stability c...
First approach:
CDN
 Pros
 Scaling, flexibility
 Bigger capacity
 Cons
 Costs
 Origin requests
 Limitations
 Regio...
Multiple CDNs  Global coverage solved
 Costs high
 Unified configuration problem
 How to select which one to use
Own capacity  Why
 Cost-effectiveness
 Keep CDNs for peaks
 Transit lines
 Peering
 Billing model
 95% / Flat rate
How to utilize own
network
 Basic decision logic
 Peering first
 Transit second
 CDNs last
 Cache layer
 Varnish
 M...
Viewer side
switching - QoS
 List all providers
 In case of problem – switch
 Challenges:
 How to detect a problem
 S...
Server side
prediction
 Initial guess for QoS
 Minimize useless switches (UX)
 Cost optimization
 How?
 Viewers repor...
UMS  Ustream Media Server
 Continuous connection to all clients
 Real-time push-based updates
 First use: viewer numbe...
UMS  Frontend layer
 Up to 150.000 clients / machine
 3 IP addresses per machine
 1.5 billion outgoing messages per
ma...
The data source:
Streamstat
 Viewer reports
 UMS process
 Log
 Historic data processing
 From log
 Hadoop + Tableau
...
Streamstat reports  Types of reports
 Playing
 Buffering
 Bandwidth
 Errors, QoS events
 Verticals
 Geo
 AS number...
Chandra  Real-time data aggregation layer
 Replaceable data store
 Redis
 Simple logic: based on increments
 Easily s...
Chandra demo
Decisions  Ways of routing viewers
 Initial setup
 Updates
 Stream priority
 Stream module
 Ways of decision
 Rule-...
Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists
Stream module
generation flow
 Filter the Stream Priority list
 Dedications / whitelists
Stream module
generation flow
Rule blocking UCDN
Routing on own
network
 Provider resolution service
 Reject to CDN if full
 Capacity monitoring
 Lines
 Clusters
What we have
now
 3 big CDN providers
 ~5 smaller providers
 UMS clusters
 Core in SJC
 POPs: NRT, AMS, BUD
 Exchang...
Future plans  Growth 
 Dynamic rules
 QoS and MBR
 UMS Elastic Logic
 Open sourcing
 Framework
 Chandra
Questions
syntaxerror@ustream.tv
Building our own CDN
Upcoming SlideShare
Loading in …5
×

Building our own CDN

1,784 views

Published on

Published in: Internet, Technology, Business
  • Be the first to comment

Building our own CDN

  1. 1. Building our own CDN How to serve millions of viewers efficiently Zoltán Németh
  2. 2. at a glance Live streaming Free broadcastingSoftware as a Service Big events • Sony Playstation • Nintendo • Lady Gaga • Festivals Citizen journalism • Rescue of Chilean miners • Earthquake in Japan • Revolutions in Egypt, Syria • Protests in Ukraine • Obama campaign • Climate Reality • Football • Justin Bieber Concurrent viewership peaks at almost 2 million
  3. 3. Streaming  Definitions  Broadcaster, viewer  Media  Buffering  History  FMS, RTMP  Edge  UHS, HTTP
  4. 4. How we use the network  Live streaming properties  Continuous connection vs chunk download  Big bandwidth  Stability critical  Large audience  Global delivery  Providers performance  Mobile networks
  5. 5. First approach: CDN  Pros  Scaling, flexibility  Bigger capacity  Cons  Costs  Origin requests  Limitations  Regional and provider differences  Capacity race conditions
  6. 6. Multiple CDNs  Global coverage solved  Costs high  Unified configuration problem  How to select which one to use
  7. 7. Own capacity  Why  Cost-effectiveness  Keep CDNs for peaks  Transit lines  Peering  Billing model  95% / Flat rate
  8. 8. How to utilize own network  Basic decision logic  Peering first  Transit second  CDNs last  Cache layer  Varnish  Monitoring  POPs, Edge clusters
  9. 9. Viewer side switching - QoS  List all providers  In case of problem – switch  Challenges:  How to detect a problem  Sync  Broadcaster side problems
  10. 10. Server side prediction  Initial guess for QoS  Minimize useless switches (UX)  Cost optimization  How?  Viewers report metrics  Server side processing and aggregation
  11. 11. UMS  Ustream Media Server  Continuous connection to all clients  Real-time push-based updates  First use: viewer number display  Channel status poll  Viewer authentication and authorization  Stream information  Java, kernel tweaks
  12. 12. UMS  Frontend layer  Up to 150.000 clients / machine  3 IP addresses per machine  1.5 billion outgoing messages per machine per day (10 billion for the live cluster)  Aggregation / logic layer  Current: 400.000 clients per media limit  Future: distributed
  13. 13. The data source: Streamstat  Viewer reports  UMS process  Log  Historic data processing  From log  Hadoop + Tableau  Realtime processing in UMS  Chandra {"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPD ATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtoc ol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamPr otocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc- uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"} {"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_U PDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","netwo rkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241, "url":"http://uhs-akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117 ,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs- akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214 ,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs- akamai.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575," streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs- level3.ustream.tv/ams/ams- uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"} {"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UP DATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProto col":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamPr otocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams- uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static- cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}
  14. 14. Streamstat reports  Types of reports  Playing  Buffering  Bandwidth  Errors, QoS events  Verticals  Geo  AS number  Media
  15. 15. Chandra  Real-time data aggregation layer  Replaceable data store  Redis  Simple logic: based on increments  Easily scalable
  16. 16. Chandra demo
  17. 17. Decisions  Ways of routing viewers  Initial setup  Updates  Stream priority  Stream module  Ways of decision  Rule-based  Dynamic
  18. 18. Stream module generation flow  Filter the Stream Priority list  Dedications / whitelists
  19. 19. Stream module generation flow  Filter the Stream Priority list  Dedications / whitelists
  20. 20. Stream module generation flow Rule blocking UCDN
  21. 21. Routing on own network  Provider resolution service  Reject to CDN if full  Capacity monitoring  Lines  Clusters
  22. 22. What we have now  3 big CDN providers  ~5 smaller providers  UMS clusters  Core in SJC  POPs: NRT, AMS, BUD  Exchanges  QoS and static rules
  23. 23. Future plans  Growth   Dynamic rules  QoS and MBR  UMS Elastic Logic  Open sourcing  Framework  Chandra
  24. 24. Questions syntaxerror@ustream.tv

×