Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2016 Streaming Media West - How to improve video quality of Service with an Effective AB testing Workflow

278 views

Published on

This presentation covers how broadcasters can improve quality of service by making small changes to their ABR algorithms and analyzing via an effective AB testing workflow. We offer practical feedback on how Dailymotion was able to improve QoS significantly via small player-side ABR iterations, explaining what metrics were used and the results obtained. We also delve into how to build an effective AB testing analytics stack, honing in on the tools needed and the appropriate environment to be able effectively analyze our actions and make changes in real time, using the example of Streamroot.

Published in: Engineering
  • Boost your brainpower with brain pill! find out more... ◆◆◆ https://bit.ly/2GEWG9T
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Doctor's 2-Minute Ritual For Shocking Daily Belly Fat Loss! Watch This Video ●●● https://tinyurl.com/y6qaaou7
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Do This Simple 2-Minute Ritual To Loss 1 Pound Of Belly Fat Every 72 Hours ▲▲▲ http://scamcb.com/bkfitness3/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Doctor's 2-Minute Ritual For Shocking Daily Belly Fat Loss! Watch This Video  http://ishbv.com/bkfitness3/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

2016 Streaming Media West - How to improve video quality of Service with an Effective AB testing Workflow

  1. 1. How to improve quality of service with player-side iterations and an effective AB testing workflow Streaming Media West – Track D Tuesday, November 1, 2016 2:45 to 3:30 pm
  2. 2. ● Creators of peer-accelerated video delivery for large-scale OTT ● Pioneers in hybrid video delivery systems to accompany exponential growth ● Experts in HTML5 video through integrations into open-source & proprietary players Infinite scale, limitless delivery. Streamroot: Who are we?
  3. 3. I. How to build an AB Testing workflow When and what to test, analytics pipeline, AB testing setup II. Dailymotion use case: ABR iterations for quick, dramatic QoS improvements ABR iterations to hls.js to cut rebuffering III. Streamroot use case: using effective AB testing to improve our product Streamroot workflow and example Presentation Outline Infinite scale, limitless delivery.
  4. 4. Infinite scale, limitless delivery. AB Testing is Data-Driven Development. …here applied to Media engines & QoS. I. Building an effective AB testing workflow
  5. 5. Infinite scale, limitless delivery. I. Building an effective AB testing workflow AB testing makes sense when: You can’t test locally: distributed and unmanaged networks (ABR, P2P). Test subjective hypothesis, magic numbers. You deploy a completely new feature that breaks your old logic.
  6. 6. Infinite scale, limitless delivery. I. Building an effective AB testing workflow Before starting: Define your metrics Determine an appropriate sample size and testing timeframe Define your variables Define your populations
  7. 7. Infinite scale, limitless delivery. For testing, you need: The right data A DB to store it A BI/analytics tool I. Building an effective AB testing workflow
  8. 8. Infinite scale, limitless delivery. I. How to build: Data pipeline General workflow
  9. 9. Infinite scale, limitless delivery. Streamroot workflow I. How to build: Data Pipeline General workflow
  10. 10. Many options based on your requirements: open-source, proprietary, Cloud... Infinite scale, limitless delivery. I. How to build: Data Pipeline What to chose from Message broker / stream processor Storage + possible query/visualization BI/viz tools Advanced analytics Kafka (Kinesis) Druid Caravel R RabbitMQ InfluxDB Grafana Matlab ActiveMQ Cassandra (DynamoDB) Tableau PostGresSQL QlikView ElasticSearch Kibana
  11. 11. Infinite scale, limitless delivery. I. How to build: AB Testing workflow General setup
  12. 12. Client AB testing methods: 1) Dynamic Config Injection 2) Different builds with a reverse proxy Infinite scale, limitless delivery. I. How to build: AB Testing workflow General setup
  13. 13. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Config injection
  14. 14. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Config injection
  15. 15. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Dynamic config injection PROs Simple & easy splits Separate - fast to deploy & rollback Scalable CONs Limited scope Complications in code Need to handle scaling & high availability WARNING Async implementation!
  16. 16. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Different builds with reverse proxy
  17. 17. Infinite scale, limitless delivery. PROs Serve very different (major) versions of player Perfect for rolling out new features smoothly (10% then 30% then...) CONs Good CI/CD necessary to roll out new versions quickly Heavy stack to maintain and scale WARNING Make sure the file request is not slower! Single point of failure! I. How to build: AB Testing workflow Different builds with reverse proxy
  18. 18. 1. AB test when it makes sense. 1. Choose the right metrics. 1. Build a pipeline with tools that are adapted to your use case. 1. Use the right method depending on what you’re testing. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Tips & sum up
  19. 19. Infinite scale, limitless delivery. II. Dailymotion Some numbers: world’s 2nd largest video hosting platform 100 million video views each day 63% Desktop 31% Mobile 8% Tablet 1% SmartTV 69% HTML5 31% Flash
  20. 20. Infinite scale, limitless delivery. II. Dailymotion ABR iterations for quick, dramatic QoS improvements I should have rehearsed this more… When’s lunch?
  21. 21. Adaptive bitrate algorithms can have a huge impact on QoS. They are easy to tweak, but improvements depend on your specific use case. One size does not fit all. Infinite scale, limitless delivery. II. Dailymotion ABR iterations for quick, dramatic QoS improvements
  22. 22. Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic
  23. 23. Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic Introduce history parameter to bandwidth estimation.
  24. 24. Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic Average out the values.
  25. 25. https://github.com/dailymotion/hls.js/blob/master/src/controller/abr-controller.js EWMA: Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic Shaka Player:
  26. 26. ABR now based on two bandwidth moving averages: • Fast → adapt down quickly • Slow → adapt up more slowly Bandwidth estimation = min(fast, slow) Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic
  27. 27. Control over media engine Control over events QoS events AB testing & customization ABR algos, config, etc. Infinite scale, limitless delivery. CDN Director Analytics aggregation and visualization II. Dailymotion AB testing workflow: total control over their stack
  28. 28. 67.4% native 70.3% hls.js 78.6% flashls Infinite scale, limitless delivery. II. Dailymotion Comparing the results - live Rebuffering, percentile rank
  29. 29. native flashls hls.js,s=9,f=3hls.js,s=0,f=0hls.js,s=6,f=1 Infinite scale, limitless delivery. II. Dailymotion Comparing the results - live Rebuffering, percentile rank
  30. 30. hls.js, s=0, f=0 hls.js, s=9, f=3 Infinite scale, limitless delivery. II. Dailymotion Number of level switches - live
  31. 31. With the right workflow, big improvement in less than 2 weeks of testing. Numbers are use-case specific (segment length, network, etc.). Lots of other options to test → a continuous effort to improve QoS! Infinite scale, limitless delivery. II. Dailymotion Conclusions
  32. 32. Infinite scale, limitless delivery. III. Streamroot Peer-accelerated delivery
  33. 33. ...We have a very distributed network! AB testing on: - matching devices to get the best connections - Transfer algorithms and protocol improvements - QoS impact and improvements Infinite scale, limitless delivery. III. Streamroot Why are we using AB testing
  34. 34. Infinite scale, limitless delivery. III. Streamroot The pipeline GET build-a.js Pipeline AB testing File Distribution
  35. 35. AB Test on AskSegment Infinite scale, limitless delivery. III. Streamroot The test case
  36. 36. Infinite scale, limitless delivery. III. Streamroot Results AskSegment Timeout Samples P2P % Longer 15,772,542 67.41% Shorter 16,152,624 67.43% No effect?
  37. 37. Infinite scale, limitless delivery. III. Streamroot Results AskSegment Timeout Type Samples P2P % Longer Live 9,004,140 66.43% Shorter Live 9,340,504 68.02% Longer VOD 6,768,403 68.72% Shorter VOD 6,812,120 66.61% Live: +1.59% for shorter timeout VOD: +2.11% for longer timeout
  38. 38. - Understand and know your data - Ab testing is sometimes the only way to prove your hypothesis - You will be wrong sometimes: Make sure you can rollback quickly!! - The most difficult is to start! Once it is set up: be relentless! Continuous AB testing FTW Infinite scale, limitless delivery. III. Streamroot Takeaways
  39. 39. Infinite scale, limitless delivery. Questions?
  40. 40. Nikolay Rodionov, Co-Founder and CPO, nikolay@streamroot.io Erica Beavers, Head of Partnerships, erica@streamroot.io Infinite scale, limitless delivery. Contact us!

×