Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ClueCon 2018: Real-time Communications Monitoring 101 by Varun Singh

19 views

Published on

Our CEO Varun Singh presented at ClueCon 2018 on Real-time Communications Monitoring 101. https://www.cluecon.com/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

ClueCon 2018: Real-time Communications Monitoring 101 by Varun Singh

  1. 1. Varun Singh
 CEO, @callstatsio twitter: @vr000m
  2. 2. About Me • CEO, callstats.io • Raised $4M+ in Venture Funding • ACM MM 2016 awarded Ph.D. Special Mention • Standards Contributor: IETF, W3C, 3GPP 2
  3. 3. 3
  4. 4. http://geek-and-poke.com/ 4
  5. 5. It is not all about throughput* … it is about latency†! *)capacity bandwidth speed audio only call 5 †)delay, 
 one-way delay time (sec)
  6. 6. Measuring User Experience 6 collect metrics and user actions CallExperience
  7. 7. Measuring User Experience 7 measure setup time CallSetup CallExperience
  8. 8. Measuring User Experience 8 Ask the right question! CallSetup CallExperience CallFeedback
  9. 9. Measuring User Experience 9 Pageload CallInit(context) CallSetup Call Experience callfeedback we collect metrics across the media pipeline network events, and user actions
  10. 10. What to Measure? • Network metrics • bits per second, RTT, jitter, packet losses, … • Multimedia pipeline metrics • playout delay, frames metrics, … • Quality Models from metrics • Annoyances • resolution/frame rate changes, interface changes, … • failures (NATs, insufficient capacity for carrying media, …) • user feedback 10
  11. 11. Measuring User Experience 11 Alice Greg Is every stream from every participant important?
  12. 12. Measuring User Experience 12 Alice Greg Speaking It depends on 
 who is speaking
 who is viewing which stream Speaking
  13. 13. Measuring User Experience 13 Alice Greg Speaking The quality is an estimate of what is "said/captured" and what is "heard/rendered"
  14. 14. Measuring User Experience 14 Alice Greg Speaking Connectivity Disrupted The quality is an estimate of what is "said/captured" and what is "heard/rendered"
  15. 15. Measuring User Experience 15 Alice Greg Connectivity Disrupted disruption cross-talk chaos!
  16. 16. Measuring User Experience 16 Alice Greg Speaking Alice Greg
  17. 17. Disruption: loss of connectivity when network interfaces change, low available capacity,or high delay The light grey vertical lines show disruption, highlighted by the red bounding boxes.17 Disruptions
  18. 18. Anticipating User Behaviour User Behaviour: The user tries to correct for the disruption by turning on and off video 18
  19. 19. User Feedback 19 User feedback: the user provides feedback on audio/video and connection.
  20. 20. Frame Stability 20
  21. 21. Some observations 21
  22. 22. network.callstats.io 22 Like fast.com but for WebRTC
 Pre-call Test runs against our TURN Service
  23. 23. 0% 20% 40% 60% 80% 100% 0 100 200 300 400 500 600 700 800 900 1000 P2P TURN Bridge Round-trip time 23 95 percentile RTT for each session RTT < 240ms 80+% of P2P 70% of Bridge
  24. 24. 0% 20% 40% 60% 80% 100% 0 100 200 300 400 500 600 700 800 900 1000 P2P TURN Bridge RTT distributions 24 95 percentile RTT for each session What happened here?
  25. 25. Can we predict this before a call? 25 https://www.callstats.io/industry-reports/ 90% of P2P the conference performs better than PCT
  26. 26. Can we predict this before a call? 26 https://www.callstats.io/industry-reports/ 70% of the cases, Infra performance is worse!
  27. 27. Why is it important 27 Bad Excellent
  28. 28. Is the infra placed incorrectly? • Infra —> TURN Server, MCUs, SFUs, etc. • We do pre-call tests before the call • against our globally available TURN servers • does a loopback test with our servers 28
  29. 29. Hairpinning Metric • How big is the problem? • Where is it occurring • Number of sessions hair pinned 29
  30. 30. How big is the problem? • 30% in Asia and Oceania respectively • 50% in South America, • 10% of the sessions on most continents are hair pinned • Even Africa? • Yes, Predominant P2P (90%) 30
  31. 31. 31
  32. 32. 32
  33. 33. 33
  34. 34. 34
  35. 35. 35 RTT when the infrastructure is not on the same continent
  36. 36. Summary (1/4) • Chrome with derivatives (e.g., Electron, Opera, etc) dominates • Platforms, desktop it is windows, iOS for mobile • There is significant hairpinning in most services • Browsers and Services are improving 36
  37. 37. Summary (2/4) • Call Setup Failures: ~2% • Setup time: 70% sessions setup in < 0.5s • User Feedback: 89% find it acceptable • Objective Quality: 88% is fair 37
  38. 38. Summary (3/4) • Average RTT is lower for P2P to bridge. • Median RTT • peer-to-peer is 70ms • bridge is 140ms. • Fractional loss is lower for the bridge compared to P2P • No 95-%le Fractional loss • 90% bridge, 84% P2P 38
  39. 39. Summary (4/4) • Media quality for: • Desktop sessions over Electron are on average 2.0 (excellent) compared to browsers 1.5 (fair) • Mobile is pretty consistent on average 2.0 • Intra-continental sessions on average are 1.5. • Inter-continental sessions on average are 1.0 (Bad). 39

×