Our CEO Varun Singh presented at ClueCon 2018 on Real-time Communications Monitoring 101. https://www.cluecon.com/

1. Varun Singh
CEO, @callstatsio
twitter: @vr000m

2. About Me
• CEO, callstats.io
• Raised $4M+ in Venture Funding
• ACM MM 2016 awarded Ph.D. Special Mention
• Standards Contributor: IETF, W3C, 3GPP
2

3. 3

4. http://geek-and-poke.com/
4

5. It is not all about throughput*
… it is about latency†! *)capacity
bandwidth
speed
audio only call
5
†)delay,
one-way delay
time (sec)

6. Measuring User Experience
6
collect metrics and user actions
CallExperience

7. Measuring User Experience
7
measure setup time
CallSetup
CallExperience

8. Measuring User Experience
8
Ask the right question!
CallSetup
CallExperience
CallFeedback

9. Measuring User Experience
9
Pageload
CallInit(context)
CallSetup
Call
Experience
callfeedback
we collect metrics
across the media pipeline
network events, and user actions

10. What to Measure?
• Network metrics
• bits per second, RTT, jitter, packet losses, …
• Multimedia pipeline metrics
• playout delay, frames metrics, …
• Quality Models from metrics
• Annoyances
• resolution/frame rate changes, interface changes, …
• failures (NATs, insufficient capacity for carrying media, …)
• user feedback
10

11. Measuring User Experience
11
Alice
Greg
Is every stream from every participant important?

12. Measuring User Experience
12
Alice
Greg
Speaking
It depends on
who is speaking
who is viewing which stream
Speaking

13. Measuring User Experience
13
Alice
Greg
Speaking
The quality is an estimate of
what is "said/captured" and
what is "heard/rendered"

14. Measuring User Experience
14
Alice
Greg
Speaking
Connectivity
Disrupted
The quality is an estimate of
what is "said/captured" and
what is "heard/rendered"

15. Measuring User Experience
15
Alice
Greg
Connectivity
Disrupted
disruption
cross-talk
chaos!

16. Measuring User Experience
16
Alice
Greg
Speaking
Alice
Greg

17. Disruption: loss of connectivity when
network interfaces change, low available
capacity,or high delay
The light grey vertical lines show disruption, highlighted by
the red bounding boxes.17
Disruptions

18. Anticipating User Behaviour
User Behaviour: The user tries to
correct for the disruption by
turning on and off video
18

19. User Feedback
19
User feedback: the user provides
feedback on audio/video and
connection.

20. Frame Stability
20

21. Some observations
21

22. network.callstats.io
22
Like fast.com
but for WebRTC
Pre-call Test
runs against our
TURN Service

23. 0%
20%
40%
60%
80%
100%
0 100 200 300 400 500 600 700 800 900 1000
P2P TURN Bridge
Round-trip time
23
95 percentile RTT for each session
RTT < 240ms
80+% of P2P
70% of Bridge

24. 0%
20%
40%
60%
80%
100%
0 100 200 300 400 500 600 700 800 900 1000
P2P TURN Bridge
RTT distributions
24
95 percentile RTT for each session
What happened
here?

25. Can we predict this before a call?
25
https://www.callstats.io/industry-reports/
90% of P2P
the conference
performs better
than PCT

26. Can we predict this before a call?
26
https://www.callstats.io/industry-reports/
70% of the cases,
Infra performance
is worse!

27. Why is it important
27
Bad Excellent

28. Is the infra placed incorrectly?
• Infra —> TURN Server, MCUs, SFUs, etc.
• We do pre-call tests before the call
• against our globally available TURN servers
• does a loopback test with our servers
28

29. Hairpinning Metric
• How big is the problem?
• Where is it occurring
• Number of sessions hair pinned
29

30. How big is the problem?
• 30% in Asia and Oceania respectively
• 50% in South America,
• 10% of the sessions on most continents are hair pinned
• Even Africa?
• Yes, Predominant P2P (90%)
30

31. 31

32. 32

33. 33

34. 34

35. 35
RTT when the
infrastructure
is not on the
same continent

36. Summary (1/4)
• Chrome with derivatives (e.g., Electron, Opera, etc) dominates
• Platforms, desktop it is windows, iOS for mobile
• There is significant hairpinning in most services
• Browsers and Services are improving
36

37. Summary (2/4)
• Call Setup Failures: ~2%
• Setup time: 70% sessions setup in < 0.5s
• User Feedback: 89% find it acceptable
• Objective Quality: 88% is fair
37

38. Summary (3/4)
• Average RTT is lower for P2P to bridge.
• Median RTT
• peer-to-peer is 70ms
• bridge is 140ms.
• Fractional loss is lower for the bridge compared to P2P
• No 95-%le Fractional loss
• 90% bridge, 84% P2P
38

39. Summary (4/4)
• Media quality for:
• Desktop sessions over Electron are on average 2.0 (excellent)
compared to browsers 1.5 (fair)
• Mobile is pretty consistent on average 2.0
• Intra-continental sessions on average are 1.5.
• Inter-continental sessions on average are 1.0 (Bad).
39