Presentation @ CommConUK Video @ https://www.youtube.com/watch?v=qcSwmdbArvs

1. Advanced WebRTC
Testing
July 2019

2. Dr. “Alex” Gouaillard
（アレックス ）
2017 – CEO – CoSMo Software (SG)
2015 – Principal Architect – Citrix (SG)
2012 – CTO - Temasys Communications (SG)
2009 – Lab Head - A*STaR (SG)
2007 – Faculty - Caltech then Harvard (USA)
2005 – Senior R&D Eng. - Sun Microsystems ERI (PRC)
2005 – PhD (Fr) EE - 3D Images and Video Processing
2005 – PhD (Jp) CS/CG - Discrete Geometry Processing
2001 – M.Eng. (Signal and Image processing)
2001 – M.Sc. (Applied Math)
Hobbies: languages (more and more)
-- Professional fluency: French, English, Japanese,
-- Everyday life: German, Chinese,
-- Can understand / is learning: Thai, Tagalog, Catalan
“Incognito Mode”
(シークレット モード)

3. 2018~ Member AOMedia – Real-Time Codec Group
2015~ Co-Chair, WebRTC interoperability A.G., IMTC
2015~ Webrtc in Webkit Initiative (webrtcinwebkit.com)
2012 ~ webRTC WG, ORTC CG, @ W3C
2012 ~ rtcweb, mmusic, payload, … @ IETF
2014 WebRTC Pioneer Award (Individual)
2014 WebRTC Expo - Best Tool Award: Puppet Master @ T.
2013 WebRTC Expo - Best Conferencing Award @ Temasys
2013 WebRTC Expo - Visionary Award @ Temasys
2013~ Corp. Contributor to webrtc.org @ Temasys
Dr. “Alex” Gouaillard
（アレックス ）
“Incognito Mode”
(シークレット モード)

4. Dr Ludovic Roux
Dr. Ludovic Roux
2017 – Chief Scientist Officer - CoSMo SCPL
2015 – Deputy-CTO - Temasys Communications
2014 – Researcher Agency For Science Technology & Research (SG)
2008 – Thales Training & Simulation (France)
2007 – Thales Communications (France)
1999 – JAXA – EORC Earth Observation Research Center (Japan)
1997 – PhD (France) Data fusion, classification of satellite images
1993 – M.Sc. (France) Image and Language Processing
1990 – B.Sc. (France) Computer Science
Languages:
– Professional fluency: French, English
-- Everyday life: JapaneseHobbies: Skydiving

5. Our vision and moto
We like to give back to grow the community / ecosystem:
- We publish most of our results for everybody to use freely
- We involve other open source community projects in our advances, share the love
- We contribute to other open source projects even if they ”compete”:
- First fully functional internet explorer plugin: FREE
- H.264 Simulcast implementation: Contributed to all Browsers
- KITE testing Engine: FREE
- We spend the money to attend all standard committee meetings to make sure the
small use cases have a voice at the table
- We create and finance projects to push the tech adoption forward: webrtcinwebkit
- We organize Hackathon so that browser vendors, SFU vendors, and anybody can
finally exchange around WebRTC in an informal and practical manner.
- …

6. Publications
1. A Lemesle, A Marion, L Roux, A Gouaillard, “NARVAL a no-reference video quality tool for real-time
communications”, Proc. of Human Vision and Electronic Imaging, 2019
2. E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source
SFUs for Video Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with
Kurento, Meetecho, MediaSoup, Medooze, Jitsi
3. L. Roux and A. Gouaillard, “Real-time communication testing evolution with WebRTC 1.0”, 2017 Principles,
Systems and Applications of IP Telecommunications (IPTComm)
4. Grozev, Boris & Ivov, Emil & Budkiewicz, Arnaud & Roux, Ludovic & Gouaillard, Alexandre, “PERC double
media encryption for WebRTC 1.0 sender simulcast”. 2017 Principles, Systems and Applications of IP
Telecommunications (IPTComm). In collaboration with Jitsi.
5. Amirante, A and Castaldi, T and Gouaillard, A and Miniero, Lorenzo and G. Murillo, S and Romano, Simon
Pietro , “Bringing privacy to the Janus WebRTC server: The PERC way“ 2017 Principles, Systems and
Applications of IP Telecommunications (IPTComm). In collaboration with Meteecho.

7. Early WebRTC Testing – Pre-KITE
L. Roux and A. Gouaillard, “Real-time communication testing evolution with WebRTC 1.0”,
2017 Principles, Systems and Applications of IP Telecommunications (IPTComm)
WebRTC brings first P2P api to the browsers!
Existing tool not applicable.
Pioneers: Kurento => ElasTests
Very basic problems:
1. it does not connect (ICE, network, …)
2. I cannot differentiate between ICE errors and DTLS errors
3. It seems to connect but media is not flowing

8. WebRTC Interop Testing – KITE
L. Roux and A. Gouaillard, “Real-time communication testing evolution with WebRTC 1.0”,
2017 Principles, Systems and Applications of IP Telecommunications (IPTComm)
Very basic problems:
1. it does not connect (ICE, network, …)
2. I cannot differentiate between ICE errors and DTLS errors
3. It seems to connect but media is not flowing
At one point most browsers made it work with themselves but
the “divorced parents” syndrome hit us/them.
Interoperability

9. WebRTC Interop Testing – KITE

10. WebRTC Interop Testing – KITE

11. WebRTC Interop Testing – KITE

12. KITE Architecture
Dashboard
reporting and
output
analysis
Engine Design
Test Conductor Test Dispatcher
Test Runner
Test Runner
Test Runner
Test Runner
Test Runners
Client configuration,
and test matrix management.
Load Balancing, and managing
test retries.
Run tests, collect and
output RAW results
Test
Setup
and
Config
WebRTC Interop Testing – KITE

13. KITE: Karoshi Interoperability Testing Engine
The effortless way to test WebRTC compliance,
prevent Karoshi with KITE!
• Open-source testing engine for WebRTC – Apache 2.0 license
• Available at github.com/webrtc/KITE
KITE can be setup on Windows, Mac or Linux. The installation
process only takes 10 to 15 minutes.
Write automated interoperability test scripts in Java or Javascript
and run them on any platforms. KITE supports:
• all web browser: Chrome, Firefox, Safari, Edge, Opera... on all OS (Linux,
Windows, Mac, iOS and Android)
• Mobile Native Apps on Android, iOS
• Desktop Native Apps on Windows and MacOS
• Electron Apps
WebRTC Interop Testing – KITE

14. Native Apps Testing
• Native App automated testing for WebRTC can prove quite time
consuming at first
• To help people get quickly started on the right track, we can
provide Selenium Grid setup guides and scripts, sample tests and
tutorials to test Native Apps on the following platforms:
• Windows 10
• MacOS
• Android
• iOS
• We also have an extensive experience
writing automated tests for Electron
apps
WebRTC Interop Testing – KITE

15. Sample Tests and Tutorials
Additional free WebRTC sample tests are
available github.com/CoSMoSoftware/KITE-Sample-Tests
The sample tests cover all the major open-source SFUs demos:
• Jisti
• Janus
• Kurento/openvidu
• Mediasoup
• Medooze (Simulcast demo)
Tutorials: Writing a simple KITE Test for Jitsi in Java and/or
Javascript
Once you've completed the installation and run the sample tests, please try to write
your own KITE test following our tutorials.
• KITE Test for Jitsi in Java
• KITE Test for Jitsi in Javascript
WebRTC Interop Testing – KITE

16. Test Reports
WebRTC Interop Testing – KITE

17. Test Reports

18. WebRTC Load Testing - KITE

19. Load Testing
Highly Scalable
• Simulate up to 250,000 clients
• Run test for unlimited time (days)
 Integrates with your own cloud, you pay directly to
AWS your usage cost
 Very cost effective by using Spot Instances (<$0.01
to $0.07/h per VM)
Server-side System Metrics
Monitor your system in real-time during the test for
network traffic, CPU & memory utilization, disk I/O
Client-side WebRTC Statistics
Collect statistics from all the clients during the
full duration of the test and analyze foreach audio
and video streams, tracks, filtering by Server,
Meeting ID, User ID, Clients:
 in/out bitrates, packet losses,
 jitter, frame rate, RTT, codec, resolution
WebRTC Load Testing - KITE

20. WebRTC Load Testing - KITE
E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source SFUs for Video
Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with Kurento, Meetecho, MediaSoup, Medooze, Jitsi

21. WebRTC Load Testing - KITE
E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source SFUs for Video
Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with Kurento, Meetecho, MediaSoup, Medooze, Jitsi

22. WebRTC Load Testing - KITE
E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source SFUs for Video
Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with Kurento, Meetecho, MediaSoup, Medooze, Jitsi
https://www.3cx.com/blog/news/web-meeting-performance/

23. WebRTC Real Time Testing probes

24. WebRTC Real Time Testing probes
A Lemesle, A Marion, L Roux, A Gouaillard, “NARVAL a no-reference video quality tool for real-time communications”, Proc. of
Human Vision and Electronic Imaging, 2019

25. WebRTC Real Time Testing probes
A Lemesle, A Marion, L Roux, A Gouaillard, “NARVAL a no-reference video quality tool for real-time communications”, Proc. of
Human Vision and Electronic Imaging, 2019

26. Network Instrumentation Use Case
Once basic connection and interoperability
problems have been solved, you go from
• a binary world (works/doesn’t work)
• to 50 shades of grey ….
or maybe just 5 shades 

27. Network Instrumentation Use Case
How to test Video Adaptation and how fast it happens in
a deterministic and SFU agnostic way as to compare them
without bias?
How to change and measure the impact of different
flavors of the following algorithms families?
- BWE, (sender side? receiving side? ….)
- CC, (REMB, TIMMBR, Transport-wide-CC, Goog-cc, BBR, ….)
- Simulcast, (degradation preference, layer switching,
key frames, …)
- SVC, (same as above)
- jitter buffer (depth, ….)
Key to UX: Perceived video quality and network adaptation

28. Network Instrumentation Use Case
In other words: How do I replicate this experiment,
automatically, for any SFU:

29. Network Instrumentation
• Network conditioning
can be done on the
clients side, for
each client
independently
• Control bandwidth,
packets loss,
latency, and jitter.
• Simulate Firewall and
NATs
• on-premises or hosted
• native or web apps
• Desktop or mobile

30. Network Instrumentation
• A video is worth 1,000 words.

31. How do you make money?
Short answer: pretty much like most open-source
projects do
• Expertise (professional services)
• core is fully featured, open source and free,
• advanced modules are commercial
• Tools and advanced GUIS that save you time, are
commercial
• Think Red Hat, NGINX, ….
No dual-license. IP protection through

32. Grid Manager Workflow
Writing a test
script
Setup the grid Uploading test
script
Running the
test
and lastly..
viewing the
test result
AND
dashboard

33. Setup your grid

34. Full control over the grid configuration

35. Upload and run your tests

36. Reporting and Dashboard
Access Allure Reports and Kibana Dashboard from the Grid
Manager GUI