This document discusses the need for improved testing of real-time communication (RTC) features like WebRTC across different browsers, operating systems, and platforms. It introduces KITE (Karoshi Interoperability Testing Engine), a framework to automate interoperability testing of RTC by running tests in parallel across multiple browser and OS combinations. KITE uses the Selenium grid for browser automation and is configurable to support desktop and mobile browsers, native apps, IoT devices, and videoconferencing use cases beyond one-to-one calls. It aims to make RTC testing easier to set up and maintain.

1. Real-Time Communication
Testing Evolution
with WebRTC
Dr Alex Gouaillard
CoSMo Software Consulting
IMTC, W3C, IETF

2. RTC Interoperability: Web Apps
Chromecast

3. • W3C Standardization process (features supported or not)
• Stand-alone tests
• Single browser (one at a time)
(JS API) Compliance and Usual Testing
Test the Web Forward

4. What about WebRTC 1.0?
Why so low test coverage?
• Some parts of specifications were just written
• Tests not needed early on in standardization process
• Many tests need P2P communication / interop testing
Media devices APIs WebRTC 1.0 APIs
Test coverage as of November 2016 (W3C TPAC)

5. Ex. of RTC feature not Testable before:
Network and Protocol Testing: ICE
Client A Client B
STUN server
What is my public
IP address?
Your public IP address
is x.x.x.x:y

6. Compliance and Usual Testing vs.
Interoperability Testing
Compliance & Usual
Testing
Interoperability
Testing
• Interaction automation
• Stand-alone tests
• Single browser
• Interaction automation
• Two synchronized instances
of the same test
• 2+ browsers run in parallel
 New testing tools needed

7. Browsers Interaction Automation:
WebDriver protocol
ChromeDriver
geckodriver
Microsoft WebDriver
SafariDriver

8. Browsers as a Service:
hosted selenium nodes

9. In-Progress: WebDriver needs update
for WebRTC Security & others
• Permission prompt
– Geolocation
– Camera
– Microphone
– Bluetooth
– …
• Fake Media / Mock Capturer
• ICE Candidate Filtering
• Unsecure origin

10. Wrap-Up: WebDriver for RTC Tests
• Interaction
• Multiple browsers in parallel
• Synchronization of instances
of same test between browsers

11. Lots of tools out there
addressing a subset of the problem
WebRTC-test
Jitsi-Hammer
Kurento Testing Framework
Jattack

12. Ok, existing tools are limited but …
How bad can it be, really?
Rough Goal: Test all possible pairs of
[Browser] x [Browser Revision]
x
[OS] x [OS Revision]

13. WhatweWANTtotest
496
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

14. Whatwecouldtesttoday 115
23%
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
381
77%
1 OS
2 Oses
(Not tested)

15. © Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
6
5%
WhatweACTUALLYtesttoday
21
18%
88
77%
Tested
WIP
Not tested

16. Ok, it’s REALLY bad …
Get visibility: list and evaluate

17. Evaluation of Interop Testing Tools
Dimension 1: Tests Triggers and frequency
6 months
Weekly (2d)
Daily / Nightly (4h)
Commit-based and fast (20mn)
Experimental (manually triggered)

18. Evaluation of Interop Testing Tools
Dimension 2: Type of SE nodes I want to use
Self-Hosted, Pre-configured VM
Hosted, as-a-service (e.g. BrowserStack)
Self-Hosted, Ephemeral VM
Hosted, Ephemeral VM (e.g. Travis)
Local / physical machines
[Apple]
[Only stable browsers]
[Extensions and plug-ins]
[Modified browsers]
[no manual settings]
[Extensions and plug-ins]
[Apple]
[limited OS set]

19. Evaluation of Interop Testing Tools
Dimension 3: OS I want to support

20. Evaluation of Interop Testing Tools
Dimension 4: Which Browsers?
Easy, kind of tested already, great WebDriver support
No Apple VM on non-apple Hardware
Webdriver is more recent and less mature than others
WebRTC? What’s your flavor~~ ?
Permission prompt? Fake Media?
36+% of Enterprise market (Windows 7)
which plugin?

21. Evaluation of Interop Testing Tools
Dimension 5: How early do I want to test?
=> How fast do you want your fix in
Ex: chrome 58  chrome 61
Stable
Windows Insider
Windows Insider

22. Evaluation of Interop Testing Tools
Own cloud
On-premises
Dimension 6: Freedom, Convenience and Price
Hosted

23. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
testRTC
Hosted

24. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
Travis CI
Hosted

25. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
BrowserStack
Hosted

26. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
Sauce Labs
Hosted

27. Ok, I see what’s missing for each.
What’s the right design now?

28. KITE: Karoshi Interoperability
Testing Engine
Because for most of us, manually testing webrtc
interop felt like “death by overwork”

29. KITE: Karoshi Interoperability Testing Engine
Automating RTC P2P tests
• Generic (any app, any back-end)
• Reusable (any process, trigger, …)
• Easy to maintain (no death  )
Stable / Beta / Dev / Canary
+
Own cloudOn-premises Hosted

30. KITE
Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
KITE
Hosted
+
On-premises

31. KITE Design
Easy maintenance by divide and conquer
• Selenium Grid
• Test engine
• Test
• Dashboard

32. KITE Design
Conductor
Dispatcher
Test Runner Test Runner
Test Runner
Config => BrowserObj
Compute list< N-Tuple< BrowserObj > >
• { Test, Callback }
• list< N-Tuple< BO > >
handle concurrency
if resource: 1 N-tuple => 1 test runner =>1 thread
1 2 … N
• { Test, Callback }
• 1 N-Tuple< BO >
Instruments N BO
Run Test
Callback( results )
BrowserObj available
• Cr/FF - Mac - Rem-BB
• BrX - OSX - Rem-BaaS (SauceLabs)
• BrY - OSY - Rem-VM (AWS)
Result
Dashboard
• { N, Test, Callback }
• list< configs >

33. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}

34. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
Location of dashboard
to send tests results
"callback": "http://localhost:8080/kiteweb/datacenter",

35. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
Registration of
Selenium server
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],

36. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
List of tests
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],

37. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
List of browsers to
run the tests
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]

41. KITE Dashboard

42. KITE Dashboard

43. That’s very nice but it feels like
something is missing …

44. Global RTC Interoperability
Chromecast

45. Global RTC Interoperability:
MOBILE (Browsers)?
Chromecast

46. Global RTC Interoperability
Beyond Desktop Browsers
Appium
• Mobile testing framework
• For native, hybrid and mobile web apps

47. Global RTC Interoperability:
NATIVE, IoT and embedded
Chromecast
IoT
Electron

48. Global RTC Interoperability
Beyond Desktop Browsers
Electron
• Open source framework
• To build cross platform desktop applications
(HTML, CSS, JS)
• Combines Chromium and node.js into a single
runtime

49. Global RTC Interoperability:
Beyond 1-1
Jitsi
• Open source multiplatform
• Videoconferencing
• Instant messaging

50. Global RTC Interoperability:Beyond 1-1
Jitsi

51. Global RTC Interoperability
Make it easy: Grid Manager
• Set up a grid through a web app
• Set up Selenium hubs and nodes
• Download and set up
– Browsers
– Browsers WebDrivers
• Backup everything for follow-up uses
• Tear-down and clean-up after tests

52. Thank you!