(c) WebKit Contributors Meeting 2023 October 2023, 24 to 25 Cupertino - California (USA) https://webkit.org/meeting

1. WebRTC in WPE/GTK ports:
Current status and challenges
WebKit Contributors Meeting 2023
Cupertino (CA), 24-25 October 2023
Philippe Normand
1

2. Outline
Brief summary of the timeline (libwebrtc, ...)
Current status (what's working, how, ...)
Challenges
2

3. WebRTC in WebKit, the timeline
3

4. 2015 to 2017: OpenWebRTC era
Work group lead by Dr Alex Gouaillard
OpenWebRTC maintained by Ericsson
Involvement of several GStreamer consultants
Integration in WebKitGTK / WPE by Igalia
Monolithic "black box", providing a custom API
Didn't have hw/accelerated support for Android/iOS
4

5. 2017 to 2019: LibWebRTC attempts
LibWebRTC bundled in WebKit for Apple WebKit ports
Integration with GStreamer encoders
Hardware accelerated support requiring custom decoder/sinks
BoringSSL licensing issues (SSLeay & OpenSSL licenses)
Complexity of LibWebRTC (tarballs size)
5

6. 2019 onwards: GstWebRTC
GstWebRTC announced at GstConf 2017
Early WebKit experimentation already around November 2017
Initial GstWebRTC backend upstreamed in 2020
6

7. WebRTC in WebKit,
current status
7

8. Stream capture (getUserMedia)
Permission handling deferred from UIProcess to WebProcess
IPC specific to WPE/GTK ports
Not easily testable with current tests infrastructure
One pipeline per capture device (Camera, Microphone)
8

9. Stream capture (getDisplayMedia)
Two layer of permissions:
1. WebKit Permission Request
2. Desktop portal (third-party) over IPC (DBus)
Desktop portal permissions can be persisted
Capture pipeline connects to PipeWire, gets frames as DMABufs
9

10. MediaStream handling
webkitmediastreamsrc GStreamer source element acting as Observer for:
Capturer pipelines
Canvas / <video> elements
Incoming WebRTC MediaStreams
Each MediaStreamTrack maps to a source GstPad
10

11. PeerConnection handling (1/2)
11

12. PeerConnection handling (2/2)
webrtcbin implements the JavaScript Session Establishment Protocol (JSEP)
Almost seamless integration with WebKit's PeerConnection infrastructure
GstPromise hand in hand with WebKit's Promise<T>
12

13. Incoming MediaStream playback
In JS: videoElt.src = mediaStream
=> In WebCore::MediaPlayerPrivateGStreamer: playbin3
uri=mediastream://uuid
==> Internally hooks the MediaStream to webkitmediastreamsrc
13

14. Additional WebRTC APIs
Statistics
Queried by WebCore::GStreamerStatsCollector to webrtcbin
Some informations can't be filled by webrtcbin
frames-decoded, frames-encoded, bitrate, ... filled in by WebKit
DataChannel, GStreamerDataChannelHandler maps fairly well to
GstWebRTCDataChannel 🤗
14

15. Implementation focus: Tests
DIRECTORY TOTAL PASS SKIP FLAKY FAIL
fast/mediacapturefromelement 11 6 4 1 0
fast/mediastream 192 134 16 23 19
http/tests/webrtc 9 5 2 2 0
webrtc 131 58 29 30 14
Almost 60% passing tests
WPT tests not covered yet
15

16. Implementation focus: End-users
Cloud-gaming: Amazon Luna 🥳
User-Agent quirk
Legacy WebRTC ( offerToReceiveAudio, offerToReceiveVideo)
Chrome-specific specs ( keyboard-lock)
A/V conferencing: Jitsi 🥺
Not working yet
Requires features we don't support yet (renegotiation, simulcast, at least)
=> Every platform requires a dedicated bring-up => Being spec-compliant is the tip of
the iceberg
16

17. WebRTC in WebKit,
the challenges ahead
17

18. Video encoding
Simulcast (GstWebRTC has some support for that, not used by WebKit yet)
Temporal scalability
Spatial scalability
https://www.w3.org/TR/webrtc-svc/
18

19. Network sandboxing
The current streaming pipeline runs in WebProcess (BAD!)
Ideally the network usage should be restricted to the NetworkProcess
GstWebRTC now supports custom ICE implementations
Plan: Leverage this new infrastructure, confine the ICE Agent to NetworkProcess
19

20. Capture devices sandboxing
The current capturer pipelines run in WebProcess (BAD!)
Tentative plan:
GTK/Desktop: Interact with Camera portal => Improved permissions handling
WPE/Embedded: Allow a fallback when user input is lacking(?)
GTK/WPE: Move all GStreamer pipelines to GPUProcess
20

21. Missing features backlog
ICE candidates filtering
SFrame encryption
Improve stats coverage
Transceiver direction changes
(DTMF? Not sure who uses that?)
21

22. Thanks. Any question(s)?
22

23. 23