In this lightning talk we will showcase the current support for the W3C WebCodecs spec, with GStreamer, in WebKit WPE and GTK ports! (c) GStreamer Conference 2023 September 25-26, 2023 A Coruña, Spain https://gstreamer.freedesktop.org/conference/2023/

1. WebCodecs in WebKit
WebCodecs in WebKit
With GStreamer!
With GStreamer!
Philippe Normand
Philippe Normand
GStreamer conference 2023
GStreamer conference 2023
1
1 /
/ 8
8

2. WebCodecs, what is this?
WebCodecs, what is this?
“This specification defines interfaces to codecs for encoding and decoding of
“This specification defines interfaces to codecs for encoding and decoding of
audio, video, and images.”
audio, video, and images.”
Gives JS direct access to encoders and decoders provided by the WebEngine
Gives JS direct access to encoders and decoders provided by the WebEngine
Ability to modify
Ability to modify VideoFrame
VideoFrame s and
s and AudioData
AudioData before encoding or after
before encoding or after
decoding
decoding
No need for 3rd-party WASM-based framework
No need for 3rd-party WASM-based framework
Muxing and demuxing handled at JS layer (MP4Box.js, etc)
Muxing and demuxing handled at JS layer (MP4Box.js, etc)
https://www.w3.org/TR/webcodecs/
https://www.w3.org/TR/webcodecs/
2
2 /
/ 8
8

3. WebCodecs, use-cases
WebCodecs, use-cases
Video editors
Video editors
WebRTC “Funny hats”
WebRTC “Funny hats”
Integration with WebAudio, WebGL, WebTransport
Integration with WebAudio, WebGL, WebTransport
{en,de}coding can be moved to another thread with WebWorkers
{en,de}coding can be moved to another thread with WebWorkers
3
3 /
/ 8
8

4. WebCodecs, in WebKit!
WebCodecs, in WebKit!
First approach with
First approach with appsrc
appsrc !
! {en,de}codebin
{en,de}codebin !
! appsink
appsink worked, but was not
worked, but was not
spec compliant
spec compliant
We need “direct” access to the codec element, no additional thread (queue)
We need “direct” access to the codec element, no additional thread (queue)
Push data to sink pad, pull from src pad
Push data to sink pad, pull from src pad
Something a bit similar to GstHarness
Something a bit similar to GstHarness
4
4 /
/ 8
8

5. WebKit’s GStreamerElementHarness
WebKit’s GStreamerElementHarness
A C++ class, no pipeline per-se
A C++ class, no pipeline per-se
Simple API,
Simple API, pushBuffer()
pushBuffer() ,
, pushEvent()
pushEvent() ,
, flush()
flush() ,…
,…
Internal state: output buffers and events stored in Double Ended Queues (deque)
Internal state: output buffers and events stored in Double Ended Queues (deque)
Simple case: one input pad, one output pad:
Simple case: one input pad, one output pad:
decoder or encoder
decoder or encoder
sink
sink src
src
src
src sink
sink
Push data on src pad
Push data on src pad
Processing ends in … sink pad
Processing ends in … sink pad
C++ closure-based processing triggered with
C++ closure-based processing triggered with ::processOutputBuffers()
::processOutputBuffers()
5
5 /
/ 8
8

6. Chaining
Chaining
one to many pads
one to many pads
callback-based “plugging”
callback-based “plugging”
“sometimes” pad handling, with possibility of sending buffers to an
“sometimes” pad handling, with possibility of sending buffers to an identity
identity -
-
based harness
based harness
decoder
decoder
parsebin
parsebin
sink
sink src
src
sink
sink src
src sink
sink src
src
src
src sink
sink
6
6 /
/ 8
8

7. Implementation status in WebKitGTK &
Implementation status in WebKitGTK &
WPEWebKit
WPEWebKit
(H.264, H.265, AV1, VP8, VP9) video encoding/decoding: shipping in 2.42
(H.264, H.265, AV1, VP8, VP9) video encoding/decoding: shipping in 2.42
(AAC, mp3, opus, vorbis, fLaC, alaw, ulaw, LPCM) audio encoding/decoding:
(AAC, mp3, opus, vorbis, fLaC, alaw, ulaw, LPCM) audio encoding/decoding:
landed in main, will ship in 2.44
landed in main, will ship in 2.44
WebPlatform tests coverage is OK
WebPlatform tests coverage is OK
Room for improvement regarding interaction with WebAudio and WebGL
Room for improvement regarding interaction with WebAudio and WebGL
7
7 /
/ 8
8

8. Demo
Demo
All usable from the GNOME Web Canary flatpak
All usable from the GNOME Web Canary flatpak
Ask me!
Ask me!
8
8 /
/ 8
8