The document discusses the integration of WebKit and GStreamer to enhance multimedia capabilities in web browsers, highlighting architecture, features, and upcoming developments. It covers WebKit's background, multimedia features like WebAudio, and the use of GStreamer for media playback and video compositing. The document also outlines future improvements, including on-disk buffering and adaptive streaming support.

2. WebKit and GStreamer
Bringing lolcatz videos to your desktop
Xabier Rodríguez Calvar
Igalia

3. Who is this dude?
· Born in the Basque Country
· Grown and living in Galicia
- One of the seven Celtic Nations, like Scotland
· Studied Computer Science at Universidade da Coruña
· Partner and developer at Igalia
- Working in Multimedia and WebKit

4. From html5demos.com

5. Index
· Quick review of WebKit
· WebKit Multimedia
· Power features
- WebAudio
- Hardware acceleration
- Accelerated video compositing
5/36

6. Quick Review

7. WebKit
· Web content engine
· Processes and renders web content
· Started as a fork of KTHML and KJS
· Open source since 2005
· Goals: open source, compatibility, compliance, stability,
performance, security, portability, usability, hackability
· Non-goals: being a full web browser, being a science project,
having reusable components, unlimited scope
· Forked by Google as Blink
· Split into ports: GTK+, Qt, EFL, Mac, Chromium, Windows
7/36

8. Architecture
8/36

9. WebKitGTK+
· Each WebKit port is composed of
- Platform interfaces
- API layer
· WebKitGTK+ platform layer:
- libsoup for networking
- cairo for rasterization
- OpenGL for making the scene graph and WebGL
- GStreamer for media
- Various GTK+ APIs for talking with the system
· API layer is a GtkWidget and a set of GObject APIs
· WebKitGTK+ is used by Epiphany, Midori, yelp, devhelp
9/36

10. WebKit2
· Give the web rendering parts of WebKit their own process
· Page crashes don't crash the browser
· Can put vulnerable data into a separate address space
· Sandbox web rendering
· Prevent pages from accessing the disk and operating system
interface
10/36

11. WebKit2 Architecture
11/36

12. The Near Future
12/36

13. WebKit Multimedia

14. WebKit Multimedia
· Mac port: QuickTime, QtKit, AVFoundations
· Apple Win port: QuickTime
· GTK+ and EFL ports: GStreamer 1.0 backend
· Blackberry port: Bridges over to its own Media playback framework
· Chromium: bridges over to WebKit/Chromium backend which
relays to FFMPeg implementations
· Qt port: GStreamer on Linux, QtKit on Mac, QuickTime on
Windows
14/36

15. 15/36
remaertSG
mroftalP
- - - - - - - - - - - reyalPaideM
eroCbeW
tiKbeW
WebKit Multimedia Architecture

16. GStreamer MediaPlayer - Overview
· Mainly developed and tested on Linux platform
· Successfully ported to MacOS/Win platfoms (not fully upstream
yet though)
· Playbin-based: the best GStreamer element for playback usecases
- Playbin2 with GStreamer 0.10
· MediaPlayer registers and uses two WebKit-specific GStreamer
elements: webkitwebsrc and a video-sink
16/36

17. GStreamer MediaPlayer - Main Features
· On-disk-buffering
· Frame-accurate seeking
· Basic metrics reporting
· Trick-modes
· WebAudio
· Hardware decoding support with VA-API (gst 1.2.x)
· Video accelerated compositing
· Codec installer support
17/36

18. WebKit and GStreamer Architecture
MediaPlayerClient
-eiPaeCin
mdalyrlet
+lyakhne(
pabcCagd)
+ouehne(
vlmCagd)
HTMLMediaElement
-eiPae
mdalyr
+ly)
pa(
+lyakhne(
pabcCagd)
+eVlm(nvlm:la)
stouei ouefot
+ouehne(
vlmCagd)
-_rvt
mpiae
MediaPlayer
+ly)
pa(
+eVlm(nvlm:la)
stouei ouefot
MediaPlayerPrivateInterface
+ly)
pa(
+eVlm(nvlm:la)
stouei ouefot
MediaPlayerPrivateGStreamerBase
+eVlm(nvlm:la)
stouei ouefot
HTMLVideoElement
HTMLAudioElement
GstPlaybin
-lyi
pabn
MediaPlayerPrivateGStreamer
+ly)
pa(
18/36

19. GStreamer MediaPlayer - HTTP source element
· a GStreamer bin implementing the URI handler interface for
http(s):// and blob:// uris.
· A single source pad provides raw data access to downstream
elements.
· appsrc is used internally to interface with the WebCore
SubResourceLoader object
· Element is registered with high rank in the GStreamer registry so
playbin is more prone to pick it up to read http:// URIs.
19/36

20. GStreamer MediaPlayer - Video rendering
· Element inheriting from GstVideoSink
- In practical sense, it is a GstAppSink with video capabilities
negotiation
· Takes video GstBuffers as input
· Pass those to the MediaPlayerPrivateGStreamer via a signal
20/36

21. GStreamer MediaPlayer - Audio rendering
· Use autoaudiosink
· Check for pulseaudio for playback request messages
· Pitch preservation
· Preserve audio passthrough when possible
· Volume managed by the sink
21/36

22. GStreamer MediaPlayer - Next steps
· Improvements and bug fixes in the player:
- Store on-disk buffered media in WebCore's cache directory
- Make the webkitwebsrc element private to WebKit
· MediaStream (getUserMedia) and WebRTC spec implementation
· MediaSource adaptive streaming API spec implementation
· Integration of WebAudio with media elements and mediastream
stack
· Finish porting to 1.2
· Remove GStreamer 0.10 code path
22/36

23. WebKit Multimedia Power Features

24. WebAudio
· A JavaScript API allowing to create a new generation of Web
games and applications
· Effects, convolution engine for room simulation, etc
· Realtime analysis, playback and visualization
· W3C spec driven and implemented mainly by Chromium and
Firefox
· In WebKit, currently implemented for:
- Mac
- Chromium
- GStreamer based Ports: GTK+, EFL, Qt
24/36

25. WebAudio GStreamer - Reading Audio
· Spec allows two ways to read audio data:
- from memory: we can use giostreamsrc GStreamer element
- from a file path: we can use the filesrc GStremer element
· Pipeline: src ! decodebin2 ! audioconvert ! audioresample !
capsfilter ! deinterleave ! queue ! appsink
· deinterleave separates multi-channel audio to n audio mono
channels (Pipeline above can contain multiple queue ! appsink
chains)
· appsink used to collect mono audio buffers, store them in a list and
convert data to floating point values
· Data is stored in a WebCore::AudioBus, containing one or multiple
channel Float arrays
25/36

26. WebAudio GStreamer - Playing Audio
· The WebAudio stack processed or generated some audio data to be
sent to the sound card
· Similarly to the "Reading Data" step, a WebCore::AudioBus is used
to store the data
· Custom GStreamer element to convert an AudioBus to an
interleaved audio data flow: webkitwebaudiosrc
· webkitwebaudiosrc: a bin with n internal pads pushing to an
interleave element. Source pad outputs a single, interleave, wavencoded data flow
· Pipeline: webkitwebaudiosrc ! wavparse ! audioconvert !
autoaudiosink
26/36

27. WebAudio GStreamer - Work in Progress
· Some layout tests don't pass yet
· Not yet enabled by default in release builds
· Support for stereo+ streams in the reader side. Playback side
already handles those.
· Integration with <video> and <audio> elements: WebAudio
MediaElementSourceNode
27/36

28. Hardware Accelerated Video
· We use GStreamer VA-API. (1.2.x)
· That's it! Hardcore comes when compositing that.
28/36

29. Accelerated Video Compositing - Acceleration in
WebKit
From Joone Hur
29/36

30. Accelerated Video Compositing
· Element inheriting from GstVideoSink
- In practical sense, it is a GstAppSink with video capabilities
negotiation
· Takes video GstBuffers as input.
· Video frame is transferred to the MediaPlayerPrivateGStreamer via
a GLib signal.
· MediaPlayerPrivateGStreamer uploads it to a GL texture and
renders it asynchronously in the WebKit context.
30/36

31. Accelerated Video Compositing - Architecture
TextureMapperPlatformLayer
MediaPlayerPrivateInterface
+an(
pit)
MediaPlayerPrivateGStreamerBase
+an(
pit)
GstPlaybin
-lyi
pabn
MediaPlayerPrivateGStreamer
*
GstElement
...
WebKitVideoSink
<cets>
<rae>
31/36

32. Accelerated Video Compositing - Rendering paths
· Software rendering path
- It uses Cairo
· Accelerated compositing paths
- TextureMapperLayer:
- Light-weight implementation of a scene-graph
- The TextureMapperLayer tree is equivalent to the
GraphicsLayer tree.
- TextureMapperGL
- Provides a GPU accelerated implementation of the
drawing primitives, using shaders compatible with GL/ES
2.0
32/36

33. Accelerated Video Compositing - Next steps
· More video decoders with GstVideoGLTextureUploadMeta support
· Support EGLImage in WebKit as video frames handlers
- And video decoders that deliver this kind of capabilities.
· Handle the vertical blank interval in WebKit
- GTK+ 3.8 seems to fix this issue.
33/36

34. From html5demos.com

35. From html5demos.com
35/36

36. Thank You!
(q&a)
twitter @calvaris
www blogs.igalia.com/xrcalvar