WebKit and GStreamer (GStreamer Conference 2013)


Published on

By Xabier Rodríguez Calvar.

WebKit is a well known open source browser engine used by Apple, GNOME, EFL, Blackberry and others and shares quite a big codebase with Blink. It was the result of a fork from KHTML and KJS.

As a simplification we can say that it has three main layers, the API, the core and the backend. The different ports implement their API and their backends and the core is shared. WebKit 2 is robust and transparent multiprocess architecture allowing safer interactions with webpages.

GStreamer is the multimedia backend of some WebKit ports and in this talk we will see its architecture, status, challenges and future.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

WebKit and GStreamer (GStreamer Conference 2013)

  1. 1. WebKit and GStreamer Bringing lolcatz videos to your desktop Xabier Rodríguez Calvar Igalia
  2. 2. Who is this dude? · Born in the Basque Country · Grown and living in Galicia - One of the seven Celtic Nations, like Scotland · Studied Computer Science at Universidade da Coruña · Partner and developer at Igalia - Working in Multimedia and WebKit
  3. 3. From html5demos.com
  4. 4. Index · Quick review of WebKit · WebKit Multimedia · Power features - WebAudio - Hardware acceleration - Accelerated video compositing 5/36
  5. 5. Quick Review
  6. 6. WebKit · Web content engine · Processes and renders web content · Started as a fork of KTHML and KJS · Open source since 2005 · Goals: open source, compatibility, compliance, stability, performance, security, portability, usability, hackability · Non-goals: being a full web browser, being a science project, having reusable components, unlimited scope · Forked by Google as Blink · Split into ports: GTK+, Qt, EFL, Mac, Chromium, Windows 7/36
  7. 7. Architecture 8/36
  8. 8. WebKitGTK+ · Each WebKit port is composed of - Platform interfaces - API layer · WebKitGTK+ platform layer: - libsoup for networking - cairo for rasterization - OpenGL for making the scene graph and WebGL - GStreamer for media - Various GTK+ APIs for talking with the system · API layer is a GtkWidget and a set of GObject APIs · WebKitGTK+ is used by Epiphany, Midori, yelp, devhelp 9/36
  9. 9. WebKit2 · Give the web rendering parts of WebKit their own process · Page crashes don't crash the browser · Can put vulnerable data into a separate address space · Sandbox web rendering · Prevent pages from accessing the disk and operating system interface 10/36
  10. 10. WebKit2 Architecture 11/36
  11. 11. The Near Future 12/36
  12. 12. WebKit Multimedia
  13. 13. WebKit Multimedia · Mac port: QuickTime, QtKit, AVFoundations · Apple Win port: QuickTime · GTK+ and EFL ports: GStreamer 1.0 backend · Blackberry port: Bridges over to its own Media playback framework · Chromium: bridges over to WebKit/Chromium backend which relays to FFMPeg implementations · Qt port: GStreamer on Linux, QtKit on Mac, QuickTime on Windows 14/36
  14. 14. 15/36 remaertSG mroftalP - - - - - - - - - - - reyalPaideM eroCbeW tiKbeW WebKit Multimedia Architecture
  15. 15. GStreamer MediaPlayer - Overview · Mainly developed and tested on Linux platform · Successfully ported to MacOS/Win platfoms (not fully upstream yet though) · Playbin-based: the best GStreamer element for playback usecases - Playbin2 with GStreamer 0.10 · MediaPlayer registers and uses two WebKit-specific GStreamer elements: webkitwebsrc and a video-sink 16/36
  16. 16. GStreamer MediaPlayer - Main Features · On-disk-buffering · Frame-accurate seeking · Basic metrics reporting · Trick-modes · WebAudio · Hardware decoding support with VA-API (gst 1.2.x) · Video accelerated compositing · Codec installer support 17/36
  17. 17. WebKit and GStreamer Architecture MediaPlayerClient -eiPaeCin mdalyrlet +lyakhne( pabcCagd) +ouehne( vlmCagd) HTMLMediaElement -eiPae mdalyr +ly) pa( +lyakhne( pabcCagd) +eVlm(nvlm:la) stouei ouefot +ouehne( vlmCagd) -_rvt mpiae MediaPlayer +ly) pa( +eVlm(nvlm:la) stouei ouefot MediaPlayerPrivateInterface +ly) pa( +eVlm(nvlm:la) stouei ouefot MediaPlayerPrivateGStreamerBase +eVlm(nvlm:la) stouei ouefot HTMLVideoElement HTMLAudioElement GstPlaybin -lyi pabn MediaPlayerPrivateGStreamer +ly) pa( 18/36
  18. 18. GStreamer MediaPlayer - HTTP source element · a GStreamer bin implementing the URI handler interface for http(s):// and blob:// uris. · A single source pad provides raw data access to downstream elements. · appsrc is used internally to interface with the WebCore SubResourceLoader object · Element is registered with high rank in the GStreamer registry so playbin is more prone to pick it up to read http:// URIs. 19/36
  19. 19. GStreamer MediaPlayer - Video rendering · Element inheriting from GstVideoSink - In practical sense, it is a GstAppSink with video capabilities negotiation · Takes video GstBuffers as input · Pass those to the MediaPlayerPrivateGStreamer via a signal 20/36
  20. 20. GStreamer MediaPlayer - Audio rendering · Use autoaudiosink · Check for pulseaudio for playback request messages · Pitch preservation · Preserve audio passthrough when possible · Volume managed by the sink 21/36
  21. 21. GStreamer MediaPlayer - Next steps · Improvements and bug fixes in the player: - Store on-disk buffered media in WebCore's cache directory - Make the webkitwebsrc element private to WebKit · MediaStream (getUserMedia) and WebRTC spec implementation · MediaSource adaptive streaming API spec implementation · Integration of WebAudio with media elements and mediastream stack · Finish porting to 1.2 · Remove GStreamer 0.10 code path 22/36
  22. 22. WebKit Multimedia Power Features
  23. 23. WebAudio · A JavaScript API allowing to create a new generation of Web games and applications · Effects, convolution engine for room simulation, etc · Realtime analysis, playback and visualization · W3C spec driven and implemented mainly by Chromium and Firefox · In WebKit, currently implemented for: - Mac - Chromium - GStreamer based Ports: GTK+, EFL, Qt 24/36
  24. 24. WebAudio GStreamer - Reading Audio · Spec allows two ways to read audio data: - from memory: we can use giostreamsrc GStreamer element - from a file path: we can use the filesrc GStremer element · Pipeline: src ! decodebin2 ! audioconvert ! audioresample ! capsfilter ! deinterleave ! queue ! appsink · deinterleave separates multi-channel audio to n audio mono channels (Pipeline above can contain multiple queue ! appsink chains) · appsink used to collect mono audio buffers, store them in a list and convert data to floating point values · Data is stored in a WebCore::AudioBus, containing one or multiple channel Float arrays 25/36
  25. 25. WebAudio GStreamer - Playing Audio · The WebAudio stack processed or generated some audio data to be sent to the sound card · Similarly to the "Reading Data" step, a WebCore::AudioBus is used to store the data · Custom GStreamer element to convert an AudioBus to an interleaved audio data flow: webkitwebaudiosrc · webkitwebaudiosrc: a bin with n internal pads pushing to an interleave element. Source pad outputs a single, interleave, wavencoded data flow · Pipeline: webkitwebaudiosrc ! wavparse ! audioconvert ! autoaudiosink 26/36
  26. 26. WebAudio GStreamer - Work in Progress · Some layout tests don't pass yet · Not yet enabled by default in release builds · Support for stereo+ streams in the reader side. Playback side already handles those. · Integration with <video> and <audio> elements: WebAudio MediaElementSourceNode 27/36
  27. 27. Hardware Accelerated Video · We use GStreamer VA-API. (1.2.x) · That's it! Hardcore comes when compositing that. 28/36
  28. 28. Accelerated Video Compositing - Acceleration in WebKit From Joone Hur 29/36
  29. 29. Accelerated Video Compositing · Element inheriting from GstVideoSink - In practical sense, it is a GstAppSink with video capabilities negotiation · Takes video GstBuffers as input. · Video frame is transferred to the MediaPlayerPrivateGStreamer via a GLib signal. · MediaPlayerPrivateGStreamer uploads it to a GL texture and renders it asynchronously in the WebKit context. 30/36
  30. 30. Accelerated Video Compositing - Architecture TextureMapperPlatformLayer MediaPlayerPrivateInterface +an( pit) MediaPlayerPrivateGStreamerBase +an( pit) GstPlaybin -lyi pabn MediaPlayerPrivateGStreamer * GstElement ... WebKitVideoSink <cets> <rae> 31/36
  31. 31. Accelerated Video Compositing - Rendering paths · Software rendering path - It uses Cairo · Accelerated compositing paths - TextureMapperLayer: - Light-weight implementation of a scene-graph - The TextureMapperLayer tree is equivalent to the GraphicsLayer tree. - TextureMapperGL - Provides a GPU accelerated implementation of the drawing primitives, using shaders compatible with GL/ES 2.0 32/36
  32. 32. Accelerated Video Compositing - Next steps · More video decoders with GstVideoGLTextureUploadMeta support · Support EGLImage in WebKit as video frames handlers - And video decoders that deliver this kind of capabilities. · Handle the vertical blank interval in WebKit - GTK+ 3.8 seems to fix this issue. 33/36
  33. 33. From html5demos.com
  34. 34. From html5demos.com 35/36
  35. 35. Thank You! (q&a) twitter @calvaris www blogs.igalia.com/xrcalvar