• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
WebKit and GStreamer
 

WebKit and GStreamer

on

  • 2,967 views

These are the slides of my talk WebKit and GStreamer of the GStreamer Conference on 2013, cohosted with LinuxCon. ...

These are the slides of my talk WebKit and GStreamer of the GStreamer Conference on 2013, cohosted with LinuxCon.

The HTML5 version with its effects can be found at http://people.igalia.com/xrcalvar/talks/20131022-GstConf-WebKit

Statistics

Views

Total Views
2,967
Views on SlideShare
2,965
Embed Views
2

Actions

Likes
1
Downloads
39
Comments
0

1 Embed 2

https://twitter.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    WebKit and GStreamer WebKit and GStreamer Presentation Transcript

    • WebKit and GStreamer Bringing lolcatz videos to your desktop Xabier Rodríguez Calvar Igalia
    • Who is this dude? · Born in the Basque Country · Grown and living in Galicia - One of the seven Celtic Nations, like Scotland · Studied Computer Science at Universidade da Coruña · Partner and developer at Igalia - Working in Multimedia and WebKit
    • From html5demos.com
    • Index · Quick review of WebKit · WebKit Multimedia · Power features - WebAudio - Hardware acceleration - Accelerated video compositing 5/36
    • Quick Review
    • WebKit · Web content engine · Processes and renders web content · Started as a fork of KTHML and KJS · Open source since 2005 · Goals: open source, compatibility, compliance, stability, performance, security, portability, usability, hackability · Non-goals: being a full web browser, being a science project, having reusable components, unlimited scope · Forked by Google as Blink · Split into ports: GTK+, Qt, EFL, Mac, Chromium, Windows 7/36
    • Architecture 8/36
    • WebKitGTK+ · Each WebKit port is composed of - Platform interfaces - API layer · WebKitGTK+ platform layer: - libsoup for networking - cairo for rasterization - OpenGL for making the scene graph and WebGL - GStreamer for media - Various GTK+ APIs for talking with the system · API layer is a GtkWidget and a set of GObject APIs · WebKitGTK+ is used by Epiphany, Midori, yelp, devhelp 9/36
    • WebKit2 · Give the web rendering parts of WebKit their own process · Page crashes don't crash the browser · Can put vulnerable data into a separate address space · Sandbox web rendering · Prevent pages from accessing the disk and operating system interface 10/36
    • WebKit2 Architecture 11/36
    • The Near Future 12/36
    • WebKit Multimedia
    • WebKit Multimedia · Mac port: QuickTime, QtKit, AVFoundations · Apple Win port: QuickTime · GTK+ and EFL ports: GStreamer 1.0 backend · Blackberry port: Bridges over to its own Media playback framework · Chromium: bridges over to WebKit/Chromium backend which relays to FFMPeg implementations · Qt port: GStreamer on Linux, QtKit on Mac, QuickTime on Windows 14/36
    • 15/36 remaertSG mroftalP - - - - - - - - - - - reyalPaideM eroCbeW tiKbeW WebKit Multimedia Architecture
    • GStreamer MediaPlayer - Overview · Mainly developed and tested on Linux platform · Successfully ported to MacOS/Win platfoms (not fully upstream yet though) · Playbin-based: the best GStreamer element for playback usecases - Playbin2 with GStreamer 0.10 · MediaPlayer registers and uses two WebKit-specific GStreamer elements: webkitwebsrc and a video-sink 16/36
    • GStreamer MediaPlayer - Main Features · On-disk-buffering · Frame-accurate seeking · Basic metrics reporting · Trick-modes · WebAudio · Hardware decoding support with VA-API (gst 1.2.x) · Video accelerated compositing · Codec installer support 17/36
    • WebKit and GStreamer Architecture MediaPlayerClient -eiPaeCin mdalyrlet +lyakhne( pabcCagd) +ouehne( vlmCagd) HTMLMediaElement -eiPae mdalyr +ly) pa( +lyakhne( pabcCagd) +eVlm(nvlm:la) stouei ouefot +ouehne( vlmCagd) -_rvt mpiae MediaPlayer +ly) pa( +eVlm(nvlm:la) stouei ouefot MediaPlayerPrivateInterface +ly) pa( +eVlm(nvlm:la) stouei ouefot MediaPlayerPrivateGStreamerBase +eVlm(nvlm:la) stouei ouefot HTMLVideoElement HTMLAudioElement GstPlaybin -lyi pabn MediaPlayerPrivateGStreamer +ly) pa( 18/36
    • GStreamer MediaPlayer - HTTP source element · a GStreamer bin implementing the URI handler interface for http(s):// and blob:// uris. · A single source pad provides raw data access to downstream elements. · appsrc is used internally to interface with the WebCore SubResourceLoader object · Element is registered with high rank in the GStreamer registry so playbin is more prone to pick it up to read http:// URIs. 19/36
    • GStreamer MediaPlayer - Video rendering · Element inheriting from GstVideoSink - In practical sense, it is a GstAppSink with video capabilities negotiation · Takes video GstBuffers as input · Pass those to the MediaPlayerPrivateGStreamer via a signal 20/36
    • GStreamer MediaPlayer - Audio rendering · Use autoaudiosink · Check for pulseaudio for playback request messages · Pitch preservation · Preserve audio passthrough when possible · Volume managed by the sink 21/36
    • GStreamer MediaPlayer - Next steps · Improvements and bug fixes in the player: - Store on-disk buffered media in WebCore's cache directory - Make the webkitwebsrc element private to WebKit · MediaStream (getUserMedia) and WebRTC spec implementation · MediaSource adaptive streaming API spec implementation · Integration of WebAudio with media elements and mediastream stack · Finish porting to 1.2 · Remove GStreamer 0.10 code path 22/36
    • WebKit Multimedia Power Features
    • WebAudio · A JavaScript API allowing to create a new generation of Web games and applications · Effects, convolution engine for room simulation, etc · Realtime analysis, playback and visualization · W3C spec driven and implemented mainly by Chromium and Firefox · In WebKit, currently implemented for: - Mac - Chromium - GStreamer based Ports: GTK+, EFL, Qt 24/36
    • WebAudio GStreamer - Reading Audio · Spec allows two ways to read audio data: - from memory: we can use giostreamsrc GStreamer element - from a file path: we can use the filesrc GStremer element · Pipeline: src ! decodebin2 ! audioconvert ! audioresample ! capsfilter ! deinterleave ! queue ! appsink · deinterleave separates multi-channel audio to n audio mono channels (Pipeline above can contain multiple queue ! appsink chains) · appsink used to collect mono audio buffers, store them in a list and convert data to floating point values · Data is stored in a WebCore::AudioBus, containing one or multiple channel Float arrays 25/36
    • WebAudio GStreamer - Playing Audio · The WebAudio stack processed or generated some audio data to be sent to the sound card · Similarly to the "Reading Data" step, a WebCore::AudioBus is used to store the data · Custom GStreamer element to convert an AudioBus to an interleaved audio data flow: webkitwebaudiosrc · webkitwebaudiosrc: a bin with n internal pads pushing to an interleave element. Source pad outputs a single, interleave, wavencoded data flow · Pipeline: webkitwebaudiosrc ! wavparse ! audioconvert ! autoaudiosink 26/36
    • WebAudio GStreamer - Work in Progress · Some layout tests don't pass yet · Not yet enabled by default in release builds · Support for stereo+ streams in the reader side. Playback side already handles those. · Integration with <video> and <audio> elements: WebAudio MediaElementSourceNode 27/36
    • Hardware Accelerated Video · We use GStreamer VA-API. (1.2.x) · That's it! Hardcore comes when compositing that. 28/36
    • Accelerated Video Compositing - Acceleration in WebKit From Joone Hur 29/36
    • Accelerated Video Compositing · Element inheriting from GstVideoSink - In practical sense, it is a GstAppSink with video capabilities negotiation · Takes video GstBuffers as input. · Video frame is transferred to the MediaPlayerPrivateGStreamer via a GLib signal. · MediaPlayerPrivateGStreamer uploads it to a GL texture and renders it asynchronously in the WebKit context. 30/36
    • Accelerated Video Compositing - Architecture TextureMapperPlatformLayer MediaPlayerPrivateInterface +an( pit) MediaPlayerPrivateGStreamerBase +an( pit) GstPlaybin -lyi pabn MediaPlayerPrivateGStreamer * GstElement ... WebKitVideoSink <cets> <rae> 31/36
    • Accelerated Video Compositing - Rendering paths · Software rendering path - It uses Cairo · Accelerated compositing paths - TextureMapperLayer: - Light-weight implementation of a scene-graph - The TextureMapperLayer tree is equivalent to the GraphicsLayer tree. - TextureMapperGL - Provides a GPU accelerated implementation of the drawing primitives, using shaders compatible with GL/ES 2.0 32/36
    • Accelerated Video Compositing - Next steps · More video decoders with GstVideoGLTextureUploadMeta support · Support EGLImage in WebKit as video frames handlers - And video decoders that deliver this kind of capabilities. · Handle the vertical blank interval in WebKit - GTK+ 3.8 seems to fix this issue. 33/36
    • From html5demos.com
    • From html5demos.com 35/36
    • Thank You! (q&a) twitter @calvaris www blogs.igalia.com/xrcalvar