Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)


Published on

By Philippe Normand and Miguel Gómez.

Since the Raspberry Pi platform was introduced it dramatically changed the picture of the embedded (affordable) micro-computer market.

WebKit is a popular Web rendering engine developed by a wide community of Open-Source developers including major actors such as Apple, Samsung, Intel and Adobe.

The purpose of this talk is to explain how we managed to leverage the hardware components of the Raspberry Pi to deliver acceptable video rendering performance of a QtWebKit(2) browser using the GStreamer 1.2.x APIs. We will explain how we integrated zero-copy rendering in the multi-process WebKit2 architecture. Other advanced use-cases such as video/canvas, video/webgl and getUserMedia will also be presented.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)

  1. 1. Efficient multimedia support in QtWebKit on Raspberry Pi
  2. 2. Talk outline • Raspberry Pi • WebKit architecture(s) with emphasis on Media playback • Platform-level improvements for QtWebKit
  3. 3. Raspberry Pi - Model B • 700MHz single-core ARMv6 CPU • Broadcom BCM2835 VideoCore IV • 512 MB RAM (partly allocated for GPU)
  4. 4. GStreamer OpenMax (gst-omx) • Hardware decoders for H.264, VP8, Theora • meta:EGLImage • Custom audio sinks: HDMI and analog • Very good integration within playbin
  5. 5. Premises of the project • Fullscreen browser, no tabs, no WM • Focus on media consumption web-apps and Youtube TV • OpenGL for rendering • Qt5 platform, EGLFS QPA
  6. 6. WebKit rendering - from HTML to the screen
  7. 7. WebKit rendering - the forest
  8. 8. WebKit1 rendering (with OpenGL) • GraphicsLayers are backed by GL textures • The video player has its own GraphicsLayer • RenderLayers are traversed and painted recursively (to their GraphicsLayers): children below -> self -> children above • Composition: GraphicsLayers are stacked to produce the final result
  9. 9. WebKit2 rendering (OpenGL) - processes • Rendering split into 2 processes: UI and Web • Web process renders the layers (similar to WebKit1 rendering) • UI process composites and shows them • GraphicsLayers shared using GraphicsSurfaces • A GraphicsSurface encapsulates a method to share gl textures among processes (EGLImages)
  10. 10. WebKit2 rendering (OpenGL) - Coordinated graphics
  11. 11. The MediaPlayer in WebCore
  12. 12. MediaPlayer - platform level
  13. 13. Video rendering - the inefficient way • Sink negotiates xRGB or ARGB video caps • Optional buffer copy in ::render, conversion to Cairo's ARGB pre-multiplied alpha • Buffer data copy in MediaPlayerPrivate::paint to a WebCore::BitmapImage • BitmapImage rendered in a QPixmap • Another approach: GLTextureUpload meta (CPU -> GPU buffer copy)
  14. 14. WebKit2 video rendering (inefficient)
  15. 15. Video sink - improved integration within the platform • Sink negotiates meta:EGLImage with upstream video decoder • Buffer pool and EGLImage allocator (like in glimagesink) • EGLImage encapsulated in a GstBuffer passed to MediaPlayerPrivate • Efficient rendering in the player, different approaches for WebKit1 and WebKit2
  16. 16. Improvements in WebKit2 rendering
  17. 17. Improving media resources loading
  18. 18. Using libsoup within QtWebKit • The Qt-based resource loader doesn't cope well with large media resources • The libsoup-based resource loader is more memory efficient and better integrated in WebCore • Pre-allocation of GstBuffers within our HTTP source element
  19. 19. WebRTC / getUserMedia
  20. 20. Local devices listing/playback • In JS: webkitGetUserMedia({audio: true}, successCallback, failureCallback); • function successCallback(stream) { video.src = URL.createObjectURL(stream);; } • Pipeline should also allow encoding and RTP payloading if required by webapp (WebRTC PeerConnection)
  21. 21. WebRTC - RPiCam integration • rpicamsrc: emits H.264 byte stream • Leverage H.264 hardware rendering • rpicamsrc ! h264parse ! omxh264dec ! sink • with a tee downstream rpicamsrc to allow encoding/streaming later on
  22. 22. WebAudio
  23. 23. WebAudio - basics • JS API for processing and synthesizing audio in web applications • Pipeline based. Examples of nodes: BufferSource, GainNode, BiQuadFilter, Destination... • Head-related transfer function: tricking the human ear to spatial sound
  24. 24. WebAudio - decoding incoming files • First scenario: local file. Leverage decodebin and appsink to create an in-memory representation of the audio file PCM data (FloatArray). • Second scenario: memory buffer. Decodebin again! With giostreamsrc • No RPi-specific optimisations
  25. 25. WebAudio - playback • Playback of WebAudio FloatArrays PCM samples to soundcard • Custom GStreamer source element encoding internal WebCore audio buffers to WAV • Again, no RPi-specific optimisations
  26. 26. So which part of the backend needed optimisations?
  27. 27. WebAudio - HRTF Database • Loaded at AudioContext creation, from a separate thread • 240 very small (256 frames) WAV files => lots of context switches • Improvement: concatenate all samples and internally split when building the database • => only one file to read, less context switches!
  28. 28. WebAudio - FFT experiments • default WebAudio FFTFrame backend using GstFFT (itself based on KissFFT) • math functions hot on perf profiles • GPUFFT library (from rpi-userland): leveraging VideoCore for FFT calculations • Major drawback: requires allocated RAM, so less RAM for the other libraries
  29. 29. Next tasks • video/canvas and video/webgl integration • WebRTC • ORC ARMv6 backend?
  30. 30. Demo rendering from youtube at 720p
  31. 31. Demo rendering from youtube at 1080p
  32. 32. Flickr pictures • • • This is the last slide, let's have lunch now!