gtkgst: Video in Your Widgets!
https://github.com/ystreet/gst-plugins-bad/tree/gtk
Matthew Waters (ystreet00)
GNOME Asia 2015
9 May 2015

Who Am I
● Australian
● GStreamer
● OpenGL

What is GStreamer?
● Pipeline-based media framework
● Basic building blocks: elements connected via pads
● Set of libraries with abstract API
● Plugins for specific features
– Often wrapping other libraries, e.g. libav/ffmpeg, OpenCV,
various codec libraries

What is GStreamer?
● Should have gone to Nirbheek's talk yesterday!

What is Gtk+?
● Cross-platform GUI widget toolkit
– OS X, Windows, Linux (X11 + Wayland)
– Input (Keyboard, Touch, Mouse)
– Standard widgets (Buttons, Text, Containers)
– Custom widgets

Computer Video
● Colour spaces/gamut
● Chroma sampling
● Scaling
● Splitting/Cutting/Editing
● Effects
● Generation

Video in GUI Toolkits
● Embed a 'window' into another window
● Watch out for threads!
● App has to know about the window system in use

GstVideoOverlay/GstNavigation
● Provide a window handle to render video into
● Notify the video sink of size changes
● Push key and mouse events into GStreamer

GstVideoOverlay
GstElement *playbin = gst_element_factory_make (“playbin”, NULL);
g_object_set (playbin, “uri”, “file:///path/to/file.mp4”, NULL);
GtkWidget *window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
GtkWidget *video_window = gtk_drawing_area_new ();
gtk_container_add (GTK_CONTAINER (window), video_window);
gtk_widget_show_all (window);
GdkWindow *video_window_xwindow = gtk_widget_get_window
(video_window);
gulong embed_xid = gtk_window_get_xid (video_window_xwindow);
gst_video_overlay_set_window_handle (GST_VIDEO_OVERLAY (playbin),
embed_xid);
gst_element_set_state (playbin, GST_STATE_PLAYING);

GstVideoOverlay - demo

Alternative - gtkgst
● Renders your video into a GtkWidget provided by
the GStreamer sink
● Software implementation uses cairo into a
GtkDrawingArea
● Place the provided GtkWidget wherever in the
widget hierarchy

gtkgst - GL
● Gtk 3.16 has new GL support
● GStreamer 1.4 has GL support
● Why don't we try integrating them?

What is OpenGL?
● Low level 3D graphics API
● Been around since the 1990's
● Uses dedicated hardware
● Cross-platform

OpenGL in GStreamer
● Contained in the libgstgl library
● GstGLDisplay, GstGLContext are your two main
objects. Analogous to GdkDisplay and
GdkGLContext
● Element base classes, color conversion, uploaders,
GL version/extension checking
● Also contains various elements – gltestsrc,
glvideomixer, glimagesink, gleffects, etc

Demo
videotestsrc ! glfiltercube ! gtkglsink

Implementation - gtkgst
● ~1000 lines of code
● Relies heavily on the GStreamer GL library and the
GL implementation
● Similar in style and scope to
avsamplebuffersink/caopengllayersink

gtkgst - GL
● Currently a proof of concept
● Currently X11 only
● System load causes frame jitter
● Uses Gtk+'s GL context
● Wayland might not be needed due to
efficient/correct subsurfaces

Uses
● Anytime you want to display video from GStreamer
into a Gtk+ application
– e.g. Videos/Totem

Future
● Other window systems planned (wayland, OS X,
Windows)
● GSK

Thanks!
● https://github.com/ystreet/gst-plugins-bad/tree/gtk
● ystreet00 on #gstreamer on freenode