Development of hardware-based elements
for GStreamer v1.0
Víctor Manuel Jáquez Leal
2012-08-27 Mon
Once upon a time. . .
Once upon a time. . .
Yeah. . .
Yeah. . .
What I learned

Everything should be made as simple as possible,
but not simpler.
 Einstein / Zukofsky / Sessions / Ockham...
Meanwhile in GUADEC 2012

Drop abstraction, make our layers thin
Don't stack things anymore
 Lennart Poettering
Then I moved to Spain
The love is still in the air. . .
GStreamer v0.10

Workarounds
Reinventing the wheel
What's AMP?

Asymmetric MultiProcessing
What's IPC?

Inter-Processors Communication
What do I want?
Wat!?

A simple,
bare bones,
home brew

video player
What do I need?

A simple video sink
A video decoder which uses IVAHD
Let's go
Video Sink
No frame buer

But there is DRM/KMS
Direct Rendering Management
Kernel Mode Settings

framebuer
CRTC
Overlay
Encoder
Connector
Kernel Mode Settings
Video Decoders
SysLink 2 is deprecated

omapdcee
Distributed Codec Engine (DCE)

It provides access to the Codec Engine interface on the
co-processor (ducati/M3) from the ...
Ducati
omapdce
History of the TI's IPCs?

- OMAP1/OMAP2
DSPLink - OMAP2/OMAP3/DaVinci
DSP/Bridge - OMAP3
SysLink2 - OMAP4
RPMsg - OMAP4/O...
RPMsg

remoteproc
rpmsg
remoteproc

Loading  Bootup
Power Management
Exception Management
Error Recovery
rpmsg

virtio-based messaging bus that allows kernel drivers to
communicate with remote processors
GStreamer v1.0
HIC SVNT DRACONES
GstBuer

gst_buffer_map()
gst_buffer_unmap()

Buers can contain a list of GstMemory objects
GstMemory

It is a lightweight  refcounted object.
It wraps a region of memory.
Memory is usually created by allocators.
GstKMSMemory

struct _GstKMSMemory {
GstMemory mem;
uint32_t fb_id;
uint32_t handle[4];
void *bo[4];
};
GstAllocator
Allocate memory blocks
Virtual methods
alloc
free

Callbacks
map
unmap
copy
share

Parameters
align
prex
padd...
GstKMSAllocator (alloc)

static GstMemory *
gst_kms_allcator_alloc (GstAllocator * allocator,
gsize size,
GstAllocationPar...
GstKMSAllocator (free)

static void
gst_kms_allocator_free (GstAllocator * self,
GstMemory * mem)
{
GstKMSMemory *buf = GS...
GstKMSAllocator (map / unmap)
static void *
gst_kms_allocator_map (GstMemory * mem,
gsize maxsize,
GstMapFlags flags)
{
Gs...
GstMeta

Describes the low-level properties of the buer content.
It contain variables that can change between each buer
ti...
GstBuerPool

Pre-allocate and recycle buers with the same properties.
After the buer pool is created, it needs to be congu...
GstVideoBuerPool

GstBuerPool + Video Meta + Video Alignment
GstVideoContext

It shares video context between neighbour elements and
the application.
gst-plugins-bad / GST_USE_UNSTABL...
gst_kms_sink_query (GstBaseSink * bsink,
GstQuery * query)
{

GstKMSSink *sink = GST_KMS_SINK (bsink);
const char **types ...
GstVideoDecoder
Base class fore converters of encoded data into raw video
frames.
Conguration.
_start ()
_set_format ()

D...
@TODO

kmssink should support more DRM backends
dcevdec should support more decoders
Implement a DRM buer metadata
Push km...
Lessons I learned

I suck at communication
Communicate your development openly

GStreamer v1.0 looks great!
omapdce/rpmsg ...
Thanks to

Rob Clark
Ohad Ben-Cohen
Nicolas Dechesne
Alessandro Decina
Sebastian Dröge
GStreamer community
Credits

Clear Inner Vision - CC BY-NC-ND 2.0
The 1265 Psalter world map
Rage Comics
Questions?
Upcoming SlideShare
Loading in …5
×

Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)

1,029 views

Published on

By Víctor Jáquez.

GStreamer 1.0 brings new features for memory handling, particularly the management of buffers. Now it is possible to implement elements that make use of memory areas that are not accessed directly by CPU, such as the video memory, continuous memory areas and so on.

The purpose of this talk is to show how we can use these new interfaces for developing GStreamer elements, that are tightly integrated with the hardware. In particular, we will show how we implemented a simple video sink for the pandaboard, using the Direct Rendering Management (DRM) interface.

This element, called kmssink, uses many of the new concepts for memory handling, added in GStreamer v1.0, such as allocators, buffer pools, and so on. We will review these concepts and how they were used in the element.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,029
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)

  1. 1. Development of hardware-based elements for GStreamer v1.0 Víctor Manuel Jáquez Leal 2012-08-27 Mon
  2. 2. Once upon a time. . .
  3. 3. Once upon a time. . .
  4. 4. Yeah. . .
  5. 5. Yeah. . .
  6. 6. What I learned Everything should be made as simple as possible, but not simpler. Einstein / Zukofsky / Sessions / Ockham / . . .
  7. 7. Meanwhile in GUADEC 2012 Drop abstraction, make our layers thin Don't stack things anymore Lennart Poettering
  8. 8. Then I moved to Spain
  9. 9. The love is still in the air. . .
  10. 10. GStreamer v0.10 Workarounds Reinventing the wheel
  11. 11. What's AMP? Asymmetric MultiProcessing
  12. 12. What's IPC? Inter-Processors Communication
  13. 13. What do I want?
  14. 14. Wat!? A simple, bare bones, home brew video player
  15. 15. What do I need? A simple video sink A video decoder which uses IVAHD
  16. 16. Let's go
  17. 17. Video Sink No frame buer But there is DRM/KMS
  18. 18. Direct Rendering Management
  19. 19. Kernel Mode Settings framebuer CRTC Overlay Encoder Connector
  20. 20. Kernel Mode Settings
  21. 21. Video Decoders SysLink 2 is deprecated omapdcee
  22. 22. Distributed Codec Engine (DCE) It provides access to the Codec Engine interface on the co-processor (ducati/M3) from the host. It invokes the hardware accelerated codecs on IVA-HD via rpmsg/remoteproc
  23. 23. Ducati
  24. 24. omapdce
  25. 25. History of the TI's IPCs? - OMAP1/OMAP2 DSPLink - OMAP2/OMAP3/DaVinci DSP/Bridge - OMAP3 SysLink2 - OMAP4 RPMsg - OMAP4/OMAP5 dsp-gateway
  26. 26. RPMsg remoteproc rpmsg
  27. 27. remoteproc Loading Bootup Power Management Exception Management Error Recovery
  28. 28. rpmsg virtio-based messaging bus that allows kernel drivers to communicate with remote processors
  29. 29. GStreamer v1.0 HIC SVNT DRACONES
  30. 30. GstBuer gst_buffer_map() gst_buffer_unmap() Buers can contain a list of GstMemory objects
  31. 31. GstMemory It is a lightweight refcounted object. It wraps a region of memory. Memory is usually created by allocators.
  32. 32. GstKMSMemory struct _GstKMSMemory { GstMemory mem; uint32_t fb_id; uint32_t handle[4]; void *bo[4]; };
  33. 33. GstAllocator Allocate memory blocks Virtual methods alloc free Callbacks map unmap copy share Parameters align prex padding
  34. 34. GstKMSAllocator (alloc) static GstMemory * gst_kms_allcator_alloc (GstAllocator * allocator, gsize size, GstAllocationParams * params) { GstKMSAllocator *self = GST_KMS_ALLOCATOR (allocator); GstKMSMemory *buf = g_slice_alloc (sizeof (GstKMSMemory)); buf-fb_id = -1; buf-bo[0] = omap_bo_new (self-dev, size, OMAP_BO_SCANOUT | OMAP_BO_WC); buf-handle[0] = omap_bo_handle (buf-bo[0]); gst_memory_init (GST_MEMORY_CAST (buf), GST_MEMORY_FLAG_NO_SHARE, allocator, NULL, size, 0, 0, size); return GST_MEMORY_CAST (buf); }
  35. 35. GstKMSAllocator (free) static void gst_kms_allocator_free (GstAllocator * self, GstMemory * mem) { GstKMSMemory *buf = GST_MEMORY_CAST_KMS (mem); omap_bo_del (buf-bo[0]); g_slice_free1 (sizeof (GstKMSMemory), mem); }
  36. 36. GstKMSAllocator (map / unmap) static void * gst_kms_allocator_map (GstMemory * mem, gsize maxsize, GstMapFlags flags) { GstKMSMemory *buf = GST_MEMORY_CAST_KMS (mem); omap_bo_cpu_prep (buf-bo[0], OMAP_GEM_WRITE); return omap_bo_map (buf-bo[0]); } static void gst_kms_allcator_unmap (GstMemory * mem) { GstKMSMemory *buf = GST_MEMORY_CAST_KMS (mem); omap_bo_cpu_fini (buf-bo[0], OMAP_GEM_WRITE); }
  37. 37. GstMeta Describes the low-level properties of the buer content. It contain variables that can change between each buer timestamp, duration, oset, interlacing video alignment, cropping, panning information extra container information global buer properties
  38. 38. GstBuerPool Pre-allocate and recycle buers with the same properties. After the buer pool is created, it needs to be congured. Buer pools increase performance reduces allocation overhead add more possibilities to implement zero-copy memory
  39. 39. GstVideoBuerPool GstBuerPool + Video Meta + Video Alignment
  40. 40. GstVideoContext It shares video context between neighbour elements and the application. gst-plugins-bad / GST_USE_UNSTABLE_API Sharing the DRM le descriptor between the decoder and the sink.
  41. 41. gst_kms_sink_query (GstBaseSink * bsink, GstQuery * query) { GstKMSSink *sink = GST_KMS_SINK (bsink); const char **types = gst_video_context_query_get_supported_types (query); if (!types) return GST_BASE_SINK_CLASS (parent_class)-query (bsink, for (i = 0; types[i]; i++) { if (!strcmp (types[i], drm-fd)) { GstStructure *structure = gst_query_writable_structure (query); gst_structure_set (structure, video-context-type, G_TYPE_STRING, drm-fd, video-context, G_TYPE_INT, sink-fd, NULL); } }
  42. 42. GstVideoDecoder Base class fore converters of encoded data into raw video frames. Conguration. _start () _set_format () Data processing. get or _parse () frames _handle_frame () call gst_video_decoder_finish_frame () or gst_video_decoder_drop_frame () Shutdown phase. _stop ()
  43. 43. @TODO kmssink should support more DRM backends dcevdec should support more decoders Implement a DRM buer metadata Push kmssink and dcevdec (??) to bugzilla Watch a movie with my video player
  44. 44. Lessons I learned I suck at communication Communicate your development openly GStreamer v1.0 looks great! omapdce/rpmsg + remoteproc too!
  45. 45. Thanks to Rob Clark Ohad Ben-Cohen Nicolas Dechesne Alessandro Decina Sebastian Dröge GStreamer community
  46. 46. Credits Clear Inner Vision - CC BY-NC-ND 2.0 The 1265 Psalter world map Rage Comics
  47. 47. Questions?

×