Some hardware devices implement the whole codec, but many recent hardwares don’t have the whole codec in hardware. So there needs to be a userspace software module to control it, possibly implement codec parsing. Many APIs exist for this such as OpenMAX IL and VA-API, but even GStreamer. We’ll analyse their pros and cons and suggest a path forward. Olvier Crête, Collabora

1. Open First
Better APIs for Video Codec
accelerators
Olivier Crête

2. Who am I ?
At Collabora since 2007
Close to 5 years on VVoIP for Nokia Maemo / Meego
GStreamer core maintainer
Multimedia Lead
Embedded Recipes 2017 2

3. Type of codecs
●
Software
●
Hardware
– Small CPU/GPU to parse bitstream
●
Hardware Accelerators
– Need CPU software
●
USERSPACE LIBRARY!
Embedded Recipes 2017 3

4. Users of codecs
●
Players
●
Encoders / streamers
●
Transcoders
●
VoIP
●
Content creation software
Embedded Recipes 2017 4

5. Use cases
●
Broadcast production
– High quality, high bit-rate, high expectations
●
User generated content
– Medium quality, low expectations
●
Physical security
– Low quality, high latency, low cost
Embedded Recipes 2017 5

6. Use cases
●
Broadcast production
– High quality, high bitrate
●
End user playback
– Mobile
– TV / STB
Embedded Recipes 2017 6

7. Use cases
●
Video calls
– Latency, latency, low bitrate
●
Transcoding
– Live: Latency!
– Stored: Quality per bit
Embedded Recipes 2017 7

8. ●
Exchange formats
– Encoded
– Raw
●
Memory allocation & registration
●
Metadata
●
Long term maintainability
Embedded Recipes 2017 8
Requirements

9. Exchange formats: Encoded
●
Packetization
– Frame
– Subframe
●
Variants
– H.264 AVC: AVC[1-4], byte-stream
Embedded Recipes 2017 9

10. Exchange formats: Raw content
●
Subsampling: 4:4:4, 4:2:2, 4:2:0, etc
●
Color space: BT.601, BT.709, BT.2020, BT.2100
– Critical for HDR
Embedded Recipes 2017 10

11. Exchange formats: Raw memory layout
●
Planar vs packed
●
Multiple planes
– Multiple DMAbuf fds
●
Alignment requirements
●
Tiled formats
●
Opaque, possible compressed formats
●
Padding, etc
Embedded Recipes 2017 11

12. Memory allocation & registration
●
External vs Internal allocation
– Import vs Export
●
Types of memory
– DMAbuf
– malloc()
Olivier Crête 12

13. Memory pre-allocation & dynamic allocation
●
Pre-registration
– No run-time allocation
– Hard real time
●
Dynamic allocation/registration
– Dynamic pipeline modifications
Embedded Recipes 2017 13

14. Metadata
●
Per frame metadata
– Timestamps and timecodes
– Vertical Ancillary data (VANC): AfD, etc
●
Inter-frame data
– SCTE-35/104
●
Ad insertion points
Embedded Recipes 2017 14

15. Latency information
●
Algorithmic latency
●
Processing latency
●
Deadline modes
Embedded Recipes 2017 15

16. Processing modes
●
Push based
●
Pull based
Embedded Recipes 2017 16

17. Long term maintainability
●
Living, maintained project
●
Open Source code, not a specification
●
Simple C API style
●
No large library dependency
Embedded Recipes 2017 17

18. ●
OpenMAX IL
●
libv4l
●
VA-API
●
GStreamer
Embedded Recipes 2017 18
Current solutions

19. OpenMAX IL
●
Android's subset
●
Minimal largely incomplete vendor implementations
●
Dead upstream , Khronos WG has not met in years“ ”
●
Spec lacks important features leading to fragmentation
●
Forces threading and allocation model
●
Whole framework
Embedded Recipes 2017 19

20. libv4l
●
“Transparent wrapper over kernel API”
●
Similar limitations to v4l API
– Tied to kernel API rules
– Still breaks dmabuf/userptr
●
Only limited maintenance
Embedded Recipes 2017 20

21. VA-API
●
Requires complex code to use it
– See gst-vaapi or mfx
●
Specific codecs only
●
Almost Intel only
●
Only video
Embedded Recipes 2017 21

22. GStreamer
●
Whole framework
●
Forces specific way of working
– Threading
– Pipeline formation
– Allocations
●
Not designed for low latency
Embedded Recipes 2017 22

23. Others
●
FFmpeg / libav
●
MFT (Windows)
Embedded Recipes 2017 23

24. Simple Plugin API (SPA)
Pipewire
Embedded Recipes 2017 24

25. The solution I wish I had designed
●
Designed
– for Pipewire
– by Wim Taymans, GStreamer architect
Embedded Recipes 2017 25

26. Design principles
●
Very inspired by LV2 audio plugin API
●
No pipeline, no framework
●
Registered buffers
– but re-registerable in pause
●
Synchronous and asynchronous
●
Externally provided thread contexts
●
Not limited to codecs
Embedded Recipes 2017 26

27. Thank you!