The document is a report on the GStreamer VA-API presented by Víctor Jáquez at the GStreamer Conference 2017, detailing the API's specifications, implementations, and recent developments. It highlights the improvements in GStreamer VA-API, including enhancements in performance, support for various codecs, and contributions from multiple developers over the past year. Additionally, it addresses future plans and ongoing tasks related to the VA-API and its components.

1. The GStreamer VA-API Report
Víctor Jáquez
<vjaquez@igalia.com>
ceyusa
GStreamer Conference 2017
21-22 October (Prague)

2. Agenda
What's VA-API?
What's new?
What's next?

3. What is VA-API?
An API specification
A library implementation
https://github.com/01org/libva

5. Back-ends
Intel driver
https://github.com/01org/intel-vaapi-driver
Mesa drivers
Others (?)

6. Intel driver on Kayblake
$ vainfo
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.0 (libva 2.0.1.pre1)
vainfo: Driver version: Intel i965 driver for Intel(R) Kaby Lake - 2.0.1.pre1 (1.8.3-156-g37613a5)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileH264MultiviewHigh : VAEntrypointVLD
VAProfileH264MultiviewHigh : VAEntrypointEncSlice
VAProfileH264StereoHigh : VAEntrypointVLD
VAProfileH264StereoHigh : VAEntrypointEncSlice
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointEncSlice
VAProfileVP9Profile2 : VAEntrypointVLD

7. GStreamer-VAAPI

8. One Year (more or less)
20 contributors
411 commits
~119 bugs

9. Releases
13 + next stable + master
1.12.3
1.12.2
1.12.1
1.12.0
1.11.91
1.11.90
1.11.2
1.11.1
1.10.5
1.10.4
1.10.3
1.10.2
1.10.1

10. Contributors
Thank you all
Dominique Leuenberger (1)
Edward Hervey (1)
Hyunjun Ko (82)
Jagyum Koo (1)
Jan Schmidt (4)
Julien Isorce (17)
Matt Staples (1)
Nicolas Dufresne (1)
Orestis Floros (9)
Scott D Phillips (6)
Sebastian Dröge (12)
Sreerenj Balachandran (23)
Thomas Petazzoni (2)
Thibault Saunier (1)
Tim-Philipp Müller (8)
Tomas Rataj (1)
Ullysses Artie Eoff (2)
Victor Toso (1)
Yi A Wang (2)

11. What's new?

12. GstObject-ification
GstVaapiDisplay ✓
GstVaapi{Filter, Decoder, Encoder, …}
(leak tracer for the worth!)
Register properties at class initialization

13. Direct Rendering and Upload
Use vaDeriveImage when possible
(problems with Mesa backend)
exporting environment variable GST_VAAPI_ENABLE_DIRECT_RENDERING

14. DMABuf
Improved of DMABuf's usage both upstream and dowstream
Negotiate memory:DMABuf caps feature

15. VA Display initialization
GstGL decides the type of GstVaapiDisplay instance
Fixed initialization for headless systems

16. VA Display sharing
Improved GstVaapiDisplay spread among pipeline
External VA display via gst.vaapi.app.Display context

17. VA Display cache
Removed!

18. Build
Support meson
Support libva 2.0 (VA-API 1.0)

19. Coverity
Fixed all detected issues by static analyzer

20. GstValidate
Normally™ passes all the gst-validate tests

21. Misc
Redirect libva's messages into GStreamer logs

22. Decoders
Faster upstream re-negotiation
low-latency mode for H.264 decoding (flush DPB for streaming)

23. Google Summer of Code 2017
Drops MVC & SVC frames when
back-end doesn't support it
base-only property is on

24. vaapidecodebin
vaapipostproc is plugged "dynamically"
You can control it by exporting the environment variable GST_VAAPI_DISABLE_VPP

25. Encoders
Query possible upload formats
Merge Gst Tags for downstream info
Support upstream reconfiguration

26. Encoders
Profile in caps negotiation (H.264 & H.265)
Handle Regions-Of-Interest (H.264)
Support H.265 main-10 profile

27. Encoders
Constant bit-rate mode VP8 & VP9
Variable bit-rate mode H.265
Added quality level parameter

28. Encoders
AU delimeter in H.264 (streaming)
H.264 macro-block bit-rate control
H.264 & H.265 multi-reference
H.264 & H.265 QP setting either for I/P/B

29. Sink
Automatic color transformation if VPP available
Revamp Wayland backend (Weston)

30. Flexible Encoding Infrastructure
It's an extension to VA-API
Allows more control over different phases of encoding
Trade off quality for speed with their on IPs.
https://bugzilla.gnome.org/show_bug.cgi?id=784667

31. Flexible Encoding Infrastructure
Split the encoding in two phases
1. ENC motion vector calculation and prediction
2. PAK transformations and entropy coding

32. Flexible Encoding Infrastructure
Without FEI:
ENC+PAK is a black box to middle-ware,
With FEI:
Extract the output of ENC and feed PAK
(custom motion vectors and macro-block prediction modes)

33. Flexible Encoding Infrastructure
vaapih264feienc
H.264 encoder provides low-level per macroblock tuning parameters
vaapih264feipreenc (TBI Shall we?)
Pre-processing element which can extract MVs and statistics before calling encode function

34. Flexible Encoding Infrastructure
Highly experimental & unstable
Skylake only
H.264 only

35. What's next?

36. libva's pipeline
FEI
DMABuf
Multi frame processing (?)
Encoding statistics

37. bugzilla's pipeline
Ton of pending tasks and bugs

38. Questions?