The Vulkan Video extension has been in development since 2018, with the intention of exposing video codec blocks as a new acceleration API. AVC and HEVC are shipping with good support for decode, and nascent encode support. AV1 and VP9 are in the works. While several APIs exist amid the convenience/control/portability tradeoff, the Vulkan extension exposes a much greater level of fine-grained control, portability and tight-integration that none of the existing solutions offer, at the expense of convenience. In this talk we will very briefly explore the API, cover the recent developments of the extension, and discuss plans for future work. This talk will provide background information for anyone interested in integrating codec acceleration within their application. (c) X.Org Developer's Conference (XDC) 2023 October 17-19, 2023 A Coruña (Spain) https://indico.freedesktop.org/event/4/

1. Status of the Vulkan Video
ecosystem
Charlie Turner
XDC, A Coruña, October 2023

2. Recap
Status of the Vulkan Video ecosystem
Charlie Turner, 18th October 2023
● A vendor and platform agnostic, stateless, video acceleration API.
● Offers fine-grained control over video processing & scheduling.
● Distribution of stream processing across CPU cores and codec
hardware.
● Seamless integration with Vulkan graphics and presentation.
● Application portability from small embedded systems to high
performance servers.
● Video texturing, generative models, DLSS, NRD denoising, streaming,
post-processing, light fields, XR, …

3. Progress
● 2022 was the year of the codec-independent APIs, as well as finalizing
the AVC & HEVC decode extensions. Going into 2023 the TSG had a
good foundation for future work.
● AVC/HEVC have been the initial focus simply due to market share. AV1
and VP9 are on the roadmap.
● The CTS was problematic, bitstream parsing is tricky.
○ Igalia have tidied this initial version up, and extended the test
coverage for AVC and HEVC.
● 2023's focus has been on finalizing encode APIs for AVC and HEVC, as
well as working on other video extensions.
Status of the Vulkan Video ecosystem
Charlie Turner, 18th October 2023

4. Progress
Status of the Vulkan Video ecosystem
Charlie Turner, 18th October 2023
NVIDIA Sample App
FFmpeg Vulkan
FFmpeg VA-API
FPS
4.2 ms / frame 2160p
1.2 ms / frame 1080p

5. Drivers (decode)
Status of the Vulkan Video ecosystem
Charlie Turner, 18th October 2023
● AVC/HEVC
○ anv - passing CTS
○ radv - nearly passing CTS, just interleaved sessions failing
○ NVIDIA/AMD Pro - passing CTS
● AV1 Mesa vendor extension
○ radv - 270/275 pass rate on Fluster
○ The KHR version is cooking.

6. Community
● FFmpeg v6.1
○ Released earlier this year with Vulkan Video support for AVC,
HEVC and AV1.
○ Thank you Lynne!
● GStreamer
○ AVC decode hopefully will land in v1.24. MR open since May 2023.
○ AVC+HEVC encode is WIP
○ Thank you Victor & Stéphane from Igalia!
● WickedEngine:
https://wickedengine.net/2023/05/07/vulkan-video-decoding/
Status of the Vulkan Video ecosystem
Charlie Turner, 18th October 2023

7. Future work
● AV1 KHR, VP9.
● Creating libraries to ease the use of the extension.
● Android declares the extension incompatible
"[C-1-11] MUST NOT enumerate support for the
VK_KHR_video_queue, VK_KHR_video_decode_queue, or
VK_KHR_video_encode_queue extensions."
● And much more!
Status of the Vulkan Video ecosystem
Charlie Turner, 18th October 2023

8. References
● An concise and Vulkan Video specific overview of video compression topics
covering essential background information for working with the extension
https://www.rastergrid.com/blog/multimedia/2021/05/video-compression-bas
ics/
● Official Khronos blog post introducing the extension
https://www.khronos.org/blog/an-introduction-to-vulkan-video
● Vulkanized 2023: A deep dive into Vulkan Video
https://www.youtube.com/watch?v=R5x6_nBRrv4
● https://github.com/nvpro-samples/vk_video_samples
● "Video decoding in Vulkan: A brief overview
ofbVK_KHR_video_{queue,decode}" -
https://www.youtube.com/watch?v=uvz1GJ8A6ZM
Status of the Vulkan Video ecosystem
Charlie Turner, 18th October 2023