Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Embedded Recipes 2018 - Upstream multimedia on amlogic so cs from fiction to reality - Maxime Jourdan


Published on

Inexpensive set-top boxes are everywhere and many of them are powered by AmLogic SoCs. These chips provide 4K H.265/VP9 video decoding and as of the last couple years, have a very good Mainline Linux and U-Boot support.

Unfortunately, most of the multimedia features, like Hardware Accelerated OpenGL ES, Video Decoding, Audio Playback, and Video Rendering were missing from Upstream projects like Linux, Mesa, Gstreamer or FFmpeg. Thankfully a growing community of independent developers have managed to get most of these features to work using upstream only source code.

Maxime will present the challenges and benefits to have Multimedia support using upstream projects and will detail the technical challenges to offer a complete Multimedia support on these Amlogic SoCs.

The audience is anyone interested in ARM SoCs, multimedia, upstream development process or learning more about the architecture of the AmLogic application processors.

Attendees can expect a presentation on the actual work of code upstreamed for the Amlogic SoCs, a multimedia overview and challenges on non-x86 platforms, an overview of the hardware architecture, and much more.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Embedded Recipes 2018 - Upstream multimedia on amlogic so cs from fiction to reality - Maxime Jourdan

  1. 1. Upstream Multimedia on Amlogic SoCs : From Fiction To Reality Embedded recipes - 2018 Maxime Jourdan
  2. 2. Neil’s talk last year
  3. 3. 1. Amlogic Multimedia status on mainline Linux 2. Implementing a V4L2 Memory-to-Memory video decoder for Amlogic SoCs 3. Current userspace development for decoder-to-display pipelines 4. Demo! Topics
  4. 4. Amlogic Multimedia status on mainline Linux (1/3) ● Display : DRM/KMS Drivers ○ Support ■ HDMI / Composite output on S905, S905D/X, S912 ■ Primary Plane RGB with Alpha ○ WiP ■ YUV overlay plane support ■ 4K P30 (HDMI 1.4) ○ To be done ■ HDMI 2.0 support ■ Support for Composite/HDMI on older S802/S812/S805 ■ Support for upcoming SoCs A113D/S905D/X2, S922, … (+ DSI output)
  5. 5. Amlogic Multimedia status on mainline Linux (2/3) ● CEC ○ Support ■ CEC support on S805X (non tested), S905/S905X/S905D/S912 and later ○ To be done ■ Suspend support : the driver should communicate the current Logical & Physical Address before the suspend, then the Suspend firmware code can continue communicating with the TV and wake up the device.
  6. 6. Amlogic Multimedia status on mainline Linux (3/3) ● Audio : Alsa SoC ○ Support ■ AXG SoC family (A113D/X) : (v4.19 - v4.20) ● TDM input and output (max 128 channels - 192KHz) * 3 ● SPDIF output ● PDM input ● Multicodec card driver ○ WiP ■ AXG SPDIF input ■ Gx SoC family: i2s output and spdif output ■ Internal DAC Codec (GXL and G12) ○ To be done ■ G12 (S905D/X2, S922) SoC family support (AXG based)
  7. 7. Implementing a V4L2 Memory-to-Memory video decoder for Amlogic SoCs
  8. 8. V4L2 M2M Video Decoder ● Video4Linux2: kernel framework for media drivers (cameras, displays, video accelerators ..) ● Memory2Memory: both inputs and outputs are in RAM ● Device node: /dev/videoX ● Increasingly supported for many SoC vendors: Samsung, Qualcomm, MediaTek.. ○ Not upstream yet: Allwinner, Broadcom, Rockchip, Amlogic
  9. 9. Stateful Video Decoders ● Possess a hardware bitstream parser ● Autonomous ● Almost always firmware-assisted ● Examples: Amlogic, Qualcomm, Samsung, MediaTek
  10. 10. Stateless Video Decoders ● Need assistance from userspace for bitstream parsing and updating the local state ● Rely on the V4L2 Request API ● Examples: Allwinner, Rockchip
  11. 11. Amlogic video decoder MJPEG, MPEG2, MPEG4 H.264 HEVC HEVC 10-bit VP9 S805 1080p60 1080p60 1080p30 N/A N/A S905 4K30 4K60 4K60 N/A S905X S912 4K60 AML-S905X-CC “Le Potato”Odroid-C2 (S905) Khadas VIM2 (S912)
  12. 12. Amlogic video decoder ● Custom driver available in vendor kernel drivers/amlogic/amports/ ○ ~20,000 lines of code for the core ○ ~3,000 lines of code for H.264 ○ ~10,000 lines of code for HEVC ● Non standard API (~20 custom ioctls) ● In-kernel media player logic
  13. 13. Where do you start ? ● Power / Clocks / IRQs ● Where are the firmware files ? ● Where do you write the bitstream ? ● How do you get the frames back ? ● How much is in vendor kernel / userspace blob ?
  14. 14. Clocks - vendor kernel amports/arch/clkgx.c #define VDEC1_CLOCK_ON() do { WRITE_HHI_REG_BITS(HHI_VDEC_CLK_CNTL, 1, 8, 1); WRITE_HHI_REG_BITS(HHI_VDEC3_CLK_CNTL, 0, 15, 1); WRITE_HHI_REG_BITS(HHI_VDEC3_CLK_CNTL, 0, 8, 1); WRITE_VREG_BITS(DOS_GCLK_EN0, 0x3ff, 0, 10); } while (0) #define VDEC1_648M() WRITE_HHI_REG_BITS(HHI_VDEC_CLK_CNTL, (6 << 9) | (0), 0, 16)
  15. 15. Firmwares - vendor kernel amports/arch/ucode/h264/h264c_linux.h static const u32 MicroCode[] __initconst = { 0x06810001, 0x06800000, 0x0d000001, 0x07400040, 0x064c0800, 0x09800300, 0x00000000, 0x060c0800, 0x0649d019, 0x0c006580, 0x0649c134, 0x06880000, 0x060c2900, 0x080c0002, 0x06bfda80, 0x07c02d40, 0x06030400, 0x00400000, 0x0c007d80, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x0c7ffe80, ...
  16. 16. Writing the bitstream VIFIFO ESPARSER Input Buffer 16 MiB Input Buffer But no IRQ triggered by the parser..
  17. 17. Writing the bitstream VIFIFO ESPARSER Input Buffer 16 MiB Input Buffer 4KiB Padding Start Code 00 00 01 ff
  18. 18. Retrieving the decoded frames Canvas ID: 0-255 Width x Height Buffer Physical Address Block Mode Endianness Decoder Frame 0 Frame 1 Y plane: Canvas 0 U/V plane: Canvas 1 Y plane: Canvas 2 U/V plane: Canvas 3
  19. 19. VIFIFOI P ESPARSERBPBB Bitstream packets Decoder VDEC_1 Canvas Frame N Decoded picture buffer Frame 1 Frame 0 FW CPU B
  20. 20. Testing ● FFmpeg ○ ffmpeg -c:v h264_v4l2m2m -i input.mp4 -f rawvideo out.nv12 ● GStreamer ○ gst-launch-1.0 filesrc location=input.mp4 ! parsebin ! v4l2video0dec ! fakesink ● Chromium (Igalia) ○ GN_ARGS += “use_v4l2_codec use_linux_v4l2_only” ○ ln -sf video0 /dev/video-dec0
  21. 21. 64x32 tiled mode
  22. 22. Block Mode Linear.. but each 8 pixels are reversed ?
  23. 23. Correct byte swapping: Proper NV12!
  24. 24. Linux DRM - Plane For a full Description of the DRM Subsystem :
  25. 25. Linux DRM - Plane Blending The DRM Plane abstraction permits : ● Using Hardware assisted Color Space Conversion ○ RGB to YUV ○ NV12 to RGB ● Using Hardware assisted Alpha Blending between multiple planes ○ OSD Plane blended with underlying Video Plane ● Digital TV Hardware usually have an “OSD” Primary Plane and a “Video” Overlay Plane behind the OSD plane ● Generic Hardware can configure the Order of the planes : z-order
  26. 26. Linux DRM - Plane Blending 3840x2160 NV12 Plane (from Decoder) 1920x1080 ARGB8888 Plane Upscaling Color Space Convert Alpha Blending 3840x2160 YUV422 Frame for HDMI
  27. 27. Sharing buffers with dmabuf ● Kernel framework to share physical buffers between devices ● Uses file descriptors ● Consumer devices import the buffers ● Producer devices export the buffers ● V4L2 exposes VIDIOC_EXPBUF to export buffers.
  28. 28. Rendering frames in zero-copy ● DRM Prime with dmabuf ○ Requires plane that supports NV12. Preferably with a scaler. ○ No dependency on GPU for the video path ● EGL dmabuf import ○ Requires a OpenGL stack ○ A single RGB plane is enough
  29. 29. Rendering - DRM Prime Media Player V4L2 M2M decoder VIDIOC_EXPBUF RGB DRM plane NV12 DRM plane GPU drmModeAddFB2 Subtitle
  30. 30. Rendering - EGL dmabuf import Media Player V4L2 M2M decoder VIDIOC_EXPBUF GPU Subtitle eglCreateImageKHR EGL_LINUX_DMA_BUF_EXT RGB DRM plane
  31. 31. Media players Kodi mpv GStreamer Chromium V4L2 + DRM Prime kmssink V4L2 + EGL import glimagesink uses FFmpeg
  32. 32. Resources ● LibreELEC - “Just enough OS for KODI” ○ ○ PROJECT=Amlogic_GX DEVICE=LePotato ARCH=arm make image ● OpenEmbedded meta-meson layer ○ ● Communication ○ ○ #linux-amlogic on Freenode ○
  33. 33. Thanks to..! ● #linux-amlogic community ● #v4l community ● LibreElec community ○ Lukas Rusak for DRM Prime support in Kodi/FFmpeg ○ Christian Hewitt for integration and testing ● Nicolas Dufresne (Collabora) for help with gstreamer