Inexpensive set-top boxes are everywhere and many of them are powered by AmLogic SoCs. These chips provide 4K H.265/VP9 video decoding and as of the last couple years, have a very good Mainline Linux and U-Boot support. Unfortunately, most of the multimedia features, like Hardware Accelerated OpenGL ES, Video Decoding, Audio Playback, and Video Rendering were missing from Upstream projects like Linux, Mesa, Gstreamer or FFmpeg. Thankfully a growing community of independent developers have managed to get most of these features to work using upstream only source code. Maxime will present the challenges and benefits to have Multimedia support using upstream projects and will detail the technical challenges to offer a complete Multimedia support on these Amlogic SoCs. The audience is anyone interested in ARM SoCs, multimedia, upstream development process or learning more about the architecture of the AmLogic application processors. Attendees can expect a presentation on the actual work of code upstreamed for the Amlogic SoCs, a multimedia overview and challenges on non-x86 platforms, an overview of the hardware architecture, and much more.

1. Upstream Multimedia on Amlogic
SoCs : From Fiction To Reality
Embedded recipes - 2018
Maxime Jourdan

2. Neil’s talk last year

3. 1. Amlogic Multimedia status on mainline Linux
2. Implementing a V4L2 Memory-to-Memory video decoder for
Amlogic SoCs
3. Current userspace development for decoder-to-display
pipelines
4. Demo!
Topics

4. Amlogic Multimedia status on mainline Linux (1/3)
● Display : DRM/KMS Drivers
○ Support
■ HDMI / Composite output on S905, S905D/X, S912
■ Primary Plane RGB with Alpha
○ WiP
■ YUV overlay plane support
■ 4K P30 (HDMI 1.4)
○ To be done
■ HDMI 2.0 support
■ Support for Composite/HDMI on older S802/S812/S805
■ Support for upcoming SoCs A113D/S905D/X2, S922, … (+ DSI output)

5. Amlogic Multimedia status on mainline Linux (2/3)
● CEC
○ Support
■ CEC support on S805X (non tested), S905/S905X/S905D/S912 and later
○ To be done
■ Suspend support : the driver should communicate the current Logical & Physical
Address before the suspend, then the Suspend firmware code can continue
communicating with the TV and wake up the device.

6. Amlogic Multimedia status on mainline Linux (3/3)
● Audio : Alsa SoC
○ Support
■ AXG SoC family (A113D/X) : (v4.19 - v4.20)
● TDM input and output (max 128 channels - 192KHz) * 3
● SPDIF output
● PDM input
● Multicodec card driver
○ WiP
■ AXG SPDIF input
■ Gx SoC family: i2s output and spdif output
■ Internal DAC Codec (GXL and G12)
○ To be done
■ G12 (S905D/X2, S922) SoC family support (AXG based)

7. Implementing a V4L2 Memory-to-Memory video decoder for
Amlogic SoCs

8. V4L2 M2M Video Decoder
● Video4Linux2: kernel framework for media drivers (cameras,
displays, video accelerators ..)
● Memory2Memory: both inputs and outputs are in RAM
● Device node: /dev/videoX
● Increasingly supported for many SoC vendors: Samsung,
Qualcomm, MediaTek..
○ Not upstream yet: Allwinner, Broadcom, Rockchip, Amlogic

9. Stateful Video Decoders
● Possess a hardware bitstream parser
● Autonomous
● Almost always firmware-assisted
● Examples: Amlogic, Qualcomm, Samsung, MediaTek

10. Stateless Video Decoders
● Need assistance from userspace for bitstream parsing and
updating the local state
● Rely on the V4L2 Request API
● Examples: Allwinner, Rockchip

11. Amlogic video decoder
MJPEG, MPEG2,
MPEG4
H.264 HEVC
HEVC
10-bit
VP9
S805
1080p60
1080p60 1080p30 N/A N/A
S905
4K30 4K60 4K60
N/A
S905X
S912
4K60
AML-S905X-CC “Le Potato”Odroid-C2 (S905) Khadas VIM2 (S912)

12. Amlogic video decoder
● Custom driver available in vendor kernel
drivers/amlogic/amports/
○ ~20,000 lines of code for the core
○ ~3,000 lines of code for H.264
○ ~10,000 lines of code for HEVC
● Non standard API (~20 custom ioctls)
● In-kernel media player logic

13. Where do you start ?
● Power / Clocks / IRQs
● Where are the firmware files ?
● Where do you write the bitstream ?
● How do you get the frames back ?
● How much is in vendor kernel / userspace blob ?

14. Clocks - vendor kernel
amports/arch/clkgx.c
#define VDEC1_CLOCK_ON()
do { WRITE_HHI_REG_BITS(HHI_VDEC_CLK_CNTL, 1, 8, 1);
WRITE_HHI_REG_BITS(HHI_VDEC3_CLK_CNTL, 0, 15, 1);
WRITE_HHI_REG_BITS(HHI_VDEC3_CLK_CNTL, 0, 8, 1);
WRITE_VREG_BITS(DOS_GCLK_EN0, 0x3ff, 0, 10);
} while (0)
#define VDEC1_648M()
WRITE_HHI_REG_BITS(HHI_VDEC_CLK_CNTL, (6 << 9) | (0), 0, 16)

15. Firmwares - vendor kernel
amports/arch/ucode/h264/h264c_linux.h
static const u32 MicroCode[] __initconst = {
0x06810001, 0x06800000, 0x0d000001, 0x07400040, 0x064c0800,
0x09800300, 0x00000000, 0x060c0800, 0x0649d019, 0x0c006580,
0x0649c134, 0x06880000, 0x060c2900, 0x080c0002, 0x06bfda80,
0x07c02d40, 0x06030400, 0x00400000, 0x0c007d80, 0x00000000,
0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x0c7ffe80,
...

16. Writing the bitstream
VIFIFO
ESPARSER
Input Buffer
16 MiB
Input Buffer
But no IRQ triggered by the parser..

17. Writing the bitstream
VIFIFO
ESPARSER
Input Buffer
16 MiB
Input Buffer
4KiB
Padding
Start
Code
00 00 01 ff

18. Retrieving the decoded frames
Canvas
ID: 0-255
Width x Height
Buffer Physical
Address
Block Mode
Endianness
Decoder
Frame 0
Frame 1
Y plane: Canvas 0
U/V plane: Canvas 1
Y plane: Canvas 2
U/V plane: Canvas 3

19. VIFIFOI P
ESPARSERBPBB
Bitstream packets
Decoder
VDEC_1
Canvas
Frame
N
Decoded picture buffer
Frame
1
Frame
0
FW
CPU
B

20. Testing
● FFmpeg
○ ffmpeg -c:v h264_v4l2m2m -i input.mp4 -f
rawvideo out.nv12
● GStreamer
○ gst-launch-1.0 filesrc location=input.mp4 !
parsebin ! v4l2video0dec ! fakesink
● Chromium (Igalia)
○ GN_ARGS += “use_v4l2_codec use_linux_v4l2_only”
○ ln -sf video0 /dev/video-dec0

22. 64x32 tiled mode

23. Block Mode Linear.. but each 8 pixels are reversed ?

24. Correct byte swapping: Proper NV12!

25. Linux DRM - Plane
For a full Description of the DRM Subsystem :
https://events.static.linuxfound.org/sites/events/files/slides/brezillon-drm-kms.pdf

26. Linux DRM - Plane Blending
The DRM Plane abstraction permits :
● Using Hardware assisted Color Space Conversion
○ RGB to YUV
○ NV12 to RGB
● Using Hardware assisted Alpha Blending between multiple planes
○ OSD Plane blended with underlying Video Plane
● Digital TV Hardware usually have an “OSD” Primary Plane and a
“Video” Overlay Plane behind the OSD plane
● Generic Hardware can configure the Order of the planes : z-order

27. Linux DRM - Plane Blending
3840x2160 NV12 Plane (from Decoder)
1920x1080 ARGB8888 Plane
Upscaling
Color Space
Convert
Alpha
Blending
3840x2160 YUV422 Frame for HDMI

28. Sharing buffers with dmabuf
● Kernel framework to share physical buffers between devices
● Uses file descriptors
● Consumer devices import the buffers
● Producer devices export the buffers
● V4L2 exposes VIDIOC_EXPBUF to export buffers.

29. Rendering frames in zero-copy
● DRM Prime with dmabuf
○ Requires plane that supports NV12. Preferably with a scaler.
○ No dependency on GPU for the video path
● EGL dmabuf import
○ Requires a OpenGL stack
○ A single RGB plane is enough

30. Rendering - DRM Prime
Media
Player
V4L2 M2M
decoder
VIDIOC_EXPBUF
RGB DRM plane
NV12 DRM plane
GPU
drmModeAddFB2
Subtitle

31. Rendering - EGL dmabuf import
Media
Player
V4L2 M2M
decoder
VIDIOC_EXPBUF
GPU
Subtitle
eglCreateImageKHR
EGL_LINUX_DMA_BUF_EXT
RGB DRM plane

32. Media players
Kodi mpv GStreamer Chromium
V4L2 +
DRM Prime
kmssink
V4L2 +
EGL import
glimagesink
uses
FFmpeg

33. Resources
● LibreELEC - “Just enough OS for KODI”
○ https://github.com/chewitt/LibreELEC.tv/tree/amlogic
○ PROJECT=Amlogic_GX DEVICE=LePotato ARCH=arm make image
● OpenEmbedded meta-meson layer
○ https://github.com/superna9999/meta-meson
● Communication
○ http://linux-meson.com/
○ #linux-amlogic on Freenode
○ linux-amlogic@lists.infradead.org

34. Thanks to..!
● #linux-amlogic community
● #v4l community
● LibreElec community
○ Lukas Rusak for DRM Prime support in Kodi/FFmpeg
○ Christian Hewitt for integration and testing
● Nicolas Dufresne (Collabora) for help with gstreamer