This document discusses updates and improvements to the Linux media controller API. It describes flaws in the original design where only data streams could be represented, not control links. The next generation media controller API aims to represent all media devices by allowing different types of links for data and control. It will also support retrieving the full media graph topology in one call and adding custom properties. Example diagrams show how the updated API can represent complex device topologies like those found in TV sets and USB cameras.

1. Samsung Open Source Group 1
Digital TV Kernel Pipelines via Media
Controller API
Mauro Carvalho Chehab
Linux Kernel Expert
Samsung Open Source Group
Linux Kernel Summit
Seoul, South Korea: Oct.26-28, 2015

2. Samsung Open Source Group 2
Media Controller – MC - API
● Goal: to discover/modify device's internal topology for any media device.
● Devices are modeled as an oriented graph with 3 object types:
– Entity: a media building block:
● physical/logical hardware devices, IP blocks, connectors, data/userspace I/O
● Two types of entities defined: devnode and v4l2-subdev
● Entities have a sub-type, like: sensor, flash, lens, decoder
● Types/subtypes are extended via defines at linux/media.h
– Pad is data connection endpoints at the entity
●
sink pads for output data streams;
●
source pads for input data streams;
– links connect two pads
– Devnode interfaces are a property of the entities

3. Samsung Open Source Group 3
Original scope of the MC
● It was conceived with V4L2, DVB, ALSA and FB in mind.
– However, it MC core and drivers had, until last year,
only a subset of V4L2 types of devices.

4. Samsung Open Source Group 4
TV Set/STB Simplified Diagram
Note: data/control interactions with userspace is not shown, to simplify the diagram,
but it happens on all pipelines, via multiple interfaces. Remote controller is also not shown.

5. Samsung Open Source Group 5
Simple PC Customer DTV Diagram
● NOTE: on DVB subsystem:
– The demux0, frontend0 and ca0 devnodes control
the hardware;
– The dvr0 devnode typically handles I/O (demux0 may
optionally also handle I/O).

6. Samsung Open Source Group 6
Example: OMAP3 MC graph

7. Samsung Open Source Group 7
Flaws With Current MC Design
● Only elements that streams data are represented
– Exception: Flash/LED entities don't stream data, but are represented as entities
as well
● The entity devnodes are assumed to have DMA hardware engines associated with
them. This is only true for webcam/video stream only hardware:
– ALSA and DVB have control nodes, and the DMA engines are internal elements
of the pipelines. The same is true for USB media devices;
– V4L2 has radio devnodes that are only control nodes;
● There's no way to represent control links, as the Media Controller were
implemented to represent only data streams.
● The ALSA entity types and the DVB entity properties aren't OK.
So, despite being planned to be used since the beginning for
V4L2, ALSA and DVB, only a subset of media devices can
be represented with the current approach.

8. Samsung Open Source Group 8
Media Controller: Next Generation
● Goal #1: To use extend MC to be used for all media controller devices
● Goal #2: (future) To allow dynamic graph tree removals/reinserts
● For Goal#1 some concepts were changed:
– Interface: an element that represents userspace control via a Linux
Kernel API (devnode, sysfs, …);
– Entity: won't have type/subtype anymore. Subtypes were renamed to
function. One entity may have more than one function (in future);
– Link: now, there are two types of links: data links (between two pads) and
interface links (between an interface and an entity).

9. Samsung Open Source Group 9
Media Controller: Next Generation
● To keep backward compatibility, the old ioctls won't suffer any
change, but will be marked as deprecated at the documentation;
● A new ioctl with the extended functionality were added:
MEDIA_IOC_G_TOPOLOGY, likely on Kernel 4.5.
– It can get all the media graph on one call
● A new properties API will be added to allow different
subsystems/drivers to add other properties to the objects (future)
● A new ioctl will likely replace MEDIA_IOC_SETUP_LINK ioctl, in
order to allow setup a link set atomically (future).

10. Samsung Open Source Group 10
Example: OMAP3 MC Next Gen

11. Samsung Open Source Group 11
Example: au0828

12. Samsung Open Source Group 12
Interface
struct media_v2_topology {
__u32 topology_version;
__u32 num_entities;
struct media_v2_entity *entities;
__u32 num_interfaces;
struct media_v2_interface *interfaces;
__u32 num_pads;
struct media_v2_pad *pads;
__u32 num_links;
struct media_v2_link *links;
__u32 reserved[40]; /* Reserved for expansion */
};
#define MEDIA_IOC_G_TOPOLOGY_IOWR('|', 0x04,
struct media_v2_topology)

13. Samsung Open Source Group 13
Interface
struct media_v2_entity {
__u32 id;
char name[64];
__u32 function;
__u16 reserved[12];
};
struct media_v2_intf_devnode {
__u32 major;
__u32 minor;
};
struct media_v2_interface {
__u32 id;
__u32 intf_type;
__u32 flags;
__u32 reserved[9];
union {
struct media_v2_intf_devnode
devnode;
__u32 raw[16];
};
};
struct media_v2_pad {
__u32 id;
__u32 entity_id;
__u32 flags;
__u16 reserved[9];
};
struct media_v2_link {
__u32 id;
__u32 source_id;
__u32 sink_id;
__u32 flags;
__u32 reserved[5];
};

14. Samsung Open Source Group 14
Thank You!