1. Video Drivers
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com>
All Rights Reserved.

2. What to Expect?
Introduction to the Linux' video subsystem
Original Video Interfaces
Frame Buffer
Console
Video For Linux v2 (v4l2) Framework
“Video Device” structure
v4l2 buffers & the queue management
v4l2 devices & subdevices
Putting them all together: Porting
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 2
All Rights Reserved.

3. Video History
Plain Character Drivers
Lot of hardware dependency exposed (ioctls)
Getting video working was non-trivial (monitors)
Needed additional video APIs, very often
Called for better abstraction
Video Layers added over character drivers
Frame Buffer
Console
But all these were for only video
More complex devices & requirements evolved
Need of Integration between others like audio
Complete framework designed: Video For Linux (v4l)
Further evolved & matured: Video For Linux (v4l2)
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 3
All Rights Reserved.

4. A 10000 feet view
User Space
Audio/Video Players/Recorders, Video Conferencing
X Windows (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files
Vertical: Various Character Drivers w/ additional Video APIs
Video Output Video Capture Image, ...
Audio Drivers
Drivers Drivers Drivers
Horizontal: PCI, AGP, PCI Express, USB, I2C ................. PCI, I2S, USB, I2C
Kernel Space
Hardware Space
Video Output Devices Video Capture Devices Audio Out/In Devices Printers/Scanners/...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 4
All Rights Reserved.

5. Original Video Subsystem
FB-aware applications
Such as video players Console
setterm
UM
X Windows fbset
Driver User Space
/dev/fbX /dev/ttyX
Virtual Terminal lpcons usb_uart
Driver
fbcon .. vgacon
Common Frame Buffer API over /dev/fbX
Frame Buffer Driver
Kernel Space
Hardware Space
Video Card X VGA Card Printer USB UART
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 5
All Rights Reserved.

6. FB Programming Interface
Header: <linux/fb.h>
Data Structures
struct fb_info – Main data structure
struct fb_ops – Entry points
struct fb_var_screen_info – Resolution, ...
struct fb_fix_screen_info – FB start addr, ...
struct fb_cmap – RGB colour map
APIs
int register_framebuffer(struct fb_info *fb_info);
int unregister_framebuffer(struct fb_info *fb_info);
struct fb_info *framebuffer_alloc(size_t size, struct device *dev);
void framebuffer_release(struct fb_info *info);
int fb_alloc_cmap(struct fb_cmap *cmap, int len, int transp);
void fb_dealloc_cmap(struct fb_cmap *cmap);
Source: drivers/video/
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 6
All Rights Reserved.

7. struct fb_ops
fb_open – Open
fb_release – Close
fb_check_var – Check video parameters
fb_set_par – Set video controller registers
fb_setcolreg – Create pseudo colour palette map
fb_blank – Blank / Unblank display
fb_fillrect – Fill rectangle with pixel lines
fb_copyarea – Copy rectangular area between screens
fb_imageblit – Draw an image to the display
fb_rotate – Rotate the display
fb_ioctl – Ioctl interface for device specific commands
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 7
All Rights Reserved.

8. Console Programming Interface
Header: <linux/console.h>
Data Structures
struct console – top-level console driver
struct consw – bottom-level console driver
APIs
void register_console(struct console *);
int unregister_console(struct console *);
int register_con_driver(const struct consw *csw, int first,
int last);
int unregister_con_driver(const struct consw *csw);
Source: drivers/video/console
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 8
All Rights Reserved.

9. Reference Video Drivers
For Frame Buffer drivers
Browse drivers/video/
Example: vga16fb.c
For Console drivers
Browse drivers/video/console/
Example: fbcon.c
Font Example: font_8x8.c
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 9
All Rights Reserved.

10. “Video in Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/XXX)
Vertical: Video Device Drivers (Character w/ additional video ioctls)
File Operations: open, release, read, poll, mmap, unlocked_ioctl
Video Buffer Management Video Control Management
Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2C
Kernel Space
Hardware Space
Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 10
All Rights Reserved.

11. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX)
Vertical: Video Device Drivers (Character w/ additional video ioctls)
struct video_device <media/v4l2-dev.h>
struct v4l2_file_operations: open, release, read, poll, mmap, unlocked_ioctl
Video Buffer Management v4l2 Control Management
Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2C
Kernel Space
Hardware Space
Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 11
All Rights Reserved.

12. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX)
Vertical: Video Device Drivers (Character w/ additional video ioctls)
struct video_device <media/v4l2-dev.h>
open, release, read, poll, mmap, unlocked_ioctl = video_ioctl2
struct v4l2_ioctl_ops
Video Buffer Management v4l2 Control Management
2
PCI, AGP, PCI Express, USB, I S PCI, USB, I2C
Kernel Space
Hardware Space
Devices: Video Output/Capture, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 12
All Rights Reserved.

13. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX)
Vertical: Video Device Drivers (Character w/ additional video ioctls)
struct video_device <media/v4l2-dev.h>
struct v4l2_file_operations struct v4l2_ioctl_ops <media/v4l2-ioctl.h>
Video Buffer Queue Management <media/videobuf2-core.h>
Q: struct vb2_queue; Q ops: struct vb2_ops;
Memory ops: struct vb2_mem_ops
v4l2 Control Management
PCI, USB, I2C
Scatter Gather Virtually Contiguous All Contiguous
videobuf2-dma-sg.h videobuf2-vmalloc.h videobuf2-dma-contig.h
Kernel Space
Hardware Space
Devices: Video Output/Capture, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 13
All Rights Reserved.

14. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX)
Vertical: Video Device Drivers (Character w/ additional video ioctls)
struct video_device <media/v4l2-dev.h>
struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler
<media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h>
Video Buffer Queue Management <media/videobuf2-core.h>
Q: struct vb2_queue; Q ops: struct vb2_ops;
v4l2 Control Management
Memory ops: struct vb2_mem_ops PCI, USB, I2C
Kernel Space
Hardware Space
Devices: Video Output/Capture, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 14
All Rights Reserved.

15. struct video_device
#include <media/v4l2-dev.h>
struct device *parent; // Dev structure from the horizontal
char name[32];
struct v4l2_file_operations *fops;
struct v4l2_ioctl_ops *ioctl_ops;
struct v4l2_ctrl_handler *ctrl_handler;
void (*release)(struct video_device *vdev);
v4l2_std_id tvnorms;
v4l2_std_id current_norm;
int debug;
struct mutex *lock;
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 15
All Rights Reserved.

16. Video Device Programming
Header: <media/v4l2-dev.h>
APIs
int video_register_device
(struct video_device *vdev, int type, int minor);
void video_unregister_device
(struct video_device *vdev);
struct video_device *video_device_alloc(void);
void video_device_release(struct video_device *vdev);
void video_set_drvdata
(struct video_device *vdev, void *data);
void *video_get_drvdata
(struct video_device *vdev);
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 16
All Rights Reserved.

17. struct v4l2_file_operations
#include <media/v4l2-dev.h>
struct module *owner;
int (*open)(struct file *f);
int (*release)(struct file *f);
ssize_t (*read)
(struct file *f, char __user *buf, size_t cnt, loff_t *off);
ssize_t (*write)
(struct file *, const char __user *buf, size_t cnt, loff_t *off);
unsigned int (*poll)
(struct file *f, struct poll_table_struct *poll_table);
long (*unlocked_ioctl)
(struct file *f, unsigned int cmd, unsigned long arg);
int (*mmap)
(struct file *f, struct vm_area_struct *vm_area);
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 17
All Rights Reserved.

18. struct v4l2_ioctl_ops
#include <media/v4l2-ioctl.h>
Video related Handlers: QUERYCAP, FMT related
Priority Handlers
Buffer Handlers
Stream on/off, Stream type-dependent ioctls
Control Handlers
Audio related ioctls
Feature Handlers: Crop, Compression
Tuner ioctls
Sliced VBI capabilities
Log, Debugging ioctls
DV Timings ioctls
Other private ioctls
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 18
All Rights Reserved.

19. struct v4l2_ctrl_handler
#include <media/v4l2-ctrls.h>
Contains list of struct v4l2_ctrl's
Defined by various values and operations
Populated using the various APIs
APIS
int v4l2_ctrl_handler_init
(struct v4l2_ctrl_handler *hdl, int nr_ctl_hint);
struct v4l2_ctrl *v4l2_ctrl_new_xxx
(struct v4l2_ctrl_handler *hdl,
<parameters_for_values_of_fields_of_v4l2_ctrl>);
void v4l2_ctrl_handler_free
(struct v4l2_ctrl_handler *hdl);
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 19
All Rights Reserved.

20. struct v4l2_ctrl_xxx
#include <media/v4l2-ctrls.h>
struct v4l2_ctrl_ops
int (*g_volatile_ctrl)(struct v4l2_ctrl *ctrl);
int (*try_ctrl)(struct v4l2_ctrl *ctrl);
int (*s_ctrl)(struct v4l2_ctrl *ctrl);
Typical useful fields of struct v4l2_ctrl_config
u32 id;
const char *name;
enum v4l2_ctrl_type type;
const strcut v4l2_ctrl_ops *ops;
s32 minimum, maximum, default;
u32 step, flags;
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 20
All Rights Reserved.

21. struct vb2_queue (the Q)
#include <media/videobuf2-core.h>
enum v4l2_buf_type type; // <linux/videobuf2.h>
unsigned int io_modes; // Buffer Q access methods
const struct vb2_ops *ops; // Buffer Q operations
const struct vb2_mem_ops *mem_ops; // Memory ops
void *drv_priv; // Private data
unsigned int buf_struct_size; // Q element's size
...
APIs
int vb2_queue_init(struct vb2_queue *q);
void vb2_queue_release(struct vb2_queue *q);
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 21
All Rights Reserved.

22. struct vb2_ops (the Q ops)
#include <media/videobuf2-core.h>
queue_setup – Sets up the queue & give back info
wait_prepare – Release the lock(s) for waiting
wait_finish – Take the lock(s) back, after waiting
buf_init – Called after allocating a buffer
buf_prepare – Called after every queueing
buf_finish – Called before every de-queueing
buf_cleanup – Called before freeing up the buffer
buf_queue – Actually queueing the buffer
start_streaming – To start streaming of video
stop_streaming – To stop streaming of video
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 22
All Rights Reserved.

23. struct vb2_mem_ops
#include <media/videobuf2-core.h>
For operating on Video Memory
void *(*alloc)(void *alloc_ctx, unsigned long size);
void (*put)(void *buf_priv);
For operating on User Space Memory
void *(*get_userptr)(void *alloc_ctx, unsigned long vaddr, unsigned long size, int
write);
void (*put_userptr)(void *buf_priv);
Miscellaneous
unsigned int (*num_users)(void *buf_priv);
void *(*vaddr)(void *buf_priv);
int (*mmap)(void *buf_priv, struct vm_area_struct *vma);
Standard operations (struct vb2_mem_ops) available for 3 types:
vb2_dma_sg_memops <media/videobuf2-dma-sg.h>
vb2_vmalloc_memops <media/videobuf2-vmalloc.h>
vb2_dma_contig_memops <media/videobuf2-dma-contig.h>
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 23
All Rights Reserved.

24. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX)
Vertical: Video Device Drivers (Character w/ additional video ioctls)
struct video_device <media/v4l2-dev.h>
struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler
<media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h>
Video Buffer Queue Management <media/videobuf2-core.h>
Q: struct vb2_queue; Q ops: struct vb2_ops;
v4l2 Control Management
Memory ops: struct vb2_mem_ops PCI, USB, I2C
Kernel Space
Hardware Space
Devices: Video Output/Capture, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 24
All Rights Reserved.

25. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX, /dev/snd/*)
Video Device Interface V4L2 Device Interface
File Ops Ioctls Control Ops File Ops Ioctls Control Ops
Video Device Driver
... Audio Device Driver
Queue Mgmt Ctrl Mgmt Buffer Mgmt Ctrl Mgmt
Kernel Space
Hardware Space
Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 25
All Rights Reserved.

26. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX, /dev/v4l-subdevX)
struct struct
struct video_device v4l2_device V4L2 v4l2_subdev
struct video_device
File Ops Control Ops File Ops Control Ops
...
Horizontal Mgmt Horizontal Mgmt
Kernel Space
Hardware Space
Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 26
All Rights Reserved.

27. “Video for Linux” Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX, /dev/v4l-subdevX)
struct struct
struct video_device v4l2_device V4L2 v4l2_subdev
struct video_device
File Ops Control Ops Control Ops File Ops
... Subdev Ops
Horizontal Mgmt Horizontal Mgmt
Kernel Space
Hardware Space
Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 27
All Rights Reserved.

28. struct v4l2_device
#include <media/v4l2-device.h>
struct device *dev; // dev structure from the horizontal
struct list_head subdev; // List of registered sub-devices
char name[V4L2_DEVICE_NAME_SIZE]; // Unique dev name
struct v4l2_ctrl_handler *ctrl_handler; // Global ctrl handler
void (*release)(struct v4l2_device *v4l2_dev); // Called when ref == 0
void (*notify) /* Notify from sub-devices */
(struct v4l2_subdev *sd, unsigned int notification, void *arg);
struct kref ref; // Reference count
spinlock_t lock; // Access protection
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 28
All Rights Reserved.

29. V4L2 Device Programming
Header: <media/v4l2-device.h>
APIs
int v4l2_device_register
(struct device *dev, struct v4l2_device *v4l2_dev);
If (dev is passed NULL) => v4l2_dev->name must be set
int v4l2_device_unregister
(struct v4l2_device *v4l2_dev);
Mostly called in the v4l2_dev->release callback
void v4l2_device_disconnect /* No equivalent connect call possible */
(struct v4l2_device *v4l2_dev);
Sets v4l2_dev->dev to NULL – to be called in disconnect of the horiz.
void v4l2_device_get(struct v4l2_device *v4l2_dev);
int v4l2_device_put(struct v4l2_device *v4l2_dev);
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 29
All Rights Reserved.

30. struct v4l2_subdev
#include <media/v4l2-subdev.h>
struct list_head list; // Being a node of the subdev list
struct module *owner;
struct v4l2_device *v4l2_dev;
struct video_device *devnode;
char name[V4L2_SUBDEV_NAME_SIZE];
const struct v4l2_subdev_ops *ops
struct v4l2_ctrl_handler *ctrl_handler;
u32 grp_id; // For driver-specific grouping
void *dev_priv, *host_priv;
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 30
All Rights Reserved.

31. V4L2 Subdevice Programming
Header: <media/v4l2-subdev.h>
APIs
void v4l2_subdev_init /* No equivalent shut call reqd */
(struct v4l2_subdev *sd, const struct v4l2_subdev_ops *o);
void v4l2_set_subdevdata/subdev_hostdata
(struct v4l2_subdev *sd, void *p);
void *v4l2_get_subdevdata/subdev_hostdata
(const struct v4l2_subdev *sd);
v4l2_subdev_call /* For use by the v4l2_device */
(sd, struct, fn_ptr, args...);
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 31
All Rights Reserved.

32. struct v4l2_subdev_ops
#include <media/v4l2-subdev.h>
const struct v4l2_subdev_core_ops *core;
const struct v4l2_subdev_tuner_ops *tuner;
const struct v4l2_subdev_audio_ops *audio;
const struct v4l2_subdev_video_ops *video;
const struct v4l2_subdev_vbi_ops *vbi;
const struct v4l2_subdev_ir_ops *ir;
const struct v4l2_subdev_sensor_ops *sensor;
const struct v4l2_subdev_pad_ops *pad;
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 32
All Rights Reserved.

33. V4L2 Subdevice Registration
Header: <media/v4l2-device.h>
APIs
int v4l2_device_register_subdev
(struct v4l2_device *v4l2_dev, struct v4l2_subdev *sd);
void v4l2_device_unregister_subdev
(struct v4l2_subdev *sd);
v4l2_device_call_all
(v4l2_dev, grpid, struct, fn_ptr, args...);
Iterators
v4l2_device_for_each_subdev(sd, v4l2_dev)
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 33
All Rights Reserved.

34. struct video_device - revisted
#include <media/v4l2-dev.h>
struct v4l2_device *v4l2_dev;
struct device *parent;
char name[32];
struct v4l2_file_operations *fops;
struct v4l2_ioctl_ops *ioctl_ops;
struct v4l2_ctrl_handler *ctrl_handler;
void (*release)(struct video_device *vdev);
v4l2_std_id tvnorms;
v4l2_std_id current_norm;
int debug;
struct mutex *lock;
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 34
All Rights Reserved.

35. V4L2 Subsystem
User Space
Audio/Video Players/Recorders, Video Conferencing
(vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...)
Character Device Files (/dev/videoX, /dev/v4l-subdevX)
struct struct
struct video_device v4l2_device v4l2_subdev
struct video_device
File Ops Control Ops V4L2 Control Ops File Ops
... Subdev Ops
Horizontal Mgmt Horizontal Mgmt
Kernel Space
Hardware Space
Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ...
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 35
All Rights Reserved.

36. Porting a Video Driver
Standard Video Chipset
Mostly involves changing pin assignments as
per the Board Design
New Video Chipset
Complete Driver as per the preceeding
discussions, need to be implemented
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 36
All Rights Reserved.

37. What all have we learnt?
Evolution of the Linux Video subsystem
Original Video Interfaces
Frame Buffer Programming
Console Programming
Video For Linux v2 (v4l2) Programming
“Video Device” structure
v4l2 buffers & the queue management
v4l2 devices & subdevices
Putting them all together: Porting
Porting through Examples
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 37
All Rights Reserved.

38. Any Queries?
© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 38
All Rights Reserved.