Video Drivers

15,485 views

Published on

Published in: Technology
2 Comments
14 Likes
Statistics
Notes
No Downloads
Views
Total views
15,485
On SlideShare
0
From Embeds
0
Number of Embeds
10,711
Actions
Shares
0
Downloads
0
Comments
2
Likes
14
Embeds 0
No embeds

No notes for slide

Video Drivers

  1. 1. Video Drivers© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> All Rights Reserved.
  2. 2. What to Expect?Introduction to the Linux video subsystemOriginal Video Interfaces Frame Buffer ConsoleVideo For Linux v2 (v4l2) Framework “Video Device” structure v4l2 buffers & the queue management v4l2 devices & subdevicesPutting them all together: Porting © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 2 All Rights Reserved.
  3. 3. Video HistoryPlain Character Drivers Lot of hardware dependency exposed (ioctls) Getting video working was non-trivial (monitors)Needed additional video APIs, very oftenCalled for better abstractionVideo Layers added over character drivers Frame Buffer ConsoleBut all these were for only videoMore complex devices & requirements evolvedNeed of Integration between others like audioComplete framework designed: Video For Linux (v4l)Further evolved & matured: Video For Linux (v4l2) © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 3 All Rights Reserved.
  4. 4. A 10000 feet viewUser Space Audio/Video Players/Recorders, Video Conferencing X Windows (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files Vertical: Various Character Drivers w/ additional Video APIs Video Output Video Capture Image, ... Audio Drivers Drivers Drivers Drivers Horizontal: PCI, AGP, PCI Express, USB, I2C ................. PCI, I2S, USB, I2CKernel SpaceHardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Printers/Scanners/... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 4 All Rights Reserved.
  5. 5. Original Video Subsystem FB-aware applications Such as video players Console setterm UM X Windows fbsetDriver User Space /dev/fbX /dev/ttyX Virtual Terminal lpcons usb_uart Driver fbcon .. vgacon Common Frame Buffer API over /dev/fbX Frame Buffer Driver Kernel Space Hardware Space Video Card X VGA Card Printer USB UART © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 5 All Rights Reserved.
  6. 6. FB Programming InterfaceHeader: <linux/fb.h>Data Structures struct fb_info – Main data structure struct fb_ops – Entry points struct fb_var_screen_info – Resolution, ... struct fb_fix_screen_info – FB start addr, ... struct fb_cmap – RGB colour mapAPIs int register_framebuffer(struct fb_info *fb_info); int unregister_framebuffer(struct fb_info *fb_info); struct fb_info *framebuffer_alloc(size_t size, struct device *dev); void framebuffer_release(struct fb_info *info); int fb_alloc_cmap(struct fb_cmap *cmap, int len, int transp); void fb_dealloc_cmap(struct fb_cmap *cmap);Source: drivers/video/ © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 6 All Rights Reserved.
  7. 7. struct fb_opsfb_open – Openfb_release – Closefb_check_var – Check video parametersfb_set_par – Set video controller registersfb_setcolreg – Create pseudo colour palette mapfb_blank – Blank / Unblank displayfb_fillrect – Fill rectangle with pixel linesfb_copyarea – Copy rectangular area between screensfb_imageblit – Draw an image to the displayfb_rotate – Rotate the displayfb_ioctl – Ioctl interface for device specific commands © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 7 All Rights Reserved.
  8. 8. Console Programming InterfaceHeader: <linux/console.h>Data Structures struct console – top-level console driver struct consw – bottom-level console driverAPIs void register_console(struct console *); int unregister_console(struct console *); int register_con_driver(const struct consw *csw, int first, int last); int unregister_con_driver(const struct consw *csw);Source: drivers/video/console © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 8 All Rights Reserved.
  9. 9. Reference Video DriversFor Frame Buffer drivers Browse drivers/video/ Example: vga16fb.cFor Console drivers Browse drivers/video/console/ Example: fbcon.c Font Example: font_8x8.c © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 9 All Rights Reserved.
  10. 10. “Video in Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/XXX) Vertical: Video Device Drivers (Character w/ additional video ioctls) File Operations: open, release, read, poll, mmap, unlocked_ioctl Video Buffer Management Video Control Management Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2CKernel SpaceHardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 10 All Rights Reserved.
  11. 11. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations: open, release, read, poll, mmap, unlocked_ioctl Video Buffer Management v4l2 Control Management Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2CKernel SpaceHardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 11 All Rights Reserved.
  12. 12. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> open, release, read, poll, mmap, unlocked_ioctl = video_ioctl2 struct v4l2_ioctl_ops Video Buffer Management v4l2 Control Management 2 PCI, AGP, PCI Express, USB, I S PCI, USB, I2CKernel SpaceHardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 12 All Rights Reserved.
  13. 13. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops <media/v4l2-ioctl.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; Memory ops: struct vb2_mem_ops v4l2 Control Management PCI, USB, I2C Scatter Gather Virtually Contiguous All Contiguous videobuf2-dma-sg.h videobuf2-vmalloc.h videobuf2-dma-contig.hKernel SpaceHardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 13 All Rights Reserved.
  14. 14. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler <media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; v4l2 Control Management Memory ops: struct vb2_mem_ops PCI, USB, I2CKernel SpaceHardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 14 All Rights Reserved.
  15. 15. struct video_device #include <media/v4l2-dev.h>struct device *parent; // Dev structure from the horizontalchar name[32];struct v4l2_file_operations *fops;struct v4l2_ioctl_ops *ioctl_ops;struct v4l2_ctrl_handler *ctrl_handler;void (*release)(struct video_device *vdev);v4l2_std_id tvnorms;v4l2_std_id current_norm;int debug;struct mutex *lock; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 15 All Rights Reserved.
  16. 16. Video Device ProgrammingHeader: <media/v4l2-dev.h>APIs int video_register_device (struct video_device *vdev, int type, int minor); void video_unregister_device (struct video_device *vdev); struct video_device *video_device_alloc(void); void video_device_release(struct video_device *vdev); void video_set_drvdata (struct video_device *vdev, void *data); void *video_get_drvdata (struct video_device *vdev); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 16 All Rights Reserved.
  17. 17. struct v4l2_file_operations #include <media/v4l2-dev.h>struct module *owner;int (*open)(struct file *f);int (*release)(struct file *f);ssize_t (*read) (struct file *f, char __user *buf, size_t cnt, loff_t *off);ssize_t (*write) (struct file *, const char __user *buf, size_t cnt, loff_t *off);unsigned int (*poll) (struct file *f, struct poll_table_struct *poll_table);long (*unlocked_ioctl) (struct file *f, unsigned int cmd, unsigned long arg);int (*mmap) (struct file *f, struct vm_area_struct *vm_area); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 17 All Rights Reserved.
  18. 18. struct v4l2_ioctl_ops #include <media/v4l2-ioctl.h>Video related Handlers: QUERYCAP, FMT relatedPriority HandlersBuffer HandlersStream on/off, Stream type-dependent ioctlsControl HandlersAudio related ioctlsFeature Handlers: Crop, CompressionTuner ioctlsSliced VBI capabilitiesLog, Debugging ioctlsDV Timings ioctlsOther private ioctls © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 18 All Rights Reserved.
  19. 19. struct v4l2_ctrl_handler #include <media/v4l2-ctrls.h>Contains list of struct v4l2_ctrls Defined by various values and operations Populated using the various APIsAPIS int v4l2_ctrl_handler_init (struct v4l2_ctrl_handler *hdl, int nr_ctl_hint); struct v4l2_ctrl *v4l2_ctrl_new_xxx (struct v4l2_ctrl_handler *hdl, <parameters_for_values_of_fields_of_v4l2_ctrl>); void v4l2_ctrl_handler_free (struct v4l2_ctrl_handler *hdl); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 19 All Rights Reserved.
  20. 20. struct v4l2_ctrl_xxx #include <media/v4l2-ctrls.h>struct v4l2_ctrl_ops int (*g_volatile_ctrl)(struct v4l2_ctrl *ctrl); int (*try_ctrl)(struct v4l2_ctrl *ctrl); int (*s_ctrl)(struct v4l2_ctrl *ctrl);Typical useful fields of struct v4l2_ctrl_config u32 id; const char *name; enum v4l2_ctrl_type type; const strcut v4l2_ctrl_ops *ops; s32 minimum, maximum, default; u32 step, flags; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 20 All Rights Reserved.
  21. 21. struct vb2_queue (the Q) #include <media/videobuf2-core.h>enum v4l2_buf_type type; // <linux/videobuf2.h>unsigned int io_modes; // Buffer Q access methodsconst struct vb2_ops *ops; // Buffer Q operationsconst struct vb2_mem_ops *mem_ops; // Memory opsvoid *drv_priv; // Private dataunsigned int buf_struct_size; // Q elements size...APIs int vb2_queue_init(struct vb2_queue *q); void vb2_queue_release(struct vb2_queue *q); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 21 All Rights Reserved.
  22. 22. struct vb2_ops (the Q ops) #include <media/videobuf2-core.h>queue_setup – Sets up the queue & give back infowait_prepare – Release the lock(s) for waitingwait_finish – Take the lock(s) back, after waitingbuf_init – Called after allocating a bufferbuf_prepare – Called after every queueingbuf_finish – Called before every de-queueingbuf_cleanup – Called before freeing up the bufferbuf_queue – Actually queueing the bufferstart_streaming – To start streaming of videostop_streaming – To stop streaming of video © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 22 All Rights Reserved.
  23. 23. struct vb2_mem_ops #include <media/videobuf2-core.h>For operating on Video Memory void *(*alloc)(void *alloc_ctx, unsigned long size); void (*put)(void *buf_priv);For operating on User Space Memory void *(*get_userptr)(void *alloc_ctx, unsigned long vaddr, unsigned long size, int write); void (*put_userptr)(void *buf_priv);Miscellaneous unsigned int (*num_users)(void *buf_priv); void *(*vaddr)(void *buf_priv); int (*mmap)(void *buf_priv, struct vm_area_struct *vma);Standard operations (struct vb2_mem_ops) available for 3 types: vb2_dma_sg_memops <media/videobuf2-dma-sg.h> vb2_vmalloc_memops <media/videobuf2-vmalloc.h> vb2_dma_contig_memops <media/videobuf2-dma-contig.h> © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 23 All Rights Reserved.
  24. 24. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler <media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; v4l2 Control Management Memory ops: struct vb2_mem_ops PCI, USB, I2CKernel SpaceHardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 24 All Rights Reserved.
  25. 25. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/snd/*) Video Device Interface V4L2 Device Interface File Ops Ioctls Control Ops File Ops Ioctls Control Ops Video Device Driver ... Audio Device Driver Queue Mgmt Ctrl Mgmt Buffer Mgmt Ctrl MgmtKernel SpaceHardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 25 All Rights Reserved.
  26. 26. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device V4L2 v4l2_subdev struct video_device File Ops Control Ops File Ops Control Ops ... Horizontal Mgmt Horizontal MgmtKernel SpaceHardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 26 All Rights Reserved.
  27. 27. “Video for Linux” SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device V4L2 v4l2_subdev struct video_device File Ops Control Ops Control Ops File Ops ... Subdev Ops Horizontal Mgmt Horizontal MgmtKernel SpaceHardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 27 All Rights Reserved.
  28. 28. struct v4l2_device #include <media/v4l2-device.h>struct device *dev; // dev structure from the horizontalstruct list_head subdev; // List of registered sub-deviceschar name[V4L2_DEVICE_NAME_SIZE]; // Unique dev namestruct v4l2_ctrl_handler *ctrl_handler; // Global ctrl handlervoid (*release)(struct v4l2_device *v4l2_dev); // Called when ref == 0void (*notify) /* Notify from sub-devices */ (struct v4l2_subdev *sd, unsigned int notification, void *arg);struct kref ref; // Reference countspinlock_t lock; // Access protection © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 28 All Rights Reserved.
  29. 29. V4L2 Device ProgrammingHeader: <media/v4l2-device.h>APIs int v4l2_device_register (struct device *dev, struct v4l2_device *v4l2_dev); If (dev is passed NULL) => v4l2_dev->name must be set int v4l2_device_unregister (struct v4l2_device *v4l2_dev); Mostly called in the v4l2_dev->release callback void v4l2_device_disconnect /* No equivalent connect call possible */ (struct v4l2_device *v4l2_dev); Sets v4l2_dev->dev to NULL – to be called in disconnect of the horiz. void v4l2_device_get(struct v4l2_device *v4l2_dev); int v4l2_device_put(struct v4l2_device *v4l2_dev); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 29 All Rights Reserved.
  30. 30. struct v4l2_subdev #include <media/v4l2-subdev.h>struct list_head list; // Being a node of the subdev liststruct module *owner;struct v4l2_device *v4l2_dev;struct video_device *devnode;char name[V4L2_SUBDEV_NAME_SIZE];const struct v4l2_subdev_ops *opsstruct v4l2_ctrl_handler *ctrl_handler;u32 grp_id; // For driver-specific groupingvoid *dev_priv, *host_priv; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 30 All Rights Reserved.
  31. 31. V4L2 Subdevice ProgrammingHeader: <media/v4l2-subdev.h>APIs void v4l2_subdev_init /* No equivalent shut call reqd */ (struct v4l2_subdev *sd, const struct v4l2_subdev_ops *o); void v4l2_set_subdevdata/subdev_hostdata (struct v4l2_subdev *sd, void *p); void *v4l2_get_subdevdata/subdev_hostdata (const struct v4l2_subdev *sd); v4l2_subdev_call /* For use by the v4l2_device */ (sd, struct, fn_ptr, args...); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 31 All Rights Reserved.
  32. 32. struct v4l2_subdev_ops #include <media/v4l2-subdev.h>const struct v4l2_subdev_core_ops *core;const struct v4l2_subdev_tuner_ops *tuner;const struct v4l2_subdev_audio_ops *audio;const struct v4l2_subdev_video_ops *video;const struct v4l2_subdev_vbi_ops *vbi;const struct v4l2_subdev_ir_ops *ir;const struct v4l2_subdev_sensor_ops *sensor;const struct v4l2_subdev_pad_ops *pad; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 32 All Rights Reserved.
  33. 33. V4L2 Subdevice RegistrationHeader: <media/v4l2-device.h>APIs int v4l2_device_register_subdev (struct v4l2_device *v4l2_dev, struct v4l2_subdev *sd); void v4l2_device_unregister_subdev (struct v4l2_subdev *sd); v4l2_device_call_all (v4l2_dev, grpid, struct, fn_ptr, args...); Iterators v4l2_device_for_each_subdev(sd, v4l2_dev) © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 33 All Rights Reserved.
  34. 34. struct video_device - revisted #include <media/v4l2-dev.h>struct v4l2_device *v4l2_dev;struct device *parent;char name[32];struct v4l2_file_operations *fops;struct v4l2_ioctl_ops *ioctl_ops;struct v4l2_ctrl_handler *ctrl_handler;void (*release)(struct video_device *vdev);v4l2_std_id tvnorms;v4l2_std_id current_norm;int debug;struct mutex *lock; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 34 All Rights Reserved.
  35. 35. V4L2 SubsystemUser Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device v4l2_subdev struct video_device File Ops Control Ops V4L2 Control Ops File Ops ... Subdev Ops Horizontal Mgmt Horizontal MgmtKernel SpaceHardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 35 All Rights Reserved.
  36. 36. Porting a Video DriverStandard Video Chipset Mostly involves changing pin assignments as per the Board DesignNew Video Chipset Complete Driver as per the preceeding discussions, need to be implemented © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 36 All Rights Reserved.
  37. 37. What all have we learnt?Evolution of the Linux Video subsystemOriginal Video Interfaces Frame Buffer Programming Console ProgrammingVideo For Linux v2 (v4l2) Programming “Video Device” structure v4l2 buffers & the queue management v4l2 devices & subdevicesPutting them all together: Porting Porting through Examples © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 37 All Rights Reserved.
  38. 38. Any Queries?© 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 38 All Rights Reserved.

×