Successfully reported this slideshow.

Video Drivers

17

Share

Upcoming SlideShare
I2C Drivers
I2C Drivers
Loading in …3
×
1 of 38
1 of 38

Video Drivers

17

Share

  1. 1. Video Drivers © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> All Rights Reserved.
  2. 2. What to Expect? Introduction to the Linux' video subsystem Original Video Interfaces Frame Buffer Console Video For Linux v2 (v4l2) Framework “Video Device” structure v4l2 buffers & the queue management v4l2 devices & subdevices Putting them all together: Porting © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 2 All Rights Reserved.
  3. 3. Video History Plain Character Drivers Lot of hardware dependency exposed (ioctls) Getting video working was non-trivial (monitors) Needed additional video APIs, very often Called for better abstraction Video Layers added over character drivers Frame Buffer Console But all these were for only video More complex devices & requirements evolved Need of Integration between others like audio Complete framework designed: Video For Linux (v4l) Further evolved & matured: Video For Linux (v4l2) © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 3 All Rights Reserved.
  4. 4. A 10000 feet view User Space Audio/Video Players/Recorders, Video Conferencing X Windows (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files Vertical: Various Character Drivers w/ additional Video APIs Video Output Video Capture Image, ... Audio Drivers Drivers Drivers Drivers Horizontal: PCI, AGP, PCI Express, USB, I2C ................. PCI, I2S, USB, I2C Kernel Space Hardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Printers/Scanners/... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 4 All Rights Reserved.
  5. 5. Original Video Subsystem FB-aware applications Such as video players Console setterm UM X Windows fbset Driver User Space /dev/fbX /dev/ttyX Virtual Terminal lpcons usb_uart Driver fbcon .. vgacon Common Frame Buffer API over /dev/fbX Frame Buffer Driver Kernel Space Hardware Space Video Card X VGA Card Printer USB UART © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 5 All Rights Reserved.
  6. 6. FB Programming Interface Header: <linux/fb.h> Data Structures struct fb_info – Main data structure struct fb_ops – Entry points struct fb_var_screen_info – Resolution, ... struct fb_fix_screen_info – FB start addr, ... struct fb_cmap – RGB colour map APIs int register_framebuffer(struct fb_info *fb_info); int unregister_framebuffer(struct fb_info *fb_info); struct fb_info *framebuffer_alloc(size_t size, struct device *dev); void framebuffer_release(struct fb_info *info); int fb_alloc_cmap(struct fb_cmap *cmap, int len, int transp); void fb_dealloc_cmap(struct fb_cmap *cmap); Source: drivers/video/ © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 6 All Rights Reserved.
  7. 7. struct fb_ops fb_open – Open fb_release – Close fb_check_var – Check video parameters fb_set_par – Set video controller registers fb_setcolreg – Create pseudo colour palette map fb_blank – Blank / Unblank display fb_fillrect – Fill rectangle with pixel lines fb_copyarea – Copy rectangular area between screens fb_imageblit – Draw an image to the display fb_rotate – Rotate the display fb_ioctl – Ioctl interface for device specific commands © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 7 All Rights Reserved.
  8. 8. Console Programming Interface Header: <linux/console.h> Data Structures struct console – top-level console driver struct consw – bottom-level console driver APIs void register_console(struct console *); int unregister_console(struct console *); int register_con_driver(const struct consw *csw, int first, int last); int unregister_con_driver(const struct consw *csw); Source: drivers/video/console © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 8 All Rights Reserved.
  9. 9. Reference Video Drivers For Frame Buffer drivers Browse drivers/video/ Example: vga16fb.c For Console drivers Browse drivers/video/console/ Example: fbcon.c Font Example: font_8x8.c © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 9 All Rights Reserved.
  10. 10. “Video in Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/XXX) Vertical: Video Device Drivers (Character w/ additional video ioctls) File Operations: open, release, read, poll, mmap, unlocked_ioctl Video Buffer Management Video Control Management Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2C Kernel Space Hardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 10 All Rights Reserved.
  11. 11. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations: open, release, read, poll, mmap, unlocked_ioctl Video Buffer Management v4l2 Control Management Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2C Kernel Space Hardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 11 All Rights Reserved.
  12. 12. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> open, release, read, poll, mmap, unlocked_ioctl = video_ioctl2 struct v4l2_ioctl_ops Video Buffer Management v4l2 Control Management 2 PCI, AGP, PCI Express, USB, I S PCI, USB, I2C Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 12 All Rights Reserved.
  13. 13. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops <media/v4l2-ioctl.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; Memory ops: struct vb2_mem_ops v4l2 Control Management PCI, USB, I2C Scatter Gather Virtually Contiguous All Contiguous videobuf2-dma-sg.h videobuf2-vmalloc.h videobuf2-dma-contig.h Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 13 All Rights Reserved.
  14. 14. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler <media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; v4l2 Control Management Memory ops: struct vb2_mem_ops PCI, USB, I2C Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 14 All Rights Reserved.
  15. 15. struct video_device #include <media/v4l2-dev.h> struct device *parent; // Dev structure from the horizontal char name[32]; struct v4l2_file_operations *fops; struct v4l2_ioctl_ops *ioctl_ops; struct v4l2_ctrl_handler *ctrl_handler; void (*release)(struct video_device *vdev); v4l2_std_id tvnorms; v4l2_std_id current_norm; int debug; struct mutex *lock; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 15 All Rights Reserved.
  16. 16. Video Device Programming Header: <media/v4l2-dev.h> APIs int video_register_device (struct video_device *vdev, int type, int minor); void video_unregister_device (struct video_device *vdev); struct video_device *video_device_alloc(void); void video_device_release(struct video_device *vdev); void video_set_drvdata (struct video_device *vdev, void *data); void *video_get_drvdata (struct video_device *vdev); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 16 All Rights Reserved.
  17. 17. struct v4l2_file_operations #include <media/v4l2-dev.h> struct module *owner; int (*open)(struct file *f); int (*release)(struct file *f); ssize_t (*read) (struct file *f, char __user *buf, size_t cnt, loff_t *off); ssize_t (*write) (struct file *, const char __user *buf, size_t cnt, loff_t *off); unsigned int (*poll) (struct file *f, struct poll_table_struct *poll_table); long (*unlocked_ioctl) (struct file *f, unsigned int cmd, unsigned long arg); int (*mmap) (struct file *f, struct vm_area_struct *vm_area); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 17 All Rights Reserved.
  18. 18. struct v4l2_ioctl_ops #include <media/v4l2-ioctl.h> Video related Handlers: QUERYCAP, FMT related Priority Handlers Buffer Handlers Stream on/off, Stream type-dependent ioctls Control Handlers Audio related ioctls Feature Handlers: Crop, Compression Tuner ioctls Sliced VBI capabilities Log, Debugging ioctls DV Timings ioctls Other private ioctls © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 18 All Rights Reserved.
  19. 19. struct v4l2_ctrl_handler #include <media/v4l2-ctrls.h> Contains list of struct v4l2_ctrl's Defined by various values and operations Populated using the various APIs APIS int v4l2_ctrl_handler_init (struct v4l2_ctrl_handler *hdl, int nr_ctl_hint); struct v4l2_ctrl *v4l2_ctrl_new_xxx (struct v4l2_ctrl_handler *hdl, <parameters_for_values_of_fields_of_v4l2_ctrl>); void v4l2_ctrl_handler_free (struct v4l2_ctrl_handler *hdl); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 19 All Rights Reserved.
  20. 20. struct v4l2_ctrl_xxx #include <media/v4l2-ctrls.h> struct v4l2_ctrl_ops int (*g_volatile_ctrl)(struct v4l2_ctrl *ctrl); int (*try_ctrl)(struct v4l2_ctrl *ctrl); int (*s_ctrl)(struct v4l2_ctrl *ctrl); Typical useful fields of struct v4l2_ctrl_config u32 id; const char *name; enum v4l2_ctrl_type type; const strcut v4l2_ctrl_ops *ops; s32 minimum, maximum, default; u32 step, flags; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 20 All Rights Reserved.
  21. 21. struct vb2_queue (the Q) #include <media/videobuf2-core.h> enum v4l2_buf_type type; // <linux/videobuf2.h> unsigned int io_modes; // Buffer Q access methods const struct vb2_ops *ops; // Buffer Q operations const struct vb2_mem_ops *mem_ops; // Memory ops void *drv_priv; // Private data unsigned int buf_struct_size; // Q element's size ... APIs int vb2_queue_init(struct vb2_queue *q); void vb2_queue_release(struct vb2_queue *q); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 21 All Rights Reserved.
  22. 22. struct vb2_ops (the Q ops) #include <media/videobuf2-core.h> queue_setup – Sets up the queue & give back info wait_prepare – Release the lock(s) for waiting wait_finish – Take the lock(s) back, after waiting buf_init – Called after allocating a buffer buf_prepare – Called after every queueing buf_finish – Called before every de-queueing buf_cleanup – Called before freeing up the buffer buf_queue – Actually queueing the buffer start_streaming – To start streaming of video stop_streaming – To stop streaming of video © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 22 All Rights Reserved.
  23. 23. struct vb2_mem_ops #include <media/videobuf2-core.h> For operating on Video Memory void *(*alloc)(void *alloc_ctx, unsigned long size); void (*put)(void *buf_priv); For operating on User Space Memory void *(*get_userptr)(void *alloc_ctx, unsigned long vaddr, unsigned long size, int write); void (*put_userptr)(void *buf_priv); Miscellaneous unsigned int (*num_users)(void *buf_priv); void *(*vaddr)(void *buf_priv); int (*mmap)(void *buf_priv, struct vm_area_struct *vma); Standard operations (struct vb2_mem_ops) available for 3 types: vb2_dma_sg_memops <media/videobuf2-dma-sg.h> vb2_vmalloc_memops <media/videobuf2-vmalloc.h> vb2_dma_contig_memops <media/videobuf2-dma-contig.h> © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 23 All Rights Reserved.
  24. 24. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler <media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; v4l2 Control Management Memory ops: struct vb2_mem_ops PCI, USB, I2C Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 24 All Rights Reserved.
  25. 25. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/snd/*) Video Device Interface V4L2 Device Interface File Ops Ioctls Control Ops File Ops Ioctls Control Ops Video Device Driver ... Audio Device Driver Queue Mgmt Ctrl Mgmt Buffer Mgmt Ctrl Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 25 All Rights Reserved.
  26. 26. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device V4L2 v4l2_subdev struct video_device File Ops Control Ops File Ops Control Ops ... Horizontal Mgmt Horizontal Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 26 All Rights Reserved.
  27. 27. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device V4L2 v4l2_subdev struct video_device File Ops Control Ops Control Ops File Ops ... Subdev Ops Horizontal Mgmt Horizontal Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 27 All Rights Reserved.
  28. 28. struct v4l2_device #include <media/v4l2-device.h> struct device *dev; // dev structure from the horizontal struct list_head subdev; // List of registered sub-devices char name[V4L2_DEVICE_NAME_SIZE]; // Unique dev name struct v4l2_ctrl_handler *ctrl_handler; // Global ctrl handler void (*release)(struct v4l2_device *v4l2_dev); // Called when ref == 0 void (*notify) /* Notify from sub-devices */ (struct v4l2_subdev *sd, unsigned int notification, void *arg); struct kref ref; // Reference count spinlock_t lock; // Access protection © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 28 All Rights Reserved.
  29. 29. V4L2 Device Programming Header: <media/v4l2-device.h> APIs int v4l2_device_register (struct device *dev, struct v4l2_device *v4l2_dev); If (dev is passed NULL) => v4l2_dev->name must be set int v4l2_device_unregister (struct v4l2_device *v4l2_dev); Mostly called in the v4l2_dev->release callback void v4l2_device_disconnect /* No equivalent connect call possible */ (struct v4l2_device *v4l2_dev); Sets v4l2_dev->dev to NULL – to be called in disconnect of the horiz. void v4l2_device_get(struct v4l2_device *v4l2_dev); int v4l2_device_put(struct v4l2_device *v4l2_dev); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 29 All Rights Reserved.
  30. 30. struct v4l2_subdev #include <media/v4l2-subdev.h> struct list_head list; // Being a node of the subdev list struct module *owner; struct v4l2_device *v4l2_dev; struct video_device *devnode; char name[V4L2_SUBDEV_NAME_SIZE]; const struct v4l2_subdev_ops *ops struct v4l2_ctrl_handler *ctrl_handler; u32 grp_id; // For driver-specific grouping void *dev_priv, *host_priv; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 30 All Rights Reserved.
  31. 31. V4L2 Subdevice Programming Header: <media/v4l2-subdev.h> APIs void v4l2_subdev_init /* No equivalent shut call reqd */ (struct v4l2_subdev *sd, const struct v4l2_subdev_ops *o); void v4l2_set_subdevdata/subdev_hostdata (struct v4l2_subdev *sd, void *p); void *v4l2_get_subdevdata/subdev_hostdata (const struct v4l2_subdev *sd); v4l2_subdev_call /* For use by the v4l2_device */ (sd, struct, fn_ptr, args...); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 31 All Rights Reserved.
  32. 32. struct v4l2_subdev_ops #include <media/v4l2-subdev.h> const struct v4l2_subdev_core_ops *core; const struct v4l2_subdev_tuner_ops *tuner; const struct v4l2_subdev_audio_ops *audio; const struct v4l2_subdev_video_ops *video; const struct v4l2_subdev_vbi_ops *vbi; const struct v4l2_subdev_ir_ops *ir; const struct v4l2_subdev_sensor_ops *sensor; const struct v4l2_subdev_pad_ops *pad; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 32 All Rights Reserved.
  33. 33. V4L2 Subdevice Registration Header: <media/v4l2-device.h> APIs int v4l2_device_register_subdev (struct v4l2_device *v4l2_dev, struct v4l2_subdev *sd); void v4l2_device_unregister_subdev (struct v4l2_subdev *sd); v4l2_device_call_all (v4l2_dev, grpid, struct, fn_ptr, args...); Iterators v4l2_device_for_each_subdev(sd, v4l2_dev) © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 33 All Rights Reserved.
  34. 34. struct video_device - revisted #include <media/v4l2-dev.h> struct v4l2_device *v4l2_dev; struct device *parent; char name[32]; struct v4l2_file_operations *fops; struct v4l2_ioctl_ops *ioctl_ops; struct v4l2_ctrl_handler *ctrl_handler; void (*release)(struct video_device *vdev); v4l2_std_id tvnorms; v4l2_std_id current_norm; int debug; struct mutex *lock; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 34 All Rights Reserved.
  35. 35. V4L2 Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device v4l2_subdev struct video_device File Ops Control Ops V4L2 Control Ops File Ops ... Subdev Ops Horizontal Mgmt Horizontal Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 35 All Rights Reserved.
  36. 36. Porting a Video Driver Standard Video Chipset Mostly involves changing pin assignments as per the Board Design New Video Chipset Complete Driver as per the preceeding discussions, need to be implemented © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 36 All Rights Reserved.
  37. 37. What all have we learnt? Evolution of the Linux Video subsystem Original Video Interfaces Frame Buffer Programming Console Programming Video For Linux v2 (v4l2) Programming “Video Device” structure v4l2 buffers & the queue management v4l2 devices & subdevices Putting them all together: Porting Porting through Examples © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 37 All Rights Reserved.
  38. 38. Any Queries? © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 38 All Rights Reserved.
  1. 1. Video Drivers © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> All Rights Reserved.
  2. 2. What to Expect? Introduction to the Linux' video subsystem Original Video Interfaces Frame Buffer Console Video For Linux v2 (v4l2) Framework “Video Device” structure v4l2 buffers & the queue management v4l2 devices & subdevices Putting them all together: Porting © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 2 All Rights Reserved.
  3. 3. Video History Plain Character Drivers Lot of hardware dependency exposed (ioctls) Getting video working was non-trivial (monitors) Needed additional video APIs, very often Called for better abstraction Video Layers added over character drivers Frame Buffer Console But all these were for only video More complex devices & requirements evolved Need of Integration between others like audio Complete framework designed: Video For Linux (v4l) Further evolved & matured: Video For Linux (v4l2) © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 3 All Rights Reserved.
  4. 4. A 10000 feet view User Space Audio/Video Players/Recorders, Video Conferencing X Windows (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files Vertical: Various Character Drivers w/ additional Video APIs Video Output Video Capture Image, ... Audio Drivers Drivers Drivers Drivers Horizontal: PCI, AGP, PCI Express, USB, I2C ................. PCI, I2S, USB, I2C Kernel Space Hardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Printers/Scanners/... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 4 All Rights Reserved.
  5. 5. Original Video Subsystem FB-aware applications Such as video players Console setterm UM X Windows fbset Driver User Space /dev/fbX /dev/ttyX Virtual Terminal lpcons usb_uart Driver fbcon .. vgacon Common Frame Buffer API over /dev/fbX Frame Buffer Driver Kernel Space Hardware Space Video Card X VGA Card Printer USB UART © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 5 All Rights Reserved.
  6. 6. FB Programming Interface Header: <linux/fb.h> Data Structures struct fb_info – Main data structure struct fb_ops – Entry points struct fb_var_screen_info – Resolution, ... struct fb_fix_screen_info – FB start addr, ... struct fb_cmap – RGB colour map APIs int register_framebuffer(struct fb_info *fb_info); int unregister_framebuffer(struct fb_info *fb_info); struct fb_info *framebuffer_alloc(size_t size, struct device *dev); void framebuffer_release(struct fb_info *info); int fb_alloc_cmap(struct fb_cmap *cmap, int len, int transp); void fb_dealloc_cmap(struct fb_cmap *cmap); Source: drivers/video/ © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 6 All Rights Reserved.
  7. 7. struct fb_ops fb_open – Open fb_release – Close fb_check_var – Check video parameters fb_set_par – Set video controller registers fb_setcolreg – Create pseudo colour palette map fb_blank – Blank / Unblank display fb_fillrect – Fill rectangle with pixel lines fb_copyarea – Copy rectangular area between screens fb_imageblit – Draw an image to the display fb_rotate – Rotate the display fb_ioctl – Ioctl interface for device specific commands © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 7 All Rights Reserved.
  8. 8. Console Programming Interface Header: <linux/console.h> Data Structures struct console – top-level console driver struct consw – bottom-level console driver APIs void register_console(struct console *); int unregister_console(struct console *); int register_con_driver(const struct consw *csw, int first, int last); int unregister_con_driver(const struct consw *csw); Source: drivers/video/console © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 8 All Rights Reserved.
  9. 9. Reference Video Drivers For Frame Buffer drivers Browse drivers/video/ Example: vga16fb.c For Console drivers Browse drivers/video/console/ Example: fbcon.c Font Example: font_8x8.c © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 9 All Rights Reserved.
  10. 10. “Video in Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/XXX) Vertical: Video Device Drivers (Character w/ additional video ioctls) File Operations: open, release, read, poll, mmap, unlocked_ioctl Video Buffer Management Video Control Management Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2C Kernel Space Hardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 10 All Rights Reserved.
  11. 11. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations: open, release, read, poll, mmap, unlocked_ioctl Video Buffer Management v4l2 Control Management Horizontal: PCI, AGP, PCI Express, USB, I2S ........................ PCI, USB, I2C Kernel Space Hardware Space Video Output Devices Video Capture Devices Audio Out/In Devices Encoders/Decoders © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 11 All Rights Reserved.
  12. 12. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> open, release, read, poll, mmap, unlocked_ioctl = video_ioctl2 struct v4l2_ioctl_ops Video Buffer Management v4l2 Control Management 2 PCI, AGP, PCI Express, USB, I S PCI, USB, I2C Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 12 All Rights Reserved.
  13. 13. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops <media/v4l2-ioctl.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; Memory ops: struct vb2_mem_ops v4l2 Control Management PCI, USB, I2C Scatter Gather Virtually Contiguous All Contiguous videobuf2-dma-sg.h videobuf2-vmalloc.h videobuf2-dma-contig.h Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 13 All Rights Reserved.
  14. 14. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler <media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; v4l2 Control Management Memory ops: struct vb2_mem_ops PCI, USB, I2C Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 14 All Rights Reserved.
  15. 15. struct video_device #include <media/v4l2-dev.h> struct device *parent; // Dev structure from the horizontal char name[32]; struct v4l2_file_operations *fops; struct v4l2_ioctl_ops *ioctl_ops; struct v4l2_ctrl_handler *ctrl_handler; void (*release)(struct video_device *vdev); v4l2_std_id tvnorms; v4l2_std_id current_norm; int debug; struct mutex *lock; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 15 All Rights Reserved.
  16. 16. Video Device Programming Header: <media/v4l2-dev.h> APIs int video_register_device (struct video_device *vdev, int type, int minor); void video_unregister_device (struct video_device *vdev); struct video_device *video_device_alloc(void); void video_device_release(struct video_device *vdev); void video_set_drvdata (struct video_device *vdev, void *data); void *video_get_drvdata (struct video_device *vdev); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 16 All Rights Reserved.
  17. 17. struct v4l2_file_operations #include <media/v4l2-dev.h> struct module *owner; int (*open)(struct file *f); int (*release)(struct file *f); ssize_t (*read) (struct file *f, char __user *buf, size_t cnt, loff_t *off); ssize_t (*write) (struct file *, const char __user *buf, size_t cnt, loff_t *off); unsigned int (*poll) (struct file *f, struct poll_table_struct *poll_table); long (*unlocked_ioctl) (struct file *f, unsigned int cmd, unsigned long arg); int (*mmap) (struct file *f, struct vm_area_struct *vm_area); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 17 All Rights Reserved.
  18. 18. struct v4l2_ioctl_ops #include <media/v4l2-ioctl.h> Video related Handlers: QUERYCAP, FMT related Priority Handlers Buffer Handlers Stream on/off, Stream type-dependent ioctls Control Handlers Audio related ioctls Feature Handlers: Crop, Compression Tuner ioctls Sliced VBI capabilities Log, Debugging ioctls DV Timings ioctls Other private ioctls © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 18 All Rights Reserved.
  19. 19. struct v4l2_ctrl_handler #include <media/v4l2-ctrls.h> Contains list of struct v4l2_ctrl's Defined by various values and operations Populated using the various APIs APIS int v4l2_ctrl_handler_init (struct v4l2_ctrl_handler *hdl, int nr_ctl_hint); struct v4l2_ctrl *v4l2_ctrl_new_xxx (struct v4l2_ctrl_handler *hdl, <parameters_for_values_of_fields_of_v4l2_ctrl>); void v4l2_ctrl_handler_free (struct v4l2_ctrl_handler *hdl); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 19 All Rights Reserved.
  20. 20. struct v4l2_ctrl_xxx #include <media/v4l2-ctrls.h> struct v4l2_ctrl_ops int (*g_volatile_ctrl)(struct v4l2_ctrl *ctrl); int (*try_ctrl)(struct v4l2_ctrl *ctrl); int (*s_ctrl)(struct v4l2_ctrl *ctrl); Typical useful fields of struct v4l2_ctrl_config u32 id; const char *name; enum v4l2_ctrl_type type; const strcut v4l2_ctrl_ops *ops; s32 minimum, maximum, default; u32 step, flags; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 20 All Rights Reserved.
  21. 21. struct vb2_queue (the Q) #include <media/videobuf2-core.h> enum v4l2_buf_type type; // <linux/videobuf2.h> unsigned int io_modes; // Buffer Q access methods const struct vb2_ops *ops; // Buffer Q operations const struct vb2_mem_ops *mem_ops; // Memory ops void *drv_priv; // Private data unsigned int buf_struct_size; // Q element's size ... APIs int vb2_queue_init(struct vb2_queue *q); void vb2_queue_release(struct vb2_queue *q); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 21 All Rights Reserved.
  22. 22. struct vb2_ops (the Q ops) #include <media/videobuf2-core.h> queue_setup – Sets up the queue & give back info wait_prepare – Release the lock(s) for waiting wait_finish – Take the lock(s) back, after waiting buf_init – Called after allocating a buffer buf_prepare – Called after every queueing buf_finish – Called before every de-queueing buf_cleanup – Called before freeing up the buffer buf_queue – Actually queueing the buffer start_streaming – To start streaming of video stop_streaming – To stop streaming of video © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 22 All Rights Reserved.
  23. 23. struct vb2_mem_ops #include <media/videobuf2-core.h> For operating on Video Memory void *(*alloc)(void *alloc_ctx, unsigned long size); void (*put)(void *buf_priv); For operating on User Space Memory void *(*get_userptr)(void *alloc_ctx, unsigned long vaddr, unsigned long size, int write); void (*put_userptr)(void *buf_priv); Miscellaneous unsigned int (*num_users)(void *buf_priv); void *(*vaddr)(void *buf_priv); int (*mmap)(void *buf_priv, struct vm_area_struct *vma); Standard operations (struct vb2_mem_ops) available for 3 types: vb2_dma_sg_memops <media/videobuf2-dma-sg.h> vb2_vmalloc_memops <media/videobuf2-vmalloc.h> vb2_dma_contig_memops <media/videobuf2-dma-contig.h> © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 23 All Rights Reserved.
  24. 24. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX) Vertical: Video Device Drivers (Character w/ additional video ioctls) struct video_device <media/v4l2-dev.h> struct v4l2_file_operations struct v4l2_ioctl_ops struct v4l2_ctrl_handler <media/v4l2-dev.h> <media/v4l2-ioctl.h> <media/v4l2-ctrls.h> Video Buffer Queue Management <media/videobuf2-core.h> Q: struct vb2_queue; Q ops: struct vb2_ops; v4l2 Control Management Memory ops: struct vb2_mem_ops PCI, USB, I2C Kernel Space Hardware Space Devices: Video Output/Capture, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 24 All Rights Reserved.
  25. 25. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/snd/*) Video Device Interface V4L2 Device Interface File Ops Ioctls Control Ops File Ops Ioctls Control Ops Video Device Driver ... Audio Device Driver Queue Mgmt Ctrl Mgmt Buffer Mgmt Ctrl Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 25 All Rights Reserved.
  26. 26. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device V4L2 v4l2_subdev struct video_device File Ops Control Ops File Ops Control Ops ... Horizontal Mgmt Horizontal Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 26 All Rights Reserved.
  27. 27. “Video for Linux” Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device V4L2 v4l2_subdev struct video_device File Ops Control Ops Control Ops File Ops ... Subdev Ops Horizontal Mgmt Horizontal Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 27 All Rights Reserved.
  28. 28. struct v4l2_device #include <media/v4l2-device.h> struct device *dev; // dev structure from the horizontal struct list_head subdev; // List of registered sub-devices char name[V4L2_DEVICE_NAME_SIZE]; // Unique dev name struct v4l2_ctrl_handler *ctrl_handler; // Global ctrl handler void (*release)(struct v4l2_device *v4l2_dev); // Called when ref == 0 void (*notify) /* Notify from sub-devices */ (struct v4l2_subdev *sd, unsigned int notification, void *arg); struct kref ref; // Reference count spinlock_t lock; // Access protection © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 28 All Rights Reserved.
  29. 29. V4L2 Device Programming Header: <media/v4l2-device.h> APIs int v4l2_device_register (struct device *dev, struct v4l2_device *v4l2_dev); If (dev is passed NULL) => v4l2_dev->name must be set int v4l2_device_unregister (struct v4l2_device *v4l2_dev); Mostly called in the v4l2_dev->release callback void v4l2_device_disconnect /* No equivalent connect call possible */ (struct v4l2_device *v4l2_dev); Sets v4l2_dev->dev to NULL – to be called in disconnect of the horiz. void v4l2_device_get(struct v4l2_device *v4l2_dev); int v4l2_device_put(struct v4l2_device *v4l2_dev); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 29 All Rights Reserved.
  30. 30. struct v4l2_subdev #include <media/v4l2-subdev.h> struct list_head list; // Being a node of the subdev list struct module *owner; struct v4l2_device *v4l2_dev; struct video_device *devnode; char name[V4L2_SUBDEV_NAME_SIZE]; const struct v4l2_subdev_ops *ops struct v4l2_ctrl_handler *ctrl_handler; u32 grp_id; // For driver-specific grouping void *dev_priv, *host_priv; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 30 All Rights Reserved.
  31. 31. V4L2 Subdevice Programming Header: <media/v4l2-subdev.h> APIs void v4l2_subdev_init /* No equivalent shut call reqd */ (struct v4l2_subdev *sd, const struct v4l2_subdev_ops *o); void v4l2_set_subdevdata/subdev_hostdata (struct v4l2_subdev *sd, void *p); void *v4l2_get_subdevdata/subdev_hostdata (const struct v4l2_subdev *sd); v4l2_subdev_call /* For use by the v4l2_device */ (sd, struct, fn_ptr, args...); © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 31 All Rights Reserved.
  32. 32. struct v4l2_subdev_ops #include <media/v4l2-subdev.h> const struct v4l2_subdev_core_ops *core; const struct v4l2_subdev_tuner_ops *tuner; const struct v4l2_subdev_audio_ops *audio; const struct v4l2_subdev_video_ops *video; const struct v4l2_subdev_vbi_ops *vbi; const struct v4l2_subdev_ir_ops *ir; const struct v4l2_subdev_sensor_ops *sensor; const struct v4l2_subdev_pad_ops *pad; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 32 All Rights Reserved.
  33. 33. V4L2 Subdevice Registration Header: <media/v4l2-device.h> APIs int v4l2_device_register_subdev (struct v4l2_device *v4l2_dev, struct v4l2_subdev *sd); void v4l2_device_unregister_subdev (struct v4l2_subdev *sd); v4l2_device_call_all (v4l2_dev, grpid, struct, fn_ptr, args...); Iterators v4l2_device_for_each_subdev(sd, v4l2_dev) © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 33 All Rights Reserved.
  34. 34. struct video_device - revisted #include <media/v4l2-dev.h> struct v4l2_device *v4l2_dev; struct device *parent; char name[32]; struct v4l2_file_operations *fops; struct v4l2_ioctl_ops *ioctl_ops; struct v4l2_ctrl_handler *ctrl_handler; void (*release)(struct video_device *vdev); v4l2_std_id tvnorms; v4l2_std_id current_norm; int debug; struct mutex *lock; © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 34 All Rights Reserved.
  35. 35. V4L2 Subsystem User Space Audio/Video Players/Recorders, Video Conferencing (vlc, mplayer, audacity, mjpg_streamer, skype, browsers, ...) Character Device Files (/dev/videoX, /dev/v4l-subdevX) struct struct struct video_device v4l2_device v4l2_subdev struct video_device File Ops Control Ops V4L2 Control Ops File Ops ... Subdev Ops Horizontal Mgmt Horizontal Mgmt Kernel Space Hardware Space Devices: Video Output/Capture, Audio Out/In, Encoder/Decoders, ... © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 35 All Rights Reserved.
  36. 36. Porting a Video Driver Standard Video Chipset Mostly involves changing pin assignments as per the Board Design New Video Chipset Complete Driver as per the preceeding discussions, need to be implemented © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 36 All Rights Reserved.
  37. 37. What all have we learnt? Evolution of the Linux Video subsystem Original Video Interfaces Frame Buffer Programming Console Programming Video For Linux v2 (v4l2) Programming “Video Device” structure v4l2 buffers & the queue management v4l2 devices & subdevices Putting them all together: Porting Porting through Examples © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 37 All Rights Reserved.
  38. 38. Any Queries? © 2012 Anil Kumar Pugalia <email@sarika-pugs.com> 38 All Rights Reserved.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

×