Code Development for Processing Video Streams


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Change the picture to something more clear!
  • Get the english picture!
  • Reconsider this slide!
  • Code Development for Processing Video Streams

    1. 1. Code Development for Processing Video Streams “NOT reinventing the wheel” Slides taken from a presentation by Dr. Gary Bradski, Intel Computer Corp.
    2. 2. Progress in Computer Vision <ul><li>1970’s vision student </li></ul><ul><ul><li>First step: Build digitizer and video display hardware </li></ul></ul><ul><li>1980’s vision student </li></ul><ul><ul><li>First step: Write image class library and interface to video capture card </li></ul></ul><ul><li>2000’s vision student </li></ul><ul><ul><li>First step: Learn to use standard libraries with standard hardware! </li></ul></ul>
    3. 3. Standard Libraries VisSDK OpenCV IPL VIPeR Toolkit An example program running under Windows which can be customized Link from network file system Download to your local disk
    4. 4. Standard Vision Libraries <ul><li>Intel IPL (now IPP) – Image processing library </li></ul><ul><ul><li>Optimized (MMX) code for filtering, FFT, etc. </li></ul></ul><ul><li>Intel OpenCV – Open Computer Vision library </li></ul><ul><ul><li>Higher level support for image representation and manipulation </li></ul></ul><ul><li>Microsoft Vision SDK – Similar to OpenCV </li></ul><ul><ul><li>Includes standard routines for image and video input and output </li></ul></ul><ul><ul><li>Otherwise, not used very much here </li></ul></ul>
    5. 5. VIPER <ul><li>This is a starter project in Visual C++ which integrates the standard libraries. It includes </li></ul><ul><ul><li>Examples of reading and writing images and video files </li></ul></ul><ul><ul><li>Examples of capturing live video and processing it </li></ul></ul><ul><ul><li>Some simple rendering and interaction hooks under Windows </li></ul></ul><ul><li>Used extensively in Digital Video Special Effects class for several years with great success </li></ul><ul><li>It is not elegant and beautiful code. </li></ul><ul><li>It is a quick way to implement experiments. </li></ul>
    6. 6. First Problem Set <ul><li>It will give you a chance to use Viper with standard libraries to do some simple processing </li></ul><ul><li>Two parts: </li></ul><ul><ul><li>Read in an image and compute its histogram </li></ul></ul><ul><ul><li>Read in an AVI file and do some simple “tracking” </li></ul></ul><ul><li>Instructions for installing VIPER and connecting to standard libraries will be provided. </li></ul><ul><li>Tutorial on VIPER, Visual C++, and histograms. </li></ul><ul><ul><li>This Friday from 4-5. Location: CCB 155 </li></ul></ul>
    7. 7. SDKs Galore! VisSDK OpenCV IPL VIPeR Toolkit
    8. 8. Image Analysis <ul><li>Thresholds </li></ul><ul><li>Statistics </li></ul><ul><li>Pyramids </li></ul><ul><li>Morphology </li></ul><ul><li>Distance transform </li></ul><ul><li>Flood fill </li></ul><ul><li>Feature detection </li></ul><ul><li>Contours retrieving </li></ul>
    9. 9. Image Thresholding <ul><li>Fixed threshold; </li></ul><ul><li>Adaptive threshold; </li></ul>
    10. 10. Image Thresholding Examples Source picture Fixed threshold Adaptive threshold
    11. 11. Statistics <ul><li>min, max, mean value, standard deviation over the image </li></ul><ul><li>Norms C, L1, L2 </li></ul><ul><li>Multidimensional histograms </li></ul><ul><li>Spatial moments up to order 3 (central, normalized, Hu) </li></ul>
    12. 12. Image Pyramids <ul><li>Gaussian and Laplacian pyramids </li></ul><ul><li>Image segmentation by pyramids </li></ul>
    13. 13. Image Pyramids <ul><li>Gaussian and Laplacian </li></ul>
    14. 14. Pyramid-based color segmentation On still pictures And on movies
    15. 15. Morphological Operations <ul><li>Two basic morphology operations using structuring element: </li></ul><ul><ul><li>erosion </li></ul></ul><ul><ul><li>dilation </li></ul></ul><ul><li>More complex morphology operations: </li></ul><ul><ul><li>opening </li></ul></ul><ul><ul><li>closing </li></ul></ul><ul><ul><li>morphological gradient </li></ul></ul><ul><ul><li>top hat </li></ul></ul><ul><ul><li>black hat </li></ul></ul>
    16. 16. What is Structuring Element? <ul><li>The rectangle mask with given position of a principal pixel: </li></ul>
    17. 17. Morphological Operations Examples <ul><li>Morphology - applying Min-Max . Filters and its combinations </li></ul>Opening IoB= (I  B)  B Dilatation I  B Erosion I  B Image I Closing I•B= (I  B)  B TopHat(I)= I - (I  B) BlackHat(I)= (I  B) - I Grad(I)= (I  B)-(I  B)
    18. 18. Distance Transform <ul><li>Calculate the distance for all non-feature points to the closest feature point </li></ul><ul><li>Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined </li></ul>
    19. 19. Flood Filling <ul><li>Simple </li></ul><ul><li>Gradient </li></ul>
    20. 20. Feature Detection <ul><li>Fixed filters (Sobel operator, Laplacian); </li></ul><ul><li>Optimal filter kernels with floating point coefficients (first, second derivatives, Laplacian) </li></ul><ul><li>Special feature detection (corners) </li></ul><ul><li>Canny operator </li></ul><ul><li>Hough transform (find lines and line segments) </li></ul><ul><li>Gradient runs </li></ul>
    21. 21. Canny Edge Detector
    22. 22. Hough Transform Detects lines in a binary image <ul><li>Probabilistic Hough Transform </li></ul><ul><li>Standard Hough Transform </li></ul>
    23. 23. Another Sample of the Hough Transform Using Source picture Result
    24. 24. Contour Retrieving <ul><li>The contour representation: </li></ul><ul><ul><li>Chain code (Freeman code) </li></ul></ul><ul><ul><li>Polygonal representation </li></ul></ul>Initial Point Chain code for the curve: 34445670007654443 Contour representation
    25. 25. Hierarchical representation of contours Image Boundary (W1) (W2) (W3) (B2) (B3) (B4) (W5) (W6)
    26. 26. Contours Examples Source Picture (300x600 = 180000 pts total) Retrieved Contours (<1800 pts total) After Approximation (<180 pts total) And it is rather fast: ~70 FPS for 640x480 on complex scenes
    27. 27. Optical Flow <ul><li>Block matching technique </li></ul><ul><li>Horn & Schunck technique </li></ul><ul><li>Lucas & Kanade technique </li></ul><ul><li>Pyramidal LK algorithm </li></ul><ul><li>6DOF (6 degree of freedom) algorithm </li></ul>Optical flow equations:
    28. 28. Pyramidal Implementation of the optical flow algorithm J image I image Image Pyramid Representation Iterative Lucas – Kanade Scheme Generic Image (L-1)-th Level L-th Level Location of point u on image u L =u/2 L Spatial gradient matrix Standard Lucas – Kanade scheme for optical flow computation at level L d L Guess for next pyramid level L – 1 Finally, Image pyramid building Optical flow computation
    29. 29. Pyramidal Implementation of the optical flow algorithm
    30. 30. 6DOF Algorithm Parametrical optical flow equations:
    31. 31. 3D reconstruction <ul><li>Camera Calibration </li></ul><ul><li>View Morphing </li></ul><ul><li>POSIT </li></ul>
    32. 32. Camera Calibration <ul><li>Define intrinsic and extrinsic camera parameters. </li></ul><ul><li>Define Distortion parameters </li></ul>
    33. 33. Camera Calibration Now, camera calibration can be done by holding checkerboard in front of the camera for a few seconds . And after that you’ll get: 3D view of etalon Un-distorted image
    34. 34. View Morphing
    35. 35. POSIT Algorithm <ul><li>Perspective projection: </li></ul><ul><li>Weak-perspective projection: </li></ul>
    36. 36. References <ul><li>M.Hu. Visual Pattern Recognition by Moment Invariant s, IRE Transactions on Information Theory, 8:2, pp. 179-187, 1962. </li></ul><ul><li>B. Jahne. Digital Image Processin g. Springer, New York, 1997. </li></ul><ul><li>M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active Contour Model s, International Journal of Computer Vision, pp. 321-331, 1988. </li></ul><ul><li>J.Matas, C.Galambos, J.Kittler. Progressive Probabilistic Hough Transfor m. British Machine Vision Conference, 1998. </li></ul><ul><li>A. Rosenfeld and E. Johnston. Angle Detection on Digital Curve s. IEEE Trans. Computers, 22:875-878, 1973. </li></ul><ul><li>Y.Rubner.C.Tomasi,L.J.Guibas. Metrics for Distributions with Applications to Image Databases. Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India, January 1998, pp. 59-66. </li></ul><ul><li>Y. Rubner. C. Tomasi, L.J. Guibas. The Earth Mover’s Distance as a Metric for Image Retrieval. Technical Report STAN-CS-TN-98-86, Department of Computer Science, Stanford University, September, 1998. </li></ul><ul><li>Y.Rubner.C.Tomasi. Texture Metric s. Proceeding of the IEEE International Conference on Systems, Man, and Cybernetics, San-Diego, CA, October 1998, pp. 4601- 4607. </li></ul>
    37. 37. References <ul><li>Gunilla Borgefors. Distance Transformations in Digital Images. Computer Vision, Graphics and Image Processing 34, 344-371,(1986). </li></ul><ul><li>G. Bradski and J. Davis. Motion Segmentation and Pose Recognition with Motion History Gradients. IEEE WACV'00, 2000. </li></ul><ul><li>P. J. Burt, T. H. Hong, A. Rosenfeld. Segmentation and Estimation of Image Region Properties Through Cooperative Hierarchical Computation. IEEE Tran. On SMC, Vol. 11, N.12, 1981, pp.802-809. </li></ul><ul><li>J.Canny. A Computational Approach to Edge Detectio n, IEEE Trans. on Pattern Analysis and Machine Intelligence, 8(6), pp.679-698 (1986). </li></ul><ul><li>J. Davis and Bobick. The Representation and Recognition of Action Using Temporal Template s. MIT Media Lab Technical Report 402,1997. </li></ul><ul><li>Daniel F. DeMenthon and Larry S. Davis. Model-Based Object Pose in 25 Lines of Code. In Proceedings of ECCV '92, pp. 335-343, 1992. </li></ul><ul><li>Andrew W. Fitzgibbon, R.B.Fisher. A Buyer’s Guide to Conic Fittin g.Proc.5 th British Machine Vision Conference, Birmingham, pp. 513-522, 1995. </li></ul><ul><li>Berthold K.P. Horn and Brian G. Schunck. Determining Optical Flo w. Artificial Intelligence, 17, pp. 185-203, 1981. </li></ul>
    38. 38. References <ul><li>J. Serra. Image Analysis and Mathematical Morphology. Academic Press, 1982. </li></ul><ul><li>Bernt Schiele and James L. Crowley. Recognition without Correspondence Using Multidimensional Receptive Field Histograms. In International Journal of Computer Vision 36 (1), pp. 31-50, January 2000. </li></ul><ul><li>S. Suzuki, K. Abe. Topological Structural Analysis of Digital Binary Images by Border Following. CVGIP, v.30, n.1. 1985, pp. 32-46. </li></ul><ul><li>C.H.Teh, R.T.Chin. On the Detection of Dominant Points on Digital Curves. - IEEE Tr. PAMI, 1989, v.11, No.8, p. 859-872. </li></ul><ul><li>Emanuele Trucco, Alessandro Verri. Introductory Techniques for 3-D Computer Visio n. Prentice Hall, Inc., 1998. </li></ul><ul><li>D. J. Williams and M. Shah. A Fast Algorithm for Active Contours and Curvature Estimatio n. CVGIP: Image Understanding, Vol. 55, No. 1, pp. 14-26, Jan., 1992. </li></ul><ul><li>A.Y.Yuille, D.S.Cohen, and P.W.Hallinan. Feature Extraction from Faces Using Deformable Templates in CVP R, pp. 104-109, 1989. </li></ul><ul><li>Zhengyou Zhang. Parameter Estimation Techniques: A Tutorial with Application to Conic Fittin g, Image and Vision Computing Journal, 1996. </li></ul>