Successfully reported this slideshow.

MSCVS2011 - OpenCV

12,012 views

Published on

Published in: Education, Technology, Business

MSCVS2011 - OpenCV

  1. 1. Open Source Computer Vision Library Victor Eruhimov ITSEEZ Microsoft Computer Vision School
  2. 2. Outline• OpenCV Overview• Functionality• Programming with OpenCV• OpenCV on CPU & GPU• Mobile vision 2
  3. 3. OpenCV History • Original goal: • Accelerate the field by lowering the bar to computer vision • Find compelling uses for the increasing MIPS out in the market • Timeline: Willow • Staffing: • Climbed in 1999 to average 7 first couple of years10 • Starting 2003 support declined between zero and one with exception of transferring the machine learning from manufacturing work I led5 (equivalent of 3 people). • Support to zero the couple of years before Willow.0 • 5 people over the last yearGa ry Bra dski 3 3
  4. 4. OpenCV Functionality Overview Image processing•General Image Transforms Fitting Optical Flow Segmentation Processing Video, Stereo, and 3D Camera Pose estimation Features Depth Maps Object detection Calibration 4
  5. 5. OpenCV Architecture and Development 5
  6. 6. OpenCV License•Based on BSD license•Free for commercial and research use•Does not force your code to be open•You need not contribute back –But you are very welcome to contribute back!
  7. 7. OpenCV sponsors
  8. 8. Where is OpenCV Used?• Academic and Industry Research• Security systems• Image/video search and retrieval• Structure from motion in movies• Machine vision factory production inspection systems• Automatic Driver Assistance Systems• Safety monitoring (Dam sites, mines, swimming pools)• Robotics Well over 3M downloads! 2M downloads
  9. 9. Robotics Operation System
  10. 10. Usage examples: ROS• Imagery infrastructure• Camera calibration• Object detection• Pose estimation• Human awareness
  11. 11. Hackathons
  12. 12. Outlet plugin
  13. 13. 3D Processing: PCL Point Cloud Library  http://pointclouds.org/
  14. 14. PCL: Finding Normals
  15. 15. PCL: Filtering by surface curvature
  16. 16. PCL:Using 3D features to classify surface types
  17. 17. OpenCV Czar
  18. 18. Outline• OpenCV Overview• Functionality• Programming with OpenCV• OpenCV on CPU & GPU• Mobile vision 18
  19. 19. How to choose which algorithms to put into the library?• Well established • Cutting edge• Works on any data • Works on Lenna – Productized by a only commercial company vs – Patented • Have to hallucinate the missing pieces• Easy to reproduce – And then it works on Lenna only
  20. 20. Imgproc 1 Robot supportGeneral Image Processing Functions Image Pyramids Integral images Transforms
  21. 21. Imgproc 2 Robot support Segmentation Shape analysis
  22. 22. Features2d contentsDetection Description Matching Matchers available • BruteForce • FlannBased • BOWDetectors available Descriptors available• SIFT • SIFT Matches filters• SURF • SURF (under construction)• FAST • Calonder • Cross check• STAR • Ferns • Ratio check• MSER • One way• HARRIS • HoG• GFTT (Good Features To Track)
  23. 23. Detector testbench• Measures of detector repeatability are taken from – K.Mikolajczyk, Cordelia Schmid, “Scale & Affine Invariant Interest Point Detectors”, IJCV 60(1), 63–86, 2004. – K.Mikolajczyk et al, A Comparison of Affine Region Detectors, IJCV 65(1/2):43-72, 2005.• Test images are taken from http://www.robots.ox.ac.uk/~vgg/data/data-aff.html• Testbench is located in opencv_extra/testdata/cv/detectors_descriptors_evaluation/ detectors
  24. 24. Descriptor testbench• Measures of descriptor matching accuracy are taken from http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolaj czyk_pami2004.pdf• Test images are taken from http://www.robots.ox.ac.uk/~vgg/data/data-aff.html• Testbench is located in opencv_extra/testdata/cv/detectors_descriptors_evaluation/ descriptors
  25. 25. Calib3d module• Camera calibration• 3D -> 2D projection• Homography in 2D and 3D• PnP problem solver• Stereo vision• Fundamental matrix• Template detectors
  26. 26. Pinhole camera model X  X 0        R 0  t Y Y     Z Z    0  X x Z Y y Z u  f x xc x v  f y yc y
  27. 27. Camera CalibrationSee samples/cpp/calibration.cpp 3D view of checkerboard Un-distorted image Gary Bradski (c) 2008 29
  28. 28. Stereo• Once the left an right cameras are calibrated internally (intrinsics) and externally (extrinsics), we need to rectify the images 31
  29. 29. Homography h11u  h12v  h13˜u h31u  h 32v  h33 h21u  h 22v  h 23˜v h31u  h 32v  h 33uw  u ˜    ˜ vw  Hv    w    1
  30. 30. Random Sample Consensus• Do n iterations until #inliers > inlierThreshold – Draw k matches randomly – Find the transformation – Calculate inliers count – Remember the best solution Problem: #matches, #inliers, k matches. How many iterations of RANSAC do you need to get the right answer? k # matches  The number of iterations required ~ C *    # inliers 
  31. 31. Machine Learning Library (MLL)CLASSIFICATION / REGRESSION(new) Fast Approximate NN (FLANN)Naïve Bayes AACBAABBCBCCCARTRandom Forests(new) Extremely Random Trees AACACB CBABBC(new) Gradient Boosting TreesStatistical Boosting, 4 flavorsSVM CCB AAA CB ABBCFace DetectorCLUSTERING CC B C B A BBCK-MeansEM(Mahalanobis distance) BB CTUNING/VALIDATIONCross validationBootstrappingVariable importanceSampling methods 34
  32. 32. ML Lib Example: Boosting: Face Detection with Viola-Jones Rejection CascadeIn samples/cpp, see:Multicascadeclassifier.cpp Gary Bradski (c) 2008 35
  33. 33. Pedestrian Detection: HOG Descriptor• Object shape is characterized by distributions of: – Gradient magnitude – Edge/Gradient orientation• Grid of orientation Magnitude Edge histograms Orientation 36
  34. 34. Object detectionP. Felzenszwalb, D. McAllester, D. Ramaman.
ADiscriminatively Trained, Multiscale,Deformable Part Model.
Proceedings of theIEEE CVPR 2008.
  35. 35. 2D Tracking 3DCamShift(); Posit(); KalmanFilter::MeanShift(); calcOpticalFlowPyrLK() Also see dense optical flow: calcOpticalFlowFarneback() Gary Bradski (c) 2008 38
  36. 36. Useful OpenCV LinksOpenCV Wiki: User Group (~40K members):http://opencv.willowgarage.com/wiki http://tech.groups.yahoo.com/group /OpenCV/joinOpenCV Code Repository:svn co https://code.ros.org/svn/opencv/trunk/opencvNew Book on OpenCV:http://oreilly.com/catalog/9780596516130/ Or, direct from Amazon: http://www.amazon.com/Learning-OpenCV-Computer-Vision- Library/dp/0596516134Code examples from the book:http://examples.oreilly.com/9780596516130/Documentationhttp://opencv.willowgarage.com/documentation/index.html Ga ry Bra dski, 2009 39 39
  37. 37. Outline• OpenCV Overview• Functionality• Programming with OpenCV• OpenCV on CPU & GPU• Mobile vision 40
  38. 38. Main Structures
  39. 39. cv::Mat cv::Mat•Image parameters•Reference counter•Pointer to data Image data RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB****** cv::Mat RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB******•Image parameters RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB******•Reference counter RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB******•Pointer to data RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB****** RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB****** RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB****** cv::Mat RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB******•Image parameters RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB******•Reference counter RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB******•Pointer to data RGBRGBRGBRGBRGBRGBRGBRGBRGBRGB****** Memory layoutRGBRGBRGBRGB******************RGBRGBRGBRGB**********************…
  40. 40. cv::Mat and std::vector std::vector<Point3f> A vector of n points XYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZ cv::Mat•Image parameters cv::Mat•Reference counter RGBRGBRGBRGBRGBRGBRGBRGBRGBRGBRGBRGB•Pointer to data Nx1 3-channel image
  41. 41. Mat are SimpleMat M(480,640,CV_8UC3); // Make a 640x480 imgRect roi(100,200, 20,40); // RectangleMat subM = M(roi); // Take a sub region, // no copy is doneMat_<Vec3b>::iterator it= subM.begin<Vec3b>(), itEnd = subM.end<Vec3b>();//Zero out pixels in subM where blue > redfor(; it != itEnd; ++it) if( (*it)[0] > (*it)[2]) (*it)[0] = 0;
  42. 42. Matrix Manipulation
  43. 43. Simple Matrix Operations
  44. 44. New C++ API: Usage Example Focus Detector C: C++:double calcGradients(const IplImage *src, i nt a perture_size = 7) double contrast_measure(const Mat& i mg){ { CvSi ze s z = cvGetSize(src); Ma t dx, dy; Ipl Image* i mg16_x = cvCrea teImage( sz, IPL_DEPTH_16S, 1); Ipl Image* i mg16_y = cvCrea teImage( s z, IPL_DEPTH_16S, 1); Sobel(img, dx, 1, 0, 3, CV_32F); Sobel(img, dy, 0, 1, 3, CV_32F); cvSobel( src, i mg16_x, 1, 0, a perture_size); ma gnitude(dx, dy, dx); cvSobel( src, i mg16_y, 0, 1, a perture_size); return s um(dx)[0]; Ipl Image* imgF_x = cvCrea teImage( sz, IPL_DEPTH_32F, 1); } Ipl Image* imgF_y = cvCrea teImage( sz, IPL_DEPTH_32F, 1); cvSca l e(img16_x, i mgF_x); cvSca l e(img16_y, i mgF_y); Ipl Image* magnitude = cvCrea teImage( sz, IPL_DEPTH_32F, 1); cvCa rtToPol ar(imgF_x, i mgF_y, magnitude); double res = cvSum(magnitude).val[0]; cvRel easeImage( &magnitude ); cvRel easeImage(&imgF_x); cvRel easeImage(&imgF_y); cvRel easeImage(&img16_x); cvRel easeImage(&img16_y); return res;} 47
  45. 45. Simple Image Processing
  46. 46. Image Conversions
  47. 47. Histogram
  48. 48. I/O
  49. 49. Serialization I/O
  50. 50. Serialization I/O
  51. 51. GUI (“HighGUI”)
  52. 52. Camera Calibration, Pose, Stereo
  53. 53. Problem: planar object detection
  54. 54. Features 2DRead two input images:Mat img1 = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE);Detect keypoints in both images:// detecting keypointsFastFeatureDetector detector(15);vector<KeyPoint> keypoints1;detector.detect(img1, keypoints1);Compute descriptors for each of the keypoints:// computing descriptorsSurfDescriptorExtractor extractor;Mat descriptors1;extractor.compute(img1, keypoints1, descriptors1);Now, find the closest matches between descriptors from the first image to the second:// matching descriptorsBruteForceMatcher<L2<float> > matcher;vector<DMatch> matches;matcher.match(descriptors1, descriptors2, matches); 57
  55. 55. Keypoints example
  56. 56. Matching descriptors example
  57. 57. Geometry validation
  58. 58. Object Recognition
  59. 59. Python Face Detector Node: 1 The Setup#!/us r/bin/python"""Thi s program is demonstration python ROS Node for face a nd object detection using haar-like features.The program finds faces i n a camera i mage or vi deo stream a nd displays a red box a round them. Python i mplementation by:Roma n Stanchak, James Bowman"""i mport roslibros l ib.load_manifest(opencv_tests)i mport sysi mport osfrom optparse import OptionParseri mport rospyi mport s ensor_msgs.msgfrom cv_bri dge import CvBri dgei mport cv# Pa ra meters for haar detection# From the API:# The default parameters (s cale_factor=2, min_neighbors=3, flags=0) a re tuned# for a ccurate yet slow object detection. For a faster operation on real vi deo# i ma ges the s ettings are:# s ca le_factor=1.2, min_neighbors=2, flags=CV_HAAR_DO_CANNY_PRUNING,# mi n_size=<minimum possible face sizemi n_size = (20, 20)i ma ge_scale = 2ha a r_scale = 1.2mi n_neighbors = 2ha a r_flags = 0 62
  60. 60. Python Face Detector Node: 2 The Coreif __name__ == __main__: pkgdir = roslib.packages.get_pkg_dir("opencv2") haarfile = os.path.join(pkgdir, "opencv/share/opencv/haarcascades/haarcascade_frontalface_alt.xml") parser = OptionParser(usage = "usage: %prog [options] [filename|camera_index]") parser.add_option("-c", "--cascade", action="store", dest="cascade", type="str", help="Haar cascade file, default %default", default = haarfile) (options, args) = parser.parse_args() cascade = cv.Load(options.cascade) br = CvBridge() if(cascade): faces = cv.HaarDetectObjects(small_img, cascade, cv.CreateMemStorage(0), def detect_and_draw(imgmsg): haar_scale, min_neighbors, haar_flags, min_size) img = br.imgmsg_to_cv(imgmsg, "bgr8") if faces: # allocate temporary images for ((x, y, w, h), n) in faces: gray = cv.CreateImage((img.width,img.height), 8, 1) # the input to cv.HaarDetectObjects was resized, so scale the # bounding box of each face and convert it to two CvPoints small_img = cv.CreateImage((cv.Round(img.width / image_scale), pt1 = (int(x * image_scale), int(y * image_scale)) cv.Round (img.height / image_scale)), 8, 1) pt2 = (int((x + w) * image_scale), int((y + h) * image_scale)) cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0) # convert color input image to grayscale cv.CvtColor(img, gray, cv.CV_BGR2GRAY) cv.ShowImage("result", img) cv.WaitKey(6) # scale input image for faster processing cv.Resize(gray, small_img, cv.CV_INTER_LINEAR) rospy.init_node(rosfacedetect) image_topic = rospy.resolve_name("image") cv.EqualizeHist(small_img, small_img) rospy.Subscriber(image_topic, sensor_msgs.msg.Image, detect_and_draw) rospy.spin() 63
  61. 61. Outline• OpenCV Overview• Functionality• Programming with OpenCV• OpenCV on CPU & GPU• Mobile vision 64
  62. 62. Hardware optimization: Intel architectures• Technologies • SSE • TBB • IPP• Highlights of C++ in-house optimization • arithmetical operations on large matrices/images: add, sub, absdiff - • image filtering: e.g. median 3x3 filter is ! • geometrical transformations: resize is • template matching: • large matrix processing: SVD of 50x50-1000x1000 matrices is
  63. 63. OpenCV GPU ModuleGoals:• Provide developers with a convenient computer vision framework on the GPU• Maintain conceptual consistency with the current CPU functionality• Achieve the best performance with GPUs – Efficient kernels tuned for modern architectures – Optimized dataflows (asynchronous execution, copy overlaps, zero-copy) 66
  64. 64. OpenCV GPU Module Contents• Image processing building blocks: Per- Color Geometrical Integrals, element conversions transforms reductions operations Template Filtering Feature matching engine detectors• High-level algorithms: FeatureStereo matching Face detection matching
  65. 65. OpenCV GPU Module Usage• Prerequisites: – Get sources from SourceForge or SVN – CMake – NVIDIA Display Driver – NVIDIA GPU Computing Toolkit (for CUDA)• Build OpenCV with CUDA support• #include <opencv2/gpu/gpu.hpp> http://opencv.willowgarage.com/wiki/InstallGui de 68
  66. 66. OpenCV GPU Data Structures• Class GpuMat – For storing 2D image in GPU // class GpuMat GpuMat(Size size, int type); memory, just like class cv::Mat GpuMat(const GpuMat& m); explicit GpuMat (const Mat& m); – Reference counting GpuMat& operator = (const GpuMat& m); GpuMat& operator = (const Mat& m); – Can point to data allocated by void upload(const Mat& m); user void upload(const CudaMem& m, Stream& stream); void download(Mat& m) const; void download(CudaMem& m, Stream& stream)• Class CudaMem const; – For pinned memory support – Can be transformed into // class Stream cv::Mat or cv::gpu::GpuMat bool queryIfComplete(); void waitForCompletion(); void enqueueDownload(const GpuMat& src, Mat& dst);• Class Stream void enqueueUpload(const Mat& src, GpuMat& dst); – Overloads with extra Stream void enqueueCopy(const GpuMat& src, parameter GpuMat& dst); 69
  67. 67. OpenCV GPU Module ExampleMat frame; Mat frame;VideoCapture capture(camera); VideoCapture capture(camera);cv::HOGDescriptor hog; cv::gpu::HOGDescriptor hog;hog.setSVMDetector(cv::HOGDescriptor hog.setSVMDetector(cv::HOGDescriptor:: ::getDefaultPeopleDetectorector()); getDefaultPeopleDetectorector());capture >> frame; capture >> frame; GpuMat gpu_frame; gpu_frame.upload(frame);vector<Rect> found; vector<Rect> found;hog.detectMultiScale(frame, found, hog.detectMultiScale(gpu_frame, 1.4, Size(8, 8), Size(0, 0), found,1.05, 8); 1.4, Size(8, 8), Size(0, 0), 1.05, 8);• Designed very similar! 70
  68. 68. OpenCV and NPP• NPP is NVIDIA Performance Primitives library of signal and image processing functions (similar to Intel IPP) – NVIDIA will continue adding new primitives and optimizing for future architectures• GPU module uses NPP whenever possible – Highly optimized implementations for all supported NVIDIA architectures and OS – Part of CUDA Toolkit – no additional dependencies• OpenCV extends NPP and uses it to build higher level CV 71
  69. 69. OpenCV GPU Module Performance Tesla C2050 (Fermi) vs. Core i5-760 2.8GHz (4 cores, TBB, SSE) – Average speedup for primitives: 33 • For “good” data (large images are better) • Without copying to GPU What can you get from your computer? – opencvsamplesgpuperfomance 72
  70. 70. OpenCV GPU: Viola-Jones Cascade Classifier• Used for face detection• Speed-up ~ 6• Based on NCV classes (NVIDIA implementation) 73
  71. 71. OpenCV GPU: Histogram of Oriented Gradients • Used for pedestrian detection • Speed-up ~ 8 74
  72. 72. OpenCV GPU: Speeded Up Robust Features • SURF (12) • Bruteforce matcher – K-Nearest search (20- 30) – In radius search (3-5) 75
  73. 73. OpenCV GPU: Stereo Vision• Stereo Block Matching (7) – Can run Full HD real-time on Dual-GPU GPU BM FULL HD• Hierarchical Dense Stereo – Belief Propagation (20) GPU CSBP FULL HD – Constant space BP (50- 100) 76
  74. 74. Epipolar geometry Fundamental matrix constraint  xR    xL , yL ,1 F   yR   0 1   
  75. 75. Block matching 78
  76. 76. Parallel algorithm 79
  77. 77. Оптимизация кода на CUDAПроблема №1: occupancy• Дефицит регистров – Отказ от текстурного механизма – Ручная оптимизация кода на CUDA• Дефицит SMEM – Использование unsigned char и short – Подбор размера блока потоков 80
  78. 78. Оптимизация кода на CUDAПроблема №3: сложные вычисления• Переход к относительной адресации памяти внутри потоков• Переход на быстрые операции• Удаление повторных вычислений 81
  79. 79. Stereo on HD in realtime
  80. 80. Outline• OpenCV Overview• Functionality• Programming with OpenCV• OpenCV on GPU• Mobile vision 83
  81. 81. Traditional cameras vs. camera phones
  82. 82. ARM is Pervasive and Open 9 Annual ARM 8 Shipments x86 7 6Units in Billions 5 4 3 2 1 0 2005 2006 2007 2008 2009 2010e 2011e 2012e 2013e 2014e Source: ARM, Mercury Research, NVIDIA
  83. 83. Using OpenCV for Android OpenCV 2.3 for Android: Native Android Camera Support Multithreading Java API (soon) Tegra HW Optimizations (soon) Wiki with the latest information: http://opencv.willowgarage.com/wiki/Android Support/discussion group:::https://groups.google.com/group/android- opencv
  84. 84. Practical session• Look for opencv_practical_session.tar.gz on your USB stick• Unzip and install OpenCV• Make sure you have your favorite compiler environment ready• CMake is optional but highly recommended
  85. 85. QUESTIONS?
  86. 86. OpenCV GPU: Viola-Jones Cascade Classifier• Used for face detection• Speed-up ~ 6• Based on NCV classes (NVIDIA implementation) 89
  87. 87. Canny Edge Detector 90
  88. 88. Distance Transform• Distance field from edges of objects Flood Filling 91
  89. 89. Hough TransformGary Bradski, Adrian Kahler 2008 92
  90. 90. Space Variant vision: Log-Polar Transform Screen shots by Gary Bradski, 2005 93
  91. 91. Scale Space Chart by Gary Bradski, 2005void cvPyrDown( void cvPyrUp( IplImage* src, IplImage* src, IplImage* dst, IplImage* dst, IplFilter filter = IPL_GAUSSIAN_5x5); IplFilter filter = IPL_GAUSSIAN_5x5); 94
  92. 92. Thresholds Screen shots by Gary Bradski, 2005 95
  93. 93. Histogram Equalization Screen shots by Gary Bradski, 2005 96
  94. 94. Contours 97
  95. 95. Morphological Operations Examples• Morphology - applying Min-Max. Filters and its combinations Image I Erosion IB Dilatation IB Opening IoB= (IB)BClosing I•B= (IB)B Grad(I)= (IB)-(IB) TopHat(I)= I - (IB) BlackHat(I)= (IB) - I
  96. 96. Image textures• Inpainting:• Removes damage to images, in this case, it removes the text.
  97. 97. Segmentation• Pyramid, mean-shift, graph-cut• Here: Watershed Screen shots by Gary Bradski, 2005 100 100
  98. 98. Recent Algorithms: GrabCut• Graph Cut based segmentation Images by Gary Bradski, © 2010 101
  99. 99. Motion Templates (work with James Davies)• Object silhouette• Motion history images• Motion history gradients• Motion segmentation algorithmsilhouette MHI MHG Charts by Gary Bradski, 2005 102
  100. 100. Segmentation, Motion Tracking and Gesture RecognitionMotion MotionSegmentation SegmentationPose GestureRecognition Recognition Screen shots by Gary Bradski, 2005
  101. 101. New Optical Flow Algorithms// opencv/samples/c/lkdemo.cint main(…){…CvCapture* capture = <…> ? cvCaptureFromCAM(camera_id) : cvCaptureFromFile(path);if( !capture ) return -1; lkdemo.c, 190 lines (needs camera to run)for(;;) { I ( x  dx, y  dy, t  dt )  I ( x, y , t ); IplImage* frame=cvQueryFrame(capture);  I / t  I / x  ( dx / dt )  I / y  ( dy / dt ); if(!frame) break; // … copy and process image G   X  b,cvCalcOpticalFlowPyrLK( …) 2 I x , I xI y  I x  cvShowImage( “LkDemo”, result ); X  (x, y), G    , b   I t   c=cvWaitKey(30); // run at ~20-30fps speed I x I y , I 2   y  I y  if(c >= 0) { // process key }}cvReleaseCapture(&capture);}
  102. 102. Tracking with CAMSHIFT• Control game with head Screen shots by Gary Bradski, 2005
  103. 103. Projections Screen shots by Gary Bradski, 2005
  104. 104. Stereo … Depth from Triangulation• Involved topic, here we will just skim the basic geometry.• Imagine two perfectly aligned image planes: Depth “Z” and disparity “d” are inversly related: 107
  105. 105. Stereo• In aligned stereo, depth is from similar triangles: T  (x l  xr ) T fT  Z l Z f Z x  xr• Problem: Cameras are almost impossible to align• Solution: Mathematically align them: 108 Al l : Ga ry Bradski and Adrian Ka ehler: Learning OpenCV
  106. 106. Stereo Rectification • Algorithm steps are shown at right: • Goal: – Each row of the image contains the same world points – “Epipolar constraint” Result: Epipolar alignment of features: 109Al l : Ga ry Bradski and Adrian Ka ehler: Learning OpenCV
  107. 107. Outline• OpenCV Overview• Cheatsheet• Simple Programs• Tour• Features2D• Applications Ga ry Bra dski, 2009 110
  108. 108. Features2d contentsDetection Description Matching Matchers available • BruteForce • FlannBased • BOWDetectors available Descriptors available• SIFT • SIFT Matches filters• SURF • SURF (under construction)• FAST • Calonder • Cross check• STAR • Ferns • Ratio check• MSER • One way• HARRIS• GFTT (Good Features To Track)
  109. 109. Detector interfacesclass FeatureDetector{public: virtual ~FeatureDetector() {} // Detect keypoints in an image. virtual void detect( const Mat& image, vector<KeyPoint>& keypoints, const Mat& mask=Mat() ) const = 0; // Detect keypoints in an image set. void detect( const vector<Mat>& imageCollection, vector<vector<KeyPoint> >& pointCollection, const vector<Mat>& masks=vector<Mat>() ) const; virtual void read( const FileNode& fn ) {} virtual void write( FileStorage& fs ) const {}protected:…};
  110. 110. Creating a detector• Statically SurfFeatureDetector detector;• Using class factory cv::Ptr<FeatureDetector> detector = createFeatureDetector(“SURF”);
  111. 111. Running detectorMat img = imread( "test.png" );vector<KeyPoint> keypoints;SurfFeatureDetector detector;detector.detect( img, keypoints );
  112. 112. Descriptor interfaces• For descriptors that can be represented as vectors in multidimensional space: DescriptorExtractor and DescriptorMatcher• More general interface (one way, decision- tree-based descriptors): GenericDescriptorMatcher
  113. 113. DescriptorExtractor interfacesclass CV_EXPORTS DescriptorExtractor{public: virtual ~DescriptorExtractor() {} // Compute the descriptors for a set of keypoints in an image. virtual void compute( const Mat& image, vector<KeyPoint>& keypoints, Mat& descriptors ) const = 0; // Compute the descriptors for a keypoints collection detected in image collection. void compute( const vector<Mat>& imageCollection, vector<vector<KeyPoint> >& pointCollection, vector<Mat>& descCollection ) const; virtual void read( const FileNode& ) {} virtual void write( FileStorage& ) const {} virtual int descriptorSize() const = 0; virtual int descriptorType() const = 0;protected:…};
  114. 114. DescriptorExtractor creating• Statically SurfDescriptorExtractor descriptorExtractor;• Using class factory cv::Ptr<DescriptorExtractor> descriptorExtractor = createDescriptorExtractor("SURF");
  115. 115. DescriptorExtractor runningPtr<FeatureDetector> detector = createFeatureDetector("FAST");Ptr<DescriptorExtractor> descriptorExtractor = createDescriptorExtractor("SURF");vector<KeyPoint> keypoints;detector->detect( img, keypoints );Mat descriptors;descriptorExtractor->compute( img, keypoints, descriptors );
  116. 116. DescriptorMatcher interfaces• Two groups of match methods – to match descriptors of image pair – to match descriptors of one image to image set• Each group consists from tree type methods – match() – knnMatch() – radiusMatch()
  117. 117. Matching of image pair// detecting keypointsSurfFeatureDetector detector;vector<KeyPoint> keypoints1, keypoints2;detector.detect( img1, keypoints1 );detector.detect( img2, keypoints2 );// computing descriptorsSurfDescriptorExtractor extractor;Mat descriptors1, descriptors2;extractor.compute( img1, keypoints1, descriptors1 );extractor.compute( img2, keypoints2, descriptors2 );// matching descriptorsBruteForceMatcher<L2<float> > matcher;vector<DMatch> matches;matcher.match( descriptors1, descriptors2, matches );
  118. 118. Visualize keypointsMat img_points;drawKeypoints( img, keypoints, img_points );namedWindow( “keypoints”, 1 );imshow( “keypoints”, img_points );waitKey(); Visualize matchesMat img_matches;drawMatches( img1, keypoints1, img2, keypoints2, img_matches);namedWindow( “matches”, 1 );imshow( “matches”, img_matches );waitKey();
  119. 119. Running the sample• Download OpenCV• Compile• Run matcher_simple: bin/matcher_simple ../../opencv/samples/c/box.png ../../opencv/samples/c/box_in_scene.png• Select a detector that gives the maximum number of keypoints• Switch SIFT and SURF descriptors
  120. 120. Cross-check outlier match filteringBruteForceMatcher<L2<float> > descriptorMatcher;vector<DMatch> filteredMatches12, matches12, matches21;descriptorMatcher.match( descriptors1, descriptors2, matches12 );descriptorMatcher.match( descriptors2, descriptors1, matches21 );for( size_t i = 0; i < matches12.size(); i++ ){ DMatch forward = matches12[i]; DMatch backward = matches21[forward.trainIdx]; if( backward.trainIdx == forward.queryIdx ) filteredMatches12.push_back( forward );}
  121. 121. Ratio test to filter matches MinDist 1Ratio   (0,1] (less is better) MinDist 2if Ratio  threshold(0.3)  inlier, else outlier
  122. 122. Calculating inliers (planar objects case)• Detect keypoints• Find matches using descriptors• Filter matches using cross-check• Calculate best homography• Filter outliers• Run bin/descriptor_extractor_matcher SURF SURF ../../opencv/samples/c/box.png ../../opencv/samples/c/box_in_scene.png 3 The last parameter is the reprojection threshold for RANSAC
  123. 123. OpenCV and ROS• Opencv2 package to fetch and compile opencv• Messages: – sensor_msgs::Image – sensor_msgs::CameraInfo• cv_bridge to convert between messages and images• image_geometry::PinholeCameraModel and image_geometry::StereoCameraModel to manage 2d <-> 3d conversions
  124. 124. Q&A• Foils will be available at http://itseez.com

×