Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modern features-part-3-software

1,644 views

Published on

  • Login to see the comments

Modern features-part-3-software

  1. 1. SoftwareFeature implementation and computationAndrea Vedaldi, University of Oxfordhttps://sites.google.com/site/eccv12features/goo.gl/jKC2J
  2. 2. 2VLFeatMATLAB ease of useCovariant detectors, MSER, SIFT, etc ...OpenCVOne-shop experienceThe most popular vision libraryOthersThere is certainly no lack of software!
  3. 3. 3VLFeatMATLAB ease of useCovariant detectors, MSER, SIFT, etc ...OpenCVOne-shop experienceThe most popular vision libraryOthersThere is certainly no lack of software!
  4. 4. VLFeat 4 An open and portable library of CV software http://www.vlfeat.org/benchmarks/index.html Quick Start with MATLAB>>"untar"http://www.vlfeat.org/download/vlfeat50.9.165bin.tar.gz">>"run vlfeat-0.9.16/toolbox/vl_setup Mac Linux Windows Since 2008 ACM OSS award
  5. 5. SIFT detector and descriptor 51. load an image imPath = fullfile(‘oxford.jpg’) ; im = imread(imPath) ;2. convert it to single precision gray scale imgs = im2single(rgb2gray(im)) ;3. run SIFT [frames, descrs] = vl_sift(imgs) ;4. visualise keypoints imagesc(im) ; colormap gray; hold on ; vl_plotframe(frames) ;5. visualise descriptors vl_plotsiftdescriptor(... descrs(:,432), frames(:,432)) ;
  6. 6. Example: matching features 6 684 tentative matchesnr : :h f t :a r / /h f t /a r h f t : /a
  7. 7. Example: stitching 7 684 tentative matches 492 (76%) inliers mosaic• Live demonstration
  8. 8. Other covariant detectors 8 frames = vl_covdet(imgs, verbose) ;imagesc(im) ; vl_plotframes(frames) ; Do not forget to convert the images to grayscale in single precision. Use im2single(im) and not single(im) !
  9. 9. Understanding feature frames 9 x     x a11 a12 b x t1 affine tf. = + 1 y a21 a22 b y t2 b x b y y• frames has a feature frame for each column containing a stacked affine transformation (A,T) 2 3 t1 6 t2 7 6 7     6a11 7 x a11 a12 b x t1 frames(:,i) = 6 6a21 7 7 = + 6 7 y a21 a22 b y t2 4a12 5 a22
  10. 10. A frame as an oriented ellipse 10 reference = two vectors uniquely identifies the affine 1 transformation b x b yreference = oriented ellipse
  11. 11. Different cornerness measures 11 frames = vl_covdet(imgs, method, Hessian) ; DoG Hessian• Difference of Gaussian (Laplacian, trace of Hessian, or SIFT)• Hessian (determinant of) HarrisLaplace HessianLaplace
  12. 12. Different cornerness measures 12 frames = vl_covdet(imgs, method, HarrisLaplace) ; HarrisLaplace HessianLaplace• Harris Laplace (Harris cornerness, Laplacian for scale selection)• Hessian Laplace MultiscaleHarris MultiscaleHessian
  13. 13. Different cornerness measures 13 frames = vl_covdet(imgs, method, MultiscaleHarris) ; MultiscaleHarris MultiscaleHessian• Multiscale Harris (just a bunch of Harris)• Multiscale Hessian
  14. 14. Affine adaptation 14frames = vl_covdet(imgs, EstimateAffineShape, true) ;
  15. 15. Upright features 15 By default the rotation is selected so that the vertical axis stays vertical. In formulas    x a11 0 b x t1 = + y a21 a22 b y t2Remark: for general affine transformationsthe horizontal direction may appear rotated even for upright features.
  16. 16. Orientation detection 16frames = vl_covdet(imgs, EstimateOrientations, true) ;
  17. 17. Computing descriptors 17 [frames, descrs] = vl_covdet(imgs) ;• Each column of descrs is a stacked SIFT descriptor• How is a SIFT descriptor associated to a feature?• What about other descriptors?
  18. 18. Computing descriptors 18[frames, descrs] = vl_covdet(imgs, Descriptor, patch) ; Each column of descrs is a stacked normalised image patch.
  19. 19. The geometry of an normalised patch 19• Normalisation and patch resampling is controlled by three parameters: (antialiasing effect) affine tf. (size of patch in normalised units) (number of pixels)
  20. 20. Example: normalised patch for SIFT 20 =1 1 3 6 f v w = 15
  21. 21. Custom frames 21Example: running affine adaptation on a grid frames = vl_covdet(imgs, ... Frames, frames, ... EstimateAffineShape, true, ... EstiamteOrientation, true) ;
  22. 22. Obtaining additional information on the features 22 [frames, descrs, info] = vl_covdet(imgs); info = gss: [1x1 struct] css: [1x1 struct] peakScores: [1x351 single] edgeScores: [1x351 single] orientationScores: [1x351 single] laplacianScaleScores: [1x351 single] 120 120 100 100 80 80 Occurences Occurences 60 60 40 40 20 20 0 0 −0.1 −0.05 0 0.05 0.1 0 2 4 6 8 10 Peak Score Edge Score
  23. 23. Obtaining the scale space 23 [frames, descrs, info] = vl_covdet(imgs); info = gss: [1x1 struct] Gaussian css: [1x1 struct] scale space peakScores: [1x351 single] edgeScores: [1x351 single] orientationScores: [1x351 single] laplacianScaleScores: [1x351 single]octave −1 octave 0 octave 1octave 2 octave 3
  24. 24. Obtaining the scale space 24 [frames, descrs, info] = vl_covdet(imgs); info = gss: [1x1 struct] cornerness css: [1x1 struct] measure peakScores: [1x351 single] edgeScores: [1x351 single] orientationScores: [1x351 single] laplacianScaleScores: [1x351 single]octave −1 octave 0 octave 1octave 2 octave 3
  25. 25. Performance comparison 25 Oxford 5K lite retrieval benchmark Num. features / image mAPVL HessianLaplace (double) VL HessianLaplace (double) 7131 79.9% VL DoG (double) 5631 DoG (double) VL 85.1% VL Hessian (double) 3411 VL Hessian (double) 77.8% VL DoG (no aff) 2794 VL DoG (no aff) 74.0% VL HarrisLaplace (double) 2316 VL HarrisLaplace (double) 77.8% CMP Hessian 2292 CMP Hessian 77.1% VL HessianLaplace 2020 VL HessianLaplace 73.8% VL DoG 1854 VL DoG 80.1% VGG hes 1517 VGG hes 74.6% VGG har 1384 VGG har 71.9% VL Hessian 1089 VL Hessian 68.7% VL HarrisLaplace 765 VL HarrisLaplace 70.4% 0 2000 4000 6000 8000 0.6500 0.7125 0.7750 0.8375 0.9000
  26. 26. Repeatability: viewpoint 26 Repeatability (graf) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (graf) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 30 40 50 60 20 Viewpoint angle
  27. 27. Repeatability: viewpoint 27 Repeatability (wall) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (wall) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 30 40 50 60 20 Viewpoint angle
  28. 28. Repeatability: scale 28 Repeatability (boat) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (boat) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 1.38 1.9 2.35 2.8 1.12 Scale changes
  29. 29. Repeatability: JPEG compression 29 Repeatability (ubc) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (ubc) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 80 90 95 98 60 JPEG compression %
  30. 30. Maximally Stable Extremal Regions (MSER) 30 [Matas et al. 2002]• Extracts MSER regions regions and fitted ellipses frames: imgs = uint8(rgb2gray(I)) ; [regions,"frames]"="vl_mser(imgs)";• Control region stability: [regions,"frames]"="vl_mser(imgs,"... """"""""""""""""""""""""""""‘MinDiversity’,"0.7,"... """"""""""""""""""""""""""""‘MaxVariation’,"0.2,"... """"""""""""""""""""""""""""‘Delta’,"10)";
  31. 31. Many other features and tools 31 superpixels MSER agglomerative information bottleneck distance transform di erent mode seeking algo- sent (some of) the data points nal to the Parzen density esti-wards the mode approximately ates mean shift trajectories byxt and in Fig. 2, medoid shifts circles. This disconnects por- alleviated (but not solved) byect. 3) seeks the energy modes s, trading-o mode over- and randomized kd-trees fast k-means feature maps, stochastic gradient SVMs, where d is the dimension- is only O(dN 2 T ), where T Fast Dense SIFT lever implementations yield
  32. 32. Many other features and tools 32 SLIC superpixels Histogram of Oriented Gradients (HOG) HARVEST ProgrammeNew SGD Learning of SVMs
  33. 33. OpenCV 33• The largest, best known vision library• Features - OpenCV 2.1: Harris, Tomasi&Shi, MSER, DoG, SURF, STAR and FAST, DoG + SIFT, SURF, HARRIS, GoodFeaturesToTrack - OpenCV 2.4: ORB, BRIEF, and FREAK Example: computing ORB features. #include <opencv2/core/core.hpp> #include <opencv2/features2d/features2d.hpp> #include <vector> Mat image = importImage8UC1(in[IN_I]); Mat descriptors; std::vector<KeyPoint> frames; ORB orb(numFeatures, scaleFactor, numLevels, edgeThreshold, 0, wtak, scoreTypeToORBPar(scoreType), patchSize); orb(image, Mat(), frames);
  34. 34. This is not all folks 34• Abundance of open source / free software for CV - Always check out authors home pages• Examples Rob Hess SIFT SURF http://robwhess.github.com/opensift/ http://www.vision.ee.ethz.ch/~surf/ Open SURF http://www.chrisevansdev.com/computer-vision-opensurf.html FAST corners http://www.edwardrosten.com/work/fast.html LIOP descriptors http://vision.ia.ac.cn/Students/bfan/index.htm ...
  35. 35. Credits 35 Karel Lenc Daniele Perrone Michal Perdoch HARVEST ProgrammeKrystian Mikolajczyk Tinne Tuytelaars Jiri Matas Cordelia Schmid Andrew Zisserman

×