SoftwareFeature implementation and computationAndrea Vedaldi, University of Oxfordhttps://sites.google.com/site/eccv12feat...
2VLFeatMATLAB ease of useCovariant detectors, MSER, SIFT, etc ...OpenCVOne-shop experienceThe most popular vision libraryO...
3VLFeatMATLAB ease of useCovariant detectors, MSER, SIFT, etc ...OpenCVOne-shop experienceThe most popular vision libraryO...
VLFeat                                 4             An open and portable library of CV software      http://www.vlfeat.or...
SIFT detector and descriptor   51. load an image  imPath = fullfile(‘oxford.jpg’) ;  im = imread(imPath) ;2. convert it to...
Example: matching features      6                      684 tentative matchesnr :   :h f    t :a r /   /h f    t /a r      ...
Example: stitching                  7                                684 tentative matches                  492 (76%) inli...
Other covariant detectors                            8    frames = vl_covdet(imgs, verbose) ;imagesc(im) ;                ...
Understanding feature frames                                           9                                                  ...
A frame as an oriented ellipse   10   reference = two vectors                               uniquely identifies            ...
Different cornerness measures                           11            frames = vl_covdet(imgs, method, Hessian) ;         ...
Different cornerness measures                           12        frames = vl_covdet(imgs, method, HarrisLaplace) ;       ...
Different cornerness measures                13      frames = vl_covdet(imgs, method, MultiscaleHarris) ;                M...
Affine adaptation                        14frames = vl_covdet(imgs, EstimateAffineShape, true) ;
Upright features   15     By default the rotation is    selected so that the vertical        axis stays vertical.         ...
Orientation detection                      16frames = vl_covdet(imgs, EstimateOrientations, true) ;
Computing descriptors          17                   [frames, descrs] = vl_covdet(imgs) ;•   Each column of descrs is a sta...
Computing descriptors                              18[frames, descrs] = vl_covdet(imgs, Descriptor, patch) ;              ...
The geometry of an normalised patch                                  19•   Normalisation and patch resampling is controlle...
Example: normalised patch for SIFT      20                                       =1         1    3     6                  ...
Custom frames                      21Example: running affine adaptation on a grid    frames = vl_covdet(imgs, ...        Fr...
Obtaining additional information on the features                                                             22           ...
Obtaining the scale space                                  23      [frames, descrs, info] = vl_covdet(imgs);            in...
Obtaining the scale space                                 24      [frames, descrs, info] = vl_covdet(imgs);            inf...
Performance comparison                                                                                    25              ...
Repeatability: viewpoint                                      26                                    Repeatability (graf)  ...
Repeatability: viewpoint                                      27                                    Repeatability (wall)  ...
Repeatability: scale                                         28                                     Repeatability (boat)  ...
Repeatability: JPEG compression                                     29                                     Repeatability (...
Maximally Stable Extremal Regions (MSER)             30                                  [Matas et al. 2002]•   Extracts M...
Many other features and tools                                             31           superpixels                MSER    ...
Many other features and tools                           32 SLIC superpixels          Histogram of Oriented Gradients (HOG)...
OpenCV                                  33•   The largest, best known vision library•   Features    -   OpenCV 2.1:       ...
This is not all folks                                 34•   Abundance of open source / free software for CV    -   Always ...
Credits                                    35         Karel Lenc         Daniele Perrone            Michal Perdoch        ...
Upcoming SlideShare
Loading in...5
×

Modern features-part-3-software

845

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
845
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
42
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Modern features-part-3-software

  1. 1. SoftwareFeature implementation and computationAndrea Vedaldi, University of Oxfordhttps://sites.google.com/site/eccv12features/goo.gl/jKC2J
  2. 2. 2VLFeatMATLAB ease of useCovariant detectors, MSER, SIFT, etc ...OpenCVOne-shop experienceThe most popular vision libraryOthersThere is certainly no lack of software!
  3. 3. 3VLFeatMATLAB ease of useCovariant detectors, MSER, SIFT, etc ...OpenCVOne-shop experienceThe most popular vision libraryOthersThere is certainly no lack of software!
  4. 4. VLFeat 4 An open and portable library of CV software http://www.vlfeat.org/benchmarks/index.html Quick Start with MATLAB>>"untar"http://www.vlfeat.org/download/vlfeat50.9.165bin.tar.gz">>"run vlfeat-0.9.16/toolbox/vl_setup Mac Linux Windows Since 2008 ACM OSS award
  5. 5. SIFT detector and descriptor 51. load an image imPath = fullfile(‘oxford.jpg’) ; im = imread(imPath) ;2. convert it to single precision gray scale imgs = im2single(rgb2gray(im)) ;3. run SIFT [frames, descrs] = vl_sift(imgs) ;4. visualise keypoints imagesc(im) ; colormap gray; hold on ; vl_plotframe(frames) ;5. visualise descriptors vl_plotsiftdescriptor(... descrs(:,432), frames(:,432)) ;
  6. 6. Example: matching features 6 684 tentative matchesnr : :h f t :a r / /h f t /a r h f t : /a
  7. 7. Example: stitching 7 684 tentative matches 492 (76%) inliers mosaic• Live demonstration
  8. 8. Other covariant detectors 8 frames = vl_covdet(imgs, verbose) ;imagesc(im) ; vl_plotframes(frames) ; Do not forget to convert the images to grayscale in single precision. Use im2single(im) and not single(im) !
  9. 9. Understanding feature frames 9 x     x a11 a12 b x t1 affine tf. = + 1 y a21 a22 b y t2 b x b y y• frames has a feature frame for each column containing a stacked affine transformation (A,T) 2 3 t1 6 t2 7 6 7     6a11 7 x a11 a12 b x t1 frames(:,i) = 6 6a21 7 7 = + 6 7 y a21 a22 b y t2 4a12 5 a22
  10. 10. A frame as an oriented ellipse 10 reference = two vectors uniquely identifies the affine 1 transformation b x b yreference = oriented ellipse
  11. 11. Different cornerness measures 11 frames = vl_covdet(imgs, method, Hessian) ; DoG Hessian• Difference of Gaussian (Laplacian, trace of Hessian, or SIFT)• Hessian (determinant of) HarrisLaplace HessianLaplace
  12. 12. Different cornerness measures 12 frames = vl_covdet(imgs, method, HarrisLaplace) ; HarrisLaplace HessianLaplace• Harris Laplace (Harris cornerness, Laplacian for scale selection)• Hessian Laplace MultiscaleHarris MultiscaleHessian
  13. 13. Different cornerness measures 13 frames = vl_covdet(imgs, method, MultiscaleHarris) ; MultiscaleHarris MultiscaleHessian• Multiscale Harris (just a bunch of Harris)• Multiscale Hessian
  14. 14. Affine adaptation 14frames = vl_covdet(imgs, EstimateAffineShape, true) ;
  15. 15. Upright features 15 By default the rotation is selected so that the vertical axis stays vertical. In formulas    x a11 0 b x t1 = + y a21 a22 b y t2Remark: for general affine transformationsthe horizontal direction may appear rotated even for upright features.
  16. 16. Orientation detection 16frames = vl_covdet(imgs, EstimateOrientations, true) ;
  17. 17. Computing descriptors 17 [frames, descrs] = vl_covdet(imgs) ;• Each column of descrs is a stacked SIFT descriptor• How is a SIFT descriptor associated to a feature?• What about other descriptors?
  18. 18. Computing descriptors 18[frames, descrs] = vl_covdet(imgs, Descriptor, patch) ; Each column of descrs is a stacked normalised image patch.
  19. 19. The geometry of an normalised patch 19• Normalisation and patch resampling is controlled by three parameters: (antialiasing effect) affine tf. (size of patch in normalised units) (number of pixels)
  20. 20. Example: normalised patch for SIFT 20 =1 1 3 6 f v w = 15
  21. 21. Custom frames 21Example: running affine adaptation on a grid frames = vl_covdet(imgs, ... Frames, frames, ... EstimateAffineShape, true, ... EstiamteOrientation, true) ;
  22. 22. Obtaining additional information on the features 22 [frames, descrs, info] = vl_covdet(imgs); info = gss: [1x1 struct] css: [1x1 struct] peakScores: [1x351 single] edgeScores: [1x351 single] orientationScores: [1x351 single] laplacianScaleScores: [1x351 single] 120 120 100 100 80 80 Occurences Occurences 60 60 40 40 20 20 0 0 −0.1 −0.05 0 0.05 0.1 0 2 4 6 8 10 Peak Score Edge Score
  23. 23. Obtaining the scale space 23 [frames, descrs, info] = vl_covdet(imgs); info = gss: [1x1 struct] Gaussian css: [1x1 struct] scale space peakScores: [1x351 single] edgeScores: [1x351 single] orientationScores: [1x351 single] laplacianScaleScores: [1x351 single]octave −1 octave 0 octave 1octave 2 octave 3
  24. 24. Obtaining the scale space 24 [frames, descrs, info] = vl_covdet(imgs); info = gss: [1x1 struct] cornerness css: [1x1 struct] measure peakScores: [1x351 single] edgeScores: [1x351 single] orientationScores: [1x351 single] laplacianScaleScores: [1x351 single]octave −1 octave 0 octave 1octave 2 octave 3
  25. 25. Performance comparison 25 Oxford 5K lite retrieval benchmark Num. features / image mAPVL HessianLaplace (double) VL HessianLaplace (double) 7131 79.9% VL DoG (double) 5631 DoG (double) VL 85.1% VL Hessian (double) 3411 VL Hessian (double) 77.8% VL DoG (no aff) 2794 VL DoG (no aff) 74.0% VL HarrisLaplace (double) 2316 VL HarrisLaplace (double) 77.8% CMP Hessian 2292 CMP Hessian 77.1% VL HessianLaplace 2020 VL HessianLaplace 73.8% VL DoG 1854 VL DoG 80.1% VGG hes 1517 VGG hes 74.6% VGG har 1384 VGG har 71.9% VL Hessian 1089 VL Hessian 68.7% VL HarrisLaplace 765 VL HarrisLaplace 70.4% 0 2000 4000 6000 8000 0.6500 0.7125 0.7750 0.8375 0.9000
  26. 26. Repeatability: viewpoint 26 Repeatability (graf) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (graf) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 30 40 50 60 20 Viewpoint angle
  27. 27. Repeatability: viewpoint 27 Repeatability (wall) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (wall) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 30 40 50 60 20 Viewpoint angle
  28. 28. Repeatability: scale 28 Repeatability (boat) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (boat) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 1.38 1.9 2.35 2.8 1.12 Scale changes
  29. 29. Repeatability: JPEG compression 29 Repeatability (ubc) 100 VL DoG VL Hessian 90 VL HessianLaplace VL HarrisLaplace 80 VL DoG (double) VL Hessian (double) 70 VL HessianLaplace (double)Repeatability (ubc) VL HarrisLaplace (double) 60 VLFeat SIFT CMP Hessian 50 VGG hes VGG har 40 30 20 10 0 80 90 95 98 60 JPEG compression %
  30. 30. Maximally Stable Extremal Regions (MSER) 30 [Matas et al. 2002]• Extracts MSER regions regions and fitted ellipses frames: imgs = uint8(rgb2gray(I)) ; [regions,"frames]"="vl_mser(imgs)";• Control region stability: [regions,"frames]"="vl_mser(imgs,"... """"""""""""""""""""""""""""‘MinDiversity’,"0.7,"... """"""""""""""""""""""""""""‘MaxVariation’,"0.2,"... """"""""""""""""""""""""""""‘Delta’,"10)";
  31. 31. Many other features and tools 31 superpixels MSER agglomerative information bottleneck distance transform di erent mode seeking algo- sent (some of) the data points nal to the Parzen density esti-wards the mode approximately ates mean shift trajectories byxt and in Fig. 2, medoid shifts circles. This disconnects por- alleviated (but not solved) byect. 3) seeks the energy modes s, trading-o mode over- and randomized kd-trees fast k-means feature maps, stochastic gradient SVMs, where d is the dimension- is only O(dN 2 T ), where T Fast Dense SIFT lever implementations yield
  32. 32. Many other features and tools 32 SLIC superpixels Histogram of Oriented Gradients (HOG) HARVEST ProgrammeNew SGD Learning of SVMs
  33. 33. OpenCV 33• The largest, best known vision library• Features - OpenCV 2.1: Harris, Tomasi&Shi, MSER, DoG, SURF, STAR and FAST, DoG + SIFT, SURF, HARRIS, GoodFeaturesToTrack - OpenCV 2.4: ORB, BRIEF, and FREAK Example: computing ORB features. #include <opencv2/core/core.hpp> #include <opencv2/features2d/features2d.hpp> #include <vector> Mat image = importImage8UC1(in[IN_I]); Mat descriptors; std::vector<KeyPoint> frames; ORB orb(numFeatures, scaleFactor, numLevels, edgeThreshold, 0, wtak, scoreTypeToORBPar(scoreType), patchSize); orb(image, Mat(), frames);
  34. 34. This is not all folks 34• Abundance of open source / free software for CV - Always check out authors home pages• Examples Rob Hess SIFT SURF http://robwhess.github.com/opensift/ http://www.vision.ee.ethz.ch/~surf/ Open SURF http://www.chrisevansdev.com/computer-vision-opensurf.html FAST corners http://www.edwardrosten.com/work/fast.html LIOP descriptors http://vision.ia.ac.cn/Students/bfan/index.htm ...
  35. 35. Credits 35 Karel Lenc Daniele Perrone Michal Perdoch HARVEST ProgrammeKrystian Mikolajczyk Tinne Tuytelaars Jiri Matas Cordelia Schmid Andrew Zisserman
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×