Modern features-part-2-descriptors


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Modern features-part-2-descriptors

  1. 1. Descriptors Cordelia Schmid Tinne TuytelaarsKrystian Mikolajczyk
  2. 2. Overview – descriptors•  Introduction•  Modern descriptors•  Comparison and evaluation
  3. 3. Descriptors Eliminate rotational Compute appearanceExtract affine regions Normalize regions descriptors + illumination SIFT (Lowe ’04)
  4. 4. Descriptors - history•  Normalized cross-correlation (NCC) [~ 60s]•  Gaussian derivative-based descriptors –  Differential invariants [Koenderink and van Doorn’87] –  Steerable filters [Freeman and Adelson’91]•  Moment invariants [Van Gool et al.’96]•  SIFT [Lowe’99]•  Shape context [Belongie et al.’02]•  Gradient PCA [Ke and Sukthankar’04]•  SURF descriptor [Bay et al.’08]•  DAISY descriptor [Tola et al.’08, Windler et al’09]•  …….
  5. 5. SIFT descriptor [Lowe’99]•  Spatial binning and binning of the gradient orientation•  4x4 spatial grid, 8 orientations of the gradient, dim 128•  Soft-assignment to spatial bins•  Normalization of the descriptor to norm one (robust to illumination)•  Comparison with Euclidean distance image patch gradient x → → y
  6. 6. Local descriptors - rotation invariance•  Estimation of the dominant orientation –  extract gradient orientation –  histogram over gradient orientation –  peak in this histogram 0 2π•  Rotate patch in dominant direction The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.•  Plus: invariance•  Minus: less discriminant, additional noise
  7. 7. Evaluation•  Descriptors should be –  Distinctive (! importance of the measurement region) –  Robust to changes on viewing conditions as well as to errors of the detector –  Compactness, speed of computation•  Detection rate (recall) 1 –  #correct matches / #correspondences•  False positive rate –  #false matches / #all matches•  Variation of the distance threshold –  distance (d1, d2) < threshold 1 [K. Mikolajczyk & C. Schmid, PAMI’05]
  8. 8. Viewpoint change (60 degrees) Detector:* * Hessian-Affine
  9. 9. Scale change (factor 2.8) Detector:* * Hessian-Affine
  10. 10. Evaluation - conclusion•  SIFT based descriptors perform best•  Significant difference between SIFT and low dimension descriptors as well as cross-correlation•  Robust region descriptors better than point-wise descriptors•  Performance of the descriptor is relatively independent of the detector
  11. 11. Recent extensions to SIFT•  Color SIFT [Sande et al. 2010]•  Normalizing SIFT with square root transformation [Arandjelovic et Zisserman’12]
  12. 12. Descriptors – dense extraction•  Many conclusions for descriptors applied to sparse detectors also hold for dense extraction –  For example normalizing SIFT with the square root improves in image retrieval and classification•  Image retrieval: sparse versus dense SIFT + Fisher vector for retrieval on the Holidays dataset Hessian-Affine MAP = 0.54 Dense MAP = 0.62
  13. 13. Overview – descriptors•  Introduction•  Modern descriptors•  Comparison and evaluation
  14. 14. Overview•  Introduction•  Modern descriptors•  Comparison and evaluation
  15. 15. Modern  descriptors  •  Efficient  descriptors  •  Compact  binary  descriptors  •  More  robust  descriptors  •  Learned  descriptors  
  16. 16. DAISY   Cita%ons:     •  Op<mized  for  dense  sampling   150    (2012)   •  Log-­‐polar  grid   •  Gaussian  smoothing   •  Dealing  with  occlusions  Engin  Tola,  Vincent  Lepe<t,  and  Pascal  Fua,  DAISY:  An  Efficient  Dense  Descriptor  Applied  to  Wide-­‐Baseline  Stereo,  TPAMI  32(5),  2010.      
  17. 17. SURF:   Speeded  Up  Robust  Features   •  Approximate  deriva<ves  with  Haar  wavelets   •  Exploit  integral  images   Cita%ons:     4500    (2012)  Herbert  Bay,  Andreas  Ess,  Tinne  Tuytelaars,  Luc  Van  Gool  "SURF:  Speeded  Up  Robust  Features",  Computer  Vision  and  Image  Understanding  (CVIU),  Vol.  110,  No.  3,  pp.  346-­‐-­‐359,  2008  
  18. 18. SURF:   Speeded  Up  Robust  Features  •  Orienta<on  assignment  
  19. 19. U-­‐SURF:   Upright  SURF  •  Rota<on  invariance  is  o`en  not  needed.       Don’t  use  more  invariance       than  needed  for  a  given  applica<on      •  Orienta<on  es<ma<on  takes  <me.  •  Orienta<on  es<ma<on  is  o`en  a  source  of   errors.    
  20. 20. Beyond  the  classics  •  Efficient  descriptors  •  Compact  binary  descriptors  •  More  robust  descriptors  •  Learned  descriptors  
  21. 21. Fast  and  compact  descriptors  •  (Very)  large  scale  applica<ons   –  >  memory  issues,  computa<on  <me  issues  •  Mobile  phone  applica<ons  
  22. 22. Fast  and  compact  descriptors  •  Binary  descriptors  •  Comparison  of  pairs  of  intensity  values    -­‐  LBP    -­‐  BRIEF    -­‐  ORB    -­‐  BRISK    
  23. 23. LBP:   Local  Binary  Paeerns   •  First  proposed  for  texture  recogni<on  in  1994.   Cita%ons:     2500    (2012)  T.  Ojala,  M.  Pie<käinen,  and  D.  Harwood  (1994),  "Performance  evalua<on  of  texture  measures  with  classifica<on  based  on  Kullback  discrimina<on  of  distribu<ons",  ICPR  1994,  pp.582-­‐585.  M  Heikkilä,  M  Pie<käinen,  C  Schmid,  Descrip<on  of  interest  regions  with  LBP,  Paeern  recogni<on  42  (3),  425-­‐436  
  24. 24. BRIEF:   Binary  Robust  Independent   Elementary  Features   •  Random  selec<on  of  pairs   of  intensity  values.   •  Fixed  sampling  paeern   of  128,  256  or  512  pairs.   •  Hamming  distance  to     compare  descriptors  (XOR).   Cita%ons:     149    (2012)  M.  Calonder,  V.  Lepe<t,  C.  Strecha,  P.  Fua,  BRIEF:  Binary  Robust  Independent  Elementary  Features,  11th  European  Conference  on  Computer  Vision,  2010.  
  25. 25. D-­‐BRIEF:   Discrimina<ve  BRIEF     •  Learn  linear  projec<ons  that  map  image   patches  to  a  more  discrimina<ve  subspace   •  Exploit  integral  images  T.  Trzcinski  and  V.  Lepe<t,  Efficient  Discrimina<ve  Projec<ons  for  Compact  Binary  Descriptors  European  Conference  on  Computer  Vision  (ECCV)  2012  
  26. 26. ORB:   Oriented  FAST  and  Rotated  BRIEF   •  Add  rota<on  invariance  to  BRIEF   Cita%ons:     43    (2012)   •  Orienta<on  assignment  based     on  the  intensity  centroid    Ethan  Rublee,  Vincent  Rabaud,  Kurt  Konolige,  Gary  Bradski,  ORB:  an  efficient  alterna<ve  to  SIFT  or  SURF,  ICCV  2011  
  27. 27. ORB:   Oriented  FAST  and  Rotated  BRIEF  •  Select  a  good  set  of  pairwise  comparisons:   minimize  correla<on  under  various   orienta<on  changes   High  variance     High  variance  +  uncorrelated    
  28. 28. BRISK:     Binary  Robust  Invariant  Scalable   Keypoints   Cita%ons:     •  Regular  grid       20    (2012)   •  Orienta<on  assignment  based  on  dominant   gradient  direc<on  (using  long-­‐distance  pairs)   •  512  bit  descriptor  based  on   short-­‐distance  pairs  Stefan  Leutenegger,  Margarita  Chli  and  Roland  Y.  Siegwart,  BRISK:  Binary  Robust  Invariant  Scalable  Keypoints,  ICCV  2011  
  29. 29. Various  others  •  FREAK:  Fast  Re<na  Keypoint  •  CARD:  Compact  and  Real<me  Descriptor  •  LDB:  Local  Difference  Binary      
  30. 30. FREAK:     Fast  Re<na  Keypoint       •  Inspired  by  the  human  visual  system  A.  Alahi,  R.  Or<z,  and  P.  Vandergheynst.  FREAK:  Fast  Re<na  Keypoint.  In  IEEE  Conference  on  Computer  Vision  and  Paeern  Recogni<on  2012.  
  31. 31. CARD:    Compact  and  Real<me  Descriptor  –  Look  up  tables    –  learning-­‐based  sparse  hashing  
  32. 32. LDB:   Local  Difference  Binary  Xin  Yang  and  Kwang-­‐Ting  Cheng,  LDB:  An  Ultra-­‐Fast  Feature  for  Scalable  Augmened  Reality  on  Mobile  Devices,  Interna<onal  Symposium  on  Mixed  and  Augmented  Reality  2012  (ISMAR  2012)  
  33. 33. Beyond  the  classics  •  Efficient  descriptors  •  Compact  binary  descriptors  •  More  robust  descriptors  •  Learned  descriptors  
  34. 34. LIOP:   Local  Intensity  Order  Paeern  for   Feature  Descrip<on   (and  predecessors  MROGH  and  MRRID)   •  Robustness  to  monotonic  intensity  changes   •  Data-­‐driven  division  into  cells  Zhenhua  Wang  Bin  Fan  Fuchao  Wu,  Local  Intensity  Order  Paeern  for  Feature  Descrip<on,  ICCV  2011.  
  35. 35. LIOP:  Local  Intensity  Order  Paeern  for   Feature  Descrip<on  
  36. 36. Beyond  the  classics  •  Efficient  descriptors  •  Compact  binary  descriptors  •  More  robust  descriptors  •  Learned  descriptors  
  37. 37. Winder  &  Brown   •  Learn  configura<on  and  other  parameters   from  training  data  obtained  from  3D   Cita%ons:     reconstruc<ons   194  (2012)  M.  Brown,  G.  Hua  and  S.  Winder,  Discriminant  Learning  of  Local  Image  Descriptors..  IEEE  Transac<ons  on  Paeern  Analysis  and  Machine  Intelligence.  2010.  
  38. 38. Winder  &  Brown  •  Training  data  =  set  of  corresponding  image   patches  
  39. 39. Descriptor  Learning     Using  Convex  Op<misa<on   •  Convex  learning  of   Pre-­‐rec<fied     keypoint  patch     –  spa<al  pooling  regions   –  dimensionality  reduc<on   Non-­‐linear  transform   •  Learning  from  very  weak     supervision   learning   Spa<al  pooling     Normalisa<on  and   cropping   Dimensionality   learning   reduc<on   Descriptor  vector  K.  Simonyan  et  al.,  Descriptor  Learning  Using  Convex  Op<misa<on,  ECCV  2012  
  40. 40. Learning  Spa<al  Pooling  Regions  •  Selec%on  from  a  large  pool  using  L1  regularisa%on  •  So`-­‐margin  constraints:     –  squared  L2  distance  between  descriptors  of  matching  feature   pairs  should  be  smaller  than  that  of  non-­‐matching  pairs  •  Convex  objec<ve  (op<mised  with  a  proximal  method)   768-­‐D   576-­‐D   320-­‐D   pooling  region  configura%ons  learnt  with  different  levels  of  sparsity  
  41. 41. Overview•  Introduction•  Modern descriptors•  Comparison and evaluation
  42. 42. Sepng  up  an  evalua<on  •  Which  problem?  Performance  in  different  applica<on/niches  may   vary  significantly.   –  Category  recogni<on,     –  Matching,     –  Retrieval    •  What  dataset?   –  Pascal  VOC  2007   –  Oxford  image  pairs   –  Oxford  -­‐  Paris  buildings  •  Protocol  and  criteria?   –  Public  dataset,     –  Avoiding  risk  to  over-­‐fipng/op<mizing  to  the  data    
  43. 43. Detector  evalua<ons   homography A B B Two points are correctly matched if # correct matches precision = T=40% # all matches A∩ B >T A∪ B # correct matchesrecall = # ground truth correspondences
  44. 44. Previous  Evalua<ons  •  2D  Scene  –  Homography   –  C.  Schmid,  R.  Mohr,  and  C.  Bauckhage,  “Evalua<on  of  interest  point   detectors,”  IJCV,  2000.   –  K.  Mikolajczyk  and  C.  Schmid,  “A  performance  evalua<on  of  local  descriptors,”   CVPR,  2003.   –  T.  Kadir,  M.  Brady,  and  A.  Zisserman,  “An  affine  invariant  method  for  selec<ng   salient  regions  in  images,”  in  ECCV,  2004.   –  K.  Mikolajczyk,  T.  Tuytelaars,  C.  Schmid,  A.  Zisserman,  J.  Matas,  F.   Schaffalitzky,T.  Kadir,  and  L.  Van  Gool,  “A  comparison  of  affine  region   detectors,”  IJCV,  2005.   –  A.  Haja,  S.  Abraham,  and  B.  Jahne,  Localiza<on  accuracy  of  region  detectors,   CVPR  2008     –  T.  Dickscheid,    FSchindler,  Falko,  W.  Förstner,  Coding  Images  with  Local   Features,  IJCV  2011    
  45. 45. Previous  Evalua<ons  •  3D  Scene  -­‐  epipolar  constraints   –  F.  Fraundorfer  and  H.  Bischof,  “Evalua<on  of  local  detectors  on  non-­‐planar,   scenes,”  in  AAPR,  2004.     –  P.  Moreels  and  P.  Perona,  “Evalua<on  of  features  detectors  and  descriptors   based  on  3D  objects,”  IJCV,  2007.   –  S.  Winder  and  M.  Brown,  “Learning  local  image  descriptors,”  CVPR,   2007,2009.   –  Dahl,  A.L.,  Aanæs,  H.  and  Pedersen,  K.S.  (2011):  Finding  the  Best  Feature   Detector-­‐Descriptor  Combina<on.  3DIMPVT,  2011.      
  46. 46. Recent  Evalua<ons  •  Recent  detectors  O.  Miksik  and  K.  Mikolajczyk,  Evalua<on  of  Local  Detectors  and  Descriptors  for  Fast  Feature  Matching,  ICPR  2012  ECCV  2012    Modern  features:    46/60  …  Detectors.  
  47. 47. Recent  detector  evalua<ons  •  Completeness  (coverage),  complementarity  between   detectors     T.  Dickscheid,    FSchindler,  Falko,  W.  Förstner,  Coding  Images  with  Local  Features,  IJCV  2011   –  Completeness,  Edgelap  (Mikolajczyk),  Salient  (Kadir),  MSER  (Matas)   –  Complementrity,  MSER  +  SFOP    
  48. 48. Descriptor  Evalua<ons  •  Matching  precision  and  recall  O.  Miksik  and  K.  Mikolajczyk,  Evalua<on  of  Local  Detectors  and  Descriptors  for  Fast  Feature  Matching,  ICPR  2012  
  49. 49. Recent  Descriptor  Evalua<ons  •  Computa%on  %mes  for  the  different  descriptors  for  1000   SURF  keypoints  O.  Miksik  and  K.  Mikolajczyk,  Evalua<on  of  Local  Detectors  and  Descriptors  for  Fast  Feature  Matching,  ICPR  2012  J.  Heinly  E.  Dunn,  J-­‐M.  Frahm,  Compara<ve  Evalua<on  of  Binary  Features,  ECCV2012    
  50. 50. Previous  Evalua<ons  •  Image/object  categories   –  K.  Mikolajczyk,  B.  Leibe,  and  B.  Schiele,  “Local  features  for  object  class   recogni<on,”  in  ICCV,  2005   –  E.  Seemann,  B.  Leibe,  K.  Mikolajczyk,  and  B.  Schiele,  “An  evalua<on  of  local   shape-­‐based  features  for  pedestrian  detec<on,”  in  BMVC,  2005.   –  M.  Stark  and  B.  Schiele,  “How  good  are  local  features  for  classes  of  geometric   objects,”  in  ICCV,  2007.   –  K.  E.  A.  van  de  Sande,  T.  Gevers  and  C.  G.  M.  Snoek,  Evalua<on  of  Color   Descriptors  for  Object  and  Scene  Recogni<on.  CVPR,  2008.    
  51. 51. Approach  •  Bags-­‐of-­‐features   1.  Interest  point  /  region  detector   2.  Descriptors   3.  K-­‐means  clustering  (4000  clusters)   4.  Histogram  of  cluster  occurrences  (NN  assignment)   5.  Chi-­‐square  distance  and  RBF  kernel  for  KDA  or  SVM  classifier     •   J.  Zhang  and  M.  Marszalek  and  S.  Lazebnik  and  C.  Schmid,     Local  Features  and  Kernels  for  Classifica<on  of  Texture  and  Object  Categories:     A  Comprehensive  Study,  IJCV,  2007   •   K.  E.  A.  van  de  Sande,  T.  Gevers  and  C.  G.  M.  Snoek,     Evalua<on  of  Color  Descriptors  for  Object  and  Scene  Recogni<on.  CVPR,  2008  
  52. 52. Evalua<on  •  PASCAL  VOC  measures   –  Average  precision  for  every  object  category   –  Mean  average  precision     Category   AP   output=>   precision   AP   recall  
  53. 53. MAP  Ranking   color/gray,  density,  dimensionality  ...   MAP   #dimensions   density  •  SIFT  s<ll  dominates  (Histograms  of  gradient  loca<ons  and  orienta<ons)  •  Opponent  chroma<c  space  (normalized  red-­‐green,  blue-­‐yellow,  and  intensity  Y  
  54. 54. Grayvalue  descriptors   MAP  Ranking   #dimensions   density    •  Observa<ons   •  Color  improves   •  All  based  on  histograms  of  gradient  loca<ons  and  orienta<ons   •  Dimensionality  not  much  correlated  with  the  performance   •  Density  Strongly  correlated  (the  more  the  beeer)   •  Results  biased  by  density   •  Implementa<on  details  maeer  
  55. 55. Break  !