Download as PDF, PPTX
Uploaded byDmytro Mishkin
HardNet: Convolutional Network for Local Image Description
The document presents HardNet, a convolutional network designed for local image description, along with a review of various methods for learning local descriptors, including L2-Net and others. It details the architecture and loss functions used, as well as training data and benchmarks from several datasets, highlighting the performance comparisons. Additionally, the paper discusses the techniques for hard-negative mining and various results achieved across different datasets.
More Related Content
Similar to HardNet: Convolutional Network for Local Image Description
![Human Reproduction [ Reproductive System ] Notes @irfanullah_mehar Irfanullah...](https://cdn.slidesharecdn.com/ss_thumbnails/humanreproductionreproductivesystemnotesirfanullahmeharirfanullahmeharjanantantra-260111172350-56e85778-thumbnail.jpg?width=640&height=640&fit=bounds)
HardNet: Convolutional Network for Local Image Description
- 1.
- 2.
- 3. 3 TRAINING DATA Discriminant Learning ofLocal Image Descriptors Brown et al, PAMI2010 3 sets, 400k patches each: • Liberty (shown) • Notredame • Yosemite Size: 64x64, grayscale. Obtained from SfM model, 3D point → DoG keypoints Used in all learned descriptors meantioned in this presentation
- 4. 4 CONVEXOPT (SIMONYAN ET AL, 2012) Global margin loss Simonyanet al, ECCV 2012 Convex optimization problem
- 5. 5 MATCHNET Han et al,CVPR2015. Works well, but rely on metric network. Approximate kNN methods, e.g. FLANN cannot be applied directly 7x7 Conv pad 1 64 24 ReLU 1 24 5x5 Conv pad 2 64 ReLU 64 3x3 Conv pad 1 ReLU 64 2x2 MP/2 32 32 2x2 MP/2 16 96 16 3x3 Conv pad 1 ReLU 96 16 3x3 Conv pad 1 ReLU 64 16 3x3 MP/2 64 8 8x8 Conv ReLU 1 128 1x1 Conv ReLU 1 256 1x1 Conv ReLU 1 256 1x1 Conv Softmax 1 2
- 6. 6 DEEPCOMPARE Zagoruyko and Komodakis,CVPR 2015 Works well, but rely on metric network. Approximate kNN methods, e.g. FLANN cannot be applied directly 7x7 Conv pad 3 64 96 ReLU 1 96 5x5 Conv pad 2 192 ReLU 192 3x3 Conv pad 1 ReLU 64 2x2 MP/2 32 32 2x2 MP/2 16 256 16 8 8x8 Conv ReLU 1 1 256 1x1 Conv ReLU 256 1x1 Conv Sigmoid 1 2x2 MP/2 256
- 7. 9 Simo-Serra et al,ICCV 2015. Balntas et al, BMVC 2016 32 32 7x7 Conv 26 TanH 1 2x2 MP/2 13 6x6 Conv TanH 8 64 8x8 Conv TanH 1 12832 TFeat (Balntas et al, 2016) • Even shallower and faster CNN, • hard-negative mining: by anchor swap in triplet. • triplet margin loss on L2 distance 1 64 7x7 Conv 58 32 TanH 2x2 L2pool/2 29 6x6 Conv TanH 23 64 5x5 Conv TanH 4 12832 3x3 L2Pool/3 8 64 4x4 L2Pool/4 1 128 DeepDesc (Simo-Serra et al, 2015 ) Relatively shallow and fast CNN. Hard negative mining: Contrastive loss on L2 distance
- 8. 10 DESCRIPTOR COMPARISON Descr. #layers w/params Loss Hardmining Kd-tree ready ConvexOpt 1 Global margin - + DeepDesc 3 Contrastive + + TFeat 3 Triplet margin +/- + MatchNet 8 Cross entropy - - DeepComp 5 Hinge - - Balntas et al, BMVC 2016
- 9. 11 L2NET. TIAN ET AL (CVPR 2017) 32 32 1616 3x3 Conv pad 1 32 32 BN + ReLU 1 3x3 Conv pad 1 32 BN + ReLU 3x3 Conv pad 1 /2 64 BN + ReLU 3x3 Conv pad 1 64 BN + ReLU 3x3 Conv pad 1 /2 BN + ReLU 8 128 3x3 Conv pad 1 BN + ReLU 8 128 8x8 Conv BN+ L2Norm 1 128
- 10. 13 L2NET: LOSS TERMS Softmax over row/columnof distance matrix
- 11. 14 L2NET: LOSS TERMS Softmax over row/columnof distance matrix Penalty on descriptor components correlation
- 12. 15 L2NET: LOSS TERMS Softmax over row/columnof distance matrix Softmax over row/column of distance matrix of intermediate features Penalty on descriptor components correlation
- 13. 16 HARDNET Triplet margin lossfor hard negative Penalty on descriptor channels correlation Softmax over row/column of distance matrix of intermediate features
- 14. 17 HARDNET (OURS) 3x3 Conv pad 1 32 32 BN+ ReLU 1 3x3 Conv pad 1 32 BN + ReLU 3x3 Conv pad 1 /2 64 BN + ReLU 3x3 Conv pad 1 64 BN + ReLU 3x3 Conv pad 1 /2 BN + ReLU 8 128 3x3 Conv pad 1 BN + ReLU 8 128 8x8 Conv BN+ L2Norm 1 128
- 15.
- 16. 19 DESCRIPTOR COMPARISON Descr. #layers w/params Loss Hardmining Kd-tree ready ConvexOpt 1 Global margin - + DeepDesc 3 Contrastive + + TFeat 3 Triplet margin +/- + MatchNet 8 Cross entropy - - DeepComp 5 Hinge - - L2Net 7 SoftMax + + HardNet 7 Triplet margin + +
- 17.
- 18.
- 19. 22 LOSSES COMPARISON, DERIVATIVES No gradient from negativeexample Small gradients
- 20. 23 LOSSES COMPARISON Contrastive Softmax (L2Net)Triplet margin FPR, Brown Yos 0.009 0.009 0.006 mAUC, W1BS 0.072 0.083 0.083 mAUC, HP-T 0.153 0.157 0.164
- 21.
- 22.
- 23. 26 RESULTS: W1BS DATASET Mishkin et al,BMVC 2015 Nuisance factor: Appearance Geometry Lighting Sensor
- 24. 27 HPATCHES DATASETDoG, Hessian, Harris– in ref.image ~1300 patches per image kept. Reprojected to other images with 3 levels of “affine frame noise” added V: 57 image sixplets – photometric changes I: 59 image sixplets – geometric changes Balntas et al, CVPR 2017
- 25.
- 26. 29 RESULTS: MATCHING WITH VIEW SYNTH Datasets are alreadysaturated On par with RootSIFT Still challenging due to multiple nuisance factors Zitnick and Ramnath, 2011, Mishkin et al 2015, Mikolajczyk et al. 2013, Hauagge and Snavely, 2012, Kelman et al, 2007, Fernando et al. 2014
- 27. 30 RESULTS: BOW OXFORD5K & PARIS 6K Philbin et al2007, Philbin et al 2008
- 28.
- 29.