발표자: 김인식, 최유경 (NAVER) 발표일: 2018.7. CVPR 2018에서 발표된 논문 중 Visual Search (Image Retrieval) and Metric Learning에 대해 설명합니다.

1. CVPR 2018 Review
In Perspective of Image Retrieval
Naver
Vision / Insik Kim
Vision / YuKyung Choi
More complete review (internal only): https://oss.navercorp.com/insikk/public/issues/2

2. Contents
• Background on Image Retrieval
• Interesting “CVPR 2018” papers
• Image Embedding (Metric Learning)
• Similarity Search

3. Background

4. Image Retrieval
• Application: Naver SmartLens (스마트렌즈)
• Search Query as Image
• Find relevant information
• Products
• Flowers, Plants
• Clothes
• Lotto
• Food

5. What people do
Similarity Search System
Approximate Nearest Neighbor Search
PQ Encoding
Query Expansion and Reranking (RANSAC)
Image Embedding
Representation Learning
Metric Learning

6. What people do
• State-of-the-art Image Retrieval Models use end-to-end image
representation learning

7. What people do
• Representation Learning
• Feature Aggregation
• Or, Learn everything in
End-to-end

8. Who are they?
• Diane Larlus (Naver Labs Europe)
• Hervé Jégou (Facebook AI Research)
• Ondrej Chum (VRG Group, Czech Technical University in Prague)
• Other groups
• Inria
• Alibaba
• SenseTime

9. Task & Dataset
Task Dataset
Instance Retrieval (Retail Product) INSTRE
Instance Retrieval (Building, Landmark Retrieval) ROxford, RParis,
Google Landmark (DeLF paper)
Instance Retrieval (Fashion Items) DeepFashion
Person Re-identification Market-1501, DukeMTMC
• We can also get ideas from
• Face verification, recognition
• Fine grained visual classification (FGVC)
• Of course from general computer vision fields
• Detection, Classification, Segmentation
• Image Matching

10. Image Embedding
• Image -> n-dim feature vector
• 𝐼 → 𝜙 𝐼
• Answer to how similar given two
images are?
• 𝑆 𝜙 𝐼 𝑝 , 𝜙 𝐼 𝑞
• Cosine similarity
• Learned similarity function
• Let’s apply deep learning
• Deep Metric Learning
ImageNet Feature from AlexNet, tSNE visualization

11. Metric Learning

12. Metric Learning
Idea:
• Additional constraint
• Exploit mini-batch
• CRF to model group consistency
• Estimate a pair of image
similarity from group
consistency
Effect:
• Achieve SOTA (State-of-the-art)
on Market-1501, DukeMTMC

13. Metric Learning
Idea:
• Triplet Loss with latent
examples
• Learning latent example
Effect:
• Robust to noise
• Faster training

14. Metric Learning
Idea:
• Hard positive, negative image mining
• Density information (Random Walk)
Effect:
• Good for triplet loss training

15. Metric Learning
Idea:
• Unsupervised feature learning
with discrimination
• N training example, N – way
softmax classification
• Direct comparison with dot-
product
Effect:
• Semi-Supervised learning
(use less labeled data)

16. Metric Learning
Idea:
• Unsupervised feature learning
with discrimination
• N training example, N – way
softmax classification
• Direct comparison with dot-
product
Effect:
• Semi-Supervised learning
(use less labeled data)

17. Similarity Search
Finding nearest neighbors from 1,000,000,000 images (1B scale)

18. What people do

19. Inverted File, Multi Index, Product Quantization
TPAMI 2010 CVPR 2012

20. Similarity Search
Idea:
• Learning coarse quantizer
in supervised manner
• Training requires
classification label
Effect:
• Learned binning works
better than unsupervised
one (e.g. IMI)

21. Similarity Search
Idea:
• Learning coarse quantizer
in supervised manner
• Training requires
classification label
Effect:
• Learned binning works
better than unsupervised
one (e.g. IMI)

22. Similarity Search
https://github.com/facebookresearch/faiss
CoRR 2016
Idea:
• Local descriptor. 1000 features / image. 8 bytes / feature.
=> Global descriptor. 1 feature / image. 128 bytes / feature
• HNSW (1M scale)
• Find nn with less comparison
• Compact representation with novel regression
Effects:
• HNSW 1M -> 1B scale

23. Similarity Search
https://github.com/facebookresearch/faiss
Idea:
• Local descriptor. 1000 features / image. 8 bytes / feature.
=> Global descriptor. 1 feature / image. 128 bytes / feature
• HNSW (1M scale)
• Find nn with less comparison
• Compact representation with novel regression
Effects:
• HNSW 1M -> 1B scale