An overview and look at the state-of-the-art in Deep Fine-Grained Image Classification.

1. Deep Fine-Grained
Image Classification
Overview and state-of-the-art
Carmine Paolino & Hagop Boghazdeklian, OLX

2. 2
Carmine Paolino
MSc in Artificial Intelligence at Vrije Universiteit Amsterdam
Data Scientist at OLX (Search and Recommendations)
carmine.paolino@olx.com
@paolino
http://paolino.me

3. 3
Hagop Boghazdeklian
MSc in DS and Econometrics from Aix-Marseille University
Data Scientist at OLX (Fraud, Trust and Safety)
hagop.boghazdeklian@olx.com

4. Offline OLX

5. 5
Deep
Fine-Grained
Image
Classification
● Introduction
● Overview of Approaches
● Standard Datasets
● State-of-the-art

6. Introduction

7. 7
Are these cars the same model/year?
Introduction

8. 8
Are these cars the same model/year?
Introduction

9. 9
Are these cars the same model/year?
Introduction
YES: BMW Series 6 Convertible 2012
High intra-class variance

10. 10
Are these cars the same model/year?
Introduction
Low inter-class variance
NO: BMW 2 series 2016 BMW 2 series 2017 BMW 2 series 2018

11. 11
Discriminating features
Introduction
BMW Series 6 Convertible 2012
High intra-class variance

12. 12
Discriminating features
Introduction
Low inter-class variance
BMW 2 series 2016 BMW 2 series 2017 BMW 2 series 2018

13. Overview of Approaches

14. 14
Deep Neural Networks
Overview of approaches
❖ Basic and pragmatic approach
❖ Two-steps classification
➢ Object detection (bounding-boxes)
➢ Classification of cropped images
❖ Increase of robustness in recent CNN architectures
❖ Transfer Learning, Data Augmentation and Fine-Tuning

15. 15
Deep Neural Networks
Overview of approaches. Valev, Schuman and Sommer
A Systematic Evaluation of Recent Deep Learning Architectures for Fine-Grained Vehicle Classification (2018)

16. 16
Part-detection and alignment
Overview of approaches
Part-detection Templates Pose alignment Classification
y

17. 17
Example of pose-normalized part-CNNs
Figure 3: Overview of the part-CNN pipeline on pose normalized parts
part-CNN
Overview of approaches. Source: Branson, Van Horn, Belogie, Peronize.
Bird species categorization using prose normalized deep convolutional nets (2014)

18. 18
Ensemble of networks
Overview of approaches. Lin, RoyChowdhury, Maji. Bilinear CNN models for fine-grained classification (2015)

19. 19
Attention mechanism
Overview of approaches. Zheng Fu, Luo. Learning Multi-Attention Convolutional Neural Network for Fine-Grained Image Recognition (2017)

20. Standard Datasets

21. 21
Caltech-UCSD Birds-200-2011 (CUB-200-2011)
Standard Datasets
200
Categories
15 Part Locations
312 Binary
Attributes
1 Bounding Box
Per image
11,788
Images

22. 22
Stanford Cars
Standard Datasets
196
Categories
1 Bounding Box
3 Labels: Make,
Model, Year
Per image
16,185
Images

23. 23
Fine-Grained Visual Classification of Aircraft
(FGVC-Aircraft)
Standard Datasets
102
Categories
1 Bounding Box
4 Labels:
Manufacturer,
Family, Variant,
Model
Per image
10,200
Images

24. State-of-the-art

25. 25
Classical approaches
State-of-the-art: H. Zheng, J. Fu, T. Mei, and J. Luo, “Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition,” Proc. IEEE
Int. Conf. Comput. Vis., vol. 2017–October, pp. 5219–5227, 2017.
Part-detection Templates Pose alignment Classification
y

26. 26
Multi-attention
State-of-the-art: H. Zheng, J. Fu, T. Mei, and J. Luo, “Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition,” Proc. IEEE
Int. Conf. Comput. Vis., vol. 2017–October, pp. 5219–5227, 2017.
Highest activations
VGG-19
Grouping
Candidate part attentions

27. 27
Final classification
State-of-the-art: H. Zheng, J. Fu, T. Mei, and J. Luo, “Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition,” Proc. IEEE
Int. Conf. Comput. Vis., vol. 2017–October, pp. 5219–5227, 2017.

28. 28
Part-CNN for final classification
Figure 3: Overview of the part-CNN pipeline on pose normalized parts
State-of-the-art. Source: Branson, Van Horn, Belogie, Peronize.
Bird species categorization using prose normalized deep convolutional nets (2014)
Region zooming

29. 29
Joint learning
State-of-the-art: H. Zheng, J. Fu, T. Mei, and J. Luo, “Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition,” Proc. IEEE
Int. Conf. Comput. Vis., vol. 2017–October, pp. 5219–5227, 2017.

30. Thank you!
H. Zheng, J. Fu, T. Mei, and J. Luo, “Learning
Multi-attention Convolutional Neural Network for
Fine-Grained Image Recognition,” Proc. IEEE Int. Conf.
Comput. Vis., vol. 2017–October, pp. 5219–5227, 2017.
B. Zhao, J. Feng, X. Wu, and S. Yan, “A survey on deep
learning-based fine-grained object classification and
semantic segmentation,” Int. J. Autom. Comput., vol.
14, no. 2, pp. 119–135, 2017.

31. 31