https://imatge.upc.edu/web/publications/video-retrieval-specific-persons-specific-locations This thesis explores good practices for improving the detection of specific people in specific places. An approach combining recurrent and convolutional neural network have been considered to perform face detection. However, other more conventional methods have been tested, obtaining the best results by exploiting a deformable part model approach. A CNN is also used to obtain the face feature vectors and, with the purpose of helping in the face recognition, an approach to perform query expansion has been also developed. Furthermore, in order to be able to evaluate the different configurations in our non-labelled dataset, a user interface has been used to annotate the images and be able to obtain a precision of the system. Finally, different fusion and normalization strategies has been explored with the aim of combining the scores obtained from the face recognition with the ones obtained in the place recognition.

1. VIDEO RETRIEVAL OF SPECIFIC
PERSONS IN SPECIFIC LOCATIONS
AUTHOR: ADVISORS:
Xavier Giró-i-Nieto
Eva Mohedano Kevin McGuinness
Andrea Calafell
Noel E. O’Connor
1

2. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
2

3. MOTIVATION
SURVEILLANCE PERSONAL VIDEO ORGANIZATION
3

4. TRECVID INSTANCE SEARCH 2016
PEOPLE AND LOCATION QUERY SET
Person
visual
examples
Binary
masks
Location
visual
examples
TARGET
DATABASE
1.5M keyframes
244 video files
(300GB)
4

5. MOTIVATION: goals
● Obtain a baseline to participate in
TRECVID Instance Search 2016 (July, 1).
● Improve the results obtained in TRECVID
using the baseline.
5

6. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
6

7. STATE OF THE ART
Image of Eva Mohedano, D3L6 Image Retrieval, Deep Learning for Computer Vision (UPC 2016)
BASIC RETRIEVAL PIPELINE:
7

8. STATE OF THE ART
Image of Eva Mohedano, D3L6 Image Retrieval, Deep Learning for Computer Vision (UPC 2016)
BAG OF VISUAL WORDS:
8

9. STATE OF THE ART
Image: Alex Krizhevsky , Ilya Sutskever , Geoffrey E. Hinton, Imagenet classification with deep convolutional neural networks, 2012
Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. A baseline for visual instance retrieval with deep convolutional networks. ICLR 2015.
CNN REPRESENTATION:
9

10. STATE OF THE ART
Eva Mohedano, Amaia Salvador, Kevin McGuinness, Ferran Marqués, Noel E. O’Connor, and Xavier Giró i Nieto. Bags of local convolutional features for scalable
instance search. ICMR 2016.
BAG OF LOCAL CONVOLUTIONAL FEATURES:
10

11. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
11

12. FRAMEWORK FOR TRECVID
12

13. FRAMEWORK FOR TRECVID
13
Mohedano, et al. ICMR 2016

14. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
14

15. FRAMEWORK FOR TRECVID
15

16. FACE DETECTION: ReInspect
Russell Stewart, Mykhaylo Andriluka, and Andrew Y. Ng. End-to-end people detection in crowded scenes. CVPR 2016.
16

17. FACE DETECTION: ReInspect
QUALITATIVE RESULTS OF REINSPECT:
Changing both the input size of the network and the image size
Changing only the image size
17
Bad detections
False negatives

18. FACE DETECTION: ReInspect
PROBLEM: Images used to train ReInspect
18

19. FACE DETECTION: Menpo
1
https://github.com/menpo/menpodetect
Python wrapper for face detectors1
:
● DLIB
● OPENCV
● Pixel Intensity Comparison-based
Object detection (PICO)
● FFLD2
:
○ Based on Deformable Part
Models (DPM)
○ Use LUV color space
2
M. Mathias, R. Benenson, M. Pedersoli, and L. Van Gool. Face detection without bells and whistles. ECCV, 2014.
Examples of FFLD results
19

20. QUALITATIVE RESULTS OF MENPO:
DLIB
FACE DETECTION: Menpo
OPENCV
PICO 20
False negatives

21. QUALITATIVE RESULTS OF MENPO:
FACE DETECTION: Menpo
FFLD
Still some false negatives Solution: Equalize image
21

22. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
22

23. DEEP FACE RECOGNITION
FACE REPRESENTATION
Image: Simonyan, Karen, and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR 2015.
O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. BMVC 2015
VGG 16-layer
23

24. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
24

25. QUERY EXPANSION
Sequence of keyframes of one shot
dilate
Mask creation pipeline
25
TEMPORAL QUERY EXPANSION:

26. Results of temporal query expansion
26
QUERY EXPANSION
TEMPORAL QUERY EXPANSION:

27. 27
QUERY EXPANSION
PSEUDO-RELEVANCE FEEDBACK QUERY EXPANSION:
Top 20 retrieved keyframes

28. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
28

29. ANNOTATION TOOL
3.991 shots
for persons
1.528 shots
for locations
794 shots
in common
29

30. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
30

31. FRAMEWORK FOR TRECVID
31

32. FUSION AND NORMALIZATION STRATEGIES
NORMALIZATION:
● Z-score:
● Max-min:
● Extreme Value Theory:
FUSION:
Linear combination, maximum, minimum.
32

33. FUSION AND NORMALIZATION STRATEGIES
RESULTS OF APPLYING DIFFERENT NORMALIZATIONS:
BASELINE 33

34. FUSION AND NORMALIZATION STRATEGIES
Brad Person distribution Laundrette Location distribution
34
EXAMPLE DISTRIBUTION:

35. FUSION AND NORMALIZATION STRATEGIES
RESULTS OF APPLYING MAXIMUM OR MINIMUM FUSION
35

36. FUSION AND NORMALIZATION STRATEGIES
RESULTS OF WEIGHTING LINEAR COMBINATION
HIGHER THAN THE BASELINE
36

37. 1. Motivation
2. State of the art
3. Framework for TRECVID
4. Face detection
5. Face representation
6. Query expansion
7. Annotation Tool
8. Fusion and normalization strategies
9. Conclusions and future work
OUTLINE
37

38. CONCLUSIONS
● FFLD, a simple approach using vanilla DPM combined with image equalization is the best option for TRECVID
dataset
38

39. CONCLUSIONS
● The temporal query expansion proposed works well, but the faces are very similar between them
However, using the top 20 faces in the ranking as new queries gives more diverse faces.
39

40. CONCLUSIONS
● An annotation tool is needed in order to obtain quantitative results.
3.991 shots
for persons
1.528 shots
for locations
794 shots
in common
TOTAL OF RELEVANT ANNOTATED SHOTS
● The best configuration is without applying normalization and combining the scores by weighting higher the
location ranking
40

41. FUTURE WORK
● Analyze deeper the location part
● Try to improve the location rankings
41

42. QUESTIONS?
42

43. FUSION AND NORMALIZATION STRATEGIES
RESULTS OF THE PARTS SEPARATELY OVER 50 KEYFRAMES
43

44. FUSION AND NORMALIZATION STRATEGIES
RESULTS OF APPLYING MAXIMUM, MINIMUM AND PRODUCT FUSION
44