More details: https://imatge.upc.edu/web/publications/visual-summary-egocentric-photostreams-representative-keyframes Author: Ricard Mestre Thesis advisor: Xavier Giró-i-Nieto This Final Degree Work approach the problem of the visual summarization of sets of images captured by an eggocentric camera for lifelogging purposes. In first place we try to group the images (which represent the day of person’s life) into distinguishable and significant events. For this purpose, we use visual features extracted with the software Caffe. In second place, we explain the design of extraction techniques of the representative images through similarity graphs. Finally we analyze the assessment scores given by different users whom we presented the different visual summaries obtained in this project. We achieve a 60% of favorable opinions of the quality of the visual summaries obtained with techniques developed in this project.

1. Visual Summary of Egocentric
Photostreams by Representative
Keyframes
Author: Ricard Mestre
Supervisor: Xavier Giró
Date: Tuesday, 17th of February 2015
1

2. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
● Conclusions and future work
2

3. Collaboration
Collaboration with UB group BCNPCL
(Barcelona Percepture Computer Laboratory)
3

4. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
● Conclusions and future work
4

5. Motivation and goals
● Lifelogging with Narrative
Clip
● Up to 2000 images/day
● A visual summary can help
the memory of Alzheimer
affected people
5

6. Motivation and goals
● Extract a visual summary
of a day
○ Clustering strategy for
event detection
○ Automatic selection of
representative frames
6

7. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
● Conclusions and future work
7

8. State of the art
Chandrasekar et al, “Efficient retrieval from large-scale egocentric visual data using a sparse graph
representation” (CVPR Workshop 2014)
8

9. State of the art
Lu and Grauman, ”Story-driven summarization for egocentric video” (CVPR 2013)
9

10. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
● Conclusions and future work
10

11. Methodology
Feature extraction Clustering Division-fusion
Keyframe
extraction
11

12. Feature extraction
● Convolutional Neural Networks (CNN) trained with
ImageNet.
12
Jia et al, “Caffe: Convolutional Architecture for Fast Feature Embedding” (ACM MM 2014)

13. Methodology
Feature extraction Clustering Division-fusion
Keyframe
extraction
13

14. Clustering
● Obtain separated events
● Agglomerative clustering
14
cutoff parameter
Talavera, E., Dimiccoli, M., Bolaños, M., Aghaei, M., & Radeva, P. (2015). “R-Clustering for Egocentric Video
Segmentation”. In 7th Iberian Conference on Pattern Recognition and Image Analysis (ACCEPTED).

15. Clustering: linkage method
● Different linkage methods
● Our case: average linkage
15

16. Methodology
Feature extraction Clustering Division-fusion
Keyframe
extraction
16

17. Division
● Long events with short events inside
● Groundtruth labelling
17
1 2 3

18. 18

19. Fusion
● Short clusters (less than 5 images) are not
representative
● Join the short events into larger ones
19
?
?

20. 20

21. Example of good segmentation
21

22. Example of good segmentation
22

23. Example of bad segmentation
23

24. Methodology
Feature extraction Clustering Division-fusion
Keyframe
extraction
24

25. Keyframe extraction
● Criterion: visual similarity-based keyframe
● Graph-based approach:
25Similarity Graph
Adjacency Matrix

26. Random walk
● One pedestrian moving along the graph
● The most visited the most representative
26

27. Minimum distance
● Adjacency matrix approach
● The minimum distance the most representative
27

28. Example of summary
28

29. Example of summary
29

30. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
○ Database
○ Clustering
○ Keyframe extraction
● Conclusions and future work
30

31. Evaluation: Database
● 5 days
● 3 users
● 4005 images
● Groundtruth available
31
Talavera, E., Dimiccoli, M., Bolaños, M., Aghaei, M., & Radeva, P. (2015). “R-Clustering for Egocentric
Video Segmentation”. In 7th Iberian Conference on Pattern Recognition and Image Analysis
(ACCEPTED).

32. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
○ Database
○ Clustering
■ Jaccard index
■ Linkage effect
■ Relabelling effect
○ Keyframe extraction
● Conclusions and future work 32

33. Evaluation: Clustering
● Jaccard index:
33

34. Linkage effect
34

35. Relabelling effect
35

36. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
○ Database
○ Clustering
○ Keyframe extraction
■ Blind taste test
■ Representative quality of keyframe
■ Summary validations
● Conclusions and future work 36

37. Evaluation: keyframe extraction
● User Surveys:
○ Representative quality of keyframe
○ Quality of summary
37

38. ● Methodology: Blind taste test
38
Lu and Grauman, ”Story-driven summarization for egocentric video” (CVPR 2013)
Figure: brandchannel.com
Blind taste test: quality of keyframe

39. 39

40. 40

41. Representative quality of keyframe
41
Do you think that the image of the left/center/right can represent the event?

42. Example of multi-event segmentation
42

43. Representative quality of keyframe
43
What image is more representative of the event, in your opinion?

44. Blind taste test: quality of summary
44

45. Summary validations
45
Can this set of images represent the complete day?

46. Summary validations
46
Which summary is the best, in your opinion?

47. Contents
● Collaboration
● Motivation and goals
● State of the art
● Methodology
● Evaluation
● Conclusions and future work
47

48. Conclusions and future work
● New methodology taking into account visual and
temporal information
● Keyframe extraction through graph-based approaches
48

49. Conclusions and future work
● 0.53 Jaccard index of segmentation
● 88-86% user acceptance with our summaries
● 58% users choose our summaries as best option
49

50. Conclusions and future work
● Temporal information causes important improvements
● First method of summary extraction for high temporal
resolution sets
50

51. Conclusions and future work
● Apply object detection
● Different criteria of representativity
● Clinical application of this work
51

52. Conclusions and future work
52
Planned
submission:
March 30, 2015

53. Thanks for your attention!
53