https://imatge.upc.edu/web/publications/semantic-and-diverse-summarization-egocentric-photo-events This project generates visual summaries of events depicted from egocentric photos taken with a wearable camera. These summaries are addressed to mild-dementia patients in order to exercise their memory in a daily base. The main contribution is an iterative approach that guarantees the semantic diversity of the summary and a novel soft metric to assess subjective results. Medical experts validated the proposed solution with a Mean Opinion Score of 4.6 out of of 5.0. The flexibility and quality of the solution was also tested in the 2015 Retrieving Diverse Social Images Task from the scientific international benchmark, MediaEval.

1. Semantic and Diverse
Summarization of Egocentric
Photo Events
Aniol Lidon Baulida
Master Computer Vision (UAB, UPC, UPF, UOC)
Advisors:
Xavier Giró Nieto, Image Processing Group, Universitat Politècnica de Catalunya
Petia Radeva, Barcelona Perceptual Computing Lab, Universitat de Barcelona
1

2. Collaboration
Barcelona Perceptual Computing Laboratory :
Marc Bolaños, Petia Radeva
Image Processing Group:
Xavier Giró
Grup de Recerca Cervell, Cognició i Conducta:
Maite Garolera
Institute of Creative Media Technologies:
Matthias Zeppelzauer
2

3. Motivation
• In 2013, 44.4 million people with dementia worldwide.
• “Cognitive Stimulation Therapy”
3

4. Motivation
• Lifelogging with Narrative Clip.
• Up to 2000~3000 images at day!
• Summarization is needed.
4

5. Goal
5
Automatically summarize events.
• Sorting by priority.
• Trade-off between relevance and diversity.
• Obtaining sorted ranks.

6. Goal
6
RELEVANCE
Automatically summarize events.
• Sorting by priority.
• Trade-off between relevance and diversity.
• Obtaining sorted ranks.

7. Goal
7
RELEVANCE
DIVERSITY
Automatically summarize events.
• Sorting by priority.
• Trade-off between relevance and diversity.
• Obtaining sorted ranks.

8. Sate of the art
• This project continues the work started by Ricard Mestre.
– Event segmentation and selecting the most repetitive image from an event.
• Off-the-shelf algorithms used:
– Informativeness network: provided by Marc Bolaños (to be published)
– Blur detection: Crete et al. The blur effect: perception and estimation with a new no-
reference perceptual blur metric
– Saliency Maps: provided by Kevin McGuinness (to be published).
– Face detection: Zhu et al. Face detection, pose estimation, and landmark localization in
the wild.
– Object Candidates: Arbelaez et al. Multiscale Combinatorial Grouping
– Object Detector: Hoffman et al. Large Scale Detection through Adaptation.
– Affective: Campos et al. Diving Deep into Sentiment: Understanding Fine-tuned CNNs for
Visual Sentiment Prediction
8

9. Pipeline
9

10. Pipeline
10

11. Prefiltering
11
Aim: Removing uninformative images.
Informativeness network
Fine-tuning by
Human Annotations
Filtering out: Discarding absolutely uninformative frames.

12. Pipeline
12

13. Pipeline
13

14. Relevance
14
What is relevance?
Frame-level:
•Repeated.
• Unusual.
• WHAT? Representative of an activity.
• WHO? Social interactions.
• WHERE? Environment.
• WHEN an event has occurred.
• HOW activity occurred.

15. Relevance
15
What is relevance?
Frame-level:
• WHAT? Representative of an activity.
• Saliency Maps
• Object detection
• WHO? Social interactions.
• Face detection
• Sentiment Analysis (Affectivity)

16. Relevance Ranking: pipeline
16
Prefiltering
Diversity
re-ranking

17. Relevance ranking
Saliency maps
SalNet CNN
Aim: Determining interesting zones.
Scoring for relevance: Averaging all saliency-map values.
17

18. Relevance ranking
18
Objects
LSDA Large Scale Detection through Adaptation
Object Detector
Aim: Finding well defined objects.
Scoring for relevance: Summing all detected objects scores.

19. Relevance ranking
19
Faces
Face detection, pose estimation, and
landmark localization in the wild.
Aim: Finding well defined faces.
Scoring for relevance: Summing exponentially all faces confidences.

20. Relevance Ranking: pipeline
20
Prefiltering
Diversity
re-ranking

21. Pipeline
21

22. Pipeline
22

23. Diversity re-ranking
Re-ranking by Soft Max Diversity Fusion
23
Color similarity
Faces similarity

24. Diversity re-ranking
Re-ranking by Soft Max Diversity Fusion
24
Color similarity
Faces similarity

25. Diversity re-ranking
Re-ranking by Soft Max Diversity Fusion
25
Color similarity
Faces similarity

26. Similarity measure
26
ImageNet
Euclidean distance between features (L2 norm).
CNN trained with ImageNet DB (1000 classes) using CaffeNet Architecture.
Fully connected layer 8 removed.

27. Pipeline
27

28. Pipeline
28

29. Assesment
29
Validation of automatic
approach
Manually annotated
summaries
• 7 dataset with labelled ground-truth • 2 Online questionnaires
• Mean Opinion Score
Psychologists feedback:
INTERMEDIATE VALIDATION FINAL EVALUATION

30. Subjective problem
30
Precision
GROUND-TRUTH SELECTED

31. Metric
31
Mean Normalized Sum of Max Similarities (MNSMS)
MNSMS n (%)
Normalization in both axes
Y: Divide by GT samples
X: Reshape samples to N bins
Ground-Truth
SortedList(Results)
n=1
Similarity Sum= + +

32. Metric
32
Mean Normalized Sum of Max Similarities (MNSMS)
MNSMS n (%)
Normalization in both axes
Y: Divide by GT samples
X: Reshape samples to N bins
Ground-Truth
SortedList(Results)
n=2
Similarity Sum= + +

33. Metric
33
Mean Normalized Sum of Max Similarities (MNSMS)
MNSMS n (%)
Normalization in both axes
Y: Divide by GT samples
X: Reshape samples to N bins
Ground-Truth
SortedList(Results)
n= 3
Similarity Sum= + +

34. Metric
34
Mean Normalized Sum of Max Similarities (MNSMS)
MNSMS n (%)
Normalization in both axes
Y: Divide by GT samples
X: Reshape samples
Ground-Truth
SortedList(Results)
Similarity Sum= + +
n= 4

35. AUC
Metric
35
Mean Normalized Sum of Max Similarities (MNSMS)
MNSMS n (%)
Normalization in both axes
Y: Divide by GT samples
X: Reshape samples
Ground-Truth
SortedList(Results)
Similarity Sum= + +
n= 4

36. Assesment
36
Validation of automatic
approach
Manually annotated
summaries
• 7 dataset with labelled ground-truth
• MNSMS (ImageNet) AUC
• 2 Online questionnaires
• Mean Opinion Score
Psychologists feedback:
INTERMEDIATE VALIDATION FINAL EVALUATION

37. Intermediate validation
37
Prefiltering
•Informativeness Network
•Hand Crafter Estimators
• Not prefitering

38. Intermediate validation
38
• SalNet
• SalNet + Gaussian
Objects Relevance
• LSDA (object detector)
• MCG (object candidates)
0,7
0,75
0,8
0,85
0,9
SalNet SalNet + Gauss
0,7
0,75
0,8
0,85
0,9
LSDA MCG
Saliency Relevance
Saliency Relevance AUC
Objects Relevance AUC

39. Intermediate validation
Affective Relevance
• Positive
• Negative
•Extremum
•Random
Sentiment analysis CNN
• 2 classes: positive / negative
39

40. Assesment
40
Validation of automatic
approach
Manually annotated
summaries
• 7 dataset with labelled ground-truth
• MNSMS (ImageNet) AUC
• 2 rounds of online questionnaires
• Mean Opinion Score
Psychologists feedback:
INTERMEDIATE VALIDATION FINAL EVALUATION

41. Final evaluation
41
SIMILARITY
• ImageNet CNN (fc8 removed)
• Places CNN (fc8 removed)
• LSDA (only spatial NMS)
• Fusion (ImageNet + Places + LSDA)
(Diversity re-ranking + Weight fusion in MNSMS)

42. Final evaluation
42

43. Final evaluation
43
MEAN OPINION SCORE
• ImageNet configuration
• Uniform Sampling
• Ground-truth (previous manual annotation)

44. Final evaluation
44

45. Final results
Representativity of summaries:
Preferred summary:
Mean Opinion Score (1 worse - 5 best)
45

46. Generalization
Mediaeval diverse task
• APPLICATION: Finding more information about a place to visit.
• GOAL: Povide a ranked list of Flickr photos for a predefined set of queries. The
refined list should be both relevant to the query and also diverse.
46
A. Lidon, M. Bolaños, M. Seidl, X. Giro-i Nieto, P. Radeva, and M. Zeppelzauer, “Upc-ub-stp @ mediaeval 2015 diversity
task: Iterative reranking of relevant images,” in MediaEval 2015 Workshop, Wurzen, Germany, 2015.
0,4
0,42
0,44
0,46
0,48
0,5
0,52
0,54
0,56
Run 1 F1@20 (Visual)

47. Conclusions
• Contributions:
– Mean Normalized Sum of Max Similarities.
– New criterion for semantic diversity (based on LSDA).
– New method for diversity fusion.
– Online evaluation questionnaires.
47

48. Conclusions
• Tested in two applications:
– Memory reinforcement for mild-dementia.
– Diverse Social Images Task from the scientific MediaEval benchmark.
• Mean Opinion Score of 4.6 out of 5.00.
• Publications:
– Working-notes paper in MediaEval challenge.
– Wearable and Ego-vision Systems for Augmented Experience of the
journal IEEE Transactions on Human-Machine Systems.
• Code available: https://imatge.upc.edu/web/resources/semantic-
and-diverse-summarization-egocentric-photo-events-software
48

49. Future work
• Further in other relevance criterion.
• Higher level of semantics.
• Determine automatically the summary length.
49

50. Thanks for your attention!
50

51. Prefiltering
51
Hand-crafted estimators
Blur
Black
Burned Color mean
Crete et al.
Informativeness network
•CNN trained with ImageNet + Places.
•Finetuned with human annotations:
relevant / irrelevant
by Marc Bolaños (UB)

52. Relevance ranking
52
Affective
• VitorNet CNN (2 classes
sentiment prediccions)
by Victor Campos (UPC)

53. Relevance ranking
53
Late fusion
• Score normalization:
•By Rank
•By Score
• Aggregate scores
Using MNSMS weights will be learned

54. Similarity measure
54
ImageNet
Places
LSDa
CNN trained with ImageNet DB (1000 classes) using CaffeNet Architecture.
Fully connected layer 8 removed.
CNN trained with Places (476 classes) DB using CaffeNet Architecture.
Fully connected layer 8 removed.
Object detector : Large Scale Detection through Adaptation (7500 classes).
Knowledgement transfer: Classifiers without bounding box annotated data into detectors
Two post-processing steps of no-maxima supression.

55. Result
Mediaeval diverse task
• APPLICATION: Finding more information about a place to visit.
• GOAL: Povide a ranked list of Flickr photos for a predefined set of queries. The
refined list should be both relevant to the query and also diverse.
Ranking for
relevance
Filtering
Distance
computation
Diversity
Informativeness network, Textual
Keep N% top results
ImageNet, Places, Textual
Diverse top results

56. Result
Mediaeval diverse task
• APPLICATION: Finding more information about a place to visit.
• GOAL: Povide a ranked list of Flickr photos for a predefined set of queries. The
refined list should be both relevant to the query and also diverse.
Visual Textual Multi Crediv. Multi