This document summarizes an approach for diversifying image retrieval results using adaptive clustering. It explores using various visual and textual features with clustering algorithms like k-means and affinity propagation. Evaluation on a benchmark task shows the best approach achieves a precision of 0.8250, recall of 0.6634, and F1-score of 0.7186. While fixed settings underperform, the adaptive approach achieves a precision of 0.6500, recall of 0.4398, and F1-score of 0.5123 on visual features alone. Lessons learned are that different queries favor different features, approaches need flexibility, and the evaluation could be made more objective.

1. AN ADAPTIVE CLUSTERING APPROACH FOR THE
DIVERSIFICATION OF IMAGE RETRIEVAL RESULTS
MAIA ZAHARIEVA
VIENNA UNIVERSITY OF TECHNOLOGY & UNIVERSITY OF VIENNA, AUSTRIA
MAIA.ZAHARIEVA@TUWIEN.AC.AT
October 20-21, Hilversum, Netherlands
RETRIEVING DIVERSE SOCIAL IMAGES TASK

2. THE IDEA

3. QUERY: SAILING BOAT
QUERY: TREES REFLECTED IN WATER

4. Different queries Different features
Different clustering approaches
Potentially highly
imbalanced groupings
Varying dimensionalities

5. THE WORKFLOW

6. APPROACHES
Affinity Propagation (AP)
Expectation Maximisation (EM)
k-Means (KM)
X-Means (XM)
CLUSTERING/DIVERSIFICATION
VISUAL
CNN (ad, gen)
DCT
Intensity histogram (IH)
KANSEI shape (KS)
MPEG-7 color layout (CL)
MPEG-7 color structure (CS)
MPEG-7 edge histogram (EH)
MPEG-7 homogeneous texture (HT)
MPEG-7 region-based shape (RS)
MPEG-7 scalable color (SC)
TEXTUAL
TF-IDF (title)
TF-IDF (tags)
TF-IDF (description)
TF-IDF (title+tags)
TF-IDF (title+tags+description)
FEATUREEXTRACTION
RANKED
IMAGE SET
INTERNALEVALUATION
Compactness:
- Sum of squares
- C-Index
Separability:
- Single linkage
Compactness + Separability
- Calinski-Harabasz
- Davies-Bouldin
- Silhouette
Consistency-based
- Gamma
- Tau
…
ROUNDROBIN
RE-RANKED
IMAGE LIST

7. EVALUATION RESULTS

8. Affinity
Propagation
Expectation
Maximization
k-Means X-Means ∑
CNN ad 11, 25 29 38, 51 5
CNN gen 4, 59 54 3
DCT 48 19, 67 50 4
Intensity histogram 45 2 28, 53 6, 65 6
KANSEI shape 9, 20, 26, 36, 46 57, 13 7
MPEG7 CL 22, 47, 55 5, 61 12, 21 7
MPEG7 CS 15 7, 62, 70 4
MPEG7 EH
MPEG7
32, 68 14, 40, 42, 52 6
MPEG7 HT 44 8, 27, 63 24 5
MPEG7 RS 23, 69 10, 41 35, 64, 66 7
MPEG7 SC 3, 43, 56 30, 37 5
TITLE 16 1
TAGS 58, 60 2
TITLE+TAGS 1, 49 2
TITLE+TAGS+DESCR 17, 39 33 18, 31, 34 6
∑ 2 26 26 16 70
OPTIMAL SOLUTION

9. Approach P@20 CR@20 F1@20
Flickr Baseline 0.6979 0.3717 0.4674
Optimal Solution
visual 0.8179 0.6575 0.7122
text 0.8136 0.6453 0.7043
visual+text 0.8250 0.6634 0.7186
Best performing
fixed settings
visual 0.6657 0.4453 0.5237
text 0.6636 0.4274 0.5045
visual+text 0.6657 0.4453 0.5237
Adaptive
approach
visual 0.6500 0.4398 0.5123
text 0.6729 0.4230 0.5029
visual+text 0.6286 0.4061 0.4803
DEVELOPMENT DATA

10. BENCHMARK RESULTS 2016

11. run configuration P@20 CR@20 F1@20
run 1 adaptive, visual 0.5141 0.4024 0.4292
run 2 adaptive, text 0.5406 0.4130 0.4463
run 3 adaptive, visual+text 0.5430 0.4130 0.4471
run 4 fixed, visual 0.4969 0.3603 0.4006
BENCHMARK RESULTS 2016

12. LESSONS LEARNED

13. MEDIAEVAL BENCHMARK 2016: RETRIEVING DIVERSE SOCIAL IMAGES TASK
LESSONS LEARNED
▸ Different queries favour different feature representations
▸ Strong requirements for higher generalization applicability
and flexibility of approaches for image search diversification
▸ Multiple solutions can be considered being correct
▸ How to make the evaluation more objective?

14. THANK YOU!MAIA.ZAHARIEVA@TUWIEN.AC.AT