We present a novel solution to the MediaEval 2014 Event Synchronization Task: Synchronization of Multi-User Event Media (SEM). The framework is based on a probabilistic graphical model. Thanks to the simple topology of the graph, the estimation of the true temporal displacement among multiple photo collections can be performed eciently through exact inference. The underlying tness function is dened in a exible way, for which it is possible to integrate easily new information (e.g., text tags or social network data). The exibility makes the framework suitable and adaptable to cope with many real situations. The method is evaluated on two datasets obtaining an overall accuracy of more than 85% in both cases

1. Synchronizing Multi-User Photo
Galleries with MRF
Emanuele Sansone, Giulia Boato Minh-Son Dao
DISI, University of Trento UIT-HCM
Trento, Italy HCMC, Viet Nam
e.sansone@unitn.it, boato@disi.unitn.it sondm@uit.edu.vn

2. Synchronization and Clustering
Multiple
Galleries
Feature
Extraction
Offset
Estimation
Synchronized
Galleries

3. Stereo Matching I
Stereo camera
Acquired Images

4. Stereo Matching II

5. Stereo Matching: Model
Observed node = pixel intensity
Latent node = possible disparity for a
given node
Potentials associated to each link in
the network
Inference algorithm
Disparity

6. Stereo Matching vs. Photo Galleries
Synchronization
• Pixels  Photos
• Pixel Intensitities  Image features (Timestamp, GPS, SURF, color
histograms)
• States: possible diparities  States: possible offsets

7. Synchronization: States

8. Synchronization: Potentials

9. Synchronization: Potentials

10. Synchronization: Potentials

11. Synchronization: Potentials

12. Synchronization: Potentials

13. Synchronization: Potentials

14. Synchronization: Final Model

15. Synchronization: Inference

16. Clustering
• K-means algorithm
• 3 runs

17. Clustering
• K-means algorithm
• 3 runs
Color Structure Descriptors
(CSD - 64 values) + Local Binary Patterns
3x3 (LBP - 18 values each block)

18. Clustering
• K-means algorithm
• 3 runs
Color Structure Descriptors
(CSD - 64 values) + Local Binary Patterns
3x3 (LBP - 18 values each block)
PCA Color Structure
Descriptors (6 values)

19. Clustering
• K-means algorithm
• 3 runs
Color Structure Descriptors
(CSD - 64 values) + Local Binary Patterns
3x3 (LBP - 18 values each block)
PCA Color Structure
Descriptors (6 values)
PCA Color Structure
Descriptors (6 values) +
TIME

20. Results: Synchronization
Dataset Precision Accuracy
Vancouver 0.35 0.86
London 0.25 0.89

21. Results: Synchronization
Dataset Precision Accuracy
Vancouver 0.35 0.86
London 0.25 0.89
• Good performance in terms of accuracy

22. Results: Synchronization
Dataset Precision Accuracy
Vancouver 0.35 0.86
London 0.25 0.89
• Good performance in terms of accuracy
• Only 1/4 galleries correctly synchronized

23. Results: Clustering
Dataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885

24. Results: Clustering
Dataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885
• Performance are decreased by using only color descriptors

25. Results: Clustering
Dataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885
• Performance are decreased by using only color descriptors
• In the Vancouver dataset time doesn’t help in clustering

26. Results: Clustering
Dataset Run Features Jaccard Index
Vancouver
1 CSD-64+LBP3x3 0.1673
2 CSD-6 0.1382
3 CSD-6+TIME 0.1315
London
1 CSD-64+LBP3x3 0.1287
2 CSD-6 0.0742
3 CSD-6+TIME 0.0885
• Performance are decreased by using only color descriptors
• In the Vancouver dataset time doesn’t help in clustering
• In the London dataset time becomes a more reliable feature

27. Lessons Learned
Synchronization:
Only 1/4 galleries correctly synchronized

28. Lessons Learned
Synchronization:
Only 1/4 galleries correctly synchronized
• new features (e.g. text tags, social network data)

29. Lessons Learned
Synchronization:
Only 1/4 galleries correctly synchronized
• new features (e.g. text tags, social network data)
• “no association” between a pair of images

30. Lessons Learned
Synchronization:
Only 1/4 galleries correctly synchronized
• new features (e.g. text tags, social network data)
• “no association” between a pair of images
Clustering:
Time is important!