Social Event Detection (SED): Challenges, Dataset and Evaluation<br />RaphaëlTroncy<raphael.troncy@eurecom.fr> VasileiosMe...
What are Events?<br />Events are observable occurrences grouping <br />01/09/2011 - <br />Social Event Detection (SED) Tas...
Two challenges (type and venue)<br />Find all soccer events taking place in Barcelona (Spain) and Rome (Italy) in the coll...
Dataset Construction<br />Collect 73,645 Flickr Photos (May 2009)<br />98,3% geo-tagged in 5 cities (Amsterdam, Barcelona,...
Ground Truth and Evaluation Measures<br />Ground Truth<br />Use EventMedia and machine tags (lastfm:event=xxx)<br />Manual...
Who Has Participated ?<br />18 Teams registered<br />7 Teams cross the lines<br />Everybody is present at the workshop!<br...
Quick Summary of Approaches<br />Up to 5 non-constrained runs per challenge<br />All participants use background knowledge...
Results (sent to participants)<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br ...
Results (updated, real?)<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 9<...
Challenge 1 (all runs) – F-measure<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy...
01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 11<br />Challenge 2 (all runs) –...
Conclusion<br />It was an easy task  … BUT people had fun<br />Looking at next year SED<br />Dataset: bigger, more divers...
EURECOM @Social Event Detection (SED) <br />Xueliang Liu <xueliang.liu@eurecom.fr>RaphaëlTroncy<raphael.troncy@eurecom.fr>...
What are Events?<br />Events are observable occurrences grouping <br />… and announced on the WEB !<br />02/09/2011 - <br ...
Approach<br />Get background knowledge about occurrences of past events<br />Information retrieval approach<br />Event inf...
Which Prior Knowledge?<br />Challenge 1<br />6 past football games in Barcelona and Roma<br />Challenge 2<br />68 past eve...
Event Model and Photo Query<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />-...
Matching Process<br />Given a photo P and an event Ewhere<br />δisthe Dirac delta function<br />N is used for scaling (var...
Visual Pruning and Owner Refinement<br />Are photos taken at the event visually similar?<br />Low-level features used:<br ...
Challenge 1<br />Run 1: basic Event Identification Model (N=3)<br />Run 2: run 1 + Owner Refinement<br />Photos for 2 game...
Challenge 2<br />Run 1 / 3: basic Event Identification Model (N=1) / (N=3)<br />Run 2 / 4: run 1 / run 3 + Owner Refinemen...
Challenge 2 Results – Paradiso<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br ...
Challenge 2 Results – Parc del Forum<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Ita...
Conclusion<br />Event information model using background knowledge:<br />Dedicated resources for Sport Events<br />General...
Upcoming SlideShare
Loading in...5
×

MediaEval 2011 SED Opening

1,132

Published on

Opening presentation of the Social Event Detection (SED) task at MediaEval 2011 and EURECOM's participation, September 2011, Pisa, Italia

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,132
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

MediaEval 2011 SED Opening

  1. 1. Social Event Detection (SED): Challenges, Dataset and Evaluation<br />RaphaëlTroncy<raphael.troncy@eurecom.fr> VasileiosMezaris<bmezaris@iti.gr>Symeon Papadopoulos <papadop@iti.gr>Benoit Huet<benoit.huet@eurecom.fr> IoannisKompatsiaris<ikom@iti.gr><br />
  2. 2. What are Events?<br />Events are observable occurrences grouping <br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 2<br />People<br />Places<br />Time<br />Experiences documented by Media<br />
  3. 3. Two challenges (type and venue)<br />Find all soccer events taking place in Barcelona (Spain) and Rome (Italy) in the collection. For each event provide all photos associated with it<br />Find all events that took place in May 2009 in the venue named Paradiso (in Amsterdam, NL) and in the Parc del Forum (in Barcelona, Spain). For each event provide all photosassociated with it<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 3<br />
  4. 4. Dataset Construction<br />Collect 73,645 Flickr Photos (May 2009)<br />98,3% geo-tagged in 5 cities (Amsterdam, Barcelona, London, Paris and Rome)<br />1,7% (1294) non geo-tagged from EventMedia<br />Altered metadata:<br />geo-tags removed for 80% of the photos (random)<br />14,465 photos still geo-tagged<br />Provide only metadata … but real media were available to participants if they asked (2,43 GB)<br />1% (697) photos disappeared in June-July 2011<br />Who studied the volatility of Flickr photos?<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 4<br />
  5. 5. Ground Truth and Evaluation Measures<br />Ground Truth<br />Use EventMedia and machine tags (lastfm:event=xxx)<br />Manual lookup at photos from Amsterdam and Barcelona<br />Discussion for the corner cases<br />14 photos discussed for challenge 1<br />No time for discussion for challenge 2 (single assessor)<br />Evaluation Measures:<br />Harmonic mean (F-score of Precision and Recall)<br />Normalized Mutual Information (NMI): jointly consider the goodness of the photos retrieved and their correct assignment to different events<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 5<br />
  6. 6. Who Has Participated ?<br />18 Teams registered<br />7 Teams cross the lines<br />Everybody is present at the workshop!<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 6<br />
  7. 7. Quick Summary of Approaches<br />Up to 5 non-constrained runs per challenge<br />All participants use background knowledge<br />Last.fm (all), Fbleague (EURECOM), PlayerHistory (QMUL)<br />DBpedia, Freebase, Geonames, WordNet<br />Classification vs Information Retrieval approach:<br />City classifier, topic/venue classifier (CERTH)<br />Linear SVM (QMUL)<br />Latent Dirichelet Allocation (LIA)<br />Hybrid (ANU)<br />Image processing:<br />ITI, EURECOM and ANU<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 7<br />
  8. 8. Results (sent to participants)<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 8<br />Challenge 1<br />Challenge 2<br />
  9. 9. Results (updated, real?)<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 9<br />Challenge 1 (ground truth contained photos not present in the dataset)<br />Challenge 2<br />
  10. 10. Challenge 1 (all runs) – F-measure<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 10<br />
  11. 11. 01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 11<br />Challenge 2 (all runs) – F-measure<br />
  12. 12. Conclusion<br />It was an easy task  … BUT people had fun<br />Looking at next year SED<br />Dataset: bigger, more diverse, training vs test sets<br />Media: photos and videos<br />Metadata: include some social network relationships, participation at events<br />Challenges: detect personal events<br />Evaluation measures: event granularity? Time/CPU?<br />…<br />01/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 12<br />
  13. 13. EURECOM @Social Event Detection (SED) <br />Xueliang Liu <xueliang.liu@eurecom.fr>RaphaëlTroncy<raphael.troncy@eurecom.fr> Benoit Huet<benoit.huet@eurecom.fr><br />
  14. 14. What are Events?<br />Events are observable occurrences grouping <br />… and announced on the WEB !<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 14<br />People<br />Places<br />Time<br />Experiences documented by Media<br />
  15. 15. Approach<br />Get background knowledge about occurrences of past events<br />Information retrieval approach<br />Event information model // metadata + photo query<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 15<br />Query<br />Event Occurrences<br />Matching<br />
  16. 16. Which Prior Knowledge?<br />Challenge 1<br />6 past football games in Barcelona and Roma<br />Challenge 2<br />68 past events recorded in Paradiso and Parc del Forum<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 16<br />
  17. 17. Event Model and Photo Query<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 17<br />E = {title, geo, time}<br />Event<br />P = {text, geo, time}<br />
  18. 18. Matching Process<br />Given a photo P and an event Ewhere<br />δisthe Dirac delta function<br />N is used for scaling (vary depending on the run)<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 18<br />p(P|E) = p(P.text|E.title) p(P.geo|E.geo)p(P.time|E.time)<br />
  19. 19. Visual Pruning and Owner Refinement<br />Are photos taken at the event visually similar?<br />Low-level features used:<br />Color moments, Gabor texture, Edge histogram<br />L1 distance on the K-nearest neighbors<br />Photos sorted according to the distance<br />Experimentally, we remove the 5% photos that are far away from the center in the visual feature space<br />Confidence that a media sharer attended an event<br />Effective way to deal with photos without any textual description<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 19<br />
  20. 20. Challenge 1<br />Run 1: basic Event Identification Model (N=3)<br />Run 2: run 1 + Owner Refinement<br />Photos for 2 games while we had knowledge for 6<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 20<br />
  21. 21. Challenge 2<br />Run 1 / 3: basic Event Identification Model (N=1) / (N=3)<br />Run 2 / 4: run 1 / run 3 + Owner Refinement<br />Run 5: run 3 + Visual Pruning + Owner Refinement<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 21<br />
  22. 22. Challenge 2 Results – Paradiso<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 22<br />
  23. 23. Challenge 2 Results – Parc del Forum<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 23<br />
  24. 24. Conclusion<br />Event information model using background knowledge:<br />Dedicated resources for Sport Events<br />General event directories for Popular Venues<br />Querying photos with occurrences of past events<br />Importance of time for structuring media collection<br />The way we used visual analysis didn’t add any value<br />02/09/2011 - <br />Social Event Detection (SED) Task - MediaEval 2011, Pisa, Italy<br />- 24<br />

×