The document discusses finding and organizing media like photos and videos that illustrate real-world events. It describes scraping various event directories and linking the data. Media can be found by searching based on geo-tags, titles, or uploader. Visual analysis is used to prune media results. The goal is to help users explore, annotate and share media related to events. Challenges include interlinking diverse data sources and detecting unscheduled events.

1. Finding Media
Illustrating Events
Xueliang Liu <xueliang.liu@eurecom.fr>
Raphaël Troncy <raphael.troncy@eurecom.fr>
Benoit Huet <benoit.huet@eurecom.fr>

2. What are Events?
Events are observable occurrences grouping
People Places Time
Experiences documented by Media
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -2

3. http://www.flickr.com/photos/crsan/3697785107

4. Searching for an event
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -4

5. Searching for media about an event
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -5

6. There’s a lot of information out there…
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -6

7. http://www.flickr.com/photos/mwparenteau/432039783
19/04/2011 - 7

8. Organize the mess
Event
Media
 Scrape event directories
 Link the information
 Find media illustrating events
 Design the application Interface
Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -8

9. Róisín Murphy at Nouveau Casino
E0-001-005971169-9
350591
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -9

10. Representing Events with
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 10

11. Interlinking
 Linking Agents with
 Freebase, Dbpedia, MusicBrainz
 Linking Venues with
 Geonames, Dbpedia, Foursquare (via Uberblic)
 Linking Events with
 Last.fm, Upcoming, Eventful
 Linking Categories with
 Facebook, Eventful, Upcoming, Zevents, LinkedIn,Eventbrite,
TicketMaster
 Linking Users with
 Social Graph API
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 11

12. Róisín Murphy at Nouveau Casino
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 12

13. Media explicitly associated with the event
APIs
Machine tags
“lastfm:events”
4790 photos, 263 1.7 million images over
videos over 110 events 108.000 events
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 13

14. Representing Media with Media Ontology
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 14

15. How much data is there?
Event Agent Location Photos User
Last.fm 57,258 50,150 16,471 1,425,318 18,542
Upcoming 13,114 0 7,330 347,959 4,518
Eventful 37,647 6,543 14,576 0 0
Total 108,019 56,693 38,377 1,773,277 23,060
1,248,021 geo-tagged photos
by propagating information from events!
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 15

16. How fast media are uploaded?
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 16

17. Finding more media that illustrate an event
A. Compute the bounding box area of a venue
B. Retrieve all media geo-tagged in this area
C. Retrieve all media with a similar title
D. Prune the results with visual analysis
E. Extend the result set with all media from the
same uploader
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 17

18. A. Bounding box of Nouveau Casino?
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 18

19. B. 74 photos taken in this area this day
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 19

20. C. 85 additional photos with a similar title
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 20

21. D. 6 photos after visual pruning
 

  
  
19/04/2011 -

Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 21


22. How is the visual pruning performed?
 Model dataset: photo id + photo geo
 Testing dataset: similar title
 Low-level features used:
 Color moments, Gabor texture, Edge histogram
 L1 distance on the K-nearest neighbors
 Threshold
 Min L1 distance between two model image pairs
 Conservative approach
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 22

23. E. 66 photos after uploader heuristics

hellerpop
DustGraph / Stefan
cartoixa
13 photos 46 photos
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 23

24. Same process for videos
1 video (id)
3 videos (geo)

26 videos (title)
Visual pruning
performed on
key frames

 Nb positive > 50%
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 24

25. How illustrated are the events?
Query By ID Query By Geo Query By Title Visual Pruning Heuristic
Photos 5 74 (74) 85 (85) 6 (6) 66 (66)
Videos 1 3 (0) 23 (0) 13 (0) -
(title)
Videos 10 (10)
(title+venue)
 20 events
 Model dataset: 785 photos
 Testing dataset: 1766 photos (1573 positive, 193 negative)
 Results: 439 photos (99% precision, 28% recall)
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 25

26. User Interface
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 26

27. Conclusions
 Method for finding media illustrating scheduled
events
 Search media with machine and geo tags
 Search media with title and normal tags
 Prune visually and retrieve all media from confirmed users
 Challenge: do not necessarily trust the geo-coordinates
 Event-based approach for users to explore,
annotate and share media
 UX can help semantics, semantics can help UX
 Outstanding challenges in interlinking and curating the data
 Detecting events from social media activity
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 27

28. http://www.slideshare.net/troncy
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 28