• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Multimedia Data Collection using Social Media Analysis
 

Multimedia Data Collection using Social Media Analysis

on

  • 742 views

The opening keynote of VIGTA 2012 – First International Workshop on Visual Interfaces for Ground Truth Collection in Computer Vision Applications...

The opening keynote of VIGTA 2012 – First International Workshop on Visual Interfaces for Ground Truth Collection in Computer Vision Applications
In conjunction with the Advanced Visual Interfaces International Working Conference in Capri Italy, May 21-25, 2012

Statistics

Views

Total Views
742
Views on SlideShare
741
Embed Views
1

Actions

Likes
2
Downloads
0
Comments
0

1 Embed 1

http://www.slashdocs.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • State of the art
  • 83 ????
  • More place
  • More place
  • Red box is the intersection with the ground truth!
  • Another place
  • Another place
  • Another picture

Multimedia Data Collection using Social Media Analysis Multimedia Data Collection using Social Media Analysis Presentation Transcript

  • Multimedia Data Collectionusing Social Media Analysis Prof. Benoit HUET EURECOM, France Benoit.Huet@eurecom.fr
  • Visual Data in the 90’s Huet & Hancock [WACV’96] Digital Map Corresponding aerial images Ground Truth taken at different aircraft altitudes 21/05/2012 B. HUET - VIGTA12 Keynote -2
  • Large Scale in the 90’ Huet & Hancock [IEEE PAMI’99] Cartographic Database  22 original images  Aerial scenes  Main features: roads  100-1000 lines per image Trademarks and logos Database [Flickner et al. ’95]  Over 1000 original images  Scanned data  B&W, Various resolution  10-5000 lines per image 21/05/2012 B. HUET - VIGTA12 Keynote -3
  • The TRECVID years (2001- to date) 2001: 11 hrs from BBC & OpenVideo Project  2003 first collaborative ground truth annotation 2005-2006: 170 hrs (Nov.’04 news in Arabic, Chinese, and English)  High-level feature extraction (10) 2007-2009: 100hrs from the Netherlands Institute for Sound and Vision (news magazine, science news, news reports, documentaries, educational programming, and archival video) 2010-2011: 600hrs of MPEG-4 Creative Commons Videos  High-level feature extraction (light=50 full=364) 21/05/2012 B. HUET - VIGTA12 Keynote -4
  • The Trend: Datasets are going Large-Scale (Web-Scale) ...slowly... Multimedia / Computer Vision researchers are tackling and experimenting with Large-Scale data Issue: 1 research objective <-> 1 data corpusAnnotation -> expensive and demanding process 21/05/2012 B. HUET - VIGTA12 Keynote -5
  • Talk Outline The scene / motivation Social Events and Big Data  Using social platforms for creating a corpus automatically Social Event Detection  Using social media for detecting events Social Event Media Mining  Enriching Event‟s Illustrations through Web Mining Conclusions 21/05/2012 B. HUET - VIGTA12 Keynote -6
  • What’s a Social Event? 21/05/2012 B. HUET - VIGTA12 Keynote -7
  • What’s a Social Event? VIGTA 2012 Capri Italy 21/05/2012 B. HUET - VIGTA12 Keynote -8
  • Big Data! 21/05/2012 B. HUET - VIGTA12 Keynote -9
  • Search For media 21/05/2012 B. HUET - VIGTA12 Keynote - 10
  • Searching for an event 21/05/2012 B. HUET - VIGTA12 Keynote - 11
  • Data Collection and Ground Truth
  • Machine Tags A way to integrate Media and Events LastFM Flickr YouTube 21/05/2012 B. HUET - VIGTA12 Keynote - 13
  • Media explicitly associated with the event 21/05/2012 B. HUET - VIGTA12 Keynote - 14
  • REST API for query 21/05/2012 B. HUET - VIGTA12 Keynote - 15
  • Conclusion The medias and events can be linked via machine tag. The relations provided by machine tags can be taken as ground truth. Thanks to the REST API, Events and Media information can be retrieved effectively. 21/05/2012 B. HUET - VIGTA12 Keynote - 16
  • Event Detectionby Temporal Analysis X. Liu, R. Troncy and B. Huet
  • Event Detection - Related Work EventBurn.com  Create summaries about given events (searching Twitter, Facebook, and Flickr) Firan et al. (CIKM’10)  Event categorization from social media data Gao et al. (WWW’11)  Employing Twitter data to enrich event information Liu et al. (ICMR’11)  Finding media illustrating events 21/05/2012 B. HUET - VIGTA12 Keynote - 18
  • How to mine events from PhotoSet… Events ?? 21/05/2012 B. HUET - VIGTA12 Keynote - 19
  • Observation Media are captured during events and shared Capture Time, Geo-localization User Tags (Annotations) Machine-Tag (lastfm:event=1337426) 21/05/2012 B. HUET - VIGTA12 Keynote - 20
  • How fast media are uploaded? 21/05/2012 B. HUET - VIGTA12 Keynote - 21
  • Experiment Data 9 Attractive Venues WorldWide Venue Name NbEvents NbPhotos NbUsers Melkweg 352 6912 266 Koko 151 3546 155 HMV Forum 106 2650 130 111 Minna Gallery 24 1369 105 HMV Hammersmith Apollo 79 2124 96 Circolo degli Artisti 148 2571 86 Circolo Magnolia 79 2190 76 Ancienne Belgique 212 7831 56 Rotown 204 3623 49 Event Ground Truth obtained from the official agendas  available from individual venue websites. 21/05/2012 B. HUET - VIGTA12 Keynote - 22
  • Detecting and Identifying Events Our solution consists of 3 steps:  Location Monitoring: finding the bounding-box of venues.  Temporal Analysis: detecting events by analyzing the uploading behavior along time.  Event Topic Identification: identifying detected events’ topics through tag analysis. 14 12 10 8 6 4 2 0 10/05/01 10/05/06 10/05/11 10/05/16 10/05/21 10/05/26 10/05/31 Location Temporal Event Topic ResultsMonitoring Analysis Identification 21/05/2012 B. HUET - VIGTA12 Keynote - 23
  • Event Detections Region Monitoring 21/05/2012 B. HUET - VIGTA12 Keynote - 24
  • Venue Bounding Box Estimation1 : INPUT : VenueName2 : OUTPUT : BoundingBox3 : PhotoSet []4 : Center GetInfo( VenueName)5 : EventSet GetPastEvents(VenueName)6 : foreach event in EventSet do7: photos GetFlickrPhoto(event)8: PhotoSet.append ( photos)9 : end10 : GeoSet GetGeoInfo( PhotoSet)11 : Filter (GeoSet, Center, threshold 1km)12 : RETURN MinRect(GeoSet) 21/05/2012 B. HUET - VIGTA12 Keynote - 25
  • Venue Bounding Boxes (a selection) Paradiso HMV Hammersmith Apollo Megwelk KoKo 21/05/2012 B. HUET - VIGTA12 Keynote - 26
  • Analyzing the number of Photos L o c a t i o n Megwelk D a t REST e Query 21/05/2012 B. HUET - VIGTA12 Keynote - 27
  • Our Media DataSet Flickr Photos  Taken in May 2010  In either one of the 9 selected locations: Number of Photos Name Overlap Total Geo-tagged Venue Name taggedKoko 372 2040 3 2409Rotown 90 273 1 362Melkweg 363 700 8 1055HMV Forum 184 412 0 596111 Minna Gallery 937 3 0 940Ancienne Belgique 2206 288 2 2492Circolo degli Artisti 70 553 1 622Circolo Magnolia 95 236 0 331Hammersmith Apollo 287 84 0 371 Total : 4604 4589 15 9178  Photos rarely have both geo-tag and venue name tag! 21/05/2012 B. HUET - VIGTA12 Keynote - 28
  • Analyzing the number of Photos 250 200 Events ?? 150 100 50 0 10/05/01 10/05/06 10/05/11 10/05/16 10/05/21 10/05/26 10/05/31 Number of Photos taken in Melkweg (NL) in May 2010 21/05/2012 B. HUET - VIGTA12 Keynote - 29
  • Analyzing the number of Photos Owners 14 12 Events ?? 10 8 6 4 2 0 10/05/01 10/05/06 10/05/11 10/05/16 10/05/21 10/05/26 10/05/31 Number of Photo Owners in Melkweg in May 2010 21/05/2012 B. HUET - VIGTA12 Keynote - 30
  • Event Detection Approach Based on media upload activity  At a given time  At a given location Events can be detected by: et arg(ti T) i  Where ti N photos * N owners T : Threshold Venue/Event popularity  Adaptive thresholding 21/05/2012 B. HUET - VIGTA12 Keynote - 31
  • Event Topics Mining Keep the top N most frequent tags Result: melkweg anouk amsterdam jemaine 2010 european flight flightoftheconchords conchords fotc mckenzie clement tour bret evelyn 21/05/2012 B. HUET - VIGTA12 Keynote - 32
  • Event Detection Example Melkweg in May 2010 Number of photos * Number of photo owners 21/05/2012 B. HUET - VIGTA12 Keynote - 33
  • Event Detection Example 111 Minna Gallery in May 2010 Number of photos * Number of photo owners 21/05/2012 B. HUET - VIGTA12 Keynote - 34
  • Event Detection Results Detection results on different conditions Source Threshold True Predict False Predict F1 mean 43 21 0.211 Image median 64 51 0.279 mean 56 56 0.246 Owner median 58 62 0.251 mean 34 18 0.172 Image*Owner median 67 53 0.289 21/05/2012 B. HUET - VIGTA12 Keynote - 35
  • Event Detection Results Event Detection Statistics Our Method Venues Ground Truth LastFM Detect Matched Precision Recall Melkweg 69 15 12 0.800 0.174 44 Koko 20 15 8 0.533 0.400 0 HMV Forum 14 12 9 0.750 0.643 14 111 Minna Gallery 23 15 2 0.133 0.087 0 Ancienne Belgique 38 15 9 0.600 0.237 28 Rotown 16 15 8 0.533 0.500 13 Circolo degli Artisti 22 15 8 0.533 0.364 12 Circolo Magnolia 25 3 1 0.333 0.040 11 Hammersmith Apollo 15 15 10 0.667 0.667 14 In total 242 120 67 0.558 0.277 136 21/05/2012 B. HUET - VIGTA12 Keynote - 36
  • Events Detection at Melkweg Detection Results Ground Truth LastFMVenue Date Tags Date Title LastFM Title Parkway Drive / Despised Icon / parkwaydrive drivemelkweg 03/05/2010 03/05/2010 Winds Of Plague / The Warriors / 50 1336473 Parkway Drive parkway Lions flight Flight Of The Conchords - Flight of themelkweg 02/05/2010 flightoftheconchords 02/05/2010 1439320 UITVERKOCHT Conchords conchords Flight Of The Conchords - Flight of themelkweg 04/05/2010 flightoftheconchords 04/05/2010 1439407 UITVERKOCHT Conchords Mayer mayerhawtorne mayermelkweg 05/05/2010 05/05/2010 Mayer Hawthorne & The County 1416229 Hawthorne & hawthorne The Countymelkweg 11/05/2010 bonobo 11/05/2010 Bonobo - UITVERKOCHT 1398102 Bonobomelkweg 14/05/2010 paulweller paul 14/05/2010 Paul Weller - UITVERKOCHT 1406677 Paul Weller Broken Socialmelkweg 18/05/2010 brokensocialscene 18/05/2010 Broken Social Scene - UITVERKOCHT 1334429 Scene Mike Stern band with special guest Richardmelkweg 19/05/2010 mikestern richardbona 19/05/2010 Bona featuring Dave Weckl & Bob Malachmelkweg 25/05/2010 beattimemelkweg 24/05/2010 Beattime - The Kika Editionmelkweg 26/05/2010 beattime 24/05/2010 Beattime - The Kika Edition Off Centre - day 3 - night met Kode 9 /melkweg 28/05/2010 offcentre 28/05/2010 Falty DL / Gold Panda / Kelpemelkweg 30/05/2010 joannanewsom 30/05/2010 Joanna Newsom 1425481 Joanna Newsom 21/05/2012 B. HUET - VIGTA12 Keynote - 37
  • Collage For illustrationShe & Him in Koko 07/05/2010 21/05/2012 B. HUET - VIGTA12 Keynote - 38
  • Conclusions on Event Detection A novel approach for automatically detecting social events is presented The key idea consists in temporally monitoring media shared on social web sites at a specific location (Geo Localized Photo) Automatic Efficient Social Event Detection and Identification can be achieved 21/05/2012 B. HUET - VIGTA12 Keynote - 39
  • Visual Event Modeling X. Liu and B. Huet
  • Objective Automatically collect training data to build event visual appearance models Model training requires both positive and negative examples/samples 21/05/2012 B. HUET - VIGTA12 Keynote - 41
  • Our proposed Automated FrameWork Positive SampleEvent tag1 Pic1 tags tag2 Event Model Top N Pic2 Top M Negative tags Photos Sample tag3 Pic3 tagN ………. PicM …… Rank tags Rank Photos by frequency by distance to tags 21/05/2012 B. HUET - VIGTA12 Keynote - 42
  • Positive Samples Collection Machine Tag Abbreviation of events name  For example “ACMMM12” is the tag to query photos from “ACM Multimedia 2012” 21/05/2012 B. HUET - VIGTA12 Keynote - 43
  • Negative Samples Collection Photos which do not originate from the event. Assumption: Photos taken near the location of the event offer better discriminating power than random photos. Collecting Approach  Collect the data taken near the event„s location and time  Extract tag from the collection, and rank them according to appearance frequency.  Keep the top tags as common tags and use them to rank photos by similarity 21/05/2012 B. HUET - VIGTA12 Keynote - 44
  • The DataSet  10 LastFM concerts, 3 international conference and 1 popular carnival Positive Negative Testing EventID Samples Candidate Pos Neg lastfm:804783 441 1063 466 64 lastfm:1830095 716 748 398 134 lastfm:1858887 408 745 431 266 lastfm:1499065 348 712 16 153 lastfm:1787326 446 913 0 313 lastfm:1351984 307 584 498 19 lastfm:1842684 602 1125 535 78 lastfm:2020655 538 745 750 6 lastfm:1301748 944 541 1157 80 lastfm:1370837 592 1025 592 115 SIGIR2010 100 557 178 23 ACMMM07 30 525 0 201 ACMMM10 118 64 15 44 NICECarnival2011 52 848 60 209 Total 5642 10195 5096 1705 21/05/2012 B. HUET - VIGTA12 Keynote - 45
  • DataSet ExamplesPositive Samples Negative Samples Test Positive Test Negative 21/05/2012 B. HUET - VIGTA12 Keynote - 46
  • Event Model Training Feature:  400D Bag of Words from SIFT features. Model:  SVM implemented with libSVM  RBF kernel  Cross validation is used to optimize the parameters 21/05/2012 B. HUET - VIGTA12 Keynote - 47
  • The (Negative Samples) Model Parameters R: the location distance between photo taken and event venue D: the time-span between photo taken and event taken time -An example on event: lastfm:804783 Conclusion: Use loose parameters for both time interval and location distance 21/05/2012 B. HUET - VIGTA12 Keynote - 48
  • Visual Event Modeling Results Random UniformEventID Query Our Algorithm k-NN Pruning Sample Negativelastfm:804783 87.92 88.68 46.98 50.00 75.85lastfm:1830095 74.81 78.38 80.26 96.62 84.96lastfm:1858887 61.84 63.41 63.56 76.47 73.89lastfm:1499065 9.47 90.53 89.94 92.90 89.35lastfm:1787326 0.00 98.40 92.65 97.12 42.49lastfm:1351984 96.32 96.32 55.32 86.65 93.81lastfm:1842684 87.28 87.93 67.86 79.28 87.11lastfm:2020655 99.21 91.80 71.69 75.00 94.58lastfm:1301748 93.53 93.53 73.73 64.83 93.21lastfm:1370837 83.73 85.15 73.83 60.25 80.62SIGIR2010 0.00 60.19 42.28 16.41 22.38ACMMM07 25.01 57.62 46.61 28.81 27.18ACMMM10 85.83 91.04 87.56 86.57 89.05NICECarnival2011 22.30 76.58 59.10 55.39 56.51Average 69.41 83.31 68.64 70.07 73.42 21/05/2012 B. HUET - VIGTA12 Keynote - 50
  • Conclusions Event-based approach for users to explore, annotate and share media  Improving user experience  Outstanding challenges in interlinking and curating the data Device and User Metadata provide interesting and valuable clues Detecting Events from social media activity Visual Event Media Enrichment 21/05/2012 B. HUET - VIGTA12 Keynote - 51
  • Conclusions and Future Work Combine multiple information sources (Tweets, Social Graph, etc…) to detect and media enrich events.  Meta-Objective: Social Event analysis based on connections between events, media and participants Can the approach be extended to private events?... MediaEval: Social Event Detection Task  www.mediaeval.org 21/05/2012 B. HUET - VIGTA12 Keynote - 52
  • Questions? IEEE Multimedia Special Issue on Large-Scale Multimedia Data Collection (to appear in summer 2012) Thank you for your attention. 21/05/2012 B. HUET - VIGTA12 Keynote - 53