1/5/2013 CUbRIK Presentation 0Building social graphs from imagesthrough expert-based crowdsourcingM. Dionisio, P. Fraternali, D. Martinenghi, C. Pasini, M.Tagliasacchi, S. Zagorac (Politecnico Di Milano, Italy)E. Harloff, I. Micheel, J. Novak(European Institute for Participatory Media, Germany)
1/5/2013 CUbRIK Presentation 1The CUbRIK project CUbRIK is a research projectfinanced by the EuropeanUnion whose main goals are:1. Advance the architecture ofmultimedia search2. Exploit the human contributionin multimedia search3. Use open source componentsprovided by the community4. Start up a search businessecosystem
1/5/2013 CUbRIK Presentation 2The CUbRIK architecture The CUbRIK architecture islayered in four main tiers1. Content and useracquisition tier2. Content processing tier3. Query processing tier4. Search tier
1/5/2013 CUbRIK Presentation 3History Of Europe use caseHoE Dataset(3924 pictures shotfrom the end ofWorld War II to themost recent years ofEU history)Automatic facerecognition tool+Crowdsourcedvalidation offace matchesSocial Graph
1/5/2013 CUbRIK Presentation 4Content processing pipeline In the initial proof of concept we designed a prototype for aface recognition service that combined automatic mechanismsfor face detection/recognition and a general purpose crowd.Group photosFacedetectionBounding boxesFacematchingAnnotated portraitsFacedetectionBounding boxesTop – 10similaritiesfor crowdvalidation
1/5/2013 CUbRIK Presentation 5Limits of a purely automatic processingFalsenegativesFalsepositives
1/5/2013 CUbRIK Presentation 6Limits of a purely automatic processingMatching score = 0.185Matching score = 0.210The matching score between two faces of thesame person is not always the highest one
1/5/2013 CUbRIK Presentation 7Using general purpose crowds We interfaced a general purpose crowd for the validation of thetop-10 matches.
1/5/2013 CUbRIK Presentation 8Results of the first proof of concept 574 faces extracted from group photos Only 17% of them were identified by the crowd Of this 17% the 66% of the matches were correct The automatic tool identified the 80% of thefaces correctly
1/5/2013 CUbRIK Presentation 9Results of the first proof of concept These weak results were influenced by severalfactors:1. Influence of image taking times2. Limited size of the ground truth3. Image resolution constraints4. Replicability and trustworthiness of the results
1/5/2013 CUbRIK Presentation 10Interfacing the expert based crowd The deficiencies encountered using a general purpose crowdcan be overcome by adopting an expert-based crowdsourcing.combined implicit and explicitexpert-based crowdsourcinginterface
1/5/2013 CUbRIK Presentation 11Interfacing the expert based crowd Indications suggest that the expert-based strategy cansucceed:1. Experts’ knowledge can overcome the drawbacks bothof the automatic tool and of the general purpose crowd2. They can use the already existing community means tocontact colleagues and cooperate to fulfill the task.
1/5/2013 CUbRIK Presentation 12Interfacing the expert based crowdThank you!