1. The CUbRIK project combines automatic face recognition tools with crowdsourcing to build social graphs from images. It uses a four-tier architecture to process content and queries. 2. An initial proof of concept used a general purpose crowd to validate top face matches identified by an automatic tool on a dataset of European history photos, but achieved only weak results due to image quality issues and the non-expert crowd. 3. The presentation argues that using an expert-based crowdsourcing approach could overcome the limitations seen with solely automatic processing and general purpose crowds, by leveraging experts' domain knowledge and ability to collaborate within their communities.

1. 1/5/2013 CUbRIK Presentation 0
Building social graphs from images
through expert-based crowdsourcing
M. Dionisio, P. Fraternali, D. Martinenghi, C. Pasini, M.
Tagliasacchi, S. Zagorac (Politecnico Di Milano, Italy)
E. Harloff, I. Micheel, J. Novak
(European Institute for Participatory Media, Germany)

2. 1/5/2013 CUbRIK Presentation 1
The CUbRIK project
 CUbRIK is a research project
financed by the European
Union whose main goals are:
1. Advance the architecture of
multimedia search
2. Exploit the human contribution
in multimedia search
3. Use open source components
provided by the community
4. Start up a search business
ecosystem

3. 1/5/2013 CUbRIK Presentation 2
The CUbRIK architecture
 The CUbRIK architecture is
layered in four main tiers
1. Content and user
acquisition tier
2. Content processing tier
3. Query processing tier
4. Search tier

4. 1/5/2013 CUbRIK Presentation 3
History Of Europe use case
HoE Dataset
(3924 pictures shot
from the end of
World War II to the
most recent years of
EU history)
Automatic face
recognition tool
+
Crowdsourced
validation of
face matches
Social Graph

5. 1/5/2013 CUbRIK Presentation 4
Content processing pipeline
 In the initial proof of concept we designed a prototype for a
face recognition service that combined automatic mechanisms
for face detection/recognition and a general purpose crowd.
Group photos
Face
detection
Bounding boxes
Face
matching
Annotated portraits
Face
detection
Bounding boxes
Top – 10
similarities
for crowd
validation

6. 1/5/2013 CUbRIK Presentation 5
Limits of a purely automatic processing
False
negatives
False
positives

7. 1/5/2013 CUbRIK Presentation 6
Limits of a purely automatic processing
Matching score = 0.185
Matching score = 0.210
The matching score between two faces of the
same person is not always the highest one

8. 1/5/2013 CUbRIK Presentation 7
Using general purpose crowds
 We interfaced a general purpose crowd for the validation of the
top-10 matches.

9. 1/5/2013 CUbRIK Presentation 8
Results of the first proof of concept
 574 faces extracted from group photos
 Only 17% of them were identified by the crowd
 Of this 17% the 66% of the matches were correct
 The automatic tool identified the 80% of the
faces correctly

10. 1/5/2013 CUbRIK Presentation 9
Results of the first proof of concept
 These weak results were influenced by several
factors:
1. Influence of image taking times
2. Limited size of the ground truth
3. Image resolution constraints
4. Replicability and trustworthiness of the results

11. 1/5/2013 CUbRIK Presentation 10
Interfacing the expert based crowd
 The deficiencies encountered using a general purpose crowd
can be overcome by adopting an expert-based crowdsourcing.
combined implicit and explicit
expert-based crowdsourcing
interface

12. 1/5/2013 CUbRIK Presentation 11
Interfacing the expert based crowd
 Indications suggest that the expert-based strategy can
succeed:
1. Experts’ knowledge can overcome the drawbacks both
of the automatic tool and of the general purpose crowd
2. They can use the already existing community means to
contact colleagues and cooperate to fulfill the task.

13. 1/5/2013 CUbRIK Presentation 12
Interfacing the expert based crowd
Thank you!