1. 1/5/2013 CUbRIK Presentation 0
Building social graphs from images
through expert-based crowdsourcing
M. Dionisio, P. Fraternali, D. Martinenghi, C. Pasini, M.
Tagliasacchi, S. Zagorac (Politecnico Di Milano, Italy)
E. Harloff, I. Micheel, J. Novak
(European Institute for Participatory Media, Germany)
2. 1/5/2013 CUbRIK Presentation 1
The CUbRIK project
CUbRIK is a research project
financed by the European
Union whose main goals are:
1. Advance the architecture of
multimedia search
2. Exploit the human contribution
in multimedia search
3. Use open source components
provided by the community
4. Start up a search business
ecosystem
3. 1/5/2013 CUbRIK Presentation 2
The CUbRIK architecture
The CUbRIK architecture is
layered in four main tiers
1. Content and user
acquisition tier
2. Content processing tier
3. Query processing tier
4. Search tier
4. 1/5/2013 CUbRIK Presentation 3
History Of Europe use case
HoE Dataset
(3924 pictures shot
from the end of
World War II to the
most recent years of
EU history)
Automatic face
recognition tool
+
Crowdsourced
validation of
face matches
Social Graph
5. 1/5/2013 CUbRIK Presentation 4
Content processing pipeline
In the initial proof of concept we designed a prototype for a
face recognition service that combined automatic mechanisms
for face detection/recognition and a general purpose crowd.
Group photos
Face
detection
Bounding boxes
Face
matching
Annotated portraits
Face
detection
Bounding boxes
Top – 10
similarities
for crowd
validation
7. 1/5/2013 CUbRIK Presentation 6
Limits of a purely automatic processing
Matching score = 0.185
Matching score = 0.210
The matching score between two faces of the
same person is not always the highest one
8. 1/5/2013 CUbRIK Presentation 7
Using general purpose crowds
We interfaced a general purpose crowd for the validation of the
top-10 matches.
9. 1/5/2013 CUbRIK Presentation 8
Results of the first proof of concept
574 faces extracted from group photos
Only 17% of them were identified by the crowd
Of this 17% the 66% of the matches were correct
The automatic tool identified the 80% of the
faces correctly
10. 1/5/2013 CUbRIK Presentation 9
Results of the first proof of concept
These weak results were influenced by several
factors:
1. Influence of image taking times
2. Limited size of the ground truth
3. Image resolution constraints
4. Replicability and trustworthiness of the results
11. 1/5/2013 CUbRIK Presentation 10
Interfacing the expert based crowd
The deficiencies encountered using a general purpose crowd
can be overcome by adopting an expert-based crowdsourcing.
combined implicit and explicit
expert-based crowdsourcing
interface
12. 1/5/2013 CUbRIK Presentation 11
Interfacing the expert based crowd
Indications suggest that the expert-based strategy can
succeed:
1. Experts’ knowledge can overcome the drawbacks both
of the automatic tool and of the general purpose crowd
2. They can use the already existing community means to
contact colleagues and cooperate to fulfill the task.
