The document discusses the CUbRIK project, which aims to build a social graph from image archives by reconstructing and exploring social ties through historical sources. It involves four pillars: close collaboration with researchers, a structured repository of historical persons and events, efficient face identification indexing, and analysis/visualization tools. Challenges include determining the truth given differing humanities and computer science perspectives, engaging crowds, and addressing image copyrights. The project seeks to refine its tools based on researcher requirements to better analyze relationships and social networks over time from multimedia sources.

1. histoGraph
Building a Social graph from image archives
La science et les effets de réseau, Lyon
24.03.2014
www.cubrikproject.eu
Lars Wieneke, CVCE, Luxembourg 0

2. The CVCE
Lars Wieneke, CVCE, Luxembourg 1

3. About CUbRIK
 European Community's Seventh Framework
Program FP7-ICT
 15 European partners
 Multimedia search
processing: Putting
humans in the loop
 Demos: History of Europe
and Fashion
Lars Wieneke, CVCE, Luxembourg 2

4. Point of departure
Images as sources
Lars Wieneke, CVCE, Luxembourg 3

5. Goal: Reconstructing and exploring social ties
through historical sources
Lars Wieneke, CVCE, Luxembourg 4

6. Towards the social graph
4 pillars
1. Close connection to the requirements of
researchers in European Integration studies
2. Structured and referencable repository of
persons, events and places in time
3. Efficient indexation process that enables the
association of faces with identities
4. Toolchain for analysis and visualization
Lars Wieneke, CVCE, Luxembourg 5

7. Towards the social graph
Sourcing researcher requirements
Lars Wieneke, CVCE, Luxembourg 6
Selection of target
user group
First draft of the app
scenario
Feedback on
technical scope
Exploratory
interviews
(daily work practices)
Second draft of the
app scenario
Focus group
(user needs and app
scenarios)
Feedback on
technical feasability
Lessons learned:
issues and features
Specification
Implementation 1.
demonstrator
Workshop: Review of
app and features
Revised specification
Implementation 2.
demonstrator
Evaluation and test
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
Users
Requirements
Technology

8. Towards the social graph
Structured repository
Lars Wieneke, CVCE, Luxembourg 7

9. Towards the social graph
Indexation process
8
Raw content
High level features
(automatic annotations)
Conflict
(e.g., “Image contains
‘Romano Prodi’ ”
? Confidence = low)
Conflict store Conflict
manager
Conflict resolution
task store
Conflict resolution
task: conflict,
required skill, priority, ..
CUbRIK app
for Conflict
resolution
Game Crowdtask Q&A
Lars Wieneke, CVCE, Luxembourg

10. 9
Towards the social graph
Indexation process II
 Human in the loop added value:
 Verification of identities/places/events ambiguous and temporal only
possible by putting humans in the loop
 Integration of multiple perspectives
 CUbRIK as an open toolbox allows
follow-up and extension through
third parties
 “Vertical” integration:
GUI, components, crowdsourcing
integrated in a platform
Lars Wieneke, CVCE, Luxembourg

11. 10
Towards the social graph
Visualization and analysis
Lars Wieneke, CVCE, Luxembourg

12. 11
Challenges & Approach
 Main challenges
 Detection and identification of identities/places/events in time
 Verification of identities/places/events in time
 Analysis of relationships (e.g. co-occurrences)
 Rights aware crawling and storage
 Verification of provenance and license information
 Truth and provenance
 Approach
 Crowd-sourced verification of detected faces (false positives/negatives)
 Verification of identities through/places/events in time social networks of
experts
 Visual knowledge discovery/exploration
 Integrated rights aware crawling and storage
 Integrated license and provenance management
27/11/2013 Lars Wieneke, CVCE, Luxembourg

13. Towards the social graph
Bringing it all together
Social Graph Network Analysis
CROWD
Research
Inquiry
Graph
Visualizat ion
Lars Wieneke, CVCE, Luxembourg 12
Image Indexation
Media
harvest ing
and upload
Face
detect ion
Face
ident ifi cat ion
Clickworkers
Crowd Face
posit ion
validat ion
Copyright
aware
crawler
Provenance
checker
License
checker
Content
provider
tools
Metadata
Ent ity
extract ion
Identity reconciliation
Ent ity
verifi cat ion &
annotat ion
Ent itypedia
Integrat ion
CROWD
pre-fi
ltering
Text Indexation
Connect ion
to the CVCE
collect ion
Ent ity
anntat ion and
extract ion
Expert Crowd
Expert
CROWD
verifi cat ion
Ent itypedia
Integrat ion
CROWD Research Inquieries
Expert Crowd
Graph
Visualizat ion
Analysis of
the social
graph
Graph Query (old: Query for Entities)
Graph
Visualizat ion
Query for
ent it ies
Context Expander
Expansion
through
documents
Expansion
through
videos
Expansion
through
images
Expansion
through
related
ent it ies
Social Graph construction
Social
Graph
Creat ion
Content Analysis and
Enrichment
Querying
Feedback acquisition
and processing
Y3 component
Query for
spat ial
constraints
EXP through
SIMILAR
images
WP Event
detect ion

14. histoGraph demo
Time for a Demo!
Lars Wieneke, CVCE, Luxembourg 13

15. Pipelining the CUbRIK components:
Human input from click-workers
Great choice for simple tasks:
 Face detection: false positives, false negatives
 Monetary motivation, via www.microtask.com
Poor performance on complex tasks:
 Low resolution images
 Different angles etc.
 Actors recurring over time
Lars Wieneke, CVCE, Luxembourg 14

16. Pipelining the CUbRIK components:
Human input from experts
Capable of complex tasks:
 In-depth knowledge of key actors
 Context knowledge allows inferences
But: Different motivational models!
 Public goods
 Reputation
Lars Wieneke, CVCE, Luxembourg 15

17. Usage for historians
 No one truth in history but interpretation,
context and discussion
 Therefore need to represent ambivalence,
contradictions and discussion
 Close ties between data representation (Social
graph) and their original context (primary
sources)
Lars Wieneke, CVCE, Luxembourg 16

18. Conclusion
 Challenges
 What is truth? Humanities vs. Computer Science
 Gathering requirements for tools that haven‘t been
developed yet
 Engaging crowds
 Image copyrights
 Scientific value?
 Refinement of the application
 Additional datasources
 Improvement of the interface
 Integration of the new components
Lars Wieneke, CVCE, Luxembourg 17

19. Outlook
CUbRIK Presentation 18

20. Outlook
CUbRIK Presentation 19

21. Outlook histoGraph
CUbRIK Presentation 20

22. Visit us on
WWW.CUBRIKPROJECT.EU
@CUBRIKPROJECT
CUbRIK Presentation 21
Or follow us on Twitter