A summary presentation of the WeKnowIt European research project that was given at the ICMR2011 conference.

1. Emerging, Collective Intelligence for Personal, Organisational
and Social Use
Symeon Papadopoulos,
Yiannis Kompatsiaris (CERTH-ITI)
www.weknowit.eu
Trento, April 20
ICMR 2011

2. overview
motivation &
concept
approach
intelligence
conclusions layers
results

3. motivation
• Users upload, tag, share, connect & search 
availability of massive amounts of user-generated
content and data
• Existing applications are limited to simple user data
management or shallow analysis
• Potential for much more if we mine the data and exploit them
in the right context

4. collective intelligence
…a form of intelligence emerging from online user activities
Collective Intelligence >> sum of individuals’ intelligences

5. an example
one of my photos @ flickr
my location: N/A
my tags: wki experiment bcn
(…pretty uninformative)

6. an example
one of my photos @ flickr others’ photos @ flickr
tags
location

7. an example
alternative views / trends / facts
one of my photos @ flickr

8. an example my friend’s photos @ flickr
one of my photos @ flickr
what did he visit next?

9. an example
one of my photos @ flickr related Linked Data

10. collective intelligence @ weknowit
personal intelligence
media intelligence
mass intelligence
social intelligence
organizational intelligence

11. overview
motivation &
concept
approach
intelligence
conclusions layers
results

12. personal intelligence

13. personal intelligence

14. media intelligence
Visual Exploration

15. media intelligence

16. mass intelligence

17. mass intelligence
Tag Clustering

18. social intelligence
Visualise
Communities

19. social intelligence
Community Browser

20. organisational intelligence
Event-based Knowledge
Sharing

21. organisational intelligence
Distributed Group
Management

22. architecture / integration
Service Integration
Knowledge and Content Storage
Scenario-driven Service Composition

23. use case: emergency response
Media Intelligence
Personal Intelligence Photo arrives at ER control centre
>> Automatic localisation of photo
>> Login, Upload
>> Photo & speech auto-tagging
>> Spam detection
>> Personalized Access
Mass Intelligence
>> Clustering
>> Enrichment from additional sources
Social Intelligence
>> ER Alert Service
>> Reputation Service
Organisational Intelligence
>> Log Merging & Viewing
>> Incident Information Access

24. use case: travel Travel Preparation
Mass Intelligence
>> Landmark & Event detection
>> Ranked facet lists of POIs
>> Hybrid Image Clustering
Media Intelligence
>> Image Localisation
>> Tag suggestions Mobile Guidance
Personal Intelligence
Post Travel >> Personal Recommendations
Social Intelligence
>> Group profiling & recommendations
>> Friends position, alert

25. case: community detection in social media (1/2)
• Structural similarity + Local expansion
(highly efficient and scalable approach)
• Not necessary to know the number
of clusters
• Noise resilient
(not all nodes need to be part of a
+
community)
• Generic approach adaptable to
many applications
(depending on node – edge
representation)
S. Papadopoulos, Y. Kompatsiaris, A. Vakali. “A Graph-based Clustering Scheme for Identifying Related Tags in Folksonomies”.
In Proceedings of DaWaK'10, Springer-Verlag, 65-76

26. case: community detection in social media (2/2)
PHOTOS & METADATA
SPATIAL CLUSTERING + TEMPORAL ANALYSIS
tags: sagrada familia,
cathedral, barcelona
taken: 12 May 2009
lat: 41.4036, lon: 2.1743
CLASSIFICATION TO LANDMARKS/EVENTS
COMMUNITY DETECTION
VISUAL
TAG
HYBRID
S. Papadopoulos, C. Zigkolis, Y. Kompatsiaris, A. Vakali. “Cluster-based Landmark and Event Detection on Tagged Photo
Collections”. In IEEE Multimedia Magazine 18(1), pp. 52-63, 2011

27. overview
motivation &
concept
approach
intelligence
conclusions layers
results

28. results: research
• User modeling & interaction (CURIO, attention streams)
• Media understanding
(photo/text localization, photo/speech auto-tagging)
• Media organization
(graph-based clustering, faceted search, event detection)
• Community analysis & management
(administration, browsing, reputation, notification)
• Knowledge representation & management
(Event Model F, dgFOAF)

29. results: applications
http://www.weknowit.eu/tr
Integrated Prototypes
• ER (desktop & mobile)
• Travel (trip planning, mobile guidance, post-travel photo management)
Stand-alone applications
• WKI image recognizer
• VIRAL (visual search and automatic localization)
• ClustTour (city exploration by use of photo clusters)
• Semaplorer++
• STEVIE (mobile POI management)

30. results: exploitation
VIRAL evaluation by Vodafone 360

31. results: public APIs
http://mklab.iti.gr/wki-apps

32. conclusions
...so far
• CI emerges from massive online activities
• it is hard to extract and manage
• ...but is definitely worth the effort.
in the future...
• other domains: news, finance, e-gov
• real-time CI
• CI  Linked Data

33. thank you!
Presentation online @ http://www.weknowit.eu > news

34. Additional Slides

35. content in weknowit
offline  model creation, training
Standard annotated corpora used for training.
• Single-modality: text (Brown corpus), speech (TIMIT database),
standard training data image (Corel database)
• Single-source: prepared by a single person/organization
• Consistent quality: absence of spam, malicious or erroneous data
• Small-moderate volume: Manually produced
Massive user generated content and feedback from Web 2.0 applications
• Multi-modality: e.g. image + tags, image + geo-location + time
massive Web 2.0 • Multi-source: may be generated by different applications, user communities,
e.g. Flickr, Panoramio, PhotoBucket
• Inconsistent quality: noise, spam, ambiguity
• Huge volume: Massively produced and disseminated
online  user profiling, method invocation
Online content and user actions by WeKnowIt users. It is mainly used for
WKI user-contributed triggering WeKnowIt services and for providing context to them, e.g.
user profile, input content to be used as example for querying, etc.

36. technical approach
Variety of approaches depending on content-metadata input.
massive Web 2.0 –
massive Web 2.0 – standard training data semi-structured
unstructured
WKI user-contributed standard
Statistical approaches Content analysis Knowledge Based
Probabilistic models Text models (n-gram, LDA, CRF) Lookup (WordNet)
(pLSA, Bag-Of-Words) Image processing Thesaurus Lookup (GeoPlanet)
Graph-based approaches (visual feature extraction) Concept detection
(SNA, community detection) Speech modeling (Wikipedia, domain
(spectral analysis, HMM) ontologies)

37. massive Web 2.0 WKI user-contributed standard training data
Locations Topics Social connections
WP1 Get recommendations Tag normalization
Emergency alert service
Tag processing WP4
Visual analysis WP2 Community analysis tool
WP2 Text classification
Text annotation
ClustTour
Events
POI recommendation WP3 Speech search
Local tag community detector WP2
WP3 POI clustering Semantic photo query
Search place POI Entities
Named entity detection Log merger
WP5 csxPOIs WP3 WP5
Entity facet extraction - ranking Semaplorer(++)
Representation Access
CURIO Storage Account Manager
WP1 WP1
VERACITY Login
WP2 Speech Indexing
WP5 Event model F + M3O WP4 Community administration platform
WP6 Data Storage
WP6 Common data model WP5 Group Management
GUI ER CSG
WP6 Mobile app Travel preparation
Manage Item Comment
System Integration
WP1 Tag Users messaging Desktop proto WP7 Mobile guidance
WP3
Search Knowledge Base Lexical Spam Detector Post ER tool Post-travel logging

38. weknowit work structure
WP9: Management
management
WP1: Personal Intelligence
WP6: Architecture / Integration
WP2: Media Intelligence
WP3: Mass Intelligence WP7.I Use Case: ER
WP4: Social Intelligence
WP7.II Use Case: Travel
WP5: Organisational Intelligence
research development
WP8: Dissemination & Exploitation
dissemination & exploitation

39. Causality Pattern in Event-Model-F
• Event (cause) implies other event (effect)
• Causal relationship holds under some justification
• Causes and effects are events, and only events

40. OntoMDE
• Specification of MoOn using eCore and OAM as UML2 class diagram
• Transformation steps implemented
• Evaluation with ontologies of different complexity

41. Content vs. Structure Concepts

42. results: dissemination
Activities
• Collective Intelligence Workshops and Special Session
• Summer schools
Publications
• 8 journal publications
• Trans. on MultiMedia, IEEE MultiMedia, J. of Web Semantics, MTAP, etc.
• 59 conference papers
• ACM MultiMedia, SIGIR, CVPR, ESWC, WWW, ICIP, WSDM, etc.
• 2 CI book chapters + 1 CI White Paper
• 3 patent applications