The WeKnowIt Project


Published on

A summary presentation of the WeKnowIt European research project that was given at the ICMR2011 conference.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The WeKnowIt Project

  1. 1. Emerging, Collective Intelligence for Personal, Organisationaland Social UseSymeon Papadopoulos,Yiannis Kompatsiaris (CERTH-ITI)www.weknowit.euTrento, April 20ICMR 2011
  2. 2. overviewmotivation & concept approach intelligence conclusions layers results
  3. 3. motivation• Users upload, tag, share, connect & search  availability of massive amounts of user-generated content and data• Existing applications are limited to simple user data management or shallow analysis• Potential for much more if we mine the data and exploit them in the right context
  4. 4. collective intelligence…a form of intelligence emerging from online user activities Collective Intelligence >> sum of individuals’ intelligences
  5. 5. an exampleone of my photos @ flickr my location: N/A my tags: wki experiment bcn (…pretty uninformative)
  6. 6. an example one of my photos @ flickr others’ photos @ flickr tags location
  7. 7. an example alternative views / trends / facts one of my photos @ flickr
  8. 8. an example my friend’s photos @ flickr one of my photos @ flickr what did he visit next?
  9. 9. an example one of my photos @ flickr related Linked Data
  10. 10. collective intelligence @ weknowit personal intelligence media intelligence mass intelligence social intelligence organizational intelligence
  11. 11. overviewmotivation & concept approach intelligence conclusions layers results
  12. 12. personal intelligence
  13. 13. personal intelligence
  14. 14. media intelligence Visual Exploration
  15. 15. media intelligence
  16. 16. mass intelligence
  17. 17. mass intelligence Tag Clustering
  18. 18. social intelligence Visualise Communities
  19. 19. social intelligence Community Browser
  20. 20. organisational intelligence Event-based Knowledge Sharing
  21. 21. organisational intelligence Distributed Group Management
  22. 22. architecture / integration Service Integration Knowledge and Content Storage Scenario-driven Service Composition
  23. 23. use case: emergency response Media Intelligence Personal Intelligence Photo arrives at ER control centre >> Automatic localisation of photo >> Login, Upload >> Photo & speech auto-tagging >> Spam detection >> Personalized Access Mass Intelligence >> Clustering >> Enrichment from additional sources Social Intelligence >> ER Alert Service >> Reputation Service Organisational Intelligence >> Log Merging & Viewing >> Incident Information Access
  24. 24. use case: travel Travel Preparation Mass Intelligence >> Landmark & Event detection >> Ranked facet lists of POIs >> Hybrid Image Clustering Media Intelligence >> Image Localisation >> Tag suggestions Mobile Guidance Personal Intelligence Post Travel >> Personal Recommendations Social Intelligence >> Group profiling & recommendations >> Friends position, alert
  25. 25. case: community detection in social media (1/2)• Structural similarity + Local expansion (highly efficient and scalable approach)• Not necessary to know the number of clusters• Noise resilient (not all nodes need to be part of a + community)• Generic approach adaptable to many applications (depending on node – edge representation) S. Papadopoulos, Y. Kompatsiaris, A. Vakali. “A Graph-based Clustering Scheme for Identifying Related Tags in Folksonomies”. In Proceedings of DaWaK10, Springer-Verlag, 65-76
  26. 26. case: community detection in social media (2/2) PHOTOS & METADATA SPATIAL CLUSTERING + TEMPORAL ANALYSIS tags: sagrada familia, cathedral, barcelona taken: 12 May 2009 lat: 41.4036, lon: 2.1743 CLASSIFICATION TO LANDMARKS/EVENTS COMMUNITY DETECTION VISUAL TAG HYBRIDS. Papadopoulos, C. Zigkolis, Y. Kompatsiaris, A. Vakali. “Cluster-based Landmark and Event Detection on Tagged PhotoCollections”. In IEEE Multimedia Magazine 18(1), pp. 52-63, 2011
  27. 27. overviewmotivation & concept approach intelligence conclusions layers results
  28. 28. results: research• User modeling & interaction (CURIO, attention streams)• Media understanding (photo/text localization, photo/speech auto-tagging)• Media organization (graph-based clustering, faceted search, event detection)• Community analysis & management (administration, browsing, reputation, notification)• Knowledge representation & management (Event Model F, dgFOAF)
  29. 29. results: applications Prototypes• ER (desktop & mobile)• Travel (trip planning, mobile guidance, post-travel photo management)Stand-alone applications• WKI image recognizer• VIRAL (visual search and automatic localization)• ClustTour (city exploration by use of photo clusters)• Semaplorer++• STEVIE (mobile POI management)
  30. 30. results: exploitation VIRAL evaluation by Vodafone 360
  31. 31. results: public APIs
  32. 32. far• CI emerges from massive online activities• it is hard to extract and manage• ...but is definitely worth the the future...• other domains: news, finance, e-gov• real-time CI• CI  Linked Data
  33. 33. thank you!Presentation online @ > news
  34. 34. Additional Slides
  35. 35. content in weknowit offline  model creation, training Standard annotated corpora used for training. • Single-modality: text (Brown corpus), speech (TIMIT database), standard training data image (Corel database) • Single-source: prepared by a single person/organization • Consistent quality: absence of spam, malicious or erroneous data • Small-moderate volume: Manually produced Massive user generated content and feedback from Web 2.0 applications • Multi-modality: e.g. image + tags, image + geo-location + time massive Web 2.0 • Multi-source: may be generated by different applications, user communities, e.g. Flickr, Panoramio, PhotoBucket • Inconsistent quality: noise, spam, ambiguity • Huge volume: Massively produced and disseminated online  user profiling, method invocation Online content and user actions by WeKnowIt users. It is mainly used for WKI user-contributed triggering WeKnowIt services and for providing context to them, e.g. user profile, input content to be used as example for querying, etc.
  36. 36. technical approachVariety of approaches depending on content-metadata input. massive Web 2.0 – massive Web 2.0 – standard training data semi-structured unstructured WKI user-contributed standard Statistical approaches Content analysis Knowledge BasedProbabilistic models Text models (n-gram, LDA, CRF) Lookup (WordNet)(pLSA, Bag-Of-Words) Image processing Thesaurus Lookup (GeoPlanet)Graph-based approaches (visual feature extraction) Concept detection (SNA, community detection) Speech modeling (Wikipedia, domain (spectral analysis, HMM) ontologies)
  37. 37. massive Web 2.0 WKI user-contributed standard training data Locations Topics Social connectionsWP1 Get recommendations Tag normalization Emergency alert service Tag processing WP4 Visual analysis WP2 Community analysis toolWP2 Text classification Text annotation ClustTour Events POI recommendation WP3 Speech search Local tag community detector WP2WP3 POI clustering Semantic photo query Search place POI Entities Named entity detection Log mergerWP5 csxPOIs WP3 WP5 Entity facet extraction - ranking Semaplorer(++) Representation Access CURIO Storage Account Manager WP1 WP1 VERACITY Login WP2 Speech Indexing WP5 Event model F + M3O WP4 Community administration platform WP6 Data Storage WP6 Common data model WP5 Group Management GUI ER CSG WP6 Mobile app Travel preparation Manage Item Comment System IntegrationWP1 Tag Users messaging Desktop proto WP7 Mobile guidance WP3 Search Knowledge Base Lexical Spam Detector Post ER tool Post-travel logging
  38. 38. weknowit work structure WP9: Management management WP1: Personal Intelligence WP6: Architecture / Integration WP2: Media Intelligence WP3: Mass Intelligence WP7.I Use Case: ER WP4: Social Intelligence WP7.II Use Case: Travel WP5: Organisational Intelligence research development WP8: Dissemination & Exploitation dissemination & exploitation
  39. 39. Causality Pattern in Event-Model-F• Event (cause) implies other event (effect)• Causal relationship holds under some justification• Causes and effects are events, and only events
  40. 40. OntoMDE• Specification of MoOn using eCore and OAM as UML2 class diagram• Transformation steps implemented• Evaluation with ontologies of different complexity
  41. 41. Content vs. Structure Concepts
  42. 42. results: disseminationActivities• Collective Intelligence Workshops and Special Session• Summer schoolsPublications• 8 journal publications • Trans. on MultiMedia, IEEE MultiMedia, J. of Web Semantics, MTAP, etc.• 59 conference papers • ACM MultiMedia, SIGIR, CVPR, ESWC, WWW, ICIP, WSDM, etc.• 2 CI book chapters + 1 CI White Paper• 3 patent applications