Knowledge Management Institute              Pragmatic Evaluation of Folksonomies                         20th Internationa...
Knowledge Management Institute               Taxonomies: Categorization by Experts                        Taxonomy: Usuall...
Knowledge Management Institute                                   Outline of this talk            1. Folksonomies          ...
Knowledge Management Institute                                   Outline of this talk            1. Folksonomies          ...
Knowledge Management Institute                   Tagging: Social classification by users                                  ...
Knowledge Management Institute                         Construction of Folksonomies   From tag centrality to tag tag centr...
Knowledge Management Institute              Semantic Evaluation of Folksonomies     Emerging Hierarchy         g g        ...
Knowledge Management Institute                                   Outline of this talk            1. Folksonomies          ...
Knowledge Management Institute                                                                        Decentralized Search...
Knowledge Management Institute                                   Outline of this talk            1. Folksonomies          ...
Knowledge Management Institute                        Pragmatic Evaluation Framework            General idea:            •...
Knowledge Management Institute                        Simulating Exploratory Navigation                                   ...
Knowledge Management Institute                                   Outline of this talk            1. Folksonomies          ...
Knowledge Management Institute   Success Rates Across Different Folksonomies                                 flickr datase...
Knowledge Management Institute             Success Rates Across Different Datasets Holds for all                          ...
Knowledge Management Institute                                 Stretch Δ = pLK-pGK                                        ...
Knowledge Management Institute                        Pragmatic Evaluation Framework                      Framework       ...
Knowledge Management Institute                            Results & Findings: Summary            1. Folksonomies are usefu...
Knowledge Management Institute                                            Thank You.                                      ...
Upcoming SlideShare
Loading in...5
×

Pragmatic evaluation of folksonomies

1,443
-1

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,443
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Pragmatic evaluation of folksonomies

  1. 1. Knowledge Management Institute Pragmatic Evaluation of Folksonomies 20th International World Wide Web Conference (WWW2011) Hyderabad, India D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. Lerman Markus Strohmaier Assistant Professor, Graz University of Technology, Austria Visiting Scientist, (XEROX) PARC, USA Markus Strohmaier 2011 1
  2. 2. Knowledge Management Institute Taxonomies: Categorization by Experts Taxonomy: Usually produced and maintained by few (e g dozens of) domain experts (e.g. experts. Alternative: Folk-generated taxonomies („Folksonomies“) ( F lk i “) But how useful are such hierarchical structures? How can they be evaluated? Markus Strohmaier 2011 2
  3. 3. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 3
  4. 4. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 4
  5. 5. Knowledge Management Institute Tagging: Social classification by users Users label and categorize Resources resources with concepts (tags) Tags Users U is a tuple V:= (U, T, R, Y) where • th th the three di j i t fi it sets U T R correspond t disjoint, finite t U, T, d to user 1 – a set of persons or users u ∈ U – a set of tags t ∈ T and – a set of resources or objects r ∈ R tag 1 res. 1 • Y ⊆ U ×T ×R, called set of tag assignments Tag similarity based on users and resources Markus Strohmaier 2011 5
  6. 6. Knowledge Management Institute Construction of Folksonomies From tag centrality to tag tag centrality: F t t lit t high generality: t lit more abstract low tag centrality: more specific Other existing folksonomy algorithms: k-means, affinity propagation, … [Heyman and Garcia-Molina 2006] Markus Strohmaier 2011 6
  7. 7. Knowledge Management Institute Semantic Evaluation of Folksonomies Emerging Hierarchy g g y Expert Hierarchy p y (Emergent) (Golden Standard) via e.g. hierarchical clustering WordNet: a lexical DB for English computers Map- Synset Hierarchy Programming ping programming distance d1 = 1 distance d2 = 2 Python Design g languages g g patterns abs. difference |d1 - d2| a Semantic simple p y for the q p proxy quality y grounding j java python of emergent semantics Markus Strohmaier 2011 8
  8. 8. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 9
  9. 9. Knowledge Management Institute Decentralized Search Idea: use folksonomies as Then, the performance of decentralized search p background knowledge g g Background knowledge: Shortest path to target depends on the suitability of folksonomies. (a tag hierarchy) In other words, we can evaluate the suitability of folksonomies for decentralized search through simulations. Folksonomy Folksonomy Folksonomy 1 ... n shortest path found with A (tag-tag) network: local k l l knowledge pLK = 4 l d Goal: Navigate from START to TARGET Δ = pLK-pGK using local and background knowledge only candidates start target shortest path with p global knowledge pGK = 3 Markus Strohmaier 2011J. Kleinberg. The small-world phenomenon: An algorithmic perspective. Proc. 32nd ACM Symposium on Theory of Computing, 2000. Also appears as Cornell Computer Science Technical Report 99-1776 (October 1999) 10
  10. 10. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 11
  11. 11. Knowledge Management Institute Pragmatic Evaluation Framework General idea: • Use the OUTPUT produced by folksonomy algorithms (hierachical structures) as INPUT (b k (hi hi l t t ) (background d knowledge) for decentralized search. Framework Instantiation K-means, Aff.Prop., 1. Generate n folksonomies DegCentrality, CloCentrality exploratory navigation 2. Model navigational task success rate, stretch 3. Select evaluation metrics decentralized search 4. Simulate navigation 4 Sim late na igation comparative evaluation 5. Evaluate performance Markus Strohmaier 2011 12
  12. 12. Knowledge Management Institute Simulating Exploratory Navigation Topically related START TARGET tags tags resources Topically related Random resources Random R d start resource Usefulness of: page: e.g. landing page from search engine We generate 100.000 search pairs (start, target) for each dataset, andFolksonomy F lkF lk Folksonomy Folksonomy F lk run simulations 1 ... n Markus Strohmaier 2011 13
  13. 13. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 14
  14. 14. Knowledge Management Institute Success Rates Across Different Folksonomies flickr dataset Tag generality approaches k-means / affinity propagation Random folksonomy Success rate: The number of times an agent is successful in finding a path using a particular folksonomy as background knowledge All approaches outperform a random folksonomy y n max hops n: the maximal number of steps an agent Tag generality approaches is allowed to perform before stopping (a tunable outperform k-means / Aff. parameter e.g., an agent only f ll t t l follows n li k ) links). Propagation Markus Strohmaier 2011 16
  15. 15. Knowledge Management Institute Success Rates Across Different Datasets Holds for all But how datasets efficient are (to diff. diff those extents) folksonomies during search? Markus Strohmaier 2011 17
  16. 16. Knowledge Management Institute Stretch Δ = pLK-pGK p Shortest Paths found with Local Knowledge Bibsonomy K M Bib K-Means Finds no path: Δ = infinite Finds paths that is +1 longer: Δ=1 Holds for all datasets d t t Finds shortest possible path: Tag T generality lit (to diff. Δ=0 approaches (d+e) extents) find much shorter paths! Markus Strohmaier 2011 18
  17. 17. Knowledge Management Institute Pragmatic Evaluation Framework Framework Instantiation Alternatives K-means, Aff.Prop., other folksonomy 1. Generate n folksonomies DegCentrality, algorithms or CloCentrality expert taxonomies exploratory other tasks 2. Model navigational task navigation success rate, stretch other evaluation metrics 3. 3 Select evaluation metrics decentralized search actual click data 4. Simulate navigation comparative other evaluation 5. Evaluate performance evaluation approaches Pragmatic evaluation produces different results for different tasks and different assumed or observed navigation behavior. The evaluation framework is completely general with regard to the task, data and evaluation metrics adopted. Markus Strohmaier 2011 19
  18. 18. Knowledge Management Institute Results & Findings: Summary 1. Folksonomies are useful b k 1 F lk i f l background k d knowledge f l d for navigation. 2. Existing folksonomy algorithms are more useful than a random baseline. baseline 3. 3 Tag generality approaches perform better than existing hierarchical clustering approaches. 4. Pragmatic results support theoretical analysis (not presented in talk – included in paper). Markus Strohmaier 2011 20
  19. 19. Knowledge Management Institute Thank You. Th k Y Markus Strohmaier markus.strohmaier@tugraz.at D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. Lerman Pragmatic Evaluation of Folksonomies 20th International World Wide Web Conference (WWW2011) Hyderabad, India, March 28 - April 1, ACM, 2011. http://kmi.tugraz.at/staff/markus/documents/2011_WWW2011_Folksonomies.pdf Markus Strohmaier 2011 21
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×