Recommended
More Related Content
Viewers also liked
Similar to Pragmatic evaluation of folksonomies
Similar to Pragmatic evaluation of folksonomies (20)
Recently uploaded
Recently uploaded (20)
Pragmatic evaluation of folksonomies
- 1. Knowledge Management Institute Pragmatic Evaluation of Folksonomies 20th International World Wide Web Conference (WWW2011) Hyderabad, India D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. Lerman Markus Strohmaier Assistant Professor, Graz University of Technology, Austria Visiting Scientist, (XEROX) PARC, USA Markus Strohmaier 2011 1
- 2. Knowledge Management Institute Taxonomies: Categorization by Experts Taxonomy: Usually produced and maintained by few (e g dozens of) domain experts (e.g. experts. Alternative: Folk-generated taxonomies („Folksonomies“) ( F lk i “) But how useful are such hierarchical structures? How can they be evaluated? Markus Strohmaier 2011 2
- 3. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 3
- 4. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 4
- 5. Knowledge Management Institute Tagging: Social classification by users Users label and categorize Resources resources with concepts (tags) Tags Users U is a tuple V:= (U, T, R, Y) where • th th the three di j i t fi it sets U T R correspond t disjoint, finite t U, T, d to user 1 – a set of persons or users u ∈ U – a set of tags t ∈ T and – a set of resources or objects r ∈ R tag 1 res. 1 • Y ⊆ U ×T ×R, called set of tag assignments Tag similarity based on users and resources Markus Strohmaier 2011 5
- 6. Knowledge Management Institute Construction of Folksonomies From tag centrality to tag tag centrality: F t t lit t high generality: t lit more abstract low tag centrality: more specific Other existing folksonomy algorithms: k-means, affinity propagation, … [Heyman and Garcia-Molina 2006] Markus Strohmaier 2011 6
- 7. Knowledge Management Institute Semantic Evaluation of Folksonomies Emerging Hierarchy g g y Expert Hierarchy p y (Emergent) (Golden Standard) via e.g. hierarchical clustering WordNet: a lexical DB for English computers Map- Synset Hierarchy Programming ping programming distance d1 = 1 distance d2 = 2 Python Design g languages g g patterns abs. difference |d1 - d2| a Semantic simple p y for the q p proxy quality y grounding j java python of emergent semantics Markus Strohmaier 2011 8
- 8. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 9
- 9. Knowledge Management Institute Decentralized Search Idea: use folksonomies as Then, the performance of decentralized search p background knowledge g g Background knowledge: Shortest path to target depends on the suitability of folksonomies. (a tag hierarchy) In other words, we can evaluate the suitability of folksonomies for decentralized search through simulations. Folksonomy Folksonomy Folksonomy 1 ... n shortest path found with A (tag-tag) network: local k l l knowledge pLK = 4 l d Goal: Navigate from START to TARGET Δ = pLK-pGK using local and background knowledge only candidates start target shortest path with p global knowledge pGK = 3 Markus Strohmaier 2011 J. Kleinberg. The small-world phenomenon: An algorithmic perspective. Proc. 32nd ACM Symposium on Theory of Computing, 2000. Also appears as Cornell Computer Science Technical Report 99-1776 (October 1999) 10
- 10. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 11
- 11. Knowledge Management Institute Pragmatic Evaluation Framework General idea: • Use the OUTPUT produced by folksonomy algorithms (hierachical structures) as INPUT (b k (hi hi l t t ) (background d knowledge) for decentralized search. Framework Instantiation K-means, Aff.Prop., 1. Generate n folksonomies DegCentrality, CloCentrality exploratory navigation 2. Model navigational task success rate, stretch 3. Select evaluation metrics decentralized search 4. Simulate navigation 4 Sim late na igation comparative evaluation 5. Evaluate performance Markus Strohmaier 2011 12
- 12. Knowledge Management Institute Simulating Exploratory Navigation Topically related START TARGET tags tags resources Topically related Random resources Random R d start resource Usefulness of: page: e.g. landing page from search engine We generate 100.000 search pairs (start, target) for each dataset, and Folksonomy F lk F lk Folksonomy Folksonomy F lk run simulations 1 ... n Markus Strohmaier 2011 13
- 13. Knowledge Management Institute Outline of this talk 1. Folksonomies Construction and E l ti C t ti d Evaluation 2. 2 Decentralized Search J. Kleinberg‘s algorithm 3. Pragmatic Evaluation Framework Presentation and discussion 4. Results & Findings Markus Strohmaier 2011 14
- 14. Knowledge Management Institute Success Rates Across Different Folksonomies flickr dataset Tag generality approaches k-means / affinity propagation Random folksonomy Success rate: The number of times an agent is successful in finding a path using a particular folksonomy as background knowledge All approaches outperform a random folksonomy y n max hops n: the maximal number of steps an agent Tag generality approaches is allowed to perform before stopping (a tunable outperform k-means / Aff. parameter e.g., an agent only f ll t t l follows n li k ) links). Propagation Markus Strohmaier 2011 16
- 15. Knowledge Management Institute Success Rates Across Different Datasets Holds for all But how datasets efficient are (to diff. diff those extents) folksonomies during search? Markus Strohmaier 2011 17
- 16. Knowledge Management Institute Stretch Δ = pLK-pGK p Shortest Paths found with Local Knowledge Bibsonomy K M Bib K-Means Finds no path: Δ = infinite Finds paths that is +1 longer: Δ=1 Holds for all datasets d t t Finds shortest possible path: Tag T generality lit (to diff. Δ=0 approaches (d+e) extents) find much shorter paths! Markus Strohmaier 2011 18
- 17. Knowledge Management Institute Pragmatic Evaluation Framework Framework Instantiation Alternatives K-means, Aff.Prop., other folksonomy 1. Generate n folksonomies DegCentrality, algorithms or CloCentrality expert taxonomies exploratory other tasks 2. Model navigational task navigation success rate, stretch other evaluation metrics 3. 3 Select evaluation metrics decentralized search actual click data 4. Simulate navigation comparative other evaluation 5. Evaluate performance evaluation approaches Pragmatic evaluation produces different results for different tasks and different assumed or observed navigation behavior. The evaluation framework is completely general with regard to the task, data and evaluation metrics adopted. Markus Strohmaier 2011 19
- 18. Knowledge Management Institute Results & Findings: Summary 1. Folksonomies are useful b k 1 F lk i f l background k d knowledge f l d for navigation. 2. Existing folksonomy algorithms are more useful than a random baseline. baseline 3. 3 Tag generality approaches perform better than existing hierarchical clustering approaches. 4. Pragmatic results support theoretical analysis (not presented in talk – included in paper). Markus Strohmaier 2011 20
- 19. Knowledge Management Institute Thank You. Th k Y Markus Strohmaier markus.strohmaier@tugraz.at D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. Lerman Pragmatic Evaluation of Folksonomies 20th International World Wide Web Conference (WWW2011) Hyderabad, India, March 28 - April 1, ACM, 2011. http://kmi.tugraz.at/staff/markus/documents/2011_WWW2011_Folksonomies.pdf Markus Strohmaier 2011 21